Quick way to query multiple search forms?
July 26, 2013 9:23 AM   Subscribe

What's the easiest way to create an automated search of custom search engines? For example, suppose I wanted to create a search form that could take a query, run it through the respective search engines of three different newspaper websites, and then return the aggregated results. What's the very simplest way of doing this?
posted by shivohum to Computers & Internet (3 answers total) 1 user marked this as a favorite
 
You'll need something to make the HTTP request, something to parse out the relevant information from the resulting HTML document, and something to output those results in your desired format.

Here's an example using python's urllib and BeautifulSoup: Scraping websites with Python
posted by RonButNotStupid at 9:35 AM on July 26, 2013


Alternative: use google site queries. So if you want to search Washington Post, New York Times, and Los Angeles Times for things having to do with cupcakes, you would use the search:

(site:washingtonpost.com OR site:nytimes.com OR site:latimes.com) cupcakes
posted by jraenar at 9:54 AM on July 26, 2013


Here's an example using python's urllib and BeautifulSoup: Scraping websites with Python

Thanks. Also, I should add that I wouldn't mind, if it would make it much easier, just to have a form that automatically opens up search queries in separate windows for each website, or else in separate frames in the same window. That might save me the parsing, no?

Alternative: use google site queries.

Unfortunately this doesn't work for me, because some of the sites I want to search return dynamic results, like online merchants.
posted by shivohum at 9:56 AM on July 26, 2013


« Older Could a cigarette lighter USB charger drain my...   |   Best 'responsible' used car for less than $10k? Newer »
This thread is closed to new comments.