Quick way to query multiple search forms?

July 26, 2013 9:23 AM

What's the easiest way to create an automated search of custom search engines? For example, suppose I wanted to create a search form that could take a query, run it through the respective search engines of three different newspaper websites, and then return the aggregated results. What's the very simplest way of doing this?
posted by shivohum to Computers & Internet (3 answers total)

You'll need something to make the HTTP request, something to parse out the relevant information from the resulting HTML document, and something to output those results in your desired format.

Here's an example using python's urllib and BeautifulSoup: Scraping websites with Python
posted by RonButNotStupid at 9:35 AM on July 26, 2013

Alternative: use google site queries. So if you want to search Washington Post, New York Times, and Los Angeles Times for things having to do with cupcakes, you would use the search:

(site:washingtonpost.com OR site:nytimes.com OR site:latimes.com) cupcakes
posted by jraenar at 9:54 AM on July 26, 2013

Response by poster: Here's an example using python's urllib and BeautifulSoup: Scraping websites with Python

Thanks. Also, I should add that I wouldn't mind, if it would make it much easier, just to have a form that automatically opens up search queries in separate windows for each website, or else in separate frames in the same window. That might save me the parsing, no?

Alternative: use google site queries.

Unfortunately this doesn't work for me, because some of the sites I want to search return dynamic results, like online merchants.
posted by shivohum at 9:56 AM on July 26, 2013