Let's make this more efficient
February 3, 2012 9:41 PM   Subscribe

There are 4-5 hobbyist forums I frequent. How do I search all of them at once?

I kind of answered my own question just before I posted because I found Google Custom Search. GCS is OK, but it doesn't show key information like when each thread was posted. It also tends to list a bunch of results from the same url in a big clump. For example, the 1st url it searches takes up the first 1.5 pages, then the 2nd url takes up some more, etc.

Is there a better tool to do this? I imagine there must be a browser extension out there. What would be really great is if I could type a keyword into one search-bar, and have 5 tabs open up that are the search results page ON each site.

I have a little C + Java experience and am not averse to solutions that are a little more technical.

posted by victory_laser to Computers & Internet (5 answers total) 3 users marked this as a favorite
If you have some C experience, perhaps you could use libcurl or similar to make an HTTP request for each site's page or pages, and then use libxml2 to parse the HTML into a data structure of your design, something you can search by keyword, for example.
posted by Blazecock Pileon at 10:58 PM on February 3, 2012

Using Firefox for example you could create a bookmark with an assigned keyword that points to a bit of Javascript code (a bookmarklet) which searches multiple URLs. The following code for example would search both Google and Altavista for a given search term and open the results in separate tabs.


Typing 'mysearch bananas' into the Firefox web address box would open two tabs to Google and Altavista with the results of the search for bananas displayed. In this example I have assigned the keyword 'mysearch' to the bookmark.
posted by wannalol at 6:49 AM on February 4, 2012 [1 favorite]

FDSE lets you customize it a lot, plus it's in Perl. You install it on your own server and index whatever pages you want. You can search several "realms" (usually domains or subdomains) simultaneously.

Here is a search for "vegetarian" over several domains. I set this up - this is kind of a self-link (I set up the search engine for this guy), but it's the best example I know of.

It's really easy to set up, too. It's been a few years since I really tweaked it, but it's been working really well all that time.
posted by amtho at 9:58 AM on February 4, 2012

Also (sorry): the domains _do not_ have to be on the same server as the search engine; in the example above, they aren't.
posted by amtho at 10:01 AM on February 4, 2012

Do the forums have RSS feeds? I put all the forums that I read into Google Reader and search there. I also use Yahoo Pipes to filter out things of interest into their own custom feed.
posted by meowzilla at 12:47 PM on February 4, 2012

« Older Please lend me your cover letter expertise!   |   Why do I have frequent urination? Newer »
This thread is closed to new comments.