Let's make this more efficient

February 3, 2012 9:41 PM

There are 4-5 hobbyist forums I frequent. How do I search all of them at once?

I kind of answered my own question just before I posted because I found Google Custom Search. GCS is OK, but it doesn't show key information like when each thread was posted. It also tends to list a bunch of results from the same url in a big clump. For example, the 1st url it searches takes up the first 1.5 pages, then the 2nd url takes up some more, etc.

Is there a better tool to do this? I imagine there must be a browser extension out there. What would be really great is if I could type a keyword into one search-bar, and have 5 tabs open up that are the search results page ON each site.

I have a little C + Java experience and am not averse to solutions that are a little more technical.

Thanks

posted by victory_laser to computers & internet (5 answers total)

If you have some C experience, perhaps you could use libcurl or similar to make an HTTP request for each site's page or pages, and then use libxml2 to parse the HTML into a data structure of your design, something you can search by keyword, for example.
posted by Blazecock Pileon at 10:58 PM on February 3, 2012

Using Firefox for example you could create a bookmark with an assigned keyword that points to a bit of Javascript code (a bookmarklet) which searches multiple URLs. The following code for example would search both Google and Altavista for a given search term and open the results in separate tabs.

javascript:void(window.open('http://www.google.com/search?q=%s'));
void(window.open('http://www.altavista.com/web/results?q=%s'))

Typing 'mysearch bananas' into the Firefox web address box would open two tabs to Google and Altavista with the results of the search for bananas displayed. In this example I have assigned the keyword 'mysearch' to the bookmark.
posted by wannalol at 6:49 AM on February 4, 2012 [1 favorite]

FDSE lets you customize it a lot, plus it's in Perl. You install it on your own server and index whatever pages you want. You can search several "realms" (usually domains or subdomains) simultaneously.

Here is a search for "vegetarian" over several domains. I set this up - this is kind of a self-link (I set up the search engine for this guy), but it's the best example I know of.

It's really easy to set up, too. It's been a few years since I really tweaked it, but it's been working really well all that time.
posted by amtho at 9:58 AM on February 4, 2012

Also (sorry): the domains _do not_ have to be on the same server as the search engine; in the example above, they aren't.
posted by amtho at 10:01 AM on February 4, 2012

Do the forums have RSS feeds? I put all the forums that I read into Google Reader and search there. I also use Yahoo Pipes to filter out things of interest into their own custom feed.
posted by meowzilla at 12:47 PM on February 4, 2012