Webbot

May 9, 2005 8:11 AM

I need my own web bot.

I need an application that will go to a set of web sites that I specify, grab information based on either (1) whether there are changes to the site, or (2) by keyword search.

Would prefer something client based. I've Googled, I've donwload.com'd, but I can't find anything that does what I want it to do.

posted by benjh to computers & internet (7 comments total)

What exactly do you mean by "grab information"?? That's a pretty vague requirement.

What you want is probably achievable with Rss/Atom feeds for most sites.. but it would help if you are more clear about what you want.
posted by bruceyeah at 8:22 AM on May 9, 2005

bleh. I should actually re-read and edit on preview.
posted by bruceyeah at 8:25 AM on May 9, 2005

Well, I work for a company that sells just such an application (targeted at corporations for media monitoring purposes, although it also covers non-media very successfully). I won't spam this space with it though, but feel free to email me if you do want to pursue it. However, I'm guessing you want a free application. I've only seen a couple I'm afraid, and they've tended to be weak. You could check this out and see if it does enough for you: Watch That Page
posted by bifter at 8:34 AM on May 9, 2005

If you or an associate is fluent in perl, the LWP modules are what you want to use. You can write your own bot in code short enough to print out on a single 8-1/2 by 11 sheet of paper.
posted by gimonca at 9:57 AM on May 9, 2005

Well, you could take a look around for software that does this kind of thing, and hopes it matches your needs and grabs what information you require.

However, if you are serious about your bot and want to have (theoretically) infinite customizability, you could do worse than reading Searchlore's Bots lore page. Might seem a little disorganised, but it should get you on your way to building your own bots in perl, Java, or PHP (or conceptually any other language you're proficient in).

It's not for the computer newbie, but great, interesting info for those who like to go a little deeper.
posted by splice at 10:02 AM on May 9, 2005

What does client based mean? Anyway, isn't this what they call "screen scraping"? Maybe that's the magic phrase to help your searching.
posted by stuart_s at 2:26 PM on May 9, 2005

You'll find that trackengine.com does exactly what you want. If will monitor pages for changes and alerts you. And if you specify a keyword, it will alert you when that keyword appears. It does not have RSS feeds, but it informs you by email.
posted by timyang at 3:57 AM on May 10, 2005