How can I get automate my daily news searches?

April 28, 2008 12:28 PM

How can I automate the news searches that I'm performing on the same subjects day after day after day?

I'm working for an employer who needs current information on what the press is saying each day. It would be great if it were an issue of what the big national papers were saying and I could simply use Google News, but I need information from our state and local papers, including those that have only a weekly distribution cycle.

Right now, I'm forced to spend time searching through ten websites a day, often to find no new information. It seems like there should be a way to automate this process so that new information pops up in my email box or RSS reader and I can see what's new right away.

So far, I've been considering applications like Dapper, Yahoo Pipes, Google Alerts, or Feed43. I can't simply subscribe to RSS feeds from the sites directly, since most of them are written in 1998-style code and don't have RSS feeds or tagging of articles.

Google Alerts seems to be out of the running since it often pops up with articles that are months old and misses the local papers.

Dapper seems like it could be great, except that I'm having some trouble tweaking it so that it doesn't send me an update just because a banner ad changes. I think I could get it to work, but I could use a tutorial.

Yahoo Pipes seems like it could be very powerful, but it's a bit intimidating.

Feed43 I've heard good things about, but it doesn't seem to do much more than turn pages into RSS feeds and this doesn't help me with the search.

If you have any tips for using the above services or could point me in the direction of a better solution, I'd really appreciate it. I'd bet someone else out there has had to work through a similar situation at some point. Thanks in advance!

posted by abkadefgee to computers & internet (9 answers total)

You can restrict the publication date for the articles that a Google News Search returns in 'advanced news search', and save that to a daily email or feed.
posted by mkb at 12:54 PM on April 28, 2008

What if you set up a Google Alert that performed a regular search restricted to the sites you regularly check (i.e. "searchterm site:paper1.com OR site:paper2.com")?
posted by designbot at 12:59 PM on April 28, 2008

Can you subscribe to something like Factiva?
posted by laukf at 1:42 PM on April 28, 2008

thepaperboy.com is a kind of portal for lots of smaller news sites. They might have stuff Google News doesn't. The search looks kind of weak but maybe you can combine it with some other resource?
posted by Brian James at 2:00 PM on April 28, 2008

You can create a personalized news section in Google News that shows you matches for set of words, or matches for a particular zip code. The word matching is a bit wonky - I don't think they support the not and the or operators (at least, not well), but it's otherwise pretty good.
posted by zippy at 3:31 PM on April 28, 2008

While all of these are good suggestions, they don't quite get at what I need. Many of the podunk local papers don't seem to be found in Google News results. Most of the suggestions also don't seem to automate the process, but rather seem to suggest where I can find better search engines. Is there anyone who can help me get closer to the automation or RSS style reading of the topics?

Don't get me wrong, I really appreciate you guys trying to help, but it's not quite the type of answer I'm looking for.
posted by abkadefgee at 9:06 PM on April 28, 2008

Ok, maybe I was wrong. I'm giving the suggestions a second look and some of them are better than I thought. Sorry to judge so quickly. I'm just still stuck at work, trying to figure things out, and more than a little cranky. Why can't internet just fix everything now!?
posted by abkadefgee at 9:21 PM on April 28, 2008

Another option is google co-op. You say you're going to the same 10 sites. Plug in the sites you want to search, and presto.
posted by cashman at 2:15 PM on April 29, 2008

Thanks everyone for your suggestions, and if you have any more, keep them coming.

For the reference of anyone who is asking the same question in the future, here's what I did:

1. Did a more in depth search for RSS feeds on some sites and found them hidden away
2. Used Yahoo Pipes to combine feeds
a. Placed all the URL's into "Fetch Feed"
b. Sent the pipe from "Fetch Feed" to a duplicate check with the uniq operator, checking by title
c. Sent this pipe through a filter that only allowed items with my search terms
d. Output this info into a new feed that would update me with new articles
3. Combined the rest of the sites I could find that did not have feeds into the Google Co-op custom search as recommended by cashman.

I'd love to just send you directly to the pipe and custom search I created, but I think that if I did, I'd violate an NDA. Maybe when this job is over, I'll update with links.
posted by abkadefgee at 11:06 AM on May 1, 2008 [1 favorite]