Managing multiple, daily Google searches
April 5, 2014 6:22 PM   Subscribe

Let's say there are a whole bunch of websites and I want to know when they use a certain phrase. Each day, I need to check and see if certain phrases appear on the website. Right now, I have a whole bunch of Google searches saved and I check each one in turn. Surely there is an easier way!

Okay, so I need to monitor a wide variety of news sources to see if they mention certain key phrases. Right now this is done by checking a bookmarked search each day. That is, there are x sites that I need to search, so I have x bookmarks that take me directly to the search results for those websites to see if the key words have popped up. Every day, I visit every one of those bookmarked search results to see if the phrase has appeared.

An additional wrinkle: different sites may be targeted for different words.

This does the job, but surely this can be made more efficient. Google Alerts, maybe? But I am hoping to avoid having to create fifty different Alerts and then wade through it all in email. (There are typically a lot of false positives in the search.) I think it used to be that I could channel the Google Alerts into Reader, but now that Reader no longer exists, that's probably not an option.
posted by synecdoche to Computers & Internet (8 answers total) 3 users marked this as a favorite
I use Feedly for this but I'm not convinced that it works terribly well. I'm also not convinced that Google Alerts work terribly well either. Both have missed things (important things, like a Wall Street Journal Article) and both have significant false positives.
posted by sciencegeek at 6:31 PM on April 5, 2014

Have you seen If This Then That? You might be able to create a recipe with your Google search results and have it combine them and alert you via email or SMS.
posted by expialidocious at 7:08 PM on April 5, 2014

wget and grep

It's built in to most unixes and very script-able, or try that will run as a service/recipe on their host. Or try any scraping program, setting ignore images will speed things. Another one called netcat might work. Unless the sites are wikipedia huge scraping directly yourself may be the most reliable. Be sure to set a limiter to slow the scraping a bit or it may trigger some filters.
posted by sammyo at 7:09 PM on April 5, 2014

What I do: set up a site-specific Google Alert query (e.g., "unique phrase"). Then, click "Manage your alerts" and deliver to Feed. Finally, paste that feed in a reader that works for you.
posted by glibhamdreck at 7:33 PM on April 5, 2014 [1 favorite]

So there have been similar commercial services for years. They used to be called 'clipping services' and now they're 'media monitoring services'. Everyone at the company I work for gets the big filter daily, if there are relevant contents.

I assume you don't need a professional clipping service. I can't point you to any specific media monitoring software that I know anything about, but I can say that there are at least some commercial offerings in this arena. Depending on budget, scope, and the value of your time, it might be worth investigating outsourcing or using someone's software solution.
posted by Mad_Carew at 9:41 PM on April 5, 2014

It sounds like the advanced filtering capabilities of Femtoo might work:

"...setting 'content filters' to monitor keywords and phrases pertinent to your organisation"

Or, on MyWebChecks is says:

"You can define words that you are looking for".

I don't think it's entirely clear on either site whether you can do what you want to do, but it sounds like you possibly can and you'd need to contact them to find out for sure.
posted by Dansaman at 9:58 PM on April 5, 2014

I think Google Alerts is what you want for this. Unless you use a site specific search as suggested above, you're going to end up with false positives no matter what service you use.

If you don't want to be overwhelmed with false positives in your regular email you could just set up a new email address just for this, then check it once a day.
posted by Jacqueline at 1:36 AM on April 6, 2014

Meltwater is the cheapest commerical "media monitoring service" I am aware of, as in an order of magnitude cheaper than competing (and I gather superior) services, and even that I believe was costing around 15k a year. I'd go with site specific Google Alerts.
posted by kaspen at 11:10 AM on April 7, 2014

« Older What genre of music is this and how can I find...   |   Kanji wear it, redux Newer »
This thread is closed to new comments.