Managing multiple, daily Google searches
April 5, 2014 6:22 PM Subscribe
Let's say there are a whole bunch of websites and I want to know when they use a certain phrase. Each day, I need to check and see if certain phrases appear on the website. Right now, I have a whole bunch of Google searches saved and I check each one in turn. Surely there is an easier way!
Okay, so I need to monitor a wide variety of news sources to see if they mention certain key phrases. Right now this is done by checking a bookmarked search each day. That is, there are x sites that I need to search, so I have x bookmarks that take me directly to the search results for those websites to see if the key words have popped up. Every day, I visit every one of those bookmarked search results to see if the phrase has appeared.
An additional wrinkle: different sites may be targeted for different words.
This does the job, but surely this can be made more efficient. Google Alerts, maybe? But I am hoping to avoid having to create fifty different Alerts and then wade through it all in email. (There are typically a lot of false positives in the search.) I think it used to be that I could channel the Google Alerts into Reader, but now that Reader no longer exists, that's probably not an option.
Okay, so I need to monitor a wide variety of news sources to see if they mention certain key phrases. Right now this is done by checking a bookmarked search each day. That is, there are x sites that I need to search, so I have x bookmarks that take me directly to the search results for those websites to see if the key words have popped up. Every day, I visit every one of those bookmarked search results to see if the phrase has appeared.
An additional wrinkle: different sites may be targeted for different words.
This does the job, but surely this can be made more efficient. Google Alerts, maybe? But I am hoping to avoid having to create fifty different Alerts and then wade through it all in email. (There are typically a lot of false positives in the search.) I think it used to be that I could channel the Google Alerts into Reader, but now that Reader no longer exists, that's probably not an option.
Have you seen If This Then That? You might be able to create a recipe with your Google search results and have it combine them and alert you via email or SMS.
posted by expialidocious at 7:08 PM on April 5, 2014
posted by expialidocious at 7:08 PM on April 5, 2014
wget and grep
It's built in to most unixes and very script-able, or try https://ifttt.com/ that will run as a service/recipe on their host. Or try any scraping program, setting ignore images will speed things. Another one called netcat might work. Unless the sites are wikipedia huge scraping directly yourself may be the most reliable. Be sure to set a limiter to slow the scraping a bit or it may trigger some filters.
posted by sammyo at 7:09 PM on April 5, 2014
It's built in to most unixes and very script-able, or try https://ifttt.com/ that will run as a service/recipe on their host. Or try any scraping program, setting ignore images will speed things. Another one called netcat might work. Unless the sites are wikipedia huge scraping directly yourself may be the most reliable. Be sure to set a limiter to slow the scraping a bit or it may trigger some filters.
posted by sammyo at 7:09 PM on April 5, 2014
What I do: set up a site-specific Google Alert query (e.g., site:online.wsj.com "unique phrase"). Then, click "Manage your alerts" and deliver to Feed. Finally, paste that feed in a reader that works for you.
posted by glibhamdreck at 7:33 PM on April 5, 2014 [1 favorite]
posted by glibhamdreck at 7:33 PM on April 5, 2014 [1 favorite]
So there have been similar commercial services for years. They used to be called 'clipping services' and now they're 'media monitoring services'. Everyone at the company I work for gets the big filter daily, if there are relevant contents.
I assume you don't need a professional clipping service. I can't point you to any specific media monitoring software that I know anything about, but I can say that there are at least some commercial offerings in this arena. Depending on budget, scope, and the value of your time, it might be worth investigating outsourcing or using someone's software solution.
posted by Mad_Carew at 9:41 PM on April 5, 2014
I assume you don't need a professional clipping service. I can't point you to any specific media monitoring software that I know anything about, but I can say that there are at least some commercial offerings in this arena. Depending on budget, scope, and the value of your time, it might be worth investigating outsourcing or using someone's software solution.
posted by Mad_Carew at 9:41 PM on April 5, 2014
It sounds like the advanced filtering capabilities of Femtoo might work:
"...setting 'content filters' to monitor keywords and phrases pertinent to your organisation"
Or, on MyWebChecks is says:
"You can define words that you are looking for".
I don't think it's entirely clear on either site whether you can do what you want to do, but it sounds like you possibly can and you'd need to contact them to find out for sure.
posted by Dansaman at 9:58 PM on April 5, 2014
"...setting 'content filters' to monitor keywords and phrases pertinent to your organisation"
Or, on MyWebChecks is says:
"You can define words that you are looking for".
I don't think it's entirely clear on either site whether you can do what you want to do, but it sounds like you possibly can and you'd need to contact them to find out for sure.
posted by Dansaman at 9:58 PM on April 5, 2014
I think Google Alerts is what you want for this. Unless you use a site specific search as suggested above, you're going to end up with false positives no matter what service you use.
If you don't want to be overwhelmed with false positives in your regular email you could just set up a new email address just for this, then check it once a day.
posted by Jacqueline at 1:36 AM on April 6, 2014
If you don't want to be overwhelmed with false positives in your regular email you could just set up a new email address just for this, then check it once a day.
posted by Jacqueline at 1:36 AM on April 6, 2014
Meltwater is the cheapest commerical "media monitoring service" I am aware of, as in an order of magnitude cheaper than competing (and I gather superior) services, and even that I believe was costing around 15k a year. I'd go with site specific Google Alerts.
posted by kaspen at 11:10 AM on April 7, 2014
posted by kaspen at 11:10 AM on April 7, 2014
This thread is closed to new comments.
posted by sciencegeek at 6:31 PM on April 5, 2014