Daily digest for passive research on multiple topics...
October 8, 2013 12:58 PM Subscribe
I want to track a variety of topics passively using services on the interwebs. Google Alerts, RSS Feeds, and IFTTT are not solving the problem (or rather, I am not sufficiently solving the problem with them). Help me use available services to get to this promised land.
posted by milqman to computers & internet (6 answers total) 5 users marked this as a favorite
I am trying to put together a (probably rube goldberg-ian) system that spots mentions of certain terms in certain publications, bundles up those publications (blog posts, for example), and sends them to me daily.
For example, let's say I was interested in what was happening with gold markets (this is not the case, but it is parallel). I would like to receive an email at the end of the day containing any posts (links to posts would be fine) among say 4 or so specified commodity investing blogs that mentioned "gold" that day.
Bonus difficulty: I want to be able to add terms to the list with relative ease. So maybe after a week of getting this daily digest, I also want to include "silver" and "frozen orange juice concentrate" but without a huge rigmarole of re-setup. By the way, I don't care if I get a post twice because it mentioned both gold and silver.
In Unix, this would be a cron job where you would sort of
for i in TOPICLIST grep i (diff TODAYSFEED YESTERDAYSFEED) >> DAILYDIGEST
-Google Alerts does a great job finding the term in the entire internet and emailing it
-IFTTT does a great job finding a specific term in a single feed and doing -something- but it cannot take in programmatic inputs for this search (or trigger off of a spreadsheet)
-RSS is good at defining the feeds to search (though, I don't know how to combine multiple channels in to a single RSS feed, that would be helpful). Maybe other RSS tools can do searching/parsing?
-I do not know of a batching/caching service that will collect data over the course of a day and deliver it in one "box"
In an ideal world I would have (I think):
-A master RSS that was the cumulative feed of my 4 or so sources
-An IFTTT recipe that triggered when the feed matched entries in
---A google spreadsheet that contained my terms
-That recorded the content of the posts -somewhere-
-That would be emailed to me at 5pm every day
The scenario I have now is a team of IFTTT recipes that trigger by individual feed and by individual term that email me when they occur (so setup for new terms is a pain and I get a barrage of emails instead of one super email).
The three hurdles are (I think):
-Having a super feed of the 4 or so feeds (maybe this is easy... it should be but maybe I am dumb/ignorant)
-Having the topic list exist on its own such that the process refers to it independently (as opposed to being baked in to a recipe or google alert, for example)
-Pooling up the matches to send a single digest email daily
Any solutions to any of these problems, new services to investigate, different ways to think about the problem, or things I may have overlooked would be very much appreciated!