vacation compatible RSS: how can I cache one month of feed items?
January 16, 2008 11:58 AM Subscribe
Say that I will go travelling for one month without internet access. For my email that poses no problem. All incoming mails wait for me in gmail when I get back. But how can I similarly avoid missing content in the many RSS feeds I regularly read? Can some online service cache all the feeds for me?
Some of the RSS feeds contain only a few new items every month so for those feeds I will get all those items when I get back. But many other feeds have a much higher throughput of items. Some have 100 or more items every day and the feed itself only includes the latest 50 or so items. For those feeds I risk missing out on almost 3000 items!
How can I remedy that? Is there some neat web service that allows me to cache all items in some feeds for a period of time? Or at least one that continually and automatically lets me filter some feeds for certain items and cache all matches? Is there even a site similar to archive.org for RSS?
What I've looked into already:
I could set up a server at home, run some RSS reader in it and write a script or change some setting so that the reader checks the feeds for new items several times every day. But I have no home server at the moment so it seems both like much work and as overkill as far as solutions go. I hope there is some alternative online solution.
I've found several online services that combine or mashup several feeds into one. Yahoo pipes, http://pipes.yahoo.com/pipes/ for example. But I haven't found one that cache one month of RSS feed items, mashed up or not.
There are several RSS to email services (see http://ask.metafilter.com/70159/RSS2Email-Headaches ). But I haven't found one that allow conversion back to RSS later on. But maybe I just haven't searched enough? If there is some solution that reliably can do these steps then tell me about it: RSS >>(through some online service)>> email >>> gmail account >>(one month later, through some online service)>> RSS >>> locally installed RSS-reader. I definitely want the items back into my regular feed reader since I very often do searches on the locally downloaded feeds (I've set the reader to not remove old downloaded feed items) and I want to avoid having to do those searches at several places (feed reader and gmail) if possible.
I would most of all prefer some service that simply takes an OPML as input, cache all items in those feeds and then let me grab them easily later.
Some of the RSS feeds contain only a few new items every month so for those feeds I will get all those items when I get back. But many other feeds have a much higher throughput of items. Some have 100 or more items every day and the feed itself only includes the latest 50 or so items. For those feeds I risk missing out on almost 3000 items!
How can I remedy that? Is there some neat web service that allows me to cache all items in some feeds for a period of time? Or at least one that continually and automatically lets me filter some feeds for certain items and cache all matches? Is there even a site similar to archive.org for RSS?
What I've looked into already:
I could set up a server at home, run some RSS reader in it and write a script or change some setting so that the reader checks the feeds for new items several times every day. But I have no home server at the moment so it seems both like much work and as overkill as far as solutions go. I hope there is some alternative online solution.
I've found several online services that combine or mashup several feeds into one. Yahoo pipes, http://pipes.yahoo.com/pipes/ for example. But I haven't found one that cache one month of RSS feed items, mashed up or not.
There are several RSS to email services (see http://ask.metafilter.com/70159/RSS2Email-Headaches ). But I haven't found one that allow conversion back to RSS later on. But maybe I just haven't searched enough? If there is some solution that reliably can do these steps then tell me about it: RSS >>(through some online service)>> email >>> gmail account >>(one month later, through some online service)>> RSS >>> locally installed RSS-reader. I definitely want the items back into my regular feed reader since I very often do searches on the locally downloaded feeds (I've set the reader to not remove old downloaded feed items) and I want to avoid having to do those searches at several places (feed reader and gmail) if possible.
I would most of all prefer some service that simply takes an OPML as input, cache all items in those feeds and then let me grab them easily later.
Try Google Reader? I haven't noticed any feed limits - it seems my unread stuff in certain feeds I ignore keeps piling up into the hundreds if I let it.
posted by mikepop at 12:23 PM on January 16, 2008
posted by mikepop at 12:23 PM on January 16, 2008
I've neglected my google reader for weeks and it tells me I have thousands of unread posts. They don't go anywhere.
posted by desjardins at 12:23 PM on January 16, 2008
posted by desjardins at 12:23 PM on January 16, 2008
I agree, Google Reader keeps RSS items forever, even if they're not in the active feed anymore. They had to do this to enable searching and all the fancy googleness.
posted by blue_beetle at 12:30 PM on January 16, 2008
posted by blue_beetle at 12:30 PM on January 16, 2008
Google Reader keeps everything, but I think the OP wants the feed reader to keep a cached copy where the original might not stay online.
Take PostSecret, for example. The site only keeps the most recent post up - so in Google Reader, the old posts are there, but the pictures are gone.
posted by puddleglum at 12:35 PM on January 16, 2008
Take PostSecret, for example. The site only keeps the most recent post up - so in Google Reader, the old posts are there, but the pictures are gone.
posted by puddleglum at 12:35 PM on January 16, 2008
How important are these feeds? Important enough that you'll spend an entire day or two on your return reading stuff from three weeks ago? Seriously?
For those feeds I risk missing out on almost 3000 items!
It's important to note that "missing out" is a relative term. But, to answer your question, Google Reader doesn't appear to have limits.
posted by pdb at 12:39 PM on January 16, 2008
For those feeds I risk missing out on almost 3000 items!
It's important to note that "missing out" is a relative term. But, to answer your question, Google Reader doesn't appear to have limits.
posted by pdb at 12:39 PM on January 16, 2008
I remember seeing a chron job that would do this in an earlier askme, where the person was looking to download blog posts en toto overnight. I'll look around, but maybe someone else remembers how to set that up.
posted by klangklangston at 12:53 PM on January 16, 2008
posted by klangklangston at 12:53 PM on January 16, 2008
Google Reader definitely does this, and I think that the Newsgator readers (NetNewsWire, FeedDemon and Newsgator) should do this when used with Newsgator online. The unread items will build up on the server and you'll get all of them downloaded to your client when you get home.
posted by andrewraff at 1:36 PM on January 16, 2008
posted by andrewraff at 1:36 PM on January 16, 2008
Why do you need it converted back to RSS? Couldn't you just read the items in whatever feed reader that you pick?
BTW, I'm the maintainer of the rss2email project. If you're interested in using it while you're away, you can MeMail if you need help.
posted by turbodog at 1:56 PM on January 16, 2008
BTW, I'm the maintainer of the rss2email project. If you're interested in using it while you're away, you can MeMail if you need help.
posted by turbodog at 1:56 PM on January 16, 2008
google reader will do this.
posted by misanthropicsarah at 2:53 PM on January 16, 2008
posted by misanthropicsarah at 2:53 PM on January 16, 2008
pdb touched on this. I doubt you'll be interested in reading a month's worth of feeds on your return. These sorts of things that seem so important beforehand turn out to not be afterwards. I've been there before, trying to play the catch-up game with online content and found it wasn't really worth my time. You won't miss anything earth-shattering. Just jump back into reading them when you return.
posted by 6550 at 5:41 PM on January 16, 2008
posted by 6550 at 5:41 PM on January 16, 2008
I've missed months worth of RSS feeds. Seriously, not worth it - in fact, the ones with 3000+ entries tend to be rather useless. I end up having to mark EVERYTHING read because it's so overwhelming. Besides, a lot of posts tend to be repeated anyway.
Make a bookmark list of blogs you follow religiously (for me that would be the blogs of my sister and best friends) and visit those when you get back.
posted by divabat at 4:45 AM on January 17, 2008
Make a bookmark list of blogs you follow religiously (for me that would be the blogs of my sister and best friends) and visit those when you get back.
posted by divabat at 4:45 AM on January 17, 2008
The beta version of Bloglines has removed the 200-item limit per feed.
posted by lukemeister at 5:48 AM on January 17, 2008
posted by lukemeister at 5:48 AM on January 17, 2008
Response by poster: Thanks for the replies so far people!
various people: regarding Google reader: thanks, I'll see if it works for my purposes. I testdrove it quite some time ago and my recollection was that it only downloaded feeds at the time of my login, but I guess I was wrong on that.
Is there some neat way to get the cached items in Google Reader to my local RSS reader (GreatNews)? (In general, is there some standard format and some tools for importing/exporting and backing up downloaded feed items?)
puddleglum: No, caching of text (not images) would really be enough for my needs.
pdb: the feeds concern various subjects but the most important ones are from a large number of academic journals and so on and contain a table of contents and/or abstracts for articles. Most of them are not updated that often and so does not pose a problem in this regard, but many others do. Since the feeds are from various disciplines I find searching all downloaded RSS items much faster than going to several online abstract search engines. But that's only reliable if I can know that the downloaded items are somewhat complete (for the last year or so. For older articles I use the regular academic search engines). I also rely on a number of "watches" in my RSS reader that gives me immediate access to sets of items that I would otherwise have to do searches for.
turbodog: I wrote unclearly when I said that I want them back to RSS. What I really want is to import them into my RSS reader of choice (GreatNews) as single items. I don't want to keep them as single email messages or as Google reader items. But I have really no idea what format GreatNews (or other local readers) store the items in. Anyway, if the Google reader solution does not work out for me I might go for rss2email and might then take you up on the kind offer of help.
pdb, 6550 & divabat: I should have explained that that for feeds with very high throughput I tend to rely heavily on filtering. So from say 3000 monthly posts only around 10 would survive the Yahoo Pipes filter I pull it through. Those 10 tend to be useful posts that I do not want to miss. So in one sense I don't miss out on all the 3000 but in another I do since I need them to get the 10. I can't grab the 10 at the end of the month because the feed only contains the latest 50 or so items, but the 10 are spread out throughout the 3000 posts.
posted by nolnar at 9:24 AM on January 17, 2008
various people: regarding Google reader: thanks, I'll see if it works for my purposes. I testdrove it quite some time ago and my recollection was that it only downloaded feeds at the time of my login, but I guess I was wrong on that.
Is there some neat way to get the cached items in Google Reader to my local RSS reader (GreatNews)? (In general, is there some standard format and some tools for importing/exporting and backing up downloaded feed items?)
puddleglum: No, caching of text (not images) would really be enough for my needs.
pdb: the feeds concern various subjects but the most important ones are from a large number of academic journals and so on and contain a table of contents and/or abstracts for articles. Most of them are not updated that often and so does not pose a problem in this regard, but many others do. Since the feeds are from various disciplines I find searching all downloaded RSS items much faster than going to several online abstract search engines. But that's only reliable if I can know that the downloaded items are somewhat complete (for the last year or so. For older articles I use the regular academic search engines). I also rely on a number of "watches" in my RSS reader that gives me immediate access to sets of items that I would otherwise have to do searches for.
turbodog: I wrote unclearly when I said that I want them back to RSS. What I really want is to import them into my RSS reader of choice (GreatNews) as single items. I don't want to keep them as single email messages or as Google reader items. But I have really no idea what format GreatNews (or other local readers) store the items in. Anyway, if the Google reader solution does not work out for me I might go for rss2email and might then take you up on the kind offer of help.
pdb, 6550 & divabat: I should have explained that that for feeds with very high throughput I tend to rely heavily on filtering. So from say 3000 monthly posts only around 10 would survive the Yahoo Pipes filter I pull it through. Those 10 tend to be useful posts that I do not want to miss. So in one sense I don't miss out on all the 3000 but in another I do since I need them to get the 10. I can't grab the 10 at the end of the month because the feed only contains the latest 50 or so items, but the 10 are spread out throughout the 3000 posts.
posted by nolnar at 9:24 AM on January 17, 2008
This thread is closed to new comments.
posted by zeoslap at 12:22 PM on January 16, 2008