What's the simplest way for me to grab a daily digest of my twitter output?

October 29, 2008 11:10 AM

What's the simplest way for me to grab a daily digest of my twitter output?

I'm planning to do a bunch of writing via twitter next month, and I'd like to be able to easily grab my output on a regular output -- once a day, probably.

I'm aware of TweetDumpr but had no luck with a test run (other reservations aside). The idea -- export data as CSV -- is more or less exactly what I'd want, though. Being able to deal with the data in a parsed-out manner is definitely key.

Are there other existing solutions that would make this easy to do? Has anyone experimented with this?

I have heard that the twitter API is pretty approachable, but I've never dug into it; if the best way to get a clean dump is to go that route with some perl, I can do so and would love pointers on how to get going on that.

posted by cortex to computers & internet (6 comments total)

Check this out - especially the since option.
posted by sanko at 11:27 AM on October 29, 2008

Use LoudTwitter and one of its support blog platforms.
posted by mkb at 11:27 AM on October 29, 2008

I don't know of any off-the-shelf tools, but the API is approachable as hell, yeah. Basically it's just a bit of javascript. If you look at the plain html widget for your websites, you see there's some pretty simple javascript there for you to dig into, should you want to roll your own. This should let you output to an xml file that you could easily dump into a db -- all you have to do is look at the script sources at those locations.

I did that for my website -- the plain html widget doesn't add links automagically if there's a url in the text, so I grabbed the twitterCallback2 function, added a little regexp, and booya.
posted by middleclasstool at 11:32 AM on October 29, 2008

Oh yeah, looking at sanko's link, you can even have it directly output to xml for you. Should be a piece of pie to do.
posted by middleclasstool at 11:34 AM on October 29, 2008

mkb, that's not a bad out-of-box solution, but I'm probably going to want more flexibility than it allows in this case.

sanko, I think that might be right on the money for me; I think I might try and just set up an hourly "since"-delimited snag on my feed using that. It's not exactly inconcievable that I'll blow past the 200 item limit of the usertimeline function on any given day, so rolling updates is probably my best bet.

mct, I'll definitely use that as a starting point for getting my bearings.

Thanks all three of you; other ideas are welcome, too. I've got a couple days to get this stuff set up, so I'll be keeping my eyes open.
posted by cortex at 12:33 PM on October 29, 2008

Seems like it should be dead simple to convert your twitter RSS feed to CSV or SQL. But, surprisingly, I'm not seeing any existing script on google that's both free and not-java. Here's a RSS2JSON perl script, though.
posted by and hosted from Uranus at 6:51 AM on October 30, 2008