What's the best way to data mine Twitter?

May 8, 2011 6:30 PM

I'm looking to start a prototype research project to look for trends in Twitter data. I've had a look through the API docs and with a lot of sharpening my python chops (?) I think it does what I want. But: does the API give me access to every query possible and I just have to learn how to use it, or is there some kind premium API so that e.g. analytics companies can pay for better access? I know about the API call limit, I'm wondering about things like the completeness of any search results, or fields/queries available. Any recommended resources also much appreciated.
posted by cromagnon to Computers & Internet (2 answers total)

Best answer: The phrase you're looking for is the twitter firehose. There is a premium api access to every tweet but you can't just walk up and pay for it. You have to apply to twitter and they only approve a select number of people to get that full access. You're probably better off working with one of the companies that are already approved. More details on the twitter blog here.
posted by jourman2 at 6:58 PM on May 8, 2011

Best answer: Some search results will be complete but past a certain number of results they start throttling you. This usually works fine for tracking a small number of keywords. If you're looking to actually sample the entire firehose, there is an API method that will give you a tiny piece of that for free, but as jourman2 notes getting the entire firehose is notoriously difficult and also quite expensive. In that case, you're probably better off going through a company like Gnip or Spinn3r.
posted by signalnine at 7:17 PM on May 8, 2011