Help me find some numbers.

April 23, 2009 12:05 PM

I'm interested in doing data-based mashups. What are some really cool, deep sources of publicly available data?

I realize I'm casting a wide net here, but I'm intrigued by the data mashup, and wonder (a) just how much, and how diverse, publicly available data is out there, and (b) where to find it.

So, just pulling some examples off the top of my head ...

* Obesity rates vs. shopping mall density
* Population vs. bankruptcies
* iPod sales over time vs. weather trends in a specific geographic area

... you get the idea. Anything goes. Is there a canonical list of publicly available data? Are there meta-indexes to indexes of the stuff? How does one go about keeping up to date with all of the numbers that are surely being published every day?

posted by jbickers to computers & internet (17 comments total)

amazon's public data sets are worth checking out
posted by phil at 12:07 PM on April 23

Netflix Prize is a contest to design a better rating algorithm. There's a data-set available if you register as a team.
posted by odinsdream at 12:10 PM on April 23

The US census bureau is a fine source, and freebase looks nice (though I've never used it).
posted by soma lkzx at 12:11 PM on April 23 [2 favorites has favorites]

The Bureau of Labor Statistics, the Federal Reserve, and the Bureau of Economic Analysis all make a bunch of data available.
posted by mhum at 12:23 PM on April 23

The US Energy Information Administration is a great source for anything to do with energy or energy consumption (movement of crude oil, comparison of electricity prices, etc...).

The OECD has lots of good transnational and comparison stats. Also sourceOECD: Taxes, GDP, death rates, education, status of women, etc...
posted by bonehead at 12:39 PM on April 23

There is a lot of Missouri data here. You're likely to find a lot of similar data if you search for state specific GIS data.
posted by schyler523 at 12:41 PM on April 23

Probably not what you had in mind, but the amount of biological data available through the NCBI is staggering.
posted by chrisamiller at 12:52 PM on April 23

Case Western's Center on Urban Poverty & Community Development in Cleveland, OH has a great publicly available dataset culled from the following sources:

* Census
* Crime data from the Cleveland Police Department
* Vital statistics from the Ohio Department of Health
* Property characteristics and sales information from the Cuyahoga County Auditor and Recorder
* Public assistance data from Cuyahoga County Employment and Family Services
* Juvenile delinquency data from the Cuyahoga County Juvenile Court
* Child maltreatment data from the Cuyahoga County Department of Children and Family Services
* Mortgage lending data (HMDA) from the FFIEC
* Enrollment and attendance from the Cleveland Municipal School District
posted by archofatlas at 12:53 PM on April 23

What you're asking for is the same thing experts in the field are asking for: publicly accessible raw data. TED has a few videos of experts explaining the situation and what it will result in:

Tim Berners Lee
Hans Rosling (sequel)
posted by pwnguin at 1:14 PM on April 23

Have you looked at ManyEyes yet? There are a lot of visualizations with associated data available for further use.
posted by Alterscape at 1:46 PM on April 23

Population Reference Bureau
Department of Energy Data Explorer
Numbrary
Datamob has an extensive list of datasets
Comprehensive Knowledge Archive Network
Infochimps.org
OpenStreetMap for geographic data
posted by motherly corn at 2:42 PM on April 23

IPUMS. They link to a lot of good stuff, too.
posted by puckupdate at 4:31 PM on April 23

The MySociety websites have lot of UK data involving national and local government.
posted by the latin mouse at 4:32 PM on April 23

Seconding Freebase (disclaimer: I worked there). I believe it's the broadest and most frequently updated open database.
posted by zippy at 5:35 PM on April 23

Check out Swivel. It's an interface to upload data sets and generate various displays of the data, but you can usually go backwards and just download the data set.
posted by tybstar at 10:05 AM on April 24

The delicious datasets tag is also interesting.
posted by davar at 3:05 AM on April 25

Just saw Google's blog post about a search interface to public data. Might be useful to you. Also look at Google Visualization toolkit (esp. the pieces from Trendalyzer) for visualizing how data changes over time. The Hans Rosling talk mentioned by a previous commenter uses Trendalyzer visualizations.
posted by thaths at 4:31 PM on April 28