Ask MetaFilter questions tagged with data

But where has the pork gone?

cabbages — Thu, 10 Dec 2009 14:22:33 -0800

Where can I find a state by state break down of pork consumption per capita in the USA? Preferably from the early 90s but I'm willing to compromise. GIS data would be nice but not necessary. I'm a student so I can't pay for it.

How to fix this?

amethysts — Thu, 10 Dec 2009 14:04:24 -0800

I'm doing a project in Access and I need help conceptualizing how to get messy data into shape... So I have some data in an Access table. I am well-versed in Access and am using it in this instance for its report-generating qualities and ease of use for the non-tech proficient. This is the only tool at my disposal.
The data came to me as an .xls with a lot of junk and stuff basically all over the place. I made a bunch of queries to get rid of the junk and re-organize (which was a lot easier to do in Access than Excel).

However this is where I'm stuck. This is what I have now:

Name | Test Date | Test data
Jane | --------------------------
------ | 3/10/09 | -----------
------- | ------------ | 1
------- | ------------ | 2
------- | ------------ | 3
John | ---------------------------
------ | 3/12/09 | -------------
------- | ------------ | 3
------- | ------------ | 1
------- | ------------ | 5
------ | 3/13/09 | -------------
------- | ------------ | 2
------- | ------------ | 5
------- | ------------ | 7

This is what I'd like to have:
Name | Test Date | Test data
Jane | 3/10/09 | 1
Jane | 3/10/09 | 2
Jane | 3/10/09 | 3
John | 3/12/09 | 3
John | 3/12/09 | 1
John | 3/12/09 | 5
John | 3/13/09 | 2
John | 3/13/09 | 5
John | 3/13/09 | 7

Some of the names have more than one test under them. How many or which names are impossible to predict. You need to SEE the data to do the updating.

I know ideally you would have Names and Test dates in their own table, but I'm trying to make this a one-table deal for ease of use by people who are not me who will be ok with importing and running the queries (which is basically one click using macros) but not much else.

I can make and run a bunch of queries and forms to do this, but I can't seem to figure out which ones.

1G or 3G?

spacefire — Sun, 06 Dec 2009 13:41:22 -0800

iPhone...first generation used or 3G new, or wait for a while? A friend might sell me an iPhone 1G for $50. Should I buy that or upgrade? I am eligible to renew my AT&T contract. It doesn't give me the iPhone option, so I would have to go to a store, but that is not an issue here. As I understand, the 1G iPhone's data plan is ten dollars cheaper than the 3G data plan. Over a year that would be $120 in savings, plus the phone being $50 cheaper than the cheapest 3G, adds up to $170 savings over a year. I am a bit cash strapped, but is it worth buying an older used phone for saving that much?

EIEIOmega

rleamon — Fri, 04 Dec 2009 15:04:20 -0800

My Iomega external hard drive died... Any chance of getting it going again? Or getting the drive(s) out and into another case? It's an Iomega HDD2H, 640GB, silver case, and I'm pretty sure it's two 320GB drives in a RAID 0 configuration. (Specs are hard to come by for this, sadly.) The case has four screws attaching the outer enclosure that mainly serves to provide feet, but the inner enclosure holding the ddrives defied my initial attempts to open it -- no apparent screws or other modes of ingress. I found a suggestion to try another external power supply with it and happened to have an exact twin that came with another drive -- but no go.

I'd rather rescue the data somehow and have a hard time believing the drives themselves are the problem; it seems more like a power issue. But even if I could get the drives out, what are the chances I could use another RAID enclosure (which I happen to have) and retrieve the data?

Lastly, yes, it's out of warranty, and Iomega quoted a minimum $900 to get the data back. Yikes!

How much memorization does art dance performance require?

Joe Beese — Tue, 24 Nov 2009 18:52:35 -0800

How much memorization does art dance performance require? I assume it varies between genres - and between roles within genres. But can useful generalizations be made about how much performance data retrieval per minute is involved compared with, say, playing in a symphony orchestra?

What interesting statistical information can I dig out of this medical billing data?

neilkod — Thu, 19 Nov 2009 11:37:28 -0800

What interesting statistical information can I dig out of this medical billing data? I have 2000 rows of medical billing information (mysql) with the following attributes. I've normalized the data and here is a very simple description:

patient:
id
dob
city/state/zip
primary insurance provider
secondary insurance provider

provider:
id

procedure:
procedure id/code

patient_procedure:(fk to patient & provider & procedure)
procedure date
billed amount

Unfortunately, the data only spans one month so I can't provide anything interesting across a date dimension beyond aggregating certain measures by weekday.

I've already produced a lot of information such as
% of patients with no secondary insurance
avg age for procedure
top-5 most frequently occurring procedures with large avg cost
billing and frequency by zipcode
so on and so forth-measure X across dimension Y

I'm looking for suggestions for some deeper analysis than the many SQL aggregates that I've performed.

Prepaid iPhone data plan in Mumbai?

jesseendahl — Tue, 17 Nov 2009 14:40:15 -0800

Hi, I will be living in Mumbai for 11 weeks next summer. I have an unlocked iPhone 3G. What is my best bet in terms of carrier and plan for the cheapest and most reliable data access? I plan to do local calls over the cell phone voice network and international calls via Skype on the iPhone. Thanks in advance!

How does the CDC measure the spread of H1N1?

autoclavicle — Fri, 13 Nov 2009 02:01:32 -0800

There's no special place to turn up if you think you've got the swine flu to be tested or otherwise counted--hospitals and clinics tell people to just stay home unless they are having actual health complications. How is the CDC able to say that 22 million people have been infected with H1N1 when if you don't have to be hospitalized, nobody will even test you for it?

How to bulk rename itunes files from external data source.

jlowen — Thu, 12 Nov 2009 15:10:40 -0800

How can I bulk rename files in itunes according to data contained in a separate file? I have 4000-some .wav files in itunes that are essentially field recordings of people talking. The files are named with a 10 digit unique id number and the date acquired. Example: 1234567890_11122009. In a separate csv file I have the identifying information of the individual with the same unique ID number.

I have rated all 4000 recordings based on quality and want to maintain those ratings, but would like to rename the files to drop the unique ID and instead insert the person's name from the csv file. I.e. 1234567890_11122009 becomes Smith_Mary_123_Fake_Street_Anytown_STATE

Can anyone picture a workflow that would accomplish this mass filename change while still maintaining the ratings?

I use a mac and have a decent understanding of automator, grep, csv files etc, but have no real programming skill. I am willing to convert recordings to a different format (mp3 etc.) if that would help. Any ideas?

Mobile Me in Melbourne?

chairish — Fri, 06 Nov 2009 14:15:02 -0800

I'm a Kiwi who travels to Australia (usually Melbourne twice a year for a week or so). I'm currently using a Telecom NZ Nokia e71 phone. If I use data roaming it will cost me $8 NZ per megabyte. I'd like to know from any Aussies out there if you know if there are any alternatives for pre-paid data. The Telecom version of the e71 will work through 900 or 1800 GSM networks as well as 850 or 2100 WCDMA. Obviously the WCDMA is better/faster than GSM.

Do you know if there is a pre-pay type deal that I can sign up for to get a few MB of data - enough for Google Maps, email, etc while I am away from home?

Rates of success?

rollick — Wed, 04 Nov 2009 09:46:36 -0800

Statistics question: is it possible to test sets of cumulative data for significant differences in rate? I have three cumulative percentage graphs, measuring the germination rates of three different seed types. Is there a way to compare them and see if there are any statistically significant differences?

The seed types were planted in triplicate, on three dishes each (nine overall). Every day for the past few weeks I've observed how many seeds on each dish have begun germinating -- so for an individual dish I would have " Day 1: 0 ... Day 7: 14 ... Day 14: 29" etc, with each day's score a cumulative total. (There are 100 seeds on each dish, so it works as a percentage rate as well)

In Excel, I've graphed the average germination rates of the replicates, for a graph that resembles this one. (with three lines plotted, and x-axis = time in days, y-axis = percent germinated).

So is there a way to compare these different rates statistically? I can use Excel, Minitab, SPSS, and R.

Network monitor linux

Petrot — Tue, 03 Nov 2009 07:41:01 -0800

I would like to be able to monitor my own bandwidth usage/data transfer rates on my laptop, whilst using an o2 mobile broadband dongle. Difficulty: Linux. I'm using O2 USB Modem E160 to get online, but it gives a limit of 3GB a month. Usually, with Windows or a Mac the modem comes with a connection manager to show what usage you're at. I got this modem to just about work with Ubuntu 9.04, but without the connection manager. So it just comes down to educated guesswork on my part to see if I'm coming close to this quota limit.

I read this previous AskMe, which suggested using BitMeter II. This looks precisely what I want:
"Using the Alerts feature, BitMeter can notify you when you have uploaded/downloaded a certain amount, or if your speed goes above or below any limit that you specify.

Does your ISP restrict how much you can download each month? Maybe the ISP Restrictions screen would be useful, just enter details of your monthly quota, and BitMeter will warn you when you approach your limit"

This would be really, really useful. But it's only for Windows. There is an open source version in the works, but it doesn't seem to have the above.

So, is there an open source network monitor that will allow me to see if I'm hitting a quota or not. Or, would it be possible to run BitMeter under WINE?

Best way to eliminate personal data on Macbook Pro?

UMDirector — Fri, 30 Oct 2009 06:44:02 -0800

After problems with my old laptop Apple sent me a free replacement, whats the best way to migrate my data, eliminate personal info but leave potential problems intact? So my 2007 MacBook Pro has been a nightmare, multiple logic boards, wireless card outages etc. So they replaced it with a brand new current model MacBook Pro. My initial plan was just to dump my time machine backup onto the new machine, but I am wondering if that would instantly slow down the new machine by adding all the old pref files and junk that accumulates over the years to slow down machines? Should I worry about this? Would it be better to do a pile of fresh installs and manually copy music directories and such?

Also I want to send me old machine back to Apple as intact as possible. I would hate for the current wireless card issue to be a software problem and then I do a entire install and wipe and it fixes it. So I want to send it back as is but delete my personal data, passwords, etc. What's the best way to do this?

How do I automagically display trending of complex data?

Spumante — Mon, 26 Oct 2009 06:41:38 -0800

I need some suggestions for data analysis and displaying trending information. At the moment I'm doing this with a huge amount of Excel pivote tables. The goal is to highlight increasing trends and top hitters in some sort of (semi)automated way. The catch is that the data is pretty deep. For example, say I have a metric for "server rebooted for unknown reason"(which, by the way, is one of 50 things measured)...I'd like to also see the break down by server type and then by, say, location and then to see it by week. What I'm not looking for is a pointer to Tufte or "sparklines" or a generic point to something like "R".

Selling raw data for fun and profit

raghuram — Sat, 24 Oct 2009 21:41:36 -0800

Anybody has experience selling raw data? Anybody here does a business out of selling raw data (formatted and presentable, of course). I know a few sites that do it (aggdata.com, for example). Would like to know the legal and ethical things to keep in mind, while collecting (scraping???) and selling data.

Doing "enough to get by" vs. "hard work"

jnnla — Wed, 21 Oct 2009 11:32:43 -0800

When is it actually worth it to attempt to do any better than average? :::A disclaimer::: The nature of my question may frame me as an "underachiever." Not true. I understand the value of hard work and not cutting corners - trying to the best of ones ability to get a job done right. My question has more to do with going beyond "trying hard" to the point where "hard work" seems to be producing diminishing returns - insofar as time, money and energy are concerned. Hours of overtime keeping people away from their home, their exercise, and their relationships in order to get something that is already "good enough" (like...the client will pay for it) to "perfect" seems to me to be a waste.

____

I work in an industry where perfectionism runs rampant. In most cases 'Good Enough' for the client isn't 'Good Enough' for my peers - and going that extra 10 percent in pursuit of perfection usually involves considerable amounts of time, energy and overtime - which I personally see as reprehensible.

So...

Are there studies that show that the extra 10 or 20 percent of work actually contributes that much to your overall return on energy? Is striving for average rather than striving for perfection better for the bottom line (as I suspect it is)?

Also, on an individual level...are there any studies that indicate that going "above and beyond the call of duty" actually results in more overall objective success. Will working OT, taking on extra work, doing more than is expected of you ACTUALLY, OBJECTIVELY pay off when contrasted with doing just enough to get by?

I already know that anecdotal evidence suggests this to be the case, but I suspect that perhaps objective data might challenge this notion. Maybe I'm wrong....but I'd LOVE to be right.

Have any data visualization recs?

k8t — Mon, 19 Oct 2009 12:34:55 -0800

I need cool data visualizations to project. Any recommendations? We want to project interesting data visualizations on the wall during an event. Ones that don't require any clicking or touching would be best. Thank you!

Common characteristics of scientific fraud or misconduct

Mapes — Thu, 15 Oct 2009 14:19:21 -0800

Article or book that discusses common characteristics of scientific fraud or misconduct? Several years ago I read an article, essay, or book chapter that listed common factors in many uncovered cases of scientific misconduct. One was that subjects typically acted alone; conspiracy was extremely unusual. Another was that researchers generally believed that the faked data represented the truth and would be confirmed if more time or resources were available.

Does anybody know what I was reading?

Humpty Dumpty played with a drive...

Static Vagabond — Thu, 15 Oct 2009 08:17:45 -0800

Damaged harddrive-- need to recover one particular file, but due to the damage, only perhaps 30% of the file is intact. What tools are available to just pull that 30% back? The drive is working to a degree-- if I freeze it I can get directory listings and see the file I want, but when I drag the file off the drive, windows errors out with a CRC error after about 30% and so the file fails to move/copy.

I'm looking for a tool that will keep what it can from the file.

So if the file is like (where G is good and B is bad) :

GGGGGGBBBGGGGGGBBGGGG I'd ideally like to 'recover' the good portions, so I'm left with either
GGGGGG___GGGGGG__GGGG or even just
GGGGGG

I don't care about the bad areas, or want the system to think too hard about querying them (drive only stays nice and chilly for so long).

I thought about installing an FTP server and ftping the file off the drive, but I'm hoping some mefites know of a more speciailized tool for the job, which might also grab those later good parts of the file. Windows options preferred.

How to get my large file from here to there?

vegetableagony — Tue, 13 Oct 2009 19:50:38 -0800

How to privately share a large set of files with a large club? Having trouble getting bittorrent to work. I have a 8gb collection of photos that I helped touch up and organize for a club I was in. Now I am across the country, but most of the other members are all on the same college network.

I tried creating a torrent with transmission using http://tracker.publicbt.com:80/announce as the tracker and gave the people the torrent directly. One person was able to download the whole thing, but now no one is able to find anyone seeding the file, even though I have transmission open, but it isn't finding any peers. I am using osx10.5 and connecting wirelessly my cable internet.

Any thoughts on what might be making the torrent not work or how to fix it?

Alternatively any suggestions on how to get the files out to the people who want them? I don't want to do anything that will have them open to any random person.

Looking for a nystagmus/artcar tracker

tacit_urn — Mon, 12 Oct 2009 15:14:55 -0800

Help me find online map based tracking tools I'm looking for software that will help me graph both time and physical location. For example, I want to keep track of every time and every place I get nystagmus. Evenings, when I'm at sea-level? Sundays near crowds?

I also envision the same tool being used by a group of friends to pinpoint when and where they spot a certain art car in town.

Maybe something like google maps, with colored pinpoints for time. If anyone has example data of this variety that is graphed in a meaningful way, I'd like a link to that as well.

How to merge data from two excel sheets into one?

willy_dilly — Fri, 09 Oct 2009 15:38:49 -0800

How to merge data from two excel sheets into one? Here's the scenario. I have two excel sheets. Column A on both sheets contain item numbers to the products that my business carry. Column B on sheet 1 contains items with product description that are incorrect but sheet 1 is also a complete data sheet with all other information on pricings, vendors, etc. Column B on sheet 2 contains the correct production descriptions but doesn't have the rest of the information (pricings, vendors, etc.).

My question is, is there anyway that I can merge the Column B (product description) of the two sheets together based on column A (item numbers)? I want sheet 1 to now have the correct product descriptions that are contained in sheet 2. Hope this makes sense.

poor quality goji berries?

skjønn — Fri, 02 Oct 2009 20:59:09 -0800

I bought some dried goji berries from my local asian market. They taste/look/smell similar to every other dried goji berry I've tried. However, according to the nutrition information on the package, they are different from other goji berries... According to the package, for 50 grams of dried berries, there are 30 calories, less than one gram of protein, 0% RDA of Vitamin A/C & Iron, and 1% RDA of calcium.

That is very different from the stats given on this site:
http://caloriecount.about.com/calories-eatraw-tibetan-goji-berries-i119927

My goji berries are packed by the "Xin Chang Trrading Company"

So what's more likely- did I buy really poor quality goji berries that are nutritionally inferior to others, or are the nutrition stats on the package wrong?

These berries were about 1/2 the price of what I would have payed for the same amount at my local fancy health food store, but everything at this asian market is loads cheaper, which is why I love to shop there.

What can you see in satellite photos?

k. — Wed, 30 Sep 2009 22:33:01 -0800

Satellite photos: what interesting features can be seen or measured in them (or other aerial photos), and where can I get a supply of images? Examples I have already seen, from some crude Google-searching:
Roads
Buildings
Degree of development (urban/suburban/rural)
Oil spills (apparently these are usually hard to see)
Blooms of algae (cyanobacteria) in the Baltic Sea
Damage to buildings from earthquakes

More would be appreciated--the more specific, the better. These will be used in a computer vision system, so I'd especially like to know where I can find a supply of images containing whatever feature I'm looking for (including the above)--easy for buildings or roads, but hard for rare events like oil spills. Natural disasters seem interesting, if there are pictures available.

I would also consider some other kind of non-photographic data as well, if it could be combined with maps or photos. Don't worry whether you think it's suitable; any idea you have will help!

Data feeds for cricket and other sports

raghuram — Sat, 26 Sep 2009 16:18:31 -0800

Where can I get sports feeds, especially cricket? I am looking to build a hobby cricket sports site. I want data, both current and historical, for records, one day matches, test matches etc. Where can I get such data, preferably free or very cheap?

Also, are there sites for other sports feeds?