Ask MetaFilter questions tagged with statistics

Math/Stats: help me analyze a data set and determine the values that created it

intermod — Wed, 23 Dec 2009 12:58:21 -0800

Mathematics / Statistics Filter: I have some pairs of numbers that are the result of a process. Given just that data set, and a rule that relates them, can you determine the integer values that could have resulted in those sets? Apologies for the phrasing of the FPP -- I know it doesn't make much sense. Hopefully some mathematics / statistics types will click through and see this longer version.

I have some sets of numbers, shown below. I'm trying to reverse engineer the numbers that could have resulted in these sets, based on some known mathematical relationships between them.

In general, a given Device(n) consumes a resource in integer Quantities at a floating point Rate(n), resulting in a total Cost for that consumption run. What I have is pairs of Device/Cost, for several different Device for several runs each, and I'm trying to determine the floating point Rate. The Rate is constant for a given Device. The Quantity consumed is different for each run, but the one key here is that I know that the Quantity values are integers.

So for a given Device run, we have:

Quantity(int) x Rate(float) = Cost(float)

All I have is Cost data for each Device, but I have multiple sets of these and am hoping there's some sort of numeric analysis that can tell me the likely Quantity values that fit.

Here's a sample of the data:

Device / Cost
Device1 / 1235
Device1 / 988
Device1 / 1003
Device1 / 1526
Device2 / 3652
Device2 / 1207
Device2 / 1729
Device2 / 518
Device3 / 745
Device3 / 2115
Device3 / 1415
Device3 / 334

So, for example, using the Device1 / 988 and Device1 / 1003 set, I could eyeball it and see that the Cost difference of 15 is due to 1 unit of Quantity difference in the runs. Thus the first run consumed 66 x 14.97 = 988 and the second run consumed 67 x 14.97 = 1003 . (Alas, the Rate values should be more in the 30-50 range, so 14.97 doesn't make much sense) But I'm hoping that with a larger population of data, there's some analysis I can do that will give a more confident answer.

Perhaps this can even be solved without ensuring that the Quantity values are integers, but it's a constraint that the data is supposed to have so I thought I'd mention it.

I'll monitor this thread for the next couple hours to answer any questions. And will add better tags! I used the science category because this seems like the kind of math that a lab scientist might be familiar with, trying to analyze a data set to work out the conditions that created it. I'm especially hoping for a statistic analysis that produces some sort of confidence measure, because a couple of these data points might be outliers, screwing up what might otherwise be a closed solution.

Note: this is not homework filter, or even do-my-job filter. It's just something I'm trying to reverse engineer.

Conditional Probability

chrisamiller — Wed, 16 Dec 2009 17:25:54 -0800

Stats-filter: Given a binary matrix, if I know the total number of ones in a given row and a given column, can I calculate the probability that a given position contains a one? I have a binary matrix, like so, where every value is either 1 or 0. So, if the first column contains 2 ones, and the first row contains 1 one, what's the probability that position A contains a one?

Example:



    2

    ________

 1 | A|  |  |

   |__|__|__|

   |  |  |  |

   |__|__|__|

   |  |  |  |

   |__|__|__|

   |  |  |  |

   |__|__|__|

(not homework-filter)

Ask a Stats Nerd

sonofslim — Tue, 15 Dec 2009 08:18:52 -0800

I need to ruin everyone's fun by adding a rigorous mathematical scoring system to the company bake-off. Li'l help? We just had an employee bake-off and while the outcome was satisfactory to all, I think the scoring methodology (average of ratings on a 1-10 scale) was too arbitrary to be statistically meaningful.

How would you build an objective and mathematically sound ranking system based on the following criteria?

* Entries will be judged by all participating employees in each of three categories: taste, presentation, and creativity
* There will be a winner within each category, as well as an overall winner
* Not everyone has to vote on every entry

We make software, so yes, the methodology by which we judge pie is quite critical.

How much money does the gossip magazine / gossip blog / tabloid industry make?

Elle Vator — Fri, 11 Dec 2009 12:23:45 -0800

So, celebrity gossip is everywhere and lots of $$$ is forked out for images of famous babies, etcetera... BUT how much money does the gossip industry MAKE? I can't find any online statistics about Gossip Industry Revenues. It's actually very odd and mysterious.

I'm looking for numbers on gossip magazines, gossip blogs and/or the tabloid industry, with the goal of comparing the Gossip numbers to the Porn Industry and the Television Industry respectively. I could use global numbers, US or North American numbers.

It would be useful to find a webpage as comprehensive as this: http://internet-filter-review.toptenreviews.com/internet-pornography-statistics.html

...but I would take a single, high impact sentence about the tabloids and gossip rags (such as this gem: "US Porn Revenues are greater than the combined revenues of ABC, CBS and NBC"), which was properly cited and verifiable.

Anyone got anything?

What are the odds?

JDHarper — Wed, 09 Dec 2009 15:59:54 -0800

I'm rolling 5 dice. I win if I roll two 1's, one 2, or one 3. What are my odds of winning, expressed as a percentage? This question is actually about the board game Battlelore, but I've simplified it above for ease of discussion.

In the game, you roll dice to score hits. Instead of numbers on the dice, there are three banner color icons (red, green, and blue), a "lore" icon, a retreat flag icon, and a bonus strike icon.

In combat, if you roll the banner color that matches the color of the target, you score a hit. In most circumstances, you also get a hit if you roll the bonus strike icon, and sometimes if you roll the lore icon or the retreat flag icon, just depending on what cards are in play and where the pieces are on the board.

I'd put together a spreadsheet for determining the odds of almost any given die roll in Battlelore. So, for example, if you want to know the odds of scoring two hits if you're rolling four dice which hit on bonus strikes and blue banners, you'd look at cell D31 and see that you have 40.74% chance of that happening.

But there's this one case from a recent game that's giving me trouble: I need to score one hit, I'm rolling 5 dice, and I score hits on the blue banner, the retreat flag, and on bonus strikes. BUT: The target ignores the first bonus strike.

No problem, I thought: That would be the odds of scoring one hit on on two die faces with five dice (cell D8, 86.83%) plus the odds of scoring two hits on one die face with five dice (cell C32, or 19.62%).

Unfortunately, those results add up to more than 100%, so I know that's wrong. The odds are high, probably over 90%, but there's still the chance that I'll roll all green and red banners or something.

How do I handle this oddly specific situation?

How many grandmothers will die in two months?

Meatbomb — Tue, 08 Dec 2009 22:55:12 -0800

Actuarial / statistics geeks: a puzzle for you. You are on a two month training program with a group of Korean teachers of English.

There are 20 teachers, ranging in age between late 20s to early 50s.

How many of their grandmothers are likely to die during the course?

Consider the current months (October-November) as the timeframe. Assume the grandmothers are all residents of Daegu, Korea.

I probably cannot provide any more specifics than those, but if you want clarification I will try.

What interesting statistical information can I dig out of this medical billing data?

neilkod — Thu, 19 Nov 2009 11:37:28 -0800

What interesting statistical information can I dig out of this medical billing data? I have 2000 rows of medical billing information (mysql) with the following attributes. I've normalized the data and here is a very simple description:

patient:
id
dob
city/state/zip
primary insurance provider
secondary insurance provider

provider:
id

procedure:
procedure id/code

patient_procedure:(fk to patient & provider & procedure)
procedure date
billed amount

Unfortunately, the data only spans one month so I can't provide anything interesting across a date dimension beyond aggregating certain measures by weekday.

I've already produced a lot of information such as
% of patients with no secondary insurance
avg age for procedure
top-5 most frequently occurring procedures with large avg cost
billing and frequency by zipcode
so on and so forth-measure X across dimension Y

I'm looking for suggestions for some deeper analysis than the many SQL aggregates that I've performed.

Paired T Test sounds like parakeet test to me

amethysts — Fri, 13 Nov 2009 11:37:29 -0800

Hi, I have a specific question about paired t-tests! On the paired t-test, the spreadsheet looks like this
Let's say I'm doing a a paired t-test for timed trials.
The spreadsheet looks like this:

Subject A Subject B Difference
5 3 2
4 4 0
5 6 -1

I don't understand why the "Difference" column allows for negative numbers. That would imply that there was a meaningful difference in the order of the two columns. If the order of the two columns isn't meaningful (it doesn't matter what data goes in col.1 and what data goes in col.2, since the trials were not completed in a set order) then shouldn't the "Difference" column transform the difference into the positive number? (For example, the -1 above would become a 1). Or is the order of the columns meaningful somehow that it would need to be preserved through the negative number?

How does the CDC measure the spread of H1N1?

autoclavicle — Fri, 13 Nov 2009 02:01:32 -0800

There's no special place to turn up if you think you've got the swine flu to be tested or otherwise counted--hospitals and clinics tell people to just stay home unless they are having actual health complications. How is the CDC able to say that 22 million people have been infected with H1N1 when if you don't have to be hospitalized, nobody will even test you for it?

Interpretation of cross-correlation and cross-covariance plots

noztran — Fri, 06 Nov 2009 06:56:44 -0800

How to interpret certain features? I am looking for some help regarding interpretation of both cross-correlation and cross-covariance plots.

Namely, I have been struggling with these features:

* sudden negative spike immediately followed by positive spike
* broad ('smeared') bulge
* oscillation of the plot (i.e. the trace somewhat resembling a sine)
* ondulation of the trace (think of it as if you were writing u's continuously)

I have been going through google over and over again, but to no avail. Are there any good sources/books/etc. that deal with actual interpretation of cross-correlation and cross-covariance plots?

BONUS question: to what extent can cross-correlation and cross-covariance be treated as a representation of impulse response function?

Any thoughts will be appreciated.

How would I pick which equation to use?

lpctstr; — Wed, 04 Nov 2009 22:49:49 -0800

I want to determine the difference between distributions. When would I use kl-divergence and when would I use rmse? It seems like both equations reduce deviation to a single number, but couldn't find a comparison between them.

Rates of success?

rollick — Wed, 04 Nov 2009 09:46:36 -0800

Statistics question: is it possible to test sets of cumulative data for significant differences in rate? I have three cumulative percentage graphs, measuring the germination rates of three different seed types. Is there a way to compare them and see if there are any statistically significant differences?

The seed types were planted in triplicate, on three dishes each (nine overall). Every day for the past few weeks I've observed how many seeds on each dish have begun germinating -- so for an individual dish I would have " Day 1: 0 ... Day 7: 14 ... Day 14: 29" etc, with each day's score a cumulative total. (There are 100 seeds on each dish, so it works as a percentage rate as well)

In Excel, I've graphed the average germination rates of the replicates, for a graph that resembles this one. (with three lines plotted, and x-axis = time in days, y-axis = percent germinated).

So is there a way to compare these different rates statistically? I can use Excel, Minitab, SPSS, and R.

Programs to work with Census data?

qldaddy — Fri, 30 Oct 2009 08:51:36 -0800

How do I create a cross-tab from Census data (SF3)? I would like to create a cross tab that shows disability status by household (or family) income at the census tract level of detail. I cannot get that table from American Factfinder. Saw the link that said "download the data and you can generate your own cross tabs". Downloaded data - got big old text (ascii) file that does not easily import into excel or access (the only two potential programs I have now). Now I have big old text file and shattered hopes.

I think I am looking for a (preferably free or shareware) program that will open and analyze this data with a minimum of fussing on my part.

Or, maybe I am looking for something completely different. You'd propbably know better than me.

Thanks in advance.

Help me out-Billy Beane my fantasy baseball league!

po822000 — Wed, 28 Oct 2009 07:06:34 -0800

How do I appropriately valuate players in my competitive fantasy baseball league? I just took over a horrendous team in a very competitive fantasy baseball league. I'm very familiar with sabermetric stats employed by real-life GMs, but it occurs to me that those same stats are not necessarily going to be helpful to me in fantasy baseball. (Especially because so many of the stats we score on are non-sabermetric in nature.)

To wit: real-life GMs use sabermetric stats to find players who avoid making outs (hitters) or get outs most efficiently (pitchers). In my league, I'm looking to maximize or minimize the cumulative results in specific statistical categories across my lineup.

To that end, I'm trying to put together my own valuation system for players, but it's been awhile since my econometrics and statistics courses in college. Can you help me put together an appropriate formula - or point me in the direction of how I should be analyzing/comparing the statistics - to properly valuate the players in my league?

The details: I'm in a weekly head-to-head rotisserie league based upon the following statistical categories:

HITTERS
Runs
Singles
Doubles
Triples
Home Runs
RBI
Walks
Strikeouts (lowest total wins this category)
Stolen Bases
Batting Average

PITCHERS
Walks (lowest total wins this category)
Strikeouts
Complete Games
Wins
Losses
Saves
Holds
ERA
WHIP (lowest total wins this category)
K/9

I'm not sure if any other parameters are needed, but any help you might be able to provide would be most appreciated.

Not all random numbers are created equal

monkeystronghold — Sun, 25 Oct 2009 23:30:14 -0800

How do I get a controlled distribution of random numbers to fairly determine a start position. In a sporting event, start position is decided based on the last digit of your registration number. Each week, random numbers are drawn to decide the start order. For example, the random draw order for a single week is 4, 0, 3, 5, 1, 8, 7, 9, 6, 2. So everyone with a number ending in 4 starts first. Everyone with a number ending in 0 starts second. And so on. The next week the draw order is again random.

While this works, the distribution of of numbers can end up being unfair (one particular number can be "lucky" or "unlucky" for many weeks). Statistically, how would one generate a set of "random" start orders so that the value of each registration number was roughly equal over the course of a season (for ease of calculation, let's say 10 weeks).

I don't know anything about math or statistics, so my description of this situation probably uses lots of words incorrectly. I'd google this, but I don't even know how to start.

Basically, is it possible for value of all of the numbers to even out. But in a random order. So, for example, that 0's aren't always going after 4's. And one group doesn't always start in the middle.

BookSuggestionFilter: I need to learn R (and about statistics) in a hurry.

kenliu — Thu, 22 Oct 2009 19:17:29 -0800

BookSuggestionFilter: I need to learn about R (and statistical modeling) in a hurry. I need to learn R in a hurry. I'm heading full steam into a new software development project at work where I will be working with some hard-core statistical modelers who are building models in R (as well as using other stuff like SAS).

I have a degree in engineering so I am ok with the math, but it has been over a decade since I did anything with statistics (and it was pretty rough even back then).

I'm looking for some book recommendations for the following:
1) a refresher on general statistics and some introductory material with statistical modeling
2) a book about R for someone who is an experienced software developer

Why are postseason baseball games so long?

madcaptenor — Sun, 18 Oct 2009 18:38:10 -0800

Why are postseason baseball games so long? (Data inside.) As I start to write this question, it's 9:09 pm. The Phillies and Dodgers started their game tonight at 8:07 pm and just finished the second inning. At this rate (31 minutes per innings) the game will take around four and a half hours.

Now, it's 6-0 Phillies, and it takes time to get all those men on base, so this isn't typical. But in Major League Baseball, playoff games in general seem to take longer than regular-season games.

Last year's postseason game times (obtained somewhat laboriously from baseball-reference.com):
World Series: 3:23, 3:05, 3:41, 3:08, 3:28
NLCS: 2:36, 3:33, 2:57, 3:44, 3:14
ALCS: 3:25, 5:27 (11), 3:23, 3:07, 4:08, 3:48, 3:31
NLDS (Phillies-Brewers): 2:39, 3:00, 3:31, 2:53
NLDS (Dodgers-Cubs): 3:10, 3:10, 3:03
NLDS (Rays-White Sox): 3:10, 3:10, 3:07, 3:13
NLDS (Red Sox-Angels): 3:14, 3:51, 5:19 (12), 2:50

If you just glance at this, you see that games over three hours predominate. Median game length is 3:13; mean is 3:24.

For comparison, the median length of games played by the same eight teams during the regular season was 2:53, the mean 2:55. (In the interest of full disclosure, games between two of those teams are counted twice.)

Why are postseason games longer? More pitching changes? Longer breaks between innings? (They don't seem longer, but an extra thirty seconds at every commercial break is ten minutes or so over the course of a game.) Something else I haven't thought of?

What's the statistical technique for combining several test results into one?

gsteff — Sat, 17 Oct 2009 20:39:39 -0800

I'm a statistics n00b trying to learn how to combine the results of several tests into one. Basically, I'd like to learn to how to categorize entities in some experimental data by combining the scores from several domain-specific tests into one unified score. A practical but hypothetical example would be writing a computer program that given the sound of a car engine will try to identify what model of car it came from. Say that there are 10 possible cars each sound can be matched with, and three independent tests that are applied to each sound, each test producing a number between zero and one for each car model indicating how likely it is that the sound came from that model. I could combine the test results naively by simply adding them together, but that could produce crappy results if one of the tests is much more accurate than the others, possibly worse results than just using that test by itself.

There are established ways to do this that I've seen used in research before, but don't know any of the math and haven't had any luck Googling for info. I'm not looking for a detailed explanation, just some pointers to what I should research to teach myself. Thanks!

Does chi-square work for this?

MoreForMad — Sun, 11 Oct 2009 15:23:30 -0800

StatisticAnalysisFilter: I took (pretty close to) scientific observations of the general populace in a neighborhood for a few months (personal project, long story). I measured the number of people who had trait X (or did not have trait X) in two locations, A and B. Now, I want to test the statistical significance of these results. Is the chi-square test sufficient for this? Or is there a better option? Effectively, I've observed that Trait X is much more common in location A than location B. But I want to let the numbers do the talking, of course, and see if this is a statistically significant difference, or if it is due to chance. Standard science stuff, but I want to make sure I'm doing this correctly, since I haven't done this in a while!

Where can I find detailed statistics on the garbage sitting in landfills?

wandergeek — Thu, 08 Oct 2009 06:20:47 -0800

I'm looking for statistics on the types of garbage that's currently sitting in American landfills. Does anybody have any good resources for this? To be clear, I'm trying to find the percentage of paper, plastics, metal, etc as they contribute to the greater whole. Statistics from other countries are welcomed too.

significantly negative about the education I got in statistics

Tandem Affinity — Mon, 28 Sep 2009 06:03:59 -0800

Is a contingency table the right thing for this kind of data (detailed within), and either way, how do I analyze it in Prism 5.0? The experiment includes at least two different treatments, sometimes more. There are always 10 experimental samples. Each treatment is replicated three times, for a total of 30 experimental samples at each treatment. Each experimental sample can have the outcomes: positive or negative, or inconclusive (if necessary, we could turn the inconclusive to "negative"). My stats seek to answer, does one treatment or another lead to significantly more positive or more negative sample?

Any recommended texts on the subject would be greatly appreciated -- I'm going to need to have sound documentation for how I'm doing this at some point, although for the time being, at least understanding would be a great help.

If it helps any, these experiments are sterilization testing (think autoclave or bleach, not tubal ligation...) and involve treatments that can achieve: total inactivation/sterilization (no positives), fractional inactivation (some positives, some negatives), or what i would view as probable lack of inactivation (all positives). Most of the results I am seeing now involve two treatments with fractional inactivation and I am trying to get a handle on whether there is a way to say that one treatment resulting in fractional inactivation is better at inactivating than another treatment resulting in fractional inactivation. Coworkers and boss want to think that more negatives = better inactivation, but when two treatments have 4/30 positives versus 10/30 positives, i am really not sure how to calculate whether that 4/30 is significantly more inactivated than 10/30.

I've search metafilter for info on contingency tables, as well as google for the same as well as trying to figure out if I should be using some other type of analysis....I am more confused than when I began. It seems like the examples I find include more complex data than that I am working with...

Help me determine rate of imprement for my players!

TheManticore — Sat, 19 Sep 2009 13:28:55 -0800

I coach a high school soccer team. Each practice we go through a routine which involves repetitions of various skills. The number of repetitions accomplished by a player for each skill is recorded in an excel spreadsheet for ranking purposes. I want to determine the rate of improvement so the players can see their progress and who's improving most. Is there a simple excel function that would help me do this?

stay awhile

dacoit — Mon, 07 Sep 2009 19:04:18 -0800

My website's stats app says that visitors stay for an average of five minutes and ten seconds. I know that this is pretty decent ... but does it put us in the 70% percentile? The 90% percentile? Is there anywhere you know of where I can go to find lots of comparative statistics for duration of visits to sites?

Parvovirus: odds of a puppy getting it?

zippy — Tue, 01 Sep 2009 23:34:45 -0800

I understand the danger parvovirus poses to puppies, but what are the odds of a puppy contracting the disease in the US (specifically Alameda County, California)? I have been reading about parvovirus in dogs (including this discussion), and understand how serious the illness is.

What I can't seem to find is any indication of risk or prevalance. What are the odds a dog will get parvo, and how many cases of it are there a year in my area?

The more mathematical and bounded the answer, the better. I know I can't be assured to the fifth decimal place about anything, but I want to know: Parvo, this terrible disease, are the odds 1%, 10%, or 100%?

More details below, in the hope that they may allow more exact bounding of the answer.

My dog is five weeks old. He was one of the larger dogs in the litter (with two or three brothers and a sister), which I understand tends to confer longer maternal immunity. I intend to start him on a full vaccine series for parvo.

He's 3/4 Australian Cattle Dog, 1/4 Fox Terrier. He was born in a remote rural area of Humboldt County, California, and as of a few days ago now lives in a semi-urban area in Alameda County.

I keep him mostly indoors, with trips to the back and front yard for exercise. I understand that completely preventing exposure to parvo is impossible (as the virus hardy and survives for long periods in the soil), but also that minimizing exposure to parvo greatly reduces the chances for infection.

I would like to know:

How common is parvo in Humbolt County and in Alameda County? Or, if these specific numbers aren't available, then whatever numbers are available for California or the US. A link to numbers of cases per year would be ideal.

What are the odds of a puppy getting parvo between the ages of 5 and 16 weeks if he's allowed to socialize with a: known dogs (with shots), or b: occasionally visit parks and meet other non-wild dogs.

Links to scholarly papers are fine, and links to the dog equivalent to the CDC would also appreciated.

If this is too specific, or if there isn't enough information, please let me know. Also, I do know how bad the illness itself is.

Maintaining Server Privacy

dacoit — Thu, 27 Aug 2009 16:27:15 -0800

If you do a search for my company’s website on Google, one of the top hits is a page at Quantcast.com that purports to display “visitor statistics” for our domain. These statistics are wildly inaccurate by orders of magnitude and make it look as though our site gets only a tiny percentage as many visitors as it actually does. What’s more, there are all kinds of demographic charts and info, purportedly about our site, that don’t seem to have any connection to reality whatsoever. How do I get them to stop misrepresenting us? In their FAQs, Quantcast answers the question “how do I remove my site from Quantcast’s listing?” with “We do not remove sites from our listing.” They propose that one join their Quantcast Publisher program, but this program forces you to give them your actual server stats, which they will use and publish whether you like it or not.

I’m galled that there seem to be only two choices:

a.) putting up with a page of highly visible misinformation about my company that causes us actual financial harm; and

b.) signing up for some company’s service and giving them access to our server stats.

Surely there's a third choice? My question is this:
Do you think there’s any way to word a letter to a company like this to get them to remove our site from their listing?

(BTW, I know there are a couple of other companies, like Alexa, that do this kind of thing, but they don’t bother me because their stats are vague and don't pretend to be accurate. It's the false "accurate" picture that I can't get over.)