Emergency Statistics Help

June 17, 2005 6:00 AM

My boss' boyfriend does human factor engineering for a pharmaceutical company. They're performing an experiment and his boss has given him a number for a sample size that the boyfriend thinks is too low. He has to be able to justify to his boyfriend that this number is too low within the next 40 minutes. I can't remember anything about statistics but I know there's a way to figure this out. (More inside!)

Here's the setup:
Population=350,000 people
Standard deviation from previous study= 1.25 the mean=2.35
P should be <.05
They expect 80 percent power (I have no idea what that means..)
---------
Further info from the sheet she gave me:
If we use 10 people and have each perform the experiment 10 times, can we use 100 as our sample size?
"I didn't think so, but was told it's ok because of the binomial theory" - from the boyfriend's comments

posted by tozturk to Work & Money (9 answers total)

Human factor engineering is something like usability? What's that mean for an experiment at a pharmaceutical company?
posted by pracowity at 6:25 AM on June 17, 2005

Is the mean supposed to be compared to a null hypothesis mean of 0?
posted by naturesgreatestmiracle at 6:46 AM on June 17, 2005

Assuming the null hypothesis is 0, with a mean of 2.35 and SD of 1.25, alpha of .05, and a two-tailed test, 5 people should reach power of .80.
posted by naturesgreatestmiracle at 6:47 AM on June 17, 2005

Response by poster: Thanks to Naturesgreatestmiracle! I appreciate the reply back.... that helps a lot! We are still stuck on the second part of the "mystery" however:

If we use 10 people and have each perform the experiment 10 times, can we use 100 as our sample size?
We don't think so, but were told it's ok because of the binomial theory" ... does this make any sense???
posted by tozturk at 6:54 AM on June 17, 2005

your description is unclear, unless someone from the same industry is here. it's too garbled/confused to give an answer based on knowing statistics in general [on preview, naturesgreatestmiracle sounds like they know what they're talking about, but what's that got to do with 10 people 10 times?]

however, in general, you cannot use the same person more than once and expect things to scale. if you need a sample of 100 that's because you need 100 different people, each with their own random variations and attitudes. using 10 people 10 times each won't get you the same range - the results for each person will be more similar than they would for 10 different people (the technical term is that results for one person give "correlated results").

but there should be someone involved who knows this kind of thing. and it's not the kind of decision that should be made against a deadline. his best approach, in my opinion, is to say "look, we could waste a lot of money here. i am not happy with this. i need more time to find someone who is an expert, explain to them what we are doing, and get an answer. i need 2 days." or similar. he might consider making his objections in writing - both to cover his back and to make a more serious impression.
posted by andrew cooke at 6:55 AM on June 17, 2005

andrew cooke is correct about making sure you don't sample the same person twice. You typically assume that observations are independent, and getting more than one measurement from the same person violates that assumption (if you are doing a one-sample or indiependent-samples comparison, which it sounds like you are).

Now, as for combining different samples, there are a few issues here.

Are you simply getting the same type of data from different at 10 different times? If so, this shouldn't be a problem (assuming conditions are controlled). Much behavioral research (what I do) is conducted this way. We simply can't get 200 observations at exactly the same time, so we split the data collection up into smaller chunks.

If, however, you are planning on sampling 10 people, checking the results, then getting 10 more, checking the results, etc, until you see the significant effect you are looking for, this isn't such a good idea. Ethically, it's questionable, and statistically, you are following your sampling error around too much. You may end up with the same results as in the first situation, but, methodologically, it's not sound.
posted by naturesgreatestmiracle at 7:04 AM on June 17, 2005

Response by poster: I agree, it's ethically questionable - but this is what was presented to me this morning.

The boyfriend wants to be able to justify not using the sample that the boss wants to use.
posted by tozturk at 7:06 AM on June 17, 2005

Also, I sent a little more info to you through email.
posted by naturesgreatestmiracle at 7:07 AM on June 17, 2005

Might they be having 10 people do it 10 times so that they can average the performance of each person and so smooth out individual variations in performance? I don't know much about statistics, so I don't remember what that's called. I want to say R-Squared or something like that.
posted by willnot at 8:05 AM on June 17, 2005