Test for significant change from a menu of choices over time.
June 16, 2014 1:54 PM Subscribe
A large group of people are asked to choose exactly one option out of 8 options, A B C D E F G H. A year later this is repeated. How to test for significance?
As the response rate is (>90%), say size m at first and size n on the repeat.
Assume that I am testing for significant change (either up or down) for each of the options A B C D E F G and H, at the 95% level.
If it matters then the options are independent of each other, and have no order.
What's the appropriate test to apply in this situation? t-test?
It's been a while since I've looked at statistics. So if you point me towards a worked example that would be most appreciated.
Thank you.
As the response rate is (>90%), say size m at first and size n on the repeat.
Assume that I am testing for significant change (either up or down) for each of the options A B C D E F G and H, at the 95% level.
If it matters then the options are independent of each other, and have no order.
What's the appropriate test to apply in this situation? t-test?
It's been a while since I've looked at statistics. So if you point me towards a worked example that would be most appreciated.
Thank you.
Response by poster: Same population, and I think with a >90% sample we can assume it is the right proportions of the population.
posted by 92_elements at 2:37 PM on June 16, 2014
posted by 92_elements at 2:37 PM on June 16, 2014
Do you care whether individual people have changed, or just overall proportions? If you care about individuals, and they're just categorical, I would do some sort of paired permutation test; shuffle the second draw 1000 times (keeping proportions the same) and see the distribution of changes in outcomes for each category (i.e., by chance, you'd expect X%+/- Y% to switch away from choice A), then put the actual changes into that distribution (Z% actually switched away from A, and Z is in the Nth percentile of the X distribution). I think that makes sense but I'd have to know more about the exact question and data set I think.
posted by supercres at 3:33 PM on June 16, 2014 [3 favorites]
posted by supercres at 3:33 PM on June 16, 2014 [3 favorites]
That's just Monte Carlo brute-forcing it, but that's often easier (and probably assumes less) than the classical tests.
posted by supercres at 3:35 PM on June 16, 2014 [1 favorite]
posted by supercres at 3:35 PM on June 16, 2014 [1 favorite]
As supercres says, the answer depends on whether you care about individual people's responses. If you don't care, then I think a Chi-square would work for you. Here's a calculator. Each "option" would be a category, its initial score would be the expected, its score a year later would be the observed.
posted by agentofselection at 4:03 PM on June 16, 2014 [1 favorite]
posted by agentofselection at 4:03 PM on June 16, 2014 [1 favorite]
...though as I understand it, a chi-square test will just tell you whether any significant changes in popularity have occurred — and not which options are significantly more or less popular.
posted by nebulawindphone at 4:45 PM on June 16, 2014 [2 favorites]
posted by nebulawindphone at 4:45 PM on June 16, 2014 [2 favorites]
A Chi-Squared or Fisher Exact Test on a 2x8 contingency table will tell you if there is a difference between the two groups, but not which differences are most significant.
posted by Blazecock Pileon at 5:30 PM on June 16, 2014 [1 favorite]
posted by Blazecock Pileon at 5:30 PM on June 16, 2014 [1 favorite]
If the chi-square comes out significant, you can just do post-hoc comparisons with Bonferroni alpha correction.
Though now that I think about it, I'm not sure if my permutation test is theoretically correct. Why should the proportions remain the same? It really all depends on your question: individuals changing (in which case I think it's valid) or overall proportion changing. I actually can't think of a good, simple permutation test for overall proportion, but here's the Bayesian approach if you have a copy of R handy. I guess it's a special case of contingency tables. (He outlines it better than I could here.)
posted by supercres at 6:37 PM on June 16, 2014 [1 favorite]
Though now that I think about it, I'm not sure if my permutation test is theoretically correct. Why should the proportions remain the same? It really all depends on your question: individuals changing (in which case I think it's valid) or overall proportion changing. I actually can't think of a good, simple permutation test for overall proportion, but here's the Bayesian approach if you have a copy of R handy. I guess it's a special case of contingency tables. (He outlines it better than I could here.)
posted by supercres at 6:37 PM on June 16, 2014 [1 favorite]
Yeah, chi-squared test is the way to go here. You don't want to test each option separately, since they're dependent: the number if responses in each category has to sum to m or n, as appropriate.
posted by matildatakesovertheworld at 7:12 PM on June 16, 2014 [1 favorite]
posted by matildatakesovertheworld at 7:12 PM on June 16, 2014 [1 favorite]
Chi-square on this is a very sensitive test, by the way, and more so with a lot of cells. So much so, that I find it meaningless in practical terms.
Many people + many choices = pretty hard to maintain the marginals over time
I would go with a simulation based approach, personally, or collapse the choices.
posted by gregglind at 7:23 PM on June 16, 2014 [1 favorite]
Many people + many choices = pretty hard to maintain the marginals over time
I would go with a simulation based approach, personally, or collapse the choices.
posted by gregglind at 7:23 PM on June 16, 2014 [1 favorite]
Response by poster: Thanks for all your great replies so far.
Do you care whether individual people have changed, or just overall proportions?
Just overall proportions will be fine in this case. We haven't linked together people's choices between years.
I'm okay with using monte Carlo and do have R installed. (supercres).
I've decided to use a chi squared test, and combine the categories down to 4 categories. Thanks gregglind.
Maltida are you saying that I can't now use stats tests if I find significant difference from the chi squared test. Even if I apply Bonferroni corrections?
Significant or not after I've done this for the whole sample I'm considering doing a subset based on if the people were subject to a type of intervention. Is this wise? What do I need to be aware of?
Many thanks.
posted by 92_elements at 5:39 AM on June 17, 2014
Do you care whether individual people have changed, or just overall proportions?
Just overall proportions will be fine in this case. We haven't linked together people's choices between years.
I'm okay with using monte Carlo and do have R installed. (supercres).
I've decided to use a chi squared test, and combine the categories down to 4 categories. Thanks gregglind.
Maltida are you saying that I can't now use stats tests if I find significant difference from the chi squared test. Even if I apply Bonferroni corrections?
Significant or not after I've done this for the whole sample I'm considering doing a subset based on if the people were subject to a type of intervention. Is this wise? What do I need to be aware of?
Many thanks.
posted by 92_elements at 5:39 AM on June 17, 2014
Multiple comparisons. Always multiple comparisons. I'm guessing you know this, but it's my reflexive response to, "What else do I need to be aware of?" :)
posted by supercres at 2:27 PM on June 17, 2014 [1 favorite]
posted by supercres at 2:27 PM on June 17, 2014 [1 favorite]
Maltida are you saying that I can't now use stats tests if I find significant difference from the chi squared test. Even if I apply Bonferroni corrections?
Sort of. I'm saying the tests aren't independent in a way that goes beyond multiple comparisons. Suppose you only had two categories. If one increased by x, then the other decreased by x and so if one of your tests was significant, the other would be too. But of course both tests are testing essentially the same hypothesis, so there's no need to correct. In short, the bonferoni correction is being even more conservative than usual.
Note if you're interested in testing that only one or two of the proportions changed, that can be done, but is a little too involved for my phone keyboard. I'll try and come back later
posted by matildatakesovertheworld at 1:31 AM on June 18, 2014 [1 favorite]
Sort of. I'm saying the tests aren't independent in a way that goes beyond multiple comparisons. Suppose you only had two categories. If one increased by x, then the other decreased by x and so if one of your tests was significant, the other would be too. But of course both tests are testing essentially the same hypothesis, so there's no need to correct. In short, the bonferoni correction is being even more conservative than usual.
Note if you're interested in testing that only one or two of the proportions changed, that can be done, but is a little too involved for my phone keyboard. I'll try and come back later
posted by matildatakesovertheworld at 1:31 AM on June 18, 2014 [1 favorite]
Response by poster: Thanks for the follow up. I'll probably want to test 3 of the 4.
posted by 92_elements at 2:30 PM on June 19, 2014
posted by 92_elements at 2:30 PM on June 19, 2014
« Older Nice car stranger, can I interview you about it? | Offline task time tracker, stopwatch, and alarm Newer »
This thread is closed to new comments.
posted by PMdixon at 2:04 PM on June 16, 2014 [1 favorite]