ANOVA says there is a significant difference, but Tukey shows nothing
April 8, 2011 3:06 PM Subscribe
I need help with SPSS- ANOVA says there is a significant difference between my variables but when I run Tukey there aren't any significant differences?
I have a paper due tonight and this is bugging me. When I run a t-test it is also significant. Is SPSS acting wonky or are there just not any significant differences? How can I fix this so SPSS shows me the differences?
I have a paper due tonight and this is bugging me. When I run a t-test it is also significant. Is SPSS acting wonky or are there just not any significant differences? How can I fix this so SPSS shows me the differences?
Best answer: IIRC, choosing Tukey from the posthoc menu will run all pairwise contrasts and then wipe out your significant differences due to overcorrection. If you expect to find significant differences between particular levels of your variables, consider doing those simple contrasts individually. If it's really a posthoc analysis, then "significant omnibus test and no significant differences" is actually a possible outcome.
posted by Nomyte at 3:14 PM on April 8, 2011 [1 favorite]
posted by Nomyte at 3:14 PM on April 8, 2011 [1 favorite]
Best answer: I'm really rusty on stats, but I'm pretty sure Tukey isn't very robust when you are comparing groups of different sizes.
posted by Maude_the_destroyer at 3:14 PM on April 8, 2011
posted by Maude_the_destroyer at 3:14 PM on April 8, 2011
Response by poster: In a video I watched about ANOVA, some guy said that doing individual t-tests would greatly increase my chances of a type I error. Is there a post-hoc test I can do that won't wipe out the significant differences between my rather large group and the other tiny ones? Or was the video incorrect about that (or maybe I misunderstood)?
posted by tweedle at 3:19 PM on April 8, 2011
posted by tweedle at 3:19 PM on April 8, 2011
Your simplest option is doing individual t-tests using the Bonferroni correction, which is just testing at a significance level equal to your default value (usually 0.05) divided by the number of contrasts. If you have many contrasts and you're basically "fishing around" in your data, this will obviously place the bar for statistical significance very high very fast.
Again IIRC, t-tests are robust to group size differences as long as they aren't egregious (I think up to 8:1).
posted by Nomyte at 3:34 PM on April 8, 2011
Again IIRC, t-tests are robust to group size differences as long as they aren't egregious (I think up to 8:1).
posted by Nomyte at 3:34 PM on April 8, 2011
A Bonferroni correction is very conservative. You might take a look into setting a reasonable FDR. Multiple comparison corrections are discussed here.
posted by Blazecock Pileon at 4:33 PM on April 8, 2011
posted by Blazecock Pileon at 4:33 PM on April 8, 2011
I actually don't think SPSS defaults to applying a correction when you run Tukey (see, e.g., this excerpt). What I think is happening to you is, indeed, related to the fact that you have different N in your groups; this is called an "unbalanced" design. Tukey's test tends to be conservative when the design is unbalanced. [See this Wikipedia article]. You should probably use a Tukey-Kramer instead.
(This is, of course, a separate issue to the problem of multiple contrasts. I would apply the Bonferroni correction to the results of the Tukey-Kramer test. Unfortunately, based on the sound of your data, this will probably make your findings non-significant unless you are doing very few comparisons).
posted by forza at 6:24 PM on April 8, 2011
(This is, of course, a separate issue to the problem of multiple contrasts. I would apply the Bonferroni correction to the results of the Tukey-Kramer test. Unfortunately, based on the sound of your data, this will probably make your findings non-significant unless you are doing very few comparisons).
posted by forza at 6:24 PM on April 8, 2011
... Okay, I just discovered that SPSS doesn't let you do the Tukey-Kramer test. (Sorry I didn't realise that immediately; I normally use a different software called R). I still think the fact that your design is unbalanced is what is driving these results, though. If I were you I would probably do the Tukey's test, apply the Bonferroni corrections, and then explain that your post-hoc tests were non-significant because you had to adjust for multiple comparisons and because Tukey's is conservative when the design is unbalanced.
posted by forza at 6:38 PM on April 8, 2011
posted by forza at 6:38 PM on April 8, 2011
I believe forza is incorrect. In SPSS if you select Tukey for post-hoc testing, it automatically applies the Tukey-Kramer correction for unbalanced groups. Moreover, it doesn't make any sense to consider applying both Tukey/Tukey-Kramer and Bonferroni to the same data -- all of these approaches treat multiple comparisons.
Unfortunately, you have run into a problem that does relate to multiple comparisons and underpowering. Remember that the ANOVA is providing a hypothesis test result based on a GLOBAL, omnibus F statistic. When sample size is small and that F statistic is significant but not impressively so, the multiple comparison problem and any correction for it may lead to an inability to perform (potentially many) additional pair-wise comparisons.
One option based on your description of the data is to use Dunnett's test with the large group as the 'control.' This is an approach that simplifies the multiple comparisons problem by reducing the number of comparisons in your case from 6 to 3, and may help if you accept that the relevant comparisons are between the large group and the other 3. Indeed, with such small numbers in the other groups, you are really reaching to do anything else statistically, particularly with parametric assumptions and such.
posted by drpynchon at 6:53 PM on April 8, 2011 [1 favorite]
Unfortunately, you have run into a problem that does relate to multiple comparisons and underpowering. Remember that the ANOVA is providing a hypothesis test result based on a GLOBAL, omnibus F statistic. When sample size is small and that F statistic is significant but not impressively so, the multiple comparison problem and any correction for it may lead to an inability to perform (potentially many) additional pair-wise comparisons.
One option based on your description of the data is to use Dunnett's test with the large group as the 'control.' This is an approach that simplifies the multiple comparisons problem by reducing the number of comparisons in your case from 6 to 3, and may help if you accept that the relevant comparisons are between the large group and the other 3. Indeed, with such small numbers in the other groups, you are really reaching to do anything else statistically, particularly with parametric assumptions and such.
posted by drpynchon at 6:53 PM on April 8, 2011 [1 favorite]
Hm.. okay, I could be wrong, and probably shouldn't have said anything. I just got all excited about seeing a stats question and spoke too soon. I'm rusty on my SPSS (like I said, I normally use R) so drpynchon could very well be right about what it is doing. I hadn't gotten that impression from those webpages I linked to but I presume he actually uses SPSS more than I do.
/resolves to not answer SPSS questions in future
/slinks away in embarrassment
posted by forza at 8:07 PM on April 8, 2011
/resolves to not answer SPSS questions in future
/slinks away in embarrassment
posted by forza at 8:07 PM on April 8, 2011
No worries mate. Here is one relevant link about that issue in SPSS.
posted by drpynchon at 11:27 PM on April 8, 2011
posted by drpynchon at 11:27 PM on April 8, 2011
We just had a discussion about this in one of my doctoral stats classes, so here's what I can surmise - and yes, I realize that your paper is probably already turned in, but hey, it is what it is.
In SPSS, Tukey does do an alpha correction for all of the pairwise comparisons its doing, though not as severely as a Bonferroni correction across multiple t-tests. Nomyte is correct in that "significant omnibus test and no significant differences" is a totally ok outcome when running post-hocs, especially if your group sizes were weird. What you've run into is a sample size issue (blah blah, each group should have 30 or more, blah blah), but if this wasn't your data, you didn't have the choice.
Or, you run LSD post-hoc and feel dirty about doing it (don't do it).
posted by SNWidget at 6:59 AM on April 9, 2011
In SPSS, Tukey does do an alpha correction for all of the pairwise comparisons its doing, though not as severely as a Bonferroni correction across multiple t-tests. Nomyte is correct in that "significant omnibus test and no significant differences" is a totally ok outcome when running post-hocs, especially if your group sizes were weird. What you've run into is a sample size issue (blah blah, each group should have 30 or more, blah blah), but if this wasn't your data, you didn't have the choice.
Or, you run LSD post-hoc and feel dirty about doing it (don't do it).
posted by SNWidget at 6:59 AM on April 9, 2011
This thread is closed to new comments.
posted by tweedle at 3:09 PM on April 8, 2011