Statistics help
March 15, 2011 1:44 AM Subscribe
How do I calculate whether these figures are statistically significant?
How do I test for statistical significance in the following scenario?
I'm doing a speech study in which I test how often people use the grammatical variant "X" as opposed to its alternative, "Y." I want to find out whether men's and women's rates of X-usage differ. For men, I've collected a total of 1229 tokens, out of which 519 contain the X-variant (so the rate of X-use is 42.2%). For women, I've collected a total of 340 tokens, out of which 160 contain the X-variant (so the rate of X-use is 47.1%). My question is-- have I collected enough tokens to be sure that the difference in rates of X-use are statistically significant?
I've been reading on the internet about how to do T-tests (if that's indeed the right test I should be using here), but I keep failing to understand how to apply them to my particular data set. Can anyone walk me through how to do this? Any help would be much appreciated.
How do I test for statistical significance in the following scenario?
I'm doing a speech study in which I test how often people use the grammatical variant "X" as opposed to its alternative, "Y." I want to find out whether men's and women's rates of X-usage differ. For men, I've collected a total of 1229 tokens, out of which 519 contain the X-variant (so the rate of X-use is 42.2%). For women, I've collected a total of 340 tokens, out of which 160 contain the X-variant (so the rate of X-use is 47.1%). My question is-- have I collected enough tokens to be sure that the difference in rates of X-use are statistically significant?
I've been reading on the internet about how to do T-tests (if that's indeed the right test I should be using here), but I keep failing to understand how to apply them to my particular data set. Can anyone walk me through how to do this? Any help would be much appreciated.
This page on choosing the correct statistical test should provide you guidance on which test to use and how to do it in a number of different statistics software packages. The site also provides guidance on interpreting the results of statistical tests, including significance.
To help you use the table, your independent variable (IV) is gender, which is categorical with 2 levels (male and female). You have 1 dependent variable (DV), how often people use the grammatical variant "X" as opposed to its alternative "Y." One thing that's not clear to me from your description is whether your DV can be treated as categorical (you counted whether a person used X or not, so that for person 1 the value is 1 if they used it, 0 if not, regardless of how many times the person used X, and so on for all the subjects of your study) or interval (you counted how often each person used X, so that for person 1 the value may be 5, for person 2 the value may be 3, etc.). If categorical, you would use a Chi-squared test as tvdveer says. If interval (or continuous), you would use the t-test.
posted by needled at 4:58 AM on March 15, 2011 [5 favorites]
To help you use the table, your independent variable (IV) is gender, which is categorical with 2 levels (male and female). You have 1 dependent variable (DV), how often people use the grammatical variant "X" as opposed to its alternative "Y." One thing that's not clear to me from your description is whether your DV can be treated as categorical (you counted whether a person used X or not, so that for person 1 the value is 1 if they used it, 0 if not, regardless of how many times the person used X, and so on for all the subjects of your study) or interval (you counted how often each person used X, so that for person 1 the value may be 5, for person 2 the value may be 3, etc.). If categorical, you would use a Chi-squared test as tvdveer says. If interval (or continuous), you would use the t-test.
posted by needled at 4:58 AM on March 15, 2011 [5 favorites]
I asked a similar question last year and received many helpful answers. Go forth and quantify, linguist! :)
posted by iamkimiam at 5:21 AM on March 15, 2011
posted by iamkimiam at 5:21 AM on March 15, 2011
FYI, the limiting factor in the precision of this analysis tends to be the smallest entry in the table. You (at least) need a larger corpus by women. If you are observing X/Y more than once per-person, the analysis is more complicated. I assume that the contexts are all the same, or as close as you can make them.
posted by a robot made out of meat at 6:46 AM on March 15, 2011
posted by a robot made out of meat at 6:46 AM on March 15, 2011
Response by poster: This is really helpful, thanks so much! I've determined that I need to use a chi-squared test, and needled, to answer your question, the presence/absence of x is categorical. I'm using the Vassar webpage that was suggested by tvdveer and I'm still trying to understand some of the components of chi-squared. What's the relation of the "phi coefficient of association" to the chi-squared value? And (sorry to be so stupid here) I'm hoping for a "p value" that's UNDER .05 or .01 or whatever p-limit I decide is adequate if I'm hoping for the results to be statistically significant, right? Thanks again for any help!
posted by ms.codex at 12:15 PM on March 15, 2011
posted by ms.codex at 12:15 PM on March 15, 2011
For a 2×2 contingency table, a Fisher's exact test would probably be better than a chi-squared test.
Yes, you want p < 0.05, at least.
posted by grouse at 12:23 PM on March 15, 2011
Yes, you want p < 0.05, at least.
posted by grouse at 12:23 PM on March 15, 2011
The Vasser software is good, but I personally prefer OpenEpi. Though designed for Epidemiology work, it has all the relevant tools you'd need for any statistical testing. It also handily highlights the most relevant test outcome in blue, which in this case would be Fisher's Exact.
Also, since what you appear to be doing is a two-tailed test (X ≠ Y), you'll want a p-value of 0.25 or less to really be statistically significant.
posted by Panjandrum at 12:42 PM on March 15, 2011
Also, since what you appear to be doing is a two-tailed test (X ≠ Y), you'll want a p-value of 0.25 or less to really be statistically significant.
posted by Panjandrum at 12:42 PM on March 15, 2011
The descriptive statistic of choice for a 2x2 table is either the difference in frequencies or the odds-ratio. Both of these are spit out by standard stat software like stata and R. For example,
------------------------------
> x<-matrix(c(1229-519,519,340-160,160), ncol=2)
> fisher.test(x)
Fisher's Exact Test for Count Data
data: x
p-value = 0.1221
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
0.9480009 1.5587588
sample estimates:
odds ratio
1.215853
------------------------------
(A few) people argue about using fisher's test, but it's fairly standard and some journals essentially require it.
posted by a robot made out of meat at 1:27 PM on March 15, 2011
------------------------------
> x<-matrix(c(1229-519,519,340-160,160), ncol=2)
> fisher.test(x)
Fisher's Exact Test for Count Data
data: x
p-value = 0.1221
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
0.9480009 1.5587588
sample estimates:
odds ratio
1.215853
------------------------------
(A few) people argue about using fisher's test, but it's fairly standard and some journals essentially require it.
posted by a robot made out of meat at 1:27 PM on March 15, 2011
Also, since what you appear to be doing is a two-tailed test (X ≠ Y), you'll want a p-value of 0.25 or less to really be statistically significant.
I assume you mean 0.025. The default fisher.test for R (as reported by a robot made out of meat) is for a two-sided test, so you don't need to adjust your p-value. It looks like this difference is not statistically significant.
posted by grouse at 6:27 PM on March 15, 2011
I assume you mean 0.025. The default fisher.test for R (as reported by a robot made out of meat) is for a two-sided test, so you don't need to adjust your p-value. It looks like this difference is not statistically significant.
posted by grouse at 6:27 PM on March 15, 2011
Oh yeah, R is awesome, and I did mean 0.025 (or α/2, for those of you keeping score).
Point remains though, from what you are reporting, you do not have a statistically significant difference. Go collect some more data on females and let us know how that goes.
posted by Panjandrum at 11:32 PM on March 15, 2011
Point remains though, from what you are reporting, you do not have a statistically significant difference. Go collect some more data on females and let us know how that goes.
posted by Panjandrum at 11:32 PM on March 15, 2011
This thread is closed to new comments.
as your variable X is dichotomous (either present or absent), you need a Chi-squared test instead of a t-test. T-tests are for continuous variables.
The test is based on the probability of the given distribution of the variables over a 2x2 table. This website can do the test for you: http://faculty.vassar.edu/lowry/odds2x2.html
At the bottom it gives you the probabilty of the found results as the P-value.
posted by tvdveer at 3:23 AM on March 15, 2011 [2 favorites]