How do I learn to read peer-reviewed studies comfortably?

May 28, 2013 6:31 PM

I come from an engineering background rather than a research background, and I find myself lacking in vocabulary when it comes to understanding research papers, particularly when they start talking about ANOVA analyses, F(x) effect sizes and p values. I can skim through the results of a study and see that certain numbers are bigger than other numbers, but I don't really know how to tell whether what I'm seeing is significant. I'm guessing that I'm missing basic education in statistics. Can I fix this in a simple way?
posted by sdis to Education (9 answers total)

Reading through a basic undergrad level quantitative Research Methods text should get you up to speed well enough, I'd think. A used copy from the past few years would be fine - too old and it's less likely to address effect sizes or advanced correlational models.
posted by bizzyb at 6:52 PM on May 28, 2013

I read a lot of these sorts of papers in grad school, but my own work didn't require that sort of statistical rigor, so I spent a while trying to get better at that sort of stuff.

Ultimately, I sort of gave up. Yes, you could learn that stuff eventually, but the easier approach is simply to take the analysis on face value and hope that peer review is doing its job at catching the obvious errors. You're not going to get savvy enough to really dissect these analyses without a ton of work. If the paper is well written, they'll tell you which results are significant and which hypotheses were proven or disproven. In my experience, if you literally skip the entire analysis section you're not missing that much. Exactly how they created their factors for the factor analysis and exactly how they constructed their sample and normalized for different biases is usually not that important. If they have a big result that's significant and has a notable effect size, they'll spend at least few hundred words talking about why that's important. If there's any major flaws in the approach, their reviewers will probably catch is and force them to address it in somewhat more accessible language somewhere in the paper to justify their decision.

Obviously, YMMV if you're reading somewhat less carefully reviewed papers. But unless you're trying to actually repeat the analyses, I think you can get 90% of the value from a paper from the first third and the last third of it.
posted by heresiarch at 7:32 PM on May 28, 2013 [1 favorite]

Uh, yeah, read a basic statistics primer like Statistics in a Nutshell. It'll cover the basics of the enterprise of statistics (what it is and what it isn't), give you an appreciation for the most popular models (generally, ANOVA, linear regression, and logistic regression), and give you the vocabulary to evaluate fit. Some alternative approaches have been gaining in popularity, but it will be a long while before authors will start assuming familiarity with them on the reader's part, at least outside of some narrow niches.
posted by Nomyte at 8:07 PM on May 28, 2013

If you're reading actual research papers, as in stuff that's appearing in journals and intended for other professional dorks, then I would guess it's unlikely that an undergraduate research methods text will tell you what you need to know.

Basically you want the online equivalent for the first one or two graduate methods courses in whatever field you're reading the papers of. The math should be pretty trivial for an engineer.

Some alternative approaches have been gaining in popularity, but it will be a long while before authors will start assuming familiarity with them on the reader's part, at least outside of some narrow niches.

That must be something that varies by field. In my own field, articles are likely to just say "We ran a negative binomial model because of overdispersion and blah foo wibble" or "We ran a multinomial logit with fixed effects for year and we won't tell you the coefficients but here are five plots of marginal effects that don't explain what marginal effects are except we made them using Clarify so fuck you." There's even a good reason for that; journal space is too scarce to allow every article that uses methods that aren't truly bleeding edge to waste space describing them.
posted by ROU_Xenophobe at 9:48 PM on May 28, 2013 [1 favorite]

If the paper is well written, they'll tell you which results are significant and which hypotheses were proven or disproven. In my experience, if you literally skip the entire analysis section you're not missing that much. Exactly how they created their factors for the factor analysis and exactly how they constructed their sample and normalized for different biases is usually not that important.

i agree and disagree with this. yes, if it's well written it will tell you what they think is significant and what is not. but, actually, the analysis section is the most important part, at the graduate level. (at the undergraduate level, just read the abstract and finish your paper. there is beer that needs drinking.) exactly what question they asked, and how they did it, is very important to you, because this will tell you how relevant or not the research is you you. you can also see where your own research takes what's been done before and extends it to something new.

are you doing this reading for fun? have you changed careers? the answers will determine to what extent you need to dive into X statistical subject.
posted by cupcake1337 at 11:33 PM on May 28, 2013

Response by poster: I'm basically trying to pick up the job of a science journalist; I'm running a blog, and would like to read through most of the available research on a couple topics (memory, language learning) and summarize any relevant bits for my readers. But I often find that science journalism is relatively irresponsible when it comes to numbers, and I'd like to report results in a meaningful, informed way in my posts.
posted by sdis at 11:48 PM on May 28, 2013

You are correct, you're missing the basic statistics background. +1 that a graduate level research methods course is what you need, but you may be able to pick up some through self-teaching given that you have an engineering background. Try the Khan Academy courses.

I don't agree that you should skip the analysis section, you cannot assume the authors are reporting their own study in an unbiased way. I've read studies that seem to cherry-pick results and that report only very specific hypotheses that support their conclusion in the face of other more substantial evidence against their assumptions. For your purposes (which sound kind of like this blog but in a different field, correct?) you will definitely want to be able to understand the authors' findings for yourself.
posted by epanalepsis at 6:57 AM on May 29, 2013

That must be something that varies by field. In my own field, articles are likely to just say "We ran a negative binomial model because of overdispersion and blah foo wibble" or "We ran a multinomial logit with fixed effects for year and we won't tell you the coefficients but here are five plots of marginal effects that don't explain what marginal effects are except we made them using Clarify so fuck you."

Absolutely, but the author is asking about ANOVA and effect sizes, which suggests that at least a good chunk of the material involves basic linear models.
posted by Nomyte at 8:49 AM on May 29, 2013 [2 favorites]

Coursera also offers classes in both statistics and biostatistics (which would probably be more useful for studies that come out of the medical literature).

You might also benefit from reading a book on research design (or taking a free class) which would demonstrate some of the pitfalls that are easy to fall into when you're setting up an experiment or a study, especially if you're working with cohort or observational data and not randomized controlled trials.
posted by The Elusive Architeuthis at 2:55 PM on May 29, 2013