Please help me meta
April 24, 2020 2:38 PM Subscribe
I have never done a meta-analysis before. I am struggling to find some basic information I need and am hoping some stats-y mefites can answer my query!
I am currently data charting and calculating effect sizes for a meta-analysis using an online calculator (the Campbell Collaboration's, if anyone cares). Most of the studies are RCTs and all are controlled trials. My unit of analysis will be standardized mean differences, Cohen's d, because the measures of outcomes vary between studies. Here is my issue:
Most of the time, I can easily calculate d using means and standard deviations (or occasionally standard errors) for whatever follow up time points were used in the study. For a couple of studies however, all I have to work with are change scores, and for one I only have t-tests with p values comparing the intervention and control groups. I can still calculate Cohen's d using this alternate info. But am I correct that I cannot pool these in the meta with the standardized mean differences that were calculated using raw means? And if this is correct, could someone explain why in simple terms?
Thank you mefites for any guidance. Feel free to also post any tips and tricks that you think might be helpful for a research team attempting their first meta-analysis!
I am currently data charting and calculating effect sizes for a meta-analysis using an online calculator (the Campbell Collaboration's, if anyone cares). Most of the studies are RCTs and all are controlled trials. My unit of analysis will be standardized mean differences, Cohen's d, because the measures of outcomes vary between studies. Here is my issue:
Most of the time, I can easily calculate d using means and standard deviations (or occasionally standard errors) for whatever follow up time points were used in the study. For a couple of studies however, all I have to work with are change scores, and for one I only have t-tests with p values comparing the intervention and control groups. I can still calculate Cohen's d using this alternate info. But am I correct that I cannot pool these in the meta with the standardized mean differences that were calculated using raw means? And if this is correct, could someone explain why in simple terms?
Thank you mefites for any guidance. Feel free to also post any tips and tricks that you think might be helpful for a research team attempting their first meta-analysis!
Best answer: This book is kind of a bible on calculating effect sizes: https://us.sagepub.com/en-us/nam/practical-meta-analysis/book11092
posted by nixxon at 6:41 PM on April 24, 2020
posted by nixxon at 6:41 PM on April 24, 2020
« Older what's the thing from the 80's that makes shirts... | COVID19 bringing out unpleasant sides of people... Newer »
This thread is closed to new comments.
I'm assuming you're looking at between-group SMDs, comparing intervention vs. control at follow-up, right?* If you're using a between-group SMD, without more information from the study, it wouldn't be possible to convert from a change score--which, if you mean the difference between T2 and T1 scores, is a within-group effect size--to a between-group effect size, because it only contains information about one of the two groups. See more about this here. The t-test, in contrast, is a between-group statistic, and so you should be able to convert it into a between-group SMD and pool it in the meta with the other SMDs.
That said, as a rule, I always request supplemental data from studies that I can't code effect size info from. After emailing 300+ different people, I'm running about a 40% rate of successfully getting the data. With RCTs, you're probably working with a much smaller list of people to need to email, so I'd strongly encourage it. Some of the first emails I sent to my current collaborators were data requests for metas!
Feel free to message me, I would absolutely LOVE to help with this!
Within-group (pre-post) SMDs should be avoided for metas.
posted by quiet coyote at 3:28 PM on April 24, 2020 [4 favorites]