Stats for Experiments Refresher Needed

September 2, 2022 10:16 PM

I'm going to be doing a test for a job on Wednesday. The test will involve doing statistics for randomized controlled trials. I do other kinds of statistics, but I haven't done so much as a problem set like this in 35 years. Please point me to some primers to refresh my memory on how to do them and how to interpret outputs. I assume I want t-tests, means comparisons, ANOVA etc. SPSS, Stata, or R. And maybe excel. Do people use excel?

I can do all kinds of advanced statistics. But I haven't ever had the chance to analyze randomized experiment data. So I learned about it in school many years ago but haven't never practiced and I was in school many years ago.

I assume I want to know how to do means comparisons, t-tests, cross-tabs/chi-square, and ANOVA, but if those aren't the right things, then please tell me what the right things are. I think the likely software candidates are SPSS, STATA, or R, but this is not a stats lab and the job ad did include the word "Excel" so maybe I should know how to do it that way too?

I need to know how to do it and then how to interpret outputs. I seem to recall it mentioned many times that interaction effects in ANOVA are interpreted very differently from interaction effects in multivariate regression models. Please something on interpreting interaction effects in ANOVA.

I'm open to tests, marked up syntax or output files or videos. I would especially love cheat sheets I can print out to have on hand. I have access to linkedin learning. Because I actually DO understand statistics, I don't want to spend 30 hours on this. Just enough to go "Oh yeah, I remember now...". I think 5 hours would probably be plenty for that.

posted by anonymous to Work & Money (3 answers total)

I used to like Andy Field's "statistics hell" site.
It is now redeployed less infernally as discovering statistics.

zip files for data / Youtube for walkthroughs / pdfs for using SPSS.

Speaking of SPSS, I recently used Jamovi for the first time, and found the downloadable guide better than my old SPSS texts.

Hope this helps!
posted by Calvin and the Duplicators at 11:17 PM on September 2, 2022 [1 favorite]

PM me or have a mod reach out. This is basically the content that was covered in my stats class for my master in public health and I have a google doc “cheat shear” that might be a good start for you (or could be too low level, but would take about 3 minutes to figure out if it’s good for you or not
posted by raccoon409 at 5:15 AM on September 3, 2022

Hard to answer without more context: what industry, what the position expects, what your work experience is. If I was hiring a senior statistician (35 years experience ...) today to work on RCTs, I would like them to understand and be able to figure out even if they don't remember the details off hand, in addition to the basics you mention:

• estimands beyond ITT, inference with non-adherence, analyses like instrumental variables
• covariance adjustment and parametric models (i.e. the Freedman type results)
• block-randomized design analysis, limitations of fixed effects
• group-randomized and stepped-wedge design analysis
• methods for survival data
• false discovery rate control, sequential hypothesis testing, and other multiple testing strategies
• trial sequential analysis, basic early stopping criteria
• analytic and inferential reproducibility strategies and workflows
• power and sample size calculations including simulation-based for complex designs

But that's in the context of complex human study designs in a big academic clinical trial center. We don't have a lot of appetite for adaptive or Bayesian trial designs, and I think of that as pretty sub-specialized. If I was looking for analysis of randomized A/B website designs or psychology experiments, I doubt you would need any of that! If I was supporting rodent studies where the randomization is almost incidental, a small subset of those (FDR, simple sample sizes) would be enough.

and the job ad did include the word "Excel" so maybe I should know how to do it that way too
There are people that use excel. It is often used as an exchange format for medium-sized data and to prototype basic reporting / figures. Groups that don't need FDA-level documentation often use it for data collection. In an MS focused group, PowerQuery or BI familiarity would signify not being a novice. Excel may have been included as part of standard language (expertise in office software such as ...) without expecting for you to do meaningful analysis in it.
posted by a robot made out of meat at 11:49 AM on September 4, 2022 [1 favorite]