Comments on: Fallacies and Errors in Inferential Statistics
http://ask.metafilter.com/233950/Fallacies-and-Errors-in-Inferential-Statistics/
Comments on Ask MetaFilter post Fallacies and Errors in Inferential StatisticsSat, 26 Jan 2013 00:17:38 -0800Sat, 26 Jan 2013 00:49:53 -0800en-ushttp://blogs.law.harvard.edu/tech/rss60Question: Fallacies and Errors in Inferential Statistics
http://ask.metafilter.com/233950/Fallacies-and-Errors-in-Inferential-Statistics
I have recently been introduced to the concept of <a href="http://en.m.wikipedia.org/wiki/Pseudoreplication">pseudoreplication</a> as a mistake that people often make when using inferential statistics to evaluate treatment outcomes. My field (evolutionary and conservation biology) makes heavy use of inferential statistics, including techniques that are vulnerable to pseudoreplication, yet nowhere in my formal education have I been taught about how poor experimental design and lack of statistical rigor can lead to fallacies like this. My personal statistical proficiency is poor, but I am working to remedy that. To that end, could folks help me by identifying and ideally explaining whatever other potential pitfalls you can think of, and explaining how they can be avoided through careful experimental design and data-analysis?post:ask.metafilter.com,2013:site.233950Sat, 26 Jan 2013 00:17:38 -0800ScientiststatisticsstatslogicinferencesciencemathfallacieserrorspitfallsBy: wayland
http://ask.metafilter.com/233950/Fallacies-and-Errors-in-Inferential-Statistics#3389089
Are you familiar with <a href="http://yudkowsky.net/rational/bayes">Bayes' theorem</a>, and especially the <a href="http://en.wikipedia.org/wiki/Base_rate_fallacy">base rate fallacy</a>?comment:ask.metafilter.com,2013:site.233950-3389089Sat, 26 Jan 2013 00:49:53 -0800waylandBy: lambdaphage
http://ask.metafilter.com/233950/Fallacies-and-Errors-in-Inferential-Statistics#3389090
<i> My personal statistical proficiency is poor, but I am working to remedy that.</i><br>
<br>
Hearing a little more about this might help to offer you specific advice. The fact that you're even worried about pseudoreplication at all suggests that it might not be as bad as you think?<br>
<br>
<i>To that end, could folks help me by identifying and ideally explaining whatever other potential pitfalls you can think of, and explaining how they can be avoided through careful experimental design and data-analysis?</i><br>
<br>
It sounds like you're asking how to get the right answers and avoid getting the wrong ones :) Although I can definitely sympathize with your desire not to send a mistake out with your name on it in the short run, I think it's ultimately more difficult to do science by treating statistics as a list of do's and don'ts. You might want to consider signing up for a statistics course, preferably a grad course for statisticians rather than a service course. This could be especially plausible if you're a grad student, but might be worth thinking about even if you're not. Rutherford, I once read, checked himself into freshman chem around the time he started work on the gold foil business. This was <i>after</i> he won the Nobel, mind you. <br>
<br>
Anyway, two more things. You're probably already familiar with multiple hypothesis testing; it seems like it's gotten some much-needed attention in recent years, but I still see issues with it all the time. A candidate from Prestigious U recently gave a job talk at our department which strongly suggested that they had never heard of it.<br>
<br>
A last point, which might be more generally applicable, is to try to develop some kind of generative model of your data and check yourself against it. The first problem discussed in Lazic 2010, for example, could have been avoided by generating like 100000 replicates of the experiment under H<sub>0</sub> in silico and comparing the distribution of the test statistic to the expected distribution given the df. Sometimes the model is too complex for that to be feasible, and sometimes it's overkill. But it's a thought.comment:ask.metafilter.com,2013:site.233950-3389090Sat, 26 Jan 2013 00:55:35 -0800lambdaphageBy: googly
http://ask.metafilter.com/233950/Fallacies-and-Errors-in-Inferential-Statistics#3389095
The differences in <a href="http://www.nature.com/neuro/journal/v14/n9/full/nn.2886.html">differences</a> fallacy (<a href="http://www.metafilter.com/108390/I-hope-this-is-all-just-incompetence">previously</a>).comment:ask.metafilter.com,2013:site.233950-3389095Sat, 26 Jan 2013 02:02:31 -0800googlyBy: paultopia
http://ask.metafilter.com/233950/Fallacies-and-Errors-in-Inferential-Statistics#3389494
Simpson's paradox is a favorite...comment:ask.metafilter.com,2013:site.233950-3389494Sat, 26 Jan 2013 12:55:12 -0800paultopiaBy: cnanderson
http://ask.metafilter.com/233950/Fallacies-and-Errors-in-Inferential-Statistics#3389807
I think you probably have these covered, but if I were to compile a list of statistical pitfalls I would definitely put these two maxims near the top:<br>
<br>
Absence of evidence is not evidence of absence. (in other words, don't interpret a negative result as proving the null).<br>
<br>
Correlation does not equal causation. (especially with observational studies, think carefully about the potential influence of variables not included in the analysis).comment:ask.metafilter.com,2013:site.233950-3389807Sat, 26 Jan 2013 19:15:18 -0800cnanderson