Comments on: Covid-19 testing, statistics, and time
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time/
Comments on Ask MetaFilter post Covid-19 testing, statistics, and timeTue, 16 Jun 2020 10:54:21 -0800Tue, 16 Jun 2020 11:30:20 -0800en-ushttp://blogs.law.harvard.edu/tech/rss60Question: Covid-19 testing, statistics, and time
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time
Has anyone proposed a method whereby your "true" odds of having Covid-19 can be determined given the rates of false-negative/false-positive results for a particular test? <br /><br /> I know three people whose doctors said "yeah, you've got it". All three recovered, and all three later tested negative (two of them using the same IgG test, third I'm not sure of).<br>
<br>
It's been established that the test has a very significant error rate, which fluctuates between roughly 15% to nearly 70% over time, relative to the apparent infection date. Given this, how many tests would a person need to take in order to have some statistical certainty that they either did, or did not, have Covid-19 at a point in the past?<br>
<br>
For the purpose of this question, let us assume the infection dates are reasonably knowable (pretty rapid onset), but I'm not averse to running the algo multiple times using different dates.<br>
<br>
The ideal result from all this would be something like "If you are tested X times using the IgG test, the majority of the result will be the correct answer Y% of the time."<br>
<br>
It may very well be that no such answer is yet possible; I'm mainly curious given the experiences of people I know, and some light reading in the Annals of Internal Medicine which looked at test error rates over time (the tests appear to be mostly appallingly bad), which made me wonder if multiple testing and ensuing analysis could override the error rate.post:ask.metafilter.com,2020:site.345726Tue, 16 Jun 2020 10:54:21 -0800aramaiccovid19covidcoronavirusmedicaltestsmedicalstatisticsstatisticsBy: basalganglia
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time#4951612
My statistics knowledge is not quite good enough to figure out if this is do-able. Important pieces in your model would be characteristics inherent to the test (sensitivity, specificity), characteristics inherent to the population prevalence (from which you derive your false positive/false negative rates), characteristics inherent to the individual being tested (i.e. pre-test probability).<br>
<br>
But as the wise ones say: Garbage in, garbage out.comment:ask.metafilter.com,2020:site.345726-4951612Tue, 16 Jun 2020 11:30:20 -0800basalgangliaBy: MeadowlarkMaude
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time#4951621
This is exactly what Bayes theorem and likelihood ratios are for, but I don't think you have all the information you need to answer your questions. To know about the chances of a <em>particular</em> test result being a true positive or true negative, then you need to know not only the sensitivity and specificity of the test, but the base rate in the population, and if you want to be even more accurate, more information about the particular person you are discussing. If Person A is a prison doctor living in New York City (high base rates, high risk individual), a positive antibody test is more likely to be correct than for Person B, who lives in a low base rate city and has low exposure, even if they are taking the same test with the same sensitivity and specificity rate. Counterintuitively to many people, if the base rates are very low, the probability of a positive test being a true positive is also very low, even if you have a very sensitive and specific test. Conversely, if base rates are low, a negative called test is more likely to be a true negative. <br>
<br>
Adding in multiple tests to this Bayesian equation has a number of complex dimensions. First, your question presupposes that the antibody tests are independent events. But that's not necessarily true. A person who has antibodies that aren't detected by one test (false negative) may be more likely to have those antibodies fail to be detected in a second or subsequent test, such that taking additional tests won't change your posterior odds (the odds have having the disease post-test) much (i.e., it's not telling you any additional information). Or perhaps the tests all fail for the reasons under similar scenarios, etc. If you can find out that the tests are independent, then its just a matter of calculating a likelihood ratio for each test and adding them together, then multiplying by the prior odds (the odds of having the disease pre-test). If they are dependent in some way, the math gets more complicated, but the general through line is the same. There's a ton on Bayes online, not all of it good, but <a href="http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Probability/BS704_Probability6.html">this article</a> has a good example of this kind of medical math.comment:ask.metafilter.com,2020:site.345726-4951621Tue, 16 Jun 2020 11:54:37 -0800MeadowlarkMaudeBy: j_curiouser
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time#4951690
<a href="https://mobile.twitter.com/taaltree/status/1248467731545911296">Bayes Theorem and COVID with math.</a>comment:ask.metafilter.com,2020:site.345726-4951690Tue, 16 Jun 2020 15:11:25 -0800j_curiouserBy: zengargoyle
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time#4951707
3blue1brown gives an explanation of <a href="https://www.youtube.com/watch?v=HZGCoVF3YvM">Bayes theorem</a> if you want something a bit more visual.<br>
<br>
MeadowlarkMaude is right on point. The usual medical Bayes explanation is usually breast cancer, or some other medical problem that we've studied long enough to know the base rate of entire populations and different groups of base risk factors. That's needed to fill in the whole negative-negative, negative-positive, positive-negative, positive-positive table of outcomes.comment:ask.metafilter.com,2020:site.345726-4951707Tue, 16 Jun 2020 15:56:19 -0800zengargoyleBy: zengargoyle
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time#4951713
Ah, I knew there was one about medical screening: <a href="https://www.youtube.com/watch?v=Vkv8dDAAeF0">Can We Trust Maths? - with Kit Yates</a>. The first part is medical and screening for breast cancer.comment:ask.metafilter.com,2020:site.345726-4951713Tue, 16 Jun 2020 16:27:31 -0800zengargoyleBy: peacheater
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time#4951735
The FDA actually provides a handy-dandy calculator on their page to estimate the Positive Predictive Value and Negative Predictive Value given the sensitivity and specificity of a given test. On <a href="https://www.fda.gov/medical-devices/emergency-situations-medical-devices/eua-authorized-serology-test-performance">this page</a> , scroll until you come to this paragraph: <br>
<em>Always refer to the complete instructions for use to put these estimates into the proper context and to understand how to use and interpret these tests. FDA also is providing a calculator that will allow users to see the estimated performance of a single test or two independent tests based on their performance characteristics and the estimated prevalence of SARS-CoV-2 antibodies in the target population.</em><br>
There should be a link to an Excel spreadsheet from the word calculator which will do what you are asking for (and yes, it does look like an intern put it together in an afternoon's work).comment:ask.metafilter.com,2020:site.345726-4951735Tue, 16 Jun 2020 18:34:08 -0800peacheaterBy: stripesandplaid
http://ask.metafilter.com/345726/Covid-19-testing-statistics-and-time#4952074
Yes, like peacheater mentions, you'll want to calculate the positive predictive value and the negative predictive value. These ask the questions:<br>
<br>
1. If I test positive for the test, what is the probability that I truly have the disease?<br>
2. If I test negative for the test, what is the probability that I truly don't have the disease?comment:ask.metafilter.com,2020:site.345726-4952074Wed, 17 Jun 2020 17:55:35 -0800stripesandplaid