How does clinical trial efficacy work (simply put)?
December 1, 2020 4:05 PM Subscribe
Confused about the data provided on the recent COVID studies, and why they mean vaccines were judged to be 95% effective
The Pfizer and Moderna press releases (Pfizer's linked here) provide three main pieces of data: the number of trial participants, the number of participants who contracted the virus, and the number of participants contracting the virus who were in the control group, receiving a placebo. (Apologies for any mistakes in terminology or anything else I'll be making here.)
So for Pfizer's trial, there were more than 43,000 participants, 170 of them contracting COVID, and 8 of them were in the vaccine group, meaning 162 were in the placebo group. 8/162 is ~5%.
The press releases don't say that this is why the vaccines are judged to be 95% effective, though papers like the Washington Post connect the dots. But how is this not the wrong way to judge effectiveness? Wouldn't the right way be to say 162 out of 16,200, e.g., taking the placebo contracted it, and that's 1%, and 8 of let's say 16,200 receiving the vaccine contracted it, and that's 5% of 1%, meaning the vaccine is 95% effective?
The Pfizer and Moderna press releases (Pfizer's linked here) provide three main pieces of data: the number of trial participants, the number of participants who contracted the virus, and the number of participants contracting the virus who were in the control group, receiving a placebo. (Apologies for any mistakes in terminology or anything else I'll be making here.)
So for Pfizer's trial, there were more than 43,000 participants, 170 of them contracting COVID, and 8 of them were in the vaccine group, meaning 162 were in the placebo group. 8/162 is ~5%.
The press releases don't say that this is why the vaccines are judged to be 95% effective, though papers like the Washington Post connect the dots. But how is this not the wrong way to judge effectiveness? Wouldn't the right way be to say 162 out of 16,200, e.g., taking the placebo contracted it, and that's 1%, and 8 of let's say 16,200 receiving the vaccine contracted it, and that's 5% of 1%, meaning the vaccine is 95% effective?
Epidemiologist here, but IANYE. You can actually completely ignore the size of the participant group (43,000 or 21,500 in each arm - placebo and vaccine) for the calculation you are trying to do. Because the two groups are matched, meaning similar in all other respects other than having been given the vaccine, the number of cases in the placebo group (162) represents the number of COVID cases you would EXPECT to see if no vaccine existed. The number of cases actually observed (8) shows what happens when the vaccine was given. You can then compare what you did get (8) to what you would expect (162) and -- as you note -- 8/162 is about 5%. So in other words, the vaccine eliminated about 95% of the COVID cases you would have expected. As a thought experiment, if COVID rates were much, much higher you might have had 1,000 cases in the placebo group, and only 50 in the vaccine group - you still get the same efficacy rate: the vaccine is preventing 95% of the cases you would expect to see. The efficacy rate of the vaccine isn't related to the underlying rate of disease in the population. (A note: that is very different than testing for COVID, where the accuracy of the test hugely relies on the rate of the disease in the population.)
posted by sonofsnark at 4:19 PM on December 1, 2020 [12 favorites]
posted by sonofsnark at 4:19 PM on December 1, 2020 [12 favorites]
Statistician here.
I think the formula is 1 - RR, or 1 - P(sick|vaccine)/P(sick|placebo).
= 1 - [P(sick&vaccine)/P(vaccine)]/[P(sick&placebo)/P(placebo)]
In a balanced trial where P(vaccine) = P(placebo), this will reduce to
= 1 - N(sick&vaccine)/N(sick&placebo)
= 1 - 8/162.
posted by eirias at 4:22 PM on December 1, 2020 [2 favorites]
I think the formula is 1 - RR, or 1 - P(sick|vaccine)/P(sick|placebo).
= 1 - [P(sick&vaccine)/P(vaccine)]/[P(sick&placebo)/P(placebo)]
In a balanced trial where P(vaccine) = P(placebo), this will reduce to
= 1 - N(sick&vaccine)/N(sick&placebo)
= 1 - 8/162.
posted by eirias at 4:22 PM on December 1, 2020 [2 favorites]
So for Pfizer's trial, there were more than 43,000 participants, 170 of them contracting COVID, and 8 of them were in the vaccine group, meaning 162 were in the placebo group. 8/162 is ~5%.
The placebo resulted in 162 cases.
If the vaccine was no better than the placebo, there would also be 162 cases in the vaccine group. But there were only 8 cases in the vaccine group.
Therefore, there were 8/162=0.049 (~5%) of the cases that we expected, so the vaccine stops 95% of cases occurring.
As sonofsnark says, it doesn't matter how many participants there are in the trial. There could have been 1 million people taking the vaccine and 1 million on the placebo - if 162 on the placebo get covid, and 8 on the vaccine get it, the vaccine is still 95% effective.
posted by EndsOfInvention at 4:25 PM on December 1, 2020 [1 favorite]
The placebo resulted in 162 cases.
If the vaccine was no better than the placebo, there would also be 162 cases in the vaccine group. But there were only 8 cases in the vaccine group.
Therefore, there were 8/162=0.049 (~5%) of the cases that we expected, so the vaccine stops 95% of cases occurring.
As sonofsnark says, it doesn't matter how many participants there are in the trial. There could have been 1 million people taking the vaccine and 1 million on the placebo - if 162 on the placebo get covid, and 8 on the vaccine get it, the vaccine is still 95% effective.
posted by EndsOfInvention at 4:25 PM on December 1, 2020 [1 favorite]
Wouldn't the right way be to say 162 out of 16,200, e.g., taking the placebo contracted it, and that's 1%, and 8 of let's say 16,200 receiving the vaccine contracted it, and that's 5% of 1%, meaning the vaccine is 95% effective?
That's the same thing. You're just dividing both numerator and denominator y 16200.
posted by GCU Sweet and Full of Grace at 5:00 PM on December 1, 2020 [5 favorites]
That's the same thing. You're just dividing both numerator and denominator y 16200.
8 8/16200 --- = ------------- 162 162/16200
posted by GCU Sweet and Full of Grace at 5:00 PM on December 1, 2020 [5 favorites]
What you're describing is important in trials where the treatment and control arms aren't balanced, though.
If you're confident in your treatment (and regulators agree), and you put 2/3 of participants in the treatment arm and only 1/3 in the control arm, *then* it would be unreasonable to directly compare 162 in the control arm to 8 in the treatment arm.
posted by GCU Sweet and Full of Grace at 5:13 PM on December 1, 2020 [3 favorites]
If you're confident in your treatment (and regulators agree), and you put 2/3 of participants in the treatment arm and only 1/3 in the control arm, *then* it would be unreasonable to directly compare 162 in the control arm to 8 in the treatment arm.
posted by GCU Sweet and Full of Grace at 5:13 PM on December 1, 2020 [3 favorites]
You can actually completely ignore the size of the participant group (43,000 or 21,500 in each arm - placebo and vaccine)
Non-epidemiologist here, but wouldn't the size of the group (proportion of cases) impact the confidence interval? i.e. a 95% CI of, say, 92-96 is very different from a 95% CI of 80-99.
posted by basalganglia at 5:20 PM on December 1, 2020 [1 favorite]
Non-epidemiologist here, but wouldn't the size of the group (proportion of cases) impact the confidence interval? i.e. a 95% CI of, say, 92-96 is very different from a 95% CI of 80-99.
posted by basalganglia at 5:20 PM on December 1, 2020 [1 favorite]
Great question, @basalganglia, and yes. But you are right that what would be impacted is the confidence interval and not the efficacy rate.
posted by sonofsnark at 5:50 PM on December 1, 2020
posted by sonofsnark at 5:50 PM on December 1, 2020
sonofsnark, that is not right. Imagine a trial with 10 participants in one arm and 10,000 in the other. In each group, 10 participants get sick. Which vaccine do you want?
Wikipedia’s entry on vaccine efficacy.
posted by eirias at 6:12 PM on December 1, 2020
Wikipedia’s entry on vaccine efficacy.
posted by eirias at 6:12 PM on December 1, 2020
eirias, I believe sonofsnark is referring specifically to differences between studies with different numbers of participants but still with a balanced design, such that the different groups within the study are matched. In that case the mean would be independent of the number of participants.
posted by biogeo at 7:13 PM on December 1, 2020
posted by biogeo at 7:13 PM on December 1, 2020
Response by poster: So is the key piece of information that I missed that such studies have equal sample sizes/cohorts/whatever smart people properly call them -- 50/50? E.g., of the 43,000 in Pfizer's, half got vaccines and half placebos?
posted by troywestfield at 5:06 PM on December 3, 2020 [1 favorite]
posted by troywestfield at 5:06 PM on December 3, 2020 [1 favorite]
This thread is closed to new comments.
posted by brainmouse at 4:15 PM on December 1, 2020 [5 favorites]