StatsFilter: Evidence challenging/contradicting conventional controlled/placebo trials?

July 25, 2006 3:15 PM

StatsFilter: Evidence challenging/contradicting conventional controlled/placebo trials?

I'm looking for articles, evidence, books that challenge the validity, accuracy, and/or adequacy of what's commonly been ascribed as the gold standard for clinical trials: controlled placebo trials (CPT).

I understand that the CPT has evolved to become the standard for scientific research for good reason, and that, like capitalism, is often cited as the "best system/solution when weighed against the alternatives". I also understand the hazards of anecdotal research and bias.

However, I suspect there must be some legitimate contrarian views concerning CPT, as well as some proponents of alternative research techniques.

For example, since the validity of findings of significance in such trials relies in large part on the adequacy of the size and composition of the control group, there is necessarily an assumption that the etiology underlaying the condition being examined is sufficiently similar across controls and test subjects. If the etiology is complex, consisting of multiple confounding variables, it seems that the control and test groups could in many cases, by virtue of the heterogeneity of etiologies manifesting in common symptoms, be deceptively too small.

In other words, suppose 100 individuals with a diagnosis of say severe Autism of a specific age and within a specific profile of symptoms, take Ginseng for six months, and another 100 individuals with the same diagnosis take Sugar Pills for six months. Neither group knows to which group they are assigned. The target is reducing incidences of dangerous behaviors.

Suppose results show Ginseng causes no significant decrease in dangerous behaviors. Sugar Pills also show no decrease. Now suppose that a statistically insignificant 18 individuals in the Ginseng group showed a decrease in dangerous behaviors. And a statistically insignificant 20 individuals in the Sugar Pill group showed a decrease in dangerous behaviors.

Do these results indicate that Ginseng is as ineffective as Sugar Pills? Or is it as likely that, considering the true imprecision of a diagnosis of Autism, and the possibility that within the test group of 200 individuals, there were only about 20 or so with truly similar etiologies and biological idiosynchrocies, the results tell us little of significance about the influence of Ginseng and Sugar Pills on dangerous behaviors in individuals with Autism.

Obviously the larger the size of the test groups, the less precise you need to be about the underlying causes of the conditions manifesting in common symptoms.

Certainly for things such as burns and skin wounds, it would seem the number of confounding variables may be manageable. However for conditions for which there remains relatively little understanding of the underlying genetic, biological, and environmental factors, such as MS, most Cancers, Leukemia, etc. -- it seems as though traditional CPT, unless the trial sizes are enormous, is a blunt technique.

So often, anecdotal reports of efficacy of this or that treatment are dismissed by empiricists as coincidence rather than causal. But isn't it just as likely that alternative and complimentary treatments for various conditions/individuals may indeed be effective, given the genetic/bio/environ factors present for a particular individual? But that finding is buried when the target is evaluated within in a sea of individuals with different genetic/bio/enviro conditions?

An implication of this may be that a reasonable course of trx. for individuals with conditions of unclear genetic/bio/enviro basis, is rapid-fire trial-and-error and assessment of a wide range of conventional, alternative, and complimentary treatments. This is hardly the 'wishful' thinking approach, as most believers in CPT would proclaim.

Am I missing something?

posted by pallen123 to science & nature (10 comments total)

Hróbjartsson and Götzsche published a study in 2001 and a follow-up study in 2004 questioning the nature of the placebo effect. (Hrobjartsson 2001, Hrobjartsson 2004) They performed two meta-analyses involving 156 clinical trials in which an experimental drug or treatment protocol was compared to a placebo group and an untreated group, and specifically asked whether the placebo group improved compared to the untreated group. Hróbjartsson and Götzsche found that in studies with a binary outcome (patients were classified as improved or not improved) the placebo group had no statistically significant improvement over the no-treatment group. Similarly, there was no significant placebo effect in studies in which objective outcomes (such as blood pressure) were measured by an independent observer. The placebo effect could only be documented in studies in which the outcomes (improvement or failure to improve) were reported by the subjects themselves. The authors concluded that the placebo effect does not have "powerful clinical effects," (objective effects) and that patient-reported improvements (subjective effects) in pain were small and could not be clearly distinguished from bias.

These results suggest that the placebo effect is largely subjective. This would help explain why the placebo effect is easiest to demonstrate in conditions where subjective factors are very prominent or significant parts of the problem. Some of these conditions are headache, stomach ache, asthma, allergy, tension, and the experience of pain, which is often a significant part of many mild and serious illnesses.

- Wikipedia

References:

# Hrobjartsson A, Norup M. 2003. The use of placebo interventions in medical practice--a national questionnaire survey of Danish clinicians. Eval Health Prof. 26:153-165. PMID 12789709.
# Hrobjartsson A, Gotzsche P. 2001. Is the Placebo Powerless? An Analysis of Clinical Trials Comparing Placebo with No Treatment. N Engl J Med. 344:1594-602. PMID 11372012.
# Hrobjartsson A, Gotzsche P. 2004. Is the placebo powerless? Update of a systematic review with 52 new randomized trials comparing placebo with no treatment. J Intern Med. 256:91-100. PMID 15257721
posted by furtive at 3:49 PM on July 25, 2006

I'll leave the heavy lifting to someone who knows more about clinical trials, but a statistical point:

Do these results indicate that Ginseng is as ineffective as Sugar Pills?

Nope. Strictly speaking, they show that (by the evidence given) we fail to reject the null hypothesis that the two are equivalent. Nothing more, nothing less.

Your null hypothesis is the baseline case; the default; the fallback position. For your research to be valid, one should test whether this null hypothesis is wrong; if the statistics suggest that it is, then you may feel comfortable rejecting it. If your null hypothesis was simple (x > 0, for example), then the alternative hypothesis is also simple (x < 0) and is likely to be true.br>
In one sense (the statistical one), your negative result doesn't mean a thing - it is negative in the sense that it provides no knowledge and violates no assumptions.[*] In another (the popular one), it means that ginseng == sugar. There are more complicated statistics that one may do to estimate the probability that we would observe this negative result even though the difference between ginseng and sugar is real. This is usually called the power of the statistical test (or beta), and this kind of error would be called a type II error.

If, on the other hand, you do see a significant difference, then there is an estimate for the probability that you observed this difference by chance. This is usually called alpha, and is measured as the "p-value" of your statistical test.

[*]Okay, perhaps that's overstating the case, but I think the hyperbole is useful. Negative results don't determine or define anything at all.
posted by metaculpa at 3:51 PM on July 25, 2006

Your alternative would suffer from the same problems that you criticize clinical trials for having, only to a much greater extent. If you were to try such a rapid-fire trial and error approach, you might feel better during the course of one treatment for a reason entirely unrelated to the treatment, leading to a false positive. Alternately, a sudden aggrivation of the condition during an effective treatment could negate any positive effects and lead to a false negative. Furthermore, the trial would have to be double blind to elimate bias on the part of the patient and the drug administrator.

The only effective method for determining the theraputic value of a treatment is a placebo-controlled trial. Anything less rigorous is going to be less reliable. This is why it is an unquestioned gold standard. Many treatments described as "alternative" simply fail any double blind placebo-controlled trial. Prayer, for instance, is widely believed to be an effective treatment; however, no doctor will prescribe prayer. This is because in all sufficently rigorous trials, prayer was found to do fuck all.

Basically, no, your proposal is not reasonable. Yes, clinical trials are not perfect, but this is no reason to think that snake oil is a panacea. If all conventional treatments are unsucessful, then alternative treatments become an option, but don't hold your breath. Avoid treatments that have alread repeatedly failed to show any theraputic value when tested to the standards of modern medicine. Yes, there is a slim chance that maybe the researchers were missing something important, but much more likely, it simply isn't effective.
posted by [expletive deleted] at 4:18 PM on July 25, 2006 [1 favorite has favorites]

One method of dealing with this is by using "surrogate endpoints". So, in one example, a disorder is caused by the inability of the body to break down a given substance, and its buildup in cells. (Lysosomal storage disorders are like this). Maybe you don't have a large enough population to have uniform symptoms, but you can still measure the presence of that substance in the urine, and hypothesize that lower levels imply more breakdown of that substance.

One such drug that was tested this way was Aldurazyme (alpha-L-iduronidase). The FDA wasn't thrilled with the whole surrogate marker thing, but they accepted it in the end.

That doesn't help with your varying etiology example, but I would advance the idea that a drug is intended to treat either the etiology, in which case the etiology should be somewhat uniform, or the symptoms, in which case you measure the symptoms.
posted by spaceman_spiff at 4:19 PM on July 25, 2006

Isn't this why God gave us a variety of techniques for multivariate analysis, complete with interaction terms?

If you're right, a study should show an interesting (if not necessarily 0.05-significant) result for black autistic kids, or for autistic kids with a certain genetic marker, or kids exposed to $chemical as an infant / prenatally. So, follow that up with another study limited to the relevant population, or with that population oversampled.

WRT your use of "as likely" and "just as likely," there's a big giant yawning chasm of a gap between "it is possible that alternative treatments are effective for a particular individual" and "it is just as likely that alternative treatments are effective..."
posted by ROU_Xenophobe at 4:25 PM on July 25, 2006

furtive:

I think you misunderstood. The FPP was not looking for studies that investigated the placebo effect. The FPP was wondering about controlled placebo trials as the gold standard among methods of scientific investigation.
posted by argybarg at 4:29 PM on July 25, 2006

Many areas of social science cannot conduct CPTs. As such, they have developed statistical techniques to overcome this difficulty. Thus, CPTs are not necessary, but they are certainly statistically "cleaner."

Recently there has been concern that CPTs lead to ethical concerns under certain circumstances. For example, if the test involves a treatment for a terminal condition with no known cure. The placebo group will certainly die, but the treatment group has some probability of surviving with treatment. Some argue that it would be unethical to deny one group the possibility of survival for the sake of experimental control.
posted by GarageWine at 4:53 PM on July 25, 2006

In such cases, you can just unblind the study when a discernible-enough test shows up and then (assuming it's the treatment group doing better) start everyone on the treatment.
posted by ROU_Xenophobe at 5:37 PM on July 25, 2006

Recently there has been concern that CPTs lead to ethical concerns under certain circumstances. For example, if the test involves a treatment for a terminal condition with no known cure. The placebo group will certainly die, but the treatment group has some probability of surviving with treatment.

In that case I'd have to agree that it would be irresponsible to run a randomized trial. If you're testing a treatment for a virus that, lets say, in all observed cases has resulted in death within a week, you don't really need a control group. You can give your treatment to everyone, and if some of the recipients don't end up dead at the end of the week then you've made some progress.

Unfortunatley, that kind of clear cut situation is rare. In the vast majority of cases, our best bet is to rely on the rules of logic (experimentation using the scientific method). That said, if I came down with some deadly disease I'd happily try any treatment that sounded remotely plausibly effective -- whether or not it had gone through rigorous trials. If it didn't work, then I (presumably) wouldn't be any worse off than if I'd done nothing.
posted by nixxon at 7:46 PM on July 25, 2006

I don't have references for you, although I'll look for some, but another thing to keep in mind is the difference between sugar pills and some other kind of more "active" placebo. This comes up in psychotropic med trials, where reported symptoms are the basis for evaluating and staging disorder. Basically, if you've got a double-blind study, but everyone is cautioned about side effects and only some people get side effects, it isn't too hard for doctors, and perhaps patients, to begin to deduce which arm of the study they're in. Since there aren't external markers (like blood tests) in anti-depressant trials, the observer effect is a real potential problem. (Most anti-depressants have pretty marked side effects when they are first started.)
posted by OmieWise at 10:53 AM on July 26, 2006