What counts as a high quality study?
June 25, 2020 8:42 AM Subscribe
Are there high-quality studies that show that turmeric is anti-inflammatory?
(I studied history in college, so I never really got the hang of evaluating scientific studies before they told me I'd taken enough classes to graduate and had to go.)
I look at the studies I can find on turmeric and it seems to me like it's pretty clear that turmeric has anti-inflammatory properties. I know that in my own personal experience that turmeric is anti-inflammatory on par with children's Ibuprofen. I also know about the placebo effect.
I believe that there are editors on Wikipedia who are excessively opposed to herbalism. What standards are Wikipedia editors using to make these decisions about what is a good enough study? What other high-quality standards are used in assessing herbal medicines?
My question is: How do you decide whether something is a high quality study? And my example that I would like explained (because I do really well with examples) is herbalism, specifically whether studies show that turmeric is antiinflammatory.
Bonus: Is there a current science-based encyclopedic source of herbal medicines on the web anywhere?
Thanks!
(I studied history in college, so I never really got the hang of evaluating scientific studies before they told me I'd taken enough classes to graduate and had to go.)
I look at the studies I can find on turmeric and it seems to me like it's pretty clear that turmeric has anti-inflammatory properties. I know that in my own personal experience that turmeric is anti-inflammatory on par with children's Ibuprofen. I also know about the placebo effect.
I believe that there are editors on Wikipedia who are excessively opposed to herbalism. What standards are Wikipedia editors using to make these decisions about what is a good enough study? What other high-quality standards are used in assessing herbal medicines?
My question is: How do you decide whether something is a high quality study? And my example that I would like explained (because I do really well with examples) is herbalism, specifically whether studies show that turmeric is antiinflammatory.
Bonus: Is there a current science-based encyclopedic source of herbal medicines on the web anywhere?
Thanks!
The best evidence-based encyclopedic source for herbal and natural medicines is probably the (paywalled) database Natural Medicines, available at most medical libraries and some public and academic libraries. I no longer have access to it, or I would look up turmeric for you.
The National Center for Complementary and Integrative Health is also a good source of info, but their page on turmeric is pretty sketchy, but does have links to articles that they consider significant (and which would probably also be considered signficant by wikipedians).
"High quality" is a tricky concept in evidence-based medicine, because something can be a high-quality study (as mark k describes above) but still not tell us anything particularly significant for patient health. A lot of studies to a great job of measuring things that you can measure via bloodwork or see in the lab or in mice but which do not necessarily translate to improved patient outcomes. With "inflammation" it's tricky to know what to measure - there are markers associated with inflammation, but do they necessarily correlate to improved patient outcomes (legit question - I have no idea)?
posted by mskyle at 9:11 AM on June 25, 2020
The National Center for Complementary and Integrative Health is also a good source of info, but their page on turmeric is pretty sketchy, but does have links to articles that they consider significant (and which would probably also be considered signficant by wikipedians).
"High quality" is a tricky concept in evidence-based medicine, because something can be a high-quality study (as mark k describes above) but still not tell us anything particularly significant for patient health. A lot of studies to a great job of measuring things that you can measure via bloodwork or see in the lab or in mice but which do not necessarily translate to improved patient outcomes. With "inflammation" it's tricky to know what to measure - there are markers associated with inflammation, but do they necessarily correlate to improved patient outcomes (legit question - I have no idea)?
posted by mskyle at 9:11 AM on June 25, 2020
Sidestepping what makes an individual experimental study especially convincing or not, there are two other kinds of scientific articles that have very high value both in the scientific community and also for the public/practitioners/pop-sci writers. Those are review articles andmeta-analysis studies.
Review articles summarize and synthesize the results of many other studies related to a topic. If they are published relatively recently in a good venue, they are a pretty reliable source of "what do we know about this topic". Two I found for turmeric are here and here.
Meta-analyses are studies that more systematically analyze the results presented in other studies. This allows researchers to mimmick some aspects of doing a very large scale study that is not viable due to e.g. cost and time considerations.Here is a 2016 article that is both a review and meta-analysis, narrowly focused on randomized control trials testing turmeric in providing relief of joint arthritis. Here's a 2019 meta-analysis which is focused on how turmeric affects inflammatory markers in chronic inflammatory disease.
Quoting the 2016 meta-analysis: In conclusion, these RCTs provide scientific evidence that supports the efficacy of turmericextract (about 1000 mg/day of curcumin) in the treatment of arthritis. However, the total number of RCTs included in the analysis,the total sample size, and the methodological quality of the primary studies were not sufficient to draw definitive conclusions
The 2019 meta-analysis says: Turmeric or curcumin did not decrease several inflammatory markers in patients with chronic inflammatory diseases.
So, for me, as a whole, these are a good set of material that can well inform a scientific opinion. It seems there is a lot of evidence for anti-inflammatory properties of turmeric, from many angles, but there are too many variables to allow us to support a blanket statement like "turmeric is a effective and proven treatment for X ailment". It's also worth noting that the two meta-analyses aren't even really in contradiction. The first is using pain scores and an osteoarthritis index, the other is looking at specific markers, and it's entirely possible e.g. that turmeric can really reduce pain comared to placebo, but not involve the markers that were used in the second meta-analysis.
The advantage here is that we're looking at bodies of work, rather than just single experiments. And additionally, when we look at a handful of reviews and meta-analyses, it is unlikely that there's much "bad science" creeping in. Even if there are many flawed studies on turmeric out there, the four works I link above together summarize maybe a hundred studies, and it is extremely unlikely that they are all rubbish. One thing to keep in mind is that drug companies don't throw much money at herbal studies, so there's just much less work done on them, because there's much less profit motive.
And just for the heck of it, here's a 2013 review article that cites dozens of studies showing anti-inflammatory effects of cururmin-free turmeric, indicating many other bioactive compounds are at play.
posted by SaltySalticid at 9:48 AM on June 25, 2020 [8 favorites]
Review articles summarize and synthesize the results of many other studies related to a topic. If they are published relatively recently in a good venue, they are a pretty reliable source of "what do we know about this topic". Two I found for turmeric are here and here.
Meta-analyses are studies that more systematically analyze the results presented in other studies. This allows researchers to mimmick some aspects of doing a very large scale study that is not viable due to e.g. cost and time considerations.Here is a 2016 article that is both a review and meta-analysis, narrowly focused on randomized control trials testing turmeric in providing relief of joint arthritis. Here's a 2019 meta-analysis which is focused on how turmeric affects inflammatory markers in chronic inflammatory disease.
Quoting the 2016 meta-analysis: In conclusion, these RCTs provide scientific evidence that supports the efficacy of turmericextract (about 1000 mg/day of curcumin) in the treatment of arthritis. However, the total number of RCTs included in the analysis,the total sample size, and the methodological quality of the primary studies were not sufficient to draw definitive conclusions
The 2019 meta-analysis says: Turmeric or curcumin did not decrease several inflammatory markers in patients with chronic inflammatory diseases.
So, for me, as a whole, these are a good set of material that can well inform a scientific opinion. It seems there is a lot of evidence for anti-inflammatory properties of turmeric, from many angles, but there are too many variables to allow us to support a blanket statement like "turmeric is a effective and proven treatment for X ailment". It's also worth noting that the two meta-analyses aren't even really in contradiction. The first is using pain scores and an osteoarthritis index, the other is looking at specific markers, and it's entirely possible e.g. that turmeric can really reduce pain comared to placebo, but not involve the markers that were used in the second meta-analysis.
The advantage here is that we're looking at bodies of work, rather than just single experiments. And additionally, when we look at a handful of reviews and meta-analyses, it is unlikely that there's much "bad science" creeping in. Even if there are many flawed studies on turmeric out there, the four works I link above together summarize maybe a hundred studies, and it is extremely unlikely that they are all rubbish. One thing to keep in mind is that drug companies don't throw much money at herbal studies, so there's just much less work done on them, because there's much less profit motive.
And just for the heck of it, here's a 2013 review article that cites dozens of studies showing anti-inflammatory effects of cururmin-free turmeric, indicating many other bioactive compounds are at play.
posted by SaltySalticid at 9:48 AM on June 25, 2020 [8 favorites]
What standards are Wikipedia editors using to make these decisions about what is a good enough study?
The standards they should be following are described at WP:MEDRS. However, just because a study meets those criteria doesn't mean it will be included. If you have a lot of time and patience, you can attempt to address these issues yourself, i.e. be WP:BOLD.
posted by SaltySalticid at 10:19 AM on June 25, 2020 [1 favorite]
The standards they should be following are described at WP:MEDRS. However, just because a study meets those criteria doesn't mean it will be included. If you have a lot of time and patience, you can attempt to address these issues yourself, i.e. be WP:BOLD.
posted by SaltySalticid at 10:19 AM on June 25, 2020 [1 favorite]
Is there a current science-based encyclopedic source of herbal medicines on the web anywhere?
Memorial Sloan Kettering has a database on herbs and supplements that provides science-based information. It's provides information on what supplements and herbs are used for as well as a summary of the evidence and contraindications. I use it all the time. The section that's labeled "For healthcare professionals" provides references for their information.
posted by FencingGal at 1:35 PM on June 25, 2020 [3 favorites]
Memorial Sloan Kettering has a database on herbs and supplements that provides science-based information. It's provides information on what supplements and herbs are used for as well as a summary of the evidence and contraindications. I use it all the time. The section that's labeled "For healthcare professionals" provides references for their information.
posted by FencingGal at 1:35 PM on June 25, 2020 [3 favorites]
This is a bit of an indirect answer, but you might find the US Preventive Services Task Force guidelines on evidence interesting/helpful in assessing studies: https://www.uspreventiveservicestaskforce.org/uspstf/grade-definitions
USPSTF is the group that decides whether various health preventive services (screenings, etc) should be used.
posted by john_snow at 1:54 PM on June 25, 2020
USPSTF is the group that decides whether various health preventive services (screenings, etc) should be used.
posted by john_snow at 1:54 PM on June 25, 2020
The only way to test whether an intervention works or not is through a randomised controlled trial (RCT). A single RCT typically does not provide enough evidence to change practice. So we combine multiple RCTs in a systematic review and meta-analysis, as mentioned above. Any systematic review and meta-analysis will define a priori a set of inclusion and exclusion criteria (so, for example, humans only, no mice, adults not children, etc) that are applied rigorously.
Then, the studies that are included are assessed for study quality aka risk of bias. The most common tool for this is the Cochrane Risk of Bias Tool. Another method for assessing the quality or certainty of evidence is the Grading of Recommendations Assessment, Development and Evaluation (GRADE). GRADE can be applied to RCTs as well as observational studies.
Depending on the specific outcome of interest, the final component of an assessment of 'does something work' may take into consideration the minimum clinically important difference (MCID). Many validated questionnaires ('instruments') have a defined MCID. If an RCT finds that there is a difference in the outcome of interest between the intervention and control groups, it may be:
1. small and not meaningful (i.e. a 1-point difference on a 100-point scale is unlikely to be clinically meaningful)
2. small and statistically significant, but not clinically meaningful i.e. the difference is smaller than the MCID (i.e. a 1-point difference on a 100-point scale in a big study can be statistically signficant, but is not likely to be meaningful to a patient)
3. at least as big as the MCID, so the difference is likely to be clinically meaningful.
posted by lulu68 at 4:07 PM on June 25, 2020 [2 favorites]
Then, the studies that are included are assessed for study quality aka risk of bias. The most common tool for this is the Cochrane Risk of Bias Tool. Another method for assessing the quality or certainty of evidence is the Grading of Recommendations Assessment, Development and Evaluation (GRADE). GRADE can be applied to RCTs as well as observational studies.
Depending on the specific outcome of interest, the final component of an assessment of 'does something work' may take into consideration the minimum clinically important difference (MCID). Many validated questionnaires ('instruments') have a defined MCID. If an RCT finds that there is a difference in the outcome of interest between the intervention and control groups, it may be:
1. small and not meaningful (i.e. a 1-point difference on a 100-point scale is unlikely to be clinically meaningful)
2. small and statistically significant, but not clinically meaningful i.e. the difference is smaller than the MCID (i.e. a 1-point difference on a 100-point scale in a big study can be statistically signficant, but is not likely to be meaningful to a patient)
3. at least as big as the MCID, so the difference is likely to be clinically meaningful.
posted by lulu68 at 4:07 PM on June 25, 2020 [2 favorites]
« Older Can I deduct a backyard office "shed"? | How do you recommend I document (or not) this... Newer »
This thread is closed to new comments.
There's a lot in there so some things that are perhaps legitimate and useful science but not necessarily convincing in that sense:
- Studies on mice or other animal models
- Studies on humans that measure, say, cyotkine production in undergrads who have just taken turmeric
- Studies on a half-dozen people
- Studies with no control arms
- Studies that reported a really big effect that didn't make biological or observational sense
- Studies that used endpoints that are non-standard in the inflammation field
- Studies where the researchers may have tested multiple endpoints, then reported only the one that worked
All of these happen in peer reviewed papers but signs for skepticism. Some (like studying mice) are fine but you can only draw so many conclusions from them; others are really big warning signs. You generally want a whole set of study results that are all showing consistent effects before you believe something is scientifically demonstrated.
I believe the prevailing wisdom is that if turmeric does something it's because of curcumin (a chemical it contains) so a good study might also show anti-inflammatory effects correlate with curcumin absorbed from the dose.
To be clear, I'm listing things that are pretty stringent. There's probably a lot you could discuss just on the topic of whether a small mouse study is wikipedia-worthy; the answer doesn't seem like it should be "automatically no."
posted by mark k at 9:07 AM on June 25, 2020 [3 favorites]