Context on retracted study for extremely science-y trans ally
November 27, 2024 3:15 PM Subscribe
Can anyone offer more background or context on this incident in trans research? Where does it leave the evidence for blockers improving mental health? (In case this looks like shit-stirring: I'm trans, and I'm trying to help a parent who found that article extremely frightening to read, since she has a trans kid on blockers.)
The person I'm talking to is extremely science literate and extremely evidence focused. "You can't even trust the NYT on trans issues anymore" won't fly with her. Neither will "Just listen to the AMA." She cares about the evidence, and knowing there are methodological shenanigans going on makes her extremely anxious.
What she needs is either "Here's why that study was garbage and deserved to be withheld," or "Here's why, although valid, that study doesn't threaten the consensus on blockers," or, if it really comes down to it, "Yeah, sorry, the evidence is currently mixed and the best rationale right now is doctors' own clinical experience. But in their experience, it helps."
I promise I am trans and I promise I am trying to help a trans kid. I know it is suspicious seeing this from a barely-used account, but I promise I am not trying to start shit.
The person I'm talking to is extremely science literate and extremely evidence focused. "You can't even trust the NYT on trans issues anymore" won't fly with her. Neither will "Just listen to the AMA." She cares about the evidence, and knowing there are methodological shenanigans going on makes her extremely anxious.
What she needs is either "Here's why that study was garbage and deserved to be withheld," or "Here's why, although valid, that study doesn't threaten the consensus on blockers," or, if it really comes down to it, "Yeah, sorry, the evidence is currently mixed and the best rationale right now is doctors' own clinical experience. But in their experience, it helps."
I promise I am trans and I promise I am trying to help a trans kid. I know it is suspicious seeing this from a barely-used account, but I promise I am not trying to start shit.
I am not a trans person not someone familiar with this area of research. I am a person familiar with research and how research is done and the logic of how research works and with interpreting statistics. I am an expert in research methodology within my own field, which is not this field. I have read this New York times article. I have not read any actual proper research write-up about this research.
Two things strike me.
1. First, a version of this is what came t my mind immediately: "Dr. Tishelman also noted that, even if the drugs did not lead to psychological improvements, they may have prevented some of the children from getting worse. “No change isn’t necessarily a negative finding — there could be a preventative aspect to it,” she said. “We just don’t know without more investigation.”
So the study finds that mental health didn't improve, meaning that the mental health wasn't better after going on the drugs than it was before. But I (as a person with neither lived experience, nor close personal ties to a trans person, nor scientific expertise in this area) wouldn't expect that to be the outcome anyway. Like what I would want to know wouldn't be about Person A when they were 10 years old before drugs to Person A when they were 15 after drugs. What I would want to know compare is Person A at 15 with drugs vs. Person A at 15 if they hadn't taken the drugs. Obviously you can't actually compare that which is the purpose of a randomized control group in studies that use one. So ok, people don't get better mental health. But that doesn't mean that if they hadn't gone on the drugs they wouldn't subsequently have had worse mental health years down the line.
2. The sample had poor mental health when it was measured initially at the beginning of the study and then supposedly ok mental health (at both time 1 and time 2) at the end of the study. The researcher says they're not done analyzing the data. I'm wondering if what's happened here is that there's been some non-random attrition in the study (i.e. some people left the study) and the people who left the study were the people with worse mental health (at the beginning, obviously no way of knowing what their mental health was like afterwards, because they left the study). People drop out of studies part way through all the time. This is never random. Depending on what their consent protocol looked like and what their ethics clearance looks like, it might be hard to check for this. Some ethics boards require that when someone drops out of the study you don't just stop collececting data from them you also destroy any data you previously collected from them. So if they had at the beginning of the study analyzed the first wave of data and said "hey there sure are a lot of people with poor mental health situations here" and now they're saying "people in this study have been doing pretty well all along and were doing well to begin with and were still doing well after the drugs" I'm kind of wondering if maybe some of that data on the people who weren't doing so great maybe fell out of the data set between time 1 and time 2.
posted by If only I had a penguin... at 3:52 PM on November 27 [3 favorites]
Two things strike me.
1. First, a version of this is what came t my mind immediately: "Dr. Tishelman also noted that, even if the drugs did not lead to psychological improvements, they may have prevented some of the children from getting worse. “No change isn’t necessarily a negative finding — there could be a preventative aspect to it,” she said. “We just don’t know without more investigation.”
So the study finds that mental health didn't improve, meaning that the mental health wasn't better after going on the drugs than it was before. But I (as a person with neither lived experience, nor close personal ties to a trans person, nor scientific expertise in this area) wouldn't expect that to be the outcome anyway. Like what I would want to know wouldn't be about Person A when they were 10 years old before drugs to Person A when they were 15 after drugs. What I would want to know compare is Person A at 15 with drugs vs. Person A at 15 if they hadn't taken the drugs. Obviously you can't actually compare that which is the purpose of a randomized control group in studies that use one. So ok, people don't get better mental health. But that doesn't mean that if they hadn't gone on the drugs they wouldn't subsequently have had worse mental health years down the line.
2. The sample had poor mental health when it was measured initially at the beginning of the study and then supposedly ok mental health (at both time 1 and time 2) at the end of the study. The researcher says they're not done analyzing the data. I'm wondering if what's happened here is that there's been some non-random attrition in the study (i.e. some people left the study) and the people who left the study were the people with worse mental health (at the beginning, obviously no way of knowing what their mental health was like afterwards, because they left the study). People drop out of studies part way through all the time. This is never random. Depending on what their consent protocol looked like and what their ethics clearance looks like, it might be hard to check for this. Some ethics boards require that when someone drops out of the study you don't just stop collececting data from them you also destroy any data you previously collected from them. So if they had at the beginning of the study analyzed the first wave of data and said "hey there sure are a lot of people with poor mental health situations here" and now they're saying "people in this study have been doing pretty well all along and were doing well to begin with and were still doing well after the drugs" I'm kind of wondering if maybe some of that data on the people who weren't doing so great maybe fell out of the data set between time 1 and time 2.
posted by If only I had a penguin... at 3:52 PM on November 27 [3 favorites]
Oh, and it's very difficulty in virtually any field (as far as I know) to publish null findings. So that's probably another reason (or possibly the real reason) this wasn't published.
posted by If only I had a penguin... at 3:56 PM on November 27 [1 favorite]
posted by If only I had a penguin... at 3:56 PM on November 27 [1 favorite]
FYI I read “the scientist didn’t publish this” as “the scientist didn’t release the paper/study/submit the paper to a journal” not “the scientist submitted it but it kept getting rejected because of the null finding”.
posted by MisantropicPainforest at 3:58 PM on November 27 [2 favorites]
posted by MisantropicPainforest at 3:58 PM on November 27 [2 favorites]
FYI I read “the scientist didn’t publish this” as “the scientist didn’t release the paper/study/submit the paper to a journal” not “the scientist submitted it but it kept getting rejected because of the null finding”.
Yes, I also assume it was never submitted, and note that she says she's still analyzing it, so that's another reason not to publish it. But given the amount of work it is to go from "we ran the models and know the basic answer to the main research question" to "we have put this all in publishable form and have met all the technical requirements of the journal to submit our findings for review" if you have a null finding and you know null findings have a snowballs chance in hell of getting published, you're not going to do all the work of getting a null-finding into submitable form for nothing. So "because it's a null finding" can be the real reason it's not published even if it was never submitted.
posted by If only I had a penguin... at 4:39 PM on November 27
Yes, I also assume it was never submitted, and note that she says she's still analyzing it, so that's another reason not to publish it. But given the amount of work it is to go from "we ran the models and know the basic answer to the main research question" to "we have put this all in publishable form and have met all the technical requirements of the journal to submit our findings for review" if you have a null finding and you know null findings have a snowballs chance in hell of getting published, you're not going to do all the work of getting a null-finding into submitable form for nothing. So "because it's a null finding" can be the real reason it's not published even if it was never submitted.
posted by If only I had a penguin... at 4:39 PM on November 27
Coincidentally, just an hour or so I got a newsletter from a survey called the Trans Youth Survey. My trans son and I have been participating in this study with yearly surveys (verrrry extensive and time-consuming surveys) since he was, I think, 11. He is now 17. He was on puberty blockers and then did testosterone puberty. This is a big study run by the four pre-eminent gender and sexuality development clinics in the country; my son and I are affiliated with the researcher at Lurie Children's Hospital in Chicago because he has been a patient of theirs since he was five.
The newsletter briefly covers questions about puberty blockers and hormones, and includes links to scholarly papers that have been published from its findings. This link will take you to the pdf in google drive; anyone with the link should be able to access it, but let me know if it doesn't work.
This is a longer-term, bigger study than the one discussed in the article, so you may find it useful/interesting.
posted by Well I never at 5:06 PM on November 27 [4 favorites]
The newsletter briefly covers questions about puberty blockers and hormones, and includes links to scholarly papers that have been published from its findings. This link will take you to the pdf in google drive; anyone with the link should be able to access it, but let me know if it doesn't work.
This is a longer-term, bigger study than the one discussed in the article, so you may find it useful/interesting.
posted by Well I never at 5:06 PM on November 27 [4 favorites]
What I would want to know compare is Person A at 15 with drugs vs. Person A at 15 if they hadn't taken the drugs.
Yes, exactly this.
Commentary from Erin Reed:
Yes, exactly this.
Commentary from Erin Reed:
Ultimately, the purpose of puberty blockers is not to "improve" a trans youth’s mental health. Gender dysphoria arises from a mismatch between a person’s physical development and their internal sense of gender identity. Puberty blockers pause physical changes, ensuring that when a transgender youth is ready to begin hormone therapy, as determined by their medical team, they won’t have undergone unwanted pubertal changes.posted by away for regrooving at 5:06 PM on November 27 [2 favorites]
...
“TGD youth who received pubertal blockade at Tanner 2 or 3 were found to have less anxiety, depression, stress, total problems, internalizing difficulties, and suicidal ideation than TGD peers who had been through more of a nonaffirming puberty.”
To add: the newsletter from the Trans Youth Survey mentions that the results can be somewhat skewed because, by definition, participants have supportive parents who have helped them access gender affirming care; it doesn't necessarily have much to say about youth who don't have supportive adults or who can't access care.
posted by Well I never at 5:08 PM on November 27 [2 favorites]
posted by Well I never at 5:08 PM on November 27 [2 favorites]
Beyond a broad "can't trust the NYT on trans topics" this specific writer has a history of producing bad reporting that sure reads like anti-trans bias.
posted by away for regrooving at 5:11 PM on November 27 [4 favorites]
posted by away for regrooving at 5:11 PM on November 27 [4 favorites]
"Here's why that study was garbage and deserved to be withheld," or "Here's why, although valid, that study doesn't threaten the consensus on blockers,"
We don't know if the study was good or not because it hasn't been published. a study that hasn't been published basically doesn't exist. it hasn't been peer reviewed! it's not a good source of information, and it's especially not a good source of information to inform medical conditions. Probably don't make medical decisions based on unpublished, non-peer-reviewed studies, unless there's some sort of severe, imminent danger.
posted by BungaDunga at 6:19 PM on November 27 [5 favorites]
We don't know if the study was good or not because it hasn't been published. a study that hasn't been published basically doesn't exist. it hasn't been peer reviewed! it's not a good source of information, and it's especially not a good source of information to inform medical conditions. Probably don't make medical decisions based on unpublished, non-peer-reviewed studies, unless there's some sort of severe, imminent danger.
posted by BungaDunga at 6:19 PM on November 27 [5 favorites]
Your friend should read the article carefully, and distinguish between what the researcher says and what the reporter implies.
The direct quote from the researcher is:
Will she release the data?
The article says it is based on "a wide-ranging interview" with Dr. Olson-Kennedy, and yet we get less than five quotes. I expect if we had access to the full transcript, it would paint a picture that's very different from what is shown in the article.
A good portion of the article is quotes from Dr. Tishelman, responding to questions we don't see. We don't know what questions the reporter asked Dr. Tishelman. Her response, once again, implies that the data is being withheld because the researcher didn't like it. So it reinforces that narrative. But we don't know if the questions she was asked represent what actually happened, or only what the reporting is implying happened.
tl;dr: I wouldn't take this article very seriously. A study was done. The data is messy. It hasn't been published yet. The researcher is still working on it. There's not much more than that if you look at the substance and set aside the reporter's position.
posted by Winnie the Proust at 6:53 PM on November 27 [8 favorites]
The direct quote from the researcher is:
I do not want our work to be weaponized. It has to be exactly on point, clear and concise. And that takes time.What does this mean? It means the data needs to be presented carefully. It does not say that she isn't publishing because she doesn't like the results.
Will she release the data?
"She said that she intends to publish the data, but that the team had also been delayed because the N.I.H. had cut some of the project’s funding."The article implies that she decided not to publish because she didn't like the results. But that's not what she says, and it doesn't offer her quotes substantiating that. What she says is that the data has to be explained carefully.
The article says it is based on "a wide-ranging interview" with Dr. Olson-Kennedy, and yet we get less than five quotes. I expect if we had access to the full transcript, it would paint a picture that's very different from what is shown in the article.
A good portion of the article is quotes from Dr. Tishelman, responding to questions we don't see. We don't know what questions the reporter asked Dr. Tishelman. Her response, once again, implies that the data is being withheld because the researcher didn't like it. So it reinforces that narrative. But we don't know if the questions she was asked represent what actually happened, or only what the reporting is implying happened.
tl;dr: I wouldn't take this article very seriously. A study was done. The data is messy. It hasn't been published yet. The researcher is still working on it. There's not much more than that if you look at the substance and set aside the reporter's position.
posted by Winnie the Proust at 6:53 PM on November 27 [8 favorites]
And btw, the title of this question says "retracted study." The study has not been retracted. It just hasn't been published, possibly "yet".
posted by If only I had a penguin... at 7:14 PM on November 27 [8 favorites]
posted by If only I had a penguin... at 7:14 PM on November 27 [8 favorites]
if she's evidence focused, you could find one of the numerous analyses of NYT trans coverage that show how negatively biased they are towards trans people.
posted by kokaku at 3:04 AM on November 28 [2 favorites]
posted by kokaku at 3:04 AM on November 28 [2 favorites]
As someone who used to assign editorial pieces sometimes based on studies, there is no reason to have published this piece except that it supports some editorial goal. I highly doubt this doctor was sending out press releases saying please cover my incomplete research. Someone at some point in the research -> research PR -> publication -> publication PR -> media stream had an agenda in order to dig this up.
posted by warriorqueen at 8:14 AM on November 28 [3 favorites]
posted by warriorqueen at 8:14 AM on November 28 [3 favorites]
idk if it would be persuasive, but it might also be worth noting that, even if this study is unimpeachable, it says nothing about how kids who are already on blockers do if you take them off. she can't time travel and retroactively not put the kid on blockers, so it's really neither here nor there whether blockers improve mental health or not.
if anything, she can be reassured that the study didn't find any evidence that it hurts kids mental health, so there's nothing new to be afraid of about continuing them.
the question for people in this kid's situation isn't "would starting blockers help", it's "would stopping blockers help or hurt", and that's a completely different question, scientifically speaking.
posted by BungaDunga at 9:31 AM on November 28
if anything, she can be reassured that the study didn't find any evidence that it hurts kids mental health, so there's nothing new to be afraid of about continuing them.
the question for people in this kid's situation isn't "would starting blockers help", it's "would stopping blockers help or hurt", and that's a completely different question, scientifically speaking.
posted by BungaDunga at 9:31 AM on November 28
Fellow trans person here. I am.. confused about why this parent finds the general description of the results frightening? Like, if you can get trans kids' mental health to hold steady during years when they would otherwise be forced to experience the wrong puberty and suffer greatly as a result, during the exact period of time when anti-trans rhetoric began to really skyrocket, that seems.. very good? (I also wonder how and if this discussion is situating these children within the broader social context of abysmal mental health for teens, period: are we expecting that blockers are going to catapult trans kids still living in a virulently transphobic society beyond their cis peers and into some magical land of sunshine and rainbows? Be for real.)
Nobody can really say much re: the quality of this particular study because nothing has been released for us to review. It is alarming that your friend is discounting the by now extensive evidence that the NYT has an anti-trans bias and what that suggests about ideological reasons for publishing an article breathlessly reporting on a paper not having been submitted yet and a researcher having concerns about people using the findings in bad faith (!). (More on the NYT in a "You're Wrong About" interview with journalist Tuck Woodstock here or wherever one listens to podcasts; transcript and citations here.)
Beyond that I recommend Julia Serano's writing as a scientist with deep knowledge of the issues. Good places to start:
Gender-Affirming Care for Trans Youth Is Neither New nor Experimental: A Timeline and Compilation of Studies
The Cass Review, WPATH Files, and the Perpetual Debate over Gender-Affirming Care
posted by wormtales at 9:46 AM on November 28 [8 favorites]
Nobody can really say much re: the quality of this particular study because nothing has been released for us to review. It is alarming that your friend is discounting the by now extensive evidence that the NYT has an anti-trans bias and what that suggests about ideological reasons for publishing an article breathlessly reporting on a paper not having been submitted yet and a researcher having concerns about people using the findings in bad faith (!). (More on the NYT in a "You're Wrong About" interview with journalist Tuck Woodstock here or wherever one listens to podcasts; transcript and citations here.)
Beyond that I recommend Julia Serano's writing as a scientist with deep knowledge of the issues. Good places to start:
Gender-Affirming Care for Trans Youth Is Neither New nor Experimental: A Timeline and Compilation of Studies
The Cass Review, WPATH Files, and the Perpetual Debate over Gender-Affirming Care
posted by wormtales at 9:46 AM on November 28 [8 favorites]
« Older What legal docs should I have my pal fill out? | seeking small boombox with CD player and great... Newer »
You are not logged in, either login or create an account to post comments
We do know that one of the authors is a buffoon for either 1) not releasing a study because it may be used by bad faith actors or 2) not releasing the study for other reasons and saying this is the reason publicly. Either possibility does not reflect well on them.
If they’re really science literate, and not just someone who pretends to be so, they should know that any one study means nothing, and that the totality of evidence is what matters. They should also know that self reported mental health measures are notoriously noisy.
Also “evidence is mixed” would be consistent with sometimes we find positive effects of these blockers, other times we find negative effects. Sometimes finding positive effects and sometimes failing to find positive effects is consistent with a positive treatment.
posted by MisantropicPainforest at 3:52 PM on November 27 [9 favorites]