How to find poor-quality qualitative research articles?
April 10, 2020 7:14 AM Subscribe
I am a sociology professor teaching a qualitative methods course, and I'd like my students to be able to pick apart poorly designed/conducted studies. I'm not sure how to find good examples without crawling through tons of literature - and Googling things like "examples of poor qualitative research" is not getting me anywhere. I know that there was a previous Ask regarding bad research studies more generally, but I couldn't find what I was looking for. Any ideas?
Best answer: Why not make it a student exercise? For one of the first classes they each have to bring an article they think is bad and talk about why. They'll probably learn more that way, as well as it being easier on you.
If you teach this course again, you can also save an archive of the worst articles students find, to use to make your points with in other classes.
posted by lollusc at 7:28 AM on April 10, 2020 [18 favorites]
If you teach this course again, you can also save an archive of the worst articles students find, to use to make your points with in other classes.
posted by lollusc at 7:28 AM on April 10, 2020 [18 favorites]
I think lollusc's idea is really best, but I would also want to figure out how give the students some guidance in how to do this, since it's proving difficult or impossible for you and would be frustrating for them, too. I'm an academic but not in Sociology. . To start, I would also probably use the searchable database of journals and publications from my discipline's website database -- not Google -- to search reviews that have phrases alluding to a poorly designed study. If you figure it out, it would be great if you'd do a follow up with what worked!
posted by nantucket at 7:38 AM on April 10, 2020 [4 favorites]
posted by nantucket at 7:38 AM on April 10, 2020 [4 favorites]
https://retractionwatch.com will probably give you enough examples for a lifetime
RABBIT HOLE ALERT!!!
posted by Melismata at 7:55 AM on April 10, 2020 [3 favorites]
RABBIT HOLE ALERT!!!
posted by Melismata at 7:55 AM on April 10, 2020 [3 favorites]
Search for papers over X years old in prestigious journal Y with under Z citations?
This is semi-effective in my field, but I’m looking at traditional experiments.
posted by SaltySalticid at 8:33 AM on April 10, 2020
This is semi-effective in my field, but I’m looking at traditional experiments.
posted by SaltySalticid at 8:33 AM on April 10, 2020
What if you look through articles in journals that are low-prestige in your field: low impact factor, high acceptance rate, maybe even scam/vanity journals. Presumably you can find things there that were first pitched to higher-prestige venues and rejected, and the authors couldn't fix them so just published wherever they could.
posted by xris at 9:23 AM on April 10, 2020 [3 favorites]
posted by xris at 9:23 AM on April 10, 2020 [3 favorites]
Depends on what you're looking for. Retraction watch is good for deliberate frauds, but things get more murky around p-hacking, file drawer effects, etc. The reproducibility project did a pretty good review of such things using the best tool we have: more science.
There's also a small collection of deliberately bad journal articles. My favorite being a study that showed that you could reduce software defects by removing letters from the keyboard, but the MRI of a dead fish was a pretty good takedown too.
posted by pwnguin at 9:59 AM on April 10, 2020
There's also a small collection of deliberately bad journal articles. My favorite being a study that showed that you could reduce software defects by removing letters from the keyboard, but the MRI of a dead fish was a pretty good takedown too.
posted by pwnguin at 9:59 AM on April 10, 2020
This is not sociology, but there was a paper The Cost of Convenience: Ridesharing and Traffic Fatalities that surmised that the introduction of Uber and Lyft lead to higher numbers of traffic fatalities.
The website City Observatory then reviewed the data, suggested some changes, and ultimately surmised that the paper couldn't make that claim, and suggested additional tests and data to compare to shore up the paper.
Which ever side you agree with, I think this is a good example of a paper and some legitimate complaints, a rejected peer review, and a good starter to identify issues with a study and how to fix them.
Going farther, I'd look at b-school papers generally, because they love to make giant claims backed up by questionable science and data.
posted by The_Vegetables at 12:07 PM on April 10, 2020
The website City Observatory then reviewed the data, suggested some changes, and ultimately surmised that the paper couldn't make that claim, and suggested additional tests and data to compare to shore up the paper.
Which ever side you agree with, I think this is a good example of a paper and some legitimate complaints, a rejected peer review, and a good starter to identify issues with a study and how to fix them.
Going farther, I'd look at b-school papers generally, because they love to make giant claims backed up by questionable science and data.
posted by The_Vegetables at 12:07 PM on April 10, 2020
Response by poster: Just to clarify, when I say "qualitative research" I mean research that isn't numbers-driven/statistics-based. This might include textual analysis of visual media, interviews, participant observation, documents, etc.
A lot of these ideas are fantastic ones to keep in mind for when I'm teaching a general research methods course, though! And Retraction Watch is super interesting, as well. Thanks, all!
posted by amyshmamy at 12:58 PM on April 10, 2020 [2 favorites]
A lot of these ideas are fantastic ones to keep in mind for when I'm teaching a general research methods course, though! And Retraction Watch is super interesting, as well. Thanks, all!
posted by amyshmamy at 12:58 PM on April 10, 2020 [2 favorites]
Why not make it a student exercise? For one of the first classes they each have to bring an article they think is bad and talk about why.
I'm an academic librarian, and I do not think this would be a good exercise. If you, as a professor and expert in your field, do not know how to identify articles with bad methods without crawling through tons of literature, how do you expect your students to do that? They won't know what databases to use and what articles to read. What is bad in this context? A bad study is often quite subtle. I appreciate the sentiment here. I think it would be a far better exercise to give them a few examples and ask them to point out flaws.
A few ways to find those flaws: ask other professors in your field for suggestions; find examples of unethical studies; talk to your subject librarian.
posted by bluedaisy at 3:05 PM on April 10, 2020 [7 favorites]
I'm an academic librarian, and I do not think this would be a good exercise. If you, as a professor and expert in your field, do not know how to identify articles with bad methods without crawling through tons of literature, how do you expect your students to do that? They won't know what databases to use and what articles to read. What is bad in this context? A bad study is often quite subtle. I appreciate the sentiment here. I think it would be a far better exercise to give them a few examples and ask them to point out flaws.
A few ways to find those flaws: ask other professors in your field for suggestions; find examples of unethical studies; talk to your subject librarian.
posted by bluedaisy at 3:05 PM on April 10, 2020 [7 favorites]
Going purely on citations is unlikely to work in sociology, since I cite bad work with bad conclusions all the time in order to discuss why it was bad.
I would start by narrowing the field significantly. I can give you examples from my very specific slice of sociology - evolutionary psychologists hypothesizing sexual inclinations based on reading habits, or the survey about fast food and class that appeared to be 'poor people are dumb' until they actually did some proper qual analysis and found that fast food was a way for poor parents to say yes to happiness. So narrow down to one little segment, then go gossip/grar hunting. It's not always based on poor research but it can be and will be picked apart for it.
(Honorable mention: the throwaway line describing lesbian erasure by assuming lesbians couldn't be fans of a male actor because the group name mentioned sex hormones)
posted by geek anachronism at 3:18 PM on April 10, 2020
I would start by narrowing the field significantly. I can give you examples from my very specific slice of sociology - evolutionary psychologists hypothesizing sexual inclinations based on reading habits, or the survey about fast food and class that appeared to be 'poor people are dumb' until they actually did some proper qual analysis and found that fast food was a way for poor parents to say yes to happiness. So narrow down to one little segment, then go gossip/grar hunting. It's not always based on poor research but it can be and will be picked apart for it.
(Honorable mention: the throwaway line describing lesbian erasure by assuming lesbians couldn't be fans of a male actor because the group name mentioned sex hormones)
posted by geek anachronism at 3:18 PM on April 10, 2020
Fairly recently a group of philosophy professors intentionally submitted a series of fake studies (I think at least one may have been qualitative?). Although this may not be a good example of a bad qual study this statement hoax did bring up some interesting questions about “soft” social science type research and quality.
https://www.google.com/amp/s/www.nytimes.com/2018/10/04/arts/academic-journals-hoax.amp.html
I took a grad level qualitative research course recently and I was struck by differing ideologies around the idea of “rigor” in the field. I can’t find the specific paper that was used as an example but we read a phenomonological interview of a single woman who did sex work - n of 1 obviously is almost never considered rigorous in quant fields but depending on the purpose of the qual study may be totally reasonable. We had a very engaging discussion about whether this particular qual study could ever be considered high-quality.
posted by forkisbetter at 3:31 PM on April 10, 2020
https://www.google.com/amp/s/www.nytimes.com/2018/10/04/arts/academic-journals-hoax.amp.html
I took a grad level qualitative research course recently and I was struck by differing ideologies around the idea of “rigor” in the field. I can’t find the specific paper that was used as an example but we read a phenomonological interview of a single woman who did sex work - n of 1 obviously is almost never considered rigorous in quant fields but depending on the purpose of the qual study may be totally reasonable. We had a very engaging discussion about whether this particular qual study could ever be considered high-quality.
posted by forkisbetter at 3:31 PM on April 10, 2020
The Stanford Prison Experiment was a research project in social psychology rather than sociology, but there are plenty of scholars who argue that the methodology was terrible. Example: Why Zimbardo’s Prison Experiment Isn’t in My Textbook
posted by obscure simpsons reference at 10:57 AM on April 11, 2020
posted by obscure simpsons reference at 10:57 AM on April 11, 2020
Might the Behind the Headlines feature at NHS provide any good examples?
posted by kristi at 5:17 PM on April 11, 2020 [1 favorite]
posted by kristi at 5:17 PM on April 11, 2020 [1 favorite]
Something I don't see recommended above: find a good systematic review of qualitative studies that does quality scoring as part of their review process, and pull the lowest-scoring studies cited. For example, Attree (2004), in a review of qualitative evidence on children’s accounts of growing up in disadvantage, graded papers on a scale from A to D. Papers graded D were excluded, so those would be likely to be good examples of poor-quality papers.
You may also find it useful to look at/introduce your students to some of the quality scoring metrics.
posted by acridrabbit at 3:22 PM on April 21, 2020
You may also find it useful to look at/introduce your students to some of the quality scoring metrics.
posted by acridrabbit at 3:22 PM on April 21, 2020
This thread is closed to new comments.
posted by Weftage at 7:24 AM on April 10, 2020 [7 favorites]