When should you inquire about the academic rigor of research?

May 31, 2014 9:28 AM

A few weeks ago I saw this FPP and found it to be very enlightening. Since then I was exposed to some similar but much lesser academic work and found questionable references in it. My issue - should I just drop it or do something more substantial?

I work with folks who enjoy sending out papers regarding ethics and how to be better leaders. I received the following paper from a senior person in my organization. This paper has been utilized in training many senior people, here is just one example. I've seen the paper more than once in both formal and informal training settings. So, I decided to look into some of the many references the authors, Ludwig and Longenecker, used and cited in their material. They most often cited Blotnick's work from his book, "Ambitious Men" - a simple googling found this. Note Blotnick was exposed in 1987, while the paper was written in 1993.

I realize there are a plethora of other issues with the paper as a tool for teaching what you'll likely surmise - military officers. However, beyond these obvious issues with the content, the paper should be retracted in my opinion and the authors should contact people they've licensed it to - as I think they are making (and have made) some significant money and notoriety off of this "Bathsheba Syndrome" idea. I simply don't know if its worth undertaking or how to do it or if I should just drop it. I do think they may already know and question the ethics... and find it ironic. Or maybe I'm wrong about the whole thing?

posted by vonstadler to Education (4 answers total)

Critical readers find flaws in methodology all the time. It's the reason you read the methods section of the article. However, even appropriate and rigorous methods lead to incorrect findings. It is simply the nature of academic inquiry.

Would I be thrilled with research that's longitudinal but unstructured and loosely coded? No. Crappy methods lead to outcomes which can't be validated. Assuming he actually did all those interviews, his whole method of inquiry is problematic...confirmation bias, selection bias, unstructured coding. (Note: I'm basing that on the article you shared. I didn't go to his book. As such his methods may be better than reported.)

However, just because the method lacks rigor it doesn't mean the finding is incorrect. (Even a stopped watch...) The standard thing to do when you run into research with weak methods is to look for other times the theory was tested in more rigorous methods. I think any researcher writing on that topic will likely cite Blotnick - his work appears to be seminal and to have spawned other thinkers. That doesn't mean you cite it as gospel. You cite it as the genesis of an idea which was further researched.

I think your next step is to look to other academic literature that tests the same ideas.
posted by 26.2 at 11:52 AM on May 31, 2014 [1 favorite]

I think one test I would use for trying to decide whether or not to go forward with criticizing it and trying to get it removed is "Do you have a better answer available which is adequately supported in the rigorous manner to which you are trying to hold this piece?" If not, then leave well enough alone.

I have not managed to read the entire Bathsheba Syndrome piece, just skimmed parts of it, but it seems that at least part of the subtext is "officers should not have affairs with the wives/girlfriends of their subordinates." This is something I know to be a legally covered issue in the military -- that an officer can be jailed for it and that it is treated similar to statutory rape in that her testifying that it was "consensual, honest!" is irrelevant. The legal assumption, as I understand it, is that it cannot really be consensual due to the enormous power (including even potential life and death in that an officer can order him on a doomed mission or whatever) that an officer can exercise over her man. Plus her swearing "it was consensual, honest!" -- if he pressured her into the affair, he can pressure her into that too. So you just can't consider that meaningful testimony.

Anyway, this is a serious issue in the military rooted in a critical morale issue in that, since an officer can ask a soldier to give his life for the good of his country, it is really critical to national security that the man so ordered feel confident that it really is about national security and not about "and then I can have your wife to myself! woohoo!" So officers fooling around with soldier's wives is a big no no for important practical reasons and are not some kind of frivolous nice-to-have idea about "in theory, a gentleman would never do that" or whatever. It is about making sure that when the ultimate sacrifice is required, it gets honored -- for the good of the country.

So the piece may be popular in the military culture in part because it somewhat subtly drives that point home and sometimes doing that as subtext is more effective than being more blunt on such a delicate, touchy topic. Thus, it may have inherent value for reinforcing the idea that officers need to keep their pants zipped and not get involved with their soldier's wives or girlfriends. That is a not insignificant point to make and any material which helps make it may have some inherent value, regardless of what you think about its academic rigor or other criticisms.
posted by Michele in California at 3:28 PM on May 31, 2014 [1 favorite]

Response by poster: My issue is why use Blotnick's work at all since it would only cast doubt over the research Ludwig and Longenecker. A follow up article from the NYT in 1987 reveals Blotnick falsified his credentials, was never a psychologist and never substantiated his work. An academic in the article was quoted as saying the books were not research books and could find no researchers familiar with his work. In short, he was a fraud.

The piece is on ethics... it uses the work of a fraud to support its thesis. I have a problem with that. As for the UCMJ, that is an entirely separate issue.
posted by vonstadler at 3:43 PM on May 31, 2014

Best answer: Thank you for those extremely interesting links; the "Bathsheba Syndrome" paper that you think should be retracted (the first link in the second paragraph) is more meta-sermon than research study, and unlike anything I can recall running into previously, but I don't think it depends significantly on Blotnick, and is not at all undermined by his fraud as far as I can see.

I find it disturbing that the US military is basing very important policy decisions on research by Christian scholars who take the Bible as much for granted as chemists do the periodic table, yet I don't think it would be possible to challenge such research by showing that even important references were fraudulent, though I am fairly certain it would suffice to ruin a military career-- namely yours, if you happen to have one.
posted by jamjam at 4:02 PM on May 31, 2014 [1 favorite]