Woo. Wooooo. Wooooooo.

March 28, 2022 10:18 AM

Inspired by Mayor West's comment, what is something in your profession that has been proven to be ineffective (or has no basis in science) but is still done by many/most people in that profession?
posted by Mitheral to Work & Money (50 answers total)

In teaching, "learning styles" - the idea that different learners are "visual", "auditory" learners, etc. There's been a consistent failure to find any evidence of this at all, despite much effort and even a prize. It just feels like it should be true, and that's enough that it was treated as fact during my training when I first became a teacher.
posted by wattle at 10:23 AM on March 28, 2022 [48 favorites]

In libraries, a whole whackload of cataloging and metadata practices that don't appreciably help library patrons (especially now that search is mediated through computers rather than card catalogs and bibliographic indexes) but were endorsed by Charles Ammi Cutter or Lubetzky the ding-dang Nitpicker, so they May Not Be Challenged In Any Way Ever.
posted by humbug at 10:43 AM on March 28, 2022 [16 favorites]

In art, and graphic design: The idea of the Golden Mean and that certain proportions are inherently both more beautiful and based on proportions in nature (shells, the human body etc). It's confirmation bias. Not based on science.
Many writers and artists also firmly believe in the "left brain =logic, right brain =creativity" thing. Which is false and only distantly based on science in that different parts of the brain are used for different things.
posted by Zumbador at 10:45 AM on March 28, 2022 [20 favorites]

The belief that taking the needs of several disparate customers and trying to combine them in to one combined acquisitions program to "save money" will ever, ever work. (Alternately - any Defense program with "Joint" in the name is almost certainly doomed to failure. See JTRS as a recent notable example.) There are other reasons for doing this (arguably the most important being that big flashy programs seem to get funded more reliably than smaller, achievable programs), but it will never save money.

While I have never worked in this specific space per se, boarding groups for airlines never actually speed up the boarding process and potentially make it slower than a free-for-all.
posted by backseatpilot at 10:53 AM on March 28, 2022 [5 favorites]

In art history, connoisseurship and too-heavy reliance on tech-y analysis both have their pitfalls in authenticating artwork.

Of course, the worst possible solution is a lawsuit, since then you have a judge (almost certainly not an expert in the field) ruling on the instance of authenticity/forgery.
posted by Lawn Beaver at 10:55 AM on March 28, 2022 [4 favorites]

Twelve step programs are not an evidence-based intervention for substance use. There is no such thing as a randomized, controlled study of 12 step programs.
posted by assenav at 10:57 AM on March 28, 2022 [40 favorites]

Rotating passwords every 90 days. The better solution is properly implemented multifactor authentication and monitoring.
posted by Candleman at 10:58 AM on March 28, 2022 [38 favorites]

A lot of the "old school" techniques of arson investigation were mostly based on "I learned it from this fire marshal who has been working for 20 years". Things like "pour patterns" and "crazing on glass" have been given over importance in "proving" a fire was an arson, and while they can be indicators of a suspicious fire, they also can be present under accidental/non-intentionally set fire conditions as well.

There ARE verifiable science based techniques which can be used to determine if any ignitable liquids were present at a scene (GC/MS) but even those need to be used in context. Gasoline found in samples from a garage is less useful then say gasoline found in samples from the bedroom closet.

Luckily as the newer generations of investigators are being trained some of the old-school stuff is falling out of favor but it still is used far too often.
posted by Captain_Science at 11:22 AM on March 28, 2022 [10 favorites]

Quite a lot of structural engineering and fabrication techniques exist solely because they've never failed, not because they're known to work (if that makes sense).

There's a LOT of waste in construction simply because none of the participants have the time and money needed to investigate the details and come up with a thoroughly-vetted solution so everyone just goes with the "empirical solution" (when you catch someone using that phrase it means the solution they're proposing has never been tested, does not have any real math behind it, and is just "the way it's always been done and it usually doesn't fail so let's just do it instead of having to think any more about it").

...which is basically unavoidable, alas. Just testing one configuration of seismic column base cost over half a million dollars, took two years, and only like three professors in North America possessed both the expertise and equipment necessary to run the tests in a timely fashion. Ain't nobody gonna pay, or have the time, to test 'em all so you just overengineer, over-build, over-pay, and then pray.
posted by aramaic at 11:53 AM on March 28, 2022 [10 favorites]

Job interviewing. No one in my experience is scientific about job interviewing. People just do whatever the org has decided to do and the beat goes on.

(to be fair, I bet it's a super hard problem to study. the inputs are as diverse as humanity, and you've got to do it at pretty massive scale in order to get enough data)
posted by Sauce Trough at 12:00 PM on March 28, 2022 [18 favorites]

In comics, there are lettering standards that came about due to printing processes that are no longer in use.
posted by fillsthepews at 12:00 PM on March 28, 2022 [14 favorites]

In finance, trading individual stocks in your own personal accounts instead of just buying index funds. Everyone still does it, despite all the evidence that folks in finance tend to do even worse than the average joe (who tends to do slightly worse than index funds) when they pick their own stocks.
posted by Grither at 12:02 PM on March 28, 2022 [12 favorites]

Software Development Methodologies. Just excuses for endless theoretical meetings which never change anything, at least for very long.
posted by Rash at 12:10 PM on March 28, 2022 [15 favorites]

I thought of another one - there's some good evidence that Emergency Locator Transmitters (ELTs) are, at best, not worth the money. ELTs are mandated equipment in all aircraft since a Congressman died in a plane crash in the early '70s, and are designed to transmit a radio signal upon detection of an impact so that search and rescue teams can find you.

The evidence for their lack of usefulness generally falls into these arguments:
-Very high false positive rates (the devices can be inadvertently activated if, say, you make a hard landing)
-Lack of coverage (older systems relied on VHF signals that don't travel very far; however, modern ELTs work on a different band that ties into a global SAR satcom system)
-The type of crashes where you most likely would need the device (basically something that would result in severe injury/death) are also the kinds of crashes that tend to destroy the ELT
-More minor crashes have a tendency to not trigger the ELT, and the victim can usually use a cellphone or get local help
-Due to the high false positive rate and other logistical issues (mostly aircraft owners failing to keep the ELTs registered), it can take quite a while to activate SAR response
posted by backseatpilot at 12:31 PM on March 28, 2022 [2 favorites]

Substance use treatment. How about treatment centers not allowing MAT drugs (specifically methadone sometimes suboxone too), or other drugs that may be prescribed for reasons.
posted by AlexiaSky at 12:40 PM on March 28, 2022 [10 favorites]

Zumbador: The idea of the Golden Mean and that certain proportions are inherently both more beautiful and based on proportions in nature (shells, the human body etc). It's confirmation bias. Not based on science.
Whoa! As I just made such an assertion over on MetaTalk I'd better qualify it with some data gathered from and by 200 students over several years at my last place of work. The average ratio between height vs height to navel = 1.63; standard deviation = 0.09; range (1.48-1.79) which is significantly (p ~ 0.02) higher than φ = 1.618. I'm glad we've cleared that up. The point I made in class was that humans in all their diversity still maintain consistent proportion in their growth and development that makes each of us recognisably different from, say, koalas.
posted by BobTheScientist at 12:52 PM on March 28, 2022 [2 favorites]

It's confirmation bias. Not based on science.

Fib-o-natch eh?
posted by snuffleupagus at 1:11 PM on March 28, 2022 [4 favorites]

The War on Drugs. Horrifying and expensive and lacks empirical support.

the 2x-annual Daylight Savings Time switch -- I loathe tha goblin Rubio but if he manages to make DST permanent I will downgrade to mere hatred.
posted by Sauce Trough at 1:13 PM on March 28, 2022 [19 favorites]

Just excuses for endless theoretical meetings which never change anything, at least for very long.

I was going to be more specific but whatever brand of "agile" is in fashion at the time. Sprint planning, retrospectives, sprint pre-planning, etc. I've been on projects where I calculated something like 30% of the entire team's time was on meetings. Not everything lends itself to two week releases. Hard to argue against it or ask the value when someone's role is literally agile coach. The construction industry manages to complete complex projects without someone coaching them. No one measures the effectiveness of agile.
posted by geoff. at 1:29 PM on March 28, 2022 [17 favorites]

This is a tiny example used by only a few people in my profession, but when I was doing my graduate research I used one chemical as an analog for another, because (I assume now, decades later) my advisor told me to do so. In writing up my thesis, I followed literature back to maybe the 1930s trying to find the original paper proving that it was a good substitute. Never found proof, just paper after paper in which the same substitution had been made.
posted by 2 cats in the yard at 1:47 PM on March 28, 2022 [7 favorites]

Response by poster: My own: As an electrician (and this is going to apply to practically all mechanical trades) I am routinely forced to install "lock" washers even though they do nothing vs a plain flat washer. Engineers spec them All. The. Time. even though it's been known for decades that they are useless at preventing bolts or nuts from coming loose.
posted by Mitheral at 1:53 PM on March 28, 2022 [21 favorites]

It probably goes to show how legal positivism works that everything I can think of is ritual knowingly maintained as such out of prestige, decorum, etc (like suits and ties) rather than anything anyone would argue is functional but in fact is not (though I am sure that exists too, were I not too close to see it....)
posted by snuffleupagus at 1:59 PM on March 28, 2022 [1 favorite]

In Emergency Medical Services there is a depressing number of these, including
- oxygen in many cases
- spinal "immobilization" in trauma patients
- transport of cardiac arrest victims
- lights and siren use
- critical incident stress debriefing for EMS providers

Oxygen helps when blood oxygen is low or breathing is labored. It quite possibly harms many others including heart attack and stroke patients. But we're addicted to giving it.

Spinal "immobilization" doesn't immobilize well at all. Thank goodness it never really needed to. It's an ineffective, often harmful, solution to a problem that didn't exist. D'oh!

People whose hearts stop are sometimes resuscitated, but if it doesn't happen right there at the scene, it's not going to happen in the E.R. Transport just endangers the crew and needlessly disrupts things further for the family.

Using lights and siren enables medics to save an astonishingly small amount of time, which has never been shown to benefit patients, but has been clearly shown to be associated with a large number of dangerous motor vehicle accidents.

EMS providers are forced to confront again and again just how porous the border is between their day-to-day existence and nightmare. "Stress debriefing" is the remedy agencies throw at this problem, despite a fair consensus among researchers that it doesn't help much and may be overall harmful. I've been guilty of it myself; I pushed people to go to sessions who were dealing with the stress relatively well, then watched them leave the session in tears, and thinking to myself "see, they really weren't dealing as well as they thought," rather than, "gosh, that made things worse!"
posted by wjm at 3:50 PM on March 28, 2022 [23 favorites]

In the mainframe world - claiming that "PCs" (used as a stand-in term for servers or racks of servers) are incapable little toy boxes.

Mainframes _are_ extremely capable, and can run modern stuff; but so are, and so can, servers. It's ineffective to argue that one is better than the other. It's better to argue that one can do an organizations work, and any new work, cheaper and more efficiently - but which one depends on the organization and what they've done so far.
posted by TimHare at 4:12 PM on March 28, 2022 [1 favorite]

In all sorts of communications and marketing, mass methods of communications still get way too many resources and attention for their effectiveness. If you have limited resources, you're almost certainly going to get more bang for your buck with something that is more targeted and tailored (for instance, a facebook ad tailored for a specific audience) than something big and splashy and meant for a mass audience (like an ad on broadcast TV). This is especially salient to me as someone who has worked for nonprofits and political campaigns, which almost always have limited budgets, but I think it's true for most small or medium-sized businesses as well. But big, splashy media or ad campaigns get a lot of attention and make donors/investors happy.
posted by lunasol at 4:26 PM on March 28, 2022 [4 favorites]

Haha, lock washers are toys of the Devil. I’ve had designers try to spec them in an A490 slip-critical connection, believe it or not. Like, my dude (it’s always a dude):

1) if that thing worked then you wouldn’t have a slip-critical connection

2) if you do have a fully-tightened slip-critical connection then your washer has been fully crushed have you looked at the ratings for an A490??

3) point me to the paragraph in the RCSC that allows lock washers, and no, Direct Tension Indicators are not lock washers, nor are lock washers some peculiarly arcane form of DTI (I had one guy try to make that argument believe it or not).

I think the people that sell lock washers have sold their souls to Satan because I’m not sure what else can explain the continued existence of the product.
posted by aramaic at 4:30 PM on March 28, 2022 [6 favorites]

>I've been on projects where I calculated something like 30% of the entire team's time was on meetings. ... No one measures the effectiveness of agile.

The myth that meetings and planning are unnecessary and get in the way of software engineering "productivity." You have to have some co-ordination of workers, most plainly set out by Amdahl's Law, to spread out the tasking that needs to be done.

Geoff.'s right to point out that metrics aren't captured and rarely improve a system that's intended to be continually improving. "What you measure, you improve" and "every measure that becomes a goal loses its insight as a measure" come into play here. It's so easy to gather metrics that eventually get weaponised against the workforce. Alternatively, do you want to measure how often the goals changed and the team pivoted? Do you want to measure how often the leadership messed about and left critical information unavailable so that delivering the product was compromised?

Further cognitive dissonance happens when you're trying to industrialise and scale out a sequence of tasks that are each bespoke and artisanal.
posted by k3ninho at 4:36 PM on March 28, 2022 [10 favorites]

I think multitasking has been empirically proven to be counterproductive in terms of work quality, accuracy and general efficiency. And still, every single job description I've ever seen in my industry cites 'multitasker' as one of the desired skills.
posted by windbox at 4:45 PM on March 28, 2022 [17 favorites]

That p=0.051 means the test failed. That p=0.049 means it’s a true result. That p=0.001 is even more significant.
posted by hydrobatidae at 4:57 PM on March 28, 2022 [6 favorites]

Ooooo wjm, talk about pushing epi during cardiac arrest next!

ETA: we stopped transporting cardiac arrest patients that don’t have ROSC during about a 30-40 min window of work years ago. Doing CPR in the back of a moving bus is ineffective and it sucks, and it doesn’t allow the family/friends to be with the patient at the end.
posted by sara is disenchanted at 7:20 PM on March 28, 2022

Any kind of "level" system in programs for people with behavioral health problems (in-patient or day psychiatric treatment for adults, schools/programs for children and teens with "behavioral problems"). These are point-based programs in which clients earn or lose various privileges based on whether they start fights, participate in group, etc. Theoretically they eventually "earn" their way to release or eligibility for release by getting to Level 7 or whatever. There is absolutely no evidence that these programs lead to improvements in client outcomes, help clients acquire new skills, decrease the amount of time clients spend in programs, decrease dangerous behavior within the programs, or do basically anything beneficial at all, and yet they are essentially industry-wide. This is an uncharitable read, but it seems like staff think they are "scientific" or "quantitative" because they involve a LOT of obsessive data tracking and numbers.
posted by Snarl Furillo at 7:34 PM on March 28, 2022 [9 favorites]

Net Promoter Score (NPS) as a measure of customer happiness and business success. It's calculated in a way that makes no logical sense and is based on a ridiculous and flawed premise. And yet it's a performance metric many ops, support, and sales leaders are held to.

Customer Satisfaction (CSAT) surveys are also generally poorly designed and do not measure what people think it measures, and support agents are often held to it as if they have full control over the outcome.
posted by rhiannonstone at 9:15 PM on March 28, 2022 [9 favorites]

Typical human body. Does not exist. Vehicles are designed for a typical 5'10 male driver and are thus uncomfortable for taller or shorter men and most women. Seat belts are not adjustable, and only the driver's side door is reinforced, only the driver's seat is properly cushioned.
More on the world designed for the typical male can be found in Invisible Women by Caroline Criado Perez.
posted by Enid Lareg at 9:28 PM on March 28, 2022 [11 favorites]

The QWERTY keyboard.
posted by slimeline at 12:33 AM on March 29, 2022 [4 favorites]

*In teaching, "learning styles" - the idea that different learners are "visual", "auditory" learners, etc. There's been a consistent failure to find any evidence of this at all*

So, double counter to this - when I looked into it, it turns out those studies have been designed ridiculously badly.
For starters, they didn't even tend to catch that a bunch of people can't even visualise, which is one heck of a 'learning style'.
Somehow that's only become a 'widely known thing' since about 2015 - https://en.m.wikipedia.org/wiki/Aphantasia

I can testify that teaching and memory techniques that rely on visualising are really useless for people with Aphantasia (I didn't even understand how the memory palace or number association things were supposed to work before I could visualise - it just gave me more things to remember by rote). Some people have no audio imagination, and so on for each sensory imagination.

As someone mildly aphantasic (I've got 'better' over the last 20 years by doing woowoo dream incubation stuff etc), the learning styles thing seems to have been a good proxy to get teachers to understand that some people *cannot imagine* in your preferred sensory teaching style, eg 'visual' so *please* mix it up a bit.

Again, Aphantasia has only gotten more widespread attention in the last 7 years, but as a child, at least some teachers were using this concept in order to stop torturing me with pointless makework like highlighting different topics in different colours so I'd 'remember it better'.

If the studies to prove whether people have sensory preferences in remembering information were better designed, they would have picked up on the population of people with Aphantasia, among others, but they all seemed pretty badly designed, so they didn't.
posted by Elysum at 2:30 AM on March 29, 2022 [4 favorites]

From my sister - in social work, trying to rehabilitate home children aged 16 who have been kicked out by their parents.
posted by paduasoy at 5:34 AM on March 29, 2022 [4 favorites]

Ugh, NPS at least makes some intuitive sense in its original context of a retail business where word-of-mouth is an important driver of new business, but it's been applied so many places outside of that context where it makes no sense. Right now my company's HR is using an employee NPS survey as its main measure of employee mood which seems bonkers. In particular with the way NPS buckets responses, if you're trying to measure customer promoter behavior, maybe there's not a big difference between "0 - it was the worst customer experience I've ever had" and "6 - it was fine, I guess, but I wouldn't really recommend it" but as a manager I care a huge amount about the difference between "0 - this is the worst place I've ever worked, I'm about to run screaming into the night" and "6 - this team is ok, I guess, but it could do a lot better".

The software development methodology stuff is a great example of "Plans are useless, planning is everything". Adopting one of those frameworks is useful almost exactly insofar as it causes a team to spend time thinking about process and experimenting with ways to improve how they work, and actively destructive to the extent that it shuts down those conversations and replaces them with "We do it this way because that's the Agile(c) way."
posted by firechicago at 5:48 AM on March 29, 2022 [4 favorites]

Psychologists regularly rail against how teachers employ "learning styles" (seriously - every few years there is a new prominent editorial, literature review, or metastudy in a prominent psychology journal on this exact topic). The argument isn't that people don't have preferences or opinions about how they like to learn or even how they think they learn best. The specific hypothesis that psychologists fight against is the unproven idea that people learn best when their learning preferences are catered to (the "meshing" hypothesis).
posted by ElKevbo at 6:08 AM on March 29, 2022 [1 favorite]

"Done by most" is a strong criterion, but in medicine it is easier to list "things we do with a strong evidence basis" than "things we do for no reason." A lot of that comes from well intended theories with varying levels of basis in facts vs historical anachronism. The set of things that "need doing" in medicine is so large and each individual different enough that you're forced to draw analogies.

With that prologue, Choosing Wisely is a somewhat famous campaign basically begging clinicians to stop that has expanded a bit to have other recommendations dumped in. "Things we do for no reason" was the title of a long-running series of journal articles on specific unfounded or futile practices and still works as a search term.
posted by a robot made out of meat at 6:37 AM on March 29, 2022 [1 favorite]

Bagel quality has not one ding-danged thing to do with the properties of NYC municipal tap water.
posted by la glaneuse at 6:49 AM on March 29, 2022 [9 favorites]

My field is metrics reporting and I often have to explain that you shouldn't average an average unless you know that the denominators are the same. If I do 10 widgets per hour for two hours and you do 7 widgets per hour for 8 hours, we did not average 8.5 ((10+7)/2) widgets per hour for the day. In fact, we made 76 widgets in ten hours, so we averaged 7.6 widgets per hour. If everyone worked the same numbers of hours, then you can just take their hourly rates and average them.
posted by soelo at 7:00 AM on March 29, 2022 [5 favorites]

It probably goes to show how legal positivism works that everything I can think of is ritual knowingly maintained as such out of prestige, decorum, etc (like suits and ties) rather than anything anyone would argue is functional but in fact is not (though I am sure that exists too, were I not too close to see it....)

How about the bar exam itself? Many people do sincerely believe that it ensures that attorneys have some baseline of knowledge and competence to practice, but the skills it tests (almost entirely focused on memorization and regurgitation across a long list of topics, many of which a lawyer in any given specialty might never encounter again) bear zero resemblance to those needed in the actual practice of law (e.g., research, analysis, client counseling), plus it has the negative effect of reinforcing inequality and lack of diversity in the legal field, and furthermore the evidence we have suggests that things would be fine without it (see Wisconsin, which licenses UW and Marquette grads to practice without taking the exam). The view that it's just a really expensive and pointless hazing ritual that needs to go is admittedly still far from winning out, but it's definitely growing -- read more here, here, here, here, here, etc. (Took it in July, passed, am still salty.)
posted by naoko at 1:42 PM on March 29, 2022 [4 favorites]

The bar exam is an interesting example, also in the context of the state bar system's protectionist aspects (as reflected by varying reciprocity rules). I wasn't thinking beyond practice itself. I passed the CA three day bar (in a rough year); that doesn't make me want to inflict the same experience on others. It needs to be completely overhauled (maybe you take each MBE during the break after you take the LS class, bite sizes while the material is fresh; and then do a couple performance tests after graduation, ditching the other essays -- you've already passed your finals). Or replaced with something better (maybe some kind of conditional licensure and apprenticeship system).

And congratulations!
posted by snuffleupagus at 2:28 PM on March 29, 2022 [2 favorites]

Covid pretty much put the final nail in the coffin of belief that the primary purpose of public education is anything other than childcare.

Also, the false claim that kids "needed to be back in school for their mental health"? Then please, explain why it's taken an across the board nosedive since schools have reopened? Parents, employers, and governments just want the daycare back in place.
posted by stormyteal at 7:06 PM on March 29, 2022 [3 favorites]

Most non-profit fundraising events (dinners, galas, etc.) don't track staff or volunteer time accurately. They feel like you are raising money but that claim is dubious in most cases. It is just the way things have always been done and it seems right.
posted by zerobyproxy at 7:54 AM on March 30, 2022 [3 favorites]

In my field, we rely extensively on risk models premised on loss events being normally distributed. Every few years, there is a notable event that was not predicted by the models and it is promptly removed from data presentations and footnoted as an outlier.

Management is wedded to the idea that we should be able to predict the unpredictable and regulators expect us to have these models so….
posted by clark at 5:32 PM on April 2, 2022 [1 favorite]

re the bar exam: an ex-girlfriend (a law librarian) once passed the bar exam after an attorney in the practice said he could coach her to pass it in less than a month—even though she had never taken a course in law school in her entire life.

This was a wager, of course, and she passed with a middling average. She and the attorney split the winnings.
posted by ivanthenotsoterrible at 3:32 PM on April 16, 2022

I'd be curious to know where it happened (stories about people who manage to 'read law' are always interesting). Usually you can't just study and take it (unless maybe your sponsor is willing to falsely affirm an apprenticeship over time meeting the state's requirement). I'd have to look, but I'd expect in California they'd also have to pass the 'baby bar' required of students at unaccredited law schools.
posted by snuffleupagus at 6:40 PM on April 16, 2022

FWIW, you also don't usually get your bar score back if you pass. I'm not sure that happens in any US jurisdiction.
posted by snuffleupagus at 6:49 PM on April 16, 2022

I did meet someone IRL who read her way to a law degree. I think she was pretty sick of being a lawyer by the time I met her, incidentally, but she did do it. And we can always check the example of Kim Kardashian and her baby bar experiences these days.
posted by jenfullmoon at 9:30 AM on April 17, 2022