IFTTTT&T, TTTTTT&T. And also that other thing.

October 29, 2013 2:41 PM

I'm looking for a way to capture decision processes as repeatable, adaptive heuristics to guide people in making situational assessments. Difficulty: not testing software.

I work in an environment where our job description, in two words, is "change detection." If something within our purview changes, we need to A) recognize it; B) describe it; C) attempt to account for or explain it; and D) search out downstream effects and go back to step A.

What we do is both technical and conceptually simple, so the learning curve of the tools and the notional ease of understanding the problem set makes it hard to see at first that it's also very deeply contextualized. That is, it probably takes your doing the job for 18-24 months before you're really proficient because of all of the inputs, possibilities, and moving parts. There's no way to substitute tricks for training, but I'd like to make our tools and processes as smart as possible to reduce the human error factor.

What I'm looking for is a way to capture what the more experienced among us do by virtue of a mix of skill, repetition, and intuition:

Oh, this happened? Well, you probably want to check these three things, and you may want to look back a day or two to see if you might've missed X. And watch for Foo because Bar. And look over at Y to see if this follows the pattern we've seen before, because that's notable. Oh, and, it's even more notable if it's somehow a departure from the ordinary.

Is this a thing? Is there some tool that helps capture these things? Bonus if it helps you recognize connections between and among things.

posted by Emperor SnooKloze to Grab Bag (6 answers total)

Flowcharts and process diagrams document this kind of process. Machine learning techniques like decision trees and artificial neural networks attempt to imitate it through training (though the final model "weights" aren't usually interpretable as heuristics that make sense).

Honestly, it just sounds like generalized expertise. At least, that's what I'd call a mix of skill, repetition, and intuition. This is any mid-level-or-higher engineer's job in a nutshell; probably extends outside engineering fields as well.

It doesn't sounds like you're trying to reproduce this ability artificially, but "expert systems" might be a good starting keyword.
posted by supercres at 2:56 PM on October 29, 2013

And to truly capture it, you'd have to record every decision you make and the variables that informed it. In some systems, this could be highly automated, and it others, nearly impossible. It's such a highly-dimensioned problem that to create any sort of model (again, I can't quite tell if that's what you're after), it would take a massive number of observations.
posted by supercres at 2:59 PM on October 29, 2013

Checklists. Highly skilled experts use checklists to achieve what you are describing to a certain degree, especially the eliminate human error part.

Atul Gawande's The Checklist Manifesto: How to Get Things Right is the book to read and it covers checklist usage in a variety of complex fields such as medicine:

First introduced decades ago by the U.S. Air Force, checklists have enabled pilots to fly aircraft of mind-boggling sophistication. Now innovative checklists are being adopted in hospitals around the world, helping doctors and nurses respond to everything from flu epidemics to avalanches. Even in the immensely complex world of surgery, a simple ninety-second variant has cut the rate of fatalities by more than a third.

NPR has a piece on Gawande. Project Check also has examples of good checklists and even a checklist for creating checklists.

Properly done checklists might not cover everything you require but they are so cheap and simple that you'll surely find some usage for them.
posted by Foci for Analysis at 3:25 PM on October 29, 2013

Google up ethnographic decision tree modeling for an anthropological approach to this.
posted by Monsieur Caution at 4:47 PM on October 29, 2013

I guess it would be useful to know a little bit more about the context. Regardless of context, however, you could say that there is a change/response continuum, with two poles - normal/regular, and easier to codify, and abnormal/irregular, requiring more expertise.

Julian Orr wrote an ethnography a while back, on photocopier repair technicians, that described how some problems could be solved by following the manual, but that other problems required talking with other techs in coffee shops and on breaks.

I read a study (but can't remember what it is at the moment) about engineering shop workers who kept an annotated scrapbook of particular issues, with photos and drawings, on a bench in the shop.

Organizational theories of 'communities of practice' also look at this, although the literature is vast.

So - and without knowing the specifics - some kind of ethnographic/interview/focus group type work might get at the normal events, while having some kind of local knowledge 'store' (and probably not a proprietary knowledge base tool with a crappy data entry interface) would support analysis and mitigation of the non-normal events.

I hope this makes sense; it's kind of hard to be more precise without more specifics.
posted by carter at 6:16 PM on October 29, 2013

The whole area of "Cognitive Task Analysis" is designed to identify the strategies (including automated or tacit processes) that experienced workers have developed over time.

The book "Working Minds" is a great introduction. A TON of different methods and templates are available in an accompanying PDF online.
posted by neutralmojo at 9:25 AM on October 30, 2013