Encoding for transcription

September 30, 2013 8:44 PM

Let's say I've got some arbitrary data (a jpg image, or a PGP-encrypted message) that will need to transmitted through manual transcription (a human reading from a piece of paper or listening to someone reading from one). Are there any encodings designed to aid speed and accuracy of human transcription?

A person asked to transcribe...

hQIOA7/zOZmBNYGsEAgAgNDDkOUCOXOp2hFUP126Am8WW1upPHQw5RmzFnBX9s4

is probably going to have a rough time. But if instead presented with...

pardon mismeasure slew alpha cat honest sedate tensile kitcat solecistic unmelting demise pindy lachrymae petroglyph stairless hypercreaturely lazyhood unfrizz leftwards dotingness

... which, assuming drawn from a wordlist of 235886 words, has the same ~384 bits of entropy all the same, but I'm able to type it in 3/4 of the time.

Have the comparative effective data rates (that is, time required to correctly transmit message, error-correction time included) of encodings designed for human transcription been studied? This is the only attempt at improvement over base64 I've been able to find.

posted by phrontist to Computers & Internet (7 answers total)

Language itself is an encoding, particularly useful for compressing cultural memories and abstract concepts into small, easily communicated units.

For instance, if I asked you to read out all the hexadecimal bytes of a PNG-formatted photo of a 5-inch by 5-inch photo of a unit circle, that could take a while. But if we both have a shared understanding of algebraic language, then you could say, "Draw me the equation x² + y² = 1 on a 5-inch square card," which takes very little time, and the message would be clear and correct.

This is the principle behind vector graphics, music on compact discs, compressed human genome libraries, etc.

If your arbitrary data can be reduced to this kind of abstraction, then it becomes a matter of picking an efficient language to encode that abstraction, e.g., a mathematical equation (or set of equations).
posted by Blazecock Pileon at 9:05 PM on September 30, 2013 [1 favorite]

Best answer: Mnemonic encoder is possibly a good starting point with a few references, including the PGP word list (although that's focused on voice communication rather than visual transcription).
posted by lantius at 9:13 PM on September 30, 2013 [1 favorite]

Response by poster: Blazecock: Sure. I'm interested in encoding encrypted data for transfer between air-gapped systems, but for fun, let's just insist the data to be transmitted is totally arbitrary, so the designer of the encoding can't assume it has any particular properties (e.g. apparent semantics). Using a system which doesn't rely on human comprehension potentially has other advantages. Certain kinds of mistakes - semantic paraphasias, etc. - are avoided.
posted by phrontist at 9:16 PM on September 30, 2013

Best answer: Pronounceable hashing is a similar idea.
posted by regularfry at 9:39 PM on September 30, 2013

Best answer: Bubble babble is another. Link goes to google cache because the original source seems to be having trouble.
posted by regularfry at 9:44 PM on September 30, 2013

Best answer: Maybe also review the NATO phonetic alphabet -- there are studies linked in the Wikipedia article.
posted by RobotVoodooPower at 11:22 PM on September 30, 2013

Best answer: A lot depends on whether productivity of the human is in question. It would actually be extremely hard to type a string of entirely unrelated words--including uncommon words with multiple syllables--as fast as someone could do basic data entry given a string of common characters. Listening to transcription of individual words is much, much slower than strings of words in a proper grammatical context. If each word were fairly distinct--when you say 235000 words, I don't know if you mean really unique words or root words with varying prefixes/suffixes--you could at least guarantee some form of error correction. For more normal English transcription, error rates in the range of one-thousandth of an error per character would not be uncommon.

If you are using human transcription and uncommon words, however, the basic productivity boost would be to then use abbreviation techniques--ptgl for petroglyph, say--which would build speed and possibly accuracy.

(Now, if you could have each character represented by multiple possible words, and have those words chosen by the software to mimic English grammar, that might involve both a productivity and accuracy increase..."alpha cat mismeasured the petroglyph" better than "alpha cat mismeasure petroglyph".)
posted by mittens at 6:32 AM on October 1, 2013