Satisfaction with Mechanical Turk work?

July 15, 2008 1:16 PM

Have you utilized Mechanical Turk workers for anything even remotely complicated? Were you satisfied with the results?

This particular project involves hand transcription of seven or eight digital documents -- one per Mechanical Turk worker -- of fairly short length (600-1000 words). So yeah, just writing something out longhand and then mailing them to me. Pretty non-demanding, and it would pay well (compared to most Mechanical Turk HITs that I've seen). Has anyone utilized MT for this purpose or something similar? Did the end result(s) turn out to your satisfaction?

Thanks.

posted by cog_nate to computers & internet (4 comments total)

Ironically, I just got my first set of results back from Mechanical Turk for a project roughly equal to the complication level of your task. Essentially, it involved cutting and pasting from a web source to a list I provided. Very repetitive, but it did call for a little bit of searching, thoroughness and common sense.

I have mixed feelings about the results. Typically, they weren't great, and in most cases, I had to do so much spot-checking of the work that it didn't save me as much time as I would have liked.

On the other hand, although I paid slightly better than average for this work (at least from what I can tell), it was still dirt cheap. My HITs were picked up within minutes, so clearly a lot of people didn't feel that it was too little pay, although some HIts were apparently abandoned.

But! The good news is that a couple of people who responded did a very precise, accurate and fine job. Both of these people were in India and seemed to be professional types. I mention the India thing as evidence that you get what you pay for (the amount I paid being, relatively speaking, much better pay for someone living in India, hence a better quality of worker.) Finding these two people has been great, and I've figured out a way to provide them with all of my work from now on. I'm paying them more (twice the pay, actually) too, since I feel a better sense of security about the work they'll do, and they're delighted at that. It still seems cheap, and everyone's happy. It saves me from doing tedious work, and I can just afford it.

So for me, MT worked out well. I have continuing work to do and I found great people through it. I think the key is really to not pay too much until you know what sort of quality you're getting. Anyone can accept a HIT, so you're at the mercy of luck until you know.

If you're trying to get this done quickly, I'd offer each HIT a couple or three times at a low price. You don't mention what you're willing to pay, but you'd probably get better results offering each HIT three times at $3 each than doing one for $9. In other words, paying a lot will not guarantee results in the slightest. Having each piece of work done three times at one-third the price increases the likelihood of quality greatly. Such is the odd way in which MT promotes the hard-edge of capitalism!
posted by Dee Xtrovert at 2:46 PM on July 15 [2 favorites]

I work on MT and I really like those kind of hits and I'm a pretty good worker.
posted by Melsky at 6:45 PM on July 15

I did a search for podcast transcription and noticed this company, CastingWords, used an interesting process. They make full use of Turk as a multi-pass system where a podcast would go through multiple steps of initial transcription, grading, then improvement with a slight bump in the payment at each pass. So the initial transcription HIT would be offered at say, $0.80, then a grading HIT would be offered at $0.07 to grade the transcription on a scale of 1-8, then improvement HITs would be offered again at $0.80 to improve that transcription quality through edits or re-transcription tasks. I assume this would be cycled until the transcription exceeded level 8.

A process of this complexity is probably a bit much for your 6 documents. But you may get better results with with say an initial transcription HIT at $1.50 with 2 improvement HITs at $1.00 (3-pass system) than a single HIT at $5.00. The idea is that for more mundane tasks, an assembly line system may yield better results and at lower cost than hiring a "craftsman."
posted by junesix at 8:29 PM on July 15 [1 favorite]

Thank you!
posted by cog_nate at 11:45 AM on July 18