Mechanical Turk and the IRB

June 17, 2014 7:07 AM

I'm planning to use Amazon's Mechanical Turk to perform a survey, but I'm a bit confused about how to work this into my IRB application.

Specifically, how does the informed consent process work when using the MT platform?

If you have actually used MT for research, I'd love some help navigating the specifics of getting it through the IRB, and keep in mind that this may be the first time my School's IRB has seen an application with MT as the recruitment vehicle.

posted by JanewayJunior to Work & Money (7 answers total)

I have not personally used MT, but I know several people who have. If you're just using it to recruit for a survey, then the consent process should work the same way that any other web survey would--the first page of the survey would have the consent language, with a button that indicates that by clicking, they are consenting to participate in research.

If your IRB has truly never heard of MT before, then you will likely need to take some time in you application describing the population & recruitment process (i.e. HITs, acceptable scores for inclusion, etc). My experience with IRBs and unfamiliar methods is that it will help if you include citations to other research, preferably in your discipline, that has used MT and can provide a sense of acceptable standards. I know MT has become very popular in psychology, for example, so there should be lots of research you can cite that will give general descriptions of the population, etc.
posted by DiscourseMarker at 8:01 AM on June 17, 2014

I've seen two methods - one was a qualifying quiz (same way you might weed out people not fluent in a language) and the other used an application built on MT that solicited the necessary affirmations. However, as a MT user I had no perspective on the approval process.
posted by michaelh at 8:02 AM on June 17, 2014

We've had a few MT-based research projects go through IRB from our department. Our IRB has prepared some guidelines that might be helpful (link to PDF from Univ. of Texas at Austin IRB).
posted by pantarei70 at 8:25 AM on June 17, 2014 [1 favorite]

There's no one answer to this. My university does a ton of MT work and we have people do consent several different ways - some embedded in the MT survey, some take you to an outside page, some ask for consent waivers. It all depends on the specific research. I work closely with the IRB committee but am not actually part of it, so I can't tell you too much detail how it works, just that I'm aware there are different models for how to do it.

I would strongly recommend that before you submit an application, you request a one-on-one meeting with an IRB staff member, and educate that person about what MT involves, find out what questions they have, discuss your thoughts on how you might work this into an application. You'll probably save yourself several rounds of review down the road.
posted by Stacey at 8:31 AM on June 17, 2014 [2 favorites]

what does irb mean, it can't be interactive ruby
posted by bensherman at 9:01 AM on June 17, 2014

IRB=Institutional Review Board
posted by kathrynm at 9:05 AM on June 17, 2014 [1 favorite]

I was taking a class last quarter where the prof mentioned that they make sure that the pay for the MT tasks works out to at least minimum wage.
posted by Blue Jello Elf at 9:58 AM on June 17, 2014