How to know your qual research pre-test interview was a success?

February 9, 2014 7:11 PM

Hello, qualitative researchers! I'm doing a qualitative research project. i have a series of interview questions i wanted to ask some people. to make sure the questions were good, i asked them to my pre-test subject. the questions seemed clear and their responses seemed interesting. but is there some more scientific way of saying (or deciding) "Yup, seems good."? I need to write in my thesis how i came to the conclusion that the pre-test was successful. I can see how this would be easy(or more precise) in quantitative research, but i don't have any numbers. How do you do this? And how do you word it? Thank you!
posted by andreapandrea to Science & Nature (5 answers total)

Look for the term pilot not pre test. Any methods textbook should give you some guidelines. Generally questions that didn't bomb. :-)

This also depends how strictly you're following a particular question writing protocol.
posted by k8t at 7:46 PM on February 9, 2014 [1 favorite]

I haven't actually done this kind of research before, but if I had to, I would think about administering a secondary survey to your pilot subjects asking them to score the questions on whether they were clear, easy to answer, likely to elicit appropriate answers, etc, as you like.

If it's too late to do this, then I would try to document the sort of information the question was designed to elicit, document the information that was actually elicited, and then try to show that the responses fit within what was expected. More or less "yup, seems good" with some structure to it.
posted by PercussivePaul at 8:11 PM on February 9, 2014

Best answer: Most qualitative researchers don't do the kind of thing you describing quantitative researchers doing because qual researchers are usually not postpositivist, like many quant researchers. That is, qual researchers are typically not trying to assess an objective reality with reliability and validity- they're trying to co-create a subjective reality in the interview space. These more constructed understandings have different sets of criteria to determine whether they're high quality than reliability and validity (which is what the numbers you mentioned serve to demonstrate). Guba and Lincoln (1989) is a classic cite for quality criteria for qualitative research- take a look at their trustworthiness and authenticity criteria. Memail me if you need the original article as I can't find a link to it. Honestly, though, all you really need to do (and you don't even really need to do this generally in qualitative research- at least I never have) is say that the protocol was pilot tested. The most I've said about a protocol is that it was revised in collaboration with stakeholders, but that doesn't sound like what happened here.

PS- "Pre-test" and "subject" are both very non-qualitative terms- as k8t says, look for "pilot" instead of "pre-test." "Participant" is strongly preferred over "subject"- I don't think I've ever seen published qualitative analyses that use the term "subject."
posted by quiet coyote at 9:01 PM on February 9, 2014 [7 favorites]

Best answer: It depends on your specific qualitative method. A good book on interviewing for qualitative research is InterViews: Learning the craft of qualitative research interviewing (Kvale & Brinkmann, 2009) - although their methods are more semi-structured interviews than structured interviews, they give a lot of great information about how to actually conduct an interview.

In the book, they talk about how there are a bunch of different types of questions in qualitative research; broadly speaking, they break down to:

1. Introductory questions ("Can you describe to me in as much detail as possible ____")
2. Follow-up questions (Repeating phrases back, directly questioning what is just said, etc.)
3. Probing questions ("Can you tell me more about that?")
4. Specifying questions ("What did you do when ___")
5. Direct questions ("Have you ever ___")
6. Indirect questions ("How do you think other people experience ___")
7. Structuring questions ("Let's introduce another topic")
8. Silence (Pauses in questioning to let participant think)
9. Interpreting questions ("Do you mean ____")

So you might want to make sure you have some of these things written into the guide. You also generally want to think about four "groups" of questions (just a different typology, from the Berg & Lune book cited below, basically):

1. Essential questions (the meat of the research)
2. Extra questions (same as essential questions, but worded differently)
3. Throw-away questions (build rapport, switch focus, adjust pace)
4. Probing questions ("Can you tell me more")

You also want to avoid double-barreled questions (two issues in one question) and complex questions. Affectively worded questions are also problematic - remain neutral in your questioning.

You might want to also check out the book Qualitative research methods for the social sciences (Berg & Lune, 2011) - it has a lot of really great information on interviews as well.

There are two general verification techniques for interview guides: have peers/mentors/experts look over the guide and identify any technical problems with the questions, and also do a pretest with potential subjects (which you've done). Your documentation of the pretest results gives you a justification for changing/rewording/removing/adding questions. Just make sure you document everything you do, like with all research projects.

Again, though, the way to handle this really does depend on your method and how much structure your interviews have. Totally structured interviews are essentially just surveys done in person or on the phone. Unstructured interviews are basically just informal conversations with no pre-determined guide. Semi-structured interviews - which are probably the most common type of interview (I think) in qualitative research fall in the middle. They're not as rigid as structured interviews, but they are more systematic than unstructured interviews.

Good luck, and have fun. Interviews are a great data collection method!
posted by k8lin at 9:05 PM on February 9, 2014 [5 favorites]

Have you read good (perhaps published in a respected journal in your field) qualitative research? How have they done it? I don't have a specific citation to suggest, but it may be a route to try.

Generally, my impression of qualitative research is that there aren't really summative "this is significant because z>2" kind of tools. Instead, you use words to describe it. So you might say your interview protocol was finalized during successful piloting, when participants answered questions as they were intended and did not ask for clarification as to the meaning of the question. The whole point of methods is to allow the reader to judge whether they can reasonably believe your conclusions. Often, you include the interview protocol as an appendix so that the reader can say, yes, this looks clear and like the interviewer may have learned what she claims to have learned when she asked people these questions.

Also, what k8t said. If you're googling, try the word "pilot." Pre-test will probably throw you into quantitative experimental design results.
posted by kochenta at 11:07 AM on February 10, 2014