Survey research

July 22, 2005 5:07 AM

Why is it a bad idea to give the same person a survery within a six month period?

I am looking for backup for an argument I am formulating. I remember that it is not the best idea to present a survey to the same person within a six month period, but forget the reasons - anyone know? I am specifically thinking of web surveys.

posted by xammerboy to computers & internet (7 comments total)

Bias. Repeated measurements from the same individual can increase the error and call the reliability of the test — and the conclusions you draw from it — into question, at least compared with repeated measurements from different individuals.

I don't know why the six month period would matter — that seems a bit arbitrary. My educated guess is that a time period like six months would remove bias from temporary conditions, e.g., a recent news-making event influences the person's view on a survey question related to that event.
posted by Rothko at 5:30 AM on July 22, 2005

I'm a sociologist and I've never heard any such general rule of thumb. I could imagine why it might be a bad idea to give the same person surveys within 6 month periods in particular situations:

1) If you're doing a cross-sectional study and not a panel study. In a panel study you're looking to see change over time or to see some things at time 1 and some things at time 2. For example, medical trials are panel studies: Measure illness pre-drug and illness post-drug. In a cross-sectional study you basically want to know what a population looks like: How many people do X. How often do people who do X also do Y? In the case of a crossectional study, all your observations need to be independent of one another. Ideally you wouldn't re-survey the same person *ever*, but I could see some people arguing for some kinds of surveys that the same person 6 months later is sort of kind of a new observation and letting it slide. (It's a dodgy thing to do, but if it couldn't be avoided, you'd probably make the argument that it was kind of sort of ok).

2) If you're in a situation where you don't want answers from time 1 to affect answers at time 2. If you were worried that people would think "Oh, last time I picked #3, so I should pick #3 this time, too or they'll think I'm lying/inconsistent/flight/whatever" or they'll just think "#3" and not really think about the answer because they figure they already picked an answer last time and it's probably still right. 6 months would probably be long enough that they would forget what they had said on the previous survey.

Can you give us some context on what kind of survey this is and what the purpose is?
posted by duck at 6:24 AM on July 22, 2005

Oh, and another reason (maybe particularly an issue with web surveys): It annoys them.
posted by duck at 6:25 AM on July 22, 2005

I used to program web surveys, and in any case where we were sending out a similar or same study more than once, we usually filtered out our panel members who had already responded for the reasons duck enumerated.

There are many times when you do interview the same people over time to study changes in attitude and perception, but that's much juicier data, which costs the client more and (in our case) paid the respondents better each time than a one-off of similar length and difficulty.
posted by Lyn Never at 7:37 AM on July 22, 2005

This survey would be to:

1. Meaure the customer satisfaction and goal attainment of users visiting the site.

2. Gather feedback regarding problem areas / Suggestions for improvements.
posted by xammerboy at 7:43 AM on July 22, 2005

I think the answer was fatigue. What I am also interested in is thoughts on surveying the same people within the same time frame. But I think the answer to that is bias, as said above. I think I answered my own question, but your responses were very helpful - they helped me do that and do a quick check that nothing glaring was wrong.

If anyone has thoughts to add, please do.
posted by xammerboy at 8:00 AM on July 22, 2005

Gather feedback regarding problem areas / Suggestions for improvements.

Slightly off-topic, but a survey isn't the best way (by far) to identify problem areas or get suggestions. Site metrics (for example, points at which people abandon shopping carts) or simply watching some users (who are NOT seasoned employees) doing typical things are much better ways to identify problems. Another alternative is to provide a feedback link ("We want to hear your suggestions - click here!).

Surveys are for statistical information; suggestions can be extremely valuable even if only one person makes it. Surveys need to ask consistent questions over time in order to see trends; identifying specific problems via a survey is problematical because, by definition, those doing the survey don't know exactly what problems users are likely to identify.
posted by WestCoaster at 10:54 AM on July 22, 2005