Implied timeframe in survey question-did I miss the memo?

March 17, 2023 11:37 AM

I was recently asked to participate in a local government survey. One of the questions was "Do you believe that race relations in the United States are getting worse, getting better or staying about the same?" There's no specified timeframe, which confuses the heck out of me. Better since the civil war? Better since Trump is out of office? Better this year? I see this sort of question and I wonder if I've missed something obvious- "everybody knows they mean....X"

Perhaps the survey just meant to discern whether you're feeling optimistic about things getting better or not and they don't care if you're thinking of it in a "long arc of history bends towards justice" way or how things are going this month? (Please note: I have seen this format for questions on a wide variety of topics and there's nothing specific about race relations to my question, it's just the most recent example- I'm just curious how to interpret a "trend" question when there's no time frame specified and my answers could be completely opposite depending on what the time frame they're asking about is)

posted by Larry David Syndrome to Society & Culture (10 answers total)

I don't know about your survey but I read that question in the context of something asked annually, so the implication is "since last survey." But it's also probably just fine to answer in a general way.
posted by Alensin at 11:42 AM on March 17, 2023

Best answer: It doesn't mean ARE they better or worse than some previous moment. It means what is the direction of movement right now. Will tomorrow be better than today, worse, or about the same. (Though obviously barring some big event the movement in a single day will be tiny.) So picture a line graph of race relations starting from whenever before now (1776, civil rights, Reagan, 2001, 2020, whenever) to sometime after now (again, whenever). If you look at the little part of the graph that represents this moment, is the slope positive or negative or flat?

I am a survey researcher. I am not your survey researcher. I have used questions like this though not this exact question.
posted by If only I had a penguin... at 11:44 AM on March 17, 2023 [20 favorites]

It's asking "are they getting better" not "have they got better", so I would read it as a question about my expectations of the future, not my interpretation of the past. (So, yes - mostly a question about optimism.) Imagine a year from now: if current trends continue (i.e. there are no big surprises or upheavals), do you think things will be better, worse, or the same?
posted by ManyLeggedCreature at 12:19 PM on March 17, 2023

Best answer: As someone who writes surveys as part of my job, and could conceivably write a very similar survey question, 1. Thank you for alerting me to how a question like this could seem vague or confusing and 2. If I were writing a question like that, I would mean, "do you feel they are getting better or worse RIGHT NOW?" It's intended to be a sentiment gauge as to how people feel they're going at the moment, however you define "this moment."

Also, I think this gets at a central tension in writing survey questions - specificity might seem better, but the more specific you get, often the harder a question is for folks to answer, and the fewer people who will answer it. So we try to only make questions as specific as they need to be for the widest group of respondents. In this case, the survey writer might not have cared much which time frame the answerer was using mentally, preferring to get more responses from people, understanding that different people will mentally use different time frames.

Finally, this kind of question is very common in surveys and tends to be a proxy for gauging a number of different things. When you see this question structure in the future, feel free to just answer it in whatever way makes sense to YOU.
posted by lunasol at 12:57 PM on March 17, 2023 [9 favorites]

I also find this type of question maddeningly vague to the point of unanswerability. It turned out that the obvious thing that I was missing for the first 44 years of my life was that I'm autistic.
posted by heatherlogan at 1:46 PM on March 17, 2023 [5 favorites]

The question is about the current direction of travel, not the current location relative to some other point in time.

"Is the amount of water in the bucket going up, going down or staying the same?" is a perfectly good question. It can be answered without a timeframe and is not ambiguous.

"Is the amount of water in the bucket more, less or the same as last year?" is also a perfectly good question, but it is a different question.

You've been asked something like the first question. I don't think this is a useful question in the context of race relations, but I don't think it's ambiguously worded, and there's no need to interpret it as the second question, or to come up with some other time to compare to.
posted by caek at 3:55 PM on March 17, 2023 [1 favorite]

It's a question measuring people's beliefs.

Just to further illuminate this, the Gallup survey has asked a similar question about crime; in this case they explicitly say the time frame ("Is there more crime in the US than there was a year ago?")*. Since 2000, every survey has has somewhere between 60-80% of people saying that there is more crime than a year ago. In reality, over that same time period the US crime rate has dropped across every major category of crime; there were about 70% as many violent crimes and 60% as many property crimes in 2019 as in 2000.

The point of the survey isn't to establish whether the crime rate is going down or not -- the FBI statistics I linked will tell anyone that. The point of the survey is to establish what people's beliefs about crime are, whether or not they are linked to reality. Because people don't actually act based on reality, they act (in this case, vote) based on their beliefs about reality.

* and by the way, I'd assume a year or so is generally the time frame that is meant about this sort of question unless otherwise specified. "Is your life getting better or worse?" isn't intended to be answered by people thinking that 100 years ago they didn't exist, so their life must be better, nor by people who are thinking that their life is worse because yesterday there was plenty of milk in the carton but today there's less and they're soon going to need to go to the store.
posted by Superilla at 5:26 PM on March 17, 2023 [2 favorites]

It’s asking for a derivative, not a difference
posted by mr_roboto at 9:29 PM on March 17, 2023 [1 favorite]

Mod note: One deleted. Let's stick to answering the OP's question directly rather than discussing someone else's answer. Thanks.
posted by taz (staff) at 11:39 PM on March 17, 2023 [1 favorite]

Response by poster: Thanks to all who answered. I think I figured out what was throwing me- if someone asked me "are things improving for you, staying the same or getting worse?" the implied short term nature of the question would be clear- it's obvious they don't want to hear that things are looking up since I no longer have to deal with that mean teacher I had in 5th grade. A question about a big picture, long term issue like race relations or women's rights could be answered in terms of the long term slope of the graph or the short term slope of the graph. "Things are getting better for women as they've gained the right to vote, own property, have jobs" is true but so is "things are getting worse for women with the assault on reproductive freedoms in many states." There was never any ambiguity that the question involved direction of change vs. absolute levels, but change over what span of time. Seems like a clear consensus that it's a year or less.
posted by Larry David Syndrome at 4:54 AM on March 18, 2023