How can I be a genuine poll statistic, not a target for shills?
July 12, 2010 9:20 PM Subscribe
How can I identify a legitimate political election phone poll at the beginning of the call? I don't mind participating in real polls but I often get ones with a hidden agenda of convincing me a candidate or proposition is bad.
After a few minutes participating in a phone survey for elections I can usually tell whether it's a legitimate "random phone call for polling purposes" situation or one with a disguised political agenda. The one I had yesterday spent a considerable amount of time 5 minutes into the call basically asking me questions that were phrased like "Would you still vote for Senator BigWig if I told you he kicks puppy dogs? Did you know he steals children's allowances and gives the money to oil companies? Would you still vote for him?" (I'm exaggerating for comedic effect of course.) And the caller kept saying "Thanks! We're almost done!" Are there questions I can ask up front to verify that the poll is for legitimate purposes and not to convince me to vote for someone? (Assuming they won't lie of course.)
After a few minutes participating in a phone survey for elections I can usually tell whether it's a legitimate "random phone call for polling purposes" situation or one with a disguised political agenda. The one I had yesterday spent a considerable amount of time 5 minutes into the call basically asking me questions that were phrased like "Would you still vote for Senator BigWig if I told you he kicks puppy dogs? Did you know he steals children's allowances and gives the money to oil companies? Would you still vote for him?" (I'm exaggerating for comedic effect of course.) And the caller kept saying "Thanks! We're almost done!" Are there questions I can ask up front to verify that the poll is for legitimate purposes and not to convince me to vote for someone? (Assuming they won't lie of course.)
I had a push poller hang up on me when I refused to give a biased answer i.e.
Poller: How do you feel about Candidate X? Favorable, moderately favorable, moderately unfavorable, or unfavorable? Or do you not know of candidate x's existence.
me: I feel neutral about X.
Poller: there is no neutral, So i'll put you down as not knowing.
me: No, I know who X is, I am simply neutral about him.
Poller: Well there is no neutral. So just pick one.
me: Neutral.
Poller:-click-
me: sniff
Meaning that if there is no way to have a neutral response to the questions, I smell an agenda and as above don't play along. So perhaps listen for whether there are any neutral possible answers to your first question and hang up if there's an obvious tilting one way or another.
It should likely be noted that since that call, I haven't been called again (hopefully i've been blacklisted for non-responsiveness) so I can't fully test my theory.
posted by Cold Lurkey at 9:44 PM on July 12, 2010
Poller: How do you feel about Candidate X? Favorable, moderately favorable, moderately unfavorable, or unfavorable? Or do you not know of candidate x's existence.
me: I feel neutral about X.
Poller: there is no neutral, So i'll put you down as not knowing.
me: No, I know who X is, I am simply neutral about him.
Poller: Well there is no neutral. So just pick one.
me: Neutral.
Poller:-click-
me: sniff
Meaning that if there is no way to have a neutral response to the questions, I smell an agenda and as above don't play along. So perhaps listen for whether there are any neutral possible answers to your first question and hang up if there's an obvious tilting one way or another.
It should likely be noted that since that call, I haven't been called again (hopefully i've been blacklisted for non-responsiveness) so I can't fully test my theory.
posted by Cold Lurkey at 9:44 PM on July 12, 2010
Just hang up once you realize that they're asking the leading questions. You don't actually need to be polite to these callers. They'll just move on to the next house.
Although I bet a lot of the time you can figure out what kind of thing it will be by asking who they're affiliated with/calling for.
posted by that girl at 9:46 PM on July 12, 2010
Although I bet a lot of the time you can figure out what kind of thing it will be by asking who they're affiliated with/calling for.
posted by that girl at 9:46 PM on July 12, 2010
AAPOR (American Association for Public Opinion Research) statement about push polls. Some advice on what to do (if you want to be aggressive about it).
posted by nangar at 10:05 PM on July 12, 2010 [1 favorite]
posted by nangar at 10:05 PM on July 12, 2010 [1 favorite]
I don't think this is a push poll. It is likely that the polls you mention are legit.
Sure, "push polls" (fake polls that are really intended to spread disinformation) exist. But successfully completed phone calls to statistically significant numbers of voters is simply not a cost-effective way to communicate with voters--it would be way cheaper and more effective to run (more) negative TV ads for, say, a Congressional than it would be to have a liver operator actually reach 100,000 people on the phone (that's like a million dials).
What I would guess was going on was message testing. They run people through different batteries of positive and negative messages, then ask you again if you'd vote for them. The idea is to gauge movement from the first "ballot test" to the second one, to see which positive/negative messages it correlates to, and to generally learn which of their stuff works and which doesn't.
Now, you may or may not be OK with this, but it is definitely legit public-opinion research. They are testing messages with (what they hope is) a valid statistical sample of the electorate (as they model it). It's way more likely they're spending about $15k to make 500 successful interviews than so as to inform larger media buys than that they are spending $500k at the least to try to make a dent with a push poll.
posted by Ignatius J. Reilly at 10:10 PM on July 12, 2010 [1 favorite]
Sure, "push polls" (fake polls that are really intended to spread disinformation) exist. But successfully completed phone calls to statistically significant numbers of voters is simply not a cost-effective way to communicate with voters--it would be way cheaper and more effective to run (more) negative TV ads for, say, a Congressional than it would be to have a liver operator actually reach 100,000 people on the phone (that's like a million dials).
What I would guess was going on was message testing. They run people through different batteries of positive and negative messages, then ask you again if you'd vote for them. The idea is to gauge movement from the first "ballot test" to the second one, to see which positive/negative messages it correlates to, and to generally learn which of their stuff works and which doesn't.
Now, you may or may not be OK with this, but it is definitely legit public-opinion research. They are testing messages with (what they hope is) a valid statistical sample of the electorate (as they model it). It's way more likely they're spending about $15k to make 500 successful interviews than so as to inform larger media buys than that they are spending $500k at the least to try to make a dent with a push poll.
posted by Ignatius J. Reilly at 10:10 PM on July 12, 2010 [1 favorite]
I have a marketing degree and for years felt it would be hypocritical and bad karma for me to not play along with phone survey people. In the past five+ years, I've changed my tune, and here is my reasoning:
1) nearly all of them, even the relatively legit ones, are engaging in some form of respondent abuse, meaning that we were taught in school that we should design surveys that would be over and done with in a reasonable period of time. If the industry EVER really did this, they don't any more. It's too expensive to get a sucker on the line, and it's too easy to just keep adding on questions. Start playing along, and fifteen minutes into those tedious and excessively personal questions, you'll really wish you hadn't.
2)Perhaps as recently as the early '90s, telephoning households could be done under the color of reaching some random sample of the population and getting good data that could be of benefit to society or help develop better products. This is a ridiculous assumption today. I still have a landline, for reasons that are becoming vague even to me (mainly has to do with inertia, and hating to update all those records out there which have this number), but I hardly know anyone under 30 who does. And those who do, have caller ID, and increasingly anyone with half an oz. of gumption doesn't answer the damn thing unless they recognize the number as someone they want to talk to. So if an NBC poll, for example, really got their data from a telephone survey, rest assured the sample is skewed toward older, less technologically savvy and/or lower-income people who don't have caller ID and/or the gumption to ignore the call or hang up on the surveyor. IOW, the data is garbage. Why be a part of it?
3) You really don't have any way of knowing who you're talking with. A LOT of political polling is push polling, or college kids (like I was) doing surveys for a class project (and I feel bad for them, in a way, and if I see a university on my caller ID I will admit to at least giving them a shot sometimes), or it could be some complete scammers. Even when it doesn't quite feel like a push poll, it's usually someone with an agenda who didn't take the time or spend the money to get a professional to write the survey questions (or again, because this whole technique is going by the wayside, maybe a pro just wouldn't be doing this any more). Hell, I'm not saying *I'm* an expert; I just got a bachelor's and I have no real-world experience being paid to do this, and I can tell the bias in the questions and I could have done a better job of writing it in a neutral way.
4) The more you play along, the more they'll call. The industry is supposed to be calling people completely at random - taking exchanges (area code + first 3) and filling in the last 4 digits randomly. This is to get unlisted numbers and give everyone in the population an even chance to be called (I was taught this before the DNC lists were the law; presumably they filter out those numbers, but I'm not sure they're required to). You may laugh now if you wish. They're under pressure to get surveys done, often in a short time frame; they'll call their sucker lists - those who have completed surveys in the past.
5) Let's face it - they're taking up your time, paying you nothing, often trying to manipulate you, sell you something, or sell your responses to someone (i.e. tagging you for followup by a salesrep or a political campaign if you answer a certain way). Why? Why bother? You pay for your phone line; it's for your convenience, not theirs.
I guess finally, on the political side, I've developed some doubts (profound ones) about the modern process of political polling as a means of feedback or shaping of public thought. Why, exactly, does it benefit society to have the candidates or elected officials knowing exactly what percentage of us think what? Even if the data was great, which it isn't, why should candidates get to find out exactly what pandering messages stroke the electorate just right? Why shouldn't they just stay in touch with their constituents by personal contacts and listening to the ones who call in or write about stuff, and make qualitative decisions about what they think is right, not based on what will or won't matter enough to swing their election?
posted by randomkeystrike at 10:10 PM on July 12, 2010 [4 favorites]
1) nearly all of them, even the relatively legit ones, are engaging in some form of respondent abuse, meaning that we were taught in school that we should design surveys that would be over and done with in a reasonable period of time. If the industry EVER really did this, they don't any more. It's too expensive to get a sucker on the line, and it's too easy to just keep adding on questions. Start playing along, and fifteen minutes into those tedious and excessively personal questions, you'll really wish you hadn't.
2)Perhaps as recently as the early '90s, telephoning households could be done under the color of reaching some random sample of the population and getting good data that could be of benefit to society or help develop better products. This is a ridiculous assumption today. I still have a landline, for reasons that are becoming vague even to me (mainly has to do with inertia, and hating to update all those records out there which have this number), but I hardly know anyone under 30 who does. And those who do, have caller ID, and increasingly anyone with half an oz. of gumption doesn't answer the damn thing unless they recognize the number as someone they want to talk to. So if an NBC poll, for example, really got their data from a telephone survey, rest assured the sample is skewed toward older, less technologically savvy and/or lower-income people who don't have caller ID and/or the gumption to ignore the call or hang up on the surveyor. IOW, the data is garbage. Why be a part of it?
3) You really don't have any way of knowing who you're talking with. A LOT of political polling is push polling, or college kids (like I was) doing surveys for a class project (and I feel bad for them, in a way, and if I see a university on my caller ID I will admit to at least giving them a shot sometimes), or it could be some complete scammers. Even when it doesn't quite feel like a push poll, it's usually someone with an agenda who didn't take the time or spend the money to get a professional to write the survey questions (or again, because this whole technique is going by the wayside, maybe a pro just wouldn't be doing this any more). Hell, I'm not saying *I'm* an expert; I just got a bachelor's and I have no real-world experience being paid to do this, and I can tell the bias in the questions and I could have done a better job of writing it in a neutral way.
4) The more you play along, the more they'll call. The industry is supposed to be calling people completely at random - taking exchanges (area code + first 3) and filling in the last 4 digits randomly. This is to get unlisted numbers and give everyone in the population an even chance to be called (I was taught this before the DNC lists were the law; presumably they filter out those numbers, but I'm not sure they're required to). You may laugh now if you wish. They're under pressure to get surveys done, often in a short time frame; they'll call their sucker lists - those who have completed surveys in the past.
5) Let's face it - they're taking up your time, paying you nothing, often trying to manipulate you, sell you something, or sell your responses to someone (i.e. tagging you for followup by a salesrep or a political campaign if you answer a certain way). Why? Why bother? You pay for your phone line; it's for your convenience, not theirs.
I guess finally, on the political side, I've developed some doubts (profound ones) about the modern process of political polling as a means of feedback or shaping of public thought. Why, exactly, does it benefit society to have the candidates or elected officials knowing exactly what percentage of us think what? Even if the data was great, which it isn't, why should candidates get to find out exactly what pandering messages stroke the electorate just right? Why shouldn't they just stay in touch with their constituents by personal contacts and listening to the ones who call in or write about stuff, and make qualitative decisions about what they think is right, not based on what will or won't matter enough to swing their election?
posted by randomkeystrike at 10:10 PM on July 12, 2010 [4 favorites]
I should probably have said "live operator" instead of "liver operator."
posted by Ignatius J. Reilly at 10:17 PM on July 12, 2010
posted by Ignatius J. Reilly at 10:17 PM on July 12, 2010
Yay! Finally, an askme I have some inside info on!
I was a telephone pollster for years. The company I worked for conducted polls for...well, anyone who paid them to do it.
Usually, these were indeed politicians, but the purpose was twofold...the push polling to try to get you to vote for our guy was one. But, even in those cases, your answers are indeed recorded, and then the data is crunched and analyzed. This analysis, in turn, lets the candidate know which issues really "win" for him, and with who, and which don't.
So even if it's for a politician, the answers to your questions are still counting for something. Obviously, whether it's a politician you care to give a helpful data point to is another question entirely.
But! One of our clients was a major television network/newspaper partnership, and this was the "legit" poll. And the way to tell the two apart is simplicity itself.
If I am a "legit" poll, particularly one with a lot of name respect and name recognition, I can't WAIT to tell you who I am calling on behalf of! And then you'll say, "Oooh! Am I gonna be on TV?" And I will say, "Yes, if you are one of the people the reporters select to get more detail and color on their opinions!" (True, BTW, at least in my case).
It was SO easy to put in a lot of volume on such polls, and so very hard to on the politician-paid ones, especially since oftentimes the politician wanted a certain demographic. So now we have to find only black females age 40-65 on our randomly-generated (true) phone number sheets.
Secondly, if you asked us who was paying for the survey, I would say something like, "I am not permitted to say as it might influence certain respondent's answers."
Bottom line, if they are "legit," they know it will only help THEM to tell you that. If they don't want to say, it's a push poll.
posted by mreleganza at 10:29 PM on July 12, 2010 [2 favorites]
I was a telephone pollster for years. The company I worked for conducted polls for...well, anyone who paid them to do it.
Usually, these were indeed politicians, but the purpose was twofold...the push polling to try to get you to vote for our guy was one. But, even in those cases, your answers are indeed recorded, and then the data is crunched and analyzed. This analysis, in turn, lets the candidate know which issues really "win" for him, and with who, and which don't.
So even if it's for a politician, the answers to your questions are still counting for something. Obviously, whether it's a politician you care to give a helpful data point to is another question entirely.
But! One of our clients was a major television network/newspaper partnership, and this was the "legit" poll. And the way to tell the two apart is simplicity itself.
If I am a "legit" poll, particularly one with a lot of name respect and name recognition, I can't WAIT to tell you who I am calling on behalf of! And then you'll say, "Oooh! Am I gonna be on TV?" And I will say, "Yes, if you are one of the people the reporters select to get more detail and color on their opinions!" (True, BTW, at least in my case).
It was SO easy to put in a lot of volume on such polls, and so very hard to on the politician-paid ones, especially since oftentimes the politician wanted a certain demographic. So now we have to find only black females age 40-65 on our randomly-generated (true) phone number sheets.
Secondly, if you asked us who was paying for the survey, I would say something like, "I am not permitted to say as it might influence certain respondent's answers."
Bottom line, if they are "legit," they know it will only help THEM to tell you that. If they don't want to say, it's a push poll.
posted by mreleganza at 10:29 PM on July 12, 2010 [2 favorites]
And oh yes...as mentioned above, if you are not interested in doing the survey but still want to be nice, just HANG UP on us. We won't think it rude. It will let us move onto the next call faster.
As you might imagine, this is a shitty, lowpaying job, and the caller is very likely NOT an acolyte of the politician they are working on behalf of, so PLEASE, never take out your frustrations on the practice, or the sponsor of it, on the operator.
posted by mreleganza at 10:37 PM on July 12, 2010 [1 favorite]
As you might imagine, this is a shitty, lowpaying job, and the caller is very likely NOT an acolyte of the politician they are working on behalf of, so PLEASE, never take out your frustrations on the practice, or the sponsor of it, on the operator.
posted by mreleganza at 10:37 PM on July 12, 2010 [1 favorite]
Not that I've done a lot of polling work, but I always appreciated at least a "no, thanks" or a "no, but have a nice day" before the call ended. It costs you nothing, while hangups with no warning say "I will not obey even minor social niceties and basically consider you to be the same as a machine." I suppose if you're going for sheer volume, the hangup could be useful, but for delicate blossoms like me a little politeness went a long way--especially because I always tried to sound at least a little human and appreciated similar humanity from those I called.
posted by verbyournouns at 11:30 PM on July 12, 2010
posted by verbyournouns at 11:30 PM on July 12, 2010
I used to work in a place like mreleganza described- mostly obvious push polls for candidates I had never heard of in far away states. I'm not sure if the company I worked for was especially shady, but it definitely wasn't all above board.
We were given front company names for each different poll, usually something like The Jefferson Group, or American Surveys, or something like that. My company told us to be as evasive as possible, short of seriously breaking the law. We would make up names, lie about our location, use fake accents, call people too late at night etc. If the pollster is actually working for reputable company conducting fair surveys they will make sure you know that. In short, if it sounds fishy, it definitely is.
Tell them you aren't interested and would like to be placed on the do-not-call list. If you don't specifically ask to be placed on the list you can almost guarantee we will mark you for a call-back, just to be vindictive. Also, be polite, it's an awful job and you get cursed out constantly all day. Even polite no's can make a pollsters day a little better.
posted by ryaninoakland at 1:10 AM on July 13, 2010
We were given front company names for each different poll, usually something like The Jefferson Group, or American Surveys, or something like that. My company told us to be as evasive as possible, short of seriously breaking the law. We would make up names, lie about our location, use fake accents, call people too late at night etc. If the pollster is actually working for reputable company conducting fair surveys they will make sure you know that. In short, if it sounds fishy, it definitely is.
Tell them you aren't interested and would like to be placed on the do-not-call list. If you don't specifically ask to be placed on the list you can almost guarantee we will mark you for a call-back, just to be vindictive. Also, be polite, it's an awful job and you get cursed out constantly all day. Even polite no's can make a pollsters day a little better.
posted by ryaninoakland at 1:10 AM on July 13, 2010
I consider it shady when the caller won't identify his/her employer beyond "an organization" or "an activist group" (and that's when you push!).
I like to hear the questions even if I believe it to be a push poll, but unless they work for a news organization, candidate, company or organization I've heard of or can quickly verify, I don't have much faith in the validity of their results, and I tend to nitpick questions (e.g., how can I rate my experience if that items isn't relevant to me?) more than surveys that I consider to be legitimate.
posted by casualinference at 5:25 AM on July 13, 2010
I like to hear the questions even if I believe it to be a push poll, but unless they work for a news organization, candidate, company or organization I've heard of or can quickly verify, I don't have much faith in the validity of their results, and I tend to nitpick questions (e.g., how can I rate my experience if that items isn't relevant to me?) more than surveys that I consider to be legitimate.
posted by casualinference at 5:25 AM on July 13, 2010
There are valid methodological reasons to withhold the name of the end client. If you know McCain is paying for a poll it might make you act differently than if you knew Obama was, for example.
I have commissioned or been involved in the commissioning or analysis of hundreds of legit polls (always with reputable pollsters), and not once have we gone through the process without being accused on "push polling." Quite simply, calling people and asking "You gonna vote for X or Y?" doesn't give you any actionable information and is a waste of money. The whole point of internal polling (as opposed to "horse race" polling by a newspaper or something) is to learn the effectiveness of various message for and against the candidates.
One thing that you can count on like clockword in every campaign is having one of your candidate's supporters call up and report on this nasty "push poll" which is actually the very survey that the candidate herself paid for -- you've got to know which of the other guy's attacks work, as well.
People are right to observe that the call centers that are contracted to conduct the interviews can sometimes be shady, but the answer to the OP's question (are these "random phone call for polling purposes?") is yes.
posted by Ignatius J. Reilly at 10:48 AM on July 13, 2010
I have commissioned or been involved in the commissioning or analysis of hundreds of legit polls (always with reputable pollsters), and not once have we gone through the process without being accused on "push polling." Quite simply, calling people and asking "You gonna vote for X or Y?" doesn't give you any actionable information and is a waste of money. The whole point of internal polling (as opposed to "horse race" polling by a newspaper or something) is to learn the effectiveness of various message for and against the candidates.
One thing that you can count on like clockword in every campaign is having one of your candidate's supporters call up and report on this nasty "push poll" which is actually the very survey that the candidate herself paid for -- you've got to know which of the other guy's attacks work, as well.
People are right to observe that the call centers that are contracted to conduct the interviews can sometimes be shady, but the answer to the OP's question (are these "random phone call for polling purposes?") is yes.
posted by Ignatius J. Reilly at 10:48 AM on July 13, 2010
I actually enjoy the creepy push polls. I just give them totally ludicrous answers. They'll be dismissed as outliers in their statistics, but I'm also wasting the time, and therefore money, of shady organizations.
posted by oneirodynia at 10:50 AM on July 13, 2010
posted by oneirodynia at 10:50 AM on July 13, 2010
oneirodynia and others who talk about "feeding bad data" to push pollers (and I'd be interested from hearing from "researchers" who will admit to constructing or being a call center for push polls as well) - are you even interested in the data collected back? It always seemed to me that the "answers" were beside the point in a true push poll, as the "questions" are really just vehicles for getting the respondees (and anyone else they talk with) some grist from the rumor mill. IOW, to deliver a "message" which would not be believed if someone just called and asserted these things to you, but which might suck a naive person in as part of this bogus survey:
"Will it affect your decision in Tuesday's run-off if candidate Wintergreen admits to having buttsex with underage boys?"
(next day in the barber shop)
"You know I got a call from one of these survey fellers last night and they were asking about Wintergreen having buttsex with underage boys..."
MISSION ACCOMPLISHED
posted by randomkeystrike at 1:27 PM on July 13, 2010
"Will it affect your decision in Tuesday's run-off if candidate Wintergreen admits to having buttsex with underage boys?"
(next day in the barber shop)
"You know I got a call from one of these survey fellers last night and they were asking about Wintergreen having buttsex with underage boys..."
MISSION ACCOMPLISHED
posted by randomkeystrike at 1:27 PM on July 13, 2010
I guess Ignatius actually sorta answered my question. Yeah, good old message testing.
posted by randomkeystrike at 1:31 PM on July 13, 2010
posted by randomkeystrike at 1:31 PM on July 13, 2010
Response by poster: Thank you for the informative answers! I feel much better equipped to handle the next few months of phone calls. And I learned the term "push poll"!
posted by girlhacker at 9:19 PM on July 13, 2010
posted by girlhacker at 9:19 PM on July 13, 2010
This thread is closed to new comments.
posted by Nanukthedog at 9:40 PM on July 12, 2010