The increase in cell phone ownership during the last decade has led some to conclude that the traditional phone survey must be a defunct public opinion research tool incapable of yielding accurate data on issues that matter. As the reasoning goes, doesn’t everybody just have cell phones now (with which they screen calls)? Who answers a landline to talk to a stranger around dinner time?
Although the answer to the first question is “mostly” – meaning, yes, a lot of people are cell phone-only now — the trend’s impact on data quality has been minimal. Furthermore, the people answering telephone surveys, sometimes on landlines and other times on cell phones, are more or less just like you and me, with a few minor exceptions that statistical tools easily correct. This may sound surprising until you understand how phone surveys work.
It is true that, according to the Pew Research Center, 20 years ago 36% of people contacted responded to the survey and that’s dropped to 9% now. However, this 9% response rate has been steady for the last four years and has stabilized. Now, one might suppose, doesn’t this mean that whoever those 9% of people are they probably are biased on a whole range of issues; maybe they’re politically affiliated a certain way, or elderly, or uneducated, or skewed in some other way? Such a low response rate means that the potential scope of phone survey research is severely limited, right?
The answers here are, yes, there is just a little bias (but a lot less than you would think) and nope, it shouldn’t make hardly a whit of difference for phone polling.
Let’s start with response rate. Scholarly paper after scholarly paper has shown time and again that low response rates rarely lead to much bias when the surveys themselves are high quality. One big reason we know this is because we compare response rates from this 9% to very high response rate general surveys – like the General Social Survey – conducted in person. Such high response general surveys are a benchmark against which phone surveys can be tracked. And once those comparisons are made, there’s very little difference on the important measures of party affiliation, political ideology or religious beliefs between the two.
Now, regarding demographics, the difference between people who answer phone surveys and overall U.S. demographic profile is relatively small. As more phone survey calls are made to cell phones, the representation in surveys of young people and Hispanics, for example, has increased. As long as cell phone-only respondents are included in a sample, the bias is minimal and, regardless, statistical weighting easily corrects for minor demographic variable imbalances. Is this simple to do? No. Is it cheap and easy to get a reliable sample? No. But good research never has been simple, cheap or easy. To generate sound research conclusions, the pollster must be up to the challenges posed by lower response rates.
Telephone polls do tend to consistently over-represent one key variable – civic engagement. People who participate in phone surveys are more likely to be “joiners” and more like to be aware of what’s going on in their community. Fortunately, however, this doesn’t correlate much with partisan political attitudes or most other things pollsters are trying to determine. Where this does make slight a difference is that those predisposed to “join” are also more politically engaged. Yet even here, sound data collection and weighting methods will conform samples to the general population.
There’s a more complex story to be told here, of course, in the technical language of statistics. Our expert researchers at Competitive Edge Research & Communication speak this language. They would be happy to answer any general or technical questions you may have. Please feel free to give us a call if you’ve got an issue into which you need insight and analysis. We’ve served San Diego for 30 years, and have done it well by listening closely to our client’s needs. We offer a full range of public opinion and market research methodologies – including on-line, and we’re proud to offer a cutting-edge, professionally staffed call center as a key component of our research mix.