RReThose Happy Numbers May Not Be Real
Researchers know the feeling: a client’s satisfaction scores come back looking great, and something about them still feels off. A new CERC study helps explain why, pointing straight back to a problem we’ve tracked for a while.
In a just-completed study of 650 interviews conducted over three months, we tested two ways of reaching customers: a texted invitation to complete a web survey, and a live phone interview. The results are striking. Among text respondents, 65% said they were “extremely satisfied” with the organization, and only 8% landed below “very satisfied.” Among phone respondents, just 49% said “extremely satisfied,” while 12% fell below “very satisfied.”
Same organization. Same customers, more or less. Two clearly different pictures of how customers feel.
If you stopped there, you might conclude the phone respondents just had worse experiences. But we that’s not what’s happening. Because it echoes findings we presented earlier this year at the annual conference of the American Association for Public Opinion Research, we think it results from a mode effect.
In that presentation, we compared a mixed-mode political survey to a text-only version of the same instrument. The pattern was consistent: text-only respondents offered fewer “unsure” answers and skewed toward people with strong, settled opinions. Weighting the data to match demographics didn’t erase the gap. That’s because the divide wasn’t about who people are: it was about who chooses to click a survey link in the first place.
Text-initiated surveys reward enthusiasm. The person who taps through a text invitation is by definition someone wanting to engage, often because they already feel strongly and are already amenable. A skilled phone interviewer, by contrast, collects data from many people who never intended to take a survey. Phone respondents don’t select themselves based on their desire to take a survey or based on how they feel about the organization; a good interviewer coaxes them through the call anyway.
That’s why the satisfaction numbers diverge the way they do. Texts catch people who are already primed to take a survey. Phone surveys are catching everyone, including customers with a complaint they otherwise never would have volunteered.
Here’s the uncomfortable part for anyone chasing bigger, cheaper samples: the “worse” numbers may be the more useful ones. An organization that wants to actually improve doesn’t need 65% of respondents telling it everything is wonderful. It needs the 51% of phone respondents who didn’t say “extremely satisfied” because those are the voices that flag what to fix.
Does this mean text-based data collection is worthless? No. It’s fast, it’s less expensive, and for some questions it may be adequate. But when the goal is an honest read on customer sentiment — not just a flattering topline number — live interviewing remains the standard. It reaches the reluctant, the ambivalent, and the quietly dissatisfied: the type of people a text link can’t persuade to respond.
The industry’s push toward automation isn’t going away, and it shouldn’t. But clients — and the researchers selling numbers to them — should be clear-eyed and honest about what each mode actually measures.