What’s the Benefit of Interviewers? What’s the Problem with Texts?

The Push Toward Automation
Caller ID and call screening have made live interviewing increasingly difficult and expensive, accelerating the industry’s movement toward self-administered survey modes. CERC’s question was not simply whether text-based research is cheaper or easier, but what changes when researchers move away from mixed-mode studies and rely exclusively on text invitations.

A Live Field Comparison
CERC compared two voter studies conducted just weeks apart. The mixed-mode study included 802 voters contacted through text, phone, and email; the text-only study included 307 voters contacted entirely by text. Both studies used comparable weighting parameters and shared key measures of civic mood and elected-official approval.

Different Modes Reach Different People
Although the studies drew from the same underlying voter-data environment, meaningful differences emerged in both the available contact frame and the respondents who ultimately participated. Textable voters differed from those without usable mobile numbers, while the completed text-only sample also differed from the electorate in ways that required greater weighting adjustments.

The Opt-In Factor
CERC’s central hypothesis is that a text invitation creates a stronger self-selection barrier: recipients must actively decide that they want to take a survey. Phone respondents, by contrast, are contacted proactively and may not initially know that the call is a survey. Skilled interviewers can therefore bring less-engaged respondents into the research process—respondents who may be systematically missing from text-only samples.

Why It Matters
Text-based research is cheaper—the presentation notes that CERC’s profit margin on the text-based study was 13% higher—but the findings suggest a potential tradeoff in data quality. CERC’s conclusion is not that researchers should reject self-administered methods, but that they should understand and disclose the tradeoffs and optimize across modes rather than reflexively eliminating live interviewers.

Is America truly becoming more polarized… or is it just a mode effect?

Those Happy Numbers May Not Be Real

Researchers know the feeling: a client’s satisfaction scores come back looking great, and something about them still feels off. A new CERC study helps explain why, pointing straight back to a problem we’ve tracked for a while.

In a just-completed study of 650 interviews conducted over three months, we tested two ways of reaching customers: a texted invitation to complete a web survey, and a live phone interview. The results are striking. Among text respondents, 65% said they were “extremely satisfied” with the organization, and only 8% landed below “very satisfied.” Among phone respondents, just 49% said “extremely satisfied,” while 12% fell below “very satisfied.”

Same organization. Same customers, more or less. Two clearly different pictures of how customers feel.

If you stopped there, you might conclude the phone respondents just had worse experiences. But we that’s not what’s happening. Because it echoes findings we presented earlier this year at the annual conference of the American Association for Public Opinion Research, we think it results from a mode effect.

In that presentation, we compared a mixed-mode political survey to a text-only version of the same instrument. The pattern was consistent: text-only respondents offered fewer “unsure” answers and skewed toward people with strong, settled opinions. Weighting the data to match demographics didn’t erase the gap. That’s because the divide wasn’t about who people are: it was about who chooses to click a survey link in the first place.

Text-initiated surveys reward enthusiasm. The person who taps through a text invitation is by definition someone wanting to engage, often because they already feel strongly and are already amenable. A skilled phone interviewer, by contrast, collects data from many people who never intended to take a survey. Phone respondents don’t select themselves based on their desire to take a survey or based on how they feel about the organization; a good interviewer coaxes them through the call anyway.

That’s why the satisfaction numbers diverge the way they do. Texts catch people who are already primed to take a survey. Phone surveys are catching everyone, including customers with a complaint they otherwise never would have volunteered.

Here’s the uncomfortable part for anyone chasing bigger, cheaper samples: the “worse” numbers may be the more useful ones. An organization that wants to actually improve doesn’t need 65% of respondents telling it everything is wonderful. It needs the 51% of phone respondents who didn’t say “extremely satisfied” because those are the voices that flag what to fix.

Does this mean text-based data collection is worthless? No. It’s fast, it’s less expensive, and for some questions it may be adequate. But when the goal is an honest read on customer sentiment — not just a flattering topline number — live interviewing remains the standard. It reaches the reluctant, the ambivalent, and the quietly dissatisfied: the type of people a text link can’t persuade to respond.

The industry’s push toward automation isn’t going away, and it shouldn’t. But clients — and the researchers selling numbers to them — should be clear-eyed and honest about what each mode actually measures.