The respected news data site Five Thirty Eight and the website On The Media have combined their efforts this political season with a look at the media coverage including polling during the 2016 campaign season. As part of this project, the pair produced its “Consumer Handbook to Election Polls.” I couldn’t wait to read it myself.
It turned out the “handbook” was a list of 12 tips on how to judge polls. In the spirit of public service, allow us to elaborate on a few of these tips.
Tip Number 2 says “Ignore national primary polls – they measure nothing.” We agree. Don’t let them influence your voting choices and don’t let them cause you to place a bet on the eventual nominee. But bear in mind that the polls do influence some donors. Because results from national polls are data points that can be strung together, campaign contributors may look to these highly imperfect metrics for signs that a campaign is gelling or flagging. So national polls can greatly influence fundraising success.
As for the TV networks, their polling thresholds to get into the debates have been argued about for a few election cycles. One academic pollster took the unprecedented step of telling debate organizers polls shouldn’t be used in that way. Marist’s Lee Miringhoff tried to be pithy, writing, “Herding is for horses. Not for pollsters doing horserace polls. Neither should the media herd the field in a political horserace via debates.” Miringhoff’s criticisms were dumb. He should get over the fact that polls are the only way to objectively limit a large field of candidates.
We were pleased to read Tip Number Seven: “Know the polling firm – some are waaay better than others.” (They included all three As in waaay). Knowing something about the reputation and performance of the polling firm, and being able to tell the experienced from the inexperienced, can be important.
In Iowa, you can count on J. Ann Selzer’s polling for the Des Moines Register newspaper. Selzer is legendary in polling circles for her abilities. Not only has she been getting her state right for a long time, but getting it right in Iowa is more challenging than most other places. Here’s why.
First, Iowa has caucuses, not primaries and that means far fewer Iowans participate. As a result, the pollster has to do a lot more meticulous work to find the actual voters, and leave out the non-voters. Iowans also get surveyed a lot more than those in other states. A pollster with shoddy methods and sloppy interviewers is liable to screw up in such a well-tilled soil. On top of that, its first-in-the-nation status puts intense pressure on pollsters. Others have been known to wilt, but not Selzer. So trust Selzer in Iowa.
Tip Number Eight needs a little help: “Margin of error and sample size matter less than who’s in the sample.” Not exactly. Yes, the demographic composition of a poll is important, but it should be obvious that small sample sizes can lead to more error. I recommend studying both the key demographics – partisanship, age, gender, ideology, geographic region – and the sample size. Anything that seems off or sample sizes less than 400 should be taken with a grain of salt.