Margin of error is polls’ fine print

By J. Patrick Coolican (contact)

Tuesday, Oct. 14, 2008 | 2 a.m.

Chris Morris

The headline couldn’t be more clear: “Poll gives big lead to Heller.”

Indeed, a Las Vegas Review-Journal poll showed voters in the 2nd Congressional District prefer Rep. Dean Heller to former higher education Regent Jill Derby 51 percent to 38 percent.

The problem with the poll, and thus the headline, could be found in the fine print.

Mason-Dixon Polling & Research, the company commissioned to gauge the opinion of the voting public, received responses from just 221 registered voters, giving the poll a margin of error of plus or minus 7 percentage points. That’s a larger margin of error than those of nearly all of the hundreds of polls published this election cycle.

“I’ve never seen a published poll with a sample that small,” said Scott Keeter, director of survey research at the Pew Research Center. (By contrast, Gallup talks to at least 2,700 people for its presidential tracking poll, which has a margin of error of plus or minus 2 percentage points.)

Given the large margin of error of the Review-Journal poll, Heller could actually be at 44 or 58, and Derby could be at 31 or 45. The odds are small that Heller is preferred by 44 percent and Derby by 45 percent, but it’s possible.

“That headline is not statistically correct,” said Nancy Mathiowetz, a past president of the American Association for Public Opinion Research and chairwoman of the sociology department at the University of Wisconsin-Milwaukee.

By many indications, including the Republican voter registration advantage in the district, his easygoing rapport with constituents and his conservative credentials, Heller appears to be in solid shape to return to Congress next year. But statisticians and other pollsters, as well as journalism scholars, question the accuracy of the Review-Journal poll because of its margin of error.

Political scientists say the very existence of polls like that one can affect public opinion, with a “bandwagon effect” or an “underdog effect,” while also influencing the decisions of potential campaign donors.

Brad Coker, the Mason-Dixon pollster, said his company was hampered by a tight schedule. The Review-Journal wanted to wait until after the second presidential debate Oct. 7 to gauge opinions, but also wanted to have the poll ready for the Sunday paper.

(The paper ran Sunday stories on a poll of the presidential race, as well as the contest in the 3rd Congressional District, where Mason-Dixon polled just 236 registered voters, who preferred Rep. Jon Porter to state Sen. Dina Titus 43 percent to 40 percent, with a margin of error of plus or minus 6 percentage points.)

Thomas Mitchell, the Review-Journal editor, said driving down the margin of error by polling more voters is difficult to do “at a decent price.”

The larger the sample, the higher the cost. Coker said tripling the sample size would multiply the cost by roughly three, though not quite because there are fixed costs. Neither the Review-Journal nor Coker would say how much the polls cost.

Mathiowetz said the paper’s ethical requirement is to disclose the margin of error, which, as Mitchell pointed out, the paper did.

Coker said he doesn’t think the results would be significantly different with a larger sample and a smaller margin of error. And he remarked, “I don’t write the headlines.”

While noting the margin-of-error disclosure, Stephen Bates, a UNLV journalism professor, called the decision to run the big headline on the Heller story “dubious” and the poll itself “meaningless.”

Some analysts defended the paper’s decision, though often tepidly.

“A poll like that is a very small poll, though not necessarily invalid,” said Keeter, of Pew. But he noted that with such a small sample, it’s harder to pull out the often more valuable data found amid larger samples. For example, this election will likely turn on what subgroups are thinking — say, suburban independent women or Hispanic professionals. With such a small sample, those data are even less reliable.

Jennifer Duffy, an analyst for the nonpartisan Cook Political Report and a close watcher of polls, said the Mason-Dixon poll was “far from an ideal sample, but not unacceptable in a House district.” Still, “the variation is wide enough to raise legitimate doubts about whether it’s right.”

More troubling, what if this snapshot of public opinion is wrong, and that in turn affects actual public opinion? Or, as Brookings Institution scholar Michael McDonald put it, “Is the tail wagging the dog?”

Early in the primary season, opinion polls can have a big effect. A poll that shows a candidate in a strong position can bring in money from fundraisers who want to be with the winner. So, for instance, polls showing Illinois Sen. Barack Obama in the hunt in the Iowa caucus race last year helped him remain viable with fundraisers and the news media.

Also, campaign committees such as the Democratic Congressional Campaign Committee make decisions about where to spend money based in part on polls, though they likely put less stock in polls with small samples.

As for voters, they too are thought to be subject to influence from polls.

Who, after all — besides Cubs fans — wants to back a loser?

Keeter said there’s no strong evidence of a “bandwagon effect” or “underdog effect” that would increase votes for the favorite or underdog as manifested in poll results.

He said polls seem to be part of the overall information environment — the mass of data voters pick through, often inscrutably — to make decisions.

McDonald, who’s also a political scientist at George Mason University, asked, “Is it possible voters could be swayed? Possible.”

But McDonald doesn’t argue that a voter looks at a poll and goes with the candidate who is ahead.

He thinks a poll might spur a reevaluation. So, if a candidate performs surprisingly well in a poll, a voter might suddenly see the candidate as a viable choice, whereas before the poll the voter never would have considered the candidate.

“It’s like, ‘Oh, I really have a choice to make here,’ ” he said.