Can we trust political polls? In-depth on what went wrong in 2016

Why "weighting" is critical to a poll's accuracy
Posted at 3:31 PM, Oct 08, 2020
and last updated 2020-10-08 18:31:06-04

The latest ABC News national polling average shows former Vice President Joe Biden leading President Donald Trump by 8 points.

But a lot of people are wondering, can we trust the polls after what happened in 2016?

The last time Donald Trump was on the ballot in 2016, the polls had him trailing former Secretary of State Hillary Clinton by an average of 3.2 percentage points, and we know what happened.

However, pollsters weren’t off by as much as you might think.

“At the national level, the polling was, remarkably, given all things, precise,” said Jay Leve, CEO of the polling firm SurveyUSA.

Trump lost the popular vote by 2.1 points instead of 3.2, the most accurate these national polls had been in 80 years, according to an analysis by the American Association for Public Opinion Research.

Where the polls did miss badly was at the state level, particularly in Pennsylvania, Michigan and Wisconsin, three states that were critical in the Electoral College.

Leve said there were several reasons for the polling problems at the state level.

“Polling is a very expensive undertaking and so it is not possible for the handful of media organizations with pockets deep enough to afford a public opinion poll to be able to poll in every critical battleground state,” he said.

Another reason? “Some of it has to do with what’s called ‘weighting,’” he added.

To understand weighting, you have to know the two R’s of a good poll: it needs to be representative and random.

Random samples are critical to the accuracy of polling, and you can look to your kitchen for an example why. Picture adding salt to a soup. If you mix it right, you can check the taste with any one spoonful -- you don’t have to eat the entire pot. That’s because each spoonful is a truly random sample.

If you don’t mix the salt in, you could easily wind up sampling a part of the soup without any salt.

When you’re trying to sample the American public with a political poll, either over the phone or most of the time now online, it’s more challenging to get a perfectly random spoonful.

“The challenge is to find the individuals in the right numbers and secure their cooperation. Those two things don’t automatically work in sync,” Leve said. “People don’t want to be disturbed. They want privacy and a pollster by definition is an interruption.”

It turns out, certain people tend to resist taking polls, while others are more willing. Research shows people with college degrees are more likely to respond to surveys than high school grads.

That means surveys run the risk of not being representative of the voter population at large, and Leve said that kind of imbalance played a big role in 2016.

To make a sample representative, pollsters gather up as many responses as they can, then adjust them with a process called weighting -- basically boosting or shrinking responses from people with certain demographics to match census data and the expected turnout.

“The weighting criteria that was in issue in 2016 was whether you had enough non-college educated white voters in your sample,” Leve said. “If you did, you got the Trump forecast correct.”

State polls that didn’t weight by education level missed badly, because to an extent far greater than in previous elections, voters with a college education broke for Clinton while voters with a high school education backed Trump.

There’s some evidence that pollsters have learned from their 2016 mistakes. Polling in the 2018 midterms was very accurate -- a full point better than the average over the last 20 years.

So can we trust the polls this time around?

Leve says yes, as long as you remember that polls are just a snapshot in time and Donald Trump is difficult to predict.

“Don’t be surprised if something happens in the final four, five, six days of the election, right before November 3rd, that’s so unforeseeable that neither you nor I nor anyone watching us could have imagined. And if so, that’s going to throw all the polls off,” he said.