Can we trust polling data? Experts break down how this data works and why it matters

With less than a month to go before the 2024 U.S. presidential election, some polls show the race is essentially neck-and-neck for both candidates. Although Vice President Kamala Harris is consistently polling ahead of former President Donald Trump in the national popular vote, those same polls show the two candidates effectively tied in the seven swing states which will decide the election in the Electoral College: Pennsylvania, Michigan, Wisconsin, North Carolina, Georgia, Arizona and Nevada.

It is at this point that public trust in polling itself begins to break down. If an observer assumes that current polling as a science is basically reliable, one can look at rigorous probabilistic forecasts at sites like Fivethirtyeight.com or The New York Times, both of which currently project Harris eking out an electoral college victory of 276 to 262. Yet in the last two presidential elections, polls have made small but crucial errors that undercut their accuracy. Given that on both occasions the errors understated Trump's support, the Republican nominee and his backers often accuse the pollsters of having a deliberate bias.

Yet for most polling agencies, these errors were not only accidental, but a nightmare come true. In an industry with a business model that literally depends on being able to make accurate predictions, pollsters want nothing less than to be wrong while the whole world is watching them, regardless of which candidate benefits.

The problem facing pollsters is that the science behind their craft relies on an ever-shifting set of variables. While in 2016 pollsters believe they inadvertently under-sampled low-propensity voters — who turned out in unusually large numbers for Trump — in 2020, an even more unpredictable confounder emerged: the COVID-19 pandemic. The public health crisis caused "the largest forced behavior change in modern American history, affected everything, including polling," according to Chris Jackson, senior vice president at the polling firm Ipsos. Because most of society was so disrupted by the virus, especially as social distancing was practiced, it became more challenging to contact and interview people in ways that were unprecedented — and therefore not fully understood.

"I think that goes from everything to who was home to answer phones, to how people were early voting, to the sort of political lean of early voting, especially with Trump discouraging Republicans from using mail-in vote," Jackson told Salon. "All of those things were complicating factors in doing good quality public opinion polls."

In terms of Trump's meddling, it's important to recall that he also undermined the post office to kneecap mail-in voters out of fear it would favor his Democratic opponent, then-former Vice President Joe Biden. Trump also generally cast doubt over the legitimacy of any election result that did not declare him to be the winner, while actively encouraging voter suppression from Republicans and further sowing chaos into voter behavior patterns.

"I think sort of extra-legal activities are definitely something that polls cannot account for," Jackson said. "You actually see this more in research in other countries, particularly in semi-legal democracies where the government does actively interfere with the vote in a somewhat significant way."

While Jackson doubts that this type of interference could account for the entire 8 million vote gap between the popular vote margin of victory polls anticipated for Biden and the actual results, he acknowledges that it could have been an important factor. Another variable is simply that the type of person who votes for Trump was, temperamentally and economically, less inclined in 2020 to answer polls.

Want more health and science stories in your inbox? Subscribe to Salon's weekly newsletter Lab Notes.

"It created biases in sample composition (survey responses) because certain types of people were more likely than others to be sequestered in their homes, sitting on Zoom calls or whatever, and potentially somewhat bored,"  David Barker, division director for Social, Behavioral and Economic Sciences at the National Science Foundation, told Salon. "Those people were disproportionately 'knowledge economy' professionals — i.e. Democrats."

Overall, Barker believes a number of possible factors could cause the 2024 polls to understate Democratic support, foremost among them the desire to overcorrect from understating Republican support in the last two cycles. Additionally, the pollsters could be underestimating Latino support for Harris "because Latinos who respond to survey invitations are more likely than other Latinos to be part of families who have been U.S. citizens for multiple generations, and (relatedly) they are more likely to speak English as a first language or to be fully bilingual. Spanish-only Latinos are less likely to be included in survey samples."

Finally, Barker expressed concern that the polls may underestimate turnout by Democratic constituencies because "the Harris campaign has more resources and a better ground operation than the Trump campaign does. That may pay dividends in terms of disproportionate turnout."

This is not to say that there are not also factors which could understate the level of support for Trump for a third consecutive cycle. Just like the last two elections, Barker said the polls could fail to capture enough white working class voters and/or underestimate those voters' propensity to turnout. In that scenario, the 2024 election will likely go the way of its two predecessors, with Trump flipping the so-called "blue wall" swing states of Pennsylvania, Michigan and Wisconsin as well as previously pro-Biden states like Nevada, Arizona and Georgia to win both a popular majority and by 312 to 226 in the Electoral College.

Alternatively, if the polls are unintentionally understating Harris' support, Harris could get as much as 52 or 53 percent of the popular vote, retain all of the swing states that Biden won and add North Carolina (which Trump won in 2016 and 2020) to the Democratic column for a 319 to 219 Electoral College victory.

It would not require major polling errors to create either of these dramatically different outcomes, just a few quirks in how the pollsters misjudged this year's electorate. As Jackson was quick to point out, pollsters' jobs are especially complicated because they are not only trying to predict the behavior of the millions of Americans who vote — they also must predict the behavior of the millions of Americans who choose not to vote.

"It's important to remember that when we talk about an election's results, people will oftentimes say such as in 2020 that Biden got 51% of Americans voting for him, and Trump got 47% of Americans voting for him, but that's not accurate," Jackson said. "Actually Biden got 32% of Americans voting for him, and Trump got 29% of Americans voting for him. The rest didn't vote." This means that when analysts think about an election, they have to account for the millions of Americans who are eligible to vote but for various reasons choose not to show up.

"That is a big, complicated thing because you're trying to measure people's behavior and sometimes people don't even know their own behavior," Jackson said. "They're going to say, 'Yeah, I'll totally go vote,' but then Election Day comes along and their kid's sick or something else and they have to deal with it, so they don't vote. It's a very complicated puzzle and there's always a lot more support out there for a candidate than actually is registered on Election Day."

In spite of all this uncertainty, America's elections are still not able to be "rigged," at least not in the manner claimed by Trump. Although chicanery happens on a local level, Jackson observed that the unwieldy and labyrinthine nature of America's electoral system serves as a built-in protection against outright election theft.

"We've done a lot of work talking with the people administering the election at the local level, and honestly, because our system is so messy and complicated, it's actually hard to tamper with at any sort of scale because it is essentially administered by 5,000 individual county clerks across the country, each one of which is basically sort of running their own system," Jackson explained. "If there is any sort of Republican futzing with things, it's at the front end, messing with registration, making it where people are not eligible to vote — it's not messing with the actual counting or tabulation of the votes."