Skeptical of Recent Polls? You Should Be...

The failure to accurately measure Trump's support has dogged the polling industry for nearly a decade. But now pollsters know their shortcomings all too well and have taken many steps to address the issue. Perhaps too many?

“There are a lot of shitty polls out there” -John Anzalone, Resident Fellow, Harvard Institute of Politics

Pollsters' Trump-Voter Obsession

In the wake of the 2016 election, there was much hand-wringing about how the polls completely missed the mark on the makeup of the electorate. They underestimated the amount of non-college-educated voters who showed up to vote and this demographic broke heavily toward Trump. As a result, pollsters have put an intense focus on increasing outreach to these voters and weighting their answers more heavily to cover for non-response bias.

"In 2016, there were a lot of legitimate concerns about polling error. What we found internally, in a group of pollsters, is that we weren’t getting the right proportion of non-college-educated voters. We were getting too many service-oriented, non-college-educated voters. We weren’t getting enough people who work with their hands or work in the factory or in agriculture, drivers, and things like that. We also saw that a lot of our small-town rural interviews were in the county seat, not in the rural areas. And so, we changed up a lot in terms of how we’re getting our interviews and quotas with non-college-educated voters."
-Christina Pazzanese, The Harvard Gazette

The 2020 election then pushed pollsters to shift their focus beyond white non-college-educated voters after Trump saw an increase in support from Black (+2%) and Hispanic voters (+10%) during his failed run for re-election. This leads us to the polls that shape our collective view on the current election.

Pollsters have made a clear effort to dive deeper into areas that supported Trump and were previously underrepresented in past elections. They've also gotten creative at addressing their Republican non-response issues by doing things like draping their invitations in more patriotic imagery and explicitly calling for views that aren't commonly represented in polling. After many years of employing this work, their difficulties with polling Trump voters are incredibly well known and documented. Even if, despite this meticulous attention, they're still failing to reach these voters en masse, there are numerous statistical methods at their disposal to properly estimate this support. One controversial method that pollsters are increasingly leaning on is called "weighting on recall vote," primarily because it insulates them from claims that they're underestimating Trump's base of support.

"Here’s how it works. First, the pollster asks respondents whether they voted for Joe Biden or Mr. Trump in the last election. Then they use a statistical technique called weighting, in which pollsters give more or less “weight” to respondents from different demographic groups, such that each group represents its actual share of the population. In this case, the pollster weights the number of Biden ’20 or Trump ’20 voters to match the outcome of the last election. This approach had long been considered a mistake... pollsters have avoided it over the years. But they increasingly do it today, partly as a way to try to make sure they have enough Trump supporters after high-profile polling misfires in 2016 and 2020." -Nate Cohn, New York Times

So if it took pollsters nearly eight years to accurately poll Trump voters, how long would it take them to adjust their methods to account for a last-minute candidate swap on the Democratic side?

What About Harris Voters?

"It’s not going to be very hard to swap in another name." -Nick Beauchamp, associate professor of political science at Northeastern University

When met with a historic political moment where the sitting president and presumptive Democratic presidential nominee dropped out of the race roughly 100 days before Election Day, pollsters didn't approach this new challenge with the same meticulousness as we've seen put toward polling Trump voters. In interviews about how pollsters are attempting to get it right this year, most of the attention is still focused on their efforts to accurately poll Trump voters. Meanwhile when asked about the challenge of tracking the nascent Harris campaign, some believed that the polling transition from Biden to Harris would be "relatively seamless." There are many issues with this laissez-faire approach toward Kamala Harris' entry into the race.

First and foremost, the makeup of the electorate has undoubtedly changed with Kamala Harris leading the Democratic party. Harris' surge in polling compared to Biden quickly cooled fears amongst pollsters. They pointed to this dramatic shift as evidence that they're correctly capturing the electorate's reaction to the candidate switch from Biden to Harris. But I think that's a naive reading of their own polls as it doesn't account for how Biden's historic unpopularity (and his relentless support of Israel) pushed many left-leaning Democrats out of the electorate and dissuaded them from voting altogether. Harris' late summer polling surge wasn't due to polls correcting their assumptions about how many new voters Harris might bring to the polls but simply represented a shift in likely Democratic voters' enthusiasm with their new candidate.

Another way of putting it: polls are accurately reflecting voter preferences of the expected electorate in a Biden-Trump matchup, but they aren't reflecting the preferences of voters who were previously sitting out this election due to their displeasure with their choices. Most notably, enthusiasm amongst Black women has surged in the last few months and I don't believe polls have adjusted to account for this expected increase in turnout. Second, the polls may be failing to catch the increasing number of Republicans that are explicitly backing Harris. If polls are expecting Republicans to support Trump at the same rate as before and weighting their responses such, that's effectively erasing "Harris Republicans" from polling results.

And while Harris has echoed President Biden's approach toward foreign policy regarding Israel's right to defend itself, the removal of a self-identified Zionist from the ticket will do a lot to pull left-leaning voters back into the fold. While leaders of the uncommitted movement are still withholding their endorsement, Harris' willingness to engage with this group may do just enough to convince previously unlikely voters to participate.

The Numbers Don't Add Up

The news often likes to report on the margins of polls: focusing on who's up and who's down rather than the percentages of support for each candidate. But if you look at the raw polling numbers, they literally do not add up to 100% in many states. This is primarily due to the large number of so-called 'undecided' voters: specifically those who are withholding their vote to utilize their leverage over either party. In the past, many of these undecided holdouts were assumed to break for Trump, helping create the media phenomenon of the "shy Trump voter". But this election feels different. Trump voters have only grown brasher and bolder over the years with their willingness to spread easily debunked conspiracy theories with reckless abandon. There's also been the emergence of movements like the Courage Tour: a series of multi-day events in swing states that explicitly call upon Christians to speak up on political matters.

"Let me put it to you real clearly. You are the voice to the nation's conscience and if you don't lift up your voice, they have no truth... Who's ultimately to blame? It is going to be us because we didn't lift up our voice and speak." -Lance Wallnau, leader of the Courage Tour

Meanwhile, left-leaning Democrats have called on those who support Palestinians to withhold their support for the party, despite knowing that Donald Trump would be exponentially worse on the issue than any Democratic president. There's also the small group of Haley voters that refuse to support Trump and seem increasingly willing to break for Harris. And again: pollsters don’t seem to know who low-propensity Harris voters are, how many there may be and where they live. 

All of this together leads me to think that we've gone from underestimating Trump's support to overestimating the size of his base. The million dollar question is where are polls overestimating this support and will it ultimately affect the outcome of the electoral college? Because who cares if Kamala is polling at +1.8% nationally and actually wins 3% of the national vote... What matters is if the Harris voters that polls are missing live in states like Pennsylvania, North Carolina and Georgia as they'll likely be the ones who decide who runs our country for the next four years.

Leftover Links

How Kamala Harris Is Changing the 2024 Electorate

Vice President Kamala Harris has changed the face of the upcoming presidential election, but she also appears to be changing the face of this year’s electorate. A pattern has existed

Ms. Magazine •Jodi Enda, The Fuller Project

How different turnout models change the NBC News poll results

A few percentage points of difference in turnout by gender or education can produce very different results, both in election polls and in elections themselves.

NBC News •Ben Kamisar