Polling’s Prognosis: Wary Conservatives and Eager Liberals

Political pollsters know they’re having trouble reaching entire swaths of the Trump-supporting electorate, and they’re working on ways to address it.

But some of them fear that might not even be the extent of the problem, and a report released today by Data for Progress, a left-leaning think tank and polling firm, bolsters that view.

Data for Progress researchers have gotten used to being told to take a hike — and worse — by Trump supporters they try to poll, and the report detailed their difficulties in reaching that group.

But perhaps most significantly, it also found that they were having trouble reaching a representative sample of Democrats, as well as Republicans — partly because liberals with strong political views were almost too willing to be polled.

Data for Progress’s results have been on the more accurate side among its peers, though it relies entirely on so-called nonprobability methods — namely reaching people by email and text message — that haven’t gained full acceptance as an industry standard for political polls, at least not yet. (That’s partly because these methods involve using an already-limited universe — say, people who have cellphone numbers attached to their names on publicly accessible voter files, which accounts for only about three in every five voters — to draw one’s sample.)

The report by Data for Progress amounts to an earnest recognition that it has encountered issues drawing a representative sample across the board. “We want this to be out in the open so that we can all help to identify the root causes,” Johannes Fischer, a lead methodologist at Data for Progress, said in an interview via Zoom.

With Democratic respondents, unlike Trump supporters, the report found that the most common problem was over-participation: “Liberal activist”-type voters, the report said, were so eager to express their views that they could quickly fill up much of the poll’s Democratic sample, unless proper adjustments were made.

So-called response bias is a big part of the reason innovative methods like text-to-web surveys have yet to win the full faith and confidence of the polling establishment. Still, it’s safe to say the report’s findings speak to the broader issue of who is and who isn’t willing to respond to surveys, across modes. And as more and more polling is done via the web and text message, these techniques demand the same scrutiny as traditional phone surveys.

‘Activist’ overrepresentation

By using voter files attached to people’s email addresses, researchers can piece together profiles of respondents based on publicly available data, including things like whether they’ve ever volunteered for a political campaign.

Using an algorithm that takes into account a user’s entire profile, methodologists will create a model that uses the set of people who have volunteered for campaigns to identify other people with similar interests and lifestyles — the kind of “activist” voters mentioned in the report.

Going forward, Fischer said, the team is working on ways to prune its sample, investigating ways “that we can down-weight people that we’re relatively confident are activists,” so that other respondents identified as Democrats can have their perspectives represented in polls.

Education isn’t everything

After the 2016 election, when pollsters underestimated Donald Trump’s support in many key states, post-mortem reports indicated that firms had often failed to account for the differences between white voters with and without college degrees.

“Weighting” data by education — that is, making sure that a survey had the right level of representation across education levels, particularly within racial groups — was cemented as standard operating procedure.

But the Data for Progress team has found that this may not be enough. Its report showed significant, consistent differences within education groups that had an effect on whom they reached. Not all white conservative voters with college degrees will vote similarly — and not all of them will be equally likely to respond to a survey.

A particularly big factor was where people lived: The team found that a respondent’s social environment — who their neighbors are — may matter more than their educational level.

Ultimately, said Colin McAuliffe, a co-founder of Data for Progress, the team found that among white voters, when it comes to candidate choice, “the percentage of voters in their ZIP code with a college degree was more predictive than their own education level.”

In particular, Fischer said: “There are substantial differences among white conservatives that we aren’t picking up on. A college-educated white Republican in a rural, consistently Trump-supporting ZIP code is different from one in an urban or Biden-supporting ZIP code.”

“We’re hearing from more of the Biden-supporting ZIP codes, and less of the non-Biden-supporting ones,” he added. “And because of that, we can’t just weight by education and expect that to be solved.”

Finding fixes

Over the course of the 2020 campaign, Data for Progress researchers said they got profane responses from thousands of people they texted — often topped off with a “MAGA” or “Trump 2020” reference. This was all of the evidence they needed to support social scientists’ frequent observation that support for Trump correlates to suspicion of institutions, which can have a big impact on surveys.

So as they conduct polls on political races and issues this year, Data for Progress researchers have begun sending out different types of invitations via text — all leading to the same survey. “We’re experimenting with the language that we use when we text somebody,” Fischer said. “Can we use language that elicits more authoritarian responses, or keys into language used more often on the right, to engage those voters?”

And by specifically factoring in the social and political climate of their respondents, as well as the respondents’ own personal and political profiles, Fischer said that Data for Progress had cut its measured error in half.

When building a representative sample now, “we care about the neighbors,” Fischer said, “more or at least as much as we care about the respondent themselves.”