The idea behind a poll average is simple: Just average the most recent polls together.
The reality is more complicated. The New York Times’s polling average is a weighted average, meaning it gives more weight to some polls than others. It also makes a series of statistical adjustments to account for the different biases among different kinds of surveys.
Despite all this fine-tuning, the poll average still isn’t a forecast. Polls are just measures of where public opinion stands today, and an average of different measurements is still ultimately a measurement — not a prediction.
Here’s how it works.
Why a weighted and adjusted poll average?
Poll averages may sound simple, but even the simple ones face important challenges. Which polls should be included? How much weight should go to a poll that’s three days old compared with one that’s three weeks old? What about pollsters whose polls lean reliably toward one party or the other?
One option in facing these challenges is to take a curated, simple average of the most recent polls. The other option: Weight and adjust polls to account for varying quality and biases of different polls and pollsters.
The simple average has the advantage of clarity. It offers the most straightforward answer to the question, “What do the polls say?” But the simple average has a major disadvantage: It risks being flooded by low-quality or biased polls. As a result, the simple average may not offer the best answer to the question, “Who’s ahead in the presidential race?”
The Times’s polling average generally takes the more complicated approach. It weights polls based on a variety of factors. It also adjusts them for potential biases. There’s one big limit, though. It’s based only on the polls themselves. It doesn’t account, for instance, for the demographics of a state or whether a state voted for President Biden or former President Donald J. Trump in the last election.
Or put differently, the average represents an estimate of the state of the race, based on the polls — and polls alone.
Which polls do we use?
The Times uses every poll collected by FiveThirtyEight, a political data site affiliated with ABC News. Times staff members will also seek out the latest surveys from a variety of pollsters.
FiveThirtyEight includes every poll that attempts to reach a representative sample of the electorate, though it has dropped a few polling firms for failing to meet a minimum standard of transparency.
When given different versions of results from pollsters, we prefer the results among likely voters over those from registered voters or all adults. If a pollster releases multiple results of the same population and same candidates, and does not clearly emphasize one over the others, we average them together.
What about Kennedy?
This cycle, one big question is how to account for Robert F. Kennedy Jr., the independent candidate who commands nearly 10 percent of the vote in recent national polls. He hasn’t yet gotten on the ballot in most states, and in the meantime every pollster is making different choices about how to handle him. Some pollsters are naming him on the first question asking about the presidential race. Others are naming him on a second question. Still more aren’t asking about him at all.
A poll average faces two big questions when figuring out how to deal with Mr. Kennedy. The obvious one is whether to emphasize the head-to-head poll result or the three-or-more-way race including Mr. Kennedy and perhaps the other minor-party candidates. At this stage of the race, the average emphasizes the head-to-head result. That’s because Mr. Kennedy hasn’t yet obtained access to the ballot in most states, and there is no guarantee that he will.
A less obvious question is whether to adjust for which polls have or have not included Mr. Kennedy. Suppose, for instance, that a relatively Democratic-leaning pollster asked only about the three-way race, while a Republican-leaning pollster asked only about the two-way race. If so, the head-to-head polling average might be more favorable to Mr. Trump than the three-way average, simply because of which pollsters chose to conduct the survey. The FiveThirtyEight polling average makes such an adjustment, and attempts to convert all two-way polls to the multicandidate race. RealClearPolitics does not.
While our average usually takes the “complex” approach, here we take the simple route. We calculate two entirely separate averages for the two-way and multicandidate races, without any attempt to adjust for which pollsters are taking which approach. As the race goes on, pollsters will probably converge on the appropriate solution.
Weighting polls
The Times polling average gives more weight to some polls than others, depending on the characteristics of the poll and the pollster.
Four main factors determine the weight that a poll will receive in the average:
-
Recency. More recent polls get more weight than older ones. The average becomes more aggressive about weighting the most recent data as the election gets closer, or in the wake of conventions and debates. There’s less emphasis on recency if only a few high-quality polls have recently surveyed a race.
-
Sample size. The larger the sample size, the better — up to around 2,300 responses. Beyond that, polls receive no additional weight for additional sample size.
-
The number of polls recently conducted by a pollster in a race. If a pollster conducts many surveys in the same race in a short period, each poll result will receive less weight. A single pollster’s set of polls can never be worth more than about four recent surveys in a race.
-
The pollster. The Times average gives more weight to surveys by some pollsters, and less weight to polls by other pollsters, as described below.
Weighting pollsters
The amount of weight assigned to a particular pollster is based on five rough proxies for quality.
-
Professionalism, defined by whether a pollster contributes microdata to the Roper Center or whether it’s a member of the American Association for Public Opinion Research.
-
Methodology, specifically whether a poll uses a nonprobability or opt-in convenience sample. A poll receives less weight if it doesn’t at least partly use a more systematic way to sample the population.
-
Partisanship. A poll receives less weight if it’s sponsored by partisans or a campaign.
-
Experience. A pollster in its first year receives significantly less weight than a more established pollster with the same attributes.
It’s important to emphasize that these are merely rough proxies for the actual quality of a pollster. There are rigorous polling firms with poor track records worth taking very seriously. There are “professional” pollsters with good track records who still don’t follow best practices, and might not be set for success in the future. But historically, these traits help predict whether a poll will be accurate and unbiased going forward.
‘Select pollsters’
To help readers understand at a glance which pollsters tend to receive more weight in the average, certain pollsters will be labeled “select pollsters.” They are denoted with a diamond symbol.
The designation is based on whether a pollster meets two of these three criteria:
-
Has a track record of accuracy in recent elections.
-
Is a member of a professional polling organization.
-
Is conducting polls only with probability-based samples.
We consider pollsters that meet at least two of the three criteria to be select pollsters, so long as they are conducting polls for nonpartisan sponsors.
Strictly speaking, this is not exactly the same as which polls or pollsters get the most weight in the average, but it’s pretty close.
Adjusting polls
In the end, the poll average strives to represent what polls say about the state of the race among likely voters heading into the election. But most polls won’t be conducted immediately before the election; many aren’t limited to likely voters; and only a handful of pollsters typically poll a particular state.
The poll average attempts to account for these challenges.
-
Likely voters. After Labor Day, polls that report results from registered voters or all adults, as opposed to likely voters, are adjusted to account for the differences between likely voters and registered voters. The adjustment is based on the difference between registered and likely voters in polls that released results for both populations.
Today, polls of registered voters would be adjusted slightly toward Mr. Biden, as polls tend to show him faring better among those who are likeliest to vote. In prior elections, the same adjustment shifted registered voter polls toward Mr. Trump.
-
Shifts in the race. Older surveys are adjusted to account for shifts in the race since the poll was conducted. The adjustment is based on how polls by the same pollster have shifted over time, and it becomes much more aggressive near Election Day.
-
Partisanship. Polls sponsored by partisans or campaigns are presumed to be biased by about six percentage points toward their party, based on how biased the polls by firms like these have been in the past.
-
Pollster tendencies, sometimes known as house effects. The average accounts for whether pollsters have a track record of leaning Democratic or Republican compared with other polls taken around the same time. If, for instance, a new poll comes out from a pollster that has leaned Democratic compared with other polls by three points, it might not shift our average toward the Democrats unless it’s at least three points better for Democrats than prior surveys.
