In presidential elections, fifty-fifty the smallest changes in horse-race poll results seem to get imbued with deep significant. Simply they are frequently overstated. Pollsters disclose a margin of error so that consumers can have an understanding of how much precision they can reasonably expect. Merely cool-headed reporting on polls is harder than it looks, because some of the amend-known statistical rules of thumb that a smart consumer might think apply are more than nuanced than they seem. In other words, as is so ofttimes true in life, information technology'southward complicated.

Here are some tips on how to recollect about a poll's margin of error and what it ways for the different kinds of things we often try to larn from survey data.

1 What is the margin of mistake anyhow?

Because surveys only talk to a sample of the population, we know that the consequence probably won't exactly lucifer the "true" result that we would get if we interviewed everyone in the population. The margin of sampling error describes how shut nosotros can reasonably expect a survey outcome to fall relative to the true population value. A margin of error of plus or minus 3 per centum points at the 95% confidence level ways that if we fielded the same survey 100 times, we would expect the result to be within iii percent points of the true population value 95 of those times.

The margin of error that pollsters customarily written report describes the amount of variability we can await around an individual candidate's level of support. For instance, in the accompanying graphic, a hypothetical Poll A shows the Republican candidate with 48% back up. A plus or minus three percentage signal margin of fault would mean that 48% Republican support is inside the range of what nosotros would wait if the true level of support in the total population lies somewhere iii points in either direction – i.east., between 45% and 51%.

2 How do I know if a candidate's lead is 'exterior the margin of error'?

News reports well-nigh polling will often say that a candidate's lead is "outside the margin of error" to signal that a candidate'southward pb is greater than what we would expect from sampling error, or that a race is "a statistical tie" if it's besides close to phone call. It is non enough for one candidate to be ahead by more than the margin of error that is reported for private candidates (i.e., alee by more than than three points, in our case). To determine whether or not the race is besides close to call, we demand to summate a new margin of error for the difference betwixt the two candidates' levels of support. The size of this margin is generally about twice that of the margin for an individual candidate. The larger margin of mistake is due to the fact that if the Republican share is besides high by risk, it follows that the Democratic share is likely besides low, and vice versa.

For Poll A, the 3-pct-point margin of error for each candidate individually becomes approximately a vi-point margin of error for the deviation between the 2. This means that although we have observed a v-point lead for the Republican, we could reasonably look their true position relative to the Democrat to lie somewhere between –one and +eleven percent points. The Republican would demand to be ahead by half-dozen per centum points or more for us to be confident that the lead is not merely the result of sampling error.

In Poll B, which also has a three-bespeak margin of mistake for each private candidate and a half-dozen-indicate margin for the departure, the Republican lead of 8 percentage points is large plenty that information technology is unlikely to be due to sampling error solitary.

three How do I know if at that place has been a change in the race?

With new polling numbers coming out daily, information technology is common to come across media reports that describe a candidate's pb as growing or shrinking from poll to poll. Simply how tin can we distinguish existent modify from statistical racket? As with the difference between two candidates, the margin of mistake for the difference between two polls may be larger than y'all call up.

In the example in our graphic, the Republican candidate moves from a lead of 5 pct points in Poll A to a lead of 8 points in Poll B, for a net change of +3 percent points. Merely taking into account sampling variability, the margin of error for that iii-betoken shift is plus or minus 8 pct points. In other words, the shift that we have observed is statistically consistent with anything from a 5-point decline to an 11-point increment in the Republican's position relative to the Democrat. This is not to say such large shifts are likely to have really occurred (or that no change has occurred), but rather that we cannot reliably distinguish real change from dissonance based on just these two surveys. The level of observed alter from i poll to the adjacent would need to be quite large in order for us to say with conviction that a change in the horse-race margin is due to more than sampling variability.

Even when we do come across large swings in support from one poll to the side by side, one should practise caution in accepting them at face value. From Jan. 1, 2012, through the ballot in Nov, Huffpost Pollster listed 590 national polls on the presidential contest between Barack Obama and Mitt Romney. Using the traditional 95% threshold, we would expect 5% (nigh xxx) of those polls to produce estimates that differ from the true population value by more than the margin of error. Some of these might be quite far from the truth.

Still often these outlier polls stop up receiving a great deal of attention because they imply a large change in the state of the race and tell a dramatic story. When confronted with a particularly surprising or dramatic issue, it's always all-time to be patient and see if it is replicated in subsequent surveys. A result that is inconsistent with other polling is not necessarily wrong, simply real changes in the state of a campaign should show up in other surveys also.

The amount of precision that can be expected for comparisons between two polls will depend on the details of the specific polls being compared. In exercise, virtually any two polls on their own will evidence insufficient for reliably measuring a change in the horse race. Merely a series of polls showing a gradual increase in a candidate's lead tin frequently exist taken as evidence for a existent trend, even if the difference between individual surveys is within the margin of fault. As a general rule, looking at trends and patterns that sally from a number of unlike polls can provide more than confidence than looking at only ane or ii.

4 How does the margin of error apply to subgroups?

Generally, the reported margin of error for a poll applies to estimates that use the whole sample (e.m., all adults, all registered voters or all likely voters who were surveyed). Only polls ofttimes report on subgroups, such as young people, white men or Hispanics. Because survey estimates on subgroups of the population have fewer cases, their margins of error are larger – in some cases much larger.

A simple random sample of ane,067 cases has a margin of error of plus or minus three per centum points for estimates of overall support for individual candidates. For a subgroup such as Hispanics, who make up about 15% of the U.S. developed population, the sample size would be virtually 160 cases if represented proportionately. This would mean a margin of error of plus or minus 8 percent points for individual candidates and a margin of mistake of plus or minus 16 percentage points for the difference between two candidates. In practice, some demographic subgroups such as minorities and young people are less probable to respond to surveys and need to be "weighted up," meaning that estimates for these groups often rely on fifty-fifty smaller sample sizes. Some polling organizations, including Pew Research Centre, written report margins of mistake for subgroups or brand them bachelor upon request.

5 What determines the amount of mistake in survey estimates?

Many poll watchers know that the margin of error for a survey is driven primarily past the sample size. Only at that place are other factors that as well affect the variability of estimates. For public opinion polls, a particularly important contributor is weighting. Without aligning, polls tend to overrepresent people who are easier to attain and underrepresent those types of people who are harder to interview. In guild to brand their results more representative pollsters weight their information so that it matches the population – unremarkably based on a number of demographic measures. Weighting is a crucial step for avoiding biased results, but it also has the effect of making the margin of mistake larger. Statisticians call this increment in variability the design result.

It is important that pollsters take the design result into account when they report the margin of mistake for a survey. If they exercise not, they are challenge more than precision than their survey actually warrants. Members of the American Association for Public Opinion Research'south Transparency Initiative (including Pew Research Centre) are required to disclose how their weighting was performed and whether or non the reported margin of error accounts for the design effect.

It is besides important to bear in listen that the sampling variability described by the margin of error is only one of many possible sources of error that tin can bear on survey estimates. Dissimilar survey firms apply different procedures or question wording that can touch on the results. Sure kinds of respondents may be less probable to be sampled or reply to some surveys (for example, people without internet access cannot have online surveys). Respondents might not be candid about controversial opinions when talking to an interviewer on the telephone, or might answer in ways that present themselves in a favorable low-cal (such as claiming to exist registered to vote when they are not).

For election surveys in particular, estimates that look at "likely voters" rely on models and predictions nigh who will turn out to vote that may also introduce error. Dissimilar sampling error, which can be calculated, these other sorts of fault are much more hard to quantify and are rarely reported. Just they are present nonetheless, and polling consumers should proceed them in mind when interpreting survey results.

Andrew Mercer is a senior research methodologist at Pew Research Center.