Throughout the election cycle, polls are released practically every
day showing who is more likely to win the presidency.
A poll in the field starting on Friday night...
Polls showing the president's standing among older voters...
And then there's consideration of whether the polls are even right...
Despite the amount of polls we see out there, according to a 2018 The
Hill/HarrisX poll, a majority of Americans don't believe polls are
accurate, even though polling has become more and more accurate over
time. Polls are a story, but you can't just
take the headline.
You have to read the entire story.
Most of the polls that grab headlines are political polls, but most
election polls are not meant to make money.
So the only people cashing in on polls are the in-house campaign
strategists. Instead, they're considered free advertising to attract
new clients for private firms and prospective students to
universities. People see that you're doing these polls and then you
can get clients to do market research, image evaluations that are a
significantly greater proportion of their work.
Polling as part of a much larger industry of market research, worth
over $19 billion in the U.S.
alone. So how reliable are political polls and how is the industry
changing?
The 2016 election shook people's confidence in polls, as most said
that Hillary Clinton would win the presidency.
Polls have gotten more accurate over time as technologies improve and
the polls were right in terms of the popular vote in 2016, when they
were off by about two points, but...
Polls don't poll the Electoral College, we poll populations.
What happened is the polls were basically accurate at the national
level, but there were some state polls that consistently showed
Hillary Clinton with a lead that she didn't end up having in the end.
Polls in states like Michigan also called the Democratic primary for
Clinton, but the state went to Senator Bernie Sanders by 1.5%.
In post-mortems after the election, there were theories that polls
were biased. White voters without a college degree voted
significantly more Republican than white voters with a college
degree. You'd never seen that phenomenon before like that.
It also speaks to other biases that polling can have.
Sample sizes of racial and ethnic minorities in these polls are not
large and the cross sections are small.
What tends to happen is that the individuals who are in communities
of color that respond to surveys are individuals that are more high
education leaning, right?
And so if that's all we're basing the sample off of, then we already
expect these individuals to turn out.
To make a sample more representative of the population as a whole...
Weighting the survey data on certain characteristics like race,
ethnicity, gender, age and education to help make sure that we're
benchmarking our sample against what we know to be U.S.
Census data. So what should you look for when looking at polls?
Polling experts agree that transparency about the methodology is key
when looking at polls. You have to ask those questions about who the
sampled, about the question wording, the question order, if that's
available. You have to know when the survey was conducted.
There are so many questions that don't get answered immediately with
just a one paragraph or scan of the survey.
And the click here for more details.
Clicking it. Most of these surveys will actually, in their like top
line report, which is what the link usually goes to, will break down
some of their responses to questions by race, by ethnicity, by
gender, by age, by partisanship, so that you can understand, you
know, where individuals that might have similar group identities as
you, you know, where they fall on some of these questions that they
are reporting. The
polling and market research industry helps brands and politicians
gain insight into what constituents and target demographics are
thinking at a particular moment in time.
Public affairs polling is usually an arm of a news media
organization. The Pew Foundation or Gallup, independent,
nonpartisan groups that are doing polling, that's a very small
universe.
According to the U.S.
Census, in 1998, market research and public polling firms made over
$9.7 billion. 20 years later, in 2018, the industry grew to over $19
billion, an annualized growth of about 3.5%.
When adjusted for inflation, the real growth was about 1.24%.
Worldwide, the industry made about $49 billion in revenue, according
to ESOMAR and BDO.
Candidates spent big on polling firms in 2016.
According to the FEC, Donald Trump spent almost $4 million in polling.
Hillary Clinton spent more than $10 million.
The biggest players in the industry globally are market research
companies that have internal public opinion departments within their
ranks. The largest of these companies by far is Nielsen Holdings
P.L.C., most famously known for their TV ratings systems.
The company's revenue was $6.5 billion in 2019.
Closing out the top 3 companies in the US by worldwide revenue are IQVIA, with about $4.4 billion in
revenue in 2019 and Gartner Research with an estimated revenue of
$3.3 billion in 2019.
Many of the more well-known polling houses are within universities.
The main difference between university polls and private firms is
where they get their funding and the required review process.
University pollsters are usually funded through grants and donations,
while private firms are for profit and usually have candidates or
parties as clients.
At the university, you have to clear an institutional review board.
You have to show them the survey and this review board says, "Well,
this questions might be traumatic for someone to answer," and so
they work with the scholar who's writing the survey and they have to
change it. That doesn't mean that professional pollsters don't have
high standards. You can be guaranteed that if they're doing a
political poll, they're going to have a pair of conservative oriented
eyes on it and a pair of liberal oriented on it, because you need
people playing devil's advocate.
You need people trying to dissect what the question might possibly
generate. There are so many pollsters out there doing political polls
that it may be hard to figure out which ones are trustworthy.
But there are tools to help with that.
The website 538 has a pollster rating system that calculates the
polling error, statistical bias and how many races each pollster
calls correctly to give each of them a letter grade.
The six pollsters with an A+ rating are Marist College, Monmouth
University, ABC News/Washington Post, Siena College/New York
Times/Upshot, Selzer and Company and Muhlenberg College.
Survey Monkey and Swayable have D- ratings, the lowest grade before
538 bans pollsters from their aggregations.
It would be almost impossible to poll every adult in the United
States, so pollsters sample a certain amount of people that represent
the population being studied.
You may think that the more people who take a survey or answer a poll
will make it more accurate and representative, but that's not
necessarily true. It's not a sample size that's representative, it's
the sampling that makes it representative.
Sampling can be divided into two categories: probability and
nonprobability sampling.
Probability samples are based on everyone having a known chance of
being included in the sample.
Thinking about a pot of chili, right?
We don't have to taste the entire pot to know the chili is good and
evenly balanced.
We only need a few small tastes or samples and so the same is true of
public opinion.
If it's a scientific sample and a good sample, you will be able
to generalize back to the population from which the respondents were
drawn with that margin of error around it.
The margin of error is how wrong a poll can be, but it's more than
that. If we're going to take a poll of 1000 registered voters in
Pennsylvania, the survey results would say something along the lines
of: "the results of this survey are accurate at the 95% confidence
level, plus or minus 3 points." And the plus or minus 3 points simply
tells us that if we were to ask the same questions on another sample
of a thousand voters, that we can be confident that the results will
be within 3 percentage points of the true population
value. While probability samples are traditional, they are becoming
more and more expensive to conduct, so there is a trend away from
them to nonprobability sampling.
We're allowing people generally on the Internet to tell us they want
to participate in polls.
Now, if you can ever get to the point where everybody wants to be
part of polls, it wouldn't matter.
It's going to be representative.
Used to be there wasn't a very good way to really do nonprobability
sampling. But now with online methods where you can basically recruit
millions of people, we have much more sophisticated ways to do
nonprobability sampling than we did just 10 years ago.
As more people let go off their landlines for cell phones, response
rates have dropped from around 36% in 1997 to 6% in 2018.
The problem that we're getting is that more and more and more people
just simply don't want to talk to somebody that they don't know or
want to pick up the phone to even see who's on the other end and that
simply has driven up the cost.
That has been the biggest challenge.
In election polling you have three kinds of polls: benchmark, heat
trials and exit polls.
A benchmark survey is one in which a candidate might decide that he or
she is going to run for office, so they ask questions such as: What
do you think is the most important problem facing the nation today?
Or this city or this state?
Have you ever heard of...
candidate's name?
And they use this as a baseline.
Trial heat polls are the ones we see throughout the election cycle
asking about specific races or issues and most reputable pollsters
take great care to make sure that they are not priming a respondent's
answer. Instead of asking: "For whom would you vote?
Donald Trump or Joe Biden?" You asked: "For whom would
you vote? The incumbent President Donald Trump or
challenger Joe Biden?" Or you could ask: "For whom would you vote?
Republican Donald Trump or Democrat Joe Biden?"
So notice how the words that are included now, trigger people to
think a certain way. They're primed. Exit polling is done outside of
election centers. Polling houses recruit hundreds or even thousands
of people to conduct interviews as people leave their polling place.
We've done a lot of work before you see it.
Like, the short part is putting it in the field and analyzing the
data. The long part is, you know, putting it together.
After the poll is conducted, the data is analyzed and then weighted to
make up for any lack of representation.
First and foremost, you want to make sure the overall results are
representative of everybody.
So you bring up groups that are too low in your sample and then you
want to try to make sure that if you're looking at that group in
particular, that it's going to be representative of that group.
And then ideally, if you can, you might be able to break out that
group by age and gender within that group.
So it depends how far you want to go with your analysis, how much you
have to do.
The polling industry is going through changes as phone response rates
go down and non probability sampling on the internet improves.
As more internet polls are conducted, questions about representation
come up. We're not getting less diverse, less racially and ethnically
diverse, we're getting more racially and ethnically diverse and we
can really no longer afford to collapse any of these variables
or any of these samples into white/nonwhite and then look at opinion
and behavior. Another concern for the industry is the Telephone
Consumer Protection Act, or TCPA, which prohibits calling cell phones
with automated machines, which also raises the price of conducting
probability sampling. Depending on whether you do a telephone pole or
an online call or text message polling or whatever, the cost can vary
significantly. But you're talking about thousands and thousands or
tens of thousands of dollars for a standard poll.
One way pollsters are getting around that is by emailing and texting
the people on their list to do the poll that way, creating hybrid
models to increase their response rates.
There has also been a move for surveys and polls to social media and
opt-in panels, but results from those polls should be taken
cautiously. Polling can actually serve as a counterpoint
to this kind of bubble that's formed on social media that people
think: "This is what everybody's talking about." And then polling can
kind of set you right and say, "No, that's not what everybody's
talking about. You're just a small proportion of the public that are
existing in that." But polling firm clients are wanting to use these
methods more since they are cheaper to conduct.
We have to look beyond telephone research for what's going to be the
next thing and invest our brain trusts within the organization and
making those things work.
And they are less expensive.
Certainly clients that want to do national public opinion polling or
any particular subgroup in particular that can be hard to reach on
the telephone wants to use panels.
So there's a lot of pressure to do it and the money's
coming in for that kind of research.
How valid are these panels based on the individuals that opt in?
Especially when you're thinking about, you know, internet surveys and
the issues we have pertaining to the digital divide.
While those divides have been in the forefront during the covid-19
crisis, some pollsters have actually gotten better response rates as
people stay home. We've been getting a lot more success in people
picking up their cell phones at different times of the day and people
actually wanted to talk to us.
Actually our response rates actually popped up just a little bit.
The polling landscape is changing as people remain skeptical of polls.
However, pollsters suggest that "a robust public polling industry is
a marker of a free society."
Most people think, you know, the true mark of democratic health is how
many people turn out to vote and people say: "Oh, my word,
how can you live in a country where there are only 32% of its
residents voting?" But if you think about countries where you have
99, 100% voting, these are totalitarian states where you have to go
and vote. So the idea of polling, the practice of polling,
is a sign of democratic health and it will not go away.