Imagine it’s your first time voting.
Maybe you carefully filled out your mail-in ballot or waited in line on a Sunday or Monday
or even Tuesday to cast your vote and maybe show off your “I voted” sticker.
You read up on all the candidates and ballot initiatives and according to the news, some
of the races could be pretty close.
But where did that data come from, and how do you know what to pay attention to as polls
close to get accurate updates?
Every vote counts! Our opinions and votes even before Election Day are valuable data
that take a snapshot of where a country stands.
How blurry it is depends on how the poll is crafted. So let’s learn how to keep it in focus!
I’m Jessica Pucci and welcome to Study Hall: Data Literacy presented by Arizona State University
and Crash Course.
It seems like there’s always an election coming up, whether it’s for your local school
board or for the top job at the national level.
Which also means there’s always someone trying to win our support.
Political parties and candidates need to know what the public thinks about current issues
so they can be effective leaders…
but they also want to strategize to get the job. How do they know what the electorate
is thinking?
They ask them! Well, sort of.
They can’t ask everyone what they’re thinking, but they can talk to a sample of the population
to get a general sense of opinions.
Surveys that have questions related to politics and elections are often referred to as polls.
And the people doing the actual question asking are pollsters, or teams that organize, conduct,
and analyze public opinion surveys.
An important survey tracks approval ratings and asks the public whether they approve or
disapprove of the way the president or prime minister or legislature or whatever are handling
their job.
But pollsters ask lots of different questions, from your opinion on the economy to social issues.
George Gallup, statistician and polling extraordinaire, founded the American Institute of Public Opinion
in 1935 to collect the US public’s opinions.
Demand for the information was high, so more pollsters followed.
Now there are many different organizations — including the Gallup Poll which continues
today — that keep track of the public’s political leanings.
In the European Union, the Eurobarometer is a set of public opinion surveys that have
been collected since 1973,
and the Afrobarometer initiative has been active across Africa since 1999. Or the Pew
Research Center has conducted a Global Attitude Survey since 2002.
But like we learned in our episode on Data Collection, even the best surveys have their
limits. When deciding which pollster to trust, transparency is a good sign.
Organizations like the National Council on Public Polls in the US and the British Polling
Council in the UK
provide guidelines for how opinion poll results are reported and pressure pollsters to explain
their process thoroughly.
But we can do our own investigation of a pollster’s methodology too.
We know both sample size and margin of error are two important numbers to look for when
fact-checking any statistic.
Like with an approval rating, we want to know how many people were asked and get a range
of where we expect the whole population’s approval rating to be.
But in countries with more than two major parties like Iceland, Tunisia, or France,
the margins of error for each party’s approval might overlap and confuse everything.
Like for three fairly equally popular parties, the margin of error ranges could be between
30 and 35 for each party.
So no matter what the poll reports, the country might be split ⅓-⅓-⅓ and it’s still
anybody’s game.
Plus, the sample size may be split into so many sub-groups that for less prominent parties,
you might not actually have that much information at all.
Like if for one smaller party you only get two supporters out of 100 people, that isn’t
a lot to go on for what that party thinks.
The survey method, or how the pollster contacts people, also makes a huge difference.
Common ways of reaching people for a poll are the internet, mail, and phone.
But deciding which method to use is harder than you’d think.
Take the 2016 US presidential election where the end result left many pollsters very surprised.
Among the polling giants, only a poll conducted by USC and the LA Times and a few smaller
pollsters predicted Donald Trump might actually win.
A popular explanation for what happened is the “shy Trumper” hypothesis.
Some experts suspected the 2016 election had such a surprise ending because voters were
embarrassed to admit they supported Donald Trump in phone polls, which skewed the results.
But actually, the news organization Politico compared telephone polls and online polls
where there isn’t social pressure
by randomly assigning people to take one or the other, and there actually wasn’t much
evidence to support the theory.
But if we cross the pond and look at the referendum for whether or not the UK should withdraw
from the European Union,
a study did find evidence that there were differences between results from phone and
online polls.
“Shyness” might have played a role, or there could be differences in the type of
person who spends more time on phones versus the internet.
We want our sample to be a good representation of the whole population, but if our best efforts
to pick a survey method fail, there’s more we can do.
In 2010 the US Census told us that about 13% of the US population is 65 years old or over.
So if 30% of the people we reached by phone were 65+, we could downweight, or multiply
the results from that group by a number less than one, so that their opinions count for
closer to 13% of the overall results.
Then our sample and results are a better reflection of the population and public opinion.
Even the wording of the question matters. In the same study on the UK referendum, researchers
found if “don’t know” was an option, a large group picked that.
But if there were only two choices — to leave or remain in the EU, you’re in or you’re
out — undecided voters tended to say they’d remain in the EU.
And finally, if we’re trying to be data literate, it never hurts to follow the money
and see who commissioned the poll.
If a candidate or specific party pays for the poll we might wonder about their motivations,
especially if the results are favorable for whoever wrote the check.
With so much to look out for, it would be nice to know who’s been pretty reliable,
and we can find that by checking a pollster’s record.
If an organization has been around long enough, we can compare the election polls to the actual
election outcomes.
Nate Silver and the FiveThirtyEight team have graded a variety of US pollsters based on
their history and methods.
In the US as of mid-2020, polling groups like the ABC News and Washington Post team and
the Siena College and The New York Times Upshot team have “A+” track records, among others.
Despite the challenges, pollsters still make predictions and won’t be stopping any time
soon. And maybe you’d like to give it a try!
Like if you decide to provide your community with polling data for the local election.
Let’s see some of the hurdles an analyst must overcome to collect data in the Data Point.
If you were an analyst during the late 1940s, just calling random phone numbers would be problematic.
Your sample couldn’t possibly represent the US population because not everyone had
a home phone.
That's what happened in the Dewey versus Truman election.
The polls favored Dewey because people with the money to have a phone were more likely
to be Republicans.
Now landlines are widely accessible but you still have a problem.
Many people only have a cell phone, so you’ll still miss a large chunk of the population
if you call random landlines.
This wouldn’t be so bad if the people without landlines were basically the same as those
with them, but the cell-phone-only people tend to be younger.
And in the US, the younger generations tend to lean more Democratic, according to 2017
PEW research data.
Even adding cell phone numbers to your list of phone numbers doesn’t get you in the
clear if they don’t pick up.
Non-response bias occurs when the people who fail to respond are different from the average
person who did respond.
Maybe the phone number belongs to someone working the night shift, which makes them
different from all the other working people in your sample.
This could be a problem if your town is deciding on a ballot measure to block loud construction
during particular hours of the day.
Even if they answer the phone, you could have to deal with response bias,
which happens when someone changes their answer because they’re uncomfortable telling the
truth to whoever’s asking the question.
Like if you said you were a college student and then asked how someone felt about Generation Z.
You might not get the truth because people won’t want to offend someone they think
is probably in that generation.
It’s harder than it looks to produce a good poll, even at the local level!
Even though the pollsters are hard at work collecting data way in advance,
we may not pay much attention to all the numbers and predictions swirling around when the election
is far off.
But as one looms closer, we start to tune in.
Closer to elections, pollsters focus on likely voter polls to try to get the best possible
picture of what could happen on Election Day.
But predicting who is likely to vote is opening up its own can of worms.
In the 2016 US presidential election, there’s evidence that some likely voter polls didn’t
do a good job of picking samples that matched the people who actually ended up voting.
This is a risk a pollster takes when focusing a poll only on “likely” voters; our expectations
don’t always match reality.
Come Election Day, it seems like news organizations get vote tallies almost in real time.
And you might even see races called before all of the votes are counted.
Real time data often uses exit polls, which are interviews with voters after they’ve
voted...but they might be going out of style.
Nowadays about 40% of ballots in the US are cast early, absentee, or by mail, so exit
polls are less representative of the entire electorate than they used to be.
And after exit polls were consistently skewed towards Hillary Clinton in 2016, the Associated
Press announced in 2018 they would no longer rely on them
and risk reporting election results prematurely.
If one candidate has a lead that is way beyond the margin of error though, media entities
may “call” the election in favor of the candidate before all the votes are officially counted.
But this too comes at a risk. Early poll results could keep some people from voting if they
think their vote won’t make a difference.
Election data is a numbers game and how we report that real-time data will probably change
in the coming years.
But if you have a vote, use it!
Today we learned what political polls are and how they monitor public opinion over time,
particularly in the US because that's where we're filming this.
And in the spirit of data literacy we want to acknowledge our own information bias.
But in the ten minutes we had, we tried to include some other places all over the world
and encourage you to do research about where you live!
Political polls can provide useful information, but even the best pollsters can get it wrong
since the survey method (and even the question wording or type of response) can affect the results.
And we learned it’s hard to predict what will happen on Election Day. Since who participates
has a big impact, as the day approaches,
likely voter polls replace broader public opinion polls as our best guess of the outcome.
Next time the doctor will be in, and we’ll learn about data in health.
Thanks for watching Study Hall: Data Literacy, which is produced by Arizona State University
and the Crash Course team at Complexly.
If you liked this video and want to keep learning with us here in Study Hall, be sure to subscribe.
You can learn more about ASU and the videos produced by Crash Course in the links in the description.
See you next time!