- [Instructor] In this video, we're gonna think about
what makes a poll or a survey credible.
Because remember, the whole reason why we're going
to do a poll or a survey is
we wanna understand public opinion.
But if it's not statistically credible, if we can't believe
what it's saying or if we don't understand exactly
what it's saying then we might not have a good sense
of what public opinion actually is.
So the first really important thing,
if you're writing any type of survey, whether it's
in politics or government or not, is that you are taking
a random, random, sample.
What does that mean?
Well we go into a lot more depth in it in our statistics
content on Khan Academy, but it means that you should look
at the people, the voting population.
And it does get a little bit nuanced and tricky
on who are likely voters, but you would go to say
the voting population or the people in an area
whose opinion you care about, and you would want
any one of them to have an equal shot
of being selected for your survey.
So good examples of random samples is maybe you take,
you have a random phone number generator of likely voters
in your district and you call them up.
Maybe, and even then you might say well hey,
maybe certain types of people pick up the phone
and certain people don't, or certain people have a phone
and certain people don't.
So you have to be very careful about this design.
But examples of non-random sample would be to go
hang outside of the Democratic party headquarters
in your district and just survey people coming out
of that building, that would be very not random
and you would get very skewed results.
Now, the next thing that you would want when you are
taking a survey is that you want a large population size.
Now, typically speaking, it's going to be
at least 500 folks, but you're going to see a lot of surveys
that are about 500 to 1,000 participants.
And sometimes in statistics they'll say your N,
which is the number of people you surveyed is 500 to 1,000.
And this is so you have a good chance of getting close
to the true public opinion.
We're gonna talk in a second about margin of error.
The larger this is, and especially if you're doing
a true random sample, this is so,
larger, larger, it reduces the, you'll have a lower,
margin, margin, of error.
And I'll explain in depth what
a margin of error is in a second.
Now, another key thing is whatever you ask in your survey,
it needs to be, the language, needs to be
as neutral and unbiased as possible.
Let's say there's a new proposition,
you could get very different results
if the wording of the question is do you support funds
for those in need or if you said do you support funds
for those who are not working.
Those could get very, very different results.
And this is, there's really an art to trying to get
a neutral, unbiased question there.
Last, but not least, you wanna make it very transparent
to the public how you conducted your poll
so that they can decide for themselves
how credible your poll actually is.
So with that out of the way,
let's look at real poll results.
And then I'm gonna dig into what the idea of margin of error
is, because you'll see this a lot when you read the news.
So this is from the Sun Sentinel in Florida.
They have a senatorial race going on, and it says
despite millions of dollars in television ad spending,
Florida's US Senate race between
Bill Nelson and Rick Scott hasn't budged.
A Florida Atlantic University poll shows Republican Scott
with 44% of voters surveyed, Democrat Nelson with 40%.
So the first thing you might say is well, what about
this only adds up to 84%, what about the other 16%?
Well, those could be undecided voters or maybe there's
a third candidate there.
That's a four-point advantage in Scott's favor,
but it's within the survey's margin of error,
which means the race could be tied,
or Scott could have a lead.
The FAU Business and Economics Polling Initiative
survey of 800 Florida registered voters was conducted
online and through automated calls to people
with landline telephones.
Researchers said it had a margin of error
of plus or minus three percentage points.
So let's think about whether this is a credible poll
and what inferences we can make from it.
So the first thing is,
were they transparent about their methods?
And it seems like they were, this whole,
this paragraph right over here, they're very transparent
about their methods, how many people did they survey,
they say how they actually conducted the survey,
and so check on transparency.
And then because they were transparent, we can check
whether these other things are true.
So first, random sample.
They don't go into a lot of depth
on how it was conducted online,
but if it is a neutral site where any Florida voter
is equally likely to show up,
well then that might be a good random sample.
Now, you have to be careful with online because
who has access to online,
who does not have access to online,
who might go to whatever site it is being conducted on.
So this has a couple of question marks right over here.
And then they say through automated calls to people
with landline telephones.
So once again, this seems reasonably random,
it's along the lines that I talked about earlier.
Maybe they had some random phone number picker
in Florida and they called those folks,
but maybe, you know, there's certain people have landlines,
a lot of people now only have mobile phones.
There's certain people who pick up and might answer things
and certain people who might not.
So as you can see, there's an art to getting
a truly random sample, but you can tell that the
Florida Atlantic University group
tried to get a random sample.
The next was the large population size.
Well, they said a survey of 800 Florida voters,
so that's pretty good.
And it's not just 800 for 800 sake,
it's the number of voters
that's gonna drive the margin of error.
So this is a good time to say what is the margin of error?
So a margin of error is going to be associated
with a confidence level.
It's typically going to be a 95% confidence level
if they don't tell you otherwise.
And so what this means is, is that 95% of the time,
when you take a random sample of 800 Florida residents,
you will get a result that is within three percentage points
of the true result.
Remember, there's some true result we don't know
unless we could perfectly get into everyone's head
and this poll is a way of trying to estimate that.
So one way to think about it, within 3% of what you got,
Republican Scott is really, you could create a confidence
interval where you could say the confidence interval's
going to be 3% less than this, which is 41%,
all the way to 3% more than this, which is 47%.
And one way to think about it is,
there's a 95% chance that this will overlap
with the true result.
And you might say hey, well the low end of this range
is still higher than Democrat Nelson,
but that confidence level applies
to Democrat Nelson as well.
So Democrat Nelson's confidence interval would be
between 37%, 3% less than this, and 43%, 3% more than this.
There's still a reasonable chance maybe the Democrat's
at the higher end of this range which could still mean
that the Democrat could win even though the headline numbers
show a four-point advantage for the Republican.
And outside of the statistical error, the margin of error
we're talking about, you have to remember that this poll
is before the actual election.
People change their minds,
there's a whole campaign going on,
so even if you were to get the exact result,
this is just a snapshot in time.
It could change on election day.
Now this last question where they asked a neutral,
unbiased question, we don't know exactly,
at least just from this article,
they might have published it in the details of the poll,
but if they asked something like who will you vote for
on the election on this day, that would be
very neutral and unbiased.
But if they say are you going to vote for the public servant
who has served Florida for many years
or that Democrat guy, well that would be not unbiased.