We like to think of numbers as being pretty objective.
1 + 1 is 2.
Always.
But just like words or images, numbers are open to a whole smorgasbord of possible interpretations
when it comes to data.
Two people can take the same data set and end up with completely different results just
based on how they decided to analyze it.
Like, say this came across your feed: “This just in!
Scientists prove exposure to trees causes hair loss!”
But then a bit later you see: “Human hair loss is the leading cause of deforestation,
new study finds.”
That sounds absolutely ridiculous.
But when data is reported, there usually isn’t enough space or time to get into the original
source material.
Or even what methods led to the conclusion.
So as consumers of data, we want a simple story that synthesizes a whirlwind of information,
but we need to also cultivate some healthy skepticism.
Today we’ll build the tools we need to avoid being misled -- or mislead other people! -- and
sniff out common misinterpretations of data.
I’m Jessica Pucci and this is Study Hall: Data Literacy presented by Arizona State University
and Crash Course.
Even though data just won’t leave us alone, most of us are not statistics pros.
And that’s okay!
Being data literate doesn’t necessarily mean we’re completely self-sufficient.
It just means that we know how to ask questions and are aware of common misconceptions.
Maybe the person interpreting the data is an experienced journalist, or maybe they're
just our downstairs neighbor Casey who likes to share interesting factoids when we pass
on the stairs.
Hopefully they both considered how the original data was collected before using it to tell
a story or combining it with other information.
But no one is infallible, and misconceptions can sneak through.
Now maybe you’ve heard a data scientist harp about it endlessly, but the misconception
that seems to get talked about the most is the difference between correlation and causation.
Correlation describes patterns in mutual relationships between variables.
So if one variable changes, the other one also tends to, and vice versa — it doesn’t
matter which one we start with.
Usually we focus on two types of correlations.
If large values of one variable tend to show up with large values of a second variable,
we say they have a positive correlation.
But if large values of one variable tend to go with small values of a second variable,
we say they have a negative correlation.
Like the longer you spend working out in the morning, the less you sleep.
But causation goes a step further and describes relationships between variables that have
an action and a direction.
If the value of one variable increases, it causes the second variable to increase.
But not the other way.
It’s much bolder to say that one variable causes the other to change.
In fact, working out and calorie burn aren’t just correlated, working out causes us to
burn more calories.
We can think about this more using information collected during the U.S. Census,
which counts the U.S. population every 10 years, and the annual American Community Survey
which tracks more details about people and households in the U.S.
So, suppose while we asked families about where they lived, we also asked how many rooms
their dwelling had and how many TVs they owned.
The important questions!
We probably expect these two variables to be positively correlated.
It seems plausible that people living in places with more rooms might also own more televisions.
Right?
Bathroom TV?
Who’s with me!?!?
But we know that buying a new television doesn’t cause you to buy a new house with an extra
room, and having an extra room in your house doesn’t cause you to buy a new television.
There must be something else going on that influences both of these variables.
And if we think about it, what would impact both the number of televisions and the number
of rooms a household has?
Income.
Households with higher incomes can afford more TVs and bigger houses.
Income is an example of a confounding variable, or confounder for short.
It’s a variable that clouds other relationships and makes variables seem directly related
when they’re really not.
And they can be sneaky.
Even if we suspect there might be a confounding variable hiding in our data, we may not have
the rest of the data we need to detangle the real relationship --
or we might not even see the relationship at all!
Confounding variables can flip a correlation between positive and negative, completely
mask a relationship, or even create the illusion of a relationship.
Or they’re something tricky to measure.
For instance, people might not mind talking about their number of TVs and rooms, but they’re
probably more unwilling to reveal detailed info on their income.
Talking about money is really awkward for a lotta people!
We’re pretty confident about our guess that income is a confounder, but figuring out if
two variables are linked by correlation or causation is not always straightforward.
The best way to determine if a relationship between two variables is a truly causal one
is to craft a study that controls the scenario to avoid confounding variables.
Alas, that isn’t always possible.
For example, even though the connection between lung cancer and smoking appeared very strong
in the 1950s, there could have been confounding variables,
like a sneaky genetic factor that makes people more likely to smoke and more likely to get
lung cancer.
To be sure smoking caused lung cancer, researchers could’ve formed a study where they forced
some people to smoke more than others and see who got lung cancer more often.
But, you know, that’s more than a little problematic since trying to give people lung
cancer is REALLY unethical.
So whenever you see someone claim one variable causes another variable, it’s time to put
your data literacy skills to work.
Take a moment to brainstorm what kinds of other variables could be confounding the relationship.
Typical confounding variables are demographics like age, sex, income, education level, and race.
If you can’t think of any logical confounders, then you may have more confidence in the claim.
So, now that we’ve seen how others live, let’s make moves to find our own dream house
in the Data Point — complete with the perfect room-to-television ratio, of course!
Suppose you like to drive around looking for houses for sale.
As you house hunt, you record the asking price and address of each house where you see a
“for sale sign” so you can look back on what the houses you like really cost.
After a long day of driving and some snooping on Zillow, you decide to find the average
house price per neighborhood.
Eeek!
One neighborhood sticks out with an average house price of over a million dollars!
Yikes, maybe you won’t be moving there.
But wait, you know people who live in that neighborhood, and you’re pretty sure they
aren’t millionaires.
What’s going on with this data?
Like we learned last time, the average isn’t the only way to measure the center of a data
set, and each measure comes with its own weakness.
You remember that the average can be skewed, or pulled in the direction of an outlier,
or data point that is not representative of the rest of the distribution.
So now it’s time to do more digging — and by that I mean you need more data.
This time you collect data on the house owners in the neighborhood that’s maybe full of
secret millionaires.
Sifting through publicly available property records online or down at the county clerk’s
office, you realize a reclusive celebrity lives in that neighborhood,
and the value of that one house significantly skews the average housing price for the
whole neighborhood.
With this in mind, you may want to investigate the median, or middle, house price in each
neighborhood to get a more reliable estimate of what you can expect to pay.
There’s still hope of finding the house of your dreams!
What we see here is an example of another common misconception.
Measures of center can be skewed so we might want to compare the mean to the median for
a quick sense of whether outliers could be lurking.
The final misconception we’ll talk about today is one so pervasive that it even has
a law named after it: the law of small numbers.
Which isn’t a real law but a clever term for when people are too quick to generalize
from just a small amount of data.
So we did lots of driving on our house-hunting expedition, but let’s stop to think.
If we could only find listings for three houses in a neighborhood, would we really trust any
statistic we’d calculate — even the median price — to truly represent the many,
many other houses in the neighborhood?
With only three houses, it’s possible we ended up with three odd-ball houses, and their
median won’t help us understand the neighborhood at all.
Personally, I’d feel a lot better about the median price if it was calculated from
300 houses.
With a lot more houses in our data set, it’s pretty unlikely they all have prices way off
from the true median of the whole neighborhood.
In fact, the law of large numbers is a real law — at least in the world of probability
— that says the larger a random sample is, the closer its average should be to the true
population average.
So basically, don’t jump to conclusions with only a little data.
Let’s stick to the law of large numbers when making decisions about data and always
ask about the sample size to help decide if we can trust the summary statistics,
like the mean, median, or mode.
Now we’ve talked about a few data misconceptions to look out for, but those certainly aren’t
the only tricks to be aware of.
In general,
it’s a good idea to know how to interrogate each story that relies on data.
Let’s go back to those trees!
First, ask where the data came from.
Whatever the overall conclusion is, it might be misleading if participants were all from,
say, a retirement home.
Then we might expect its older residents to be losing their hair for other reasons than
long hikes in the forest.
Second, think about who analyzed the data.
Maybe they have the same dedication to data literacy as we do, but maybe not.
Or they might have a particular point of view that introduces some bias whether they mean
to or not.
And third, look for anything that’s missing from the data analysis.
Beyond the headline, find out how the sample was collected and how big it was.
Are we talking about 5 tree climbers with thinning hair, or 500?
There are steps we can take to evaluate all data we find, whether it comes from a trusted
source or not.
But when something seems especially fishy, there’s more we can investigate.
Putting multiple data sources together is tricky because there’s lots of different
ways to collect data and the two sets might not really fit together.
We might be more wary of a conclusion if data comes from studies with different
collection methods.
Or we could look into extreme hair loss cases to see if there were outliers affecting the
overall conclusion, like we did for the celebrity house.
Basically, being data literate means we have to be willing to play detective every time
we’re presented with a data finding.
So today we learned to interrogate the data collection process, the data collectors, and
the broader context of the finding to see if anything seems suspicious.
No potential confounder is safe from our scrutiny!
Common data mishaps don’t scare us now!
We might even feel confident enough to go to the original source of data for more information
by following a link in a news story or searching for the original study ourselves.
Next time we’ll talk about how to critically assess data that’s presented visually.
Thanks for watching Study Hall: Data Literacy, which is produced by Arizona State University
and the Crash Course team at Complexly.
If you liked this video and want to keep learning with us here in Study Hall, be sure to subscribe.
You can learn more about ASU and the videos produced by Crash Course
in the links in the description.
See you next time!