Vsauce!
Kevin here -- when KGB Chairman Yuri Andropov was conducting a job interview, the man said,
“Let me tell you about myself...”
Andropov stopped him and asked, “What makes you think that you know more about you than
we do?”
That’s really creepy -- and also probably true.
Data collected via self-reporting compared to outside observation tends not to be the
same.
You’d think that the most reliable way to get crime data is to just ask the people involved,
both the victims and the criminals.
Going straight to the source makes the most sense, right?
WRONG -- because not only can we not trust you, you can’t trust you.
Analyzing crimes, criminals, and victims is rife with problems -- it’s a running theme
in my videos.
The Worst Math Ever Used in Court shows how probability can be twisted and misapplied
to be flat out wrong -- and the Sally Clark video showed how that can ruin lives.
If you think eyewitness testimony is bulletproof, ask Jennifer Thompson and Ronald Cotton.
And if reducing crimes to objective data seems smart, check out the mad lib that is the US
Federal Sentencing Guidelines or the data-driven monster called CompStat.
If getting reliable information from the criminal justice system is too complex, why not make
it as simple as possible by collecting data directly from the people involved in the crimes?
LET’S FIND OUT.
Even in the 1940s and 1950s, researchers knew systems were biased and highly variable, so
they wanted to bypass the bias by gathering data from primary sources.
That way there’s nothing getting in the way of the truth.
Except us.
Some of the problems with self-reporting are obvious -- you need to ask people the right
questions.
Bad questions, bad responses.
And you need to ask the right people, too.
Great questions to the wrong people just… aren’t useful at all.
I don’t know what it’s like to go into space.
Ask an actual astronaut.
They would know.
Especially the ones that have been into space.
But even a really straightforward and objective question like measuring the average height
and weight of the population can be a total mess.
We’re just dealing with numbers, right?
This should be simple.
Let’s ruin this in three ways.
One.
Doctors and nurses measuring during hospital appointments -- this is professional data
collection, the OPPOSITE of self-reporting; Two.
Tracking the average sizes of clothes purchased over time, this one’s part professional,
part self-reporting.
Three.
Just asking people.
Self-reporting.
Hospital measurements are going to be accurate… but the sample is not likely to be representative
of the broader population.
By default, acutely sick and chronically ill people are overrepresented.
The result is a great measurement of an atypical population.
So bad data.
Trends and styles affect the size of clothing people buy -- are the $186.9 million worth
of JNCO jeans sold in 1998 an accurate measure of weight that year?
Well, puffy pants still have the same waist and inseam, right?
Maybe!?
Take a look at this chart detailing how “Size 8” is actually 10 completely different measurements
depending on the retailer.
Vanity sizing ruins the data.
And do larger people buy more or fewer clothes than smaller people to skew the data set?
And which stores should we even track?
Brooks Brothers?
Tommy Bahama?
Ross Dress For Less?
Who knows?!
Bad data.
Okay, so let's just ask people -- a 2009 study showed that men over-report their height by
about half an inch, and women under-report their weight by an average of 3 lbs.
It doesn’t mean anyone is lying -- they probably really believe they’re giving accurate
information.
The problem is that it’s wrong so… again bad data.
Is it hopeless?
YEAH, yeah, it’s probably hopeless.
In 1964, a photographer asked if she could shoot Andy Warhol’s Marilyn Monroe paintings.
He thought she meant photos but she pulled out a pistol and fired a shot through Marilyn's
forehead.
It took 4 minutes for "Shot Sage Blue Marilyn" to sell for $195 million -- which happened
as $9 trillion was wiped from the stock market and inflation ran rampant.
Morgan Stanley reports that the average fine art piece is selling for 26% more at auction
compared to last year -- because art has some unique qualities as an investment.
You can buy shares in works from great artists -- from Banksy to Picasso -- with Masterworks.
They have over half a million members because they've gotten results, like a sale in October
that netted a 21.5% return for its investors.
As a result, demand for Masterworks is so high that there’s a wait list.
But you can skip it by clicking the link in the description for priority access -- and
do every bit of research you can.
All investments carry risks, so be smart about it.
Go to Masterworks.art/Vsauce2 -- and thanks to Masterworks for sponsoring this video.
James Short and Ivan Nye sensed they were getting misleading and incomplete crime data
in the 1950’s, and… they were.
They recognized that criminals can manipulate to sway the tone of interviews, so they threw
in questions to expose over- or under-reporting of delinquency.
Like questions about whether criminals had ever disobeyed their parents or ever lied
-- unrealistic answers like claiming you’ve never once lied to your parents presented
an image of manipulative “over-conformity,” so those respondents were excluded.
So if you know data is routinely unreliable, at least you can filter out the very worst
of it.
The thing is, most people do try hard to be accurate -- but their own worldview and the
limits of language can dramatically skew the results.
Let’s use some of Short and Nye’s real self-reporting questions to generate some
BAD DATA.
“Have you ever had a fist fight with one person?”
-- Well, what does that mean exactly -- does horsing around with a sibling that gets out
of hand constitute a fist-fight, are we talking competitive combat in a kumite, or instigating
or defending yourself in a violent altercation?
Those are all pretty different fist fights.
“Have you ever drank wine, beer, or liquor in your own home?”
-- Does that mean stealing your dad’s bottle of Pappy Van Winkle and guzzling it in your
bedroom, or your mom giving you one single sip of giggle juice on New Year’s Eve?
I don’t know.
“Have you ever driven too fast or recklessly?”
To me, 15MPH over the speed limit is too fast.
To my grandma, 2 MPH over the limit is a death race.
Individual judgment matters, and we can’t guarantee that over- and under-interpretations
will cancel each other out to give us an accurate picture.
Probably the most obvious problem with self-reporting is that…
people lie.
Especially as the behavior to which they’d be admitting -- even anonymously -- becomes
more serious.
You’re more likely to admit to swiping a sip of wine than overtaking an international
drug trafficking syndicate.
But occasionally self-reporting gives us a nugget of extremely valuable truth.
In 1978, the RAND Corporation conducted surveys of inmates in California penitentiaries to
assess their levels of criminal behavior prior to their incarceration.
Inmates were asked how many times per year they committed each of the following crimes,
and they got a median value:
Burglary, Robbery, Assault, Theft, Forgery and Credit Card fraud,Fraud, and
All crimes except drug dealing.
We’ll say that someone at the 50th percentile of crime frequency for each of these offenses
represents the ‘average’ criminal -- but what about the criminals at the 90th percentile,
the very worst ones?
Burglary, 232 Robbery, 87
Assault, 13 Theft, 425
Forgery and Credit Card fraud, 206 Fraud, 258
All crimes except drug dealing, 605
605.
8 thefts a year is fewer than 1 a month.
425 is more than 1 a DAY.
The differences here range from about 5x more crime, like assault, to FIFTY times more on
thefts and fraud -- even if the median inmate underreported his crimes and the top 10% worst
criminals over-reported because they were bragging, the data reveals a substantial gap
we didn’t know existed.
We wouldn’t need the exact numbers to understand that the criminal class is highly varied,
and that most criminals haven’t dedicated their entire life to crime -- but a portion…
absolutely, quantifiably have.
The good news is that informs our approach to both punishment and rehabilitation.
The bad news is… it doesn’t tell us the truth.
We don’t know exactly what the inmates considered to be crimes, whether they were honest, or
whether they even remembered correctly.
And it doesn’t answer the question about how we can make sure criminals don’t re-offend,
or whether there are people we truly will never be able to help.
But it does tell us something -- and it’s up to us to use that information in conjunction
with everything else that we’ve measured and collected.
Sometimes that is helpful; sometimes we have to admit it’s not.
The strange thing is, we know we can’t trust what people say...
But if we never ask, we’ll never know.
And as always, thanks for watching.