Hi, I’m Adriene Hill, and welcome back to Crash Course Statistics.
And sadly, we’re nearing the end of this course.
We’ve covered a LOT of topics.
From probability, to t-tests, to Machine Learning to Bayesian statistics.
Today, we’re going to go more “Real World.”
We’re going to talk about how statistics is used in the courtroom to make...pretty
important...decisions.
In particular, we’re going to look at 3 individuals whose lives changed by the use
of statistics: Alfred Dreyfus, Sally Clark, and Jonathan Dorfman.
INTRO
In 1894, a political scandal began in France that would last until 1906 when Alfred Dreyfus--a
Jewish Officer in the French army--was convicted of treason.
His conviction hinged on an unsigned letter--referred to as the bordereau.
The letter offered French military secrets for sale, and the courts decided that it was
Dreyfus who had written it.
Dreyfus was convicted, and for a time, sent to live in a prison cell at Devil’s Island.
Which is about as pleasant as the name suggests.
This story is complicated.
Many books have been written about it.
But we’ll focus in particular on a handwriting analysis done by Alphonse Bertillon--the founder
of the first crime laboratory for police in France.
Bertillon alleged that Dreyfus purposely made the bordereau look like a forgery of his own
handwriting.
That way, if he ever got caught, he could claim someone had tried to frame him.
Bertillon theorized that Dreyfus created the bordereau by tracing words and letters from
various sources, including one in his own handwriting and his families’ handwriting.
That would make the document look more like an forgery than something he’d actually
written.
One of the words Bertillon believed got traced over and over was “intérêt” in Dreyfus’s
brother’s handwriting.
He’d found a letter written by the brother and thought that the word in that letter looked
similar to some words in the bordereau.
The book Math on Trial: How Numbers Get Used and Abused in the Courtroom breaks down his
process from here.
“Intérêt” contains five of the most commonly used letters in the French language
-- e, n, r, i, and t.
So it would have been a logical one to pick for repeated tracing.
Bertillon created a key to test his theory.
He traced “intérêt” in the brother’s handwriting over and over with no spaces on
a line.
This is how he imagined Dreyfus would have created a key of his own:
Conveniently, the bordereau was slightly transparent, so Bertillon could place his key underneath
it, then look for places where letters overlapped.
And yes, there was some overlap, but the majority of letters didn’t.
Then, he moved his key over a bit and saw some new overlaps.
(And again, a lot of non-overlap).
But, he got excited by the overlapping letters that he HAD seen, so he decided to make two
keys and put them both underneath the bordereau with a little distance between them.
Then he counted up how many times the letters e,n,r, and t overlapped.
And Bertillon concluded that the letters lined up MUCH more than should be expected by chance.
Which...makes some sense.
I mean he had two keys.
Bertillon used the frequencies of these letters in the bordereau to come up with the expected
frequencies of the overlaps.
So, the bordereau contained about 800 letters and there were about 60 r’s.
If his key was just a bunch of R's, then every r in the bordereau would line up with an r
on the key.
But since only 1 out of 7 letters in the key were r’s, he expected 1 out of every 7 R's
in the bordereau to overlap with the key by chance.
So, he expected the letter r to line up 9 times.
But it actually lined up 20 times.
All the other letters--with the exception of ê--matched up more times than he expected.
But this logic was suspect.
A group of famous French mathematicians was asked to inspect Bertillon’s “analysis.”
It didn’t hold up.
They went as far as to call it “completely unfortunate”...which is about as mean as
French mathematicians got back then.
The expected probabilities that Bertillon found may have been relatively accurate--but
only when he overlapped the key and the bordereau ONE time.
Bertillon had overlapped them TWICE.
Which means he counted the overlapping letters BOTH in the regular position (red) AND the
offset position (green).
And it was the rebuttal by Three French mathematicians helped finally free Dreyfus.
Our next story starts in England in 1996 when Sally Clark’s firstborn son Christopher
had the sniffles.
Doctors told Sally that it was a cold.
But later that month, she found Christopher in distress and called an ambulance.
He later died.
In 1998, Sally’s second son, Harry, died in a similar way.
A doctor’s examination of Harry found things that could be indicative of abuse, such as
retinal hemorrhaging.
So, an investigation was conducted.
Steve and Sally Clark were arrested for murdering their sons, but Steve was exonerated.
During the trial, Dr. Roy Meadow, a well respected pediatrician, said the probability that one
of Sally’s children would die from Sudden Infant Death Syndrome, or SIDS, was 1 in 8,543.
He then made an incredibly consequential probability mistake by declaring that the probability
of TWO of these deaths happening in the same family was 1 in 73 million.
Sally’s lawyer confronted Dr. Meadow on whether the probability of Harry’s death
was still 1 in 8,543, even though his brother had earlier died with the same diagnosis.
Essentially, he was asking whether these two events were independent or if having one SIDS
related death in a family increases the probability of having another.
And he was right to ask.
SIDS-related deaths in a family may not be independent of each other.
We don’t know the cause of SIDS, but if it’s even in part genetic or environmental,
it could be possible for babies in the same family to have similar risks.
If Meadow had done the calculation with this in mind, he would have most likely come to
a number MUCH lower than 1 in 73 million.
But that is not the only statistics error that affected the Clark trial.
Whether the lawyers suggested it or the jury assumed it, the unspoken assumption was that
there was only one other option: Murder.
And that since SIDS was so unlikely, murder
MUST be more likely.
The jury sentenced Clark to life in prison for murdering her sons.
But there was another possibility...the deaths could have been caused by naturally occurring
disease or circumstances.
That was found to be the case in Harry’s death.
In 2003, Clark’s conviction was overturned by a Court of Appeal because Harry’s medical
records indicated that he could have been suffering from a Staph infection, which wasn’t
disclosed by a pathologist during the original trial.
When we do statistics, we often only consider one, or possibly two different hypotheses.
But it’s important to remember that there are other OPTIONS out there.
When possible, we should try to consider those other options.
Otherwise, we can accidentally commit this Prosecutor’s fallacy, which incorrectly
claims that since the evidence found is SO unusual or unlikely, that the jury could assume
that the accused defendant isn’t innocent.
And secondly, the fact that one hypothesis is unlikely does not mean that another must
be more likely.
Fast forward to 2011, a student at the University of California, San Diego, Jonathan Dorfman,
was accused of cheating on a midterm by his Chemistry professor.
After two Academic Integrity Review Board hearings, Dorfman was expelled from UCSD,
because he had a previous incident on his record.
He then filed a lawsuit against the school.
After the midterm, the professor had noticed that Dorfman had changed the Test Version
on his answer sheet.
There were four versions of the test -- A, B, C, and D. Students were told that if their
test version wasn’t the same as the letter on their answer sheet, they should tell their
test proctor.
Dorfman HAD changed the letter on his answer sheet from D
to A. But, he said that he arrived late, so he didn’t hear those instructions.
He just saw that his test booklet had a different version than his answer sheet, and changed
the answer sheet to match the test.
After looking at all the exams, the professor also noticed that Dorfman’s test matched
another student--they called them Student X--who had test version A, the same one Dorfman
claimed to have.
24 of the 26 answers matched between the two exams.
8 of the 10 incorrectly answered questions, and all 16 of the correct ones.
The professor of the class took this as further proof that Dorfman was cheating, and even
went as far as to get a statistician to say that the probability of those same eight wrong
answers happening by chance was a billion to one.
Though the court documents do not reveal the exact math that the statistician used, it
seems possible that they, like Dr. Meadow in Sally Clark’s case, probably assumed
independence when no such assumption should be made.
The wrong answers that students choose aren’t always totally random.
On multiple choice tests, many times there’s one answer that looks good, even if it’s
slightly inaccurate.
And since these students were all taking the same course, and reading the same textbook,
their incorrect answers aren’t independent of each other.
Their misconceptions of the material were VERY likely to be dependent on their learning
environment and therefore related.
Dorfman’s lawyer displayed this, during the second review board hearing, by showing
44 of the 618 students who took this test (in all its versions) had 23-25 answers matching
Dorfman’s.
So this one in a billion stat that was presented to the court can be misleading.
To most people, the stat seems believable at first glance.
And much of the University’s initial argument relied on it.
So much so, that they refused to identify Student X--off of whom Dorfman allegedly cheated.
The evidence, they claimed, was so strong that further information about Student X wasn’t
required for the case to move forward.
The court determined that by not identifying Student X, the university had given Dorfman
an unfair disadvantage in his hearings.
By not showing who Student X was there was no way to prove that they weren’t cooperating
or that they weren’t sitting next to each other.
More information about Student X would allow us to update our 1 in a billion chance with
new information, much like we update beliefs in Bayesian Statistics.
Taken alone, the probabilities can be misleading.
The Court ruled in favor of Dorman, saying that if UCSD ever wanted to bring more charges
against him, that they’d need to find Student X.
Many cases are influenced by poor probability calculations.
Understanding basic rules about probability and statistics, and being skeptical of the
probabilities you may hear can have a huge impact on whether or not you come to the right
conclusions.
Thanks for watching, I’ll see you next time.