[MUSIC PLAYING]
DREW BINHAIR: This is Drew Binhair, reporting live
from the Discretown Medical Clinic.
The town is grappling with an outbreak of hypertangleosis, a condition which
makes hair tangled and gnarled.
As you can see, the line is out the door.
Let's go to Dr. Sarah Straighthair for the latest.
Doctor, how is your clinic coping with this horrifying outbreak?
DR. STRAIGHTHAIR: Please, everybody, stay calm.
Hypertangleosis is treatable and does not cause long-term health issues.
We treat it at our clinic with a special conditioner which
usually cures the disease immediately.
It's not always effective, though.
We have limited capacity and can only treat 10 patients per day.
We're working to address these issues.
DREW BINHAIR: Thank you, Doctor.
Good luck, and please keep us posted.
We go now to Professor Joe Blitzstein from the Harvard statistics department,
to tell us more about how to make sense of this disturbing situation.
Professor, can you tell us how effective the treatment is?
JOE BLITZSTEIN: Based on the limited data we have so far,
we estimate that the treatment has a 60% chance of success
and a 40% chance of failure.
DREW BINHAIR: Let me get this straight.
So, each day, you expect that 6 out of the 10 treated patients will be cured?
JOE BLITZSTEIN: On average, 6 out of the 10 will be cured.
But there could easily be more than six, or fewer than six on a particular day.
We have to consider the whole distribution.
DREW BINHAIR: You mean, who distributes the treatment to the clinic?
JOE BLITZSTEIN: We should try to find out
as much as we can about the treatment.
But I meant distribution in the statistics sense.
When we are looking at a quantity that can take on various possible values,
the distribution tells us how likely any particular value or range of values is.
Let's say we want to predict X, the number of patients
who will be cured tomorrow.
The distribution of X specifies the answers
to questions like, what's the probability that X is 6?
Or what's the probability that X is between 4 and 8?
DREW BINHAIR: So what is the distribution of X?
JOE BLITZSTEIN: We don't have enough data to be sure.
A lot is unknown about this treatment.
For example, is it more effective on young people than on older people?
Assuming that each patient has probability 0.6 of being treated
successfully, and that these patients are independent of each other,
the distribution of X is called Binomial 10, comma, 0.6.
Here's a picture of this Binomial distribution,
showing how likely each possible value is.
DREW BINHAIR: Hey, that looks just like my hair!
JOE BLITZSTEIN: You can see that 6 is the most likely value,
but that doesn't mean it's especially likely.
Its probability is about 0.25.
There's about a 90% chance, though, that X will be between 4 and 8.
[MUSIC PLAYING]
DREW BINHAIR: Breaking news.
This just in.
Bimodality detected.
Day 100 of our continuous coverage of the hypertangleosis crisis
has brought us a shocking statistical development.
We go to Professor Blitzstein for more.
JOE BLITZSTEIN: A few months ago, I suggested the Binomial distribution
as a plausible model for how many patients will get cured on a given day.
Now, with much more data, it has become clear
that the Binomial model is inadequate.
DREW BINHAIR: How so?
JOE BLITZSTEIN: Here's a histogram of how many patients were cured in a day
compared with the Binomial 10, comma, 0.6 distribution.
As you can see, they look very different.
We have a case of bimodality.
There are two peaks in the distribution.
I will be working closely with Dr. Straighthair to figure out why.
DREW BINHAIR: So is everything you told us about Binomials useless?
JOE BLITZSTEIN: There were many problems where the Binomial fits well, this just
isn't one of them.
Even so, the Binomial gives us benchmarks
we can compare the real data to and serves as a building block
for various more complicated models.
DREW BINHAIR: Please, Professor, keep us updated.
[MUSIC PLAYING]
Dr. Straighthair and Professor Blitzstein
have been working tirelessly to unravel the mystery of bimodality.
They're here now live to update us on the results.
DR. STRAIGHTHAIR: Joe and I have been poring over the data,
and have found an explanation we hope will not only
resolve the statistical anomaly, but also
let us improve the percentage of patients we can treat successfully.
The conditioner comes from two different suppliers--
Normal distributors and Cauchy distributors.
JOE BLITZSTEIN: We discovered that the conditioner from Normal distributors
is far more effective than the conditioner from Cauchy distributors.
On each day, it was completely random which supplier was used.
On days when the conditioner came from Normal distributors,
the success rate was 80%.
But for Cauchy distributors, the success rate was only 40%.
The overall distribution is a mixture of Binomials, not a Binomial.
Here's the histogram of the data compared with the mixture distribution.
As you can see, the mixture fits much better than the Binomial did.
[MUSIC PLAYING]
DREW BINHAIR: We're back, live, on day 110 of our coverage
of the hypertangleosis crisis.
Cauchy distributors' stock price has plummeted in the last week,
as it has become clear that their quality control is
far inferior to their competitor.
In other news, Normal distributors has just
released a new formula of the conditioner
that patients can apply at home instead of at a clinic with limited capacity.
Professor Blitzstein is here to help us predict the ramifications.
JOE BLITZSTEIN: Hypertangleosis is rarer than you
may think, possibly because the media have been covering this 24 hours a day.
It would be much better to obtain good data on how many cases there
are as a function of time.
According to preliminary estimates, the probability
of coming down with hypertangleosis tomorrow
for someone whose hair is currently fine, is only 0.001.
The number of people in Discretown with healthy hair is currently about 20,000.
So we can expect about 20 new cases tomorrow.
DREW BINHAIR: But what is the distribution?
JOE BLITZSTEIN: I'm glad you asked.
Assuming the estimates I gave are correct,
and that people come down with hypertangleosis independently,
the number of new cases is Binomial 20,000, comma, 0.001.
That's pretty unwieldy to work with, since calculations with it involve
extremely large numbers like 20,000 factorial,
which is about 10 to the 77,337th power.
In general, the Binomial (n,p) distribution counts the number
of successes in n,p independent trials, each of which has probability, p,
of success.
If we have a Binomial (n,p) distribution, where p is small,
it turns out that we can approximate it well with a beautiful distribution
known as the Poisson.
The Poisson has one parameter, usually called lambda.
The probability of a Poisson of lambda--
random variable equaling k--
is e to the minus lambda, lambda to the k over k factorial
For any non-negative integer k.
For the problem at hand, or should I say, at hair,
we can use the Poisson of 20 as a good approximation
to the Binomial 20,000, comma, 0.001.
DREW BINHAIR: That looks very pretty, like a bell-shaped curve.
JOE BLITZSTEIN: That's another story.
DREW BINHAIR: But tell us, Professor, how many of the new cases
do you predict will be cured?
JOE BLITZSTEIN: Let's assume that the number of new cases is Poisson of 20,
and that they all try the new treatment with success probability 0.08.
Then, the expected number of new cases treated successfully
is 0.8 times 20, which is 16.
And the expected number treated unsuccessfully is 4.
The distribution of the number of successes is Poisson of 16.
And the distribution of the number of failures is Poisson of 4.
Surprisingly, these two quantities are independent,
with a Poisson number of new cases, knowing
the number of successes tells us nothing about the number of failures.
DREW BINHAIR: Thank you, Professor, for explaining all of this to us.
Any parting words?
JOE BLITZSTEIN: We need good data and good statistical thinking
in order to make good predictions.