Today we're talking about the hot hand
in sports, particularly basketball. I grew
up watching basketball, the New York
Knicks. We knew whoever was on a shooting
streak should get the ball because the
chances are that they would make the
next shot. People are more likely to make
a basket given they've made a whole stream of them,
this was what we believed. From a math
perspective a hot hand usually means
that you have a higher chance than usual
of making a basket, given that you just
made a whole bunch of 'em in a row. Think
about it, you have some life time - see it
as an average or some average for your
game or some kind of base measurement,
and then all of a sudden you've made a
whole bunch of shots - maybe you think the
chances are that the next shot is going
to be a winner or higher than your
baseline average; because you've got
confidence, because you're doing great,
because you're throwing that ball well.
That's the hot hand. - (Brady: You're in the zone.)
Yea- oh yeah in the zone and of course
it has implications for strategy, right?
Because if a guy's on a streak and if
he's really doing better than his own
usual you're more likely to want to feed
that guy the ball. So this is a tale
with a long history; in 1985 a team of
three scholars, including famed
psychologist Amos Tversky, published a
study. Two professional teams were
involved: the Philadelphia 76ers and the
Boston Celtics; and also some Cornell
varsity and junior varsity players. And
what they did was something that
scientists do a lot, they challenged
conventional wisdom. They looked in shot
strings of hits and misses and asked the
question, from a statistical perspective.
Is there a hot hand? So what they did
specifically, imagine you have a string
with zeros representing misses and ones
representing hits, and you could look and
see after a player or a team has made a
bunch of shots is the next shot more
likely to go in than say after an
equally long string of misses? That's
exactly what they did.
Tversky was a very very big basketball
fan so I'm sure he was- brought his
passion into the
study, but I think he was looking at the
strings of zeros and ones and found no
hot hand. This paper's published; it's got
thousands of citations. And it had two
very interesting effects.
One was the non effect in the sporting
industry; those who thought they knew
better than scientists shrugged their
shoulders, this is not right - we know
there's a hot hand, we see it, we're
experienced. And then there was the field
of scholars who thought this was pretty
interesting and started doing more
studies. Different teams, different time
periods, different sports - there's a big
hot hand literature. Launched a thousand
academic ships but got a shrug of the
shoulders from professional sports. If
you search for hot hand fallacy on the
web you're gonna find tons of hits. It's
not universally agreed, because as usual
with a scientific experiment there could
be lots of different ways to do it. Maybe
you use a different data set- - (Brady: What possible different)
(data set could you use other than the ball going in the basket?)
Well you could use a different
team, you could use a different season
or you could look for streaks in a
different sport. You can do all sorts of
things. You always make a lot of choices
in experiments and it's easy to forget,
when you have a conclusion, that your
conclusion depends not just on the
hypothesis you want to test but also the
way you implemented your experiment. 30
years later two statisticians, Josh Miller
and Adam Sanjurjo, published a paper.
How did it differ from the enormous
literature on the subject in a really
important way? It challenged the original
result; not by saying maybe we need a
different experimental design, maybe we
need a different idea for what a hot
hand should be in mathematics - it said
these guys did their study wrong. So I
was really stopped in my tracks when I
saw this- - (Brady: The equivalent of saying)
(they they made an addition mistake or)
(something?) - It is like that and especially
when you consider a scholar of the
importance of Amos Tversky. So just a
little background on Tversky: he's a
psychologist, as I said in Israeli, he's a
psychologist. He's partner of Daniel
Kahneman who went on to win the Nobel
Prize for their joint work. Tversky died
of melanoma in 1996 and did not then receive
the price, because it's not awarded
posthumously. But the work was all about
how people make mistakes.
So here's Tversky, a connoisseur of human
error, and he it seems has made a mistake
or so say Sanjurjo and Miller. This gets
to the question of the most basic kind
of hypothesis testing that statisticians
do. So it always goes pretty much the
same way; there's a concept of the
ordinary and statisticians, for better or
worse, call this thing a null hypothesis
which may be off-putting to some but all
they mean is what's typical. Put a metric
of what's typical and then you look at
your observation - maybe it's a drug
effect or maybe it's whether you've just
made an unusual- your your probability of
making a shot is unusually high compared
to a notion of typical; and then you say
well is there an effect or not an
effect? That's how hypothesis testing
works. And so there's a lot of art and
science in deciding what ordinary means,
maybe not everybody agrees. But the
mistake that was found in this paper,
that Sanjurjo and Miller noticed, was
in the concept of saying what the null
hypothesis, what the ordinary was. And it
was particularly related to the fact the
data; even though mathematicians like to
think in terms of infinite strings or
infinite number of observations or laws
of large numbers, very often are dealing
in practice with small finite samples of
data and small stuff does not need to
behave like big stuff. It's pretty simple
actually; if you see three things happen
in a row you think, wow maybe a trend!
That's a small sample error, right? Maybe
the fourth one's going to be going in a
different direction; but we are very
quick to find reasons for things, to find
trends, to find patterns - we're wired this
way, okay. So we know about this, and no
one knew it better than Amos Tversky, and
yet he had a small sample of data and he
treated that small sample along with his
co-authors as though, as though it were
an infinite set. Maybe the best way to
start is with coin flipping, okay? So this
is like a toy example of the sort of
thing they were doing, coin flips could
be like hitting or missing shots: heads,
tails, hits and misses maybe. So here's
the kind of thing
we're thinking about because, after all,
we're looking at the probability of
making a shot given that you've just
made some shots or making a shot after
you've just missed some shots. So we want
to look at what's happening after
something else happens. So we could play
a game with a coin and we're gonna think
that we have a completely unbiased coin,
50% heads 50% tails, and we're going to
think that my flips of the coin are
independent. So what happens on the next
flip has nothing to do with what
happened on the previous flip or
anything else in history. We're going to
flip the coin three times and we're
thinking we're looking for what happens
after something. So here's the game: I
flip the coin and then we're going to
write down what happens after I flip
a head. So if I flip tails we don't think
about anything, then I flip the heads
we become alert, then flip something
else, we write down an H if it's a heads
or a T if it's not a head. So we're gonna
do this three times. And so the question
is what do we think before we do the
experiment is gi- given this is an
unbiased 50/50 coin and my flips are
completely independent you can ask what
do you expect to see on the piece of
paper; like half heads and half tails, why
not? Because being cued by a head
shouldn't have any effect on the next, on
the next flip. But in fact it did. I can
write down all the outcomes, there's only
eight of them; so I think I got all eight
of them. And let's look at what we would
have written down if we were keeping
track of what happened after heads. So
TTT is a non-starter. TTH is also a
non-starter because even though you
got a heads here we don't know what
happened afterwards, because we were only
looking at streaks of length- total
length three. Okay, so TTH we don't write
anything down. So you could already see
something funny's going wrong because
out of my eight cases two of them have
already been problematic. The next guy is
not problematic because here's an H
right in the middle, so you flip the T
you don't get do anything
flip the H it's right in the middle - you're
alert. Then you get a T so you write down
a T. In this next string HTT; H starts
off right away, gets your attention. You
write down a T. Here in this string you
start with the T, don't write anything.
Then we see an H, we become alert, we get
an H we write it down. Okay and then
we're finished; this H would have
queued us if we were doing strings of
length four but we aren't. Here HTH so
the first H tells us to write down the T,
the second T tells us to do nothing then
we end with an H which is kind of
dangling. Here HHT. So the first H
tells us to write down what happens
after, which is an H. Second H tells us to
write down what happens which is a T, we
don't do anything when we see a T. And
here finally we have HHH so the first
two H's each tell us to write an H, and
we don't get to do anything after the
third. In the first two cases we write
nothing in the last six cases we write
something; so let's see what fraction had
heads. So in the first one the fraction
is zero; and in the second one the
fraction is zero; and the third one the
fraction is one. Again we have a zero.
Here we have a heads and a tails, half
the time we saw heads half the time we
saw tails - that's 0.5. And here a hundred
percent of the time we saw heads again
so we have a one. So we take these
numbers, their average represents
expectation. So I add them up I get 1
plus 1/2 plus 2, that's 2.5. The average
is equal to 2.5 over 6, which is
definitely less than 1/2. 6 is the number
of examples that I'm averaging over - so
don't forget we're asked, what do you see
what happens given that you have a head?
And so here we never- the first one we never
had a head at all; in the second one we
had a head at the end but we don't know
what happened given that we had that
head so those things are out of the
sample. This is exactly how you would
have treated a sample with heads being
make a shot and tails being miss a shot,
if you were doing this for basketball
strings. So if we were happening to look
at only three shots, bringing this back
to basketball, we would have seen that
there was a bias downward from the 1/2
that we would have expected to see given
the fact that the coin is 50/50 and that
the flips are all totally independent.
Now importantly if we looked at longer
strings that average would have tended
toward 1/2. And if we looked at very long
strings, even not all the way infinite, it
might have been so close to 1/2 that
this difference didn't matter. But in
fact what we're seeing is that in data,
in this finite data set, the reversals
were more likely than the continuation;
and that is a finite data a small sample
effect and this was not considered when
the original hot hand scholars made up
their sense of what was ordinary. What-
we can now ask, what happens when we look
at really high-end basketball players of
today? Take the original experiment but
put in this correction, allow for the
fact that we're looking at finite data
sets, small samples; which effectively
lowers the bar from 1/2 or whatever is
the right number down to what ordinary
looks like in a data set and see if we
do find hot hands. So this is an
experiment that we did. So the way we did
it, I'll tell you a little bit about our
experiment; is that we took some
basketball players, I'll tell you about
them in a second some well-known
basketball players, over the 2016-2017
season and playoff game by game, and we
looked their strings of hits and misses;
and we asked the question in each game,
were they having a hot hand or not? Were
they more likely to make a shot after a
string of shots than they were after a
string of misses? That's the question
that we asked. Following the original
researchers we have to set up a concept
of ordinary - no effect. And then look at
the observed string and see if it's
different from ordinary in an important
way. - (Brady: So if Steph Curry gets 10 shots)
(in a row there's a chance that was just luck,)
(he just, you know, if he was a 50/50)
(player he'd get 10 shots in a row sometimes)
Sometimes, right? Could be like
somebody's got to be on top, right, is
that luck or skill? It's exactly the same
sort of question and it's great to see that
people in England know about Steph Curry.
He's one of our star players, one of the
featured players of the Golden State
Warriors, which is the team
we looked at and also one of the
featured players in this hot hand study
that we did, we looked at two Warriors. So
the Warriors were the champions, NBA
champions, last season and we looked at
two actually three of their star players:
Steph Curry and Klay Thompson, who are
known as the "Splash Brothers" for very
good reason.
Klay is- Thompson's probably the
streakiest of the sh- the shooters; and
then we also looked at Kevin Durant who
is undeniably the heart and soul of the
championship win. So if you take a string
for the player, it's got some number of
streaks in it,
suppose the player takes 20 shots and
makes 11 of them or something, right; and
it's got some zeros and some ones in it
this is- we did this per game. And then
what we did was we looked to see in that
string what was the probability of a hit
following two hits, and a hit following
two misses; but to see if there was any
difference in that string. And then we
took that string and we scrambled it, did
a random permutation, and made the same
calculation. Think about it,
scrambling that thing ruins the order.
All- no, no streaks - I mean whatever
streaks emerge emerge by chance, even
though you've got the same fraction of
hits and misses. And we did that
thousands of times. And each time we
scrambled it we looked to see what
happened, so we're kind of mimicking
randomness. And we looked at all those
outcomes over the scrambled string and
then asked whether the observed string
was unusual; could have been in the highest
corner of hotness compared to all of the
experiments we did. And you can see that
there's no asymptotics, there's no 'what
happens if I had a longer string?' That
doesn't even come into the discussion,
which is why this type of experiment
which is more and more commonplace in-
all throughout science, called a
permutation test. This experiment
automatically accounts for the size of
the sample.
[Cheers]
So on December 6th 2016 Klay Thompson,
one of the Splash Brothers, one of our
featured players, scored 60 points in a
game. So if you don't know basketball,
typical score in a game is like a
hundred points or 110 points; he scored
more than half of them all by himself.
The Warriors won that game by a big
amount. And here is the sequence of ones
and zeroes, one being a shot he made and
zero being a shot he missed. He hit 31 of
his 44 shots, we just strung together the
free throws and the field goals, the two
point and the three-point shots - we didn't
distinguish in this sequence. And the
question is, did he have a hot hand?
Now everybody in the audience thought he
had a hot hand; I thought he had a hot
hand - it was obvious that he had a hot
hand. But if you go to our permutation
test where we scrambled that string,
which after all does have a lot of ones
in it, and look at what ordinary is - this
odd shape is our null hypothesis, it's
our concept of ordinary. Unusually high
streakiness is this red zone over here,
and Klay's observed strength was down
there. So according to this measure of
hot hand he didn't have one. - (Brady: so from all)
(the ways he could have shot 31 from 44,)
(in terms of hot streaks or cold streaks,)
(or just like normal-ness;)
(he was actually at the colder end of)
(things?) - He was, he was indeed. And so this
is what we saw game after game. Of the 99
games the Warriors played; Curry played
96, Thompson played 95. We looked whether
there was a difference between what
happened after making one shot and
missing one shot, that's what we call our
conditioning set depth. Making two shots
and missing two shots, making three shots
and missing three shots - we looked in how
many games did we see an unusual effect.
And so for Curry, if we looked what
happened after one shot we saw hotness
in 7 out of 96 games. If we looked at
what happened after two shots, 2 of 96;
and after 3 shots in 3 of 96. We turned
to looking at the team as a whole and
looked quarter by quarter. So these stars
shoot like 20-25 shots in a game - the team
shoots like 20-25 shots
in a quarter. And so it was a kind of a
comparable length, and we found that of
the 396 quarters that were in the 99
games, again we did not see a high
fraction of these, so different from 5%,
showing significance. Here I've used a
5% significance level, which is
fairly common. What we saw is that while
there was an occasional bout of hotness,
not at all outside of what can be
explained by mere chance. It's pretty
consistent. When we add superstar Kevin
Durant; he played fewer games than the
others because he was injured for a long
stretch, and showed no particularly
unusual hotness either - only rarely. - (Brady: I walk away from your office)
(today now thinking there's no such thing as a)
(hot hand, being on fire - being in the zone)
(on the basketball court is a fallacy.)
Well coming back to what we were saying
right at the very beginning: a study is
both a test of what you're trying to
test plus your own implementation of it.
So this- we've looked at many variants on
this, but we haven't looked at all of
them. I'm pretty sure there's no hot hand
in these data - if you formulate hot hand
as we have formulated it. But there are
other formulations and one really
interesting source of information, that
we haven't paid attention to here, is timing.
Maybe when you're hot you're doing
things faster. We've just looked at
strings, we haven't looked at how quickly
they've occurred. Maybe we've missed a
piece of data like that that's really
important for understanding a true
phenomenon hot handedness. But another
explanation, which- for which there is
mounting evidence and which could also
be true, is that people are not very good
at understanding randomness. That there's
much more chance out there than we think
there is and while we are seeking for
patterns and explanations, as we look
backward, we're not giving a fair shot to
the explanation that this was just
really a random event. [Extras] ...especially like
the true believer in the law of small
numbers commits his multitude of sins
against the logic of statistical
inference in good faith.