- [Voiceover] As a teacher thinking about using a tool
inside or outside of your classroom,
the first natural question is,
well does that tool work?
And that's also very important to us
here at Khan Academy with our mission.
We don't want just people to use it.
We want to feel good that that usage
is actually driving real learning.
And so what I'm going to give an overview of in this video,
is a very large study conducted in the state of Idaho.
It was across the entire state,
understanding whether Khan Academy is driving real learning.
And this is the largest study that has ever been done
on a personalized learning tool.
And so I'm going to summarize this.
We'll also give a link to the study,
later on in this tutorial.
It was done and funded in conjunction with
the J.A. and Kathryn Albertson Family Foundation.
And so I'm just going to summarize the report here.
So just as a big picture,
we'll go right down here.
There's a lot of really cool stories from teachers
and educators and district officials here
talking about how they implemented the pilot.
But the big picture,
this was 47 schools, 33 districts, 173 teachers,
several thousand students throughout the state of Idaho.
And what they did in the study,
is the students took the NWEA map exam
at the beginning of the year.
They worked on Khan Academy,
and then they took the NWEA map exam at the end of the year,
and the way that efficacy was measured
is well the map exam can see for any student,
whether they are a student who is more advanced
or a student who is re-mediating,
they can see what their expected growth
is in a traditional model
and then they can compare that expected growth
to what the actual growth was using Khan Academy.
And so let's just cut to the chase
and see what the results are.
There's a lot of videos that we have on our educator portal
that talks about how the different teachers
use Khan Academy and what they found valuable.
But I'll go straight to the data here
for the sake of this video.
And so if we go down here,
this is just the distribution of students by grade level.
You can see kind of late elementary and middle schools
were the bulk of the students in the study were.
But even in third grade,
we're talking about over 200 students.
So this is a very large scale study
and this first one says mission completion
versus spring percentile.
Mission completion,
these are the grade level personalized learning experiences
that the students can pick.
So there is a third grade mission.
There's actually an early learning mission.
There's a third, fourth, fifth,
all the way up to Algebra, Geometry, Trigonometry,
there's even a calculus mission.
And these are aligned to the important state standards,
especially the common core standards
for K through pre-calculus.
And they're aligned to the AP standards for calculus.
And so what you see here is,
depending on how much a student completed of their mission,
over the course of the pilot,
it correlates very strongly with
how they performed on the map exam
during the spring assessment.
Now you might say, well of course, naturally,
a student who completes more,
well that's maybe a student
who's a little bit more advanced,
and so they're going to be in a higher percentile.
And I would agree with you.
That makes sense.
So this is, this right here, is not evidence of,
or clear evidence of efficacy just yet,
but you definitely wanna see this trend.
If for whatever reason this was a flat line,
if it had a downward slope for some reason,
then you would have an issue.
But this is the direction that you wanna see,
but we wanna see more evidence.
We wanna see for a given student,
regardless of where they started,
that usage of Khan Academy is not
driving just expected learning.
For all we know, this might be just
what you would expect for different students.
It's driving learning above and beyond the expectations.
And so that's where we see this data down here.
And they boiled it out,
students who completed zero to 10%
of their mission on average, grew as expected.
These students barely used Khan Academy.
So once again, grew as expected,
this means if you are an advanced student
and you are learning at a fairly quick pace,
well you grew as expected,
you grew at that quick pace.
And if you are a student who took a little bit more time.
Well you also grew as expected
and so when you didn't use Khan Academy,
just what you would expect happened.
But then when you see students who
had significant usage of Khan Academy,
students who completed 40% of more
of their mission on average grew 1.5 times more
than their expected growth in one year.
Students who completed 60% or more of their mission
grew an average 1.8 times their expected growth.
So they almost had two years of growth in one year.
And once again, I want to be very clear,
this is, what we're plotting on the vertical axis here.
This isn't just growth,
this is average actual growth divided by target growth.
So in a normal model,
if you just look at the map data on a nationwide basis,
whether a student is strong or weak,
if Khan Academy had no effect
you would expect just a flat line right over here.
You would expect everything here to be just at one.
If Khan Academy was actually less effective
than whatever students were doing before,
then you would say well these students
who are spending a lot of time on Khan Academy,
then it would be taking away from those,
so you would see a downward slope.
But over here we see that the students
who are actually using Khan Academy deeply,
and you can imagine any large scale studies like this
you have variation, and the various fidelity
to which folks are using Khan Academy,
how integrated it is in the classroom.
But the classrooms that are using it more deeply,
more regularly, are not just seeing expected growth,
they're seeing 1.8 times.
They're seeing a significant multiple
of what you would expect their growth to be.
So this is nearly two years of growth
in one year of learning time.
And that's once again, this is 60% plus,
so you could only imagine what it would be
if students complete 70%, 80%, 90% of the mission
or if it's used in an even deeper way
in different classrooms.