Hello, my name is Burr Settles. I'm a Research Director at Duolingo
and I'm very excited to talk to you
about some of the ways we're using Artificial Intelligence
to improve the ways that we teach through Duolingo.
Now, to start things off, I want to talk a bit about our mission,
which is to develop the best education in the world
and make it universally available.
And we're not kidding when we say universally available.
I've been with Duolingo for over 8 years and from day one
we were committed to building a free app
that does not put educational content behind any kind of paywall.
That's also why we use the Internet and mobile devices
to deliver our lessons to as many people as possible.
But we care equally about it being the best education.
Now, to a lot of people, the best education comes from having
a private one-on-one tutor or a really great teacher.
And while that may be true in many cases,
unfortunately not everybody in the world has access to a one-on-one private tutor
or a great teacher.
We believe that Artificial Intelligence is the best way
to connect the dots and really achieve our mission.
And to be clear, we're not interested in using A.I. to replace great teachers.
On the contrary, through platforms like Duolingo for Schools,
we wanna use technology to enable teachers to do an even better job in the classroom.
But until everybody in the world has access to a really great teacher,
we think A.I. is the best way to scale that kind of experience
to as many people as possible.
Now, when you think about a really great teacher,
I claim that they have three properties.
One, is that they know the material really well.
Great teachers are domain experts in their chosen field.
Number two, they know how to make that material engaging,
exciting and interesting, and motivate you to keep learning.
And number three, and perhaps more importantly,
they are able to get inside your head.
So great teachers,
because of the one on one time they spend with you,
and the time they spent with hundreds of other students in the past,
they know how to understand
the things that you know and don't know,
things that come easily, and things you struggle with,
things you've probably forgotten by now and it's time to review.
So, what we've done in Duolingo is we've taken each of these three properties
and turned them into research programs for our A.I. efforts.
So, for example, when it comes to the material,
we use machine learning and natural language processing
to create tools that assist our world class content developers
in auditing and improving our courses.
So, this can be anything
from analyzing the vocabulary and grammar content of our lessons,
to looking at scripts for Duolingo Stories or Duolingo Podcast or audio lessons
to make sure that we're hitting a target level of proficiency
be it beginner, intermediate or advanced.
When it comes to making the material engaging,
we're using state-of-the-art speech technologies
to create custom voices
for our Duolingo world characters, which is super cool.
And you're also gonna be hearing more about that today.
And believe it or not, whenever you get a push notification
to remind you to do your daily Duolingo lesson,
it turns out there's an A.I. algorithm
that chose which, out of hundreds of possible messages
we could send you at this precise moment,
to maximize the chances that you want to go and do a lesson on Duolingo.
Now, those are just a few examples, but I'm gonna spend most of our time today
talking about how we use A.I. to get inside your head
and personalize your learning experience.
And I'm gonna focus on a new system we're really excited about
that we affectionately called BirdBrain.
Before I tell you what BirdBrain is,
let me tell you a little bit about why we built it.
In educational psychology,
there's a concept known as the "zone of proximal development"
and it goes something like this:
Whenever you're learning a new skill or ability,
there's this sphere of things that you understand,
that come easily, that you've already mastered,
and that you can do without receiving any help.
Now, just outside of that sphere is "the zone of proximal development."
Now, this is the frontier of what you're able to do.
You can still maybe do those tasks, learn those things,
but you might need extra time or extra help.
And beyond that, there are things that you're just not ready for,
they're too hard now.
There's a lot of evidence to suggest that you're most challenged and motivated
when you learn at the edge of your abilities,
or if you spend most of your time here in the zone of proximal development.
What that means for Duolingo is that whether you realize it or not,
when you go into a lesson,
there are hundreds or thousands of possible exercises
that we can try to cram into that five minutes chunk of time.
And so we want to maximize that time by filling it with lessons that are
in your zone of proximal development for you personally.
Things that are not too easy or not too hard
but are just right to keep you challenged and motivated.
So, you might be wondering how can A.I. help get you "in the zone"?
That's where BirdBrain comes in. And here is how it works.
So, for every learner and every exercise,
BirdBrain looks at these two things and makes a prediction
that this learner has an 81% chance,
for example, of getting this exercise correct.
Now, some other exercise that might be more basic,
it'll predict 98%, like this is super easy.
But other more challenging exercises,
BirdBrain might make a prediction like 45%.
This is probably something that it's too soon
to try to put into your lessons.
But that's all for the same learner.
Some other learner who is further along on their language learning journey,
might find this particular exercise a little bit easier, like 79%,
and this might be in or closer
to their personal zone of proximal development.
So, BirdBrain is able to do this because our algorithms learn
to make accurate predictions from more than 5 billion exercises
every single week.
And this works because each of our individual learners
have their own unique profile of the sorts of things
they tend to get right and tend to get wrong.
So, think about this.
The next time you go and do a lesson on Duolingo,
you're not only learning something new for yourself,
but you're teaching BirdBrain a bit about your personal abilities,
as well as the difficulties of all of the exercises we give you.
And this, in turn, helps us to improve and personalize the learning experience
for hundreds of millions of learners all around the world.
It's very exciting for us.
Now, you might be wondering how do we know it's effective?
One way we can do this
is to look whether or not BirdBrain is making accurate predictions.
Now, this chart shows the prediction quality of BirdBrain
over the first six months or so
after we built it at the beginning of last year.
This particular chart is specifically
for the course teaching English to Spanish speakers.
But we got almost identical results for all of our courses.
It's important to note that the prediction quality is high,
but it's also increasing and improving over time
as more learners use the platform.
Another thing that we do for almost all of our product changes
is A/B testing.
We take all of our learners
and split them up randomly into two different groups.
And in the case of BirdBrain, one half of those learners
get lessons that are constructed using pre-existing heuristics
that we've developed over the course of years.
This is the control group.
And for the other half of our learners,
they get custom personalized lessons using BirdBrain.
This is the treatment group.
And we compare learning outcomes between these two groups
and see if BirdBrain is having a significant impact.
So, here are the results from one such A/B test
where we used BirdBrain to personalize level 1 lessons.
And here we saw a 3.5% increase in content length.
So, what this means is the sentences
that you needed to translate or transcribe as part of the lessons,
were on average 3.5% longer
and this is one of the many measures that we look at
of how difficult the lessons are.
Now, not only were they more challenging but also more motivating
because we saw a 6.3% increase in the time spent learning.
So these learners were spending more time in the app.
And we saw these same results again
when we used BirdBrain to try to improve Skill Practice.
We saw an almost 9% increase in content length
and a 3% increase in time spent learning.
This is a pattern that we saw over and over again.
In this chart,
each dot represents a different A/B test that we ran
trying to use BirdBrain to improve the learning experience in some way.
And as you can see, in most cases we were able to increase
both content length and time spent learning.
And in fact, there's a positive statistically significant relationship
between these two.
Meaning that if BirdBrain succeeds in making the lessons more challenging,
it's also likely to succeed in making it more motivating by a proportional amount.
Now, this may seem obvious, or even easy,
but it turns out it's really not.
So, let's compare these green dots to these blue dots
that represent A/B tests that were run using
more conventional product development and software engineering techniques.
For these other non-BirdBrain A/B tests, there's almost no relationship
between content length and time spent learning.
And a lot fewer of those experiments succeeded in improving both.
But BirdBrain, because it takes into account
both the difficulty of the exercises
and where you're at on your own personal learning journey,
and puts the two together, is able to strike a good balance
and balance the tensions between these two extremes.
And improve the language learning experience.
Now, today BirdBrain is used in some way
for almost all session types that are in the app.
And it's used in particular to choose the really difficult exercises
that go into hard mode practice for extra XP
or for the Legendary skills that you'll also be hearing about today.
And of course, we're working to make BirdBrain even better.
We have ongoing research to make it more accurate,
more linguistically detailed,
so not only is there a 90% chance that you'll get this right
but if you get it wrong, it's probably because of this tricky noun,
adjective, gender agreement, for example.
And all of this makes BirdBrain more useful.
So… if you're excited or interested to learn more
about the ways we're using A.I., machine learning
natural language processing, etc. here at Duolingo,
I encourage you to check duolingo.ai. Thank you very much.
And enjoy the rest of Duocon.