Welcome to Unconfuse Me. I’m Bill Gates.
Let’s talk about AI.
You and I have both been lucky enough
over the last six months or so to have engagement
with the mix of both Microsoft and OpenAI
and have early access.
I remember some of the best examples
of how to get the AI to do fun things,
where, when you came up and saw me
and we were brainstorming about this,
you were the one who said,
"Hey, if you tell it to write a speech like various politicians,
including Trump and others," that it’s stunning
how it captures the voice of different people.
Give us how you first were using the AI,
and then this is super timely because
you’ve just recently come out with the Khanmigo,
if I’m saying that right.
That’s right, very good.
The OpenAI folks reached out, and they said,
"Hey, we’re a couple of weeks away
from having the first training of our model."
They wanted to reach out
to Khan Academy for two reasons.
One was they said we want to
make this really good at AP Biology,
and I only found out later –
I don’t know if this is true –
they told me you gave them the challenge.
That’s right. In June, they kept showing me this thing,
and I was like, "Yeah, it’s kind of an idiot savant.
I don’t think it’s practical.
Why don’t you see if it can do the AP Biology exam.
I’m not going to pay any attention
until you can get a 5."
I thought, "Okay, that’ll give me three years
to work on HIV and malaria."
Then it was so bizarre because
Sam Altman and Greg Brockman in late August said,
"Hey, we want to come show you this thing."
It was early September when
there were like thirty people at my house.
I’ve said it’s the most stunning demo
I’ve ever seen in my life.
I mean, right up there
with seeing the Xerox PARC graphics user interface
that set the agenda for Microsoft for about fifteen years.
This demo was so surprising to me,
the emergent depth that
as they scaled up the training set,
its fluency, and you have to say understanding,
that computers could not read
in the sense humans do,
and it couldn’t write in the sense humans do.
Now, with lots of footnotes
about hallucination and things like that,
but I’m still personally in a state of shock at
wow, it is so good, and okay, therefore,
let’s see where we can put it to good use.
Yes, 100%. So thank you
for that challenge to them.
I think that helped.
They came to us because we have a large library
of AP Biology questions, etc.
They’re like, "Hey, can we use that
to either evaluate or train?"
At the time I was like,
"Well, what’s in it for Khan Academy?"
You know, I’m in it.
They’re like, "Well, maybe you get access to the model."
I, too, was skeptical.
I had seen GPT-3 at that point,
and it was cool,
but I don’t see how we were going to apply it.
Two weeks later,
they showed us the AP Bio question and said,
"So Sal, what’s the answer to this?"
I read it. I was like,
"Okay, I think it’s C." It said, “the answer is C,"
and I was like, "Oh, that’s interesting."
I started getting a few goosebumps.
Then I said,
"Well, ask it why that’s the answer." It explained it.
Oh, yes, it’s so good at that!
I think what folks need to realize –
because everyone had that moment with ChatGPT,
but this was like that, but more,
because GPT-4 is even better.
Way better.
I said, "Can you say why
the other answers aren’t correct?"
Did it. Then I said –I’m almost shaking – I said,
"Can you write ten more questions like this?"
Bam, bam, bam.
The first ten I saw were all pretty good.
I’m like, "Yep, yep, yep, yep. That’s all legit."
Then the implications for Khan Academy
started to go through my mind.
Then we did start to get into
some of its hallucinations
and some of its math errors in those early days.
But that weekend, they gave me
and our chief learning officer access on Slack,
and I couldn’t sleep. I was having these
‘in the rabbit hole’ conversations with it.
Then we had a hackathon for our team;
we got about 40 people on our team under NDA,
and we said “just come up with stuff.”
We were having the debates
that everyone was having around,
well, the information is not 100% airtight.
The math isn’t great right now.
The costs are not trivial. It can introduce bias.
What’s the safety?
The use of people’s information, etc.?
But then we were starting to get it to work well
as something that helps you answer questions
while you’re watching a video,
to work well as a tutor.
Every ten minutes we thought about it like,
wait, it could also do this.
It could also do that. It can also do that.
And so we said, "Well, what if we could make it
so you could talk to any historical character?
What if you could make it
so it gets into a debate with you?
What if you could make it
so it doesn’t write your paper,
but it writes it with you?
What if it could do lesson plans for teachers?"
It could be the end of static curricula.
The imagination kept going.
By December, January we had our team
in full, rapid prototyping mode.
Just recently we launched Khanmigo.
So far we’re starting to titrate it out to the world,
giving people access to it.
But the feedback is very promising.
What we’re hearing overwhelmingly
from social media and the press
is that they’re really happy that we’ve engaged in this
and that we’re taking a safe approach
where parents and teachers can monitor it.
We have a moderation filter.
OpenAI has also gone through great pains
to make sure that things stay appropriate.
You and I have talked extensively
about the math issue,
and we’ve done some things
that I think make it quite robust
on top of the things that the model does.
The costs are coming down dramatically.
I mean, that’s really impressive.
Even ChatGPT isn’t bad,
and then GPT-4 is dramatically better.
It makes mistakes, to be clear.
They both make mistakes.
But one of the things we realized is
when you do math, especially if I’m tutoring you,
let’s say you do some work,
I don’t just immediately say, "Correct, incorrect."
I say, "Well, let me see this."
Okay. Let me see what he did. Okay.
“Yeah, yeah. Good job.”
And one of the hacks – I don’t think it’s actually a hack,
I think it’s a principle we’re doing –
which is we weren’t getting good results
when we just asked it to decide,
when it acts as a tutor,
whether a student is right or wrong.
But as soon as we said,
you know what, construct your thought,
and these thoughts are private to you,
write that down first, and then evaluate
the student’s response to your thoughts.
And then say something publicly to the student,
then the math improved dramatically.
So it is funny.
As soon as one day you think like,
"Oh, this is so not like a human being,”
and then the next year it’s like,
wow, that’s kind of how we operate.
We kind of need that thought before we can talk.
The way I think of it is,
it’s like a human
that’s not very good with context,
that the math context we know
is a special context of ‘check your answer.’
It’s also sometimes, even in conversation,
if you get it into a mode where it’s telling jokes,
with humans, we have kind of a look,
or the new question is quite different.
It thinks it’s supposed to just keep telling jokes,
and you almost have to do a reset to get it out of this
“Hey, everything is a joke”-type mode.
So it’s like kind of a very naive person
in terms of all these different contexts we’re in.
My favorite one is where it tries to do Sudoku,
which it can’t do.
And you point out,
"Hey, that’s not a good solution.”
And it says, "Oh, I must have mistyped,”
because it’s like,
where is the typewriter? You have a typewriter?
I like how it gets defensive.
It sees how humans deal with being accused,
“My dog ate my homework,”
Subscribe to Unconfuse Me wherever you listen to podcasts.