One of the areas
that the Gates Foundation
would love to see is a kind of math tutor.
There’s a question, do you need
a big, big, big, big model to do that?
Because if you make it so big,
then our ability to know how it behaves,
it’s hard to test.
We’re hoping that one of the
more medium-size models
that mostly learn math textbooks,
and won’t have such a
broad knowledge of the world,
we’re hoping that that will let us
do quality assurance.
In academia, there’s actually
a lot of such effort going on,
but without a lot of compute.
Including my own work
that tries to develop a special model.
Usually, the smaller models
cannot win over ChatGPT in all dimensions,
but if you have a target task,
like a math tutoring,
I do believe that definitely,
not only you can close the gap
with larger models,
you can actually surpass
the larger model’s capability
by specializing on it.
This is totally doable, and I believe in it.
Certainly for something like drug discovery,
knowing English isn’t necessary.
It’s kind of weird,
these models are so big
that very few people get to probe them
or change them in some way.
And yet, in the world of Computer Science,
the majority of everything
that was ever invented
was invented in universities.
To not have this in a form
that people can play around with,
and take a hundred different approaches
to play around with,
we have to find some way to fix that,
to let universities be pushing these things,
and looking inside these things.
I couldn’t agree with you more.
It cannot be very healthy
to see this concentration of powers
so that the major AI is only
held by a few tech companies,
and nobody knows
what’s going on under the hood.
That’s just not healthy.
Especially when it is extremely likely
that there is a moderate size solution
that is open, that people can investigate
and better understand
and even better control, actually.
Because if you open it,
it’s so much easier for you to adapt it
into your custom use cases,
compared to the current way
of using GPT-4,
which all that you can do is sort of
a prompt engineering, and then hope
that it understood what you meant.
The math tutoring case seems to be the case,
where the language models have seen
a lot of educational material
already out there online.
So, that probably is, indeed,
much more around the corner,
because it has seen a lot of data.
Whereas the drug discovery,
now the challenge is for AI to come up with
something new that doesn’t exist yet.
I suspect that that’s a different type
of a challenge for AI,
because now it truly needs to reason more
in a symbolic manner
that is grounded in knowledge,
as opposed to,
‘oh, there’s a bunch of the sequences,
and let’s predict what comes next
and get lucky.’
That’s inspiring for me to think about,
the different types of challenges
and what it might take
in order to push things to the next level.
I think that’s basically the future.
I am excited to see a lot more
open-source effort,
really catching up rapidly right now,
the fact that it’s just so opaque.
Current learning is unbelievably brute force,
which I don’t think is the correct way
of doing intelligence.
There must be a better solution.
And for that, we have to open it.
In order to be able to really promote
better science around it, we need to open it.
We don’t have to open
the largest or best one, however,
because even if you open it,
it’s not like academic people
can do anything with it.
If GPT-4 is open for me,
there’s no compute for me to run all those!
I think to deal with the complexity
and the accuracy you probably want
to build these things from scratch.
I believe, with a bit more effort,
something like that could be built.
And with that wishful thought,
I’m also working toward that sort of a system
where we might have a little bit more
explainable, descriptive knowledge
that we can give to the machine
to really, truly learn and memorize.
Then when it does make mistakes,
being able to control the machine through,
‘Oh, what kind of knowledge
are you assuming for that kind of answer?,’
and being able to provide,
‘Oh, you know, your assumption
is wrong that way.
From here on, learn this knowledge.’
Those kind of problems
unlock really exciting new types
of machine learning problems,
where you need to be able to unlearn,
not just learn,
but unlearn the incorrect knowledge,
and then be able to revise over that
in the way that humans also are able to.
Whereas right now, everything,
like you said, is a bit too black box.
But I do think that with effort,
that this sort of technology could happen.
Subscribe to ”Unconfuse Me” wherever you listen to podcasts.