My guest today is Sam Altman.
He, of course, is the CEO of OpenAI.
He’s been an entrepreneur and a leader
in the tech industry for a long time,
including running Y Combinator,
that did amazing things
like funding Reddit, Dropbox, Airbnb.
Sam is also involved in companies
that could help solve climate like Helion and Oklo,
so a broad range of activity.
Today we’re going to focus mostly on AI,
because it’s such an exciting thing,
and people are also concerned.
Welcome, Sam.
Thank you so much for having me.
I was privileged to see your work as it evolved,
and I was very skeptical.
I didn’t expect ChatGPT to get so good.
It blows my mind,
and we don’t really understand the encoding.
We know the numbers,
we can watch it multiply,
but the idea of where is Shakespearean encoded?
Do you think we’ll gain
an understanding of the representation?
A hundred percent.
Trying to do this in a human brain is very hard.
You could say it’s a similar problem,
which is there are these neurons,
they’re connected.
The connections are moving
and we’re not going to slice up your brain
and watch how it’s evolving,
but this we can perfectly x-ray.
There has been some very good
work on interpretability,
and I think there will be more over time.
I think we will be able to understand these networks,
but our current understanding is low.
The little bits we do understand have,
as you’d expect,
been very helpful in improving these things.
We’re all motivated to really understand them,
scientific curiosity aside,
but the scale of these is so vast.
We also could say,
where in your brain is Shakespeare encoded,
and how is that represented?
We don’t know.
We don’t really know,
but it somehow feels even less satisfying
to say we don’t know yet
in these masses of numbers
that we’re supposed to be able to
perfectly x-ray and watch
and do any tests we want to on.
I’m pretty sure, within the next five years,
we’ll understand it.
In terms of both training efficiency and accuracy,
that understanding would let us do
far better than we’re able to do today.
A hundred percent.
You see this in a lot of the history of technology
where someone makes an empirical discovery.
They have no idea what’s going on,
but it clearly works.
Then, as the scientific understanding deepens,
they can make it so much better.
Yes, in physics, biology,
it’s sometimes just messing around,
and it’s like, whoa –
how does this actually come together?
In our case, the guy that built GPT-1
sort of did it off by himself and solved this,
and it was somewhat impressive,
but no deep understanding
of how it worked or why it worked.
Then we got the scaling laws.
We could predict how much better
it was going to be. That was why,
when we told you we could do a demo,
we were pretty confident it was going to work.
We hadn’t trained the model,
but we were pretty confident.
That has led us to a bunch of attempts
and better and better scientific understanding
of what’s going on.
But it really came from a place
of empirical results first.
When you look at the next two years,
what do you think some of
the key milestones will be?
Multimodality will definitely be important.
Which means speech in, speech out?
Speech in, speech out. Images. Eventually video.
Clearly, people really want that.
We’ve launched images and audio, and it had
a much stronger response than we expected.
We’ll be able to push that much further,
but maybe the most important areas of progress
will be around reasoning ability.
Right now, GPT-4 can reason
in only extremely limited ways.
Also reliability.
If you ask GPT-4 most questions 10,000 times,
one of those 10,000 is probably pretty good,
but it doesn’t always know which one,
and you’d like to get the best response
of 10,000 each time, and so
that increase in reliability will be important.
Customizability and personalization
will also be very important.
People want very different things out of GPT-4:
different styles, different sets of assumptions.
We’ll make all that possible,
and then also the ability
to have it use your own data.
The ability to know about you,
your email, your calendar,
how you like appointments booked,
connected to other outside data sources,
all of that.
Those will be some of the most important
areas of improvement.
In the basic algorithm right now,
it’s just feed forward, multiply,
and so to generate every new word,
it’s essentially doing the same thing.
I’ll be interested if you ever get to the point where,
like in solving a complex math equation,
you might have to apply transformations
an arbitrary number of times,
that the control logic for the reasoning
may have to be quite a bit more complex
than just what we do today.
At a minimum, it seems like we need
some sort of adaptive compute.
Right now, we spend the same amount of compute
on each token, a dumb one,
or figuring out some complicated math.
Yes, when we say, "Do the Riemann hypothesis …"
That deserves a lot of compute.
It’s the same compute as saying, "The."
Right, so at a minimum,
we’ve got to get that to work.
We may need much more
sophisticated things beyond it.
Subscribe to “Unconfuse Me” wherever you listen to podcasts.