If you look at what's happening, really we are seeing
the confluence of three trends coming together.
First is the availability of large amounts of unlabeled data
from the Internet for unsupervised training.
The second important trend, is the evolution of model architectures towards Transformers.
The third part is the emergence of domain specialization in hardware.
I recently sat down with Sudipta Sengupta and Dan Roth,
both of whom are Distinguished Scientists at AWS.
I wanted to speak with them about architectures and models and learn more about the role they
play in the seemingly sudden popularity of Generative AI.
Sudpita and Dan have a long history and deep knowledge of large language models
and the need for custom hardware to support the growth and development in this space.
I was curious how they see traditional architectures evolving and what they
think might be empowering this growth.
Dan, Sudipta, thank you for taking time to meet with me today and talk about this
magical area of generative AI. You both are distinguished scientists at Amazon.
How did you get into this role? Because it's a quite unique role.
All my career has been in academia, so I've been for about 20 years.
I was a professor at University of Illinois in Urbana Champagne, then the last five,
six years at University of Pennsylvania doing work in a wide range of topics in AI,
machine learning, reasoning, and natural language processing.
Sudipta-
I was before this at Microsoft Research, and before that at Bell Labs. And one of the
best things I liked in my previous research career was not just doing the research, but getting it
into products, kind of understanding the end to end pipeline from conception to production
and meeting customer needs. So when I joined Amazon and AWS, I kind of doubled down on that.
So if you look at your space and this whole generative AI seems to have just
come around the corner out of nowhere, but I don't think that's the case, is it?
I mean, you've been working on this for quite a while already.
It's a process that, in fact, has been going for 30-40 years.
In fact, if you look at the progress in machine learning, and maybe even more significantly in
the context of natural language processing and representation of natural languages, say in the
last ten years, and more rapidly, let's say in the last five years since Transformers came up.
But a lot of the building blocks actually were there ten years ago.
And some of the key ideas actually earlier,
only that we didn't have the architecture to support this one.
Really we are seeing the confluence of three trends coming together.
First is the availability of large amounts of unlabeled
data from the Internet for unsupervised training.
The models get a lot of their basic capabilities from this unsupervised training.
Examples are like basic grammar, language understanding, and knowledge about facts.
The second important trend is the evolution of model architectures towards Transformers,
where they can take input context into
account and dynamically attend to different parts of the input.
The third part is the emergence of domain specialization in hardware,
where you can exploit the computation structure of deep learning to keep riding on Moore's Law.
Transformer-based neural network architectures have led to the rise of large-scale models
that contain hundreds of billions of parameters.
A parameter is a configuration variable internal to the model
whose value can be adjusted to optimize the model's performance
For example, in a neural network, the connections between neurons
are defined by parameters such as weights and biases,
which can be tuned during training in order to improve the network's accuracy.
During training, the parameters are updated iteratively in order to minimize the error
between the network's predictions and the actual values,
and to improve the model's performance on the given task.
To give you a sense of scale, model sizes have increased from 100s of millions
to 100s of billions in just a few years.
I wanted to understand what that really means, and where we are headed next.
Parameters is just one part of the story. It's not just about number of parameters, but also training
data and volume and the training methodology. So you can think about increasing parameters
as kind of increasing the representational capacity of the model to learn from the data.
So as this learning capacity increases, you need to satisfy it with diverse high quality
and large volume of data. In fact, in the community today, there is understanding of
empirical scaling laws that predict the optimal combinations of model size and data volume to
maximize accuracy for a given compute budget. We have these models that are based on billions
of parameters. Yeah, and the corpus is the complete data on the internet
and customers can fine tune this by adding just a few hundred examples of what it is.
How is that possible? That it's only a few hundred that
are needed to actually create a new task model?
If all you care about is one task, if you want to do text classification or
sentiment analysis and you don't care about anything else, it's still better perhaps to
just stay with the old machine learning with strong models but annotate data.
The model is going to be small, no latency, less cost.
But AWS has a lot of models like this that solve specific problems very very well.
Now, if you want models that you can actually very easily move from one task to another that
are capable of performing multiple tasks, then these abilities of foundational model come in.
Because these models kind of know language in a sense, right?
So they know how to generate sentences,
they have an understanding of what comes next in a given sentence.
And now if you want to specialize it to text
classification or to sentiment analysis or to question answering or summarization,
you need to give it supervised data, annotated data and fine tune on this.
And basically it kind of massages the space of the function that we
are using for prediction in the right way and hundreds of examples are often sufficient.
So the fine tuning is basically supervised.
Supervised, yeah.
So you combine supervised and unsupervised learning in the same bucket.
Basically, again, this is very well aligned with our understanding in the cognitive sciences of
early childhood development that the kids, babies, toddlers learn really well just
by observation who is speaking, pointing correlating with spoken speech and so on.
So a lot of this unsupervised learning is going on and kind of quote unquote,
free unlabeled data that's available in vast amounts on the Internet.
One component that I want to add that really led to
this breakthrough is the issue of representation.
So if you think about how to represent words,
it used to be in old machine learning that words for us were discrete objects.
So you open a dictionary, you see words and they are listed this way.
So there is a “table” and there's a “desk” somewhere there,
and there are completely different things.
So what happened about ten years ago is that we moved completely to continuous representation of
words, where the idea is that we represent words as vectors, dense vectors, where similar words,
semantically, are represented very close to each other in this space.
So now table and desk are next to each other. So that's the first step that allows us to
actually move to more semantic representation of words and then sentences and larger units.
So that's kind of the key breakthrough.
And the next step was to represent things contextually.
So the word “table” that we sit next to now,
versus the word table that we are using to store data in, are now going to be different
elements in this vector space because they come, they appear in different contexts.
A big reason for the rise of foundation models is the transformer architecture.
Transformers pre-process text inputs as embeddings,
which are mathematical representations of a word.
When encoded in vector space, words that are closer together
are expected to be closer in meaning.
These embeddings are processed through an encoder component
that captures contextual information from an input sequence.
A decoder component can apply this encoded context
and emit output text.
Compared to its predecessors, like recurrent neural nets,
transformers are more parallelizable
because they do not process words sequentially one at a time;
but instead, process the entire input all at once during the learning cycle.
Now that we have this, you can encode these kind of things in this neural architecture,
very dense neural architecture, multilayer neural architecture.
And now you can start representing larger object and you can represent semantics of bigger objects.
How is it that the Transformer architecture
allows you to do unsupervised training?
Why is that? Why do you no longer need to label the data?
So really, when you learn representation of word, what we do is self-training.
I think a better terminology for it is self-training.
The idea is that you take a sentence that is correct, that you read it of the newspaper,
you drop a word and you try to predict the world given the context, either the two sided context
or the left sided context. And so essentially you
do supervised learning, right? Because you're trying to predict
a world and you know the truth. So you can verify whether your
predictor model actually does it well or not. But you don't need to annotate data for this.
And this is the basic, very simple objective function.
Drop a word, try to predict it. That drives almost all the learning
that we are doing today. And it gives us the ability
to learn good representations of words.
If I look at not only at the past five years with these larger models,
but if I look at sort of the evolution of machine learning in the
past 10-15 years, it seems to have been sort of this lockstep where new software
arises, new hardware is being built, new software comes, new hardware.
And a sort of acceleration happened of the application of it.
So most of this was done on GPUs and the evolution of GPUs,
but they're extremely power hungry beasts. Why are GPUs the best way of training this?
And why are we moving to custom silicon? Because of the power?
Yes so one of the things that is fundamental in computing is if you can specialize the
computation, you can make the silicon optimized for that specific computation structure instead
of being very generic like CPUs are. So what is interesting about deep
learning is that it's essentially low precision linear algebra, right?
So if I can do this linear algebra really well, then I can have a very power efficient,
cost efficient, high performance processor for deep learning.
Is the architecture of, let's say,
the Trainium radically different from, let's say, general purpose GPU?
So really, it is really optimized for deep learning.
So the systolic array for matrix multiplication, you have like a
small number of large systolic arrays and the memory hierarchy is optimized for deep
learning workload patterns versus something like a GPU, which has to kind of cater to a
broader set of markets like high performance computing, HPC graphics and deep learning.
So the more you can specialize and scope down the domain, the more you can optimize in silicon.
And that's the opportunity that we are seeing currently in deep learning.
Okay, if I think about sort of the hype that is all there in the past days or the past weeks,
it looks like this is the end all of machine learning and this little magic happens.
But there must be limitations to this. There are things that they can do well
and things that it probably cannot do well at all. Do you have a sense of that?
We have to understand that language models cannot do everything right.
So aggregation is a key thing that they cannot do.
Various logical operations is something that they cannot do well.
Arithmetic is a key thing or mathematical reasoning.
So what language models can do today, if trained properly,
is to generate some mathematical expressions well, but they cannot do the math.
So you have to figure out mechanism to enrich these with calculators, if you want.
Spatial reasoning, this is something that requires grounding.
So if I tell you go straight and then turn left and then
turn left and then turn left, where are you now?
It's something that three year old’s will know,
but language models will not because they are not grounded.
And there are various kinds of reasoning, common sense reasoning.
I talked about temporal reasoning a little bit.
These models don't have a notion of time unless it's written somewhere.
Can we expect that these problems will be solved over time or...?
I think they will be solved.
Some of these challenges are also opportunities.
So when a language model does not know how to do something, it can figure out that it
needs to call an external agent, as Dan said, he gave the example of calculators, right?
So if I can't do the math, I can generate an expression which the
calculator will execute correctly, right? So I think we are going to see opportunities
for language models to call external agents or APIs to do what they don't
know how to do and just call them with the right arguments and synthesize the results back into
the conversation or their output. That's a huge opportunity.
Well, thank you very much, guys. I really enjoyed this. You really educated me on the
real truth behind large language models and generative AI. Thank you very much.
Thank you.
Thanks to Sudipta and Dan for taking us on this
journey to understand where the future might be headed.
And to begin to shed light on the scale of these vastly complex models.
And, the energy needed from custom hardware to reach new efficiencies in performance and cost.
The industry is rapidly working to better understand where these technologies are headed.
I’m looking forward to learning more from engineers and developers about the impact
Generative AI will have on everything from coding to testing to building infrastructure.