In the previous video we were talking about
transformers this architecture that uses attention to give
Unprecedented ly good performance on sort of language modeling tasks and some other tasks as well
but when were looking at language modeling and that was in preparation to make a video about
GPG 2, which is this very giant language model that has been there was recently
Well, it was recently not released actually by open AI the way that they generated the data set for this is pretty cool
to get enough text they went to Reddit and
They pulled every website that is linked to from reddit. Do we have any idea of how many days lots?
Literally, everything was everything that had more than three karma
I think or maybe more than two karma something like that like
Anything that had somebody had thought to post around it and at least two or three people who had thought was good enough to upload
They scraped the text from that. It's pretty much just a transformer. It's not the the
Architecture is not especially novel. They haven't done any like amazing new
new discovery, but
What they realized was?
Transformers it seems like
the more data you give them the better they do and the bigger you make them the better they do and
Everything that we built up until this point is clearly not
Like we haven't hit the limits of what this can do
We they thought we think we're probably
Bottle necked on data and maybe network size
So what happens if we'd like to turn that 211 what happens if we just give this all?
The data and make a really big one. It makes sense to talk about the acronym right so it's a generative pre-training
Transformer so generative same as generative adversarial network. It generates outputs to generate samples
Your pre-trained is this thing. I was talking about all of the different things
You can use a language model for right you can do you can do translation. You can try and resolve ambiguities
You can do summarization. You can answer questions. You can use the probabilities for augmenting other systems
So yeah, there's a bunch of different benchmarks for these different tasks
that you might want your language model to do and
This is what we talked about in the grid worlds video of having these like standardized problems with standardized metrics and standardized data sets
So that if you're comparing two different methods, you know that you're actually comparing apples to apples
And this is like very important it gives you numbers on these things. It's often quite difficult
Expected to like you're generating samples of text and it's like how plausible is this text? How realistic does it look like?
How do you put a number on that it's kind of difficult. So there's all of these standardized metrics and
the thing that
People came to realize which actually I mean I say that as though it's like some amazing discovery
It's fairly obvious. If you train your system in a like an unsupervised way on a large corpus of just general English text and
then you take that and
Train that with the data from this benchmark or the data from that benchmark
You can like fine-tune it so you start with something which has like a decent
Understanding of how English works more or less and then you say now I'm going to give you these
Samples for like question answering or I'm going to build a system using that to solve to go for this benchmark
So it's pre trained you start with something. That's like a general-purpose language model and then you from that a
Fine-tuned it to whichever
Actual benchmark or problem you're trying to solve
and this
Can give you better performance than to starting from nothing and training to each of the benchmarks from scratch
make sense
and so
The point of the GPT 2 paper the thing that makes it cool is they said okay if we make a really huge one
What if we?
don't
Fine tune it at all
What if we just make a giant model and then just try and run it on the benchmarks without messing with it?
Without showing it any of their specialized data for that benchmark. Just the raw
general-purpose language model, how does that perform and it turns out
surprisingly well, so this is a
Very very large data set for text
It's about 40 gigabytes
which
Actually doesn't sound like very much but like for text text that's insane, right? It's
somebody said that this was the size of
Google's entire index of the Internet in 98
So like it's yeah, it's a lot of text
and they trained it on that and they ended up with a
1.5 billion parameter model, but which is like a previous state of the art system was 345 million
This is 1.5 billion
So they've just made the thing much much bigger and it performs really well some of their samples that they published quite
captured the public imagination
You could say and now that we've talked a little about the problems that
Neural networks or any language model really?
Has with a long term dependency
we can now realise just how impressive these samples are because when you look at them as a you know,
If you look at them uninitiated, you're like yeah, that's pretty realistic
It seems to like make sense and it's cool. But when you look at it knowing how language models work, it's like
very impressive the the coherence and the
Consistency and the long-range dependencies so we can look at this one that got everybody's attention the unicorns one
right
So they prompted it with in a shocking finding scientists discovered a herd of unicorns
living in a remote previously unexplored valley in the Andes Mountains
Even more surprising to the researchers was the fact that the unicorns spoke perfect English
And from there you then say you go to your language model gbgt, and you say given that we started with this
What's the next word and what's the word after that and so on?
So it goes on the scientist named the population after their distinctive horn of its unicorn
These four horned silver white unicorns were previously unknown to science
We do have a clue here as a human being unicorns for horned doesn't quite make sense
But nonetheless we're going okay
Now after almost two centuries the mystery of what sparked this odd phenomenon is finally solved. Dr
Budetti Jorge Jorge Perez
Jo are G an evolutionary biologist from the University of La Paz
This is impressive because we've mentioned the Andes Mountains in our prompt and so now it's saying okay
This is clearly, you know in a shocking finding. This is a science press release news article
It's seen enough of those because it has every single one that was ever linked to from reddit, right?
So it knows how these go it knows. Okay third paragraph
This is when we talk about the scientist, we interview the scientist, right? Okay
First word of the scientist paragraph, dr. Obviously, right because this is the now we're in the name of the scientist
What name are we going to give?
It needs to be a name
conditioning on the fact that we have the Andes Mountains
So we need to get where we're in South America
The name probably should be Spanish or maybe Portuguese
So we get we get dr. Perez here
And then evolutionary biologist makes sense because we're talking about animals
from the University of La Paz again
This is the first sentence like when you have that first clause that introduces the scientist you always say where they're from
So we say from the University of and then university names tend to be the name of a city
What's the city where we have the Andes Mountains, so we're going to Bolivia lapaz. Perfect
And the thing that's cool about this is it's remembered all of these things that were quite a long time ago several sentences ago
Well, it hasn't remembered them. It's paid attention to them across that distance, which is impressive
But also this is encoding a bunch of understand understanding a bunch of information about the real world
Right all that was given all it knows is statistical relationships between words, but the way that it comes out to us
Is that it knows?
Where the Andes Mountains are what kind of names people in that area have what their cities are what the universities are all of those
Facts about the real world because in order to have a really good language model it turns out you have to kind of implicitly encode
information about the world because
We use language to talk about the world and knowing what's likely to come next
Requires actual real world understanding and that's something that we see in some of the other
Things that they got it to do you can see the real world understanding coming through
Let's keep going
University of a person several companions were exploring the Andes Mountains when they found a small valley with no other animals or humans peres see
We're hanging on to him. Yep. We're referring to him again
but now we've changed it to be just the surname because that's the
format that people use in news articles Peres noticed that the valley had what appeared to be a natural fountain surrounded by two peaks of
Rock and silver snow presently others, then ventured further into the valley a round about here in our article
We should have a quote from the scientist right quote
By the time we reached the top of one peak the water looked blue with some crystals on top and we're talking about this fountain
I guess it's natural fountain. We're referring back to the previous int. It's like everything is
Relying on in contingent on earlier parts of the text while examining there by snipped paragraph while examining these bizarre
Creatures the scientists discovered that the creatures also spoke some fairly regular English know when I read that I like, okay
this is now unusually good because that's the second sentence of the lead right where six paragraphs in and
It knows about this point. I've covered the first sentence of this
initial paragraph
now it's time to talk about this second sentence of the lead even more surprising to the research of us of the fact that they
spoke English and
It completely ignored the speaking English part until it got to the part of the news article where that comes in
You've gone six whole paragraphs
the idea of
Accurately remembering that the unicorn speak perfect
English is like that's very impressive to me and then it goes into its gets a little bit unhinged
Starts talking about it's likely that the only way of knowing for sure if unicorns are indeed
The descendants of a lost alien race is through DNA. That's read it really
Well, it's not actually stuff on reddit. It's stuff linked to from reddit. But yeah, this is this is news articles men
They seem to be able to communicate in English quite well
Which I believe is a sign of evolution or at least a change in social organization said the scientist
That's his evolutionary biology there. Right? Right, right. Yeah, we know here's an evolutionary biologist. So so the the
coherence of this text is
really dependent on its ability to
Condition what it's generating on
Things that it's generated a long time ago
So yeah
So it can generate really nice news articles and it can generate all kinds of text things that it anything that is
Sufficiently well represented in the original data set. So that's GPG - it's a really
Unusually powerful and like versatile
language model that can do all of these different natural language processing
Tasks without actually being trained specifically on those tasks
It's really and that's that's why it's impressive
It's not that it's a it's a brand new architecture or a brand new approach or whatever
It's just when you make these things really huge and give them tremendously large amounts of data
The results are really impressive
In the original data set. So it will it will write you the Lord of the Rings fan fiction
It will write you cake recipes if we're like, there's all kinds of examples of different samples. Here's a recipe for
Some kind of peppermint chocolate cake and it's got a bunch of different