We're six paragraphs in and it knows about this point, I've covered the first sentence of this initial paragraph
now it's time to talk about this second sentence of the lead even more surprising to the researchers than the fact that they
spoke English and
it completely ignored the speaking English part until it got to the part of the news article where that comes in and
Now it's talking about it
Which is the kind of thing that these kinds of systems would have real trouble doing I've known journalists who can't write that well
If that ties into
Something that I think is kind of fundamental to the way that people think about AI
like we used to think that you had to be really clever to be good at chess and if you could play chess then you
Were real?
Intelligence you had to be and then we realized that like playing chess to a superhuman level doesn't require something that we would call
Intelligence and it's slightly unsettling to find that like writing coherent and plausible
News prose apparently also doesn't require general intelligence
like if you just learn the statistical relationships between words
But do that really really well that seems to be enough
I mean obviously any journalist would would say that the truth is kind of important in this as well
But yeah to make it sound plausible. You're absolutely right, right
Yeah there so there's there's definitely questions about like directing this towards producing a specific article
that is correct, but
just the generating of the of the prose itself apparently requires less like
Philosophical sophistication than we thought
Then many thought anyway, I think 10 years ago
People would have a really hard time believing that
Something that's just learning from data and learning these relative probabilities could produce something this coherent
you would expect it to have all sorts of conditions and a formula and
An uncoded you would look at this and say oh this must have like a database of
names and countries
locations and
Cities and everything else because it's using that information, but it turns out all of that
Is already represented in the data set because we're talking about all these things
Here's a recipe for some kind of peppermint chocolate cake, and it's got a bunch of different completions
so you can just spit these out arbitrarily so
You know those recipe blogs where like you google the recipe for something and you go to the blog and there's like seven paragraphs
Of like my mother used to make this for me back in our home in, Indiana
I always remember sitting out on the porch with my dog
He used to you know
The important thing is I had an onion on my belt which was the style at the time and and it's doing that
It's like talking about
Different back stories. Yeah, just back story and um I wonder if anyone was trying to make any of these recipes
Yeah, it's dangerous
this one
So this is this is a recipe for meringue cookies
1 and 3/4 cups butter softened the cup of sugar and egg yolk 3 t of heavy
Critical T. What unit? Is that 3 tons? Heavy cream?
That's usually a lowercase T. I don't know what it is 3 tons of heavy cream. Let's say
three and a half to four cups of flour pinch of salt
Peppermint Jojo topping which like I have no idea what that is, but peppermint. Jo jo's is mentioned in the prompt
So one and a quarter cups powdered sugar a cup of chopped pecans a half a cup
Finely chopped mint leaves 1/2 cup chopped fresh mint about a half sheet
So it's like it doesn't quite make sense, but it's right on the edge of making sense
like we have half a cup of chopped mint leaves and
Then also half a cup of chopped fresh mint all these potentially cherry picked out of a huge number of horrendous
Right. Yes. So this is these ones specifically or not
So the unicorn one and this is something that I like and it's standard practice, but I like this the unicorn one
It says they specifically said right now we're gonna make a sample for the paper
They made 10 of them and they picked the one they liked which was this one but for the recipes here
These are not cherry picked at all. That's why they're showing 6 they just gone
This is the first six that we generated
And here they are and there so that gives you a better idea of the general quality of what it's putting out
They're all fairly sensible. I like look at this world, right?
this one comes in and says I do not substitute it with something else blah blah and then
Like this, I don't know if that is right
Here's an image or yeah
Please like this on Facebook and then it goes on I found this really cute card with cute little
Kittens on and then as your samples cut off, so it's just like next post in the blog or you know
This is why GPT-2 is so cool to me. Let's see
The first thing they tested on is a children's book test where this is. I think it's like a closed thing
Where you just have it have a data set with some children's books and then you like remove one word
And then the system has to predict which of the words is the correct
Like you give it you give it ten words that it might be or something like that and it has to pick what word fits
In this space. So that's standard kind of language model type task. The one thing that they did have to do is
they had to run analysis to check about overlap and it turns out that one of the
Children's books is the jungle book by Rudyard Kipling
Which was actually in in its entirety was in the data set that they trained the thing on so he just knew that one
So then they threw that out because I think that's not fair if you've already seen the entire book before
And it's performance on that was was good by the time they guts up to the to the very large scale models
Its scoring one. Is that eighty?
eighty nine percent where humans tend to get like
ninety 90 to 93 percent
It's like nearly a human level for guessing one missing word from a sentence in a children's book pretty good
Lambada is designed to test long-range dependencies, which is what I've been talking about a lot
The task is to predict the final word of sentences which require at least 50 tokens of context for a human to successfully predict
It's like 50 words is a pretty long sentence
and
So this kind of long-term dependency thing is a standard way of testing language models and
It ends up with an accuracy of 63 percent
Which is an improvement over state of the art by four percent. So it's state of the art on Lambada without
Being specifically trained on that just running in general the winograd schema
I don't know if it's support if you know event venal guard. Who knows
Maybe it's somebody's name
Whatever. This is about resolving ambiguities, which is really especially important in
translation tasks and things like that
it's very easy for sentences to be ambiguous in a way that
Makes translating them
Very difficult or even impossible and I have like I have an example of this check this out. So consider a sentence like
The chicken didn't cross the road because it was too black
okay, and then we can consider different versions of this sentence suppose that this is
Wide chicken didn't cross the road because it was too wide, right?
That's like one possible completion for this or you might say
The chicken didn't cross the road because it was too scared
another perfectly sensible sentence, then the question is
it
in one of these
It is referring to the chicken and in one of them. It is referring to the road color for a third and sure
stormy
Stormy. Oh, yeah. Alright. That's a good one. So stormy means it is actually neither of the things in the sentence
the other one that's fun is something like
busy
Right. Is it a busy road or did the chicken just have better things to do than crossing the road?
We don't know. I mean like I
Would say probably the road but this could be a children's book, right?
We are running this thing on children's books. The rabbit. The rabbit was too busy. It was late for an important date
why can't that she can be busy so
The point is suppose. We're trying to translate this into a language the genders everything as many languages do and
Maybe chicken is like a is a masculine noun and Road is a feminine noun and it has to know
What it's about right?
Is this like is this L or L or whatever so the idea of this benchmark is?
measuring how well it can resolve ambiguities because if this says wide if you're trying to do
Translation the old old-fashioned way
Where you're like parsing trees and looking things up in dictionaries and stuff like that
This kind of sentence is the hell nightmare because you can't you don't know. I mean, you just don't know the information is not
really present in the
Text it's not present grammatically. It's present in your understanding of the world and chickens and roads, right?
So translating this if this is it was too wide and you translate it
Into something which is the equivalent of the English sentence the chicken cross the road because the chicken was too wide you've screwed up
Right, that's a bad translation. But at the same time
Like there's nothing in the sentence that tells you that it shouldn't be right
so what you need to do is the same this the thing that we've already seen that GPT 2 is super good at
Which is pulling in knowledge of the world
Like knowing that the University of La Paz is going to be near the Andes or in the Andes right that kind of thing
It's going to know
that
Roads being wide is a thing much more than chickens being wide is a thing and so on and that like roads don't get scared
If it's scared crazy fearless, so again on this kind of thing, it does very well
It beats the state of the art by a lot. You can see on this graph here
So the way that this graph is laid out, by the way, this is the same in all of them
This is the size of the model. They made four different sizes of model. And these are like the same sizes that previous
Language models were so they were like make sense to compare them their previous to the small ones
They do worse than the state of the art
But then these seven hundred sixty two million parameter and the 1.5 billion parameter significantly past state of the art
They're getting like 70 percent. So the state of the art is the straight line across, right?
Yes
And the thing that is also kind of fun about some of these graphs is so some of them they're seven six two
million and the 1.5 billion end up doing about as well as each other, which means you've like hit the limit of I
Get maybe your data set or whatever. Whereas in this one. They're still growth which means an even bigger model
We might expect to do even better maybe reading comprehension
This is another thing you have some text you then you have to answer questions about that text, right?
the thing that's fun is
How do you do this
Without
Modifying your model. That's just a generative model
This is where we start getting into
So that's... you... by the time it's read it, it's modified itself based upon what you've given it to read? Is that what you mean?
No, what I mean is
The way that GPT 2 works is
you give it the sequence of tokens and it gives you a probability distribution for the next token and
so they're like type signature of that is
totally fine with if you're trying to fill in a missing word or
I guess I don't know how they did it for these
for this test
But you have to take you have to take the challenge that you're given and try and express it in terms of
This like predict the next word
type
setup because otherwise
You're sort of cheating, right? The whole thing is they're trying to go at this and not not modify the system at all. So
for reading comprehension
the way they do it is they
give the
thing bits to be comprehended and
then they give
Q : a question a : the correct answer to that question
newline Q : a new question they give like three or four example questions and then
Q : the question, they actually want answered a : let it generate
So they sort of prime it I think we have some examples of this
So this is how they did the question/answer thing. They gave these two paragraphs about
the
Olympic Games to torch relay moving the
Olympic torch and I have some news story and then a bunch of questions right question
What was the theme of the one world one dream and so on and so on and then at the end question and did they?
climb any mountains and then a :
Generate me the next word so they've used
This the the input text to kind of prime the model
Now we're doing question-answer pairs. This is how it works, right and
The interesting thing about this is it actually ends up giving kind of a better answer than that human generated answers
So the question did they climb any mountains the responses they got from humans were unknown. Yes. Yes, and yes because they do plain mountains
but gbg to
Dare answer is Everest. So gbt twos answer is actually kind of better than the humans. The humans just said, yes they did and
the machine learning system has named the mountain that they climbed so I don't know if that's
If that counts is not quite understanding the question or if that counts is actually providing a high-quality answer
It's up for debate because it has this ability with attention to do the long range
It has to look back past all of the previous questions
To the actual paragraph and find the relevant information and it can do it and it performs reasonably. Well at that
But the thing I love about that is that they have to like come up with tricks
To get it to actually do the task that they're trying to get it to do. So the summarization one is brilliant
I love the way they did this with the summarization. See if you can guess how they did this, right?
You want to get a summary of?
A piece of text. How do you how do you get that given a huge data set or edit?
content and what you do is
you write the whole long piece of text and then you put like a new line and then you put
TL DR too long didn't read in this data set. There will be
thousands and thousands of examples of long pieces of text followed by a short summary of that text and
In the middle is this string TL DR? I would love to have been in the room when they thought of that
So yeah, it's really
Really cool really powerful technology. I like it a lot.
So an executable binary the net effect of slotting that T diagram against here slightly downwards is to show you
That the C you've written gets converted into binary and the net output
from this process it produces out a program that you probably store in a