Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
This is an incredible paper from OpenAI, in which the goal is to teach an AI to read a
piece of text and perform common natural language processing operations on it, for instance,
answering questions, completing text, reading comprehension, summarization, and more.
And not only that, but additionally, the AI has to be able to perform these tasks with
as little supervision as possible. This means that we seek to unleash the algorithm that
they call GPT-2 to read the internet and learn the intricacies of our language by itself.
To perform this, of course, we need a lot of training data, and here, the AI reads 40
gigabytes of internet text, which is 40 gigs of non-binary plaintext data, which is a stupendously
large amount of text.
It is always hard to put these big numbers in context, so as an example, to train similar
text completion algorithms, AI people typically reach out to a text file containing every
significant work of Shakespeare himself, and this file is approximately 5 megabytes. So
the 40 gigabytes basically means an amount of text that is 8000 times the size of Shakespeare’s
works. That’s a lot of text.
And now, let’s have a look at how it fares with the text completion part.
This part was written by a human, quoting: “In a shocking finding, scientist discovered
a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains.
Even more surprising to the researchers was the fact that the unicorns spoke perfect English.”
And the AI continued the text the following way, quoting a short snippet of it:
“The scientist named the population, after their distinctive horn, Ovid’s Unicorn.
These four-horned, silver-white unicorns were previously unknown to science.”
Whoa! Now note that this is clearly not perfect, if there is even such a thing as a perfect
continuation, and it took 10 tries, which means that the algorithm was run 10 times
and the best result was cherrypicked and recorded here. And despite all of these, this is a
truly incredible result, especially given that the algorithm learns on its own.
After giving it a piece of text, it can also answer questions in a quite competent manner.
Worry not, later in this video, I will show you more of these examples and likely talk
over them, so if you are curious, feel free to pause the video while you read the prompts
and their completions.
The validation part of the paper reveals that this method is able to achieve state-of-the-art
results on several language modeling tasks, and you can see here that we still shouldn’t
expect it to match a human in terms of reading comprehension, which is the question answering
test. More on that in a moment.
So, there are plenty of natural language processing algorithms out there that can perform some
of these tasks, in fact, some articles already stated that there is not much new here, it’s
just the same problem, but stated in a more general manner, and with more compute. A ha!
It is not the first time that this happens.
Remember our video by the name “The Bitter lesson”? I’ve put a link to it in the
video description, but in case you missed it, let me quote how Richard Sutton addressed
this situation: “The bitter lesson is based on the historical
observations that 1) AI researchers have often tried to build knowledge into their agents,
2) this always helps in the short term, and is personally satisfying to the researcher,
but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough
progress eventually arrives by an opposing approach based on scaling computation by search
and learning. The eventual success is tinged with bitterness, and often incompletely digested,
because its success over a favored, human-centric approach.“
So what is the big lesson here? Why is GPT-2 so interesting?
Well, big lesson number one is that this is one of the clearer cases of what the quote
was talking about, where we can do a whole lot given a lot of data and compute power,
and we don’t need to insert too much additional knowledge into our algorithms. And lesson
number two, as a result, this algorithm becomes quite general so it can perform more tasks
than most other techniques. This is an amazing value proposition.
I will also add that not every learning technique scales well when we add more compute, in fact,
you can see here yourself that even GPT-2 plateaus on the summarization task. Making
sure that these learning algorithms scale well is a great contribution in and of itself
and should not be taken for granted.
There has been a fair bit of discussion on whether OpenAI should publish the entirety
of this model. They opted to release a smaller part of the source code and noted that they
are aware that the full model could be used for nefarious purposes.
Why did they do this? What is the matter with everyone having an AI with a subhuman-level
reading comprehension? Well, so far, we have only talked about quality. But another key
part is quantity. And boy, are these learning methods superhuman in terms of quantity…just
imagine that they can write articles with a chosen topic and sentiment all day long,
and, much quicker than human beings. Also note that the blueprint of the algorithm is
described in the paper, and a top-tier research group is expected to be able to reproduce
it.
So does one release the full source code and models or not? This is a quite difficult question:
we need to keep publishing both papers and source code to advance science, but, we also
have to find new ways to do it in an ethical manner. This needs more discussion and would
definitely be worthy of a conference-style meeting. Or more.
There is so much to talk about, and so far, we have really only scratched the surface,
so make sure to have a look in the video description, I left a link to the paper and some more super
interesting reading materials for you. Make sure to check them out!
Also just a quick comment on why this video came so late after the paper has appeared.
Since there were a lot of feelings and intense discussion on whether the algorithm should
be published or not, I was looking to wait until the dust settles and there is enough
information out there to create a sufficiently informed video for you. This, of course means
that we are late to the party and missed out on a whole lot of views and revenue, but that’s
okay, in fact, that’s what we’ll keep doing going forward to make sure you get the
highest quality information that I can provide.
If you have enjoyed this episode and would like to help us, please consider supporting
us on Patreon! Remember our motto, a dollar a month is almost nothing, but it keeps the
papers coming. And there are hundreds of papers on my reading list. As always, we are available
through Patreon.com/TwoMinutePapers, and the link is also available in the video description.
Thanks for watching and for your generous support, and I'll see you next time!