Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
OpenAI GPT-2 is a learning-based technique which can perform common natural language
processing operations, for instance, answering questions, completing text, reading comprehension,
summarization, and more.
What is absolutely amazing about this technique is that it is able to perform these tasks
with as little supervision as possible.
This means that we unleash the algorithm that they call GPT-2 to read the internet, and
it will learn our language by itself.
To perform this, of course, we need a lot of training data, and here, the AI reads 40
gigabytes of internet text, which is 40 gigs of non-binary plaintext data, which is a stupendously
large amount of text.
You see here an example of how it fares with text completion, and the results were absolutely
amazing.
And, not only that, because, hold on to your papers, we can now try it ourselves and my
results show that it can be wise as a scholar, and it can also cheat like nobody’s business.
So, let’s try something simple here.
Well, first, as everyone is talking about job security, I’d like to give it the start
of this Two Minute Papers script and see if I am about to be replaced.
Let’s see!
It continues the intro with “The history of Eastern Europe since 1989 has been an interesting
and complex story spanning four decades.
On one side you have a regime of Western-backed dictators from the mid-1990s” and so on.
Very scholarly indeed.
However, I think I might be safe …for now.
And now, something that I wanted to try since the inception of GPT-2, which is, feed it
a snippet of something that I said about Wavelet Turbulence, which is one of the best papers
ever written, and have the AI finish it.
I tried to frame it as an exam question and was not disappointed be the results.
So the text is, “There was a similar technique by the name Wavelet Turbulence, which is”,
I hope you see what I did there…and look what it did!
“There was a similar technique by the name Wavelet Turbulence, which is not at all what
we are talking about, but I'll let you decide whether it is worth your time to read it.”
Wow!
GPT-2 has been caught cheating red handed.
I love it.
Let’s continue what it had to say: “Anyway, my aim here is to describe how we use a fluid
simulation to simulate smoke and liquid effects using just a few simple equations: Solving
a few functions gives us a set of derivatives of the particle trajectories, that is equations
that describe how and where the particles will come and go across the liquid in the
simulation.
For simplicity (as a good teacher of high-school students should know) we can use a simple
equation, which simulates two particles that collide.”
I am stunned.
It recognized that we are talking about fluid simulations, which is already remarkable,
but it went much further - The completion is not bad at all and is not only coherent,
on topic, but has quite a bit of truth to it.
I will have to rethink my previous claim about my job security.
The even crazier thing is that the size of this model is about 750 million parameters,
which is only half of the size of the original full model, which is expected to be even better.
I put a link to this website in the video description for your pleasure, make sure to
play with it, this is mad fun.
And, GPT-2 will also see so many applications that we cannot even fathom yet.
For instance, here you can see that one can train it on many source code files on GitHub
and it will be able to complete the code that we write on the fly.
Now, nobody should think of this as GPT-2 writing programs for us, this is, of course,
unlikely, however, it will ease the process for novice and experts users alike.
If you have any other novel applications in mind, make sure to leave a comment below.
For now, bravo OpenAI, and a big thank you for Daniel King and the Hugging Face company
for this super convenient public implementation.
Let the experiments begin!
Thanks for watching and for your generous support, and I'll see you next time!