Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
When I was growing up, IQ tests were created by humans to test the intelligence of other
humans.
If someone told me just 10 years ago that algorithms will create IQ tests to be taken
by other algorithms, I wouldn’t have believed a word of it.
Yet, just a year ago, scientists at DeepMind created a program that is able to generate
a large amount of problems that test abstract reasoning capabilities.
They are inspired by human IQ-tests with all the these questions about sizes, colors and
progressions.
And then, they wrote their own neural network to take these tests, which performed remarkably
well.
How well exactly?
In the presence of nasty distractor objects, it was able to find out the correct solution
about 62% of the time, and, if we removed these distractors, which, I will note that
are good at misdirecting humans too, the AI was correct 78% of the time!
Awesome.
But today, we are capable of writing even more sophisticated learning algorithms that
can even complete our sentences!
Not so long ago, the OpenAI lab published GPT-2, a technique that they unleashed to
read the internet, and it learned our language by itself.
A few episodes ago, we gave it a spin, and I almost fell out of the chair when I saw
that it could finish my sentences about fluid simulations in such a scholarly way, that
I think, could easily fool a layperson.
Have a look here and judge for yourself!
This GPT-2 technique was a neural network variant that was trained using 1.5 billion
parameters.
At the risk of oversimplifying what that means, it roughly refers to the internal complexity
of the networks, or in other words, how many weights and connections are there.
And now, the Google Brain team has released Meena, an open-domain chatbot that uses 2.6
billion parameters, and shows remarkable human-like properties.
The chatbot part means a piece of software or a machine that we can talk to, and the
open-domain part refers to the fact that we can try any topic, hotels, movies, the ocean,
favorite movie characters, or pretty much anything we can think of and expect the bot
to do well.
So how do we know that it’s really good?
Well, let’s try to evaluate it in two different ways.
First, let’s try the super fun, but less scientific way, or, in other words, what we
are already doing, looking at chat logs!
You see a Meena writing on the left, and a human being on the right, and it not only
answers questions sensibly and coherently, but is even capable of cracking a joke.
Of course, if you consider a pun to be a joke, that is.
You see a selection of topics here, where the user talks with Meena about movies, and
the bot expresses the desire to see The Grand Budapest Hotel, which is indeed a very humanlike
quality.
It can also try to come up with a proper definition of philosophy.
And now, since we are scholars, we would also like to measure how humanlike this is in a
more scientific manner as well!
Now is a good time to hold on to your papers, because this is measured with the Sensibleness
and Specificity Average score, from now on, SSA in short, in which, humans are here, previous
chatbots are down there, and Meena is right there, close by, which means that it is easy
to be confused for a real human.
That already sounds like science fiction, however, let’s be a little nosy here and
also ask, how do we know that this SSA is any good in predicting what is humanlike and
what isn’t?
Excellent question.
When measuring human-likeness for these chatbots, plugging in this SSA, again, the Sensibleness
and Specificity Average, we see that they correlate really strongly, which means that
the two seem to measure very similar things, and in this case, SSA can indeed be used as
a proxy for human likeness.
The coefficient of determination is 0.96.
To put this into perspective, this is a several times stronger correlation than we can measure
between the intelligence and the grades of a student, which is already a great correlation.
This is a remarkable result.
Now, what we get out of this is that the SSA is much easier and precise to measure than
human likeness, and is hence, used throughout the paper.
So, chatbots eh?
What are all these things useful for?
Well, remember Google’s technique that would automatically use an AI to talk to your callers
and screen your calls?
Or even make calls on your behalf.
When connected to a text to speech synthesizer, something that Google already does amazingly
well, Meena could really come alive in our daily lives soon.
What a time to be alive!
This episode has been supported by Lambda.
If you're a researcher or a startup looking for cheap GPU compute to run these algorithms,
check out Lambda GPU Cloud.
I've talked about Lambda's GPU workstations in other videos and am happy to tell you that
they're offering GPU cloud services as well.
The Lambda GPU Cloud can train Imagenet to 93% accuracy for less than $19! Lambda's web-based
IDE lets you easily access your instance right in your browser.
And finally, hold on to your papers, because the Lambda GPU Cloud costs less than half
of AWS and Azure.
Make sure to go to lambdalabs.com/papers and sign up for one of their amazing GPU instances
today.
Our thanks to Lambda for helping us make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time!