Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
In this new age of AI, there is no shortage of articles and discussion about AI safety,
and of course, rightfully so: these new learning algorithms started solving problems that were
previously thought to be impossible in quick succession.
Only ten years ago, if we told someone about half of the things that have been covered
in the last few Two Minute Papers episodes, we'd have been declared insane.
And of course, having such powerful algorithms, we have to make sure that they are used for
good.
This work is a collaboration between OpenAI and DeepMind's security team and is about
introducing more human control in reinforcement learning problems.
The goal was to learn to perform a backflip through reinforcement learning.
This is an algorithm that tries to perform a series of actions to maximize a score.
Kind of like playing computer games.
For instance, in Atari Breakout - if we break a lot of bricks, we get a high score so we
know we did something well.
If we see that happening, we keep doing what led to this result, if not, we go back to
the drawing board and try something new.
But this work is about no ordinary reinforcement learning algorithm, because the score to be
maximized comes from a human supervisor and we're trying to teach a digital creature to
perform a backflip.
I particularly like the choice of the backflip here because we can tell when we see one,
but a mathematical specification of this in terms of movement actions is rather challenging.
This is a problem formulation in which humans can overlook and control the learning process,
which is going to be an increasingly important aspect of learning algorithms in the future.
The feedback option is very simple: we just specify whether this sequence of motions achieved
our prescribed goal or not.
Did it fall or did it perform the backflip successfully.
After around 700 human feedbacks, the algorithm was able to learn the concept of a backflip,
which is quite remarkable given that these binary yes/no scores are extremely difficult
to use for any sort of learning.
In an earlier episode, we illustrated a similar case with a careless teacher who refuses to
give out points for each problem on a written exam and only announces whether we have failed
or passed.
This clearly makes a dreadful learning experience, and it is incredible that the algorithm is
still able to learn using these.
We provide feedback on less than 1% of the actions the algorithm makes, and it can still
learn difficult concepts off of these extremely sparse and vague rewards.
Low-quality teaching leads to high-quality learing.
How about that!?
This is significantly more complex than what other techniques were able to learn with human
feedback.
And, it works with other games too!
A word about the collaboration itself.
When a company hires a bunch of super smart scientists and a spends a ton of money on
research, it is understandable that they want to get an edge through these projects, which
often means keeping the results for themselves.
This leads to excessive secrecy and a lack of collaboration with other groups as everyone
wants to keep their cards close to their chest.
The fact that such collaborations can happen between these two AI research giants is a
testament to how devoted they are to working together and sharing their findings with everyone,
free of charge for the greater good.
Awesome.
As the media is all up in arms about the demise of the human race I feel that it is important
to show the other side of the coin as well.
We have top people working on AI safety right now.
If you wish to help us tell these stories to more people, please consider supporting
us on Patreon.
Details are available in the video description, or just click the letter p that appears on
the screen in a moment.
Thanks for watching and for your generous support, and I'll see you next time!