Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
This episode does not have the usual visual fireworks, but I really wanted to cover this
paper because it tells a story that is, I think, very important for all of us to hear
about.
When creating a new AI to help us with a task, we have to somehow tell this AI what we consider
to be a desirable solution.
If everything goes well, it will find out the best way to accomplish it.
This is easy when playing simpler video games, because we can just tell the algorithm to
maximize the score seen in the game.
For instance, the more bricks we hit in Atari breakout, the closer we get to finishing the
level.
However, in real life, we don’t have anyone giving us a score to tell us how close we
are to our objective.
What’s even worse, sometimes we have to make decisions that seem bad at the time,
but will serve us well in the future.
Trying to save money or studying for a few years longer are typical life decisions that
pay off in the long run but may seem undesirable at the time.
The opposite is also true, ideas that may sound right at a time may immediately backfire.
When in a car chase, don't ask the car AI to unload all unnecessary weights to go faster,
or if you do, prepare to be promptly ejected from the car.
So, how can we possibly create an AI that somehow understands our intentions and acts
in line with them?
That’s a challenging question, and is often referred to as the agent alignment problem.
It has to be aligned with our values.
What can we do about this?
Well, short of having a mind-reading device, we can maybe control the behavior of the AI
through its reward system.
Scientists at DeepMind just published a paper on this topic, where they started their thought
process from two assumptions: Assumption number one, quoting the authors:
“For many tasks we want to solve, evaluation of outcomes is easier than producing the correct
behavior”.
In short, it is easier to yell at the TV than to become an athlete.
Sounds reasonable, right?
Note that from complexity theory, we know that this does not always hold, but it is
indeed true for a large number of difficult problems.
Assumption number two: User intentions can be learned with high accuracy.
In other words, given enough data that somehow relates to our intentions, the AI should be
able to learn that.
Leaning on these two assumptions, we can change the basic formulation of reinforcement learning
in the following way: normally, we have an agent that chooses a
set of actions in an environment to maximize a score.
For instance, this could mean moving the paddle around to hit as many blocks as possible and
finish the level.
They extended this formulation in a way that the user can periodically provide feedback
on how the score should be calculated.
Now, the AI will try to maximize this new score and we hope that this will be more in
line with our intentions.
Or, in our car chase example, we could modify our reward to make sure we remain in the car
and not get ejected.
Perhaps the most remarkable property of this formulation is that it doesn’t even require
us to for instance, play the game at all to demonstrate our intentions to the algorithm.
The formulation follows our principles, and not our actions.
We can just sit in our favorite armchair, bend the AI to our will by changing the reward
function every now and then, and let the AI do the grueling work.
This is like yelling at the TV, except that it actually works.
Loving the idea.
If you have a look at the paper, you will see a ton more details on how to do this efficiently
and a case study with a few Atari games.
Also, since this has a lot of implications pertaining to AI safety and how to create
aligned agents, an increasingly important topic these days, huge respect for DeepMind
for investing more and more of their time and money in this area.
Thanks for watching and for your generous support, and I'll see you next time!