Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Reinforcement learning is a learning algorithm that chooses a set of actions in an environment
to maximize a score.
This class of techniques enables us to train an AI to master a large variety of video games
and has many more cool applications.
For instance, in the game of Q*bert, at every time step, the AI has to choose the appropriate
actions to control this orange character and light up all the cubes without hitting the
purple enemy.
This work proposes an interesting alternative to reinforcement learning and is named evolution
strategies and it aims to train not one agent, but an entire population of agents in parallel.
The efficiency of this population is assessed, much like how evolution works in nature, and
new offsprings are created from the best performing candidates.
Note that this is not the first paper using evolution strategies - this is a family of
techniques that dates back to the 70s.
However, an advantage of this variant is that it doesn't require long trial and error sessions
to find an appropriate discount factor.
But wait, what does this discount factor mean exactly?
This is a number that describes whether the AI should focus only on immediate rewards
at all costs, or whether it should be willing to temporarily make worse decisions for a
better payoff in the future.
This optimal number is different for every game, and depends on how much long-term planning
it requires.
With this evolutionary algorithm, we can skip this step entirely.
And the really cool thing about this is that it is not only able to master many games,
but after only 5 hours of training, it was able to find a way to abuse game mechanics
in Q*bert in the most creative ways.
It has found a glitch where it sacrifices itself to lure the purple blob into dropping
down after it.
And much to our surprise, it found that there is a bug - if it drops down from this position,
it should lose a life for doing it, but due to a bug, it doesn't.
It also learned another cool technique where it waits for the adversary to make a move
and immediately goes the other way.
Here's the same scene slowed down.
It had also found and exploited another serious bug which was to the best of my knowledge,
previously unknown - after completing the first level, it starts jumping around in a
seemingly random manner.
A moment later, we see that the game does not advance to the next level, but cubes start
blinking and the AI is free to score as many points as it wishes.
After this video, a human player was able to reproduce this, I've put a link to it in
the video description.
It also found out the age-old trick in breakout, where we dig a tunnel through the bricks,
lean back, start reading a paper, and let physics solve the rest of the level.
One of the greatest advantages of this technique is that instead of training only one agent,
it works on an entire population.
These agents can be trained independently, making the algorithm more parallelizable,
which means that it is fast and maps really well to modern processors and graphics cards
with many cores.
And these algorithms are not only winning the game, they are breaking the game.
Loving it.
What a time to be alive!
I think this is an incredible story that everyone needs to hear about.
If you wish help us with our quest and get exclusive perks for this series, please consider
supporting us on Patreon.
We are available through patreon.com/TwoMinutePapers, and a link with the details is available in
the video description.
We also use part of these funds to give back to the community and empower research projects
and conferences.
For instance, we recently sponsored a conference aimed to teach young scientists to write and
present their papers at international venues.
We are hoping to invest some more into upgrading our video editing rig in the near future.
We also support cryptocurrencies, such as Bitcoin, Ethereum and Litecoin.
I am really grateful for your support.
And this is why every video ends with...
Thanks for watching and for your generous support, and I'll see you next time!