Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Reinforcement learning is a technique in the field of machine learning to learn how to
navigate in a labyrinth, play a video game, or to teach a digital creature to walk. Usually,
we are interested in a series of actions that are in some sense, optimal in a given environment.
Despite the fact that many enormous tomes exist to discuss the mathematical details,
the intuition behind the algorithm itself is remarkably simple. Choose an action, and
if you get rewarded for it, try to find out which series of actions led to this and keep
doing it. If the rewards are not coming, try something else. The reward can be, for instance,
our score in a computer game or how far our digital creature could walk.
Approximately a 300 episodes ago, OpenAI published one of their first major works by the name
Gym, where anyone could submit their solutions and compete against each other on the same
games. It was like Disneyworld for reinforcement learning researchers.
A moment ago, I noted that in reinforcement learning, if the rewards are not coming we
have to try something else. Hmm..is that so? Because there are cases where trying crazy
new actions is downright dangerous. For instance, imagine that during the training of this robot
arm, initially, it would try random actions and start flailing about, where it may damage
itself, some other equipment, or even worse, humans may come to harm. Here you see an amusing
example of DeepMind’s reinforcement learning agent from 2017 that liked to engage in similar
flailing activities.
So, what could be a possible solution for this? Well, have a look at this new work from
OpenAI by the name Safety Gym. In this paper, they introduce what they call the constrained
reinforcement learning formulation, in which these agents can be discouraged from performing
actions that are deemed potentially dangerous in an environment. You can see an example
here where the AI has to navigate through these environments and achieve a task, such
as reaching the green goal signs, push buttons, or move a box around to a prescribed position.
The constrained part comes in whenever some sort of safety violation happens, which are,
in this environment, collisions with the boxes or blue regions. All of these events are highlighted
with this red sphere and a good learning algorithm should be instructed to try to avoid these.
The goal of this project is that in the future, for reinforcement learning algorithms, not
only the efficiency, but the safety scores should also be measured. This way, a self-driving
AI would be incentivized to not only drive recklessly to the finish line, but respect
our safety standards along the journey as well. While noting that clearly, self-driving
cars may be achieved with other kinds of algorithms, many of which have been in the works for years,
there are many additional applications for this work: for instance, the paper discusses
the case of incentivizing recommender systems to not show psychologically harmful content
to its users, or to make sure that a medical question answering system does not mislead
us with false information.
This episode has been supported by Linode. Linode is the world’s largest independent
cloud computing provider. They offer you virtual servers that make it easy and affordable to
host your own app, site, project, or anything else in the cloud. Whether you’re a Linux
expert or just starting to tinker with your own code, Linode will be useful for you.
A few episodes ago, we played with an implementation of OpenAI’s GPT-2 where our excited viewers
accidentally overloaded the system. With Linode's load balancing technology, and instances ranging
from shared nanodes all the way up to dedicated GPUs you don't have to worry about your project
being overloaded. To get 20 dollars of free credit, make sure to head over to linode.com/papers and
sign up today using the promo code “papers20”. Our thanks to Linode for supporting the series
and helping us make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time!