artificial intelligence machine learning
and deep neural networks these are terms
that can spark your imagination of a
future where robots are thinking and
evolving creatures in this video we're
gonna look at reinforcement learning or
RL as I'll sometimes abbreviate it it's
a type of machine learning that has the
potential to solve some really hard
control problems you may have heard that
the AI company deepmind created a
program called alphago it's an AI that
uses reinforcement learning to beat the
world's best go players and then they
recently created alpha star which is
poised to dominate the Starcraft 2 scene
so naturally you might be thinking well
if it can do that why can't I use
reinforcement learning to control my
robot or cool my data center or
stabilize a drone in a highly dynamic
and turbulent flow well let's talk about
that now a quick warning before we begin
this is not intended to be an exhaustive
look at reinforcement learning I
wouldn't be qualified to explain it all
to you anyway instead I want to
introduce this topic from the point of
view of a traditionally trained controls
engineer and I hope to show you that
there's actually a lot of overlap with
control theory by the end of this series
I think you'll be better prepared to
answer questions like what is
reinforcement learning and why should I
consider it when solving my control
problem how do I set up and solve the
reinforcement learning problem and what
are some of the benefits and drawbacks
of reinforcement learning compared to a
traditional controls approach alright so
with the scope of this series set let's
get to it I'm Brian and welcome to a
MATLAB Tech Talk let's start by thinking
about the complexity of building a
walking robot from the perspective of a
traditional controls approach we might
use cameras to view the environment and
then extract image features that can be
converted into signals like position or
locations of obstacles we could combine
those observations with other sensors
that complete the state estimation which
we then use along with a model of the
plant and environment to design the
control system and more than likely this
would consist of multiple control loops
that all interact with each other for
example there would be low-level motor
controllers and high-level controllers
that are managing the leg trajectories
or the robot trunk trajectory and maybe
a higher level controller that is
managing the balance or off-nominal
behavior and everything has to work
together in an uncertain environment to
generate this complex movement of
walking which can be really challenging
instead of all this complexity let's
squeeze it down into a single black box
that simply takes in observations and
outputs the low-level motor commands
directly if we were infinitely smart we
could sit down and design a function
that could get a robot to walk without
concerning ourselves with all of the
internal steps along the way but since
we're not that's where machine learning
comes in
broadly speaking machine learning can be
subdivided into three categories
unsupervised learning supervised
learning and reinforcement learning
unsupervised learning is used to find
patterns or hidden structures and
datasets that have not been categorized
or labeled for example imagine you
collected information on a hundred
thousand animals like a bunch of
physical attributes and social
tendencies then you can use unsupervised
learning to group the animals or cluster
them into similar features this can be
something obvious like grouping them
into mammals and birds or to group them
by patterns that might not be as obvious
like finding correlations between
physical traits and social behaviors
that you didn't know about ahead of time
supervised learning on the other hand is
different in a subtle way in that we
train the computer to apply a label to a
given input for example let's say that
one of the columns of our data set of
animal features is the species we can
then treat species as the label and the
rest of the data as inputs into a
mathematical model then we can use
supervised learning to train our model
to correctly label each set of animal
features by inputting them in one at a
time
letting the model guess the species and
then systematically tweaking the model
based on whether that guess was correct
or not and if we had enough training
data to get a reliable model we could
then send through the input features for
a new animal one that we don't have
labeled and our trained model would
apply the most probable species label to
it and supervised learning is probably
the type of machine learning that most
people are familiar with because it's
what allows computers to recognize
pictures of cats or your friends and
photos
and fundamentally applying a label to an
image is exactly the same problem as
applying a label to a dataset of animal
features we input a bunch of training
images into the model and then we tweak
the model based on whether it guessed
correctly or not
until it's accurate the difference is
that the input data for an image is just
a stream of numbers representing pixel
intensities so it's not as
straightforward to understand how that
relates to a cat this is what deep
learning is good at by representing the
model as a deep neural network we have
an efficient way to input thousands of
numbers and then tweak it during
training so that it can identify
features within an array of pixel
intensities that will ultimately allow
it to apply the correct label
reinforcement learning is a different
beast altogether unlike the other two
learning frameworks which work with a
static data set
RL works with a dynamic environment and
the goal is not to cluster data or label
data but to find the best sequence of
actions that will generate the optimal
outcome optimal in this sense means to
collect the most reward it does this by
allowing a piece of software called an
agent to explore interact with and learn
from the environment the agent can take
an action which affects the environment
changing its state and the environment
then produces a reward for that action
and using this information the agent can
adjust which action to take in the
future it can learn from this process
and although you're presumably not a
piece of software you learn in
essentially the same way a software
agent learns with the reinforcement
learning framework you can be thought of
as an agent and the world around you as
the environment that you can interact
with observe its state and collect
rewards you get rewarded by the
environment by taking actions that are
good like you went to college action and
you got a job
State and that job pays well reward or
you looked both ways before crossing the
street action you got to the other side
state and you didn't get run over in the
process reward alternatively you get low
or negative rewards for taking actions
that are bad like you stayed up late
before an exam action you're tired State
and you received a poor grade reward
within the agent there is a brain that
takes in state observations the inputs
and maps them to actions the outputs and
an RL nomenclature this mapping is
called the policy given a set of
observations the policy decides which
action to take and just like with
supervised learning we can represent the
policy as a deep neural network which
we'll see later allows our agent to
input thousands of states at once and
still be able to come up with a
meaningful action this is where the term
deep reinforcement learning comes from
in a walking robot example the
observations might be the state of every
joint and the thousands of pixels from a
camera sensor the policy would take in
all of these observations and output the
actuator commands and if the robot stays
upright and continues walking the
environment would generate a reward
telling the agent exactly how well that
very specific combination of actuator
commands did of course the policy might
not be mapped correctly to take the best
actions or the environment might be
slowly changing and so the mapping is no
longer optimal and this is where
reinforcement learning algorithms come
in they changed the policy based on the
actions that were taken the observations
from the environment and the amount of
reward collected in this way the goal of
the overall agent is to use
reinforcement learning algorithms to
modify its policy as it interacts with
the environment so that eventually given
any state it will always take the most
advantageous action the one that will
produce the most reward in the long run
for example if you were tired for your
exam and you received a bad grade well
you learn from it and you adjust your
policy so that you won't stay up late
before the next exam now at its heart
reinforcement learning is an
optimization problem but there are some
very interesting concepts that set
reinforcement learning apart from other
optimization techniques first is the
idea of value reward is the
instantaneous benefit of being in a
specific state whereas value is the
total reward that an agent can expect to
collect from that state and onwards into
the future assessing the value of a
state rather than assessing the reward
helps the agent choose the action that
will collect the most reward over time
rather than a short-term benefit
for example imagine our agent is in this
situation and is trying to collect the
most reward within three steps if the
agent looks directly at the reward for
each action then it will step left first
to get a higher reward and then right
and then left again to ultimately
collect +1 however if the agent is able
to estimate the value of a state then it
will see that going right has a higher
value than going left and will
ultimately end up with +8 reward of
course often the promise of a high
reward in the future still might not
mean that the action is the best and
there's at least two good reasons for
this one like with the financial market
money in your pocket now can be better
than a little more money in your pocket
a year from now and to your prediction
of rewards further into the future
become less reliable and therefore that
high reward might not be there by the
time you reach it in both of these cases
it's more advantageous to be a little
more short-sighted when estimating value
and in RL we can control this by
discounting rewards by a larger amount
the further they are in the future
another critical aspect of reinforcement
learning is the trade-off between
exploration and exploitation when
interacting with the environment this is
the trade-off between collecting the
most rewards that you already know about
versus exploring areas of the
environment that you haven't visited yet
for example let's say the agent only
knows about the two rewards immediately
adjacent to it if it took the greedy
approach by exploiting the environment
it would only go after the highest
reward it knows about and so it would go
left to collect the +1 however if we
occasionally let the agent explore the
state space even at the risk of
collecting fewer rewards it can fill out
more of its value function and it opens
up the possibility of finding higher
rewards that it didn't know about and
this is part of our normal learning
process as humans as well a simple
example is deciding on which restaurant
you want to eat at do you choose a
restaurant you know you like and
therefore exploit your knowledge or do
you venture out and explore a restaurant
you've never been to before increasing
your knowledge now trying a new
restaurant gives you the opportunity to
find a new favorite place but it also
increases your chances of getting a meal
that you don't like
it's tricky to settle on the perfect
balance between exploring and exploiting
however at the very least our L
algorithms provide a simple way to set
that balance okay it's starting to feel
like our L has a completely different
goal than what control engineers are
trying to do when we design control
systems but it is pretty much exactly
the same problem we're trying to figure
out how to design the controller or the
policy that map's the observed state of
the plant or the environment to the best
actuator commands the actions and when
we design a controller we're basically
doing a one-time policy update and one
of the ways that we can design an
optimal controller is by minimizing a
cost function like we do with lqr and
cost is just the negative of reward
so by maximizing the reward we're
solving the same problem as minimizing
cost the difference is that with
reinforcement learning the computer
tries to learn the optimal behavior over
time rather than have the designer
solved for it explicitly it's like the
adjustment mechanism in an adaptive
controller where it's tweaking the
parameters at each sample time in this
way we can essentially design a
controller which is a policy without
knowing anything about the system itself
and without having to solve any of the
traditional control problems we just let
the computer learn the right parameters
on its own through a process that you
can think of as fancy trial and error
now even with the learning algorithm
doing most of the work for us we can't
enter this process completely ignorant
we have to know several things before we
start and the first is that we need to
understand our system that we're trying
to control and determine whether it's
better to solve the problem with
traditional control techniques or with
reinforcement learning and if we choose
the learning path then we need to set up
the policy so that it has enough
parameters and in the right structure so
that it can be tweaked successfully it
won't do us any good if we're hoping to
control a multi-dimensional system but
only give it a single parameter we also
need to know what a successful result
would be and reward the controller for
doing well
this requires crafting a reward function
so that the learning algorithm
understands when it's getting better and
ultimately settles on the result that
you're actually looking for and third we
need to apply an efficient algorithm
that looks at the reward and
the system state and knows how to tweak
the parameters so that the process
converges with any reasonable amount of
time this is where we would also set the
parameters for exploration and
exploitation and the discounting on
future rewards in the next few videos
we'll expand on all of this by looking
at the workflow of reinforcement
learning in more detail we'll look at
the structure of the policy and
introduce neural networks and we'll talk
about how crafting the proper reward
function impacts your final result and
we'll look at a very high overview of
some interesting learning algorithms so
if you don't want to miss the next Tech
Talk video don't forget to subscribe to
this channel also if you want to check
out my channel control system lectures I
cover more control theory topics there
as well thanks for watching and I'll see
you next time