in the last video we focused mainly on
setting up the environment what the
environment includes and how rewards
from the environment are used to guide
the agent's behavior in this video we're
going to focus on setting up the agent
specifically the last three steps in the
RL workflow now to fully cover these
steps is easily several semesters worth
of material luckily that's not the goal
of this 17-minute video instead I want
to introduce a few topics at a high
level that will give you a general
understanding of what's going on so that
it makes more sense when you dive in
with other more complete sources so with
that being said in this video I will
address these two main questions one why
use neural networks to represent
functions as opposed to something else
like tables or transfer functions and
two why you may have to set up two
neural networks and how they complement
each other in a powerful family of
methods called actor critic so let's get
to it I'm Brian and welcome to a mat lab
tech
talk now last time we ended by
explaining how a policy is a function
that takes in state observations and
outputs actions and then I briefly
introduced the idea of why tables and
specifically defined functions aren't a
great solution in many cases this is
because tables are impractical when the
state and action space get really large
and also it's difficult to craft the
right function structure for complex
environments using neural networks
however can address both of these
problems a neural network is a group of
nodes or artificial neurons that are
connected in a way that allows them to
be be a universal function approximator
this means that given the right
combination of nodes and connections we
can set up the network to mimic any
input and output relationship and this
is good for us because we can use say
the hundreds of pixel values in a
robotic Vision system as the input into
this function and then the output could
be the actuator commands that drive the
robot's arms and legs even though the
function might be extremely complex we
know that there is a neural network of
some kind that can achieve it and if
you're not familiar with the m
mathematics of a neural network I highly
recommend the four-part series by three
blue one brown on the topic he provides
a fantastic visualization of what's
going on within the network so I'm going
to skip most of that discussion here I
do however want to highlight just a few
things on the left are the input nodes
and there's one for each input into the
function and on the right are the output
nodes in between our Columns of nodes
called hidden layers this network has
two inputs two outputs and two hidden
layers of three nodes each with a fully
connected Network there is a weighted
connection from each input node to each
node in the next layer and then from
those nodes to the layer after that and
then again until the output
nodes the value of a node is equal to
the sum of every input node times its
respective waiting Factor plus a bias
and we can perform this calculation for
every node in a layer and then write it
out in a compact Matrix form as a system
of linear
equations now if we calculated the
values of the nodes simply like this and
then fed them as inputs into the next
layer to perform the same type of linear
operations and then again to the output
layer you'll probably have a concern how
in the world could a bunch of linear
equations act as a universal function
approximator specifically how could it
represent a nonlinear function well
that's because I left out a step which
is probably one of the most important
aspects of an artificial neural network
after the value of a node has been
calculated
an activation function is applied that
changes the value of the node in some
way and two popular activation functions
are the sigmoid which squishes the node
value down between 0er and one and the
Rayo function which basically zeros out
any negative node values there are a
number of different activation functions
but what they all have in common is that
they are nonlinear which is critical to
making a network that can approximate
any function now as to why this is the
case I really like the explanations from
Brendan Fortuner and Michael neelon who
show how it can be demonstrated with the
ru and sigmoid activations respectively
I've linked to their blogs in the
description okay so now let's remind us
where we are we want to find a function
that can take in a large number of
observations and then transform them
into a set of actions that will control
some nonlinear environment and since the
structure of this function is often too
complex for us to solve for directly we
want to approximate it with a neural
network that learns the function over
time time and it's tempting to think
that we can just plunk any network in
and let loose a reinforcement learning
algorithm to find the right combination
of weights and biases and we're done
unfortunately as usual that's just not
the case we have to make a few choices
about our neural network ahead of time
in order to make sure that it's complex
enough to approximate the function we're
looking for but not too complex as to
make training impossible or impossibly
slow for example as we've already seen
we need to choose an activation function
and the number of hidden layers ERS and
the number of neurons in each layer but
beyond that we also have control over
the internal structure of the network
should it be fully connected like the
network I've drawn or should the
connection skip layers like in a
residual neural network should they loop
back on themselves to create internal
memory with recurrent neural networks
should groups of neurons work together
like with a convolutional neural network
and so on we have a lot of choices but
as with other control techniques there
isn't one right approach a lot of times
it comes down to starting with a network
structure that has already worked for
the type of problem you're trying to
solve and then just tweak it from there
now I keep saying we use these neural
Nets to represent a policy in the agent
but as for exactly what that means we
need to look at a high level description
of a few different classes of
reinforcement learning algorithms policy
function based value function based and
actor critic now the following will
definitely be an oversimplification but
if you're just trying to get a basic
understanding of the ways RL can be
approached I think this will help get
you started at a high level I think
policy function based learning
algorithms make a lot of sense because
we're trying to train a neural network
that takes in the state observations and
outputs an action and this is closely
related to what a controller is doing in
a control system we'll call this neural
network the actor because it is directly
telling the agent how to act and the
structure looks pretty straightforward
so now the question is how do we
approach training this type of network
and to get a general feel for how this
is done let's look at the Atari game
Breakout and if you're not familiar
breakout is this game where you're
trying to eliminate bricks using a
paddle to direct a bouncing ball this
game only has three actions move the
paddle left right or not at all and a
near continuous State space that
includes the position of the paddle the
position and velocity of the ball and
the location of the remaining Bricks now
for this example we'll say that the
observation is a screen capture of the
game feeding in one frame at a time and
therefore there's one input into our
neural net for every
pixel now with the network set there's
many approaches to training it but I'm
going to highlight one broad approach
that has a lot of variations and those
are policy gradient methods policy
gradient methods can work with a
stochastic policy which means that
rather than a deterministic take a left
or right the policy would output the
probability of taking a left or the
probability of taking a right a
stochastic policy takes care of the
exploration exploitation problem because
exploring is built into the
probabilities now when we learn the
agent just needs to update the
probabilities is taking a left a better
option than right push the probability
that you take a left in this state a
little bit higher then over time the
agent will nudge these probabilities
around in the direction that produces
the most
reward so how does it know whether the
actions were good or not well the idea
is this execute the current policy
collect reward along the way and update
the network to increase the probability
of actions that increase reward if the
paddle went left missing the ball
causing a negative reward then change
the neural network to increase the
probability of moving the paddle right
next time the agent is in that state
essentially it's taking the derivative
of each weight and bias with respect to
reward and adjusting them in the
direction of positive reward increase in
this way the learning algorithm is
moving the weights and biases of the
network to ascend up the reward slope
this is why the term gradient is used in
the name
now the math behind this is more than I
want to go into in this video but I'd
encourage you to read up on the policy
gradient theorem to see how it's
possible to find the gradient without
actually taking a derivative I've left a
link in the description now one of the
downsides of policy gradient methods is
that the naive approach of just
following the direction of steepest
Ascent can converge on a local Maxima
rather than Global they also can
converge slowly due to their sensitivity
to noisy measurements which happens for
example when it takes a lot of
sequential actions to receive a reward
and then the resulting cumulative reward
has high variance between episodes
imagine having to string together
hundreds or thousands of sequential
actions before the agent ever receives a
single reward you could see how time
consuming it could be to train an agent
using that extremely sparse reward
system keep that in mind and we're going
to come back to that problem by the end
of this video all right let's move on to
Value function based learning algorithms
for these let's start with an example
using the popular grid world as our
environment in this environment there
are two discrete State variables the X
grid location and the Y grid location
and there's only one state in Grid world
that has a positive reward and then the
rest have negative rewards the idea is
that we want our agent to collect the
most reward which means getting to that
positive reward in the fewest moves
possible and the agent can only move one
square at a time either up down left or
right and to us powerful humans it's
easy to see exactly which way to go to
get the reward however we have to keep
in mind that the agent knows nothing
about the environment it just knows that
it can take one of four actions and it
gets its location and reward back from
the environment after it takes that
action with a value function based agent
a function would take in the state and
one of the possible actions from that
state and output the value of taking
that action the value is the sum of the
total discounted rewards from that state
and on like we talked aled about in the
first video in this way the policy would
simply be to check the value of every
possible action and then choose the one
with the highest value and we can think
of this function as a Critic since it's
looking at the possible actions and it's
criticizing the agent's choices and
since there's a finite number of states
and actions in Grid world we could use a
lookup table to represent this function
and this is called the Q table where
there's a value for every state and
action pairing so how does the agent
learn these values well at first we can
initialize it to all zeros so all
actions look the same to the agent this
is where the exploration rate Epsilon
comes in and it allows the agent to just
take a random action after it takes that
action it gets to a new state and
collects the reward from the environment
the agent uses that reward that new
information to update the value of the
action that it just took and it does
that using the famous Bellman
equation the Bellman equation allows the
agent to solve the Q table over Time by
breaking up the whole problem into
multiple simpler steps so rather than
solving the true value of the state
action pair in one step through dynamic
programming the agent will update the
state action pair each time it's visited
let me try to describe this equation in
words so that hopefully it makes some
sense after the agent has taken an
action it receives a reward now value is
more than just the instant reward from
an action it's the maximum expected
return into the future therefore the
value of the state action pair is is the
reward that the agent just received plus
how much reward the agent expects to
collect going forward and we discount
the future rewards by gamma so that as
we talked about the agent doesn't rely
too much on rewards far in the future
this is now the new value of the state
action pair sa and so we compare this
value to what was in the Q table to get
the error or how far off the agent
prediction was the error is multiplied
by a learning rate and the resulting
Delta value is added to the old estimate
now when the agent finds itself back in
the same state a second time it's going
to have this updated value and it will
then tweak it again when it chooses the
same action and it'll keep doing this
over and over again until the true value
of every state action pair is
sufficiently known to exploit the
optimal
path okay now let's extend this idea to
an inverted pendulum where there's still
two states angle and angular rate except
now the states are
continuous value function can handle
continuous State spaces except not with
a lookup table so we're going to need a
neural network and the idea is the same
we input the state observations and in
action and the neural network returns a
value you can see that this setup
doesn't work well for continuous action
spaces because how could you possibly
try every infinite action and find the
maximum value even for a large action
space this becomes computationally
expensive but I'm getting ahead of
myself for now let's just say that the
action space is discret and the agent
can choose one of five possible torque
values that seems reasonable so here's
the idea when you feed the network the
observe State and an action it'll return
a value and our policy would again be to
just check the value for every possible
action and then take the one with the
highest value and just like with grid
World our neural network would be
initially set to Just Junk values and
the learning algorithm would use a
version of the Bellman equation to
determine what the new value should be
and then update the weights and biases
in the network accordingly and once the
agent has explored enough of the state
space then it's going to have a good
approximation of the value function and
can select the optimal action given in E
State and that's pretty neat right but
we did encounter a drawback in that the
action space needs to be relatively
small but often in control problems we
have a continuous action space like
being able to apply a continuous range
of torqus to an inverted pendulum
problem this brings us to a merge ing of
the two techniques into a class of
algorithms called actor critic the actor
is a network that is trying to take what
it thinks is the best action given the
current state just like we had with the
policy function method and the critic is
a second Network that is trying to
estimate the value of the state and the
action that the actor took like we had
with the value base methods now this
works for continuous action spaces
because the critic only needs to look at
a single action the one that the actor
took and not try to find the best action
by evaluating all of them here's how it
basically works the actor chooses an
action in the same way that a policy
function algorithm would and it's
applied to the environment the critic
estimates what it thinks the value of
that state and action pair is and then
it uses the reward from the environment
to determine how accurate its value
prediction was the error is the
difference between the new estimated
value of the previous state and the old
value of the previous state from the
critic Network the new estimated value
is based on the received reward and the
discounted value of the current state
and it can use this error as a sense of
whether things went better or worse than
the critic expected the critic uses this
error to update itself in the same way
that the value function would so that it
has a better prediction the next time
it's in this state the actor also
updates itself with the response from
the critic and if available the error
term so that it can adjust its
probabilities of taking that action
again in the future that is the policy
now ascends the reward slope in the
direction that the critic recommends
rather than using the rewards directly
so both the actor and the critic are
neural networks that are trying to learn
the optimal Behavior the actor is
learning the right actions using
feedback from the critic to know what a
good action is and what is bad and the
critic is learning the value from the
received rewards so that they can
properly criticize the action that the
actor takes this is why you might have
to set up two neural networks in your
agent each one play plays a very
specific
role with actor critic methods the agent
can take advantage of the best parts of
policy function and value function
algorithms actor critics can handle both
continuous States and action spaces and
speed up learning when The Returned
reward has high variance now I'm going
to show you an example of using an actor
critic based algorithm to get a bipedal
robot to walk in the next video but
before I close out this video I want to
briefly discuss the last step in the RL
workflow deploying the algorithm on the
target Hardware so far the agent has
learned offline by interacting with the
simulated environment but once the
policy is sufficiently optimal then the
learning would stop and the static
policy would be deployed onto the target
just like you would any developed
control law however we also have the
ability to deploy the reinforcement
learning algorithms along with the
policy and can continue learning on the
target with the actual environment and
this is important for environments that
are hard to model accurately or that are
slowly changing over time and therefore
the agent needs to continue to learn
occasionally so that it can adjust to
those
changes okay that's where I'm going to
leave this video for now and if you
don't want to miss any future Tech talk
videos don't forget to subscribe to this
channel also if you want to check out my
channel control system lectures I cover
more control theory topics there as well
thanks for watching and I'll see you
next time