what is up everybody today you're going
to learn how to go from a paper to a
fully functional implementation of deep
deterministic policy gradients if you're
not familiar with deep deterministic
policy gradients or ddpg for short it is
a type of deep reinforcement learning
that is used in environments within
continuous action spaces you see most
environments have discrete action spaces
this is the case with say the atari
library say like breakout or space
invaders where the agent can move left
right it can shoot
but it can move left right and shoot by
fixed discrete intervals fixed amounts
right in other environments like say
robotics
the robots can move a continuous amount
so it can move in anywhere from you know
zero to one minus one to plus one
anything along a continuous number
interval and this poses a problem for
most deep reinforcement learning methods
like say q learning which works
spectacularly well in discrete
environments but cannot tackle
continuous action spaces now if you
don't know what any of this means don't
worry i'm going to give you the rundown
here in a second but for this set of
tutorials you're going to need to have
installed the openai gym you'll need
python 3.6 and you also need tensorflow
and pi torch other packages you'll need
include
matpotlib to handle the plotting of the
learning curve which will allow us to
see the actual learning of the agent as
well as numpy to handle your typical
vector operations
now
here i'll give you a quick little
rundown of reinforcement learning so the
basic idea is that we have an agent that
interacts with some environment and
receives a reward the rewards kind of
take the place of labels and supervised
learning in that they tell the agent
what is good what is it that it is
shooting for in the environment and so
the agent will attempt to maximize the
total rewards over time
uh by solving something known as the
bellman equation we don't have to worry
about the actual mathematics of it but
just so you know for your future
research the algorithms are typically
concerned with solving the bellman
equation which tells the agent the
expected future returns assuming it
follows something called its policy so
the policy is the probability that the
agent will take a set of actions given
it's in some state s it's basically
probability distribution
now many types of algorithms such as q
learning will attempt to solve the
bellman equation by finding what's
called the value function the value
function or the action value function in
this case maps the current state and set
of possible actions to the expected
feature returns the agent expects to
receive so in other words the agent says
hey i'm in some state meaning some
configuration of pixels on the screen in
the case of the atari gym atari library
for instance
and says okay if i take one or another
action what is the expected future
return assuming that i follow my policy
actor critic methods are slightly
different and that they attempt to learn
the policy directly and
recall the policy is a probability
distribution that tells the agent what
the probability of selecting an action
is given it's in some state s
so these two algorithms have a number of
strengths between them
and deep deterministic policy gradients
is a way to marry the strengths of these
two algorithms into something that does
really well for discrete action sorry
continuous action spaces you don't need
to know too much more than that
everything else you need to know i'll
explain in their respective videos so in
the first video you're going to get to
see how
i go ahead and read papers and then
implement them
on the fly
and in the second video you're going to
see
the implementation of deep deterministic
policy gradients in pi torch in a
separate environment both these
environments are in
both these environments are continuous
and so they will demonstrate the power
of the algorithm quite nicely
you don't need a particularly powerful
gpu but you do need some kind of gpu to
run these
as it does take a considerably long time
even on a gpu so you will need at least
a
like say a maxwell class gpu or above so
something from the 700 series on nvidia
side unfortunately neither of these uh
frameworks really work well with amd
cards so if you have those you would
have to figure out some sort of clues to
get the opencl implementation to
transcompile to cuda that's just a
technical detail
i don't have any information on that so
you're on your own sorry so this is a
few hours of content grab a snack a
drink and watch this at your leisure
it's best to watch it in order i
actually did the videos in a separate
order a reverse order on my channel just
so i could get it out
so i did the implementation of pytorch
first and then the video on implementing
the paper in tensorflow second but it
really is best for a new audience to go
from the
paper paper video to the pie torch video
so i hope you like it leave any comments
questions suggestions issues down below
i'll try to address as many as possible
you can check out the code for this on
my github and you can find many more
videos like this on my youtube channel
machine learning with phil i hope you
all enjoy it let's get to it
what is up everybody in today's video
we're going to go from the paper on deep
deterministic policy gradients all the
way into a functional implementation in
tensorflow so you're going to see how to
go from a paper to a real world
implementation all in one video grab a
snack a drink because this is going to
take a while let's get started
so the first step in my process really
isn't anything special i just read the
entirety of the paper
of course starting with the abstract the
abstract tells you what the paper is
about at a high level it's just kind of
an executive summary
introduction is where the authors will
pay homage to other work in the field
kind of set the stage for what
is going to be presented in the paper as
well as the need for it
uh the background kind of expands on
that and you can see here it gives us a
little bit of mathematical equations
and you will get a lot of useful
information here this won't talk too
much about useful nuggets on
implementation but it does set the stage
for the mathematics you're going to be
implementing
which is of course critical for any deep
learning or in this case deep
reinforcement learning paper
implementation
the algorithm is really where all the
meat of the problem is it is in here
that they lay out the exact steps you
need to take to
implement the the algorithm right that's
why it's titled that way so uh this is
the section we want to read most
carefully and then of course they will
typically give a table where they
outline the actual algorithm and
oftentimes
if i'm in a hurry i will just jump to
this
because i've done this enough times that
i can read this what is called
pseudocode if you're not familiar with a
pseudocode it's just an english
representation of computer code
so we will
typically use that when we outline a
problem and it's often used in papers of
course
so
typically i'll start here reading it and
then work backward by reading through
the paper to see what i missed
but of course it talks about the
performance across a whole host of
environments and of course all of these
have in common that they are continuous
control so
uh
what that means is that the
action space is a vector whose elements
can vary
on a continuous real number line instead
of having discrete actions of 0 1 2 3 4
5
so that is the really motivation behind
deep deterministic policy grades is that
allows us to use deep reinforcement
learning to tackle these types of
problems
and in today's video we're going to go
ahead and tackle the i guess pendulum
swing up also called the pendulum
problem
reason being is that
while it would be awesome to start out
with something like the
bipedal walker you never want to start
out with maximum complexity you always
want to start out with something very
very small and then scale your way up
and the reason is that you're going to
make mistakes and it's most easy to
debug most
quick to debug very simple environments
that execute very quickly so
the pendulum problem only has i think
three elements in its state vector and
only a single action so or maybe it's
two actions i forget but either way it's
very small problem uh relative to
something like the bipedal walker or
many of the other environments you could
also use the
continuous version of the card poll or
something like that that would be
perfectly fine i've just chosen the
pendulum for this particular video
because we haven't done it before
so it's in here that they give a bunch
of plots of all of the
performance of their algorithm of the
various sets of constraints placed upon
it and and different implementations so
you can get an idea and one thing you
notice right away um it's always
important to look at plots because they
give you a lot of information visually
right it's much easier to gather
information from plots in his text you
see that right away they have a scale of
one so that's telling you it's relative
performance and you have to read the
papers relative to what
i don't like that particular approach
they have similar data in a table form
and in here you see a whole bunch of
environments they used and there's a
broad broad variety they wanted to show
that the algorithm has a wide arena of
applicability which is a typical
technique and papers they want to show
that this is relevant right if they only
showed a single environment
people reading it would say well that's
all well and good you can solve one
environment but what about these dozen
other environments
right and part of the motivation behind
reinforcement learning is generality can
as can we model real learning and
biological systems such that it mimics
the generality of biological learning
one thing you notice right away is that
these numbers are not actual scores so
that's one thing i kind of take note of
and
causes me to raise an eyebrow so
you have to wonder the motivation behind
that why would the authors express
scores in a ratios a couple different
reasons one is because they want to
just to make all the numbers look
uniform maybe the people reading the
paper
wouldn't be familiar with each of these
environments so they don't know what a
good score is and that's a perfectly
valid reason another possibility is they
want to hide poor performance i don't
think that's going on here but it does
make me raise my eyebrow whenever i see
it
one exception is the torx which is a
totally open
race car simulator environment i don't
know if we'll get to that on this
channel that would be a pretty cool
project
but that would take me a few weeks to
get through
but right away you notice that they have
a whole bunch of environments these
scores are all relative to one and one
is the score that the agent gets on a
planning algorithm which they also
detail later on
so those are the results
and they talk more about i don't think
we saw the
headline but they talk about related
work which talks about other algorithms
that are similar and their shortcomings
right they don't ever want to talk up
other algorithms you always want to talk
up your own algorithm to make yourself
sound good
you know why else would you be writing a
paper in the first place and of course a
conclusion to tie everything together
references
i don't usually go deep into references
if there is something that i feel i
really really need to know i may look at
a reference but i don't typically bother
with them if you were a phd student then
it would behoove you to go into the
references because you must be an
absolute expert on the topic and for us
we're just you know hobbyists i'm a
youtuber so i don't go into too much
depth with the background information
and the next most important bit of the
paper are the experimental details and
it is in here
that it gives us the
parameters and architectures for the
networks so this is where if you saw my
previous video where i did the
implementation of ddpg in pi torch in
the continuous lunar lander environment
this is where i got
most of this stuff it was almost
identical with a little bit of tweaking
i left out some stuff from this paper
but
pretty much all of it came from here
in particular the
hidden layer sizes 400 and 300 units
as well as the initialization of the
parameters from uniform distribution of
the given ranges
so
just to recap this was a really quick
overview of the paper
just showing my process of what i look
at
the
most important parts are the
details of the algorithm as well as the
experimental details
so as you read the paper
like i said i gloss over the
introduction because i don't really i
kind of already understand the
motivation behind it i get the idea it
says it basically tells us that um you
can't really handle
just continuous action spaces with dq
networks
we already know that
and it says you know you can discretize
the state space but then you end up with
really really huge actions sorry you can
discretize the action space but then you
end up with a you know whole bootloader
actions you know what is it 2187 action
so it's intractable anyway and they say
what we present you know a model free of
policy algorithm
and uh then it comes down to this
section where it says the network is
trained off policy with samples from a
replay buffer to minimize correlations
very good and train with the target
queue network to give consistent targets
during temporal difference backups so in
this work we make use of the same ideas
along with batch normalization so this
is a key chunk of text and this is why
you want to read the whole paper because
sometimes you'll embed stuff in there
that you may not otherwise catch so
uh as i'm reading the paper what i do is
i take notes and you can do this in
paper you can do it in you know text
document in this case we're using the
editor so that way i can show you what's
going on and it's a natural place to put
this stuff because that's where you can
implement the code anyway let's hop over
to the editor
and you'll see what i take notes
so right off the bat we always want to
be thinking in terms of what sort of
classes and functions will we need to
implement this algorithm so the paper
mentioned a
replay buffer as well as a target queue
network so
the target queue network
for now we don't really know what it's
going to be but we can write it down so
we'll say we'll need a
replay buffer class
and
need a class for a target queue network
now i would assume that if you're going
to be implementing a paper of this
advanced uh difficulty you'd already be
familiar with q learning where you know
that the target network is just another
instance of a generalized network uh the
difference between the target and
evaluation networks are you know the way
in which you update their weights so
right off the bat we know that we're
going to have a single class at least
one if you know something about actor
critic methods you'll know that you'll
probably have two different classes one
for an actor one for a critic because
those two architectures are generally a
little bit different
but what do we know about q networks we
know that q networks are
um
state action value functions right
they're not just value functions so the
critic in the actor critic methods is
just a state value function in general
whereas here we have a q network which
is going to be a function of the state
and action so we know that it's a
function of s and a so we know right off
the bat it's not the same as a critic
it's a little bit different
and it also said
well we will use
we will use batch norm so batch
normalization is just a way of
normalizing inputs to prevent divergence
in a model i think it was discovered in
2015 2014 something like that
uh so they will use that so we'll need
that in our network so we know at least
right off the bat a little bit of an
idea of what the network is going to
look like so let's go back to the paper
and see what other little bits of
information we can glean from the text
before we take a look at the algorithm
uh reading along
we can say
blah blah bl
features simplicity it requires only a
straightforward actor critic
architecture and very few moving parts
and then they talk it up and say can
learn policies that exceed the
performance of the planner you know the
planning algorithm
uh even learning from pixels which we
won't get to in this particular
implementation
so then okay no real other nuggets there
the background talks about the
mathematical structure of the algorithm
so
uh
this is really important if you want to
have a really deep in-depth knowledge of
the topic uh if you already know enough
about the background you would know that
you know the formula for discounted
future rewards you should know that if
you've done
a whole bunch of reinforcement learning
algorithms if you haven't then
definitely read through this section
to get the full idea of the background
and the motivation behind the
mathematics
other thing to note is it says the
action value function is used in many
algorithms we know that from dbq
learning
and then it talks about the re cursive
relationship known as the bellman
equation that is known as well
other thing to note what's interesting
here and this is the next nugget is if
the target policy is deterministic we
can describe it as a function mu
and so you see that in the remainder of
the paper like in the algorithm they do
indeed make use of this parameter mu so
that tells us right off the bat that our
policy is going to be deterministic now
if you have
you could probably guess that from the
title right
deep deterministic policy gradients
right so you would guess from the name
that the policy is deterministic but
what does that mean exactly so a
stochastic policy is one in which the
software maps the
probability of taking an action to a
given state so you input a set of state
and out comes a probability of selecting
an action and you select an action
according to that probability
distribution so that right away bakes in
a solution to the explore exploit
dilemma so long as all probabilities are
finite right so so as long as a
probability of taking an action for all
states doesn't go to zero there is some
element of exploration involved in that
algorithm
q learning handles the explore exploit
dilemma by using epsilon greedy action
selection where you have a random
parameter that tells you
how often to select a random number
sorry random action and then you select
a greedy action the remainder of the
time of course policy gradients don't
work that way they typically use a
stochastic policy but in this case we
have a deterministic policy so you have
to wonder right away
okay we have a deterministic policy how
are we going to handle the explore
exploit dilemma so let's go back to our
text editor and
make a note of
that we just want to say that the
the policy is deterministic
how to handle
explore exploit
and that's a critical question right
because if you only take
what are perceived as the greedy actions
you never get a really good coverage of
the parameter space of the problem and
you're going to converge on a
sub-optimal strategy so this is a
critical question we have to answer in
the paper so let's head back to the
paper and see how they handle it
so we're back in the paper and you can
see the reason they introduced that
deterministic policies to avoid an inner
expectation or maybe that's just a
byproduct i guess it's not accurate to
say that's the reason they do it
but what's neat is it says the
expectation depends only on the
environment means it's possible to learn
q to the mu meaning q is a function of
mu off policy using transitions which
are generated from a different
stochastic policy beta so right there we
have off policy learning which is say
explicitly
with a stochastic policy so we are
actually going to have two different
policies in this case
so then this already answers the
question of how we go from a
deterministic policy to solving the
export dilemma and the reason is that
we're using a stochastic policy to learn
the greedy policy or a sorry a purely
deterministic policy
and of course they talk about the
parallels with q learning because there
are many between the two algorithms
and you get to the loss function which
is of course critical to the algorithm
and
this y of t parameter then of course
they talk about
what q learning has been used for they
use they make mention of
deep neural networks which is of course
what we're going to be using that's
where the deep comes from uh and talks
about the atari games which we've talked
about on this channel as well
um
and uh importantly they say
uh
the the two changes that they introduced
in q learning which is the concept of
the replay buffer and this the target
network uh which of course they already
mentioned before they're just
reiterating and reinforcing what they
said
that's why we want to read the
introduction and background type
material to get
a solid idea of what's going to happen
so now we get to the algorithmic portion
and this is where all of the magic
happens so
they again reiterate that it's not
possible to apply q learning to
continuous action spaces
uh
because you know reasons right it's
pretty obvious you have an infinite
number of actions that's a problem
um
then they talk about the deterministic
policy gradient algorithm which we're
not going to go too deep into right for
this for this video we don't want to do
a full thesis we don't want to do a full
doctoral dissertation on the field we
just want to know how to implement it
and get moving so
this goes through and gives you
an update for the gradient
of this parameter j
and gives it in terms of the gradient
of q which is the state action value
function and the gradient of the
policy the deterministic policy mu
other thing to note here is that this
gradients these gradients are over two
different parameters so the gradient of
q is with respect to the actions
such that the action a equals mu of s of
t so what this tells you is that
q is actually a function not just of the
state but is
intimately related to that policy mu so
it's not um
it's not an action chosen according to
an arg max for instance it's an action
short chosen according to the output of
the other network
and for the update of mu it's just the
gradient with respect to the weights
which you would kind of expect
so
they talk about another algorithm nfqca
i don't know what that is honestly mini
batch version blah blah our contribution
here is to provide modifications to dpg
inspired by the success of dqn which
allow it to use neural network function
approximators to learn in large state
and action spaces online we call it ddpg
very creative
as they say again we use a replay buffer
to address the issues of correlations
between samples generated on subsequent
steps within an episode
finite size cache size r
transition sample from the environment
so we know all of this so if you don't
know all of it what you need to know
here is that you have state action
reward and then new state transitions so
what this tells the agent is
started in some state s
took some action received some reward
and ended up in some new state why is
that important it's important because in
in anything that isn't dynamic
programming you're really trying to
learn the
state probability distributions you're
trying to learn the probability of going
from one state to another and receiving
some reward along the way if you knew
all those beforehand then you could just
simply solve a set a very very large set
of equations for that matter to arrive
at the optimal solution right if you
knew all those transitions you say okay
if i start in this state
and take some action i'm going to end up
in some other state with certainty
then you'd say well what's the most
advantageous state what state is going
to give me the largest reward and so you
could kind of construct some sort of
algorithm for traversing the set of
equations to maximize your reward over
time now of course you often don't know
that and that's the point of the replay
buffer is to learn that through
experience and interacting with the
environment
and it says when the replay buffer was
full all the samples were discarded okay
that makes sense it's finite size it
doesn't grow indefinitely
at each time step actor and critic are
updated by sampling a mini batch
uniformly from the buffer so it operates
exactly according to q learning
it does a uniform sampling
random sampling of the buffer and uses
that to update the actor and critic
networks
what's critical here is that
combining this statement
with the topic of the previous paragraph
is that when we write our replay buffer
class it must sample
states at random so what that means is
you don't want to sample
a sequence of subsequent steps and the
reason is that there are large
correlations between those steps right
as you might imagine and those
correlations can cause you to get
trapped in little knicks nooks and
crannies of parameter space and really
cause your algorithm to go wonky so you
want to sample that uniformly so that
way you're sampling across many many
different episodes to get a really good
idea of the
i guess the breadth of the parameter
space to use kind of loose language
and then it says directly implementing q
learning with neural networks prove to
be unstable many environments and
they're they're going to talk about
using the the target network okay but
modified for actor critic
using soft target updates rather than
directly copying the weights so in q
learning we directly copy the weights
from the evaluation to the target
network
here it says we create a copy of the
actor and critic networks q prime and mu
prime respectively they're used for
calculating the target values
the weights of these target networks are
then updated by having them slowly track
the learned networks theta prime goes to
theta
theta times tau plus one minus tau times
theta prime with tau much much less than
one
this means that the target values are
constrained to change slowly greatly
improving the stability of learning
okay so this is our next little nugget
so let's head over to the paper and make
to our text editor and make note of that
what we read was that the we have two
not in caps we don't want to shout we
have
two networks
um
uh target networks
sorry we have
two actor and two critic networks
a target
for each
updates are soft according to
theta equals tau times theta
plus
one minus tau
times uh
theta
prime
so
sorry this should be theta prime so this
is the update rule for the
parameters of our target networks and we
have two target networks one for the
actor and one for the critic so we have
a total of four
deep neural networks and so this is why
the algorithm runs so slowly even on my
beastly rig
it runs quite slowly even in the lunar
lander and a continuous lunar lander
environment
i've done the bipedal walker and it took
about 20 000 games to get something that
approximates a decent score so this is a
very very slow algorithm
and that 20 000 games took uh i think
about a day to run so quite slow um
but nonetheless quite powerful the only
method we have so far of implementing
uh deep reinforcement learning and
continuous control environments so hey
you know
beggars can't be choosers right
but we know
just to recap that we're going to use
four networks
two that are on policy into off policy
and the updates are going to be soft uh
with
tau much less than one if you're not
familiar with mathematics
this double less than or double greater
than sign means much less than or much
greater than
uh respectively so
what that means is that tau is going to
be of order .01 or smaller right 0.1
isn't much smaller that's kind of
smaller 0.01 i would consider much
smaller they use we'll see in the
in the
details we'll see what value they use
but you should know that it's of order
0.01 or smaller
and the reason they do this is to allow
the
updates to happen very slowly to get
good conversions as they said in the
paper so let's head back to the paper
and see what other nuggets we can glean
before getting to the
outline of the algorithm
and then in the very next sentence they
say this simple change moves the
relative unstable problem of learning
the action value function closer to the
case of supervised learning a problem
for which a robust solution exists
we found that having both the target mu
prime and q prime was required to have
stable targets y i in order to
consistently train the critic without
divergence
this may slow learning since the target
networks delay the propagation of value
estimates however in practice we found
this was always greatly outweighed by
the stability of learning and i found
that as well you don't get a whole lot
of diversions but it does take a while
to train
then they talk about learning in low
dimensional and higher dimensional
environments
and they do that to talk about
the need for feature scaling so
one approach to the problem
which is the
ranges of
variations in parameters right so in
different environments like in the
mountain car you can go from
plus minus 1.6 like minus 1.6 to 0.4
something like that and the velocities
are plus or minus 0.07 so you have a
two-order magnitude variation there in
the parameters that's kind of large even
in that environment and then when you
compare that to other environments
where you can have parameters that are
much larger in the order hundreds you
can see that there's a pretty big issue
with
the
scaling of the inputs to the neural
network which we know from our
experience
that neural networks are highly
sensitive to the scaling between inputs
so
it says their solution that problem is
to manually scale the features so
they're in similar
across
environments and units
and they do that by using
batch normalization and it says this
technique normalizes each dimension
across the samples in a mini batch dev
unit mean and variance and also
maintains a running average of the mean
and variance used for normalization
during testing during exploration and
evaluation so in our case
training and testing are slightly
different than in the case of supervised
learning so supervised learning you
maintain different data sets or
shuffled subsets of it of a single data
set to do training and evaluation
and of course in the evaluation phase
you'd perform no
wait updates of the network you just see
how it does based on the training
in
reinforcement learning you can do
something similar where you have a set
number of games where you train the
agent to achieve some set of results and
then you turn off the learning and
allow it to just choose actions based
upon whatever policy it learns and if
you're using batch normalization in pi
torch in particular there are
significant differences in
how batch normalization is used in the
two different uh phases so you have to
be explicit in
uh
setting training or evaluation mode
in particular in pi torch they don't
track statistics in evaluation mode
which is why when we wrote the
ddpg algorithm in pi torch we had to
call the eval and train functions so
often
okay so we've already established we'll
need batch normalization so everything's
kind of starting to come together we
need a replay network batch
normalization we need
uh four networks right we need two we
need two
each of a target
of an actor and two each of a critic so
half of those are going to be used for
on policy and have them are going to be
used for off policy for the targets
and then it says we scroll down
a major challenge
of learning in continuous action spaces
is exploration
an advantage of off policy algorithms
such as ddpg is that we can treat the
problem of exploration independently
from the learning algorithm we
constructed an exploration policy mu
prime by adding noise sampled from a
noise process n to our actor policy
okay so right here is telling us
what the basically the target
uh
actor function is it's mu prime is
basically
mu plus some noise n
n can be chosen to suit the environment
as detailed in the supplementary
materials we use in orenstein woolenbeck
process to generate temporally
correlated exploration for exploration
efficiency and physical control problems
with inertia
if you're not familiar with physics
inertia just means the
um
tendency of stuff to stay in motion it
has to do with like
environments that move like the walkers
the cheetahs stuff like that the ants
okay so we've kind of uh got one other
nugget to add to our text editor let's
head back over there and write that down
okay so
uh the
target
actor
is
just the
um evaluation we'll call it that for
lack of a better word evaluation actor
plus some noise process
they used
ornstein uh uh i don't think i spelled
that correctly um
we'll need to look that up
uh that
i've already looked it up um
my background is in physics so it made
sense to me it's basically
a
noise process that models the motion of
of browning particles which are just
particles that move around
uh under the influence of their
interaction with other particles in some
type of medium like a lossless medium
like a perfect food or something like
that
and in the orange seed ulumbat case they
are temporally correlated meaning each
time step is related to the time step
prior to it and i haven't thought about
it before but that's probably important
for the case of markov decision
processes right so in mdps the
current state is only related to the
prior state and the action taken you
don't need to know the full history of
the environment
uh so i wonder if that was chosen that
way if there's some underlying physical
reason for that uh just kind of a
question of cursing me off the top of my
head i don't know the answer to that if
someone knows play drop the answer in
the comments i would be very curious to
see the answers so
uh we have enough nuggets here so just
to summarize we need a replay buffer
class
we'll also need a class for the noise
right
so we'll need a class for noise a class
for the replay buffer
we'll need a class for the target q
network and we're going to use batch
normalization
the policy will be deterministic so what
that means in practice is that the
policy will output the actual actions
instead of the probability of selecting
the actions so
the
policy will be limited by whatever the
action space of the environment is so we
need some way of taking that into
account
so
so deterministic policy means
outputs the actual action instead
of a probability
we'll need a way to bound the actions to
the
environment
environment limits
and of course these notes don't make it
into the final code these are just kind
of things you think of as you are
reading the paper uh you would want to
put all your questions here
uh i don't have questions since i've
already implemented it but this is kind
of my thought process as i went through
it the first time
as best as i can model it after having
finished the problem
and you can also use a sheet of paper
that's some kind of magic about writing
stuff down on paper but we're going to
use the code editor because i don't want
to use an overhead projector to show you
guys a freaking sheet of paper this
isn't grade school here so
let's head back to the paper and
uh take a look at the actual algorithm
to get some real sense of what we're
going to be implementing
the
the results really aren't super
important to us yet uh we will use that
later on if we want to debug the model
performance but the fact that they
express it relative to a planning
algorithm makes it difficult right
so
scroll down to the data really quick
so
they give another thing to note i didn't
talk about this earlier but i guess now
is a good time
is
the stipulations on this tr on this
performance data says performance after
training across all environments for at
most 2.5 million steps so i said earlier
i had to train the bipedal walker for
around 20 000 games
that's around um
times
i think that's around
about about two and a half million steps
or so i think it was actually on fifteen
thousand steps so maybe around three
million steps something like that
we report both the average and best
observed across five runs so
why would they use five runs so if this
was a super duper algorithm and
which none of them are this isn't a
slight on their algorithm this isn't
meant to be snarky or anything what it
tells us is that they had to use five
runs because there is some element of
chance involved so
you know in one problem with deep
learning is the problem of
uh replicability right it's hard to
replicate other people's results if
particularly if you use system clocks as
seeds for random number generators right
using the system clock to to see the
random number generator guarantees that
if you run the simulation at even a
millisecond later right
that you're going to get different
results because you're going to be
starting with different sets of
parameters now you will get
qualitatively similar results
right you'll be able to repeat
the
the general idea of the experiments but
you won't get the exact same results
it's kind of what it's an objection to
the whole deep learning phenomenon it
makes it kind of not scientific but
whatever it works
uh has an enormous success so we won't
quibble about semantics or you know
philosophical problems but we just need
to know
for our purposes that
even these people that invented the
algorithm had to run it several times to
get some idea of what was going to
happen because the algorithm is
inherently probabilistic and so they
report averages and best case scenarios
so
that's another little tidbit and
they included
results for both the low dimensional
cases where you receive just a state
vector from the environment as well as
the pixel inputs
we won't be dealing with pixel inputs
for this particular
video but maybe we'll get to them later
i'm trying to work on that as well
so these are the results and the
interesting tidbit here is that it's
probabilistic it's going to take five
runs so okay fine
other than that we don't really care
about results uh for now we'll take a
look later but
uh that's not really our concern at the
moment so
now we have a series of questions we
have answers to all those questions we
know how we're gonna handle the explore
exploit dilemma we know the purpose of
the target networks we know how we're
going to handle the noise we know how
we're going to handle the replay buffer
um
and we know what the policy actually is
going to be it's the actual prop it's
the actual actions the agent is going to
take so we know a whole bunch of stuff
so it's time to look at the algorithm
and see how we fill in all the details
so
randomly initialize a critic network and
actor network with weights
theta
super q theta super mu so
this is handled by whatever library you
use you don't have to manually
initialize weights but we do know from
the supplemental materials that they do
constrain these updates to be within
sorry these initializations to be within
some range so
put a note in the back of your mind that
you're going to have to constrain these
a little bit
and then it says initialize target
network q prime and mu prime with
weights
that are
equal to the original network so
theta super q prime gets initialized
with theta super q and theta mu prime
gets initialized with theta super mu so
uh
we will be updating the weights right
off the bat for the target networks with
the evaluation networks
an initializer replay buffer r now this
is an interesting question how do you
initialize that replay buffer so i've
used a couple different methods
uh you can
just initialize it with all zeros and
then if you do that
when you perform the learning you want
to make sure that you have
a
number of memories that are greater than
or equal to the
mini batch size of your training so that
way you're not sampling the same states
more than once right if you
have 64 memories in a batch that you
want to sample but you only have 10
memories in your replay buffer then
you're going to sample
let's say 16 memories and you're going
to sample each of those memories four
times right so then that's no good so
the question becomes if you update if
you initialize your replay buffer with
zeros then you have to make sure that
you don't learn until you exit the
warm-up period where the warm-up period
is just a number of steps equal to your
replay buffer uh your buffer sample size
or you can initialize it with the actual
environmental play now this takes quite
a long time uh you know the replay
buffers are border a million so if you
let the uh the algorithm take in lane
steps at random then it's going to take
a long time i always use zeros and then
you know just wait until the agent fills
up the mini batch size of memories
uh just a minor detail there
then it says
for some number of episodes do so a for
loop initialize a random process n for
action exploration so this is something
now reading it i actually made a little
bit of a mistake so
uh in my
previous implementation i didn't reset
the noise process at the top of every
episode uh so it's explicit here i must
have missed that line um
and i've looked at other people's code
some do some don't but it worked uh
within uh how many episodes was it uh
within a uh
under a thousand episodes the agent
managed to beat the continuous orlando
environment so is that critical maybe
not um and i think i mentioned that in
the video
receive your initial state observation
s1 so for each step of the episode t
equals one to capital t due select the
action a sub t equals mu
the policy
plus n sub t according to the current
policy and exploration noise okay
so that's straightforward just use just
feed the state forward what does that
mean it means feed the state forward
through the network
receive the
vector output of the action and add some
noise to it okay
execute that action and receive and
observe reward and new state
simple
store the transition you know the old
state action reward a new state in your
replay buffer are okay that's
straightforward
uh each time step sample a random mini
batch of n transitions
from the replay buffer
and then you want to use that set of
transitions to set y sub i equals so i
is
sorry having difficulties here so i is
each step of
that uh is each element of that mini
batch of transitions
so you want to basically loop over that
set or do a vectorized implementation
looping is more straightforward that's
what i do i always opt for the most
straightforward and not necessarily most
efficient way
of doing things the first time through
because you want to get it working first
and worry about implementation uh sorry
efficiency later
so set y sub i equals r sub i plus gamma
where gamma is your
discount factor times q prime
of the new state sub i plus 1
uh times uh where the action is chosen
according to mu prime
uh given some weights theta super mu
prime and theta super q prime so
uh what's important here is
that and this isn't immediately clear if
you're reading this for the first time
what's this very important detail
so it's the action
uh must be chosen according to the
target
actor network
so you actually have q as a function of
the state as well as the output
excuse me of another network
that's very important
update the critic by minimizing the loss
uh basically a weighted average of that
y sub i minus the
output from the
actual q network where the a sub i's are
from the actually
the actions you actually took during the
course of the episode so this a sub i is
from the replay buffer
and
these actions
right are chosen according to the target
actor network
so
uh for each learning step you're gonna
have to do a feed forward pass
of not just this target q network but
also the target actor network as well as
the evaluation critic network i hope i
said that right so the feed forward pass
of the target critic network as well as
the target actor network and the
evaluation critic network as well
and then it says update the actor policy
using the sample policy gradient this is
the hardest step in the whole thing this
is the most confusing part
so it says the gradient is equal to 1
over n times the sum so a a mean
basically whenever you see one over n
times the sum that's a mean
the gradient with respect to actions of
q
uh where the actions are chosen
according to the policy mu of the
current states s times the gradient with
respect to the weights of mu
where you just input the set of states
okay so that'll be a little bit tricky
to implement so and this is part of the
reason i chose tensorflow for this
particular video is because
tensorflow allows us to calculate
gradients explicitly
in pi torch you may have noticed that
all i did was
set q to be a function of the
[Music]
the
current state as well as the actor
network and so
i allowed pytorch to handle the chain
rule this is effectively a chain rule so
let's let's scroll up a little bit to
look at that because this kind of gave
me pause the first 10 times i read it so
this is the hardest part to implement
if you scroll up
you see that this exact same expression
appears here right and this is in
reference to this so it's a gradient
with respect to the weights theta super
mu
of q of s and a
uh such that you're choosing an action a
according to the policy mu
so really what this is is the chain rule
so it's the
uh this gradient is a proportional to a
gradient
this quantity times a gradient of the
other quantity it's just the chain rule
from calculus so
in the
in the pi torch paper we implemented
this version
and these are these are equivalent it's
perfectly valid to do one or the other
so in pi torch we did this version today
we're going to do this particular
version
so that's good to know
all right so
next step
on each time step you want to update the
target networks according to this solved
update rule
so theta super cubed prime gets updated
as tau times theta super q plus one
minus tau theta super q prime and
likewise for theta super mu prime
and then just end the two loops so
in practice this looks very very simple
but what do we know off the bat we need
a class for our replay network
we need a class for our noise process
we need a class for the actor in a class
for the critic now you could think that
perhaps they're the same but when you
look at the details which we're going to
get to in a minute you realize you need
two separate classes so you need at
least one class to handle the deep
neural networks
so you have at least three classes and i
always add in an agent class on top as
kind of an interface between the
environment and the deep neural networks
so that's four and we're gonna end up
with five but that's four right off the
bat
so now that we know the algorithm let's
take a look at the supplemental details
uh supplemental information to see
precisely the architectures and
parameters used
so we scroll down so here are the
experimental details uh these atom
for learning the neural network
parameters for the learning rate of 10
to the minus 4 and 10 to the minus 3 for
the actor and critic respectively so
they tell us the learning rates 10 to
the minus 4 10 to the minus 3 for our
critic
for q the critic we included l2 weight
decay of 10 to the minus 2
and use a discount factor of gamma
0.99 so that gamma is pretty typical but
this important thing is that for q and
only q not mu we included l2 weight
decay of 10 to the minus 2 and use a
discount factor of gamma of
0.99 that's an important detail
for the soft target updates we use tau
equals 0.001 so one part in a thousand
that is indeed very very small
okay fine
the neural networks use the rectified
non-linearity for all hidden layers okay
the final output layer of the actor was
a tangent hyperbolic layer to bound the
actions now
tan hyperbolic goes from minus one to
plus one so
in environments in which you have
um
bounds of plus or minus two let's say
you're going to multiplicative factor
uh so that's just something to keep in
mind
the low dimensional networks up but that
doesn't um that doesn't impact the
tension hyperbolic it just means there's
a multiplicative factor and they're
related to your environment
the low dimensional networks had two
hidden layers with 400 and 300 units
respectively about 130 000 parameters
actions were not included until the
second hidden layer of q so when you're
calculating the critic function q
you aren't actually
passing forward the action from the very
beginning you're including it as a
separate input at the second hidden
layer of cue that's very important
that's a very important implementation
detail
and this is when learning from pixels we
use three convolution layers which we
don't need to know right now we're not
using pixels yet
and followed by two fully connected
layers the final layer weights and
biases
both the actor and critic were
initialized from a uniform distribution
of plus or minus
three by ten to the minus three for the
low dimensional case
this was this was to ensure that the
initial outputs for the policy and value
estimates were near zero
the other layers were initialized from
uniform distribution of plus or minus
one over square root of f where f is the
fan in the layer fanon is just the
number of input units
and the other layer oh i already heard
that
the um
the
the actions were not included into the
fully connected layers that's for the
convolutional
uh case so here right now i'm
experiencing some confusion reading this
so
um
it says the other layers are initialized
from uniform distributions related to
the fan in the actions were not included
into the fully connected layers
so i'm guessing since we're talking
about fully connected layers they're
talking about the
pixel case right because otherwise
they're all fully connected you know it
wouldn't make sense to say specify fully
connected layers
uh so this gives me a little bit of
confusion is this statement referring to
the way i initially interpreted it uh it
is referring to both cases for the state
vector and pixel case
but whatever i'm going to interpret it
that way because it seemed to work
but there's ambiguity there and this is
kind of an example of how reading papers
can be a little bit confusing at times
because the wording isn't always clear
maybe i'm just tired maybe i've been
rambling for about 50 minutes and
my brain is turning to mush that's quite
probable actually but anyway we train
with mini match sizes of 64 with low
dimensional problems and 16 on pixels
with a replay buffer of 10 to the six
for the exploration noise process we use
temporally correlated noise in order to
explore well in physical environments
that have momentum warrants in the
olympic process with theta equals 0.15
and sigma equals 0.2 and it tells you
what it does
all well and good okay so these are the
implementation details we need 400 units
and 300 units for our hidden layers atom
optimizer 10 to the minus 4 for the
actor 10 to the minus 3 for the critic
for the critic we need an l2 weight
decay of 10 to the minus 2 discount
factor of gamma 0.99
and for the soft update factor we need
0.001 and we need updates uh we need
initializations that are proportional to
the one over the square root of fanon
for the
lower layers and plus minus point zero
zero three
uh zero zero yeah zero zero three for
the final output layers of our fully
connected networks
okay so that's a lot of details we have
everything we need to start implementing
the paper uh and that only took us about
50 minutes
it was much shorter than when i read it
took me quite a while so
let's head back up to the algorithm here
and we will keep that up as reference
for the remainder of the video because
that's quite critical so
let's go ahead and head back to our
code editor and start coding this up
we'll start with the easy stuff first
so let's start coding and we will start
with the
probably one of the most confusing
aspects of the problem with the
orenstein ulembek action noise now
you can go ahead and do a google search
for it and you'll find a wikipedia
article that talks a lot about the
physical processes behind a lot of
mathematical derivations and that's not
particularly helpful so if you want to
be a physicist i invite you to read that
and check it out it's got some pretty
cool stuff it took me back to my grad
school days but we have a different
mission in mind for the moment the
mission now is to find a code
implementation of this that we can use
in our problem so
if you then do a google search for
orenstein ulembek
github as in you want to find someone's
github example for it you end up with a
nice example from the open ai baseline
library uh that shows you the precise
form of it so let me show you that one
second
so you can see it here uh in the in the
github
there is a whole class for this right
here that i've highlighted
and this looks to do precisely what we
want it has a previous plus a delta term
and a dt term so it looks like it's
going to create correlations through
this x previous term
there's a reset function to reset the
noise which we may want to use
and there's this representation which
we'll probably skip because we know what
we're doing and it's not really critical
for this particular application though
it would be nice if you were writing a
library as these guys were so they
included the representation
method so so let's go ahead and code
that up in the editor
and that will
tackle our first class
so i'm going to leave the notes up there
for now uh they do no harm they're just
comments at the top of the file so
the first thing we'll need is to import
numpy as np
we know we can go ahead and start and
say import tensorflow as tf we're going
to need tensorflow
we may need something like os to handle
model saving so we can go ahead and
import that as well
just a fun fact it is considered good
practice to import your
system level packages first and followed
by your
library packages second in numerical
order followed by your own personal code
in numerical order
so that's the imports that we need to
start let's go ahead and code up
our class we'll call it ou action noise
and that'll just be derived from the
base object
so the initializer will take a mu
a sigma now they said they used a
default value
of i believe 0.15
and a theta of
0.2
a dt term something like 1 by 10 to the
minus 2
and our x naught will save as none
and again if you have any doubts on that
just go check out the open ai baselines
for it
their implementation is probably correct
right
i'll give them the benefit of the now
so
go ahead and save your parameters as
usual
so we have a mu a theta a dt
a sigma
and x0
and we'll go ahead and call the reset
function at the top
now they override the call method
and what this does is it enables you to
say noise equals oh you action noise
and then when you want to
get the noise you just say
our noise equals noise you can use the
parenthesis that's what overall writing
call does
that's a good little tidbit to know
so we want do you implement the equation
that they gave us it's l self dot x
previous plus
sub dot theta times
mu minus x previous
times
self.dt
plus
sigma times
numpy
dot square root
self.dt times numpy random normal
with a size of
mu
and set the x previous to the current
value that's how you create the temporal
correlations and return the value of the
noise
so
we don't have a value for x previous so
we have to set that with the reset
function
which takes no parameters
self.x previous equals x naught if
self.x
not is
not none else
numpy zeros like
uh self.mu
and that's it for the noise function
that's pretty straightforward
uh so that's all well and good so we
have one one function down so
uh we've taken care of the noise sorry
one class now we've taken care of the
noise and now we can move on to the
replay buffer so this will be something
similar to what i've implemented in the
past there are many different
implementations and ways of implementing
this
many people will use a built-in python
data structure called a dq or a deck i
think it's dq is the pronunciation
basically it's a q
that you fill up over time and that's
perfectly valid uh you can do that
there's no reason not to i prefer using
a set of arrays
and
using numpy to facilitate that
the reason being that we can tightly
control the data types of the
stuff that we're saving for this
pendulum environment it doesn't really
matter but as you get more involved in
this field you will see that you are
saving stuff that
has varying sizes so if you're trying to
save images let's say from the one of
the atari libraries or a mujoko
environment or something like that
hope i pronounce that correctly uh
you'll see that this
memory can explode quite quickly you can
eat into your ram so the ability to
manipulate the underlying data type
representation whether you want to use
single or double precision for either
your floating point numbers or integers
is critical to memory management
as well as taking advantage of some
other optimizations for nvidia gpus
in the touring class and above so i
always use the numpy arrays because it's
a clean implementation that allows
manipulation of data types you can use
the dq if you want
it's perfectly valid
so our separate class
has its own initializer of course
so we're going to want to pass in a
maximum size
the input shape and the number of
actions
right because we have to store the state
action reward and new state tuples we're
also going to want to facilitate the use
of the done flag so we'll have an extra
parameter in here and the reason behind
that is
intimately related to how the
how the bellman equation is calculated
it took me a second to think of that
at the end of the episode the agent
receives no further rewards and so the
expected future reward the discounted
feature reward if you will is
identically zero so
you have to multiply the
reward for the next state by zero if
that next state follows the terminal
state if your current state is terminal
and the next state is following the
terminal state so you don't want to take
into account anything from that the
expected future rewards because they're
identically zero so we need a done flag
this is all i wanted to say
so we save our parameters
um
uh yeah we only need to save that we can
say cell dot mem counter equals zero and
that'll keep track of the position of
our most recently saved memory
state memory an array of numpy
zeros mem size by input shape
uh
new state memory
same same deal you know it's just just
totally the same
good grief okay so now we have the let's
say action memory and that's numpy zero
self.mem size by
self.n actions
uh we didn't save an actions did we so
we'll just call it n actions
we have the
reward memory and that's just a scalar
value
so that only gets
shape self mem size
we also need the terminal memory
terminal memory
and that'll be
shape self.men size
and i have numpyfloat32 if i recall
correctly that is due to the
data types in the pi torch
implementation it's probably not
necessary here in the tensorflow
implementation but i left it the same
way just to be consistent it doesn't
hurt anything
so next we need a function to store a
current transition so that'll take a
state action reward
new state and a done flag as input
so the index of where we want to store
that memory
is the memory counter modulus the
mem size so for any mem counter less
than mem size this just returns mem
counter
for anything larger than mem size it
just wraps around so
if you have a million memories all the
way up from zero to 999 999 it will be
uh that number and then once it's a
million it wraps back around to zero and
then one and two and so on and so forth
and so that way you're overriding
memories at the earliest part of the
array with the newest memories
precisely as they describe in the paper
uh if you're using the dq mem method
then i believe you would just uh pop it
off of the left i believe now don't
quote me because i haven't really used
that implementation but from what i've
read that's how it operates
new state memory sub index
equal state underscore
reward memory
equals sub index
is reward
as action memory
now keep in mind that the actions in
this case are arrays themselves so it's
an array of arrays just keep that in the
back of your mind so that you can
visualize the problem we're trying to
solve here
next up we have the terminal memory
now a little twist here is that we want
this to be
one minus int of done so done is either
true or false so you don't want to count
the rewards after the episode has ended
so when
done is true
you want to multiply by zero so one
minus into true is one minus
one which is zero and when it's not over
it's one minus zero which is just one so
it's precisely behavior we want
and finally you want to increment mem
counter by one every time you store
m new memory
next we need a function to sample our
buffer
and we want to pass in a batch size
alternatively you could make batch size
a member variable of this class it's not
really a big deal i just did it this way
for whatever reason i chose to do it
that way so what we want to do is we
want to find the
uh minimum either
so let's back up for a second so what we
want is to
sample the memories from either the
zeroth position all the way up to the
most filled memory the last filled
memory so if you have
less than the max memory you want to go
from zero to
uh mem counter otherwise you want to
sample anywhere in that whole interval
so
max mem equals
the minimum above of either mem counter
or mem
size and the reason you don't want to
just use mem counter is that mem counter
goes larger than mem size so
if you try to tell it to select
something from the range
uh mem counter when mem counter is
greater than mem size you'll end up
trying to access elements of the array
that aren't there and it'll throw an
error so that's why you need this step
next you want to take a random choice
of uh from zero to maximum of size batch
size
so then you just want to gather those
states from the respective arrays
like so
i forgot the self of course
new states
actions
rewards
and call it terminal
batch
i believe that's everything and you want
to return the states
actions
rewards new states
and terminal
okay so
so now we are done with the replay
buffer class so that's actually pretty
straightforward and if you've seen some
of my other videos on deep q learning
then you've seen pretty much the same
implementation i just typically kept it
in the aging class then uh we i'm kind
of refining my approach getting more
sophisticated over time so it makes
sense to stick it in its own class
so we're already like 40 of the way
there we got five classes in total so
that's good news so
next up we have to contend with the
actor and critic networks so
we'll go ahead and start with the actor
and um
from there keep in mind that we have two
actor networks and we're going to have
to contend with some of the
peculiarities and
way tensorflow likes to do stuff so
let's get started
the actor and of course in
tensorflow you don't derive from any
particular class where in pytorch you
would derive from nn.module
don't know what i'm doing there definite
so we'll need a learning rate number of
actions
a name and the name is there to
distinguish the regular atro network
from the target actor network
input dimms we're going to want to pass
it a session so tensorflow has the
um
construct of the session which houses
the graph and all the variables and
parameters and stuff like that
you can have each class having its own
session but it's more tidy to pass in a
single session to each of the classes
a number of dimensions for the first
fully connected layers so of course i
should be 400
if we're going to implement the paper
precisely fc2 dims is 300
an action bound a batch size that
defaults to 64.
a checkpoint directory
and the purpose of that is to save our
model and in the case of the
pendulum doesn't really matter because
it's so quick to run but in general you
want a way of saving these models
because it can take a long time to run
so we'll save our learning rate number
of actions
all of the parameters we passed in
dot
fc1 dimms
and the purpose of this action bound is
to accommodate environments where the
action is either greater than
plus or minus negative one so if you can
go from plus or minus two then the
tangent hyperbolic is only going to
sample like half your range right from
plus minus one and so you won't have an
action bound and there's a
multiplicative factor to make sure that
you can sample the full range of actions
available to your agent
and we need to say that checkpoint der
and finally we want to call a build
network function well it's not final but
we'll call the build network function
next up
since we have to do the
soft cloning the soft update rule for
the
target actor class and target critic
then we know that we have to find a way
of keeping track of the parameters in
each network so
we're going to keep track of them here
in the variable params
and it's tensorflow.trainable variables
with a scope of self.name so the
we have a single session a single graph
and you're going to have multiple deep
neural networks within that graph we
want we don't want to update the
critic network when we're trying to do
the actual network and vice versa right
we want those to be independent and so
we scope them with their own name so
that tensorflow knows hey this is a
totally different set of parameters from
this one and that'll aid in copying
stuff later and also make sure that
everything is nice and tidy
and the scope is what facilitates that
we'll also need a saver object to save
the
model
let's make a checkpoint file
and that's where we use os
and that will clone into checkpointer
and the name plus underscore
ddpg.checkpoint
so this will automatically scope the
save files for us so that way we don't
confuse the parameters for the
target actor and actor or critic and
target critic or even actor and critic
for that matter
very important
so we're going to calculate some
gradients and we're going to do that by
hand so we're going to need a series of
functions that will facilitate that and
the first of which is the
unnormalized
actor gradients
and that is
given my tensorflow.gradients
self.mu
self.params
and minusself.action
gradient
so we're going to calculate so mu will
be the mu from the paper the actual
actions of the agent params are our
network parameters and this action
gradient so let's go back to the paper
for a second so we can get some idea
what i'm talking about here so if we
look at the algorithm we can see that
we're going to need these
gradients of the critic function with
respect to the actions taken
we're also going to need the gradient of
the actual mu with respect to the
weights of the network so we're passing
in into tensorflow gradients this
function mu
uh the parameters to get the gradient
with respect to those parameters and
then we're gonna have to calculate the
gradient of the critic with respect to
the actions taken so we'll have to
calculate this later that's a
placeholder that's the minus self.action
gradient and we're going to calculate
that in the learning function um
but that's where all of that comes from
so now let's go back to the code editor
and continue
so that's our unnormalized actor
gradients and unnormalized is just
because we're going to take 1 over the
sum 1 over n times the sum
we need a function for performing that
normalization so the actor gradients
has to be this parameter has to be a
list so we cast it as a list and we're
just going to map a lambda function
into x tf.div
x and batch size
no big deal there
so optimize is our optimization step
and of course that's the atom optimizer
we want to optimize with the learning
rate of self.learning rate
and we want to apply gradients so
typically you'd use dot minimize loss
but in this case we're calculating our
gradients manually so we need to apply
those gradients
and what do we want to apply we want to
apply the
actor gradients
to the params
and i would encourage you to go look at
the
tensorflow code for all of these the
video is getting a little bit long
already up to like an hour and 20 hour
and 10 20 minutes something like that so
i'm not going to go through all the
documentation for tensorflow feel free
to look that up
i had to when i was building this out so
next up we need to build our network
so
this is how we're going to handle the
scoping
tf.variable
underscore scope self.name so that way
every network gets its own scope
we need a placeholder for the input
that'll be a 32-bit floating number
shape of
none which is batch size
input dimms
dms that won't work
let's put this on its own line
and we're going to give it a name it's
not critical the name
parameter is just for
debugging if something goes wrong then
you can kind of trace where it went
wrong makes life a little bit easier
the action gradient is also a
placeholder that is what we're going to
calculate in the learn function for the
agent
and
that gets a shape of
none by and actions
so it'll be the gradient of q with
respect to each action so it has number
of dimensions with of actions
and so those are our two placeholder
variables now we get to construct the
actual network so let's handle the
initialization first
so f1 is the fan in
it's one divided by numpy square root
fc1 dimms
um and our dense one
tf layers
dot dense
and that takes self.input as input
with self.fc1 dims as units
our kernel initializer
equals random ah i forgot an import it's
random uniform
one second minus f1 to f1
and the bias initializer
currentl no that's not right kernel bias
initializer
random
uniform
minus f1 to f1
that gets two parentheses so
we have to come back up to our imports
uh import tensor
flow dot
uh
initializers no
so we have to come back up here and say
from tensorflow.initializers
initial
iser's import random uniform
and now we're good to go
let's come back down here
we have dense one now we want to do the
batch normalization so
batch one equals tf layers
batch normalization
dense one and that doesn't get
initialized
so now let's uh activate
our first layer
and that's just the value activation of
the batch normal batch normed now it is
an open debate from what i've read
online about whether or not you should
do the activation
before or after the batch normalization
i'm in the camp of doing it after
the activation after the batch norm
that's because um the radioactivation
function at least at least in the case
of ryu so in rel you you lop off
everything lower than zero so your
statistics might get skewed to be
positive instead of maybe they're zero
maybe they're negative who knows so i
think the batch norm is probably best
before the activation and indeed this
works out this is something you can play
with uh so go ahead and fork this repo
and play around with it and see how much
of a difference it makes for you maybe i
missed something when i try to do it the
other way it's entirely possible i miss
stuff all the time
so you know it's something you can
improve upon
that's how i chose to do it and it seems
to work and who knows maybe other
implementations work as well
so now let's add some
space so f2 is one over
square root
sc2 dimms
dense two
is similar that takes layer one
activation as input
with sc2 dimms
we don't need that i'm going to come up
here and
copy this
there we go except we have to change f1
to f2
perfect
then we have
batch two
and that takes dense two as input
your two activation
batch two
now finally we have the output layer
which is the actual policy of our agent
the deterministic policy of course
and from the paper that gets initialized
with the value of 0.003
and we're going to call this
mu that gets layered two activation as
input
and that needs n actions as the number
of output units
what's our activation that is tangent
hyperbolic tanch
and i will
go ahead and copy the
initializers here of course that gets f3
not f2
perfect can i tab that i can okay
so that is mu
and then we want to take into account
the fact that our environment
may very well
require actions that have values plus
greater than plus or minus one
so self.mu
is tf.multiply with mu
and the action bound
an action bound would be you know
something like two it needs to be
positive so that way you don't flip the
actions
but that's pretty straightforward so now
we've built our network built our
network
next thing we need is a way of getting
the
actual actions out of the network so we
have a prediction function that takes
some inputs
and you want to return
self.sess.run
self.mu
with a feed dictionary of
self.input
and pass in inputs that's all that is
there is to uh passing the doing the
feed forward kind of an interesting
contrast to how pytorch does it with
the explicit construction of the
feedboard function this just runs the
session on this and then goes back and
finds all the associations between the
respective variables
nice and simple
now we need a function to train
and that'll take inputs and gradients
this is what will perform the actual
back propagation through the network
and you want to run
self.optimize that's our
function that accommodates learning
with a feed dictionary of
inputs inputs
and
self.action gradient
gradients
so that is also reasonably
straightforward so
you know let's
format that a little bit better
all right so that is our training
function
next we need two functions to
accommodate the loading and the saving
of the model so
define save checkpoint
print
then you want to say self.saver.save
very creative
you want to save the current session to
the checkpoint file
and the
load checkpoint function is the same
thing just in reverse
so
here we want to print
loading
checkpoint
self.saver.restore.session
and
the checkpoint
file so that will
you can only call this after
instantiating the agent so it will have
a default session
with some initialized values and you
want to load the variables from the
checkpoint file into that checkpoint
and that is it for the actor class this
is reasonably straightforward the only
real mojo here is the
actor gradients and this is just two
functions that accommodate the fact that
we're going to manually calculate the
gradient of the critic with respect to
the actions taken
so let's go ahead and do the critic
class next
uh that is also very similar so
that again derives from the base object
gets an initializer
and it's pretty much the same number of
actions a name
input dimms a session
sc1 dimms
sc2 dimms
a batch size will default to 64 and a
checkpointer will default to this
just a note you have to do a make dur on
this temp slash gdp ddpg first otherwise
it'll bark at you
not a big deal just something to be
aware of since it's identical let's go
ahead and copy a good chunk of this
stuff here
uh what exactly is the same
all of this
now we need the checkpoint file let's
grab that
control c
come down here
ctrl v voila it is all the same so very
straightforward um
nothing
too magical about that so now let's
handle the optimizer
because we already have a we've already
called the function to build our network
we can define our optimizer so
self.optimize
tftrain.adam optimizer
and we're going to minimize the
the loss
which we will calculate in the build
network function
and we also need uh the function to
actually calculate the gradients of q
with respect to a so
we have self.action gradients
self.q
and
actions
let's build our network
with tnsrtf
variable scope self.name
so now we need our placeholders again we
need sub.input
placeholder
float32
we need a shape
none
by input dimms
now
if you're not too familiar with
tensorflow specifying none in the first
dimension tells tensorflow you're going
to have some type of batch of inputs and
you don't know what that batch size
would be beforehand so just expect
anything
and
uh
let's delete that say
oh you need a comma for sure
and say name equals
uh
inputs
here we go
did i forget a comma up here let's just
make sure
i did not
okay
so
we also need the actions
because remember we only take into
account the actions
uh on the second hidden layer of the
critic neural network
i have float 342
shape
none by and actions
name of actions
now much like with q learning we have a
target value
let me just go back to the paper really
quick and show you what that precisely
will be
so that target value will be this
quantity here and we will calculate that
in the learning function
as we
as we get to the agent class so let's go
back to the code editor and finish this
up
so q target is just another placeholder
and that's a floating point number
none by one it's a scalar so it is shape
batch size by one
and we will call it
targets
okay so now we have a pretty similar
setup to the actor network so let's go
ahead and come up here
and copy this
no not all that just this
and come back down here
so f1 we recognize
that's just the fan in
uh we have a
dense layer for the inputs and we want
to initialize that with a random number
uh pretty straightforward
then we can come up to
f2 and copy that as
as well sorry the second layers
it'll be a little bit different but
we'll handle that momentarily
so now f2
the layer 2 is pretty similar
the only real difference is that we want
to get rid of that activation and the
reason we want to get rid of that
activation is because after we do the
batch norm we have to take into account
the actions so we need another layer
so action in
tf layers dense
uh it's going to take in self-don
actions which will pass in from the
learning function
and that's going to output fc2 dimms
with a rally activation
so then our state actions
will be the
uh the addition of the batch two
and the action in
and then we want to go ahead and
activate that
okay so this is something else i pointed
out in my pie torch video where this is
a point of debate with the way i've
implemented this i've done it a couple
different ways i've done the different
variations and this is what i found to
work uh i'm doing a double activation
here so i'm activating the relu i'm
doing the value activation on the
actions in
on the output of that that dense layer
and then i'm activating the sum now
the
the value function is non-commutative
with respect to the addition function so
the value of the sum is different than
the sum of the values and you can prove
that to yourself
on a sheet of paper
but it's debatable on whether or not the
way i've done it is correct it seems to
work so i'm going to stick with it for
now
again fork it clone it uh change it up
see how it works
and improve it for me that would be
fantastic then make a pull request and
i'll disseminate that to the community
so
we have our
state actions now we need to calculate
the actual output of the layer
and an f3 is our uniform initializer for
our final layer which is
self.q
that's all layers.dense that takes state
actions as input
outputs a single unit
and we have the
kernel initializers and
bias initializers
similar to up here
let's paste that
there we go that's a little bit better
still got a whole bunch of white space
there all right so
we are missing one thing and that one
thing is the regularizer so as i said in
the paper they have
l2 regular regularization on the critic
so we do that with kernel regular iser
equals tf keras regularizers
dot l2 or the value of zero zero one uh
zero one sorry
so
that is that for the q and notice that
it outputs one unit
and it outputs a single unit because
this is a scalar value you want the
value of
the particular state action pair
finally we have the loss function
and that's just a mean squared error
error
q target and q so q target is the
placeholder
up here and that's what we'll pass in
from the learn function from the agent
and self.q is the output of the deep
neural network
okay so now similar to our critic
we have a prediction function
it takes inputs and actions
and you want to return
self.sess.run
self.q
or the feed dictionary
of
sub.input inputs
and
actions
oops
next you need a training function
and that's slightly more complicated
than the
actor visa takes in the inputs actions
and a target
and you want to return
self.ss.run
self.optimize
with a feed dictionary
of
sub.input inputs
self.actions actions
and q target
spacing that's a little wonky whatever
we'll leave it
that's a training function next we need
a function to get the action gradients
and that'll run that action gradients
operation up above
so let's get that
again takes inputs and actions as input
and you want to return
self.session.run
self.action ingredients
with a its own feed dictionary
equals
self.input
inputs and self.actions actions
and then we also have the save and load
checkpoint functions which are identical
to the actor so let's just
copy and paste those
there we go
so
that is it for the critic class so now
we have most of what we need we have our
our
our noise our replay buffer our actor
and our critic now all we need is our
agent now the agent is what ties
everything together it handles the
learning functionality it will house the
noise the memory
the replay buffer as well as the four
different deep neural networks
and that derives from the base object
initializer is a little bit long
and that takes an alpha and a beta these
are the learning rates for the actor and
critic respectively recall from what we
read in the paper they use 0 0 0 1 and 0
0 1 for both networks
we need input dimms tau the environment
that's how we're going to get the action
bounds
a gamma of 0.99 for the paper in this
case we need number of actions equals 2
a mem size
for our memory
of 1 million
layer 1 size of 400 a layer 2 size of
300
and a batch size of 64.
so of course we want to save all of our
parameters
uh we'll need the memory
and that's just a replay buffer max size
whoops
input dimms and number of actions
we will need a batch size
here's where we're going to store the
session
and this is so that we have a single
session for all four networks
and i
and believe
believe don't quote me on this but
i tried it with an individual network
for sorry an individual session for each
network and it was very unhappy when i
was attempting to copy over parameters
from one network to another i figured
there were some scoping issues uh so i
just simplified it by having a single
session there's no real reason that i
can think of to have more than one
and that is an actor that gets
alpha a number of actions the name is
just actor input dims
the session
layer one size layer two size
action space
dot high and that's our action bounce
the action space dot high
next we have a critic
that gets beta and actions its name is
just critic
but dims
self.session
layer 1 size layer 2 size and we don't
pass in anything about the environment
there
so now we can just copy these
and instead of actor it is
target actor
and let's go ahead and clean up and be
consistent with our
pet bait style guides
always important
that actually makes you stand out i've
worked on projects where the manager was
quite happy to see that i had a somewhat
strict adherence to it um just something
to take note of
so then we have a target critic as well
and we will clean up
this
and so that is all four of our deep
neural networks that's pretty
straightforward
so now we need noise
that's an ou action noise
with
mu equals numpy zeros
in the shape and
actions
so now
we need operations to perform the soft
updates so
uh the first time i tried it i defined
it as its own separate function
and that was a disaster
it was a disaster because it would get
progressively slower at each soft update
and i don't quite know the reason
for that i just know that that's what
happened and so when i moved it into the
initializer and defined one operation
i'm guessing it's because it adds every
time you call it it probably adds
something to the graph so that adds
overhead to the calculation that's my
guess i don't know that's accurate it's
just kind of how i reasoned it but
anyway let's go ahead and
define our update operations here
so what we want to do is iterate
over our target critic
parameters
and call the assignment operation what
we want to assign
we want to assign the
product of
critic frames
and self.tau
plus
give that multiply
self dot target critic
param sub i
times or and
um no that should be a sorry a comma
comma and
one minus cell dot tau
so we get two
let's hit
do that
and that is a list comprehension
for i and range length of
target critic
dot params
now it's that
you don't need that
i don't believe no
so then we have a similar operation for
the actor
we just want to swap
actor and critic
target actor
uh i did the same thing here didn't i
target actor
and up here
dot params
okay
target actor
actor and then target actor
there so now we have our update
solved update operations
according to the paper
uh finally we have constructed all the
graphs uh you know for the whole
four networks so we have to initialize
our variables
self.session.run
tf global variables
initializer
you can't really run anything without
initializing it
and as per the paper at the very
beginning we want to update the
network parameters
and
at the beginning we
want to pass in the we want the target
networks to get uh updated with the
uh full ver the full value of the critic
of the evaluation networks and so i'm
passing in a parameter of first equals
true
so since it's confusing let's do that
first that function first update network
parameters
um
first we'll default to false
so if first
we need to say old tau equals self.tau
i need to save the old value of tau to
set it again so that i can reset it
so that's how it goes one and say
self.target
critic.session.run
update
critic
self.target actor.session.run
self dot update
actor
and
as i recall this is important which
section you use to run the
update
although maybe not since i'm using only
one session
if you want to play around with it go
ahead
and then go ahead and reset the tau to
the old value because you only want to
do
this particular update where you
update the target network with the
original network
full values on the first term on the
first
go through otherwise
just go ahead
and run the
update function
boom so next we need a way of storing
transitions
uh self state action reward
ah
new state and done
so self.memory
transition
you want to store all this stuff
this is just an interface from one class
to another
you know this may or may not be great
computer science practice but it works
next we want a way of choosing an action
and that should take a state as input
since we
have defined the
input variable to be shape batch size by
n actions
you want to reshape the state to be
one by
sorry it should be one by
the observation space
new axis all
and that's because we have the
come up here for the
actor network just so we're clear
it is because this
has shape none by input dimms so if you
just pass in the observation vector that
has shape input dimms and it's going to
get
uh it's going to get uppity with you so
you just have to reshape it
because you're only passing in a single
observation to determine what action to
take
so mu
actor.predict
state
uh noise
equals self.noise
mu prime
u is mu plus noise
return
sub-zero so
this returns a tuple uh so you want the
zeroth element
so now we have the learning function and
of course this is where all the magic
happens
so if you have not um
filled up the memory then you want to go
ahead and bail out
otherwise you want to go ahead and
sample your memory
so the memory.sample buffer
batch
size
so next we need to do the update from
the paper so let's go back to the paper
and make sure we are clear on that
so we need to um
we already sampled this so we need to
calculate this and to do that we're
going to need the
q prime the target critic network uh
as
output as well as the output from the
target actor network
and then we'll use that to update the
loss for the critic and then we're going
to need the output from the critic as
well as from the
actor network so we need to
basically pass states and actions
through all four networks to get the
training function
the learning function
so let's go ahead and head back to the
code editor and do that
so our critic value for
q prime so q critic value underscore
sub.targetcritic.predict
and you want to pass in the new states
and you also want to pass in the actions
that come from the target actor
new state
i need one extra parenthesis there
so then we want to calculate the y sub
i's for the targets so that's an empty
list for j and range
self.batch size
target.append reward
critic value underscore j
times done sub j
and that's where the
kept harping on getting no rewards after
the terminal state that's where that
comes from uh when done is
true then it's one minus true which is
zero so you're multiplying this quantity
by zero so you don't take into account
the value of the next state right which
is calculated up here you only take into
account the most recent reward
as you would want
so then we just want to reshape that
target
into something that is batch size by one
that's to be consistent with our
placeholders
and now we want to call the
critictrain function
right because we have everything we need
we have the states actions and targets
with states and actions from the replay
buffer and the target from this
calculation here
very easy
now we need to do the actor update so
the action outs
self.actor.predict
we get the predictions from the actor
for the states
our grads
the gradients
get action
gradients
state
a outs that'll get the remember that
i'll do a feed forward and get the
gradient of the
critic with respect to the actions taken
and then you want to train the actor
state
and grads the gradients it's a tuple so
you want to de-reference it and get the
zeroth element and then finally you want
to update network
parameters
okay so that is it for the learn
function now we have two other
bookkeeping functions to handle which is
save models
and
this will save all of our models so a
self.actor.save
checkpoint
self.target actor
save checkpoint
critic
and a function to load models
sounds like a dog fine out there
so we want to load checkpoints instead
of saving them
load checkpoint
load
and load
so that is it for the uh agent class it
only took about an hour so we're up to
about two hours for the video so longest
one yet so uh this is an enormous amount
of work this is already 310 lines of
code if you've made it this far
congratulations this is no mean feat
this took me you know a couple weeks to
hammer through
but we've gotten through it in a couple
hours so
this is all there is to the
implementation now we have to actually
test it so let's open up a new file
and save that
as
main tensorflow.pi
and what we want to do now is
go ahead and test this out in the
pendulum environment
called tensorflow original
import our agent we need jim we need
um
do we need numpy for this
uh yes
we do
we need we don't need tensorflow
from utils import plot learning
and
we don't need os so
we will have a score history
now you know what i don't want to do
that way let's save
name equals
main
now we want to say
env gym.make
banju
lom
v0
uh
agent
equals
agent
it gets
a learning rate of zero zero zero one
a beta of
zero one
we put dems three
three tau
zero
pass in our environment
batch size 64. i'm being verbose here
and it looks like when i ran this i
actually used different layer sizes
well that's an issue for
hyper parameter tuning
i just want to demonstrate that this
works
and actions equals one so
when i got good results for this i
actually used 800 by 600
and i cut these learning rates in half
but we'll go ahead and use the values
from the paper
sometimes i do
hyper parameter tuning other thing we
need to do is set our random seed
you know we can just put it here
whatever
the reason we want to do this is for
replicability and i have yet to see an
implementation of this where they don't
set the random seed and i've tried it
without it and you don't get very good
sampling of your replay buffer so this
seems to be a critical step and most
implementations i've seen on github do
the same thing
so
let's go ahead and play our episodes
say a thousand games
enemy that reset we forgot our score
history of course
that's to keep track of the scores over
the course of our games
don is
false and the score for the episode is
zero
let's play the episode say act
agent i choose action
takes obs as input
new state reward
done info equal cnb dot step
act
agent dot remember ops act reward new
state
and done i guess the int done really
isn't necessary we take care of that in
the replay buffer fun uh funk class but
you know
doesn't hurt to be
explicit
and then you wanna uh we wanna learn on
each time step because this is a
temporal difference method
keep track of your score and set your
old state to be the new state so that
way when you choose an action on the
next step you are using the most recent
information
finally at the end of every episode
you want to append that score to the
score history and
print episode
i
score
percent.2f
score
uh 100 game average
percent.2f
percent numpy mean score
history
um last hundred games
minus 100 on
one more
and at the end
equals
pendulum
dot png
plot learning
uh score history
file name and a window of 100. the
reason i chose a window of 100 is
because many environments define find
solved as trailing 100 games over some
amount the pendulum doesn't actually
have a
a solved amount
so
what we get is actually on par with some
of the best results people have on the
leaderboard so it looks like it does
pretty well so
that is it for the main function let's
go ahead and head to the terminal and
see how many typos i made
all right here we are let's go ahead and
run
the main file
uh invalid syntax i have an extra
parenthesis i'm just going to delete
that really quick
run that
ah i have the star out of place let's go
back to the code editor and handle that
so
it looks like it is online
95
which is here
all right back to the terminal let's try
it again
and it says
line 198
invalid syntax oh it's because it's a
comma instead of a colon
all right i'll fix that
once more
all right so that was close
so actor object
has no attribute input dimms line 95.
ah okay that's easy let's head back
so it's in line 95
just means i forgot to save
input dimms
and that probably means i forgot it in
the critic as well since i did a cut and
paste
uh
yes it does
all right now we'll go back to the
terminal hopefully that's the last one
all right
moment of truth
critic takes one positional argument but
seven were given good grief okay one
second
so that is of course in the agent
function
so
i have line two thirty four
one two three four five six seven
parameters indeed
so critic takes learning rate number of
actions name input dim session
interesting one two three four five six
seven
what have i done
oh
that's why uh
it's class critic of course all right
let's go back to the terminal and see
what happens
name action bound is not defined
and that is in the
line 148 okay
line 148
that is in the critic ah that's because
i don't need it there let's delete it
that was just for the actor it's because
i cut and pasted always dangerous i
tried to save a few keystrokes to save
my hands and ended up wasting time
instead all right let's go back to the
terminal
all right
actor has no
attribute inputs
that is on line
126.
self.inputs it's probably self.input
yes
that's why i do the same thing here
okay
perfect and it runs so
i'll let that run for a second but i let
it run for a thousand games earlier and
this is the output i got now keep in
mind it's with slightly different
parameters
the point here isn't that whether or not
we can replicate the results because we
don't even know what the results really
were because they express it as a
fraction of a planning
a fraction of the performance of a
planning agent so who knows what that
really means
i did a little bit of hyper parameter
tuning all i did was double the number
of input units and have the learning
rate
and i ended up with something that looks
like this so you can see it gets around
150 or so steps
it gets around 150 or so steps to solve
and if you check the leaderboards it
it shows that that's actually a
reasonable number some of the best
environments only have 152 steps some of
them do a little bit better sorry best
agents
solve it 152 steps or achieve a best
score of 150 steps but it's pretty
reasonable so so the default
implementation looks like it's very very
slow to learn
uh you can see how it's kind of starts
out bad gets worse and then gets starts
to get a little bit better so that's
pretty typical you see this you know
oscillation and performance over time
pretty frequently
but that is that that is how you go from
an imp uh paper to a full implementation
in about two hours of course it took me
you know many you know a couple uh many
times that uh to get this set up for you
guys but um
i hope this is helpful i'm going to milk
this for all it's worth this has been a
tough project so i'm going to present
many many more environments in the
future i may even do a video like this
for pytorch i have yet to work on a
keras implementation for this
but there are many more
ddpg videos to come so subscribe to make
sure you don't miss that
leave a comment share this if you found
it helpful that helps me immensely i
would really appreciate it and i'll see
you all in the next video
welcome back everybody in this tutorial
you are going to code a deep
deterministic policy grading agent to
beat the continuous lunar lander
environment in pytorch no prior
experience needed you don't need to know
anything about deep reinforcement
learning you just have to follow along
let's get started
so we start as usual with our imports
we'll need os to handle file operations
and all this stuff from torch that we've
come to expect as well as numpy i'm not
going to do a full overview of the paper
that will be in a future video where i
will show you how to go from the paper
to an actual implementation of deep
deterministic policy gradients so make
sure you subscribe so you don't miss
that but in this video we're just going
to get the very high level overview the
50 000
foot view if you will that will be
sufficient to get an agent to beat the
continuous lunar liner environment so
that's good enough
so the gist of this is we're going to
need several different classes so we'll
need a class to encourage exploration in
other words a type of noise
and you might have guessed that from the
word deterministic it means that the
policy is deterministic as in it chooses
some action with certainty and so if it
is purely
deterministic you can't really explore
so we'll need a class to handle that
we'll also need a class to handle the
replay memory because deep deterministic
policy gradients works by combining the
magic of actor critic methods with the
magic of deep q-learning which of course
has a replay buffer
and we'll also need um
classes for our critic and our actor as
well as the agent so
that's kind of a mouthful we'll handle
them one bit at a time and we will start
with the noise so
this class is called ou action noise
and the ou stands for orenstein
willembeck so that is a type of noise
from physics that models the motion of a
brownian particle meaning a particle
subject to a random walk based on
interactions with other nearby particles
it gives you a temporarily correlated
meaning correlated in time set type of
noise that centers around a mean of zero
so
we're going to have a number of
parameters
a mu a sigma a theta
if i could type that would be fantastic
as well as a dt as in the differential
with respect to time and an initial
value that will get an original value of
none so if you want to know more about
it then just go ahead and check out the
wikipedia article but the overview i
gave you is sufficient for
this tutorial
so of course we want to save all of our
values
um
x0
and we want to call a reset function so
the reset function will reset the
temporal correlation which you may want
to do that from time to time turns out
it's not necessary for our particular
implementation
but it is a good function to have
nonetheless
so next we're going to override the call
function uh if you aren't familiar with
this
this allows you to say
noise equals ou action noise
and then call
noise so that allows you to instead of
saying noise.get noise just say noise
with parentheses or whatever the name of
the object is so that's overriding the
call function so we'll have an equation
for that x previous
plus theta times
the quantity mu minus
self.x previous times
self.dt
plus
self.sigma
times numpy square root
of self dt times mp random normal
size equals mu shape so it's a type of
random normal noise
that is correlated in time through this
mu minus x previous term
um
and every time you calculate a new value
you want to set the old value the
previous value to the new one and go
ahead and return the value
so the reset function
all that does is check to make sure
x0 exists if it doesn't it sets it equal
to some
zero value so so x previous equals so x
naught if
is not none
else numpy zeros like in the shape of
self.mu
that's it for the action noise again
this will be used in our
actor class to add in some exploration
noise to the action selection
next we need the replay buffer class
and this is pretty straightforward it's
just going to be a set of numpy arrays
in the shape of the
action space numpa the observation space
and rewards so that way we can have a
memory of events that have happened so
we can sample them during the learning
step
if you haven't seen my videos on deep q
learning please check those out they
will make all of this much more clear as
well as checking out the videos on actor
critic methods because this again
ddpg kind of combines after critic with
deep q learning so that will really be
helpful for you
i'll go ahead and link those here as
well so that way you
can get edumacated so
um scroll down a bit we want to save our
mem size
as max size
so the mem counter will start out as
zero again this is just going to be a
set of arrays that keep track or
matrices in this case that keep track of
the
state reward action transitions
and that will be in shape mems size so
however number of memories we want to
store and input shape so
if you are relatively new to python this
star
uh variable idiom it isn't a pointer if
you're coming from c or c plus plus it
is uh
an idiom that means to unpack a tuple so
this makes our class extensible so we
can pass in a list of a single element
as in the case of the lunar lander and
continuously landing environment that
we'll use today or later on when we get
to the continuous car racing environment
we'll have images from the screen so
this will accommodate both types of
observation vectors it's a way of making
stuff
extensible
and the new state memory is of course
the same shape so we just copy it
it looks like i am missing a
parenthesis somewhere
i guess we'll find it when i uh go ahead
and run the program so
oh
it's not self.init it's def there we go
definite perfect so
we'll also need an action memory
and that of course will also be an array
of zeros in the shape of mem size by
number of actions i believe that means i
need an extra
parenthesis there yes
and we'll also have a reward memory
and that will just be in shape mem size
we also need a terminal memory so
in reinforcement learning uh we have the
concept of the terminal state so
when the episode is over the agent
enters the terminal state from which it
receives no future rewards so the value
of that terminal state is identically
zero and so the way we're going to keep
track of when we transition into
terminal states is by saving the done
flags from the open ai gem environment
and that'll be shape
numpy uh by zeros mem size
and
i've called this float32 it's probably
because torchy is a little bit
particular with data types so we have to
be
cognizant of that
we need a function to store transitions
which is just a state
action reward
new state and
done flag
so the index is going to be the first
available position
so
mem counter just keeps track of the last
memory you stored it's just an integer
quantity from zero up to mem size and so
when mem counter becomes greater than
mem size it just wraps around from zero
so when they're equal at zero and when
it's equal to mem size plus one it
becomes one and so on and so forth
so state memory subindex
equals state
action memory index
reward
equals state underscore
and the
terminal memory
good grief terminal memory index doesn't
equal done but it equals one minus done
so the reason is that when we get to the
update equation the bellman equation for
our learning function you'll see we want
to multiply by whether or not the
episode is over and that gets is
facilitated by
one minus the quantity done
just increment mem counter
and next we need to
sample that buffer so sample buffer and
that will take in a batch size as input
so the max memory
is going to be the minimum of either mem
counter
or
mem size not the max but the minimum
then match is just going to be a random
choice
of
a maximum index of maximum
number of elements
excuse me equal to batch size scroll
down a bit
and then we want to get a hold of the
respective states actions rewards and
new states and terminal flags and pass
them back to the learning function
sub
batch
new states
rewards
know it's not easy to type and talk at
the same time apparently uh
and let's get um actions
self dot action memory
match
and terminal
good grief so we want to return states
actions rewards
new states and the terminal flags
perfect so that is it for our replay
memory so
you're going to see this a lot in the
other videos on deep deterministic
policy gradients
because we're going to need it for
basically anything that uses a memory
next let's go ahead and get to the meat
of the problem with our critic network
and as is often the case when you're
dealing with pytorch you want to derive
your
neural network classes from nn.module
that gives you access to
important stuff like the train and eval
function which will set us in train or
evaluation mode very important later i
couldn't get it to work until i figured
that out so a little tidbit for you you
also need access to the parameters for
updating the weights of the neural
network
so
let's define our initialize function
the beta is our learning rate
we'll need input dimms
number of dimensions for the first fully
connected layer as well as second
connected layer
number of actions
a name the name is important for saving
the network
you'll see that we have
many different networks
so we'll want to keep track of which one
is which
very important
as well as a checkpoint directory for
saving the model they're also very
important because this model runs very
very slowly so you'll want to
save it periodically
and you want to call the super
constructor for critic network
and that will call the constructor for
nn.module i believe
input dems equals input dimms
sc1 dimms
so these will just be the parameters for
our deep neural network that
approximates the value function
the number of actions
a checkpoint file and that is os path
join
checkpoint der
with
name plus
underscore ddpg
and if you check my github repo i will
upload the trained model because this
model takes a long time to train
so i want you to be able to take
advantage of the fully trained model
that i've spent the resources and time
training up so you may as well benefit
from that so next up we need to define
the first layer of our neural network
just a linear layer
and that'll take input dimms
and output
fc1 dimms
we're also going to need a number for
initializing the weights and biases of
that layer of the neural network i'm
going to call that f1 and it's divided
by 1 over the square root of the
number of
dimensions into the network so
self dot
fc1
dot weight
dot size
and that returns a tuple so i need the
zeroth element of that
and then we want to initialize that
layer by using t
torch and then init dot uniform
underscore
the the tensor you want to initialize
which is fc1 weight
data
up uh from -1 to positive f1 so uh this
will be a small number of order uh
0.1 or so not exactly 0.1 but of order
0.1
and you also want to initialize the
biases
bias.data and that gets the same number
and again in the future video when we go
over the derivation of the paper i'll
explain all of this uh but just for now
know that this is to constrain the
initial weights of the network to a very
narrow region of parameter space to help
you get better convergence
um
perfect so oh and make sure to subscribe
so you don't miss that video because
it's going to be lit
so bn1 is a layered norm
and takes
fc1 dims as input
the batch normalization helps with um
convergence of your model uh you don't
get good convergence if you don't have
it so leave it in
so fc2 is the second layer another
linear it takes fc one dimms as input
and outputs fc
two dimms
good grief and we wanna do the same
thing with initialization so it's one
over the square root of self dot fc2
weight data dot size
zero
and you want to do tnn init uniform
underscore
c2
weight
data
minus f2 up to f2 make sure that's right
so we don't screw that up because that's
important
that looks correct
the syntax here is the first parameter
is the tensor you want to initialize and
then the lower and upper boundaries
so next we need uh bn2 which is our
second batch norm layer
fc2 dimms
and just a note the fact that we have a
normalization layer a batch norm type
layer means that we have to use the eval
and train functions later
kind of a nuisance it took me a while to
figure that out
the critic network is also going to get
a
action value because the
action value function takes in the
states and actions as input
but we're going to add it in at the very
end of the network
linear
actions
see two dimms
and this gets a constant initialization
of zero zero three
or sorry the um
the next one uh the output gets a
initialization of
zero zero three
and since this is a scalar value it just
has one output
and you want to initialize it again
uniformly
that q dot weight dot data and that gets
minus f3 up to f3
and likewise for bias data
okay so now we have our optimizer and
that will be the atom optimizer
and what are we going to optimize the
parameters
and the learning rate will be beta so
you notice that we did not define
parameters
right here right we're just calling it
and this comes from the inheritance from
nn.module that's why we do that so we
get access to the
network parameters
next
you certainly want to run this on a gpu
because it is an incredibly expensive
algorithm so you want to call the device
so t device
cuda zero if t dot cuda dot is
available
else
cuda one so i have two gpus if you only
have a single gpu it will be else cpu
but i don't recommend running this on a
cpu so next you want to send the whole
network to your device by self.2
self.device
we are almost there for the
critic class next thing we have to worry
about is the forward function
and that takes a state and an action as
input
keep in mind the actions are continuous
so it's a vector in this case length two
for the continuous lunar lander
environment it's two real numbers in a
list or numpy array format
so state value
fc1 state
and then you want to pass it through
bn1
state value
and finally you want to activate it
f.value state value now it is an open
debate whether or not you want to do the
value before or after the batch
normalization
in my mind it makes more sense to do the
batch normalization first because
when you are calculating batch
statistics
if you apply the value first then you're
lopping off everything below zero right
so that means that your statistics are
going to be skewed toward the positive
end when perhaps the real distribution
has a mean of zero instead of a positive
mean or maybe it even has a negative
mean which you wouldn't see if you used
the
value function
before the batch normalization so just
something to keep in mind
you can play around with it feel free to
clone this and see what you get but
that's how i've done it i did try both
ways and this seemed to work the best
so next we want to feed it into the
second fully connected layer
bn2 state value
and
then we want to
be into that oh sorry i already did that
one second let me let the cat out so
we've already done the batch
normalization we don't want to activate
it yet what we want to do first is
take into account the action value
and
what we're going to do is
activate the action through the action
value layer and perform a value
activation on it right away we're not
going to calculate batch statistics on
this so we don't need to worry about
that
but what we want to do is
add the two values together so state
action value f.value
t add state value
and action value other thing that's a
little bit wonky here and i invite you
to clone this and play with yourself is
that i am double valuing the action
value function so the actual value
quantity so i do a value here
and then i do a value on the ad now
this is a little bit sketchy i've played
around with it and this is the way it
works for me
so if you can clone it and get it to
work a different way the other
possibility is that you
don't do this value here but you do the
value after the add so
value is a non-commutative function with
add so what that means is that if you do
an addition first and then a value
that's different than doing the sum of
the two values right or so if you take
value of minus 10 plus value of 5
you get a
value of minus 10 to 0 plus value of 5
is 5 so you get 5 if you take the value
of minus 10 plus 5 then you get a value
of -5 or zero so it's a non-commutative
function so it does matter the order uh
but this is the way i found it to work
i've seen other implementations that do
it differently feel free to clone this
and do your own thing with it
i welcome any additions improvements or
comments
so then we want to get the actual state
action value
by passing that's uh additive quantity
through our final layer of the network
and go ahead and return that
a little bit of bookkeeping
we have a check
save checkpoint function
and you'll say print
and then you want to call t.save
self.state dict what this does is it
creates a state dictionary uh where the
keys are the names of the parameters and
the values are the parameters themselves
and where do you want to save that you
want to say that in the checkpoint file
then you also have the load checkpoint
good grief checkpoint function
and that does the same thing just in
reverse
loading checkpoint
and you want
self.loadstatedict
t load self.checkpoint
file
so that is it for our critic network now
we move on to the actor network
and then of course derives from
nn.module
we have an uh init function
it takes alpha
if we get spell correctly
input dimms fc1 dimms fc2 dimms
this is pretty similar to the
critic network it will just have
a different structure
in particular we don't have the
um
we don't have the
actions i can't type and talk at the
same time
but it's pretty similar nonetheless
so input dimms we want to save
and actions
see one dimms
and fc2 dims same deal
uh let me go copy the checkpoint file
function
just to make life easy
perfect i like to make life easy so we
have our first fully connected layer
and in dot linear take
self.input dimms as input
and fc1 dims and of course it operates
in the same way as i
discussed in the replay buffer where it
will just unpack the tuple
next we have the initialization value
very similar 1 over mp square root
self.fc1.weight
data size
0th element and we want to initialize
the first layer uniformly within that
interval
and it's uniform
underscore
se1 weight data minus f1 and f1
copy that
that will be fc1
bias data
fc2 is another linear layer it takes fc1
dimms as input and outputs
fc2 dimms
as you might expect
and the initialization for that
will be basically the same thing except
for layer two so fc2
weight data
and then you know what let's just copy
this
paste and make sure we don't mess it up
fc2
f2
good grief
and
plus minus
f2
um and that is all well and good other
thing we forgot is the batch norm
and that is nn layer norm
and that takes fc1 dims as input
likewise for layer 2 that is
another layer norm takes fc2 dimms as
input shape
and that doesn't get initialized
but we do have the
f3 and that is zero zero three this
comes from the paper we'll go over this
in a future video
uh but don't worry about it self.mu mu
is the representation of the policy in
this case it is a
real vector of shape and actions it's
the actual action not the probability
right because this is deterministic
so it's just a linear layer
takes fc2 dims as input and outputs the
number of actions
and we want to do the same thing where
we initialize the weights
let's copy
paste and instead of fc2 it will be mu
and instead of f2 it is
f3
as you might expect
am i forgetting anything i don't believe
so
so finally we have an optimizer and
that's again optim.addem
self.parameters
and learning rate equals alpha
we also want to do the same thing with
the device
t.device
cuda zero
if t dot cuda dot is available
else cpu
and finally
send it to the device
that is that
so next we have the feed forward so
that takes
the
state as input
so i'm just going to call it x this is
bad naming don't ever do this
self.fc1
do as i say not as i do
self.bn1
of state
value of x that's a mistake
it should be x
x equals
self.fc2 of x
bn2
x
and then
x equal t tan hyperbolic self.mu
of x and then return x so what this will
do is
pass the current state or whatever state
or set of states batch in this case you
want to look at
perform the first uh feed forward pass
batch normate value send it through the
second layer batch norm but not activate
send it through to the
final layer now i take that back i do
want to activate that
silly mean f.value x and then i'll pass
it through the final layer mu and
perform a tangent hyperbolic
activation so what that will do is it'll
bound it between minus one and plus one
and that's important for many
environments later on we can multiply it
by the actual action bounds so some
environments have a max action of plus
or minus two so if you're bounding it by
plus or minus one that's not going to be
very effective so you just have a
multiplicative factor later on
and then i'm going to go copy the two
save and load checkpoint functions
because those are
precisely the same
that's it for our actor next we come to
our final class the meat of the problem
the agent
and that just gets derived from the base
object
and that gets a whole bunch of
parameters alpha and beta of course you
need to pass in the learning rates for
the actor and critic networks
input dimms
a quantity called tau i haven't
introduced that yet but we'll get to it
in a few minutes
we're going to pass in the environment
that's to get the action space that i
talked about just a second ago
the gamma which is the agent's discount
factor so if you're not familiar with
reinforcement learning an agent values a
reward now more than it values a reward
in the future because there's
uncertainty around future rewards so it
makes no sense to value it
as much as a current reward so what's
the discount factor you know how much
less does it value a future reward one
percent um that's where we get a gamma
of 0.99 it's a hyper parameter you can
play around with it values like 0.95 all
the way up to 0.99 are typical
number of actions
will default it to two a lot of
environments only have two actions
the
max size of our memory that gets one
million one one two three one through
three
layer one size
equals default of 400 there to size
300 is our default and again that comes
from the paper and a batch size
for our batch learning from our replay
memory
so you want to go ahead and save the
parameters
equal tau
and you want to instantiate a memory
that's a replay buffer of size
max size
input dimms and n actions
we also want to store the batch size
for our learning function
we want to instantiate our first actor
yes there are more than one
and that gets alpha
input dims
layer one size
layer two size
and actions equals n actions
name equals
actor
so let's copy that
so next we have our target actor
so much like the deep q network
algorithm this uses target
networks as well as the base network so
it's an off policy method
and the difference here is this going to
be called
target actor it'll be otherwise
identical this will allow us to have
multiple different agents with similar
names
and you'll see how that plays into it
momentarily
we also need a critic
that's a critic network
takes beta
input dimms
they're one size
there are two size and actions equals n
actions
name equals
critic
so let's
be nice and tidy there
and
we also have a target critic as well
and that is otherwise identical it just
gets a different name
and this is very similar to q learning
where you have q eval and q next or
q target whatever you want to call it
same concept
okay so those are all of our networks
what else do we need we need noise
and that's our ou action noise
and the mu is just going to be numpy
zeros
of shape and actions so it'll give you
an array of zeros this is the mean of
the rewards over time
and next we need another function you
may be able to predict if you've seen my
videos on q learning which you should
check out
is the update network parameters
and we'll call it with an initial value
tie equals 1. so what this does is
it solves a problem of
a moving target so in q learning
if you use one network to calculate both
the
action as well as the value of that
action then you're really chasing a
moving target because you're updating
that estimate every turn right so
you are end up using the same parameter
for both and it can lead to divergence
so the solution to that is to use a
target network uh that learns the values
of these states
and action combinations and then the
other network is what learns the policy
and then of course periodically you have
to overwrite the
target parameter target networks
parameters with the evaluation network
parameters and this function will do
precisely that
except that we have four networks
instead of two
so next
we want to choose an action
and that takes whatever the current
observation of the environment is
now very very important
you have to put the actor into
evaluation mode now this doesn't perform
an evaluation step this just tells
pytorch that you don't want to calculate
statistics for the batch normalization
and this is very critical if you don't
do this the agent will not learn
and it doesn't do what you think the
name implies it would do right the
corresponding the
complementary function is trained
doesn't perform a training step it puts
it in training mode where it does store
those statistics in the graph for the
batch normalization if you don't do
batch norm then you don't need to do
this
but if you do what's the other function
um dropout dropout does the same thing
or has the same tick where you have to
call the eval and train functions
so let's start by putting our
observation into a tensor
d type because t dot float to
self.actor.device
so that'll turn it into a cuda float
tensor
now you want to get the
actual
action from the in the actor network so
feed that forward
to
cell.actor.device and just make sure
that you send it to the device so it's a
cuda tensor
so mu prime is going to be mu plus t dot
tensor what are we going to use
self.noise that will give us our
exploration noise
and that is going to be d type of float
and we will send that to
actor
device
and then you want to say
i should be actor.training shouldn't it
yes self.actor.train
yes
and then you want to return mu
prime
cpu.detach
dot numpy so this is an idiom within pi
torch
where you have to
basically do this otherwise it doesn't
um
it doesn't doesn't give you the actual
numpy value right it's going to try to
pass out a tensor which doesn't work
because you can't pass a tensor into the
open ai gym so
kind of a funny little quirk but it is
necessary
so now we need a function to store state
transitions
and this is just kind of an interface
for our
replay memory class
so
memory dot store transition state action
reward new state
done
simple
so now we come to the meat of the
problem where we actually the learning
so
you don't want to learn if you haven't
filled up at least batch size of your
memory buffer so
self.memory.mem
counter is less than self.batch size
then you just want to return
otherwise
action reward new state
done
you want to sample your memory
memory dot sample buffer
self.batch size
then you want to go ahead and turn all
of those into tensors
that's because they come back as numpy
arrays
in this case we'll put them on the
critic device as long as they're on the
same device it doesn't matter i do this
for consistency because these values
will be used
in the critic network
so
the duns
equal to tensor done
two
self self-critic device
you need the new state
peta tensor
new state
d type t float
two self critic device
you also need the actions
predict.device
and you need states
yeah tensor
device
and now we come to another quirk of
pytorch where we're gonna have to send
everything to eval mode
for the targets it may not be that
important i did it for
consistency
so we want to calculate the target
actions much like you do in the bellman
equation for q learning deep q learning
target actor
forward
new state
we want the critic value underscore
which is the new states
zelda target critic
dot forward
and that takes target actions as input
so what we're doing is getting the
target actions from the target actor
network in other words what actions
should it take based on the target
actors
estimates and then plugging that into
the state value function for the target
critic network
you also want the critic value which is
self.critic.forward
for state and action so in other words
what was the
what is your estimate of the values of
the states and actions we actually
encountered in our subset of the replay
buffer
so now we have to calculate the targets
that we're going to move towards
or j and range
so that batch size and i use a loop
instead of a vectorized implementation
because
the vectorized implementation is a
little bit tricky if you don't do it
properly you can end up with something
of shape batch size by batch size which
won't flag an error
but it definitely gives you the wrong
answer and you don't get learning so
target dot append
reward sub j plus self dot gamma times
critic
value underscore sub j times done sub j
so this is where i was talking about the
done flags if the episode is over then
the value of the resulting state is
multiplied by zero and so you don't take
it into account you only take into
account the reward from the current
state
precisely as one would want
so now let's go ahead and turn that
target into a tensor
sorry tensor
target
dot 2
self critic dot device
and we want to reshape this target
equals target dot view
self.batch size
and one
now now we can come to the calculation
of the loss functions so we want to set
the critic back into training mode
because we have
already performed the evaluation now we
want to actually calculate the values
for batch normalization
a train
cell.critic.optimizer.zero
grad
in pi torch whenever you calculate the
loss function you want to zero your
gradients that's so that gradients from
previous steps don't accumulate and
interfere with the calculation it can
slow stuff down
you don't want that
so critical loss is just
good grief f dot mse means square error
loss between the target and the critic
value so then you want to back propagate
back propagate that
backward
and step your optimizer
boom so that's it for the critic
so now we want to set the critic into
evaluation mode
for the calculation of the loss for our
actor network so
cell.actor.optimizer.0grad
and i apologize if this is confusing uh
it was confusing to me it took me a
while to figure it out uh this is one of
the ways in which tensorflow is superior
to pi torch you don't have this quirk uh
i tend to like
tensorflow a little bit better but
you know whatever we'll just figure it
out man and get it going so
mu
equals the forward propagation of the
state
i'm going to put the actor into training
mode
and you want to calculate your actor
loss that's just
minus self.critic dot forward state dot
state and mu
after loss equals t dot mean a back to
loss
again stay tuned for the derivation from
the paper this is all outlined there
otherwise it seems mysterious but this
video is already 45 minutes long so you
know that'll have to wait for a future
video
actor loss backward
and
dot self.actor.optimizer.step
and then we're done learning so now
after you finish learning you want to
update the network parameters for your
target actor and target critic networks
so self.update
network
parameters
man okay so
we're almost there i promise um
let's go ahead and do that def update
network
parameters
self and tau equals none by default
so tau is a parameter that allows the
update of the target network to
gradually approach the
evaluation networks
and this is important for a nice slow
convergence you don't want to take two
largest steps in between updates so
towel is a small number much much less
than one
so
so if tau is none
then you want to say tau equals self.tau
now this may seem mysterious the reason
i'm doing this is because
at the very beginning when we call the
initializer we say
uh update network parameters tau equals
1. this is because in that in the very
beginning we want to update or sorry we
want all the networks to start with the
same weights and so we call it with a
tau of one and uh in that case tau is
not none so tau is just one and you will
get the update rule here in a second so
this is more hocus pocus with um
pytorch
actor.named parameters
so this will do is it'll get all the
names of the parameters from these
networks
and we want to do the same thing for
target actor
parameters
uh target
critic params
okay
now that we have the parameters let's
turn them into a dictionary that makes
iterating them much easier
because this is actually a generator so
i believe
don't quote me on that
critic params
um the
actor state dict equals dict of the
actor params
target critic
state
uh let's just do target critic dict
equals dict
rams
params boom
okay
almost there so now we want to iterate
over these dictionaries
and copy parameters so for name in
critic state dict
critic state
dict sub name equals
equals tau times critic state dict
name
dot clone
plus
1 minus tau times target
critic
dict
name dot clone
excel.target
critic.load state dict
critic
state dict so what this does is it
iterates over this dictionary looks at
the key in the
in the uh in the dictionary and updates
the values
from this particular network and you can
see that when tau is one you get one
minus one is zero so it's just this
equals tau one times that so it's i the
identity and then it loads the target
critic with that parameter so at the
very beginning it'll load it with the
parameters from the initial uh
critic network and likewise for the
actor network so let's go ahead and copy
this
and just go ahead and change critic to
actor
and then we'll be done with that
function
and we'll only have one other thing to
take care of before we get to the
main
program
target actor
actor state dict
i believe that is it
yes indeed it
is
now it should be
target actor
yes
perfect okay now it's right
uh
so next up we have two other bookkeeping
functions to save the models so def save
models
and you definitely want this because
this thing takes forever to train
and self.critic
and
target actor
and
target critic
and
load models does the inverse operation
and let's just copy all this
load
keep things simple right
and again i will upload since this takes
so long i'm going to upload my saved
model parameters to the github for you
uh
but this is it this is 275 lines so this
is probably the longest project we have
worked on here at machine learning with
phil
if you've made it this far
congratulations we're already 50 minutes
in and we're almost done i promise so
let's come over to our main function
and
we want to
import our agent so ddpg torch import
agent
we want to import
gym
we want to do we want yes we want numpy
we want um
my super duper awesome
import plot learning
function
and that is it so env gem dot make
lunar lander contin us v2
agent
equals agent
alpha
two three 0.1234 two five so two point
five by ten to the minus five
beta equals zero point zero zero zero
two five so two point five by ten to the
minus four
input dimms equals a list with element
eight
tau tal
0.001
in vehicles env
well that reminds me i didn't multiply
by the action space
high in the function for the choose
action don't worry that'll be in the
tensorflow implementation uh or i can
leave it as an exercise to the reader it
doesn't matter for this environment uh
when we get to other ones that where it
doesn't matter i'll
be a little bit more diligent about that
batch size is 64
size 400 layer 2 size 300 and actions
equals two
now another interesting tidbit is that
we have to set the random seed uh this
is not something i've done before but
this is a highly
sensitive learning method uh so if you
read the original paper they do averages
over five runs
uh and that's because every run is a
little bit different and i suspect
that's why they had to initialize the
weights and biases within such a narrow
range right you don't want to go all the
way from plus and minus one when you can
constrain to something much more narrow
it gives you a little bit more
repeatability uh so
we have to set the numpy random seed to
some value instead of none so in this
case i view zero i've seen other values
used please
clone this and see what happens if you
input other seed values
so next we need a score history to keep
track of the
scores over time
and we need to
iterate over a thousand games
done equals false
score equals zero
uh observation equals env dot reset
that'll get a new observation
so while not done
agent dot
choose
action
obs
new state
reward done info equals env step act
agent.remember we want to keep track of
that transition obs act reward new state
and done
agent.learn we learn on every step
because this is a temporal difference
learning method instead of a monte carlo
type method where we would learn at the
end of every episode
keep track of the score
and set your old state to the new state
so at the end of the episode if
[Music]
at the end of every episode
we want to print print the place marker
so we'll say score history
dot append score
print episode
i score
percent.2f
score
um
100 game average
percent.2f
and what this will do is
take the last 100 games and compute the
mean so that way you can get an idea of
its learning remember with the lunar
lander environment solved means that it
has gotten a an average score of 1 200
over the last 100 games
so every 25 games we want to save
the model
agent does save models
and at the end file name equals lunar
lander png
that's not in the loop you want to do at
the end of all the loop games plot
learning
i'll name no score history
file name and a window of 100 games
wow so
an hour in we finally finished this now
we get to go to the terminal and see how
many typos i made i'm sure there's
probably 50 so let's get to it
alright so here we are let's see what we
get
we want to run torch lunar lander
fingers crossed okay so that's a stupid
one so
in line 30 we forgot an equal sign let's
go back there and fix that
all right here we are so it says
line 30
yes right here
all right did we do that again anywhere
else not that i can see
but that's no guarantee alright so let's
go back to the terminal
all right let's try it again
line 119 okay
uh typical
all right so 119
right there
so that's in the actor let's just scroll
down
that's the agent class
i don't think i did it there all right
i'm going back to the terminal
all right so i started it and it ran so
let's see
built-in function has no
function or method has no attribute
numpy all right that's an interesting
bug let's fix that
so that is on line
192 in our choose action function
and
mu prime
oh that's why
it's detached as a function not an
object that should fix it let's head
back to the terminal
rewards is not defined
so that is in line 55
okay
ah it's just called reward
there we go add the s and back to the
terminal
ah
perfect
that's easy to fix mr
temp slash udpg
it's because i didn't make the directory
first
that's easy
perfect now it's running so i'm not
going to let this run all 1000 games
because it takes about a couple hours or
so instead let's take a look here
so i was running this earlier when i was
making the uh videos for
the uh sorry the recording the agents
play while making uh for this video and
you can see that within
under 650 games it went ahead and solved
it so when you print out the trailing
average for the last hundred games we
get a reward of well over 200
now keep in mind uh one interesting
thing is that this is still
actually in training mode it's not
excuse me it's not in full evaluation
mode because we still have some noise
right if you wanted to do a pure
evaluation of the agent you would set
the noise to zero we'll do that in a set
of future videos there's a whole bunch
of stuff i can do on this topic uh but
just keep in mind that this is an agent
that is still taking some random actions
the noise is nonzero and so it is still
taking sub-optimal actions and getting a
score of 260 and still beating the
environment even though that noise is
present and you can see it in
uh like episode 626 where it gets a
score of 26 so uh and then in episode
624 where it does you know 8.58 points
so
uh that is pretty cool stuff so this is
a very powerful algorithm and keep in
mind this was a continuous action space
totally intractable for q learning right
that is simply not possible uh it's an
infinitude of of actions so you need
something like ddpg to handle this and
it handles it quite well in future
videos we're going to get to the walker
the bipedal walker we're going to get to
the
learning from pixels where we do the
continuous
racing environment and we'll probably
get into other stuff from the robo
school of the open ai gym so make sure
to subscribe so that you can see that in
future videos uh go ahead and check out
the github for this so you can get the
weight so you can play around with this
so you don't have to spend a couple
hours training it on your gpu
make sure to leave a like share this if
you found it helpful that is incredibly
helpful to me uh and leave a comment
down below i answer all my questions i
look forward to seeing you all in the
next video
you