now that we have an understanding of the
reinforcement learning workflow in this
video I want to show how that workflow
is put to use in getting a bipedal robot
to walk using an RL equipped agent we're
going to use the walking robot example
from the MATLAB and Simulink robotics
arena that you can find on github I've
left a link to it in the description
this example comes with an environment
model where you can adjust training
parameters train an agent and visualize
the results and in this video we're also
going to look at how we can modify this
example to make it look more like how we
would set up a traditional controls
problem and then show some of the
limitations of the design so I hope you
stick around for this because I think
it'll help you understand how to use
reinforcement learning for typical
control applications I'm Brian and
welcome to am a lab tech talk let's
start with a quick overview of the
problem the high-level goal is to get a
two-legged robot to walk somewhat like
how a human would our job as designers
is to determine the actions to take that
correctly move the legs and body of the
robot and the actions that we can take
are the motor torque commands for each
joint there's the left and right ankle
the left and right knee and the left and
right hip joints so in all there are six
different torque commands that we need
to send at any given time now the robot
body and legs along with the world in
which it operates makes up the
environment the observations from the
environment are based on the type and
locations of sensors as well as any
other data that is generated by the
software
for this example we're using 31
different observations these are the y&z
body positions the x y&z body velocity
and the body orientation and angular
rate there were also the angles and
angular rates of the six joints and the
contact forces between the feet and the
ground those are the sensed observations
we're also feeding back the six actions
that we commanded in the previous time
step which are stored in a buffer in
software so in all our control system
would need to take in these 31
observations and then calculate the
values of the six motor torques
continuously so you can start to see how
complex the logic has to be for even
this really simple system
now as I mentioned in previous videos
rather than try to design the logic the
loops the controllers the parameters and
all that stuff using the traditional
tools of control theory we can replace
this whole massive function in to end
with a reinforcement learning agent one
that uses an actor network to map these
thirty-one observations to the six
actions and a critic network to make the
training more efficient as we know the
training process requires a reward
function something that tells the agent
how it's doing so that it can learn from
its actions and I want to reason through
what should exist in the reward function
by thinking about the conditions that
are important for a walking robot this
might be how you'd approach building up
a reward function if you didn't know
where to start now I'm going to show you
the results of training with this
function as we create it so you can see
how the changes impact the solution
however I'm not going to go over how to
run the model because there's already a
great video by Sebastian Castro that
does just that so if you're interested
in trying out all of this on your own
I'd recommend checking out the link in
the description below
alright so on to the reward where to
start we obviously want the body of the
robot to move forward otherwise it's
just going to stand there but instead of
distance we can reward it for its
forward velocity that way there's a
desire for the robot to walk faster
rather than slower in training with this
reward we can see that the robot dives
forward to get that quick burst of speed
at the beginning and then it falls over
and doesn't really make it anywhere now
it may eventually figure out how to make
it further with this reward but it was
taking a long time to converge and not
making a whole lot of progress so let's
think about what we can add to the
reward to help with training
we could penalize the robot for falling
down so that diving forward isn't as
attractive so if it stays on its feet
for a longer time or if more sample
times elapsed before the simulation ends
then the agent should get more reward so
let's see how this one does
it's got a bit of a hop at the beginning
before ultimately falling down perhaps
if I had let this agent train longer I
could have had a robot that jumps across
the world like a frog which is cool but
not what I'm going for so it's not
enough that the robot moves forward and
doesn't fall we want some semblance of
walking instead of hopping or crouch
walking so to fix this we should also
reward the agent for keeping the body as
close to a standing height as possible
let's check out this reward function
okay this one's looking better but the
solution isn't really natural looking it
stops occasionally to sort of jitter its
legs back and forth and most of the time
it's dragging its right leg like a
zombie and putting all of the actuation
in to the left leg and this isn't ideal
if we're concerned with actuator
wear-and-tear or the amount of energy it
takes to run this robot we'd like both
legs to do equal work and not to overuse
the actuators with a lot of jittering so
to fix this we can reward the agent for
minimizing actuator effort this should
reduce extra jittering and balance the
effort so that each leg has a share of
the load let's check out this trained
agent
okay we're getting really close here
this is looking pretty good except now
we have one final problem we want to
keep the robot moving in a straight line
and not veering off to the right or to
the left like it's doing here
so we need to reward it for staying
close to the x-axis this is our final
reward and training with it needs about
3500 simulations so if we set this up in
our model and we unleash the simulation
on a computer with multiple cores or a
GPU or on a cluster of computers then
after a few hours of training we're
going to have a solution we'll have a
robot that walks in a straight line in a
human-like way
so with the reward function set let's
move on to the policy I've already
stated that the policy is an actor
neural network and along with it is a
critic neural network and each of these
networks have several hidden layers of
hundreds of neurons each so there's a
lot of calculations that go into them if
we don't have enough neurons then the
network will never be able to mimic the
high dimensional function that is
required to map the 31 observations to
the six actions for this nonlinear
environment on the other hand too many
neurons and we spend more time training
the excessive logic in addition the
architecture of the network is really
important in functional complexity these
are things like the number of layers and
how they're connected and the number of
neurons in each layer so there is some
experience and knowledge needed to find
that sweet spot that makes training
possible and efficient luckily as we
know we don't need to manually solve for
the hundreds of thousands of weights and
biases in our networks we let the
training algorithm do that for us in
this example we're using an actor critic
training algorithm called the deep
deterministic policy gradient or DDP G
the reason is because this algorithm can
learn with environments with continuous
action spaces like we have with the
continuous range of torques that we can
apply to the motors also since it
estimates a deterministic policy it's
much faster to learn than one that
learns a stochastic policy I know this
all sounds fairly complicated and
abstract but the cool thing to me about
this is that most of the complexity is
there for training the policy once we
have a fully trained agent then all we
have to do is deploy the actor network
to the target hardware remember the
actor is the function that map's
observations to actions it's the thing
that's deciding what to do it's the
policy the critic and the learning
algorithms are just there to help
determine the parameters in the actor
okay at this point here's a question
that you might have sure we can use our
L to get a robot to walk in a straight
line however it won't this policy do
only this one thing for instance if I
deploy this policy and turn on my robot
it's just going to instantly start
walking
straight forever so how can I learn a
policy that will let me send commands to
the robot to walk where I want it to
walk well let's think about that right
now this is what our system looks like
we have the environment that generates
the observations and the reward and then
we have the agent that generates the
actions there's no way for us to inject
any outside commands into this system
and there's no way for our agent to
respond to them even if we had them so
we need to write some additional logic
outside of the agent that receives a
reference signal and calculates an error
term the error is the difference between
the current exposition which we can get
from the environment and the reference
value this is the same error calculation
that we would have in a normal feedback
control system now rather than reward
the agent for a higher velocity in the
x-direction we can reward it for low
error this should incentivize the robot
to walk towards and stay at the
commanded X reference value for the
observations we need to give the agent a
way to view the error term so that it
can develop a policy accordingly since
it might help our agent to have access
to the rate of change of error and maybe
other higher derivatives I'll feed in
the error from the last five sample
times this will allow the policy to
create derivatives if it needs to
eventually the policy will be to walk
forward at some specified rate if the
error is positive and then backwards if
it's negative and since we now have 36
observations into our agent we need to
adjust our actor network to handle the
additional inputs again check out
Sebastian's video in the description if
you need guidance on how to make those
changes
I've updated the default model in
simulink with the new error term and fed
it into the observation block and the
reward block and I've trained this agent
over thousands of episodes using this
particular profile so it should be
really good at following this profile
but the hope is that the trained policy
will be robust enough to follow other
profiles as well that have similar rates
and accelerations so let's give this a
shot I'll have it walk forward and then
pause for a bit and then walk backwards
you
okay it's kind of funny looking when it
walks backwards but overall a pretty
good effort with some reward tweaking
and maybe a little more time spent
training I might be on to something
pretty good here so in this way you can
start to see how we can use an RL agent
to replace part of the control system
instead of a function that learns a
single behavior we can extract the
high-level reference signal and have the
agent work off error so that we can
retain the ability to send commands we
can also remove low-level functionality
from the agent for example instead of
the actions being the low-level torques
for each of the six joints the agent
could just learn where to place its feet
on the ground so the action would be
place left foot at some location in the
body coordinate frame this action could
be the reference command for a
lower-level traditional control system
that drives the joint motors you know
something that might feed forward a
torque command based on your knowledge
of the dynamics of the system and then
feed back some signal to guarantee
performance and stability and this is
beneficial because we can use our
specific domain knowledge to solve the
easy problems and that's going to give
us insight and control over the design
and then we can reserve reinforcement
learning for the problems that are more
difficult something to note about our
final walking solution is that it's
really only robust to its own State you
know it can walk around without falling
over which is good but only in a
perfectly flat featureless plane it's
not taking into account any part of the
world outside of the robot so it's
actually quite a fragile design for
example let's see what happens if we
place a single obstacle in the way of
our robot
well that went pretty much as expected
the problem here is that we didn't give
our agent any way of recognizing the
state of the environment beyond the
motions of the robot itself there's
nothing that can sense the obstacles and
therefore nothing can be done to avoid
them but here's the thing the beauty of
neural network based agents is that they
can handle what we'll call rich sensors
these are things like lidar invisible
cameras things that don't produce
singular measurements like an angle but
rather return arrays of numbers that
represent thousands of distances or
pixels of varying light intensities so
we could install a visible camera and a
lidar sensor on our robot and feed the
thousand or so new values as additional
observations into our agent you can
imagine how the complexity of this
function needs to grow as our
observations increase from 36 to
thousands we may find that a simple
fully connected network is not ideal and
so we may add additional layers that
incorporate specialized logic that
minimizes connections like convolutional
networks or things that add memory like
recurrent networks
these are Network layers that are more
adapted to dealing with large image data
in more dynamic environments however we
might not need to change the reward
function in order to get the robot to
avoid these obstacles the agent could
still learn that straying from the path
and therefore getting a lower reward
here
allows the robot to continue walking
without falling down thereby earning
more reward overall now I've addressed a
few of the problems with reinforcement
learning in this video and I showed how
we can modify the problem by combining
the benefits of traditional control
design with reinforcement learning now
we're going to expand on this some more
in the next video where we'll talk about
other downsides of reinforcement
learning and what we can do to mitigate
them so if you don't want to miss that
and future tech talk videos don't forget
to subscribe to this channel also if you
want to check out my channel control
system lectures I cover more control
topics there as well thanks for watching
and I'll see you next time