now that we have an understanding of the reinforcement learning workflow in this
video I want to show how that workflow is put to use in getting a bipedal robot
to walk using an RL equipped agent we're going to use the walking robot example
from the MATLAB and Simulink robotics arena that you can find on github I've
left a link to it in the description this example comes with an environment
model where you can adjust training parameters train an agent and visualize
the results and in this video we're also going to look at how we can modify this
example to make it look more like how we would set up a traditional controls
problem and then show some of the limitations of the design so I hope you
stick around for this because I think it'll help you understand how to use
reinforcement learning for typical control applications I'm Brian and
welcome to am a lab tech talk let's start with a quick overview of the
problem the high-level goal is to get a two-legged robot to walk somewhat like
how a human would our job as designers is to determine the actions to take that
correctly move the legs and body of the robot and the actions that we can take
are the motor torque commands for each joint there's the left and right ankle
the left and right knee and the left and right hip joints so in all there are six
different torque commands that we need to send at any given time now the robot
body and legs along with the world in which it operates makes up the
environment the observations from the environment are based on the type and
locations of sensors as well as any other data that is generated by the
software for this example we're using 31
different observations these are the y&z body positions the x y&z body velocity
and the body orientation and angular rate there were also the angles and
angular rates of the six joints and the contact forces between the feet and the
ground those are the sensed observations we're also feeding back the six actions
that we commanded in the previous time step which are stored in a buffer in
software so in all our control system would need to take in these 31
observations and then calculate the values of the six motor torques
continuously so you can start to see how complex the logic has to be for even
this really simple system now as I mentioned in previous videos
rather than try to design the logic the loops the controllers the parameters and
all that stuff using the traditional tools of control theory we can replace
this whole massive function in to end with a reinforcement learning agent one
that uses an actor network to map these thirty-one observations to the six
actions and a critic network to make the training more efficient as we know the
training process requires a reward function something that tells the agent
how it's doing so that it can learn from its actions and I want to reason through
what should exist in the reward function by thinking about the conditions that
are important for a walking robot this might be how you'd approach building up
a reward function if you didn't know where to start now I'm gonna show you
the results of training with this function as we create it so you can see
how the changes impact the solution however I'm not going to go over how to
run the model because there's already a great video by Sebastian Castro that
does just that so if you're interested in trying out all of this on your own
I recommend checking out the link in the description below alright so on to the
reward where to start we obviously want the body of the robot to move forward
otherwise it's just going to stand there but instead of distance we can reward it
for its forward velocity that way there's a desire for the robot to walk
faster rather than slower in training with this reward we can see that the
robot dives forward to get that quick burst of speed at the beginning and then
it falls over and doesn't really make it anywhere now it may eventually figure
out how to make it further with this reward but it was taking a long time to
converge and not making a whole lot of progress so let's think about what we
can add to the reward to help with training
we could penalize the robot for falling down so that diving forward isn't as
attractive so if it stays on its feet for a longer time or if more sample
times elapse before the simulation ends then the agent should get more reward so
let's see how this one does
it's got a bit of a hop at the beginning before ultimately falling down perhaps
if I had let this agent train longer I could have had a robot that jumps across
the world like a frog which is cool but not what I'm going for so it's not
enough that the robot moves forward and doesn't fall we want some semblance of
walking instead of hopping or Crouch walking so to fix this we should also
reward the agent for keeping the body as close to a standing height as possible
let's check out this reward function
okay this one's looking better but the solution isn't really natural looking it
stops occasionally to sort of jitter its legs back and forth and most of the time
it's dragging its right leg like a zombie and putting all of the actuation
in to the left leg and this isn't ideal if we're concerned with actuator
wear-and-tear or the amount of energy it takes to run this robot we'd like both
legs to do equal work and not to overuse the actuators with a lot of jittering so
to fix this we can reward the agent for minimizing actuator effort this should
reduce extra jittering and balance the effort so that each leg has a share of
the load let's check out this trained agent
okay we're getting really close here this is looking pretty good except now
we have one final problem we want to keep the robot moving in a straight line
and not veering off to the right or to the left like it's doing here
so we need to reward it for staying close to the x-axis this is our final
reward and training with it needs about 3500 simulations so if we set this up in
our model and we unleash the simulation on a computer with multiple cores or a
GPU or on a cluster of computers then after a few hours of training we're
gonna have a solution we'll have a robot that walks in a straight line in a
human-like way
so with the reward function set let's move on to the policy I've already
stated that the policy is an actor neural network and along with it is a
critic neural network and each of these networks have several hidden layers of
hundreds of neurons each so there's a lot of calculations that go into them if
we don't have enough neurons then the network will never be able to mimic the
high dimensional function that is required to map the 31 observations to
the six actions for this nonlinear environment on the other hand too many
neurons and we spend more time training the excessive logic in addition the
architecture of the network is really important in functional complexity these
are things like the number of layers and how they're connected and the number of
neurons in each layer so there is some experience and knowledge needed to find
that sweet spot that makes training possible and efficient luckily as we
know we don't need to manually solve for the hundreds of thousands of weights and
biases and our networks we let the training algorithm do that for us in
this example we're using an actor critic training algorithm called the deep
deterministic policy gradient or DDP G the reason is because this algorithm can
learn with environments with continuous action spaces like we have with the
continuous range of torques that we can apply to the motors also since it
estimates a deterministic policy it's much faster to learn than one that
learns a stochastic policy I know this all sounds fairly complicated and
abstract but the cool thing to me about this is that most of the complexity is
there for training the policy once we have a fully trained agent then all we
have to do is deploy the actor network to the target hardware remember the
actor is the function that map's observations to actions it's the thing
that's deciding what to do it's the policy the critic and the learning
algorithms are just there to help determine the parameters in the actor
okay at this point here's a question that you might have sure we can use RL
to get a robot to walk in a straight line however it won't this policy do
only this one thing for instance if I deploy this policy and turn on my robot
it's just going to instantly start walking
straight forever so how can I learn a policy that will let me send commands to
the robot to walk where I want it to walk well let's think about that right
now this is what our system looks like we have the environment that generates
the observations and the reward and then we have the agent that generates the
actions there's no way for us to inject any outside commands into this system
and there's no way for our agent to respond to them even if we had them so
we need to write some additional logic outside of the agent that receives a
reference signal and calculates an error term the error is the difference between
the current exposition which we can get from the environment and the reference
value this is the same error calculation that we would have in a normal feedback
control system now rather than reward the agent for a higher velocity in the
x-direction we can reward it for low error this should incentivize the robot
to walk towards and stay at the commanded X reference value for the
observations we need to give the agent a way to view the error term so that it
can develop a policy accordingly since it might help our agent to have access
to the rate of change of error and maybe other higher derivatives I'll feed in
the error from the last five sample times this will allow the policy to
create derivatives if it needs to eventually the policy will be to walk
forward at some specified rate if the error is positive and then backwards if
it's negative and since we now have 36 observations into our agent we need to
adjust our actor network to handle the additional inputs again check out
Sebastian's video in the description if you need guidance on how to make those
changes I've updated the default model in simulink with the new error term and
fed it into the observation block and the reward block and I've trained this
agent over thousands of episodes using this particular profile so it should be
really good at following this profile but the hope is that the trained policy
will be robust enough to follow other profiles as well that have similar rates
and accelerations so let's give this a shot
I'll have it walk forward and then pause for a bit and then walk backwards
you
okay it's kind of funny looking when it walks backwards but overall a pretty
good effort with some reward tweaking and maybe a little more time spent
training I might be on to something pretty good here so in this way you can
start to see how we can use an RL agent to replace part of the control system
instead of a function that learns a single behavior we can extract the
high-level reference signal and have the agent work off error so that we can
retain the ability to send commands we can also remove low-level functionality
from the agent for example instead of the actions being the low-level torques
for each of the six joints the agent could just learn where to place its feet
on the ground so the action would be place left foot at some location in the
body coordinate frame this action could be the reference command for a
lower-level traditional control system that drives the joint motors you know
something that might feed forward a torque command based on your knowledge
of the dynamics of the system and then feed back some signal to guarantee
performance and stability and this is beneficial because we can use our
specific domain knowledge to solve the easy problems and that's going to give
us insight and control over the design and then we can reserve reinforcement
learning for the problems that are more difficult something to note about our
final walking solution is that it's really only robust to its own State you
know it can walk around without falling over which is good but only in a
perfectly flat featureless plane it's not taking into account any part of the
world outside of the robot so it's actually quite a fragile design for
example let's see what happens if we place a single obstacle in the way of
our robot
well that went pretty much as expected the problem here is that we didn't give
our agent any way of recognizing the state of the environment beyond the
motions of the robot itself there's nothing that can sense the obstacles and
therefore nothing can be done to avoid them but here's the thing the beauty of
neural network based agents is that they can handle what we'll call rich sensors
these are things like lidar invisible cameras things that don't produce
singular measurements like an angle but rather return arrays of numbers that
represent thousands of distances or pixels of varying light intensities so
we could install a visible camera and a lidar sensor on our robot and feed the
thousand or so new values as additional observations into our agent you can
imagine how the complexity of this function needs to grow as our
observations increase from 36 to thousands we may find that a simple
fully connected network is not ideal and so we may add additional layers that
incorporate specialized logic that minimizes connections like convolutional
networks or things that add memory like recurrent networks
these are Network layers that are more adapted to dealing with large image data
in more dynamic environments however we might not need to change the reward
function in order to get the robot to avoid these obstacles the agent could
still learn that straying from the path and therefore getting a lower reward
here allows the robot to continue walking
without falling down thereby earning more reward overall now I've addressed a
few of the problems with reinforcement learning in this video and I showed how
we can modify the problem by combining the benefits of traditional control
design with reinforcement learning now we're going to expand on this some more
in the next video where we'll talk about other downsides of reinforcement
learning and what we can do to mitigate them so if you don't want to miss that
and future tech talk videos don't forget to subscribe to this channel also if you
want to check out my channel control system lectures I cover more control
topics there as well thanks for watching and I'll see you next time