in the last video we covered a high-level introduction to reinforcement
learning and I ended it by telling you that we can't just jump into RL by
unleashing a learning algorithm on a problem while being completely ignorant
of what it means to set the problem up correctly now in order to get started
you should have an understanding of the RL workflow and how each part of the
process contributes to solving the problem and what are some of the
decisions you'll have to make along the way so in this video we're gonna build
on our basic understanding of reinforcement learning and explore what
it means to set up the problem I'm Brian and welcome to a MATLAB Tech Talk now I
know I said that understanding the system you're trying to control is the
first step because you don't want to choose reinforcement learning if a
traditional controls approach is better however I think it's easier to make that
choice if you have a good understanding of the RL workflow therefore for now
assume that we've already decided we want to use reinforcement learning and
we'll cover when not to use it in a future video once we've built up this
good foundation okay so let's jump into the workflow first we need an
environment where our agent can learn and therefore we need to choose what
should exist within the environment and whether it's a simulation or a real
physical setup then we need to think about what we ultimately want our agent
to do and craft a reward function that will incentivize the agent to do just
that we need to choose a way to represent a policy how we want to
structure the parameters and logic that make up the decision-making part of the
agent and once we have this set up we choose a training algorithm to get to
work finding the optimal policy finally we need to exploit the policy by
deploying it onto an agent in the field and verifying the results and to put
this workflow into perspective let's think about each of these steps in the
context of two examples balancing and inverted pendulum and getting a robot to
walk so let's get to it the environment is everything that exists outside of the
agent practically speaking it's where the agent sends actions and it's what
generates rewards and observations I think this concept is a bit confusing at
first especially coming from a controls background because we tend to think of
the environment as everything outside of the controller and the plant things like
road in affections wind gusts and other
disturbances that impact the system you're trying to control but in
reinforcement learning the environment is everything outside of the controller
so this would include the plant dynamics as well for the walking robot example
most of the robot is part of the environment the agent is just the bit of
software that's generating the actions and updating the policy through learning
it's the brain of the robot so to speak and the reason this distinction is
important is because with reinforcement learning the agent doesn't need to know
anything about the environment at all this is called model free RL and it's
powerful because you can basically just plop an RL equipped agent into any
system and assuming you've given the policy access to observations actions
and enough internal states then the agent will learn how to collect the most
reward on its own this means the agent doesn't need to know initially anything
about our walking robot it'll still figure out how to collect rewards
without knowing for example how the joints move or how strong the actuators
are or the lengths of the appendages but as engineers we do typically know some
things about the environment so why are we throwing out all of our knowledge of
physics and not helping the agent at all this seems crazy right well this is
where model-based RL can help without any understanding of the environment an
agent needs to explore all areas of the state space to fill out its value
function which means that it'll spend some time exploring low reward areas
during the learning process however as designers we often know some parts of
the state space that aren't worth exploring and so by providing a model of
the environment or part of the environment we provide the agent with
that knowledge for example take an agent that's trying to determine the fastest
route to a destination should it go right or left at this point
without a model the agent would have to explore the entire map in order to know
what the best action is at this point but with a model the agent could explore
going right sort of mentally without having to take the action physically it
could then figure out that going right results in a dead end and our agent
would then choose to go left in this way a model can complement the learning
process by avoiding areas that are known to be bad and exploring and learning the
model based RL is pretty powerful but the reason model free RL is so
popular right now is because people hope to use it to solve problems we're
developing a model is difficult such as controlling a car or a robot from pixel
observations and also since model free RL is the more general case we're gonna
focus on it for the rest of this series okay so we know the agent learns by
interacting with the environment so it makes sense that we have to have a way
for the agent to actually interact with it and this might be a physical
environment or a simulation for example for the inverted pendulum we may let the
agent learn how to balance by running it with the physical pendulum setup this
might be a good solution since it's probably hard for the hardware to damage
itself or others with the walking robot however this might not be such a good
idea as you can imagine if the agent treats the robot and the world like a
black box that it knows nothing about then it's going to do a lot of falling
and flailing before it even learns how to move its legs let alone learns how to
walk not only could this damage the hardware but it would be extremely
time-consuming to have to pick the robot up each time so not ideal an attractive
alternative is to train your agent within a high fidelity model of the
environment and just simulate the learning experience and there's a lot of
benefits to doing this the first comes from the idea of sample inefficiency
learning is a process and that requires lots of samples lots of trials errors
and corrections often in the millions or tens of millions and so with the
simulation you have the ability to run the learning process at faster than real
time and you can also spin up lots of simulations and run them all in parallel
the other beneficial thing you can do with a model of the environment is
simulate conditions that are hard to test for in the real world for example
with the walking robot you could simulate walking on a low friction
surface like ice which would help the robot stay upright on all surfaces and
the nice part about needing a simulation is that for control problems we usually
already have a good model of our system and environment since we typically need
it from a traditional control design this is where if you already have a
model built in MATLAB or Simulink you can replace your existing controller
with an RL agent add a reward function to the environment and start the
learning process and this model does just that the RL agent is a block from
the reinforcement learning toolbox and these three subsystems generate the
observation the reward and sets a flag that will stop the learning process now
one of the difficulties here is figuring out how much of the environment to model
what to include and what to leave out however this is the same question you
have when modeling a plant for a controller design and so you can use the
same intuition about your system to build an RL environment model and one
approach is to just start training on a simple model find the right combination
of hyper parameters that will let training succeed and then add more
complexity to the model later on hyper parameters are the knobs that we can
turn on the training algorithms that set things like the learning rate and the
sample times and I'll cover more of this in detail in a future video ok with the
environment set the next thing to think about is what you want your agent to do
and how you're going to reward it for doing what you want this is similar to
the cost function in lqr where we think about performance versus effort however
unlike lqr where the cost function is quadratic in RL there's really no
restrictions on creating a reward function we can have sparse rewards or
rewards every time step or rewards that only come at the very end of an episode
after long periods of time they can be calculated from a nonlinear function or
calculated using thousands of parameters really it completely depends on what it
takes to effectively train your agent do you want to get an inverted pendulum to
stand upright then maybe give more rewards to your agent as the angle from
vertical gets smaller want to take controller effort into account then
subtract rewards as actuator use increases
want to encourage a robot to walk across the floor then give the agent a reward
when it reaches some state that's far away now with that being said making a
reward function is easy it can be pretty much any function you can think of
making a good reward function on the other hand is really really hard and
unfortunately there isn't a straightforward way to craft your reward
to guarantee that your agent will converge on the solution you actually
want and I think that boils down to two main reasons one often the goal you want
to incentivize comes after a long sequence of actions this is the sparse
reward system therefore your agent will stumble around for long periods of time
not receiving any rewards in the process this would be the case for the walking
robot by only giving a reward after the robot successfully walked 10 meters the
chance that your agent will randomly stumble on the action sequence that
produces the sparse reward is very unlikely
imagine the luck needed to generate all of the correct motor commands to keep a
robot upright and walking rather than just flopping around on the ground it's
possible but relying on that random exploration is extremely slow to where
it's impractical this sparse reward problem can be improved by shaping the
reward providing smaller intermediate rewards that coax the agent along the
right path but reward shaping comes with its own set of problems and this is the
second reason crafting a reward function is difficult if you give an optimization
algorithm a shortcut it'll take it and shortcuts are hidden within reward
functions and more so when you start shaping them this causes your agent to
converge on a solution that is optimal given the reward function but not ideal
an easy example to think about is giving an intermediate reward if the body of
the robot travels one meter from its current spot the optimal solution might
not be to walk that one meter but rather fall ungracefully towards the reward to
the learning algorithm walking and falling both provide the same reward but
obviously to the designer one result is preferred over the other now I don't
want to make crafting a reward function sound easy because getting it right is
possibly one of the more deaf tasks in reinforcement learning however
hopefully with this general overview you'll be in a better position to at
least understand some of the things that you need to watch out for and that might
make crafting the reward function a little less painful okay so now that we
have the environment which provides the rewards we're ready to start work on the
agent itself and the agent is comprised of the policy and the learning algorithm
and these two things are intimately intertwined many learning algorithms
require a specific policy structure and choosing an algorithm depends on the
nature of the environment and we're gonna cover all of this in the next
video but before I end here I want to introduce the topic and get us thinking
about how of the parameters and the logic within the policy can be
represented remember the policy is a function that takes in state
observations and outputs actions so really any function with that input and
output relationship can work in that way of thinking we could use a simple table
to represent policies and tables are exactly what you would expect they're an
array of numbers where you use an input as a lookup address and output the
corresponding value for example a cue function is a table that map's States
and actions to value so given a state s the policy would be to look up the value
of every possible action from that state and then choose the action with the
highest value and training an agent with a cue function would consist of
developing over time all of the actions and their values for every state
now this type of representation falls apart when the number of action value
pairs get really large or becomes infinite this is the so called curse of
dimensionality imagine our inverted pendulum the state of the pendulum can
be any angle from minus PI to PI and the action that you take can be any motor
torque from the negative limit to the positive limit and trying to capture all
of that in a table is not feasible now we could represent the continuous
nature of the state action space with a continuous function but setting up this
function where we could learn the right parameters would require us to know the
structure of the function ahead of time which might be difficult for high degree
of freedom systems or nonlinear systems so instead it makes sense to try to
represent the policy with a general purpose function approximator something
that can handle continuous state action spaces without us having to set the
structure ahead of time and we get that with deep neural networks and that's
where I'm gonna leave this for now and we'll continue this discussion in
the next video so if you don't want to miss future Tech Talk videos don't
forget to subscribe to this channel also if you want to check out my channel
control system lectures I cover more control theory topics there as well
thanks for watching and I'll see you next time