in the last video we covered a
high-level introduction to reinforcement
learning and I ended it by telling you
that we can't just jump into RL by
unleashing a learning algorithm on a
problem while being completely ignorant
of what it means to set the problem up
correctly now in order to get started
you should have an understanding of the
RL workflow and how each part of the
process contributes to solving the
problem and what are some of the
decisions you'll have to make along the
way so in this video we're gonna build
on our basic understanding of
reinforcement learning and explore what
it means to set up the problem I'm Brian
and welcome to a MATLAB Tech Talk now I
know I said that understanding the
system you're trying to control is the
first step because you don't want to
choose reinforcement learning if a
traditional controls approach is better
however I think it's easier to make that
choice if you have a good understanding
of the RL workflow therefore for now
assume that we've already decided we
want to use reinforcement learning and
we'll cover when not to use it in a
future video once we've built up this
good foundation
okay so let's jump into the workflow
first we need an environment where our
agent can learn and therefore we need to
choose what should exist within the
environment and whether it's a
simulation or a real physical setup then
we need to think about what we
ultimately want our agent to do and
craft a reward function that will
incentivize the agent to do just that we
need to choose a way to represent a
policy how we want to structure the
parameters and logic that make up the
decision-making part of the agent and
once we have this set up we choose a
training algorithm to get to work
finding the optimal policy finally we
need to exploit the policy by deploying
it onto an agent in the field and
verifying the results and to put this
workflow into perspective let's think
about each of these steps in the context
of two examples balancing an inverted
pendulum and getting a robot to walk so
let's get to it the environment is
everything that exists outside of the
agent practically speaking it's where
the agent sends actions and it's what
generates rewards and observations I
think this concept is a bit confusing at
first especially coming from a controls
background because we tend to think of
the environment as everything outside of
the controller and the plant things like
road in
affections wind gusts and other
disturbances that impact the system
you're trying to control but in
reinforcement learning the environment
is everything outside of the controller
so this would include the plant dynamics
as well for the walking robot example
most of the robot is part of the
environment the agent is just the bit of
software that's generating the actions
and updating the policy through learning
it's the brain of the robot so to speak
and the reason this distinction is
important is because with reinforcement
learning the agent doesn't need to know
anything about the environment at all
this is called model free RL and it's
powerful because you can basically just
plop an RL equipped agent into any
system and assuming you've given the
policy access to observations actions
and enough internal states then the
agent will learn how to collect the most
reward on its own this means the agent
doesn't need to know initially anything
about our walking robot it'll still
figure out how to collect rewards
without knowing for example how the
joints move or how strong the actuators
are or the lengths of the appendages but
as engineers we do typically know some
things about the environment so why are
we throwing out all of our knowledge of
physics and not helping the agent at all
this seems crazy right well this is
where model-based RL can help without
any understanding of the environment an
agent needs to explore all areas of the
state space to fill out its value
function which means that it will spend
some time exploring low reward areas
during the learning process however as
designers we often know some parts of
the state space that aren't worth
exploring and so by providing a model of
the environment or part of the
environment we provide the agent with
that knowledge for example take an agent
that's trying to determine the fastest
route to a destination should it go
right or left at this point without a
model the agent would have to explore
the entire map in order to know what the
best action is at this point but with a
model the agent could explore going
right sort of mentally without having to
take the action physically it could then
figure out that going right results in a
dead end and our agent would then choose
to go left in this way a model can
complement the learning process by
avoiding areas that are known to be bad
and exploring and learning the
model based RL is pretty powerful
but the reason model free RL is so
popular right now is because people hope
to use it to solve problems we're
developing a model is difficult such as
controlling a car or a robot from pixel
observations and also since model free
RL is the more general case we're gonna
focus on it for the rest of this series
okay so we know the agent learns by
interacting with the environment so it
makes sense that we have to have a way
for the agent to actually interact with
it and this might be a physical
environment or a simulation for example
for the inverted pendulum we may let the
agent learn how to balance by running it
with the physical pendulum setup this
might be a good solution since it's
probably hard for the hardware to damage
itself or others with the walking robot
however this might not be such a good
idea as you can imagine if the agent
treats the robot and the world like a
black box that it knows nothing about
then it's going to do a lot of falling
and flailing before it even learns how
to move its legs let alone learns how to
walk not only could this damage the
hardware but it would be extremely
time-consuming to have to pick the robot
up each time so not ideal an attractive
alternative is to train your agent
within a high fidelity model of the
environment and just simulate the
learning experience and there's a lot of
benefits to doing this the first comes
from the idea of sample inefficiency
learning is a process and that requires
lots of samples lots of trials errors
and corrections often in the millions or
tens of millions and so with the
simulation you have the ability to run
the learning process at faster than real
time and you can also spin up lots of
simulations and run them all in parallel
the other beneficial thing you can do
with the model of the environment is
simulate conditions that are hard to
test for in the real world for example
with the walking robot you could
simulate walking on a low friction
surface like ice which would help the
robot stay upright on all surfaces and
the nice part about needing a simulation
is that for control problems we usually
already have a good model of our system
and environment since we typically need
it from a traditional control design
this is where if you already have a
model built in MATLAB or
you can replace your existing controller
with an RL agent add a reward function
to the environment and start the
learning process and this model does
just that the RL agent is a block from
the reinforcement learning toolbox and
these three subsystems generate the
observation the reward and sets a flag
that will stop the learning process now
one of the difficulties here is figuring
out how much of the environment to model
what to include and what to leave out
however this is the same question you
have when modeling a plant for a
controller design and so you can use the
same intuition about your system to
build an RL environment model and one
approach is to just start training on a
simple model find the right combination
of hyper parameters that will let
training succeed and then add more
complexity to the model later on hyper
parameters are the knobs that we can
turn on the training algorithms that set
things like the learning rate and the
sample times and I'll cover more of this
in detail in a future video ok with the
environment set the next thing to think
about is what you want your agent to do
and how you're going to reward it for
doing what you want this is similar to
the cost function in lqr where we think
about performance versus effort however
unlike lqr where the cost function is
quadratic in RL there's really no
restrictions on creating a reward
function we can have sparse rewards or
rewards every time step or rewards that
only come at the very end of an episode
after long periods of time they can be
calculated from a nonlinear function or
calculated using thousands of parameters
really it completely depends on what it
takes to effectively train your agent do
you want to get an inverted pendulum to
stand upright then maybe give more
rewards to your agent as the angle from
vertical gets smaller want to take
controller effort into account then
subtract rewards as actuator use
increases
want to encourage a robot to walk across
the floor then give the agent a reward
when it reaches some state that's far
away now with that being said making a
reward function is easy it can be pretty
much any function you can think of
making a good reward function on the
other hand is really really hard and
unfortunately there isn't a
straightforward way to craft your reward
to guarantee that your agent will
converge on the solution you actually
want and I think that boils down to two
main reasons one often the goal you want
to incentivize comes after a long
sequence of actions this is the sparse
reward system therefore your agent will
stumble around for long periods of time
not receiving any rewards in the process
this would be the case for the walking
robot by only giving a reward after the
robot successfully walked 10 meters the
chance that your agent will randomly
stumble on the action sequence that
produces the sparse reward is very
unlikely
imagine the luck needed to generate all
of the correct motor commands to keep a
robot upright and walking rather than
just flopping around on the ground it's
possible but relying on that random
exploration is extremely slow to where
it's impractical this sparse reward
problem can be improved by shaping the
reward providing smaller intermediate
rewards that coax the agent along the
right path but reward shaping comes with
its own set of problems and this is the
second reason crafting a reward function
is difficult if you give an optimization
algorithm a shortcut it'll take it and
shortcuts are hidden within reward
functions and more so when you start
shaping them this causes your agent to
converge on a solution that is optimal
given the reward function but not ideal
an easy example to think about is giving
an intermediate reward if the body of
the robot travels one meter from its
current spot the optimal solution might
not be to walk that one meter but rather
fall ungracefully towards the reward to
the learning algorithm walking and
falling both provide the same reward but
obviously to the designer one result is
preferred over the other now I don't
want to make crafting a reward function
sound easy because getting it right is
possibly one of the more difficult tasks
in reinforcement learning
however hopefully with this general
overview you'll be in a better position
to at least understand some of the
things that you need to watch out for
and that might make crafting the reward
function a little less painful okay so
now that we have the environment which
provides the rewards we're ready to
start work on the agent itself and the
agent is comprised of the policy and the
learning algorithm and these two things
are intimately intertwined many learning
algorithms require a specific policy
structure and choosing an algorithm
depends on the nature of the environment
and we're gonna cover all of this in the
next video but before I end here I want
to introduce the topic and get us
thinking about how the parameters and
the logic within the policy can be
represented remember the policy is a
function that takes in state
observations and outputs actions so
really any function with that input and
output relationship can work in that way
of thinking we could use a simple table
to represent policies and tables are
exactly what you would expect there are
an array of numbers where you use an
input as a lookup address and output the
corresponding value for example a cue
function is a table that map's states
and actions to value so given a state s
the policy would be to look up the value
of every possible action from that state
and then choose the action with the
highest value and training an agent with
a cue function would consist of
developing over time all of the actions
and their values for every state now
this type of representation falls apart
when the number of action value pairs
get really large or becomes infinite
this is the so-called curse of
dimensionality imagine our inverted
pendulum the state of the pendulum can
be any angle from minus PI to PI and the
action that you take can be any motor
torque from the negative limit to the
positive limit and trying to capture all
of that in a table is not feasible
now we could represent the continuous
nature of the state action space with a
continuous function but setting up this
function where we could learn the right
parameters would require us to know the
structure of the function ahead of time
which might be difficult for high degree
of freedom systems or nonlinear systems
so instead it makes sense to try to
represent the
policy with a general purpose function
approximator something that can handle
continuous state action spaces without
us having to set the structure ahead of
time and we get that with deep neural
networks and that's where I'm gonna
leave this for now and we'll continue
this discussion in the next video so if
you don't want to miss future Tech Talk
videos don't forget to subscribe to this
channel also if you want to check out my
channel control system lectures I cover
more control theory topics there as well
thanks for watching and I'll see you
next time