the first four videos in this series
covered how great reinforcement learning
is and how you can use it to solve some
really hard control problems so you may
have this idea that you can essentially
set up an environment place an RL agent
in it and then let the computer solve
your problem for you while you go off
and drink a coffee or something
unfortunately even if you set up a
perfect agent and a perfect environment
and then the learning algorithm
converges on a solution there are still
drawbacks to this method that we need to
talk about so in this video I'm gonna
address a few possibly non-obvious
problems with RL and try to provide some
ways to mitigate them even if there
aren't straightforward ways to address
some of the challenges that you'll face
at the very least it'll get you thinking
about them so let's get to it
I'm Brian and welcome to a MATLAB Tech
Talk the problems that we'll address in
this video come down to two main
questions the first is once you have a
learned policy is there a way to
manually adjust it if it's not quite
perfect and the second is how do you
know the solution is going to work in
the first place so let's start with the
first question to answer this let's
assume that we've learned a policy that
can get a robot to walk on two legs and
we're about to deploy this static policy
on to the target hardware think about
what this policy is mathematically it's
made up of a neural network with
possibly hundreds of thousands of
weights and biases and nonlinear
activation functions and the combination
of these values and the structure of the
network creates a complex function that
map's high-level observations to
low-level actions
this function is essentially a black box
to the designer now we may have an
intuitive sense of how this function
operates you know the mathematics that
convert the observations to actions and
we may even understand some of the
hidden features that this network has
identified however we don't know the
reason behind the value of any given
weight or bias so if the policy doesn't
meet a specification or if the plant or
the rest of the operating environment
changes how do you adjust the policy to
address that which weight or bias do you
change the problem is that the very
thing that has made solving the problem
easier namely condensing all of that
difficult logic down into a single black
box function has made our final solution
incomprehensible since the human didn't
craft this function and doesn't know
every bit of it it's difficult to
manually target problem areas and then
fix them now there is active research
that is trying to push the concept of
explainable artificial intelligence this
is the idea that you can set up your
network so that it can be easily
understood and audited by humans however
at the moment the majority of RL
generated policies still would be
categorized as a black box
where the designer can't explain why the
output is the way it is therefore at
least for the time being we need to
learn how to deal with this situation
contrast this with a traditionally
designed control system where there is
typically a hierarchy with loops and
cascaded controllers each designed to
control a very specific dynamic quality
of the system think about how gains
might be derived from physical
properties like appendage lengths or
motor constants and how simple it is to
change those gains if the physical
system changes in addition if the system
doesn't behave the way you expect with a
traditional design you can often
pinpoint the problem to a specific
controller or a loop and focus your
analysis there you have the ability to
isolate a controller and test and modify
it standalone to ensure its performing
under the specified conditions and then
bring that controller back into the
larger system this is really difficult
to do when the solution is a monolithic
collection of neurons and weights and
biases so if we end up with a policy
that isn't quite right rather than being
able to fit
the offending part of the policy we have
to redesign the agent or the environment
model and then train it again in this
cycle of redesigning training and
testing and redesigning training and
testing can be time-consuming but there
is a larger issue looming here that goes
beyond the length of time it takes to
train an agent it comes down to the
needed accuracy of the environment model
the issue is that it's really hard to
develop a sufficiently realistic model
one that takes into account all of the
important dynamics in the system as well
as disturbances and noise at some point
it's not going to perfectly reflect
reality it's why we still have to do
physical tests rather than just verify
everything with the model now if we use
this imperfect model to design a
traditional control system then there's
a chance that our design won't work
perfectly on the real hardware and we'll
have to make a change but since we can
understand the functions that we created
we're able to tune our controllers and
make those necessary adjustments to the
system however with the neural network
policy we don't have that luxury and
since we can't really build a perfect
model any training we do with it will be
not quite correct therefore the only
option is to finish training the agent
on the physical hardware which as we've
discussed in previous videos can be
challenging in its own right a good way
to reduce the scale of this problem is
simply to narrow the scope of the RL
agent like I showed in the last video
rather than learn a policy that takes in
the highest level observations and
commands the lowest level actions we can
wrap traditional controllers around an
RL agent so it solves only very
specialized problems by targeting a
smaller problem with an RL agent we
shrink the unexplainable black box to
just the parts of the system that are
too difficult to solve with traditional
methods shrinking the agent obviously
doesn't solve our problem it just
decreases its complexity the policy is
more focused so it's easier to
understand what it's doing however even
with this we're still left with the
second question how do you know the RL
agent will work regardless of its size
for example is it robust to
uncertainties are their stability
guarantees and can you verify that the
system will meet the specifications
to answer these let's again start with
how we would go about verifying a
traditional control system one of the
most common ways is just through tests
like we talked about this is testing
using a simulation and a model as well
as testing with the physical hardware
and we verify that the system meets the
specifications that is it does the right
thing across the whole state space and
in the presence of disturbances and
hardware failures and we can do this
same level of verification testing with
an RL generated policy again if we find
a problem we have to retrain the policy
to fix it but testing the policy appears
to be similar however there are some
pretty important differences that make
testing a neural network policy
difficult for one with a learned policy
it's hard to predict how the system will
behave in one state based on its
behavior in another for example if we
train an agent to control the speed of
an electric motor by having it learn to
follow a step input from 0 to 100 rpm we
can't be certain without testing that
that same policy will follow a similar
step input from 0 to 150 rpm this is
true even if the motor behaves linearly
this slight change may cause a
completely different set of neurons to
activate and produce an undesirable
result we won't know that unless we test
it now with traditional methods if the
motor behaves linearly then we have some
guarantees that all step inputs within
the linear range will behave similarly
we don't need to run each of those tests
independently so the number of tests we
have to run with an RL policy is
typically larger than with traditional
methods and a second reason verification
through tests is difficult with neural
network policies is because of the
potential for extremely large and hard
to define input spaces remember one of
the benefits of deep neural networks is
that they can handle rich sensors like
images from a camera for example if
you're trying to verify that your system
can sense an obstacle with an image
think about how many different ways an
obstacle can appear if the image has
thousands of pixels in each pixel can
range from 0 to 255 think about how many
permutations of numbers that works out
and how impossible it would be to test
them all and just like with the step
input example just because your network
has learned to recognize an obstacle in
one portion of the image at some
lighting condition and orientation and
at some scale doesn't give you any
guarantees that it works in any other
way in the image the last difficulty I
want to bring up is with formal
verification these methods involve
guaranteeing that some condition will be
met by providing a formal proof rather
than using tests in fact with formal
verification we can make a claim that a
specification will be met even if we
can't make that claim through testing
for example with the motor speed
controller we tested a step input from 0
to 100 rpm and from 0 to 150 but what
about all of the others like 50 to 300
or 75 to 80 even if we sampled 10,000
speed combinations it won't guarantee
every combination will work because we
can't test all of them it just reduces
overall risk but if we had a way to
inspect the code or perform some
mathematical verification that covered
the whole range then it wouldn't matter
that we couldn't test every combination
we'd still have confidence that they
would work for example we don't have to
test to make sure a signal will always
be positive if the absolute value of
that signal is performed in the software
we can verify it simply by inspecting
the code and showing that the condition
will always be met other types of formal
verification include calculating
robustness and stability factors like
gain and phase margins but for neural
networks this type of formal
verification is more difficult or even
impossible in some cases for the reasons
I explained earlier it's hard to inspect
the code and make any guarantees about
how it will behave we also don't have
methods to determine its robustness or
its stability and we can't reason
through what will happen if a sensor or
actuator fails it all comes back to the
fact that we can't explain what the
function is doing internally ok so these
are some of the ways that using learned
neural networks make design difficult
it's hard to verify their performance
over a range of specifications and then
if the system does fail it's hard to
pinpoint the source of the failure and
then even if you can pinpoint the source
it's hard to manually adjust the
parameters or the structure leaving you
with the only option of redesigning and
starting the training process over again
but I don't want to destroy your
confidence in reinforcement learning as
a useful tool for production systems
because with all of that being said
our ways to set up learning so that the
resulting policy is more robust in the
presence of uncertainties there are also
ways to increase safety and make
reinforcement learning a viable option
for production systems and we can even
use reinforcement learning to solve a
slightly different problem one that
avoids a lot of these issues in the
first place so let's in this video on a
positive note and talk about all three
of these
we'll start with making the policy more
robust even though we can't quantify
robustness we can make the system more
robust by actively adjusting parameters
in the environment why the agent is
learning for example with our walking
robots let's say manufacturing
tolerances cause the maximum joint for a
motor torque to fall between 2 and 2.1
Newton meters we're going to be building
dozens of these robots and we'd like to
learn a single policy for all robots
that is robust to these variations we
can do this by training the agent in an
environment where the motor torque value
is adjusted each time the simulation is
run we could uniformly choose a
different max torque value in that range
at the start of each episode so that
over time the policy will converge to
something that is robust to these
manufacturing tolerances by tweaking all
of the important parameters in this way
you know things like appendage lengths
and delays and the systems and obstacles
and reference signals and so on we're
gonna end up with an overall robust
design we may not be able to claim a
specific gain or phase margin but we'll
have more confidence that the result can
handle a wider range within the
operational state space this addresses
the robustness but it still doesn't give
us any guarantees that the policy will
do the right thing on the hardware and
one thing we don't want is for the
hardware to get damaged or someone get
hurt because of an unpredictable policy
so we need to also increase the overall
safety of the system and one way we can
increase safety is by determining
situations that you want the system to
avoid no matter what and then building
software outside of the policy that
monitors for that situation if that
monitor triggers then constrain the
system or take over and place it into
some kind of safe mode before it has a
chance to cause damage this doesn't
prevent you from the
flowing and dangerous policy but it will
protect the system allowing you to learn
how it fails and adjusting the reward
and training environment to address that
failure both the fix to increase
robustness and safety our kind of
workarounds to the limitations that we
have with a learned neural network
policy however there is a way to use
reinforcement learning and still be able
to take advantage of the result being as
robust safe changeable and verifiable as
a traditionally architected control
system and that is by using it simply as
an optimization tool for a traditionally
architected control system let me
explain it this way imagine designing an
architecture with dozens of nested loops
and controllers each with several gains
you can end up with a situation where
you have a hundred or more individual
gain values that you need to tune and
rather than try to manually tune each of
these gains by hand you could set up an
RL agent to learn the best values for
all of them at once the agent would be
rewarded for how well the system
performs and how much effort it takes to
get that performance and then the
actions would be the hundred or so gains
in the system so when you initially kick
off training the randomly initialized
neural network in the agent would just
generate some random values and then
you'd run the simulation using those for
the control gains now more than likely
that first episode would produce some
garbage result but after each episode
the learning algorithm would tweak the
neural network in a way that the gains
move in the direction that increased
reward that is it's improving
performance and lowering effort in the
nice part about using reinforcement
learning in this way is that once you've
learned an optimal set of control gains
you're done you don't need anything else
we don't have to deploy any neural
networks or verify them or worry about
having to change them
we just need to code the final static
gain values into our system so in this
way you still have a traditionally
architected system one that can be
verified and manually adjusted on the
hardware just like we're used to but you
populated it with gain values that were
optimally selected using reinforcement
learning sort of a best of both worlds
approach
so hopefully you can see that
reinforcement learning is really
powerful for solving hard problems and
then it's definitely worth learning and
figuring out how to combine it with
traditional approaches in a way that you
feel comfortable with the final product
there are some challenges that exist
regarding understanding the solution and
verifying that it'll work but as we
covered a bit we do have a few ways
right now to work around those
challenges but here's the thought I want
to leave you with learning algorithms RL
designed tools like MATLAB and
verification methods are advancing all
the time we are nowhere near
reinforcement learnings full potential
perhaps it won't be far into the future
before it becomes the first design
method of choice for all complex control
systems all right that's it so thanks
for watching this video
and if you don't want to miss any future
Tech Talk videos don't forget to
subscribe to this channel and if you
want to check out my channel control
system lectures I cover more control
theory topics there as well I'll see you
next time