artificial intelligence machine learning and deep neural networks these are terms
that can spark your imagination of a future where robots are thinking and
evolving creatures in this video we're gonna look at reinforcement learning or
RL as I'll sometimes abbreviate it it's a type of machine learning that has the
potential to solve some really hard control problems you may have heard that
the AI company deepmind created a program called alphago it's an AI that
uses reinforcement learning to beat the world's best go players and then they
recently created alpha star which is poised to dominate the Starcraft 2 scene
so naturally you might be thinking well if it can do that why can't I use
reinforcement learning to control my robot or cool my data center or
stabilize a drone in a highly dynamic and turbulent flow well let's talk about
that now a quick warning before we begin this is not intended to be an exhaustive
look at reinforcement learning I wouldn't be qualified to explain it all
to you anyway instead I want to introduce this topic from the point of
view of a traditionally trained controls engineer and I hope to show you that
there's actually a lot of overlap with control theory by the end of this series
I think you'll be better prepared to answer questions like what is
reinforcement learning and why should I consider it when solving my control
problem how do I set up and solve the reinforcement learning problem and what
are some of the benefits and drawbacks of reinforcement learning compared to a
traditional controls approach alright so with the scope of this series Set let's
get to it I'm Brian and welcome to a MATLAB Tech
Talk let's start by thinking about the complexity of building a walking robot
from the perspective of a traditional controls approach we might use cameras
to view the environment and then extract image features that can be converted
into signals like position or locations of obstacles we could combine those
observations with other sensors that complete the state estimation which we
then use along with a model of the plant and environment to design the control
system and more than likely this would consist of multiple control loops that
all interact with each other for example there would be low-level motor
controllers and high-level controllers that are managing the leg trajectories
or the robot trunk trajectory and maybe a
higher level controller that is managing the balance or off nominal behavior and
everything has to work together in an uncertain environment to generate this
complex movement of walking which can be really challenging instead of all this
complexity let's squeeze it down into a single black box that simply takes in
observations and outputs the low-level motor commands directly if we were
infinitely smart we could sit down and design a function that could get a robot
to walk without concerning ourselves with all of the internal steps along the
way but since we're not that's where machine learning comes in
broadly speaking machine learning can be subdivided into three categories
unsupervised learning supervised learning and reinforcement learning
unsupervised learning is used to find patterns or hidden structures and
datasets that have not been categorized or labeled for example imagine you
collected information on a hundred thousand animals like a bunch of
physical attributes and social tendencies then you can use unsupervised
learning to group the animals or cluster them into similar features this can be
something obvious like grouping them into mammals and birds or to group them
by patterns that might not be as obvious like finding correlations between
physical traits and social behaviors that you didn't know about ahead of time
supervised learning on the other hand is different in a subtle way in that we
train the computer to apply a label to a given input for example let's say that
one of the columns of our data set of animal features is the species we can
then treat species as the label and the rest of the data as inputs into a
mathematical model then we can use supervised learning to train our model
to correctly label each set of animal features by inputting them in one at a
time letting the model guess the species and
then systematically tweaking the model based on whether that guess was correct
or not and if we had enough training data to get a reliable model we could
then send through the input features for a new animal one that we don't have
labeled and our trained model would apply the most probable species label to
it and supervised learning is probably the type of machine learning that most
people are familiar with because it's what allows computers to recognize
pictures of cats or your friends and photos
and fundamentally applying a label to an image is exactly the same problem as
applying a label to a data set of animal features we input a bunch of training
images into the model and then we tweak the model based on whether it guessed
correctly or not until it's accurate the difference is that the input data for an
image is just a stream of numbers representing pixel intensities so it's
not as straightforward to understand how that relates to a cat this is what deep
learning is good at by representing the model as a deep neural network we have
an efficient way to input thousands of numbers and then tweak it during
training so that it can identify features within an array of pixel
intensities that will ultimately allow it to apply the correct label
reinforcement learning is a different beast altogether unlike the other two
learning frameworks which work with a static data set
RL works with a dynamic environment and the goal is not to cluster data or label
data but to find the best sequence of actions that will generate the optimal
outcome optimal in this sense means to collect the most reward it does this by
allowing a piece of software called an agent to explore interact with and learn
from the environment the agent can take an action which affects the environment
changing its state and the environment then produces a reward for that action
and using this information the agent can adjust which action to take in the
future it can learn from this process and although you're presumably not a
piece of software you learn in essentially the same way a software
agent learns with the reinforcement learning framework you can be thought of
as an agent and the world around you as the environment that you can interact
with observe its state and collect rewards you get rewarded by the
environment by taking actions that are good like you went to college action and
you got a job State and that job pays well reward or
you looked both ways before crossing the street
action you got to the other side State and you didn't get run over in the
process reward alternatively you get low or negative rewards for taking actions
that are bad like you stayed up late before an exam action you're tired State
and you received a poor grade reward within the agent there is a brain that
takes in state observations the inputs and maps them to actions the outputs and
in RL nomenclature this mapping is called the policy given a set of
observations the policy decides which action to take and just like with
supervised learning we can represent the policy as a deep neural network which
we'll see later allows our agent to input thousands of states at once and
still be able to come up with a meaningful action this is where the term
deep reinforcement learning comes from in a walking robot example the
observations might be the state of every joint and the thousands of pixels from a
camera sensor the policy would take in all of these observations and output the
actuator commands and if the robot stays upright and continues walking the
environment would generate a reward telling the agent exactly how well that
very specific combination of actuator commands did of course the policy might
not be mapped correctly to take the best actions or the environment might be
slowly changing and so the mapping is no longer optimal and this is where
reinforcement learning algorithms come in they changed the policy based on the
actions that were taken the observations from the environment and the amount of
reward collected in this way the goal of the overall agent is to use
reinforcement learning algorithms to modify its policy as it interacts with
the environment so that eventually given any state it will always take the most
advantageous action the one that will produce the most reward in the long run
for example if you were tired for your exam and you received a bad grade well
you learn from it and you adjust your policy so that you won't stay up late
before the next exam now at its heart reinforcement learning is an
optimization problem but there are some very interesting concepts that set
reinforcement learning apart from other optimization techniques first is the
idea of value reward is the instantaneous benefit of being in a
specific state whereas value is the total reward that an agent can expect to
collect from that state and onwards into the future assessing the value of a
state rather than assessing the reward helps the agent choose the action that
will collect the most reward over time rather than a short-term benefit
for example imagine our agent is in this situation and is trying to collect the
most reward within three steps if the agent looks directly at the reward for
each action then it will step left first to get a higher reward and then right
and then left again to ultimately collect +1 however if the agent is able
to estimate the value of a state then it will see that going right has a higher
value than going left and will ultimately end up with +8 reward of
course often the promise of a high reward in the future still might not
mean that the action is the best and there's at least two good reasons for
this one like with the financial market money in your pocket now can be better
than a little more money in your pocket a year from now and to your prediction
of rewards further into the future become less reliable and therefore that
high reward might not be there by the time you reach it in both of these cases
it's more advantageous to be a little more short-sighted when estimating value
and in RL we can control this by discounting rewards by a larger amount
the further they are in the future another critical aspect of reinforcement
learning is the trade-off between exploration and exploitation when
interacting with the environment this is the trade-off between collecting the
most rewards that you already know about versus exploring areas of the
environment that you haven't visited yet for example let's say the agent only
knows about the two rewards immediately adjacent to it if it took the greedy
approach by exploiting the environment it would only go after the highest
reward it knows about and so it would go left to collect the +1 however if we
occasionally let the agent explore the state space even at the risk of
collecting fewer rewards it can fill out more of its value function and it opens
up the possibility of finding higher rewards that it didn't know about and
this is part of our normal learning process as humans as well a simple
example is deciding on which restaurant you want to eat at do you choose a
restaurant you know you like and therefore exploit your knowledge or do
you venture out and explore a restaurant you've never been to before increasing
your knowledge now trying a new restaurant gives you the opportunity to
find a new favorite place but it also increases your chances of getting a meal
that you don't like it's tricky to settle on the perfect
balance between exploring and exploiting however at the very least our L
algorithms provide a simple way to set that balance okay it's starting to feel
like our L has a completely different goal than what control engineers are
trying to do when we design control systems but it is pretty much exactly
the same problem we're trying to figure out how to design the controller or the
policy that map's the observed state of the plant or the environment to the best
actuator commands the actions and when we design a controller we're basically
doing a one-time policy update and one of the ways that we can design an
optimal controller is by minimizing a cost function like we do with lqr and
cost is just the negative of reward so by maximizing the reward we're
solving the same problem as minimizing cost the difference is that with
reinforcement learning the computer tries to learn the optimal behavior over
time rather than have the designer solved for it explicitly it's like the
adjustment mechanism in an adaptive controller where it's tweaking the
parameters at each sample time in this way we can essentially design a
controller which is a policy without knowing anything about the system itself
and without having to solve any of the traditional control problems we just let
the computer learn the right parameters on its own through a process that you
can think of as fancy trial and error now even with the learning algorithm
doing most of the work for us we can't enter this process completely ignorant
we have to know several things before we start and the first is that we need to
understand our system that we're trying to control and determine whether it's
better to solve the problem with traditional control techniques or with
reinforcement learning and if we choose the learning path then we need to set up
the policy so that it has enough parameters and in the right structure so
that it can be tweaked successfully it won't do us any good if we're hoping to
control a multi-dimensional system but only give it a single parameter we also
need to know what a successful result would be and reward the controller for
doing well this requires crafting a reward function so that the learning
algorithm understands when it's getting better and ultimately settles on the
result that you're actually looking for and third we need to apply an efficient
algorithm that looks at the reward and the system state and knows how to
tweak the parameters so that the process converges with any reasonable amount of
time this is where we would also set the parameters for exploration and
exploitation and the discounting on future rewards in the next few videos
we'll expand on all of this by looking at the workflow of reinforcement
learning in more detail we'll look at the structure of the policy and
introduce neural networks and we'll talk about how crafting the proper reward
function impacts your final result and we'll look at a very high overview of
some interesting learning algorithms so if you don't want to miss the next Tech
Talk video don't forget to subscribe to this channel also if you want to check
out my channel control system lectures I cover more control theory topics there
as well thanks for watching and I'll see you next time