what is the difference between
model-free and model-based reinforcement
learning
well in this video i'm going to answer
that question and as part of the answer
i'm going to show you the results of
training an agent to balance a cartpole
system with both model-free and
model-based reinforcement learning and
hopefully by the end you're going to
have a better understanding of the
situations where you may want to choose
one over the other so i hope you stick
around for it i'm brian and welcome to a
matlab tech talk
with reinforcement learning an agent
learns how to interact with an
environment in a way that maximizes a
reward and it does this by adjusting its
policy over the course of many different
attempts the agent takes an action and
then if that's deemed to be good based
on the reward that's received then that
action is reinforced and there is a
higher likelihood that it will perform
the same action again the next time it's
in that state
what i've drawn here is model free
reinforcement learning and with model
free rl an agent learns the optimal
policy solely through direct
interactions with an environment so all
of the experience that the agent uses to
learn is generated by the real
environment
now on the other hand with model based
rl the agent still uses some real
experiences but the difference is that
within the agent there's also a model of
the environment which it can use to
create additional simulated experiences
it's sort of like having a mental model
of the world that allows the agent to
learn by thinking rather than just
acting
now i know that's kind of a confusing
statement so to understand it better
let's use sports as an analogy
there are two teams a and b and they've
never played each other before and
therefore team a doesn't know how team b
plays or what to expect for their
strategy
and at the start of the match team a
will try some tactics and decide whether
those actions worked or not and then
real time adjust their strategy to
attempt to win the game
this is a model free approach to
learning because team a had to interact
with the environment or team b in this
case in order to learn how team b plays
and improve their chances of winning
however during this process team a could
have developed a model in their head of
how team b plays
this model might not be perfectly
accurate but now before the two teams
play again team a can think about
different strategies and assess whether
they will be successful or not based on
their understanding of how team b plays
this is a model based approach to
learning because team a is able to
update their policy or learn a better
playing strategy without directly
interacting with team b but just by
thinking about it
they are supplementing their real
experiences with simulated experiences
and because of this they're able to
learn and improve between episodes or
between real physical matches
in this same way a software-based agent
can have a mental model of the
environment and use that to update its
policy between physical interactions
with the real world environment
and this strategy can investigate more
of the state space than just real
experiences alone
conceptually it works like this
imagine this 2d rectangle represents the
entire state space of the environment
and we want the agent to explore the
entire thing and learn the optimal
policy
if the starting state is here and the
agent takes real actions then over time
the agent may transition to this
environment state these are the real
world experiences and with model-free
reinforcement learning the agent can
only learn from these experiences which
as we can see doesn't cover much of the
total state space area
however if the agent has a model of the
environment then at each time step it
can simulate what would happen if it had
taken a different action
so from a single real trajectory the
agent could generate thousands of
simulations each time step and even if
those simulations are only looking ahead
one step in the future
the combination of real and simulated
experiences covers a much wider swath of
the state space and so in this way it
takes fewer real interactions with the
environment overall in order to cover
the entire area
and this is why model based rl is more
sample efficient than model free rl
however it does come at a cost
and that is computations
running a simulation requires more
compute power than just interacting with
the real environment so what we gain
with fewer samples we lose in more
compute operations
all right now before we move on i want
to clear something up when i say real
environment this simply means the
environment that is external to the
agent i don't mean that the environment
has to be a real physical entity i mean
very often in reinforcement learning we
still train agents using a model of the
environment it's just that that model is
external to the agent and therefore the
only way for the agent to learn is by
taking actions and receiving a response
from the agent's perspective that model
is the environment and it learns solely
through direct interactions
it's only when the model exists within
the agent and the agent can query that
model internally without ever taking an
action that it becomes model-based
reinforcement learning
all right so now that we have a handle
on the differences between model-free
and model-based rl i want to talk about
why you may choose one versus the other
in comparison model free rl is a simple
and straightforward approach you let the
agent take an action and then update the
policy based on the observations and
reward you don't have to mess around
with having a model or setting up any
kind of simulations and so one benefit
of model free is the simplicity of it
also models are less accurate than the
real world and therefore an agent that
is updating its policy using possibly an
imperfect mental model might learn some
behaviors that don't work in the real
environment
and so the agent would have to use the
real environment to correct that policy
and if the model continues to stay
inaccurate the agent would continually
learn the wrong thing and then have to
correct it and then learn the wrong
thing again and so on in a really most
inefficient manner
so model free rl doesn't have this
problem since the agent is always
learning directly from the external
environment
now on the other hand model based rl
requires fewer interactions with the
external environment
and this is because like we said earlier
the agent can do a lot of learning using
the model in between real world
interactions and this is particularly
important if the agent is controlling a
physical system like a robot or a
vehicle because real actions have the
possibility of damaging hardware either
by the agent requesting an action that
causes that damage or just through the
wearing down of parts over many
consecutive movements
there is also the benefit of being able
to gain experience faster than real time
by running models and it's also easier
to place the agent in states that would
be difficult with the real environment
so those are some of the reasons why you
may want to choose a model-based
approach
now if the environment is based or it's
a simulation already like a simulink
model of the real environment
then sample efficiency might not matter
as much as total compute resources
it costs money and time to perform more
calculations and you might as well just
interact directly with the environment
with model-free rl
but in the case of physical systems it
can be beneficial to reduce the number
of real-world training episodes with
model-based rl even at the expense of
more computations
all right so let's say that model based
rl is what you want to use to train an
agent on a physical system but what if
you don't have a model of the
environment ahead of time how do you get
one so that you can actually do model
based rl
well one way is that we could just
create a model and then give it to the
agent and modeling is done all the time
right so this doesn't seem like a bad
idea
however often creating a model requires
data from the physical system
so we need the agent to physically
interact with the environment anyway to
generate the data that we want to fit a
model to or we want to use that data to
learn a model
so if we need to interact with the
environment to develop a model
and we want to train an agent using
reinforcement learning why not combine
the two and set up a reinforcement
learning algorithm that simultaneously
learns the environment model and learns
the optimal policy
in this approach when the agent gets
real experiences from the environment it
uses that data to update the model and
also to update its policy and so now
before the next episode with the real
environment the agent can use the model
to generate additional simulated
experiences
but a question might be do we have to
fully learn the model before the agent
can start using it
i mean we know it doesn't do us any good
to make predictions with the model if
the predictions are garbage and early in
the learning process most of the model
is garbage right
therefore if we train the agent with
that model we'll end up with reinforcing
behaviors that will perform badly in the
real world
and that is true
but when the bad actions are attempted
in the real world and a lower reward is
received then that experience will
correct the agent's policy
and it's going to correct the model for
that given state in action
so now the agent can use that corrected
model to explore the nearby states and
actions more thoroughly
so it's self-correcting over time but
there are ways of reducing the chance
that the agent learns from areas of the
model that aren't accurate in the first
place and one is by giving the agent
multiple models or an ensemble of models
here the agent is maintaining three
models of the environment and each one
is a neural network which is initialized
with random weights and biases and
therefore if they're given the same
state action input then it's likely that
each will make different random
predictions
and then after a training episode these
models will have a higher chance of
making the same prediction for that
trajectory that they were trained on
while all of the other state action
inputs would still produce random
outputs
in this way over time if the predictions
from all of the models agree then it's
likely that the agent is simulating a
state that it's already learned
and if they don't agree then it's safe
to assume that the predictions are from
random parts of the model
all right so one way to handle this is
to just check the predictions and then
only use the model experiences when they
agree and if they don't agree just
ignore them
however a simpler approach is to just
not worry about it and let the agent
learn from all of the simulated
experience whether it's correct or not
if the models agree then the agent is
going to reinforce that behavior
multiple times once for each model and
if they don't agree then the agent will
reinforce several random behaviors but
each probably in a different direction
so they kind of offset each other and so
in this way statistically there is a
strong reinforcement if the models agree
and a weaker reinforcement if they don't
all right so let's wrap all of this up
with model-based reinforcement learning
an agent has a model of the environment
that it can use to generate simulated
experiences
additionally some algorithms allow the
agent to learn that model as it
interacts with the real environment but
whether it learns that model or not just
having a model as part of the agent is
what makes it model based rl
and having multiple models provides a
way for the agent to learn more quickly
from simulated experiences that are more
accurate and more slowly from simulated
experiences that are less accurate
and all of this typically extends the
amount of time and calculations it takes
to train an agent
but it reduces the number of real-world
samples that are needed
now i told you that i would show you the
results of both a model free and model
based reinforcement learning approach to
training an agent
and to set up this problem i'm using the
matlab example called train mbpo agent
to balance cartpole system
this example uses a model-based policy
optimization agent or mbpo to learn how
to move a cart such that it balances a
freely spinning pole
and this agent is learning three neural
network models of the environment which
it also uses to generate simulated
experiences that augment the real
experiences that it gets from the
physical environment
i've left a link to this example below
and i recommend that you take some time
and play around with it so you can get a
better feel for what this example is
actually doing and how model based rl
works
but what i want to show you is the
result from running this example five
times with different random seeds and
then here i'm comparing the number of
episodes needed to exceed the training
criteria for a model-based approach with
the number of episodes needed for a
model free approach
and notice that in four of the five
cases the model-based agent reached the
training criteria in fewer episodes it
took just about half of the number of
episodes
now there is this one that took about
the same number but that's just kind of
the random luck of the draw on average
the model based agent is more sample
efficient
now i don't have the number of
calculations or the exact wall clock
time that it took for each of these
examples but on average the model-based
approach took about twice as long to run
so that's the main trade-off that we
have to consider between these two
methods calculation time versus number
of real-world interactions
all alright well that's where i'm going
to leave this video for now if you don't
want to miss any other future tech talk
videos don't forget to subscribe to this
channel and if you want to check out my
channel control system lectures i cover
more control theory topics there as well
thanks for watching and i'll see you
next time