In this video, we'll demonstrate how
to apply model based reinforcement
learning with reinforcement learning toolbox.
We'll use the MBPO, or Model Based Policy Optimization
agent, introduced in 2022 A to balance the cart-pole system.
Unlike model-free methods, model-based reinforcement
learning uses the traditional models
as a part of the training algorithm.
For example, in addition to training the policy,
the MBPO agent trains the neural network transition
model of the environment using data,
or experiences collected from interactions
with the real environment, and it's stored in a memory buffer.
The MBPO agent can also optionally train
neural network models of the environmental reward
and is-done signal if they are not known before training.
The trained models are then used to generate
the additional simulated experiences.
The generated experience is are used
by a base agent, for example, [INAUDIBLE],,
along with the experiences collected
from the real environment to train the policy.
Compared to model-free agent, the MBPO agent
can be more sample efficient, as you
can use the generated data to limit
the interactions between the agent and development.
Now, let's see how we can use the MBPO agent to balance
the cart-pole system.
First, we define a cart-pole environment.
Then we construct an MBPO agent.
This part consists of six steps.
First, we create a base agent.
You can use DDPG, TD3, or SAC for continuous actions, and DQN
for discrete actions.
We use SAC agent as a base agent in this example.
Then we need to construct a neural network transition
model of the cart-pole system.
The transitional model predicts the next observation
given the current observation and the action.
To deal with model uncertainty, then MBPO agent
allows you to specify and train multiple transition models.
These can be either deterministic or stochastic.
In this example, we use three deterministic transitional
models, but you can use more.
Let's look at how to create a traditional model.
First, we create a neural network
with the desired architecture.
The inputs for the deterministic transitional network
or current observation and action.
The output is the next observation.
After the network is created, we define
a deterministic transitional function
using our continuous deterministic transitional
function.
We did the same steps for two more transitional functions.
Note that neural network architecture does not
need to be the same across the transitional functions.
Next, we construct a reward function.
You can use the true reward function if it is available.
If you cannot access the true reward function,
or the reward is too complex to compute,
you can specify a neural network model you would like the agent
to train inside [INAUDIBLE] continuous deterministic reward
function.
In this example, we use a true reward function.
The agent can get more reward if the cart is closer
to the original position.
Next, we construct an is-done function.
This function predicts the terminal signal
given the current observation, action,
and the next observation.
Similar to the reward, you can use a true is-done function
if it is available.
If you cannot access the true is-done function,
you can use [INAUDIBLE] is-done function with your defined
neural network.
In this example, is-done is true when
the pole is more than 12 degrees from the upright position,
or the cart moves more than 2.4 meter
from the original position.
After you have created a transitional model, reward
model, and is-done model, you can
define the overall environment model with [INAUDIBLE]
neural network environment.
Now, we define a MBPO agent options.
In this setting, the MBPO agent trains the environment model
at the beginning of each episode using a 15 minute batches
from the real replay memory.
Using the real sample residual parameter,
you can specify the ratio of real
and the generated experiences in each batch used by the base
agent when training the policy.
We use these settings to generate 2,000 trajectories
at the beginning of each episode using the environmental model.
We also increase the generated trajectory length
every 100 epochs as our environment model
becomes more accurate.
Finally, we construct an MBPO agent
by specifying the base agent, the environment model,
and the MBPO agent options.
After specifying the training options,
we can begin training the agent.
After approximately 500 episodes,
the training converges.
Let's verify the training policy in simulation.
The policy is working fine.
Now, let's check one of the three transitional models
the MBPO agents trained to approximate
the cart-pull system.
You can set which transitional models to use here,
and the call-step method of the environment model
to predict the next observation, reward, and is-done signal.
These four figures show each dimension of observation.
Blue lines show GroundTruth values,
and orange lines show predicted values.
As you can see, the prediction of the trained models
are close to the GroundTruth values.
This concludes MBPO agent demonstration.