This video will demonstrate how to log data while training
reinforcement learning agents.
You can train agents using the built-in train function
in the Reinforcement Learning Toolbox.
The train function outputs a data structure
with training information.
A lot of additional data is also computed during the training
process, and it may be useful to log
this data for further analysis.
Examples of such data could be training losses,
agent experiences, exploration parameters, and so on.
The goal of this video is to cover the process
of logging such data to file.
The general workflow is to first create
a data logger object using the rlDataLogger function,
then configuring that object with one
or more callback functions that log data
at different stages of the training process.
Finally, integrating the logger object with the train function
by passing it as a name value argument.
You can create the logger object using the rlDataLogger
function.
The default syntax of this function
creates an object that logs data to MAT files.
The logger object has configurable options
that can be found under the LoggingOptions property.
Here is a list of the LoggingOptions
and a brief description for each of them.
You can specify these options to configure the LoggingDirectory,
add a prefix for the filename, specify version and compression
configuration for the MAT files, or control
the number of episodes you want to cache data
before writing the file.
The next step is to choose the data
to log by specifying callback functions to the logger object.
These functions are executed at different stages
of the training process.
For example, the episode finished
function is executed after the end of a training episode.
Here is an example to log the agent's experiences
during the training process.
You can configure the episode finished function properly
with the function handle.
The function outputs the dataToLog
after an episode-- in this case, experiences.
The function syntax is shown in the example code below.
It accepts a variable called data
as input, which is a structure containing information
from the current stage of training.
The fields of data include the current episode count,
the agent and environment objects,
which can be used to extract underlying
data such as learnable parameters, information
about the episodes such as a reward,
and also a structure array of experiences.
You can pass this Experience area
as the output of the function within a structure.
That's all you would need to do to configure
logging of experiences.
Let's go a step further and log the actor and critic training
losses.
You can do this by specifying a function
to the AgentLearnFinishedFcn property.
The function will be executed after the learn subroutine has
finished.
It can be multiple times within an episode
or after the end of the episode, depending
on the agent algorithm or the training scenario.
The function syntax is similar as before.
However, you have access to data such as actor and critic
trading losses within the input variable data.
The next step is to integrate with the train function.
In this section, data logging is configured
for training a DDPG agent with a continuous CartPole
environment.
The code shown here creates the agent, environment,
and training options, which is the usual workflow for setting
up a training.
The logger is now passed through the train function
as a name value argument.
Let's run this action to start the training.
While the training is in process,
data is saved in the MAT files in the specified logging
directory.
The data is saved in the context of an episode,
with each MAT file containing data
for the particular episode.
After the training is finished, you can analyze the logged data
by loading it into memory.
You can either write a script to load the MAT files iteratively
in a for loop or load it using a Datastore.
With the fileDatastore function, you
can read the mad files from the logging directory all at once.
The code loads the data in the MATLAB workspace
under the variable called allData.
You can now inspect the loaded data in MATLAB.
You can index into the allData variable,
and extract data for a particular training episode
and analyze it.
For example, load the episodeData field,
which contains the experiences previously logged
for the callback function.
Similarly, you can also load the actor and critic training
losses.
You can also specify a function to format the data
by loading into memory.
The loadActorLoss function loads the actor loss data
and converts it into a numeric area for convenience.
To summarize, you can log data at different stages
of the training process.
The data can then be saved to disk
and loaded into memory for further tracking and analysis.
That concludes this demo.