in this video we show how to use
reinforcement learning for field
oriented control of a permanent magnet
synchronous motor to showcase this we
start with an example that uses the
typical field oriented control
architecture where the outer loop
controller is responsible for speed
control whereas the inner loop bi
controllers are responsible for
controlling the D axis and Q axis
currents we then create and validate a
reinforcement learning agent that
replaces the inner loop controllers of
this architecture the use of RL agent is
especially beneficial when the system is
nonlinear in which case we can train a
single RL agent instead of during PA
controllers at multiple operating
conditions in this example we use a
linear motor model to showcase the work
flow of field oriented control using
reinforcement learning and this workflow
remains the same for a complex nonlinear
motor as well let's look at the Similan
model that implements the field oriented
control architecture this model contains
two control loops an outer speed loop
and an inner control loop the outer loop
is implemented in the speed control
subsystem
and it contains a PA controller that is
responsible for generating reference
currents for the inner loop the inner
loop is implemented in the current
control subsystem and contains two PI
controllers to determine reference
voltages in the DQ frame the reference
voltage is then used to generate
appropriate PWM signals that control the
semiconductor switches of the inverter
which then drives the permanent magnet
synchronous motor to achieve desired
torque and flux
let's go ahead and run the Similan model
we can see that the tracking performance
of the controllers are good and are able
to track the desired speed let's save
this result for later comparison with a
reinforcement learning controller now we
update the existing model by replacing
two pi controllers in the current loop
with a reinforcement learning agent
block
in this example we use D D P G as the
reinforcement learning algorithm which
strains an actor and a critic
simultaneously to learn an optimal
policy that maximizes long-term reward
once the symlink model is updated with a
reinforcement learning block we then
follow the reinforcement learning
workflow to set up train and simulate
the controller reinforcement learning
workflow is as follows first step is to
create an environment in this example we
already have a symlink model that
contains the permanent magnet
synchronous motor modeled using motor
control blocks it and sim scape
electrical within the plant and inverter
subsystem we then use this similar model
to create reinforcement learning
environment interface with appropriate
observations and actions here the
observations to the reinforcement
learning block are error in the stator
currents ID error and IQ error and the
stator currents ID and IQ actions are
the state of voltages V D and V Q next
we create the robot signal to let the
reinforcement learning agent know how
good or bad the action it selects during
the training are based on its
interaction with the environment here we
shape a reward based on the quadratic
reward penalty that penalizes distance
from goal and control effort then we
move on to creating network architecture
here we construct the actor and the
critic networks as required by the D D
PG algorithm programmatically using
MATLAB functions for layers and
representations the neural networks can
also be constructed using the deep
network designer app and then imported
into MATLAB the critic network in this
example takes an observations and
actions as the input and gives estimated
Q values as the output the actor network
on the other end takes and observations
as the input and gives actions as the
output
with actor and critic representations
create we can create a DD PG agent the
sample time of the DD PG agent is
configured depending on the execution
requirement of the control loop in
general agents with smaller sample time
take longer time to train as it involves
a greater number of simulation steps
each episode we are now ready to train
the agent first we specify the training
options here we specify that we want to
run training for at most two thousand
episodes and stop training if the
average reward exceeds the provided
value we then use the Train command to
start the training process in general it
is best practice to randomize reference
signals to the controller during
training process to obtain a more robust
policy this can be done by writing the
local reset function for the environment
during the training process progress can
be monitored in the episode manager once
the training is complete we can simulate
and verify the control policy from the
trained agent by simulating the model
with the trained agent we see that the
speed tracking performance of field
oriented control is good with
reinforcement learning
agent controlling the stator currents on
viewing this performance with a
previously saved output we see that
performance of field oriented controller
with reinforcement learning agent is
comparable to its bi controller
counterpart this concludes the video