In this video, we’ll be looking at using Unity’s MLAgents with a Kart Racing Game.
MLAgents, or the Unity Machine Learning Agents Toolkit, is an open-source Unity plugin that
enables games and simulations to serve as environments for training intelligent agents.
Agents can be trained using reinforcement learning, imitation learning, neuroevolution,
or other machine learning methods through a simple-to-use Python API.
In this video, we’ll be exploring a simple integration of the MLAgents Toolkit to the
Kart Racer microgame.
The sample project includes an example scene, the MLAgents package, the KartAgent implementation,
a pre-trained brain, and a basic racetrack. These assets are all available for free download
at a link in the description below.
We have provided an example KartLearningBrain trained using the default Reinforcement Learning
model, that has been trained for a few hours. When we enter Play Mode, we can see the AI
attempt to navigate the track.
Let’s take a look at how the KartLearningBrain is controlling the Kart.
We have added a custom KartAgent component that acts as a controller, and linked it to
the Input field of the KartMovement component. This replaces the default KeyboardInput component
and allows our KartAgent to control the Kart directly.
The KartAgent component inherits from the base Agent class.
These top parameters of the KartAgent component are inherited from the base Agent class. They
relate to the input of the Agent, and how it is trained. If you’d like to learn more,
visit the MLAgents GitHub repository for more information on these parameters.
The bottom parameters are custom fields we have made for this specific project.
Our KartAgent uses 9 different raycasts to “see” the world. Each raycast is defined
by a Transform in our Raycasts Array. The KartAgent uses these raycasts as input into
the brain. At every step, the KartAgent will see how far each Raycast travels before hitting
a wall, and then feed that number into the brain. The KartAgent’s brain will then process
the raycasts through its neural net and output the acceleration and steering values for the
Kart.
The brain used by the KartAgent is defined by the Behavior Parameters. The Behavior Parameters
define which brain model to use, and how it functions. Our model is set to the KartLearningBrain.
You can have multiple different brains for a MLAgent. In your game, you may want to have
different brains for different difficulty levels.
There are many ways to train a neural net brain. We have provided a sample KartAgent
brain, that has been trained in a basic Reinforcement Learning environment.
Reinforcement Learning works on a simple principle. The brain will learn by continuously changing
its behavior, trying different things to maximize its rewards, and to minimize its punishments.
We have set the Max Step to 2000, which will end the training episode after a 2000 training
steps.
We have set the Kart to be rewarded in two different ways:
Reward the agent a very small amount while it is moving forward.
Reward the agent a large amount when it reaches a checkpoint.
The KartAgent is only punished in one way: Reset the agent if it touches a wall.
To reward the agent while it’s moving forward, we used the AddReward() function inside of
KartAgent, passing in Kart.LocalSpeed, and multiplying it by a small number.
To reward the agent with checkpoints, we have placed checkpoints all around the track.
We have created a ReachedCorrectCheckpoint event on the TrackManager and subscribed the
KartAgent.OnReachCheckpoint function.
We have set up multiple KartAgents that use these rules to run in parallel. Having multiple
agents can speed up training time. Each KartAgent contributes its experiences to the same neural
net, while trying to reach as many checkpoints as it can.
If we try out the pre-trained brain, we will see the karts move along the track, detecting
walls and turning using its raycasts.
We have trained the brain for several hours, and it can navigate the track, but it can
still use some improvement. We encourage you to try training the KartAgent
yourself, and to experiment with the reward structure.
We can train our KartAgent through the KartAcademy. The KartAcademy script inherits from the base
MLAgent Academy class. It determines what parameters to run Unity in while in training
mode and while in normal playback inference mode.
We have set our Time Scale to a higher number, allowing us to run the game faster than normal
and therefore speeding up training.
Our KartAcademy will connect to the Python MLAgent API, feeding it the raycast sensors
we created.
You will need the MLAgent Python environment installed to conduct training.
With the MLAgent environment installed, you can monitor the status of the Agent using
Tensorboard during or after training.
For information on how to set up the MLAgent Python environment for your platform, visit
the MLAgents GitHub repository, which is linked below.. https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md
We hope that this project can help you get started with Unity’s MLAgents Toolkit, to
help you create machine learning AI for your own project. We encourage you to download
the free sample project, and try training the Kart yourself. Thanks for watching!