It is almost impossible to go a day in technology circles these days without
discussing machine learning: the ability to train systems to learn about complex
problems and develop nuanced and intelligent responses for them.
Well, I say almost impossible, but it's pretty easy to do if you work in the
games industry. As is perhaps evident by the suite of videos I've created over the years,
be it in the main AI and Games series, or here in AI 101, you'll notice that across
the board machine learning isn't as explored as heavily as other techniques. But why is this?
It's really a reflection of how game development has historically operated. Machine Learning has
typically not been seen as a practical solution for many of the challenges that developers face,
especially in non-player character design. With only a handful of exceptions to that rule.
But with the recent innovations in the field, particularly the boom of research in deep
learning, this is beginning to change. But while we're seeing these new ideas
creep into some familiar territory, such as your opponents in racing games like MotoGP,
Forza and Gran Turismo, or even strategic adversaries in Age of Empires IV,
it's arguably having an even bigger impact in other areas of video game production and is
beginning to fundamentally change how video games are being made.
I'm Tommy Thompson, welcome to AI 101 here on AI and Games, and in this episode we begin our
journey into the world of machine learning. In the first of many episode to come on the subject, I'm
going to give you a very high level introduction to machine learning and explain what is, how it
differs from traditional AI, the many different ways it can be created, and why it is now becoming
increasingly practical to apply in the games industry. This means looking beyond the likes of
non-player characters, and instead exploring how it's being used in player modelling, analytics,
cheat detection, texture synthesis, graphical upscaling, animation blending and much much more.
So let's start with the absolute basics, when we say 'machine learning', what do we mean?
Well to borrow from Stuart Russell and Peter Norvig's 'Artificial Intelligence:
A Modern Approach: a machine learning system is built "to
adapt to new circumstances and to detect and extrapolate patterns".
As these systems adapt and extrapolate these patterns,
in theory they improve at the task they're set, which we observe as it learning. You
might wonder why we want to get a machine to learn to solve a problem? Especially when us
humans know what the problem is, surely we can just write a program to solve it. Well,
to borrow once again from Russell and Norvig there's a number of reasons for this:
"designers cannot anticipate all possible that the agent might find itself in"
"designers cannot anticipate all changes over time"
"sometimes human programmers have no idea how to program a solution themselves"
And to embellish on that latter point, sometimes even if you do know how to get
a basic working solution, it might not be as effective as figuring out how to
get a machine to do it for you. And yeah, that's what a machine learning system is:
it learns and adapts to information given and tries to extrapolate information from within it.
If you're keeping up with the big innovations in AI research and development, you'd be forgiven for
thinking that machine learning is a new concept. But it's conceptually as old as other forms of AI.
The AI techniques we've seen in AI 101 to-date all conform to what is known as symbolic AI,
of encoding human expert knowledge using some kind of symbols and notation, and then using it to
calculate solutions. The reason you might not have heard about machine learning prior to now is that,
well, to be blunt about it: machine learning was kind of a dead end and failed to work at scale
or with larger, more complex problems while symbolic AI continued to deliver. However,
machine learning has seen a new lease on life over the past decade, courtesy of breakthroughs
in research, but critically the benefits of large-scale cloud computation platforms. This
is allowing for bigger machine learning systems to crunch through large quantities of data at speeds
considered impossible even a decade ago. Hence all the recent work in text
and image generation tools, StarCraft 2 bots and all the other weird and wacky
innovations we see in the news nowadays have benefited from this breakthrough.
Now our definition of machine learning is a very high level one, but there a myriad of
different methods and techniques arise that we consider to be machine learning. We won't get
into the specific implementation types *just yet* in this video, but I wanted to break the
definition down a little further. Machine Learning techniques, are typically broken down into three
specific categories: unsupervised learning, supervised learning, and reinforcement learning.
Unsupervised learning is the idea that we can feed data to a system,
apply an algorithm and it will make observations of from it. This is a useful technique when you
have a bunch of data to feed into the system, but you don't have a particular desired output,
you're just interested to see what output it can determine. Perhaps you have data and you just want
to figure out if there is anything interesting to find in that dataset? How can this be applied,
well back in episode 31 of AI and Games we looked at research in the Tomb Raider franchise that used
unsupervised learning on player data to understand how players perform in the game. This project is
one of the earliest examples of player analytics, which is now pretty much a requirement for any
mobile, online or live service game to monitor the behaviour and activity of the player community.
Next up, is supervised learning. This time around you're feeding data into a system, and
you have an expected outcome that you wish for it to reproduce. So your dataset doesn't just include
a set of inputs to the system, but it also has a corresponding set of outputs, and we expect the
system to learn the relationship between them. So for example if I have this list of numeric inputs,
and then this set of numeric outputs, you can see that the output is always the input multiplied by
4. By running a supervised learning algorithm, the system should learn that the underlying
relationship is that the output is always the input multiplied by four. Now the trick is that,
if trained correctly, the system should then be able to take any input that wasn't in the
dataset you trained it on, and still figure out the answer. Why is this useful? Well if we have
a bunch of data on a particular phenomenon then we can reproduce it in known and unknown situations,
and a great example of this in games is through player modelling. By feeding in information
from the player, we can then train a system that reproduces it and you can see that behaviour in
action. Some notable examples of this from previous videos include the Drivatar system of
Forza in episode 60 and as discussed in episode 48 on Google DeepMind's StarCraft player, codenamed
AlphaStar, the first phase of its development is that it learns to play like professional
StarCraft 2 players, by watching their replay data and trying to mimic their behaviour.
The third and final branch of Machine Learning is known as reinforcement learning. This is typically
used where we have an environment within which a system or agent can interact within it and
determine a reward for what is deemed as good or bad behaviour in context of the objective. But the
environment is often highly complex and difficult to learn simple, yet effective strategies to solve
it for a given situation. Hence in reinforcement learning, a system learns what is called a policy
through experimentation, it conducts an educated exploration of the problem and over time it learns
that certain actions are better than others, and learns to use these more frequently.
Now you might be wondering, hey, what about Deep Learning? Well deep learning
is really a process of training what's known as a neural network,
using a combination of supervised learning and reinforcement learning. That's a whole
topic in and of itself. But think of deep learning as a subset of machine learning,
in the same way that machine learning, is simply a subset of AI as a whole.
So having covered the absolute basics, it's worth highlighting
the main differences and strength and weaknesses of machine learning versus
more traditional AI methods. And some of this will help give context as to how it is
now being used in the industry, as well as why it has not received much love until recently.
Machine learning as we've already described, is about taking existing data and establishing the
correct answers from that data. At the start, the system will makes all the wrong decisions
and over time it will begin to improve, and from the outside in, we can pretend that it's learning
like a human does. Thought ultimately all that's really happening is the algorithm is performing
better at the task we told it to do. Now this doesn't mean humans are out of the loop, we define
the function the algorithm will optimise against, we provide the data, we design the structure and
configuration of the algorithm. But the system is then left to understand and build its own
understanding of the problem, and how to solve it. This often results in these systems developing a
much more nuanced understanding of the data it is provided, than what a human could achieve in
a similar timeframe. As discussed in my episodes on the Forza Drivatars and Gran Turismo Sophy, a
machine learning powered race car begins to recognise how to handle thousands of similar,
but slightly different configurations of turn types quite easily. But also they can
adjust strategy for different road and weather conditions and even the vehicle they're driving.
But this strength inherently results in two big weaknesses. First in order to generalise against
that dataset, it needs a significant amount of data and time to calculate the outcome,
and if it doesn't do quite what you want. You need to hit the drawing board again; rewrite
the algorithm and train it again, or gather even more data given it's not got enough to learn from.
Now further research in the field has led to a lot of ML-algorithms learning and converging on
smart behaviours much faster than before, but it's also compounded by the second weakness.
After training an ML system, depending on the approach it's nigh impossible to edit
it quickly or add specific flourishes that, in the context of games, would enhance the
experience. Designers can't crack it open to add unique behavioural flourishes with ease,
or remove a behaviour they don't like. In each case, you'd need to change the restrictions being
applied to the AI to capture these nuances (which of course incurs the expense of training all over
again). This presented three distinct failings that, for the games industry, wasn't compatible:
- First of all, it doesn't support game developments iterative nature. NPCs in
games go through a lot of refinement. They get playtested for bugs, focus tested for engagement
and iterated upon to bring a high level of polish. Machine learning didn't really fit that.
- Secondly, it doesn't support the game development pipeline. It's fine for a
programmer experienced in ML to mess around with these tools. But tweaking learning rate parameters
or neural network configurations is a whole bunch of technical mumbo jumbo that game designers
typically don't understand, and frankly shouldn't have to in order to get on with their job.
- Lastly, the resources required to train each system was a dealbreaker.
Whether it was the time taken, the data required,
or the sheer financial cost of running these systems. It wasn't worth the effort.
Now this isn't to say there hasn't been any effort to put it into practice. We already
mentioned the likes of Forza's Drivatar, which is one of the longest running machine learning
applications in the industry, but prior to that there was the likes of Lionhead's
Black & White and the AI critters in Creatures which are notable exaples of machine learning
for games. But these were anomalies rather than a reflection of an ongoing trend in the industry.
Given the more recent advancements in AI research, we're now seeing a change in the tides,
as more games utilise machine learning for NPCs and opponents. Ranging from the likes
of the AI drivers in the MotoGP series, to the enemy AI in Hello Neighbor 2,
and the campaign AI in Age of Empires IV. But for the most part? Games studios still rely on
tried and true symbolic AI techniques when designing characters for games.
And this brings us to the big secret: the real AI revolution happening in the games industry is
happening elsewhere. We're seeing significant investment in machine learning technologies
for games, but it's all happening in others areas of video game production.
Right now we're seeing an increasing number of machine learning applications in games
for areas that aren't NPCs and enemy AI. Instead it's designed to support
the future of the industry and seeking to improve game productions. And in each case
it plays to machine learnings strengths: it's capturing interesting patterns in
datasets and using that recognition to yield intelligent responses.
For example, way back in episode 45 of AI and Games on the AI of Hitman,
I discussed an approach that computed optimal blend points for animations. This concept has
expanded beyond Hitman into a more complex and nuanced approach typically referred to as motion
matching and is now used in everything from The Last of Us to FIFA. And this
is being expanded to enable animations to be increasingly more flexbiel thanks to machine
learning to capture movements across rocky terrain, handling unique geometric features,
responding to obstacles and compensating for the characters direction and rate of
motion. We'll be returning to this in a future episode of AI 101 to discuss in more detail.
Another obvious one to pull from earlier episodes is Texture Upscaling,
we talked about this at length in episode 61 of AI and Games, where the idea is to train
an system to learn how to generate a copy of an existing image, but at a higher resolution
in a way that still looks crisp. As detailed in that video, this concept was originally
advocated for by modding communities, but the more recent work in the Mass Effect: Legendary
Edition highlights that this is a concept that can be applied at scale in AAA studios.
And of course, we can't ignore the work in upscaling graphics in real-time as well. Deep
Learning is powering the likes of NVIDIA's DLSS superresolution technology. What this means is
that machine learning is being used to take the output of a game and increase the resolution of
the final image. Allowing for users to play games at 4K resolutions without melting their
hard drives along the way. This is a topic I discussed both in episode 62 of AI and Games,
as well as my dedicated DLSS 3 video that explains how this technology has evolved in recent years.
Perhaps one of the most exciting ideas that did make some public traction is quality assurance.
Having AI systems playtest games isn't just a fairly useful tool given the ever-increasing
size of games, it can help alleviate a lot of the pressures being placed on QA teams - who
lets face it, are often not treated very well by certain companies. In fact we've talked about this
already back in my videos on the AI of Division 2, in which they used symbolic AI methods to have
bots run around and test missions and the open world. This has since been expanded upon by a
variety of companies to have bots trained with machine learning that can learn to play a game,
and in doing so, find issues in it as they play. We saw Electronic Art's early work make headlines
as they had ML bots playing Battlefield One back in 2018 and they're still researching new systems
that conduct more rigorous coverage of issues in a game world. In fact this is a big priority for
EA given they started using symbolic AI bots to test Battlefield V as far back as 2018,
and have continued updating their testing tools for everything from Star Wars Battlefront 2 to
Apex Legends and Battlefield 2042. And it's not just EA, companies like modl.ai are now
developing tools that run in game engines to facilitate this for smaller dev teams.
Another fun direction is in cheat detection. Recognising and identifying erroneous behaviour by
players is often very hard to do, and if they're running cheat systems on their PC it's very hard
to catch it happening lest you run some sort of spyware on their device, which is of course,
not a popular approach. There have been some work in this area that is yielding positive results.
Why many people will have heard more recently about Richochet, which is the ML-powered cheat
detection in Call of Duty and Warzone, the longer running system is in Overwatch! No,
not that Blizzard shooter. Overwatch is the cheat reporting system that runs in Counter-Strike:
Global Offensive or CS:GO, and players are incentivised to catch cheaters and report
them to a team of human assessors who will then act on the data provided. However, since early
2017 is also a deep learning system called VACnet that is trained on human reported instances in
Overwatch system to identify trends of behaviour cheaters. Then VACnet can report cheaters to the
human moderators in the same infrastructure as humans do via Overwatch. The big difference,
is that the AI has been found to be significantly more accurate, depending on a myriad of factors
relating to the games ecosystem, a human reporting a player via Overwatch may lead to a conviction
by the moderation only 15-30% of the time. Meanwhile with VACnet, the number of successful
convictions increases drastically to 80-95% of all reports filed via Overwatch by the AI system.
These are just some of the many areas that machine learning is being employed, and to go into even
more detail would reduce in a non-exhaustive list. That said, I refrained from mentioning
the likes of text and image generation, given at the time of this video in 2023, these techniques
have not proven themselves to be valuable in regular video game productions. Time of course
will tell whether the recent boom in these techniques may result in something fruitfrul.
And so with this episode of AI 101 we're painting a picture of what machine learning is,
and how it can be used in games. But ultimately this is all at a very high level. In future
episodes we'll dig deeper, and look at specific techniques and applications, ranging from neural
networks to reinforcement learning, clustering techniques and diffusion models. We have a lot
of ground to cover, I'm hoping you'll stick around to find out more with each new video.
Thanks for watching this episode of AI 101, I hope you found it interesting and
learned a thing or two along the way. In the video description I've linked to some
useful materials be it other episodes of AI and Games, textbooks and a few other
bits and pieces that might help if you want to dive into this subject in more detail.
AI and Games is a crowdfunded series, and it's thanks to our supporters that
new videos make their way to your eyeballs. Check the links on screen
and in the description to find out how you can support AI and Games via YouTube
Membership or on Patreon and in return you get early access to new content,
bonus videos, ad-free podcasts, your names in the credits, exclusive merch and much more!