Hi I'm Tommy Thompson and welcome to Design Dive here on AI and Games. In this episode
let's talk AlphaStar - DeepMind's grandmaster level AI StarCraft 2 player. AlphaStar made
headlines throughout 2019 as the competence of the system grew, first defeating pro StarCraft
players TLO and MaNA followed by playing in public matchmaking on the Battle.net European
servers, allowing it to climb to the top 0.15% of all players. The big question I hear a
lot is how can the games industry capitalise on this and build their own Deep Learning
AI players. But it isn't as straight forward as that and despite the real innovation and
excitement around AlphaStar, this isn't going to have an immediate impact on the way game
AI is developed - or at least not in the way you think. And in this video I'm going to
explain why...
Let me stress that this video isn't intended to speak ill of DeepMind and their work. AlphaStar
is an incredible achievement that - even in academic circles - still felt like it was
years away. But rather I want to... temper peoples enthusiasm a little bit. The media
sensationalism's of AI often means understanding the capability of these systems is difficult
to grasp. But the bigger issue is that the way in which AlphaStar has been built does
not make it an easy to adapt and translate to a game development pipeline. So let's talk
about what this all really means for the video game industry in the short-term rather than
treating this as the next big innovation that will transform into Skynet and eventually
kill us all.
**On that note, it is legit both funny and depressing how everyone and their Aunty knows
what SkyNet is yet Terminator movies are bombing at the box office.**
I won't be talking about how AlphaStar works in this video, because I did that already
over in episode 48 of the main show. So if you want to get a grasp of what's actually
happening under the hood of these StarCraft AI players, then go watch that video first.
Plus I do make reference to some of the points raised in that video, but hopefully it's all
easy for you to follow along with.
The first issue is that the games industry needs to see the benefits of adopting this
approach for non-player character AI before it embraces it. This isn't the first time
machine learning has reared its head offering to fix problems for the video games industry.
In fact there was an initial exploration of machine learning back in the late 90's and
early 2000's - which led to games like the original Total War and Black & White using
neural networks and genetic algorithms - but to mixed success. One of the big reasons that
machine learning died out was the lack of control or authority designers and programmers
have once they've been trained to solve the task at hand. Deep Learning is creating complex
artificial neural networks that carry thousands if not millions of connections that are given
numeric weights. Training the connection weights is what gives the system intelligence, but
when you read that as a human - it's just numbers. Lots and lots of numbers.
So if you build an AI player using Deep Learning and it does something weird, you can't crack
it open and debug it. You need to isolate what's wrong in the design of the network,
the learning process or the training data that may have caused this erroneous behaviour
and re-train it. Then if you want to create AI that cater to particular situations or
configurations, you'd need to build the training process to reflect that. This isn't remotely
accessible or friendly to game designers who want control over how an AI character will
behave within the the game and are working with the programming team to make that a reality.
If you consider episode 47 of AI and Games where I looked at Halo Wars 2, that whole
system is built in a modular data-driven fashion to allow designers to have a huge amount of
control. Right now Deep Learning technologies do not cater to that level of interaction
and oversight for a designer to work with it. It's why behaviour trees are so pervasive
in game AI: they're arguably the most accessible for both designers and developers, allowing
each team to focus on their specialism without stepping on the toes of others.
This isn't to say machine learning isn't going to have an impact within the industry itself,
but more specifically I don't see it being used pervasively for in-game behaviours. Sure,
we've seen the likes of Forza Horizon and MotoGP adopt it for their opposing racers,
but those are very bespoke situations that actually cater quite nicely given the problem
space. The industry is still evolving and adapting to this surge in machine learning
once more and while big publishers are investing in their own AI R&D teams, that isn't reflected
in even AAA studios. Over time we're going to see Deep Learning used more and more in
games, but not in the ways you might think and I'd argue rarely for in-game character
behaviour.
The second issue is that - irrespective of the technologies capabilities - the requirements
for training AlphaStar don't allow for it to be easily replicated for games in active
development. As mentioned in my other video, AlphaStar's first phase of learning is achieved
by watching and then mimicking behaviours from match replays of human players.
So this is a chicken and egg problem: given if you want to train super intelligent AI
in your game, you need to have existing examples of high-level play that it can replicate through
supervised learning. If you want that training data, then you either need to have expert
players playing the game before release or build a separate AI player to bootstrap the
machine learning player by creating examples for it to learn from - and that kinda defeats
the point. AlphaStar benefits heavily from the ecosystem that StarCraft exists within.
The game has been out for nearly a decade and is relatively bug-free, it's been a popular
eSports title for several years, plus Blizzard's cult of personality helps maintain an active
and lively fanbase around their products. This means lots of data already exists for
AlphaStar to work with.
Now all that said, the AlphaStar is still quite a fickle system. The two version of
the AI player were built against two specific version of StarCraft 2 - with version 1 running
on 4.6.2 and version 2 on 4.9.2 of the game. Now the unspoken problem here is that any
changes made to the games design that influence the multiplayer meta in any significant way
will break AlphaStar. The reinforcement learning trains the bots against that current meta,
and that means it can't just adapt to the changes brought on by the patch, you need
to retrain it. Even the human expert play it's bootstrapped against might not even prove
applicable anymore in this context. I can't say with any certainty, but there's a small
chance that already as of version 4.10 of StarCraft 2, AlphaStar might not be able to
play as well as it once did.
The third and most critical element that prevents AlphaStar being adopted en-masse is cost.
Training the AlphaStar agents is an incredibly expensive process, you need to have dedicated
processing systems for the training to run in a large distributed heterogeneous fashion.
DeepMind utilise Google's own cloud infrastructure to achieve this and the training was executed
on their Cloud Tensor Processing Unit's or TPUs. These are custom-developed application
specific integrated circuits or ASICs designed to help accelerate machine learning training
and inference.
The more recent version of AlphaStar from November 2019 trained on 384 TPU v3 accelerators
- for a period of 44 days. Now if you consider Google's public pricing model for using these
TPUs, which runs at around $8 an hour for a single TPU, then even a naiive estimation
of cost amounts to $3,072 per hour, $73,728 a day and $3,244,032 in total. Though I'm
sure DeepMind got a heavy discount.
Now you might think this isn't a big deal when some AAA productions have budgets in
the tens if not hundreds of millions of dollars, but $3.5 million to train your AI is a ridiculous
amount of money. Sure, publishers like EA, Take Two, Ubisoft or Activision might have
that kind of cash available, but this is just the cost of running the training, not the
staff, the infrastructure, the development time and all that other critical parts of
game development. Bear in mind this is but one tiny part of a much larger puzzle when
building a game of a scale akin to StarCraft. Plus, as cool as this ridiculous expenditure
is, DeepMind are actually haemorraghing money right now - posting losses for Alphabet (the
umbrella company of Google) exceeding $1 billion dollars in the last three years. This technology
is not stable enough at this stage without further investigation for a AAA publisher
to take seriously.
Perhaps even more critically, this excludes all but the top 2% of games studios and publishers
even if they could afford it. The training costs suggested here are bigger than most
development budgets for a game. This technology can't permeate throughout the industry if
it costs that much to train it. And of course, if you need to train it again, as your design
needs force you to reconsider something - boom - that's more money being thrown at Google
to solve the problem. Alternatively, a company invests in their own Deep Learning infrastructure
or use another provider. In any case, money, money, money.
I will stress this isn't just an issue of unabated capitalism: the issue of data and
compute resource to train Deep Learning systems is not a solved issue and is one of the larger
problems being addressed not just in research of AI methodologies, but even hardware companies
such as Intel that are building the next generation of compute hardware to deliver training and
inference of machine learning cheaper and faster than is currently possible.
Now while I'm stressing that AlphaStar isn't going to change gaming just yet, that is not
to say that machine learning is not having an impact within the games industry. As I
mentioned earlier, the intial enthusiasm for machine learning largely petered out by the
mid-2000's, the recent Deep Learning revolution has seen renewed interest. But this new and
more concerted effort is being explored to address issues beyond just the creation of
traditional AI players. EA's SEED Division revealed their work in 2018 training Deep
Learning agents to play Battlefield as well as exploring imitation learning from human
play samples to bootstrap AI behaviours. Meanwhile Ubisoft's La Forge research lab in Montreal
is experimenting with machine learning for testing gameplay systems, AI assistants that
support programmers in committing bug-free code, motion matching animation frameworks
for character behaviours and lip syncing for dialogue in different languages. Plus the
most obvious applications in data sciences are long established at this point, as analytics
teams use machine learning to learn more about how people play their games and provide insight
into changes that can be made going forward. I mean let's look on the bright side, I'm
going to have plenty more to talk about on this channel in the coming years!
Thanks for watching this episode on Design Dive, I figured it was worth giving my 2 cents
in explaining why we shouldn't be expecting Deep Learning to invade all of Game AI just
yet. I hope you found it interesting! If you've got questions, comments or just flat out disagree
with me then slap that down in the comments and once I've had enough to drink I'll go
take a look! Don't forget you can help to support my work by joining the AI and Games
Patreon or by becoming a YouTube member - just like Scott Reynolds, Ricardo Monteiro and
Viktor Viktorov have done right here, plus all the other lovely folk you see right here
in the credits. Take care folks, I'll be back.