Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
This topic has been perhaps the most highly anticipated by you Fellow Scholars and I am
extremely excited to show you the first joint paper between DeepMind and Blizzard on creating
an AI program to play Stacraft II.
Hell yeah!
And fortunately, we have a paper where every detail is meticulously described, so there's
much less room for misunderstandings.
And before we start, note that this is a preliminary work, so please don't expect superhuman performance.
However difficult you thought this problem was, you'll see in a minute that it's way
more complex than most people would think.
But before we start - what is Starcraft 2?
It is a highly technical strategy game which will be a huge challenge to write a formidable
AI for because of the three reasons.
One, we have imperfect information with a partially observed map.
If we wish to see what the other opponent is up to, we have to devote resources to scouting,
which may or may not be successful depending on the vigilance of the other player.
Two, we need to select and control hundreds of units under heavy time pressure.
One wrong decision, and we can quickly lose most of our units and become unable to recover
from it.
And three, perhaps the most important part - long-term strategies need to be developed,
where a poor decision in the early game can lead to a crushing defeat several thousands
of actions later.
These cases are especially difficult to identify and learn.
However, we ran a bit too far ahead to the gameplay part.
What needs to be emphasized is that there is a step number one before that.
And that step number one is making sure that the AI is able communicate and interact with
the game, which requires a significant engineering effort.
In this paper, a Python-based interface is described to make this happen.
It is great to have companies like DeepMind and OpenAI who are devoted to lay down the
foundations for such an interface, which is a herculean task.
This work would likely had never seen the light of day if AI research would only take
place in academia.
Huge respect and much thanks for the DeepMind guys for making this happen.
To play the game, Deep Reinforcement Learning is used, which you heard about earlier in
the series.
This is a powerful learning algorithm where a neural network is used to process the video
input and is combined with a reinforcement learner.
With reinforcement learning, we're observing the environment around us and choose the next
action to maximize a score or reward.
However, defining score was very easy in Atari Breakout, because we knew that if the number
of our lives drops to zero, we lost, and if we break a lot of bricks, our score improves.
Simple.
Not so much in Starcraft 2, because how do we know exactly if we're winning?
What is the score we're trying to maximize?
In this work, there are two definitions for score: one that we get to know at the very
end that describes whether we won, had a tie, or lost.
This is the score that ultimately matters.
However, this information is not available throughout the game to drive the reinforcement
learner, so there is an intermediate score that is referred to as Blizzard score in the
paper, which involves a weighted sum of current resources and upgrades, as well as our units
and buildings.
This we have access to throughout the game.
This sounds good for a first approximation, since it is monotonically increasing if we
win battles and manage our resources well, and decreases when we're losing.
However, there are many matches where the player with the more resources does not have
enough time to spend it and ultimately loses a deciding encounter.
It remains to be seen whether this is exactly what we need to maximize to beat a formidable
human player.
There are also non-trivial engineering decisions on how to process the video stream.
The current system uses a set of feature layers, which encode relevant information for the
AI, such as terrain height, the camera location, hit points for the units on the screen and
much, much more.
There is a huge amount of information that the convolutional neural network has to make
sense of.
And I think it is now easy to see that starting out with throwing the AI in the deep water
and expecting it to perform well on a full one versus one match, at this point, is a
forlorn effort.
The paper describes a set of minigames, where the algorithm can learn different aspects
of the game in isolation, such as picking up mineral shards scattered around the map,
defeating enemy units in small skirmishes, building units or harvesting resources.
In these minigames, the AI has reached the level of a novice human player, which is quite
amazing given the magnitude and the complexity of the problem.
The authors also encourage the community to create more minigames for the AI to train
on.
I really love the openness and the community effort aspects of this work!
And we've only just scratched the surface, there is so much more in the paper, with a
lot more non-trivial design decisions and a database with tens of thousands of recorded
games.
And, the best part is that the source code for this environment is available right now
for the fellow tinkerers out there.
I've put a link to this in the video description.
This is going to be one heck of a challenge for even the brightest AI researchers of our
time.
I can't wait to get my hands on the code and also, I am very excited to read some followup
papers on this.
I expect there will be many of those in the following months.
In the meantime, as we know, OpenAI is also working on DOTA with remarkable results, and
there's lots of discussion whether a DOTA 5 versus 5 or a Starcraft 2 1 versus 1 game
is more complex for the AI to learn.
If you have an opinion on this, make sure to leave a comment below this video.
Which is more complex?
Why?
This also signals that there's going to be tons of fun to be had with AI and video games
this year.
Stay tuned!
Thanks for watching and for your generous support, and I'll see you next time!