starcraft 2 is a multiplayer game where the objective is simply to eliminate the other players
you do so with an army of attack units we like this we do not like this those attack units cost
resources to make so you probably need to set up a base build various buildings collect resources and
then you can build that army all of this involves seemingly endless strategizing and knowledge
of the game along with years of tuning muscle memory to achieve high actions per minute but
we're programmers and we can just use libraries that let us use python code to play the game for
us even this however takes a while to master all of the strategies but we're also machine learning
practitioners could we just simply import some reinforcement learning from stable based lines 3
and play the game i guess it's going to be a little harder than that but let's say that we
really do want to use deep reinforcement learning to play some starcraft 2. how might we actually
do that like most deep learning problems at the macro level we have to consider what our inputs
and what our outputs are going to be as these are the things that we actually have control of and
we have to determine the characteristics of these ahead of time for the input what information do we
think the neural network will actually need in this case it could quite literally be the game
visuals like the frames from the game it could be some other graphical representation or it could
just be a vector of values like number of units and minerals and locations of things and so on
then for the output of this model we need to consider what this neural network can actually
decide essentially with deep reinforcement learning in this case this is going to be
the actions that the model can actually take like whether to expand and build out our base to attack
the enemy and so on for the input into the model i think something like the minimap is actually
a really good candidate for model input showing the actual game would mean that the agent would
have to come up with some logic to either move the camera around or it would probably wind up
mostly paying attention to the mini-map anyways so i think building our own version of the mini-map
with the data that we think is important i think that's the best place to start from for the input
map we're going to start with a blank canvas that is just simply the size of the game map then first
we're going to draw the minerals not only are we going to draw them but we'll also make them more
bright the more minerals there are left we'll use the same concept for minerals as well as the gas
but also a similar idea for building and unit health here's the result of this so far we can
only gather information on minerals that we can see with units which is why some of them are just
very dim and then some are very bright the dim ones we just we haven't seen so we have no idea
how many minerals are there next we'll mark the enemy starting location with a red dot then we're
going to use a similar strategy for enemy units in another red and then enemy structures as well in a
red color then we're going to mark our own nexus buildings in a light blue color and then all the
other structures a greenish color we're going to mark vespine as a pink we're doing this after our
structures because if we just slap a structure on top of the vespine we wouldn't actually know what
the amount of spin was right you can only have one thing on a square so i just went with draw
the buildings first because we end up building a building on top of the gas to collect the gas
so draw the buildings first then draw the best bean then finally we'll draw all our units
which will be green by default but we're going to color the void ray unit which is our attack unit
specifically a blue i think this gives us a pretty good final result where it's fairly easy to see
what's going on in the game i did most of my r d here in linux which didn't allow me to run the
graphical representation of the game and just by watching games with this representation i think
it's really easy to see what's going on how we're doing and all that i think you could definitely
make good decisions with this information so i think this will be a pretty good starting point
for input to our model will it work maybe remember the best thing that we can do is keep things
simple to start one issue i can already imagine here is that this is just a ton
of mostly noise per input the image is mostly empty so it's going to be an area that we could
probably improve upon later but now that we have network input we need to handle for the output
logic which will be the macro actions that we're going to allow the reinforcement learning agent to
take for action 0 i'm referring to it as the expand action this is the code for it essentially
for expands we want to make sure we have enough supply to grow and build as many void rays as
desired if we have that then we'll check to make sure we have enough probes these are essentially
the worker units if not build more of those if we don't need those then let's build assimilators to
collect the vespine gas finally if we have all of this stuff then let's make another base in another
resource area for action 1 it's all about building stargates and to build stargates we need gateways
and cybernetics cores to also exist if those don't exist build them otherwise build a stargate right
now the code is limiting one stargate per base but this could be changed to allow the ai to
make endless stargates theoretically speeding up the ability for you to build void rays
faster for action number two if we can afford and we can build a void ray build one action
number three this is to scout just not too often essentially scouting is sending a poor
probe to its certain death and we're just doing this to see what the enemy is up to
initially we're going to start out completely random so theoretically one out of every six
frames we could be sending a probe and this would be faster than we could possibly build probes
so just to limit this we're adding a once every 200 frames that's when you can send another scout
action four is to attack first we're going to look for any close units then any close buildings than
any known unit then any known building per void ray if all else fails then we're going to send
the void rays to the enemy start location we could probably get a little more fancy here since it's
totally possible that we eradicate the enemy from its starting location and we have no idea where
it is from this point but it's somewhere building its base and collecting resources and all that so
we might want to have some sort of ability to hunt or seek out the enemy in other locations
finally action 5 is mostly tuck tail and flea the void rays should also be capable of just simply
returning to the base after a fight so it's not just to flee it's also just to bring our void rays
back home the void ray units are really our only base defense here so if they're not busy elsewhere
we should bring them back sometimes at this point the last thing that we need to do is calculate a
reward for the agent to learn from but first is this even a worthy experiment at all i can already
hear some people complaining that the agent isn't controlling every single action in the game and we
must ask ourselves how will we know if this agent is good or not we need a benchmark to compare to
so before we waste our time on a reward mechanism and all the r d required there which honestly
is often the hardest part of any reinforcement learning development we really do need to test
random which is really just a question of how good does an agent do if it is just randomly
picking actions we have a lot of helper code here some failsafes isn't it really just possible that
randomly picking actions will also win games or at least be remotely comparable to any reinforcement
learning agent well we can test this really easily after running 200 games running random actions
we have not won a single game okay i think we have something we can certainly work for all
right now let's talk reward in order for the agent to learn anything at all we need to reward it for
doing quote unquote good rewarding agents and reinforcement learning is often the hardest part
of all take our problem for example the actual goal here is to win games it's pretty simple
if we win we could give the agent a reward and if we lose a big negative reward the issue is as
we've seen already the agent starts off basically random and the number of observations and actions
per game can be five thousand to ten thousand or even more sometimes if we go through 10 000
actions and lose the game as we pretty much know we initially will every single time how will
the agent know which actions were the ones that lost the game and more importantly which actions
would be good to take with this many actions to sift through before getting an actual reward it's
far too challenging for a model to discern which one was the good action and which ones were the
bad actions so instead we're going to need some sort of intermediary rewards to help boost the
activity that we think will be useful for victory to do this you can just give a very small reward
per step for things that you think might be helpful and then you can still give one large
reward at the end for actually winning the game so what might we reward to help win we could
reward resource gathering but this would probably just result in an agent prioritizing lengthening
the game so it can collect as many resources as possible and it probably wouldn't want to build
many attack units since these cost resources we could go off of total units and buildings as well
as a sort of reward for the overall size of our military but again this would disincentivize the
agent from actually winning and completing the game instead it would just want to build lots
and stay alive for as long as possible so for me i personally settled on something extremely simple
a reward for attacking for every void ray which is our attacking unit that we have in our army if
that unit is actively in combat then they earn the agent a small reward per step this incentivizes
actually eliminating or at least attacking the enemy it will still possibly be the case that
keeping the enemy just barely alive to make more units and buildings for us to destroy will happen
but my hope is that a large reward for actually finishing the game will overcome this okay so
we've got some great ideas here but we still have to actually connect stable bass lines 3 in this
python starcraft 2 environment and to do that we also need to convert the python sc2 environment to
an open ai gem environment in order to work with stable baselines 3. due to the way everything
runs and can or cannot communicate while it runs this presented a far more difficult challenge
than i expected if you're looking to learn more about stable bass lines 3 and custom environments
i'll put a link in the description for a tutorial series i did it's a lot more simple and easier to
understand if you're new to it compared to what i'm going to be doing here so for everything to
communicate here i ended up going with a state file this file contains the state which is going
to be our minimap the reward action and whether or not the game is done and this shared file is
how we'll communicate between the systems here is there a better way absolutely will this work
though i hope so so this is definitely going to be very confusing with everything that has to connect
and wait on each other and all this but let's just go in logical order so to actually run everything
we're just going to have a training script train dot pi and this is what's going to actually run
and start up basically everything so for train.pi there's really nothing fancy here it's a typical
stable baselines 3 training script the script runs the sc2n file which i haven't shown yet for
our custom environment so now let's peek at this sc2mv file so what this is is a custom openai gym
environment essentially we have an init step in a reset method again to learn more about this check
out the sp3 tutorial in the description it'll be a much more easy to understand than this but
basically all this does is initialize the action and observation space for the agent immediately
then before the environment actually runs it is reset and here is where the magic or hacky code
happens in the reset method we empty the map and prepare our state pickle file note that the action
in this pickled dictionary is none to start after creating and saving that pickle we open up another
process to actually run the starcraft 2 game so this is actually running the game with the logic
that we've kind of described up into this point as well as through the blizzard api and actually
playing starcraft 2. finally the reset method actually is just returning an observation which
is that map that blink map that we've created so at this point theoretically the starcraft 2 game
is just kind of hanging out waiting for us to issue commands to it so we've returned
this observation from the reset method which stable base lines 3 via this train dot pi file
passes through the reinforcement learning model in this case that's a ppo model and
that model returns an action stable baselines 3 then passes that action back to our environment's
step method in the sc2m step method the first thing we wait for is an action to be picked by
the model once we have that action we add it to the state dictionary keep in mind again up to this
point the action has been none so what's actually waiting for an action our actual starcraft 2 game
and the script that communicates with it which is incredibot.pi once that script detects a new
action it performs the action actually in the game it redraws that map that we've created
it calculates any reward then replaces the previous state dictionary with a new one where
yet again the action is none but we're returning the reward and a new observation we'll also
handle at the end for if the game actually ends from here which script is waiting on an action
that's the sc2 end script and the cycle goes on until the game ends which will issue a done and
then the environment resets and we continue on indefinitely so yeah this is definitely the most
spaghetified code for reinforcement learning that i've written but does it work of course
it works around this time i spent quite a while tinkering with various rewards in logic but the
one that worked best was just a simple reward for attacking and of course winning or losing
while the models were training i was obviously tracking things like overall reward the reward
mechanism itself changed quite a few times so each variant isn't necessarily directly comparable to
the other one as well as i have personally found that it's best to try multiple times with the
same exact reinforcement learning algorithm as it seems the random initialization starting point of
all models can actually play quite a significant role with reinforcement learning and that actual
model learning over time as you can probably tell i made many attempts before settling on
that very simple reward mechanism of just simply rewarding for combat and then at the end a reward
or punishment for a winner loss the best model landed on a reward of about 200 at its peak
that said what does that actually translate to as models trained i also kept a log of wins and
losses since that's actually what we're truly after so i wanted to measure the winning rate
in the case of this model at the same time as its 200 reward peak we can see game victories
at around 70 against the hard computer bot which isn't too bad especially considering random was
literally zero percent we were never winning despite the amount of effort put in so far i
would still consider this just the beginning of any reinforcement learning with starcraft 2 and
i'll have to think more on how i might expand this to do better at both macro tasks like we've done
here as well as some sort of micro reinforcement learning algorithm probably in the form of two
separate algorithms running at the same time where one algorithm is focusing on macro like what we
have here and making large overarching decisions like build buildings and expand or attack and so
on and then another one that focuses solely on micro tasks for example like what coordinates to
go to or build on and so on but overall i'd say this is a pretty good start i did also attempt
to build some sort of hunting capability for the void rays to find enemies at more locations than
just their base or if they just happen to show up since it seemed like many games were lost because
the enemy just hid when we had the advantage but later came with a bigger army and defeated us
i ended up allowing void rays to look at random coordinates if no other enemies were present but
my results here were actually worse than when we weren't looking at all i still think there's
something here to be fixed logically probably fairly simply but again i wonder about maybe
instead making a reinforcement learning algorithm that chooses what coordinates to fly to rather
than me hand coding even more logic i think making a micro strategy reinforcement learning algorithm
to pick coordinates for travel is very likely the next step to try that however is a project for
another day i've hosted all the code as well as a decent model that you can play with if you'd like
you can find that in the description of this video if you're interested in learning more about neural
networks and how they work then you might want to check out neural networks from scratch a book by
myself and daniel kukiewa inside of which you'll learn how to code neurons activation functions
how to calculate loss to optimization and back propagation and of course apply everything that
you've just learned to a problem the book itself is in full color for graphs and code syntax
highlighting everything's on high quality pages with full font and a comfortable line spacing to
make reading easy finally every concept including the math and coding is broken down into the truly
atomic parts and then put back together step by step to give a full understanding of neural
networks and how they work including all the math behind them by the end of the book you will have
created your own neural network framework from scratch this is the book that i wish
existed when i was learning deep learning and i'm extremely happy to be able to offer it to you if
this sounds interesting to you then you can head over to nnfs to learn more and buy yourself a copy
you