what is going on everybody and welcome
to a machine-learning slash deep
learning example with halite so before
we get too far into that I just want to
cover real briefly at least the the
overarching structure to really any
machine learning at least classifier
obviously there's different things
there's like regressors or you might
have generative output or something like
that but for your your typical
classifier whether it's deep learning or
machine learning you're gonna have some
sort of some sort of input vector or
array I suppose but we'll call it a
vector and that's just gonna have values
so these are going to be your features
so this could be anything in the case of
halite these features might be things
like how many planets do we have how
many ships do we have how many enemies
do we have how far away is enemy number
one how far away is enemy number two so
they need to be some sort of numerical
value you couldn't pass an entity object
in this input vector and then you're
gonna have some sort of output now in
the case of deep learning it's probably
going to be an array and it's going to
be some sort of one hot array in most
cases if it was traditional machine
learning then your your outputs probably
gonna need more like a 1 a 2 a 3 a 4
something like that depending on how
many choices you have so let's say with
deep learning we're gonna use deep
learning here so we're gonna think of
the output as this sort of one hot array
which means only one value is one the
rest would be zeros so in my case we
might say for each value in here that
first value is attack the second value
is mine and then the third value might
be Explorer or something like that so in
this case where we've got the zero one
zero that would mean we want to mind so
let's mine our own planet but if instead
the output vector was maybe a 1 that
would mean attack was hot so we would
choose to attack for example
so just a really quick overview of what
we're gonna attempt basically in order
to do machine learning or deep learning
we have to translate everything we have
into this sort of structure where you've
got input features and then an output
choice so and then in the case of deep
learning that output choice needs to be
some sort of like vector basically a
one-hot array so now let's talk about
how we can go about doing this so we
have a few options with highlights so
the highlight starter bot which I can
pull up real quick which is totally
worth checking out we're not going to be
using that for a variety of reasons but
also the way this works is off of replay
files so those little files that you can
upload to watch your replays those
contain basically all the information
from the game it's basically it's like a
log file you can think of it like a log
file so what the the ML starter bot does
is it reads the log files and then just
kind of determines where we should
allocate our ships so this AI only makes
that choice of where should the ships be
allocated and it doesn't really do
anything else besides that so there's
obviously gonna need to be a lot of
logic on top of that but that's totally
fine so you can use AI for choosing your
entire strategy and where to go or you
can use AI for something really simple
like how fast should your ship go and
that's the only thing he hews AI for so
you can use it for literally anything so
what I'd like to do is use AI to choose
whether we should attack mine our own
planet or mine an empty planet just
something simple enough and for now I'm
not going to use the replay files but if
you are interested in using the replay
files and you still want to check it out
check out the ml starter bot but also
I'm just gonna copy and paste a snippet
of code here just for anybody who is
interested you paste this is code and
also there I'll put a link in the
description because we're just going to
be doing a lot of snippets of code
rather than writing everything
here otherwise we'd be here for hours
this is a snippet of code to read a
replay file again it's in the
description there's a link to this
tutorial in the description you can get
this code but for whatever reason with
the ml starter bot I couldn't read
replay files maybe it's like a Windows
thing but the code didn't decompress the
game data at all so I'm not really sure
if maybe the stuff that's in the cloud
is not compressed I don't know anyway if
you want to read your own local files
you definitely need to decompress zes
Dedes you'll have to install that as
well I think I don't think that's a
standard light anyway that's an example
in those replay files they're just
basically just these giant JSON files
and they contain here you can iterate
through the game frames you can get
events you can get information on planet
ships and then you can kind of pour
through this is just some examples of me
kind of parsing that data but again it's
like a log file so it's really
challenged like it's not like you can
just read the replay file mimic it and
directly mimic it you actually you still
have to kind of parse and translate that
log file so I think it adds kind of a
layer of complexity that can make
especially if you're not totally
familiar with deep learning I can make
that a little more challenging I think
to pick up but maybe I'm wrong but you
can also like I said besides this
tutorial you can go check out their
their ml starter bot as well and maybe
try both so now let's go ahead and get
started with the script like I said I
think I'd rather just use the snippets
here and explain its most of this stuff
we've already gone over so I would just
be wasting time if we're just gonna
rewrite the script so what I'm gonna do
like I said is I'll have a link in the
description for all the code and then
I'm just gonna move it over here and
then basically just explain it as I go
that way we're not that way you guys
don't have to type it and I don't have
to type it cuz again pretty much all
this stuff is stuff we've we've coded
before it's halite code so I'm just
going to paste this in so we're
importing that Hallett directory logging
to log order because eventually
we want to have a dictionary that for
sure retains its order numpy for numpy
thing
random for choosing a random choice and
OS for making and deleting files well
really just deleting files thank you I'm
gonna have a few more constants here
this is gonna be how many features per
entity that we want to track that's
gonna make a little more sense once we
get to that point of entities percent
changed chance kind of like before where
we had our ships and we're kind of
saving whatever plan we had for our
ships this is the chance that that plan
might change so if a ship is currently
attacking it might get to the point
where we're gonna change its plan and
make it go mine or something like that
and the finally desired ship count I'm
setting us to 20 somewhere between 20
and 50 is a good number 50 seems to time
out so once you get to many ships your
calculations are going to take too long
so you have two seconds to to iterate
through all of your ships so each turn
you have two seconds so so one option
you have is to time how long you're
taking and then once you've taken too
long you just break but then that means
a lot of ships are probably just gonna
sit there and do nothing which is not
desirable so instead what I've decided
to do is if we have more ships than
however many ships we desire to have we
send the remainder of ships off to their
death and make them attack so that's how
we're going to kind of keep control of
our ship count there's there's a few
different ways that we could handle that
best the way I'm going to handle it so
then we're gonna have our typical you
know starting information so just the
the name of the game and start our
logging information and stuff then just
like before ship plans and like I said
with that percent change chance you got
a 30% chance basically every frame per
ship to change the plans of a ship now
what I'm going to do is we're going to
if the file exists we're going to delete
it so basically every game we're going
to save all of this players moves to a
file depending on whatever version
number it is so basically we're going to
have two scripts the exact same
script it's just gonna be one's gonna be
player one one's gonna be player two and
then whichever player wins we're gonna
take those winning moves and then we'll
use them to train and at least to start
these it's just going to be totally
random so the both bots will just be
making random moves and then we use
those to train and then you can actually
just continue that iterative process can
have two AIS compete against each other
train based on the winners and continue
as long as you have some degree of
chance and randomness you should find
that you you continue to slowly improve
so once we have that we're ready to go
ahead and begin that that main loop so
let's do the wild troop game map you
should already understand that command
queue nothing new there so now the big
huge crux of this entire program is
collecting those features so there are
some features that we can collect
without doing it per ship so I'll paste
those in so we can find out all of our
team ships we can find out all of the
ships and then we can find out all of
the enemy ships we can do that without
doing it on a per ship basis because all
we're trying to figure out is like how
many ships do we have here and then we
can get out so how many ships do I have
how many ships does my enemy have and
then how many ships total are there the
next thing is I'm just gonna grab here
we're gonna get our ID and then we're
gonna prep at least prepare for empty
planets our planets and then enemy
planets and then here I'm gonna iterate
through all of the planets I'm gonna
grab their size and then if it is owned
or I'm sorry if it's not owned that's an
empty planet so we're gonna save that so
this entire these dictionaries are going
to be by size and so if it's empty we'll
throw it in there if if if we are the
owner
then we're gonna save it to our planets
and then if we're not the owner but it
is owned then we'll store it to the
enemy planets so then once we have that
we can figure out how many planets each
player has so there we have that and
then finally we can go by the actual
sizes themselves so I'm just gonna paste
that in and basically all this gives us
is it's the K for K so it's key for key
the key was the size of the planet so
this just tells us how big are these
planets and then because the sword is
gonna go from smallest to largest we
actually are going we we might want
largest to smallest planets so I'm just
reversing the list at the very end there
and then once we have all this data
we're ready to actually start iterating
through our ships because the rest of
the stuff are things that we need to
calculate on like a a per ship basis so
for example we might want to know how
close are the closest five enemies to
each individual ship or how close is the
closest planet or so on so there's a lot
of a lot of those kind of kinds of
questions we have to I have to ask that
on a per ship basis so I'll at least
start this so as we iterate through if
the ship is basically if the ship is
docked let's skip it
this is questionable whether or not you
want to leave it or not
I would test it both ways I kind of
think it's pretty inefficient to like
dock and undock and dock and undock so I
didn't leave it in there but because
we're only choosing between three
choices and one of those choices is mine
empty planet which is not likely to be
an option much past the first let's say
100 frames it's probably only gonna
learn one move which would be attack
okay so so it'll probably wind up with a
very aggressive AI if you go with this
whereas if you remove this the AI would
be much more likely to learn other
instances where hey yeah we actually can
dock so yeah
anyway continuing along we'll grab the
ship ID and then determine whether or
not we want to make a change
so here change is currently false but if
we choose a random integer that is less
than or equal to the percent change
chance so basically I think we chose 30
so if it's less than 30 30 percent
chance change equals true and then we'll
use that down the line when we actually
go to actually pick whatever move we
want to make okay now comes the massive
list of features that we're gonna grab
it's a ton of code but these are pretty
much all the same idea so I'm just gonna
paste it and then we'll go over it so
entities by distance this just grabs all
the entities by distance we've done that
before and then we sort the entities by
their distance so whatever is close up
I'm like making this bigger as if that's
gonna help you anyway
and then we're gonna sort that by the
key so it'll be closest to farthest so
now the the closest enemy planets
this will just be anything that herbs
started this is close to empty this will
be anything that is doesn't have an
owner but yet it's still going to be
sorted and then entities by distance
that distance and the 0th element this
will be the the values here are the
actual entity so you could navigate to
each value in this list and then these
are simply the distances so we we want
both because on you know to feed our
neural network or any machine learning
classifier we want to know we want
scalar values right so we want the
actual distance how far is that thing
whereas here if we say hey yeah we
actually want to go to the closest
planet number one or whatever we also
want to be able to know okay what is
that because like we've got the
distances but we also need to know the
entity so then we're just doing the
exact same thing for our owned planets
and then
enemy planets and then honestly we it's
the same thing for the team ships we're
grabbing the actual entities and then
we're also grabbing the distances same
order and all that so then finally yeah
so that's all done and then we come down
here and this is probably this will be
this will slightly different because
basically what we're doing is we're one
of our factors that we're gonna pass is
going to be the sizes right we've got
sizes and distances if there's going to
be times when like for example when the
game first starts we don't have any
planets and our enemy has no planets all
right so if we don't have those values
we're just gonna populate negative 99 so
the distance will be negative 99 we also
end up doing the same thing for like
closest empty planets all these things
like with the distances we do this fixed
data function which actually I haven't
put that up yet but I'll show it in a
second which again is just gonna do the
same thing it's gonna make sure we have
as many values as we're expecting
because I guess I didn't mention it
before but that input remember with the
input vector for example and it had I
don't know just some numbers in it the
size of this input vector at least in
most most of the time the size of this
input vector needs to be exactly the
same we can't change the size also the
meaning of this first value in second
value let's say this is how many ships
we have how many ships the enemy has
this first or the zeroeth element here
needs to always be how many ships do we
have the the the first element needs to
always be how many ships is the enemy
have so and then let's say this is the
closest enemy planet that needs to
always be the closest enemy planet but
what if the enemy has no planet well we
still have to pass a value there so in
this case we're just going to pass
negative 99 so while we're up here - let
me just go ahead and I'm gonna paste
these two helper functions that I wrote
okay so key by value this just binds the
dictionaries key by the dictionaries
value it's kind of a reverse operation
dictionaries we're never really meant to
be looked up by their value that are
meant to be looked at by their keys so
we kind of need to iterate through but
then sometimes we're not gonna have that
any value so we'll return negative 99
otherwise we'll return the actual key
which will be in most cases the distance
so it's basically we're looking for a
distance by an entity and then fixed
data is kind of like what I was saying
before so we're asking to have five
features but many times we won't own
find planets for example so we're saying
hey we want to know the distance to our
closest five planets per ship that we
own but many times we won't have that so
maybe we've only got one planet so what
we're gonna do here in fixed data is we
just had that data data out so we're
gonna say here if if this fails we're
just gonna say zero let's say if we had
no data at all we're just gonna say zero
but which would president say probably
should be like a negative 99 anyway
regardless the show must go on if there
is a first value will append that first
it's basically just gonna pad that value
out and let's say there's three values
but we need five values is if it's going
to take that that third value and Pat it
out that way again because we always
have to have fixed size of that input
vector okay so now we're gonna come back
down to where we were creating this this
is the entity lists now what we're ready
to do is actually create that input
vector so I'm gonna paste a little bit
of code first that so basically what
this is going to do is because entity
lists these are lists where as the the
input vector needs to just be flattened
right so these each of these is a list
of five values basically well actually
it's a list of quite a few values it'll
be at least five because we're running
fixed data on it but it could in theory
be many many more so we're gonna do that
then we're got our input vector
and then we're going to iterate over
this list which is a list of lists
basically and then for each element in
that list up to however many features we
require we're going to append them to
the input vector so our input vector is
going to have the five closest empty
planet distances it's going to have the
five closest my planet distances five
closest enemy planet distances and so on
that's just going to keep going now at
the end of all that we also want a few
more stats so we're gonna go ahead and
also bring in the following so we're
gonna add in how many ships do I have
how many ships is my enemy have how many
planets do we have how many are empty
planets and then how many planets does
our enemy have okay so quite a bit of
data and honestly all the way up to this
point all we've been doing is collecting
features we haven't done anything we're
just figuring features out and then also
some of these features like planet size
for example I I've almost I've basically
never seen a planet really get depleted
and actually I'm not a hundred percent
if someone can comment below if you
actually get more docking spots for a
larger planet so I can't remember if you
do or not but anyways so planet size may
or may not be important right and then
there's probably a bunch of important
factors that we actually could continue
adding to this so this is just meant to
be an example but anyways we've
calculated of all of our features and
now we're going to calculate the output
vector which at this stage is just going
to be random eventually we're going to
have our neural network actually pick
the output but for now random so if the
ship count is greater than the desired
ship count our output vector we're gonna
say is basically what we're gonna say is
if if the zeroeth element is a one so in
fact let me just do this let's say we'll
call this
attack and that would be a 1 0-0 okay
so if we have more ships than we hope to
have then we're gonna say ok the output
vector starts off as three by zero so
that's just going to be a zero zero zero
and then we're gonna say the output
vector zero with element is equal to a 1
which means one zero zero if that's the
case then we're going to ship plans
whatever that ship ideas is that output
vector then do this now if that ship
doesn't actually have a plan so if this
isn't the case and that ship doesn't
have a plan or change is true we want to
pick a new plan so the output vector
again is just that three by zero and
then here we're just gonna pick a random
choice that's it so zero one or two a
random option between those three
options will be a one and then boom we
append that value now if all else fails
is so if basically if there was no
change but we don't have more ships and
we're supposed to have then we'll just
continue with whatever the plan for the
ship was when we go to add a neural
network and once we've trained it the
only difference is going to be basically
this code here it's just the neural
network we'll choose the output vector
rather than the output vector being
random okay now we actually have to make
the choice so copy this
I think that's where I want it so if the
if the Arg max of the output vector is
the zeroth element so again we've
already talked about it but that would
mean this so if you did an NP an umpire
Arg max its it's trying to find the
element the index basically with the the
maximum value in this case it's the
zeroth element so if the Arg max is zero
that means we want to attack an enemy
ship so we're going to attack we're
gonna try to find the closest enemy ship
now there's going to be times when well
at least on the enemy ship if there's
zero enemy ships uh the game is over so
this probably isn't necessary I don't
know if it goes one more frame after the
because you the last thing you'd want to
do is like win the game but time out
because you made an error so anyway I
would still leave that there but at
least like on the planets for example we
need to make sure that the planets
aren't that kind of integer value as
when we go to pad them but anyway so so
yeah if if that was the ark max then we
attack now we're gonna do the Minor
Planet and this one's actually a little
longer because it needs we need to have
like something to fall back on just in
case we make a mistake so paste okay and
this is why we're not writing all this
out so if the ark max is one what do we
want to do well that means we want to
mine one of our owned planets so then
we're gonna check if that planet is an
entity and not an int that we just used
a pad then the target will be the
closest planet and then we have to make
this check so we have to ask if that
planets if the number so the length of
the dock ship IDs is less than the
number of docking spots that are
available then we can dock but there's
going to be times when that planet is
full we can't actually dock
that planet so when that's the case else
attack so we just go on the attack one
option to consider moving forward to is
if this is to happen you might want to
change your random vector so because
this wasn't actually an option right and
so any AI that learns what we can try to
mine but then we just fall back on
attack really that should be in a an
attack option not a proper chose to mine
option anyway and then we attack so
looks like I copied too much there if no
anyway so if if our closest planet
there's more here than I thought yeah if
if our closest planet is if we can mine
it then we mine it otherwise we're going
to attack but if this turns out to not
be like if the closest owned planet if
we don't have a closest owned planet
then we go and fall back to the closest
empty planet and then we attempt to do
the closest empty planet but if that
fails then we attack so we just have
these kind of fall back so you have two
options here you could one just change
your vector choice if you fall back on
one of these fail-safes or you could not
have any of the fail-safes
and then and then hopefully your model
might be a little more accurate or
something like that but I'm gonna have
them there then finally if if we get
that kind of third option let me make
sure I get the right amount of tabs
should be one two three four tabs over
one two three four tabs over paste and
just for the record if you go and copy
and paste the code from Python program
to net like at the bottom is the full
script so you don't have to do these
chunks like this so now if the arc max
of the output vector is the is-2 meaning
the third option we're gonna mine the
empty planet and this is basically the
same kind of logic first it tries to
mine me at closest empty planet if that
fails for whatever reason we're just
gonna go and attack okay and then
finally we just kind of end and the code
paste in the end and let me make sure we
don't have any syntax error here okay at
least when I have a syntax error so what
I'm gonna do now is we're gonna go to
the the next tutorial which will be
taking this bot will make its second
version to you
and then we're gonna write code that
will basically have these two a eyes
compete against each other and then
we'll take the winner save that data and
repeat and basically what we want to do
is write a script that will just
automatically run these games locally
like a bunch of times
saving whoever the winner was and all
that so that is what we're gonna be
doing in the next tutorial if you have
questions comments concerns whatever
feel free to leave them below if you've
got suggestions about better factors to
use or any of the factors that we use
that don't really make any sense or
whatever feel free to let us know below
otherwise I will see you in the next
tutorial