what is going on everybody and welcome
to part nine I think of the deep
learning with the halite AI competition
3 2018 all right let's go so hope you
feel working on is doing machine
learning with halide 3 and at least
where we left off we've we've got some
code here and we figured out how to at
least save the you know the frame
basically from the game at least of the
information that we're curious in we
visualized it we saw okay this is
looking like what we want except for the
flippy stuff but then we figured out
what's going on there now we're ready to
actually create and save the data so we
need to come up with some sort of
principle to why we would want to save
or not some data now historically I've
done this before with like an
evolutionary based model and if you just
go off of let's say the best between
your model that allows for the model to
actually get worse over time so if you
train a model and that new model
actually sucks for whatever reason maybe
it over fit or whatever and then you go
back and start playing some games you
you're still gonna have winning games
but actually let's say the total halite
collected or whatever is gonna be worse
than before and we don't want that so
instead what I think I would rather go
with is rather than an evolutionary
based model as in you know who wins
instead what we should do is go off of
some sort of threshold of halite that's
what this entire game is about anyways
it's all about the halite so so that's
what I'd like to do now a couple of
things like so for example it's a little
challenging to for the by like the bots
never actually knows when the game is
over because of the way halite is set up
and it's kind of unfortunate
but we can know in advance how many
turns there will be given a game size so
for example let me just zoom in a little
bit here and I guess what I'll do is
I'll just make because later we could
query so the bots can query the game map
and then we can get the size of the game
map which will always be square and then
what we can do is have some sort of
variable here called like map settings
that'll be a dictionary and then let's
say the map is a 32 by 32 that means
there's going to be 400 turns then we
can just continue this so at 40 a 40 by
40 that means it's gonna be 425 we have
a 48 oops not 49 48 which is gonna be a
450 wow I can't type I hope this doesn't
turn out like yesterday's videos anyways
56 becomes a 4 7 2 5 and finally a 64 is
500 turns so we could query the game map
and we can know okay how many turns are
there this way on the last turn we could
query and say how much highlight do we
have doesn't meet a certain threshold
yay nay if it does
now we'll save the data so in theory all
of the bots in any given game could hit
that threshold and that would be good
now there's a couple of downsides with
this one is each map is seeded randomly
so some maps if you just look at replays
on like halide io some maps have an
insane abundance of halide everywhere
especially close to the to the drop off
so these games are likely to produce
higher high light numbers because ships
didn't have to travel as far to get the
bulk of the halite also it allows you to
create many more ships immediately so
this is somewhat problematic so you know
maybe there should be some sort of
combination of winter and halide or
maybe ships produced halite or even
better some sort of maybe combination of
of like a percentage of halite and i
think you know like the at least on
holiday oh when they visualize it they
do say what percentage of the halite was
collected and I think instead probably
eventually we'll go that route and get
like just go based on a threshold of
percent collected rather than halide
collected because that might wind up
confusing a model but for now what I'd
like to do is continue on this little
trick that we've got going on so first
of all we want to set some sort of
minimum threshold so I'm gonna say
Thresh hold and let's just say I don't
know a thousand how light I don't think
we're gonna be able to collect a
thousand how light because it's gonna be
random but I just want to set it there
just to have a variable there and then
and in fact let's call this save save
threshold so it will require a thousand
halite also just for now I'm gonna throw
in another variable called total turns
so for now we're gonna limit it to 50
turns because when you call the
highlights you knows command line
interface thing there is a flag for no
Tarin
I think it's turn - limit and then you
could say 50
let me check real quick and make sure
that is true
I think it's tournament yeah turn its
just yet - - turn limit and then you can
set so it doesn't matter the game size
turn limit is 50 now why do we want to
do that we'd like to do that because the
first models are going to be totally
random
so the quote-unquote reason why one AI
was able to collect a thousand how light
let's say and the other one not when
both of them move 100% randomly is maybe
one move out of all of the moves that
was made by any one of the ships so if
we allow this to go for 500 three
in turns the results are probably it's
gonna be almost untrainable because the
little kernel of what the AI did good
during five hundred turns is going to be
you know obstructed by insane amounts of
noise so instead if we can lower the
total turn number and then maybe lower
the threshold like fifty turns to
collect a thousand how light it's gonna
be quite challenging but if we can do
that that would be beneficial also
probably safe threshold maybe we make
that well because we're gonna I can't
decide what I want to do if I want to
produce all of the because you'll
start with five thousand halite so you
know you might want to even save the
threshold is six thousand or you know
something like that I can't really
decide what I want to do there because
we're only gonna make one chip so
actually not six thousand but you'll
start with five thousand you'll make one
ship you'll have four thousand so let's
say anything about four thousand is good
that means we at least dropped off
something so in theory we could say the
safe threshold is five thousand if we
want to collect a thousand anyway we can
we can hash that one out in a little bit
but I'll leave it at five thousand for
now so okay so we've got our safe
threshold we've got our total number of
turns now what we want to do is let's
add a max max I keep forgetting my caps
lock doesn't work Mac ships equals let's
go with one for now again over time we'd
like more ships but just like I was
saying before with the total turns let's
just do one ship and because this model
is a ship only model so it's a very
micro model so we'd like first to train
a single ship to do a smart thing just
to see if that can work this would be
the easiest version of this challenge
eventually we do need the model to take
the other ships into account so we do
need a model that's training on the
presence of other friendly ships that
way if a friendly ship is closer to a
bigger plot a highlight we don't waste
our time trying to go to that spot
because our other ships probably doing
the same thing
okay so max ships one let's go ahead and
scroll down and fix that so less than
max ships okay so now what we want to do
is we will also rather than NP dot save
here what I'd like to do is I'm gonna
comment that out we're gonna come over
here and let's make a new directory and
I'm going to call this not in all caps
I'm gonna call this training underscore
data so this is where all our training
data is gonna go if we meet a certain
threshold so I'm gonna throw another if
statement down here and we're gonna say
if game dot turn number equals what was
the thing Mac turns total turns that's
gotta be it total turns so if that's the
case okay so if we are on the final turn
because again our AI does not get
information from the game that the game
is over so we have no way of knowing but
we do have a way of knowing what is the
final turn and then we can kind of
surmise okay okay that's the end of the
game so so if game dot turn number
equals the total turns what do we want
to do well if me dot halite amount is
greater than or equal to that save
threshold that we set then we want to
save so let's say NP dot save whoops and
then where do we want to save it let's
make this an F string again training
data right training data training data
yeah
training data slash and then let's save
it let's save it by two variables let's
say the halite amount because later we
might we might set an initial threshold
and then suddenly we've got like a
freaking ton of games and then we still
might want to like somehow set some sort
of limit there so for now let's just say
me dot highlight amount so that'll be
the first segment of this file name so
then later we could go in
via Python and os Lister or something
and then we could filter by that first
thing with just a simple split string by
- and then get the zero with anyways
meet a high light amount and then let's
get a int time time and that should be
fine but let's go ahead and go down to
that like millisecond just to be certain
that it's a unique file we don't
overwrite some other file but that I
really think the chance that would be
pretty much none but okay and then npy
so also I'm not sure I've brought in
time I have not
so we shall import time now later total
turns like once we don't set a turn
limit total turns would be equal to like
you price set total turns somewhere down
here like around line 39 we would do Len
I think you have game map dot width and
game map dot height or maybe it's dot
size we'll we'll figure that out so you
get that number and then you would say
total turns equals that you know map
settings that number with the key so or
as the key but in this case we're gonna
hard code 50 for now because we want the
game to stop at 50 okay
impedance a Papa Papa I think we're all
set so let's go ahead and actually let
me change that safe threshold I just I
feel like I feel dirty asking for that
let's just say the safe threshold is
4050 so anything I don't even know I
just don't know how many will collect
that random so we'll just save it at
4,000 so anything I suppose do 4001
anything any drop off we are happy with
okay and then okay so your scepter in
order so they're already moved randomly
am I forgetting anything else or are we
ready to run it I think we're ready to
rumble so let's go into the run game
okay so now we don't want two versions
this won't cause that same problem as
before this the problem before was that
like overwrite because we were using
like the same directory whereas in this
case it'll create its own little unique
file okay so run game I kind of got rid
of it okay so this is the command to run
the game now we could just run one game
really quickly like let's make sure it
works would be nice and also actually
let me just throw in replay directory
let's just throw in a - - turn - limit
50 okay copy
save break that run it again let's see
what happens if we hit any errors okay
looks good
zero highlight was collected let's run
it again still zero how light this could
get ugly man no hell I really I figured
like with random it can't be that hard
to accidentally go left than right or
you know left right right
oh wait oh no no um
we're hitting an error first of all and
because of that error we killed the
player I was missing that so we passed
the filename but not the array okay okay
okay
what are we saving here surroundings
surroundings yeah try again there we go
we got some halite look at that okay
4034 let's just run this a couple times
that one got zero extra halite okay
we've got like third no 28 and six
ooh 4099 40 83 it looks like a hundred
would not be a super challenging
threshold to hit we want something
that's challenging something that's
relatively difficult maybe a hundred
maybe we'll go with a hundred yeah so
we've definitely surpassed 100 there but
I I kind of want to go to that second to
last replay so not this one but this one
let's let's look at that because I'm
gonna bet there's like a lot of
highlight close but at least while it's
random it may not be the case it might
just be that we got very randomly lucky
but let's just see Oh what happened
I just resized it are you kidding me oh
no what now really what happened oh my
gosh see if I can do this I don't think
I can can i I thought that would give me
the last one no that is the last one
okay so at least yes in this case there
are some close clusters but that's not
why he did good he just went down down
collected some serious highlight there
oh but he's already had it at that point
or so it was the green one
oops I was looking at the red guy so he
just goes collects and happens to just
come up and then I think he must make
another deposit he does yay for a random
okay so yeah that's you know we wish you
would have collected a little more
halite but hey this is random and in 50
steps only hey that's a pretty good a
pretty good AI I mean that our that's a
pretty good you know better than random
thing we'd like to mimic so cool so
that's that is what we want so I'm gonna
say 40 100 will be our threshold and
then save that and then rather than us
sitting here manually we could do like a
while loop like I think you can do like
what is it while true maybe with a
capital T here
while true wow dude
true and then you do the thing and then
you do it done I think that works but
I'm not gonna do that especially for our
sad Linux users instead hey we can make
a Python script so let me let's
copy-paste let's call this run game and
we will open with sublime text and we
will get rid of everything and then what
we want to do is we want to simply run a
command now try to get let's see run cuz
that let's see we probably could get
away with OS dot system should run the
command so let's do import OS we will
import secrets
let's also bring in we'll set a max
turns just to set them for now we'll say
50 and then we want to have both to
player and for player games also we do
want to have varying map size so
actually let's go ahead and copy map
settings or your paste and then we'll
just set a while true loop here and then
first of all let's say map size equals
secrets dot chase map settings print map
size I just want to make sure that's
gonna do what we intended no key error -
why
I don't know okay kind do it might be
keys or get keys or something really I
feel like that should be an acceptable
option what is it is it get keys trying
to remember dot get keys maybe has no
attribute kick so it may be keys
oh so bro why is this so difficult
get list of keys from dictionary pi
thank you sir
yeah dot keys
okay so secrets must be trying to index
it so probably what we could say then is
list keys okay hey look at us
hacker man okay so that's out of the way
hey guys check out my basics Python
series now that that's out of the way
we've got our map size we've got our max
turns so the next thing will be
basically what we want to do is call
this turn limit so let's copy this let's
say commands we'll make it a list and we
basically two versions of this command
and in fact we need to we'll get away
with just putting it in single quotes
single quote so that's one of the
options the other option would be a four
player game so copy come down here paste
and then we want why did that happen
we want basically this double so to four
players boom done okay so now turn limit
oh this needs to be an F string F string
dude okay then what we want to say is
turn limit shall be max turns I really
should have done this before we copied
it yeah that would have made more sense
for D becomes what is our map size map
size so then I'll just copy paste paste
paste and then we run Oh si no no no
actually yeah we can do this uh-oh we'll
say code so the command that we want to
run ray come
and that we want to run as commands dot
or no secrets secrets choice commands
and then let's just run OS that system
command I'm trying to recall if OS that
systems gonna work anywhere
I think OS that system will work even on
like Linux and Mac and all that to write
like I'm pretty sure that that's like a
yeah so I we should get away with OS
system on any operating system if not
check out like sub process would
probably be the next best thing but this
should be fine
so OSS system command and then let's
just do like ten so for I in range 10
let's print I just want to make sure
it's working so let's come into here I
want to delete all of our replays begun
and then I want to go into train trips
not gameplay actually we could just
delete game play we're not using that
anymore training data will delete all
that and let's begin so let's say Python
run game dot pi well that's a lot of
output it looks like we're running a
game though for sure we getting any
training data Wow 43 what wow we got oh
my gosh that's a lot of Hal I bro okay
alright that appears to work so rather
than for I in range 10 we will say wow
true we don't need this anymore okay
close these
this so then what we're going to do
let's close this
oops we need to back up one mmm
Python run game PI I am not in the mood
today
alright and we're off okay
yes sir so now we can you know you could
be running this one and then we could
pop up into here
maybe run another one Python run Python
run game okay and now we are making
games I'm hoping that that's not gonna
like we might want to like shut up the
print statements or like the output I
guess it's actually I believe the output
is actually is it standard I think it's
like STD air that's coming out but I
can't remember but again you can
probably use so process and shut those
up but for now wow we've got a lot of
training Wow 4400 almost 4500 so keep in
mind so we already know each of these is
50 steps or 50 turns we might even
decrease the amount of turns but if we
go to calculator hello you don't have a
calculator okay whatever
each one is 50 though so like let's say
we had you know the target is generally
a hundred thousand and then we want to
divide by 50 that would mean our target
should be approximately 2000 games to
give us a hundred thousand steps of
samples so at some point like I would
just leave this running like overnight
all day whatever run more games
simultaneously however many you can do I
would but this kind of a CPU hog
anyway run a bunch of these gather a
bunch of training samples
about 2,000 hopefully we can get about
2,000 samples you know above like I
don't know 4200 4,300 that'd be pretty
nice look at this guy almost 5000 he's
moving randomly it's one ship in 50
turns a thousand halide that is awesome
okay anyway so I would let these run for
a while and I think I'm gonna stop it
here and then pick up in the next
tutorial where we will take the best of
the best train a model and then
basically we begin our infinitely
iteratively iteratively infinitely
iterative loop of you know train a model
on the top halide threshold or plausibly
some other rules we set like a
percentage how light collected is it's
probably the more superior route so
maybe that it will be the better way to
go anyway set some sort of threshold to
save by and just over time slowly
increment that threshold train a model
that met those criteria increment the
threshold play a bunch of games train a
model increment the threshold train play
a bunch of games anyway repeat that
process hopefully you understand anyway
because I don't I don't know what I'm
doing that's it for now questions
comments concerns whatever yeah feel
free to leave them below
also there's discord channel discord GG
so lash cent Dex will get you there
otherwise I will see you guys later in
another video will retrain an amazing
model built completely off of random
movements see you there
actually hold on it just clicked I want
to make absolutely certain before I
leave this that our size how big were we
working where is it Oh 16 because I want
to be able to use the like mobile net
and I think Inception v3 and those
things there minimum size is 32 so we'd
have to pad and
okay also sorry I know I said we were
leaving I feel bad for the people who
like click off at the end some videos
and then they we get back and there's
like news things they've never seen
let's go let's call this rate I hate to
call it radius it's not really radius I
don't know sight distance equals 16 copy
that
come down here paste good save
I kind of want to confirm that that
worked so I'm gonna break break run game
cool
also just out of curiosity how many
training files do we have yet 197 so it
definitely should not be challenging to
meet the 4000 threshold I think that was
our threshold right 4,000 oh no 4100 so
definitely not a challenge to meet that
it only take like I don't know 20
minutes probably
so maybe 4243 45 somewhere around there
we will set a limit anyway yep clearly
working cool all right I will see you
guys in the next video just kidding
we really need to stop meeting like this
but of course that is not gonna work so
so I was really looking over all this
stuff and I realized I've made an
egregious mistake and what we've done is
basically we're just we've made a couple
of mistakes one is we're not saving the
entire games worth of data we're only
saving the final surrounding image also
we are not saving the move associated so
we need to do two things we need to save
the entire game and every move
associated with every frame so as sucky
as that was forget that let me just add
that in really quickly so basically what
we're gonna do is we actually kind of I
thought I had that I did okay so we'll
take this direction or
that actually doesn't need to be where
it is so I'm gonna put it up here so no
need to have it defined every time it's
the same then what we need is we're
gonna have a training underscore data
and this is what's gonna contain every
single frame every single move for the
entire game and if we meet a certain
threshold that's what we want to
actually save so then we're gonna come
all the way down to where where we
actually make our move so in this case
append ship move secrets okay
so what we really want to say is we want
to say something like choice equals
secrets dot chase and then let's say for
the range Len of Direction orders so
basically it's going to be a zero one
two three or four five options for this
prediction so that's going to be the
choice that we make and I'm trying to
think I don't think that's going to
cause trouble later I'm a little curious
due to the way that we're gonna actually
train this model if it would make a
difference I don't think it will but we
can chant train later but anyways with
with TF learn you can train with scalars
rather than one hot vectors you can pass
those in I think behind the scenes it
just does like a quick operation to
convert it to a one hot for you I don't
think it actually matters that much but
anyways we'll say that so that's the
choice that we want to make then what we
wanted to say is a frozen okay thanks
virtual machine hopefully it'll come
back basically what we want to say is we
want to append our the surrounding data
and that specific move to training data
and then at the very end that's when we
want to save it hello all right as soon
as I stop the recording it comes back
awesome okay training data then we want
to append
surroundings and the choice that we make
and hey let's make it Pepe okay so then
what we want to do is so in this case
here we actually want to say Direction
order choice instead so move Direction
underscore order choice and then at the
very end down here if we want to save it
what we want to save is training data
rather than just the surroundings okay
let's save that let's come back up into
one of these also I just renamed this
old this is now an empty directory rerun
it hopefully everything works
immediately at least we've got a game
cool and now let me just copy this come
in to back over here let me go into
testing grounds sublime text import
numpy is NP n equals NP load and then
let's print n zero go oops
what we want is training data slash you
little okay
next there we go okay so now we have the
visual now we have the choice okay now
this is the data that we want to train
against unfortunately I had actually
built quite large
I had it running for like six hours I
came back and then it like hit me and I
was like oh my gosh I don't think I I'm
I was like I remembered that I probably
saved his surroundings rather than
actually built the training data with
targets and all of it
it was quite the heart sinking moment
okay anyways now we are going to build
the training data it is going to
hopefully not take too too long wow look
at this we've almost already got a 5k
game incredible anyway now I will see
you in the next video