[MUSIC PLAYING]
DUSTIN KIRKLAND: Hello, my name is Dustin Kirkland,
I'm a product manager at Google, and I'm
joined by two of my colleagues: Andrew Jackson, a software
engineer at Google, and Josh Hoke, also
a software engineer at Google.
They are going to tell us about a pretty exciting model being
trained on top of Kubernetes.
And they're currently training another Go AI; an AI that's
learning to play the game Go.
All right, give us a little bit about this project,
and how it got started.
ANDREW JACKSON: Sure, OK.
So, DeepMind is a different Alphabet company
based in London, and they're mostly
a research-focused company that wrote this paper that
was published in Nature about mastering
the game of Go using tree-search and deep neural networks.
And this was sort of a revelation,
because beating professional Go players has
been a longstanding goal in artificial intelligence.
It has been a long time in coming, whereas Deep Blue was,
what, 30 years ago?
Was it 30 years ago?
DUSTIN KIRKLAND: Close to, 90s, yeah.
ANDREW JACKSON: Long time, yeah.
DUSTIN KIRKLAND: [INAUDIBLE]
ANDREW JACKSON: 20, 30 years ago?
Go has seemed a really long way away.
And then out of nowhere, DeepMind's like,
"oh, by the way, we did that."
And they published this paper, and it
was kind of a revelation.
And then later the next year, I think
they played world champion Lee Sedal in Seoul.
They published another paper where
they said that they didn't just train a network
to beat a professional Go player,
they actually trained it with no human input whatsoever;
that it started from essentially random noise,
and learned how to play by itself.
JOSH HOKE: And what was crazy is that this Go AI was better.
ANDREW JACKSON: It was better than the first one.
JOSH HOKE: It was better, it was fantastic.
[INAUDIBLE]
DUSTIN KIRKLAND: That's fantastic.
OK, so that's the DeepMind team.
Now, this is a different group at Google
that you're representing here.
ANDREW JACKSON: We're actually not a group at Google at all.
So this is not a group at Google,
and this is a side project.
This is an enthusiast project, because--
JOSH HOKE: 20% project.
ANDREW JACKSON: We're Go players who love the game Go,
and we really wanted to have a chance to play against AlphaGo.
But of course we couldn't, so we had to sort of rewrite it.
And so Minigo is an open source, ground-up
rewrite- I shouldn't even call it a rewrite-
it's a implementation based on their published work.
It shares no code, we're not related to that team at all.
We have no relationship with that team
whatsoever, other than the fact that we're
able to implement their published papers
and try and reproduce their results.
So this is sort of Interesting from a reproducibility
of results perspective, and also that this is an open source
demonstration.
They had built a really fast performance solution
that was really baked into the Google stuff, as we assume.
And so what we wanted to do was, we
wanted to provide an excellent example of how you could
use the Google Cloud platform, and really amplify
and leverage some very limited developer resources
to try and achieve some really amazing results.
DUSTIN KIRKLAND: OK so, this is an open source project
built on your 20% time at Google,
our Friday afternoons where we can work on fun things related
to the field.
Now tie this to KubeCon, we here at KubeCon in Copenhagen,
and you guys just gave a talk?
JOSH HOKE: Absolutely.
So, we just gave a talk, because the whole way that we trained,
the whole way we got data, was with Kubernetes.
We had this problem where we needed
to create hundreds, thousands, actually millions of games
in order to get the sort of strong model.
So what did we do?
We used Kubernetes.
We used machine learning.
ANDREW JACKSON: So what did we do with her?
JOSH HOKE: Machine learning, that's right.
ANDREW JACKSON: Machine learning.
DUSTIN KIRKLAND: OK, so--
ANDREW JACKSON: It's kind of a punch line,
but I think the important part here
is that this is reinforcement learning, where
the algorithm has to create its own data to train on.
And that creation of data is a really expensive part
of the overall process.
So, in other reinforcement learning applications
like robotics, running these simulations
that produce the next iteration of data
is really computationally expensive,
whereas for the Go domain- for the domain of playing Go-
that can be very easily parrallelized
in a very natural way.
DUSTIN KIRKLAND: So the parallelization- now,
you guys ran a huge model to train your implementation-
and you used Kubernetes to orchestrate
the playing of the games.
Is that right, more or less?
JOSH HOKE: That's right.
So we use Kubernetes- we actually use GKE-
and we created a giant, giant cluster.
And not only--
DUSTIN KIRKLAND: Define giant.
JOSH HOKE: I think we were running, what
do you say, like 2000 nodes?
ANDREW JACKSON: 2,000 GPUs.
JOSH HOKE: Right so, yeah--
DUSTIN KIRKLAND: GPU instances?
JOSH HOKE: Right.
So not only were we using a whole bunch of machines,
we were using a whole bunch of GPUs to train our models.
ANDREW JACKSON: Right.
DUSTIN KIRKLAND: And this problem
is particularly well suited for CUDA and GPUs, right?
JOSH HOKE: Absolutely.
And so, the reason why it's suited to GPUs
is because we're running things through machine learning
models, through neural networks.
It's just math- just matrices- and the thing that GPUs
are really good at is actually running things
through matrices.
DUSTIN KIRKLAND: OK, so each game
that was simulated- some 20,000 games that
were simulated- each of those ran in a Kubernetes pod
as a container?
JOSH HOKE: That's right.
And we scheduled out all these pods, all
with the communities batch API, the job API,
and we set the parallelism way up,
and Kubernetes just took care of scheduling
on all these machines.
It was fantastic.
DUSTIN KIRKLAND: Now can I docker pull this image
and, I don't know, run a game simulation for fun
on my laptop, obviously not GPU accelerated?
ANDREW JACKSON: Absolutely.
If you have the GPU accelerated version
you can do that as well.
That will be a different image, but for sure, we
have these available.
You can pull down the image.
We actually have a UI attached to that image,
so you can actually watch it play, and see
it think as you do that.
That's actually our demo that we have.
DUSTIN KIRKLAND: I'd love to see that.
So how long does playing a game take a simulation?
ANDREW JACKSON: Just a couple of minutes.
JOSH HOKE: Just a couple minutes.
If you don't have a GPU, it can take quite a long time.
ANDREW JACKSON: But it depends how much you want it to think.
JOSH HOKE: That's right.
ANDREW JACKSON: That's the real question.
You want it to think a while, it could be done.
You want it to think a long time,
it would take a long time.
You want it to think very quickly, won't take as long.
It's your choice.
JOSH HOKE: But with GPUs, maybe a couple of minutes for a game.
DUSTIN KIRKLAND: Couple of minutes a game.
ANDREW JACKSON: Almost all of the load
is taken care of by the GPU, there's
very little work that's actually done by the processor
if you're using GPU acceleration.
DUSTIN KIRKLAND: All right, fantastic.
I would love to see your demo.
ANDREW JACKSON: You got it.
DUSTIN KIRKLAND: Talk us through this.
ANDREW JACKSON: All right, so what we're seeing here
is we are seeing- well, what we'll see in a few seconds,
I guess I should wait.
What we're seeing here is we're going to see Minigo think.
It's going to be trying to figure out
what moves will be played.
So on the left side, we see the principal variation,
and it just decided that it wants to play this move.
So now it's going to be my turn, I'm gonna play there,
and then we're going to see Minigo think again.
So I decided to play there, and Minigo is thinking,
white will play one then black will play two,
then white will play three.
And as those numbers move around,
it's changing its mind about what
it thinks is the best outcome for both sides.
The other graphs that we see here,
so this board up here at the top is
the actual inferences that are happening on the model
right now.
So each of those is 1/20 of the total number of possible board
positions that it's looking at.
So just like how a human would play the game when
we try to think ahead and anticipate what's
going to happen, and then, by visualizing what will happen,
we then try and decide whether or not
we like where we're going.
Minigo is doing exactly the same thing.
It's trying to read ahead what the best choices are
for either person, and essentially
figure out whether it likes where the game is developing.
So that's what the top one is, that's
each of those inferences.
The middle one is all of the different moves
that it's exploring.
The bottom one is the change in score for the given play-out.
This graph that you see in the upper right-hand corner
is how it thinks the game is going for each side.
As it goes up, it thinks one side
is winning, as it's going down, it
thinks the other side is winning.
And so, all of that de-bug information is all there.
JOSH HOKE: Yet I should say, one of the key things
is, of course Minigo things much faster than we do.
And that's why we're only sampling 1/20.
ANDREW JACKSON: We're only sampling
1/20 of its table positions.
JOSH HOKE: Its positions, that's right.
ANDREW JACKSON: So, as Go players,
this is really interesting for us
to watch and see what it's thinking of when it sees things
that we maybe don't think of, or moves
that we wouldn't think of.
And we can also see what it thinks the continuation is,
what will happen after that.
So this is a really great study tool, and it's useful for us
as Go players.
DUSTIN KIRKLAND: And I should say,
one thing that's super fascinating
is, I look at this game and I'm like,
this is something that I would play.
This actually looks like a very human game to me.
JOSH HOKE: That's right.
This is a smaller model, we should mention.
This is a nine-by-nine game of Go.
The real game of Go is actually four times as big as this,
but this is a smaller version that is easier for us
to run locally, and essentially give a demo on a laptop
instead of on a big beefy GPU machine.
So this is a smaller version, it's a smaller model as well.
But as a result, that doesn't really matter.
We've got full sized models as well,
you can download those also off of our bucket
with all of our data.
DUSTIN KIRKLAND: Are you ready to take on a human?
ANDREW JACKSON: Yeah, we've taken on humans.
This has been playing in public against anybody who
wants to play it on Go servers.
It's still in development, and we're
hoping that we can replicate the results,
and then also start to explore what else can be done
with it, what can be added, what other sorts of options
there are.
JOSH HOKE: But there's definitely a lot to improve.
ANDREW JACKSON: A lot to do.
JOSH HOKE: There's always bugs to figure out,
we're always making it better.
ANDREW JACKSON: If you want to find us,
this is all on GitHub, right under the TensorFlow
repository.
So github.com/tensorflow/minigo, we'll take any sort of pull
requests, comments, issues, complaints,
conspiracy theories, whole thing.
DUSTIN KIRKLAND: What a fascinating, fascinating
project.
Just tying this back to Kubernetes one last bit.
So, Kubernetes provided you some infrastructure underneath.
Without a Kubernetes,
What were your options, I guess?
ANDREW JACKSON: So, after I had finished
the initial implementation of this,
I realized that the hard part wasn't actually implementing
the neural network.
The hard part was figuring out how
to do this huge amount of self-play and create the data.
It's not just 20,000 games, it's 20,000 games per model,
and you need to do 500 models, maybe.
So it's a huge amount of this compute
that I needed to figure out how was I
going to do that when each game takes minutes to play.
And luckily I knew a guy who worked on Kubernetes,
and I was able to say, hey Josh, how should I do this?
Should I like, script a bunch of bash to manage VMs directly?
JOSH HOKE: Totally.
Bash is the right option.
ANDREW JACKSON: Oh man, I had all of these other,
like, how am I going to get this at scale
with only the open source tools?
Because I didn't have any of the Google infrastructure
to lean on.
DUSTIN KIRKLAND: I guess that's the other piece, right,
the infrastructure.
You didn't have to provision a single VM,
you didn't have to find a GPU.
ANDREW JACKSON: I did have to do a lot of work
in getting the code to work inside a docker image,
because this involved making those docker
images is very performant.
Like, Docker's abstraction is, well,
don't worry about what the actual silicon is underneath.
But TensorFlow and GPUs were very much
trying to compile with all of the processor optimizations,
we're trying to compile with the latest NVIDIA drivers.
Getting all of that to work inside the docker image
was non-trivial, but it's also gotten a lot easier.
I think in particular with the launch of GPUs
on GKE, that whole situation with getting the latest
drivers, getting CUDA DNN installed,
that's all come a long way.
And even when we started this, it was only,
I guess six months ago.
It's come a long way, just since then.
So it's pretty exciting--
JOSH HOKE: Super exciting.
ANDREW JACKSON: To see that sort of progress.
But that hard part was getting that performance
inside those containers.
So it is interesting to see that the containers are
no barrier to this sort of high performance computing.
If you want to use GPUs and go as fast as you can,
you want to use all of the extended processor
instructions, you can do that too in your instances
with is a little bit of care in how you craft them.
So I think that's a pretty interesting.
DUSTIN KIRKLAND: That's fantastic.
Josh, Andrew, thank you for your time.
JOSH HOKE: Thank you.
ANDREW JACKSON: Thank you.
[MUSIC PLAYING]