[MUSIC PLAYING]
DANIEL VISENTIN: So as Zak said, I'm
from DeepMind's applied group, where
I work on a team that applies DeepMind's research to machine
learning and reinforcement learning--
DeepMind's research in machine learning and reinforcement
learning to Google's products.
Today I'm going to briefly talk to you
about why we chose to use TensorFlow, how we use it,
and how it's helped us.
So the choice of a numerical computing
platform such as TensorFlow is incredibly important to us.
It's how we express our ideas, and it's the piece of software
our engineers and scientists spend most
of their time interacting with.
The quality of that platform has a direct and significant effect
on the quality of our work and the speed with which we
can deliver results.
And there are several aspects to what we look for in that.
So from flexibility, our researches
continue to concoct all these weird and wonderful networks,
which we're hoping to put into use.
Usability-- using this platform has to be easy and natural.
Simple things like getting some embeddings
and passing through a feed-forward network
have to be trivial.
Composing things should be easy.
We should be able to take all these components which
people are developing and knock them together
to create new things.
We need to be able to reach into the guts of that network
to see what it's doing, and we need tools for visualizing
and debugging what's going on.
The platform also has to be scalable.
We see this move towards increasingly more data,
increasingly larger networks trained
on more powerful accelerators across multiple servers.
And the more that trend continues, the better.
We need to be performant.
So training machine learning models
is the new compile link cycle.
It often takes days or even weeks for us
to get results, which reduces the speed with which we
can iterate and try new things.
So the faster it goes, the faster we can develop.
It's also important at serve time,
where latency is a critical factor in what we can do.
We often have quite tight latency boundaries.
The more performant the model is,
the more expressive and more computationally powerful
models we can use, which will give us bigger accuracy gains.
Also with training, performance matters,
because when we're training on real data at serve time,
the speed with which we can train
affects the freshness of the model, which
often has downstream effects on our accuracy at serve time.
And finally, production readiness--
we often spend all this time doing research, and evaluation,
and experimentation of our models.
And when we finally come up with a good thing,
we want to get it into production as soon
as possible to do live experiments
and ultimately scale it up to all the traffic that we have.
So previously at DeepMind, our projects
were all implemented in either Torch or Disbelief.
As TensorFlow began to mature and head
towards its first open source release,
we started looking at it as an interesting thing.
In the applied group, we began experimenting with it
by trying a few new projects.
Meanwhile, the research team, who
have this very stable, much used code
base they needed to think about, began
by carefully evaluating the suitability by reimplementing
several existing projects.
Based on the results of that, we decided
that TensorFlow was the way forward,
and so we started to migrate.
So over the past year or so, we've
been working on that process.
And as of today, most of our code that we currently use
is in TensorFlow.
And all the stuff--
the significant amount of stuff we're building from now on
will be in TensorFlow.
What's quite cool about this is we made this decision a year,
a year and a half ago, and since then, the TensorFlow
and the ecosystem around it has improved
beyond our expectations.
So we're even more happy with this decision
now than we were then.
So I'm going to spend the rest of my time talking
a bit more specifically about how we use TensorFlow
and how it helps us.
And since I'm from the applied group, I'll start there.
So one of the coolest things we've done recently
is on optimizing the energy usage in Google's data centers.
This began when the technical infrastructure team came to us
after seeing the AlphaGo matches and wondered
how we could use RL to control the data centers more
efficiently.
So we work with them, and subsequently worked
with teams in Google Brain to build models to control
the cooling infrastructure.
Briefly, this is what a cold aisle in one of Google's data
centers looks like.
This is the servers.
Some of them, maybe, are running TensorFlow,
and they're getting hot doing so.
So we have these fans at the back, which
pull that cold air over the servers
into this hot aisle, where it travels down and is transferred
into a water cooling system.
So here are some pipes and some pumps.
And the pipes take the hot water from the hot aisle
to the chillers on the roof, and the cold water from those
chillers back to the cold aisle.
And here's what those cooling towers look like.
They're just big fans which take the water
and cool it down into the air.
So we spent a while working on this offline.
We had to take into account safety.
The model couldn't go and try random things,
otherwise we'd break billion-dollar data centers,
which is never great.
And we had to work out how to do a bit of exploration
as well, because if we just learned on the data
which we had available, the model would probably
never learn to do anything more optimal
than what the human operators had done.
So we iterated a bit, and when it eventually
came time to experiment, we turned it on,
and we saw this nice drop.
>From my experience, it's kind of rare to see something
so defined and sustained like this,
so we're quite happy with it.
So we had this initial result, and it came time
to implement it.
Because the data centers didn't really
have the infrastructure set up to run, and train,
and evaluate, and verify machine learning models,
we had to write a lot of this ourselves.
So we did that.
We took our original Torch models
and moved them to TensorFlow.
We built up all this training, and validation,
and serving stuff.
But over the past year, what we've
witnessed the development in these high-level APIs
around TensorFlow and components like TensorFlow Serving, which
basically negate a lot of the work that we had to do,
which from our perspective is great because we'd rather
focus on the modeling aspects than all this kind
of infrastructure stuff.
Cool.
So another example where TensorFlow
helped us is we previously had this system
called Gorila, which stands for general reinforcement learning
architecture, with some extra letters thrown in there.
And we wrote this, initially, to speed up
training of DQN style agents.
So what this was was a framework built on top of disbelief.
We had the parameters of an agent stored
across multiple parameter servers,
and multiple instances of that agent
pulling the parameters from that parameter server
and interacting with environments to get experience.
That experience was then stored in a distributed memory where
multiple learners sampled from it to make updates
to the model parameters.
And the whole thing kind of cycled around like that.
By doing it this way, we could get more experience
faster because we had multiple agents,
and we could learn faster because we
had multiple learners.
But often, more than the speed of convergence,
we often saw that sometimes these models actually
did better than models trained on single servers.
And we think that had something to do
with the breaking of correlations
between the examples we trained on from time to time.
This also was one of the inspirations
for the A3C training algorithm, which our researchers later
developed.
So with the introduction of TensorFlow,
we were able to basically deprecate this
because much of the functionality
is handled by TensorFlow or can be
written from within TensorFlow, which is great
because it's less time we need to spend on code
and maintaining it, and more time we
can spend on our actual projects.
More generally, since we've started using TensorFlow,
we've seen a reduction across the board
in the amount of ancillary work we have to perform.
And things are only getting better as TensorFlow
and the ecosystem around it continues to develop.
So one of these is the high-level APIs
which are developing, which take care of a lot of the work we
previously had to do to start a new project,
to build a prototype, to train it and evaluate it,
meaning we can spend more time on modeling.
It's easy to experiment with models in TensorFlow.
It's easy to put together a baseline in a couple
lines of code.
It's easy to change things about it.
So for example, if we have a bunch of embeddings
from various time series, do we want
to concatenate the last 10 of them together?
Do we want to convolve over them?
Do we want to pass them through an LSTM to get the state?
It's all just a couple of lines of changes,
which is quite nice.
The distributed training helps us.
We can try more things quicker and sooner.
And finally, once we've gone through all this work
and we're excited and wanting to put them all into production,
TF Serving kind of gets rid of a lot of the pain that
was previously there.
So all this amounts to a quicker development cycle, meaning we
can support more projects.
We can experiment more, and we can deliver
quicker, which is great.
And what's more, we've yet to come across an instance in any
of these collaborations with Google
where TensorFlow has not been suited
to the problem we're working on, hasn't been flexible
enough, which is great.
So I'm excited about the possibilities here,
and I hope we can tell you more about some of these projects
soon.
So the applied team is only one part of DeepMind,
and our friends in research are also
busy doing lots of cool stuff and have
begun to take advantage of TensorFlow to do so.
So one of the obvious things and one
of the most exciting moments for us last year
was when AlphaGo succeeded in a series of matches
against Lee Sedol, one of the greatest Go players
of the last decade.
What made this so exciting was it
was a combination of a great technical achievement
in making a strong Go player, but it also
represented something of a new beginning in the game of Go.
Interest boomed.
Go boards sold out across the world, which was interesting.
And players responded, not with a sense of defeat,
but they were excited about it as a new beginning,
because they could work with these machines
to explore more deeply the mysteries of this ancient game.
More to the point, AlphaGo was one
of the first significant uses of TensorFlow
at DeepMind, where it was used to great advantage
to train these networks, powering AlphaGo.
So I won't spend too much time talking about Go itself.
Just suffice it it say, it's a perfect information game
with a very high branching factor and a lot
of moves in a particular game.
So your standard minimax search obviously won't work.
The main component of AlphaGo was composed
of these two networks--
a policy network and a value network.
The policy network was trained to, given a game state,
work out which move the opponent was going to make.
And the value network was trained to, given a game state,
predict how likely we were to win that game.
So naively, while you could just use the value network
to evaluate your next set of moves,
in practice that wasn't quite good enough for what we needed.
So we resorted to a bit of a compromise,
using these nets with a Monte Carlo tree search
to get better results.
And the way this basically works is,
if you do a full Monte Carlo search, that's
way too many possibilities.
So we use this policy network to guide the paths in the search
tree to the most likely paths.
And then at some point further on in one of those paths,
we'd use the value network to evaluate
how good that path was, meaning we didn't have
to go all the way to the end.
Why the value network performed better in that case
is because as we unrolled these paths, more of the game
had unfolded.
And things were more certain, so the value network
had a better prediction.
But the real thing where TensorFlow came in
was the training of this thing, which was quite significant.
It took quite a significant amount of work.
So we started by taking games from expert players
and training this policy network.
And then we began this iterative process
where we use reinforcement learning
to train a value network from that policy network,
and then update the policy network to produce
better moves.
Previously, doing this on Torch, it was slow.
And when we moved to TensorFlow, we
were able to bring the whole power of distributed training
to bear on this, which made the whole thing faster.
It meant our researchers could iterate quicker and try
new things.
Another cool thing from this year
was WaveNet, which we use to generate
realistic audio signals.
So this is hard in general, because sound waves have
this complex oscillating structure
with a very high sampling rate, on the order of 16 kilohertz.
Generating an audio signal requires
a model which is able to reproduce this sort of signal
at both these very fine-grained oscillations,
on the order of milliseconds, and the whole signal
over seconds, which is difficult because of the amount
of information.
So there were two main approaches to this,
previously--
concatenative models and parametric models.
So a concatenative model worked by having a speaker say
a whole bunch of sound, and chopping it
up into little bits, and then reassembling those
to say whatever you wanted.
What you often see from these is the sound
seems kind of choppy because these transitions
between these sound segments don't quite match up.
It's also difficult to modify the voice,
either adding a new speaker or altering the emphasis
or emotion.
Similarly there, the other was parametric models,
of which WaveNet is one, where all the information required
to generate speech is stored in the parameters of the model.
And you supply information about what you want to say
and the characteristics of that as inputs to the model.
However, prior to WaveNet, we noticed
that parametric text to speech tended
to sound a bit less natural than concatenative models.
So WaveNet changes the paradigm that was previously
in parametric models by directly modeling
the waveform of the audio signal, one sample at a time.
So as well as yielding more natural sounding speech
by directly modeling the [INAUDIBLE],
it means that WaveNet can produce any kind of audio, not
just speech.
So the architecture for this evolved out
of work on pixel RNNs and pixel CNNs,
which were used for generating images.
And the key piece of architecture
here was the use of a dilated convolution
with an exponentially increasing stride as the layers increased.
What this allowed the model to do
was to have a receptive field size that
was exponential in the number of the layers
in the model, which allowed it to incorporate
all this into information about the signal.
The other cool thing about this structure
is that in the first half of the input,
compared to the second half, there
are twice the number of connections,
meaning it uses that information to a greater degree.
So here are some samples generated
from the various systems.
So on the left, we have audio from a concatenative model,
[INAUDIBLE] play--
the left one, yeah.
CONCATENATIVE MODEL: "The Blue Lagoon"
is a 1980 American romance and adventure film
directed by Randal Kleiser.
DANIEL VISENTIN: So if you listen closely,
you can perceive some of the discontinuities
in the generated audio, due to the slight mismatches
in the transitions between the different recorded chunks.
So in the middle, we have a parametric model.
PARAMETRIC MODEL: "The Blue Lagoon"
is a 1980 American romance and adventure film
directed by Randal Kleiser.
DANIEL VISENTIN: Which is less choppy,
but it doesn't have that natural quality we
associate with normal speech--
and so, and finally, we have WaveNet.
WAVENET: "The Blue Lagoon" is a 1980 American romance
and adventure film directed by Randal Kleiser.
DANIEL VISENTIN: Which, compared to the other two,
is very smooth and natural--
the only slight thing that WaveNet does
is it has a bit of noise, due to the sampling procedure
and the neural network.
So maybe if we play them all again,
just so people can see, hear, from--
SPEAKER: [INAUDIBLE]
DANIEL VISENTIN: Yeah, just from left to right.
SPEAKER: You want to play them again?
DANIEL VISENTIN: Yeah, thanks.
CONCATENATIVE MODEL: "The Blue Lagoon"
is a 1980 American romance and adventure film
directed by Randal Kleiser.
PARAMETRIC MODEL: "The Blue Lagoon"
is a 1980 American romance and adventure film
directed by Randal Kleiser.
WAVENET: "The Blue Lagoon" is a 1980 American--
DANIEL VISENTIN: Thanks.
[LAUGHTER]
So we compared WaveNet against other models
through the use of opinion scores.
And what we found by human raters, and what we found
was that it narrowed the gap to human speech by about 50%,
which is great.
And because WaveNet models, they model the raw audio signal,
we can also get it to produce sound other than speech.
So here's an example where we trained it
on a corpus of classical music.
[MUSIC PLAYING]
Thanks-- as you can hear, it's able to generate--
oh, go back one, yeah, thanks.
As you can hear, it's able to generate
realistic sounding piano notes on a moment-to-moment basis.
And finally, with what must be one of my favorite paper titles
of the year, "Learning to Learn by Gradient Descent
by Gradient Descent," in which some of our researchers
trained a neural network to train a neural network,
which I think is a great example of the kind of flexibility
that TensorFlow offers.
And I think, going forward, we're
going to see more of this kind of crazy, out-there stuff,
like with models learning architectures of models,
models adding bits and pieces to themselves, and so on.
And so it's exciting to see that TensorFlow can handle that.
And I know, at least from us at DeepMind,
that TensorFlow is going to be our choice of how we model
these things going forward.
So if you're interested in more details,
you can look at our website, deepmind.com.
There's a blog post linked here, which rounds up some
of the things from last year.
And we also have links to the over 100 papers
that we've published so far.
Thanks for listening.
[APPLAUSE]
[MUSIC PLAYING]