JONATHAN HSEU: Hi, I'm Jonathan.
A lot of you are probably running a lot of infrastructure
already in your organization or you
want to think about running various other pieces
of infrastructure.
And I'm going tell you about TensorFlow fits into that.
So, the clicker--
[SIDE CONVERSATION]
OK, thank you.
So first, I'm going to talk about all the steps you
go from training to serving a model in production.
And it typically involves these three steps.
Usually, the raw data that you have isn't good for training
yet, so you have to do some form of data preparation.
After that, you do training and then you serve your model.
So, for data preparation, there are three essential steps
that are involved in data preparation.
So you'll import the data from various sources.
This might be Hive, MySQL, and so on.
You'll typically perform some sort of pre-processing
on the data.
They'll compute aggregates or do joins
and then you'll export the file in the file format supported
by TensorFlow and I'll explain that a bit later.
So, a lot of people have been using typically these three
things for doing this process--
SPARC, Hadoop, MapReduce, or Apache Beam.
And it works well with all these at the moment.
And as a visual example, we have--
you're feeding in data from MySQL.
Say it's user data.
And you have data in Hive.
An example might be web impressions or web clicks.
You want to join that data, compute aggregates
like user data per region.
Often, you'll do vocabulary generation, particularly
if you're using embeddings.
Vocabulary generation essentially
means, like, mapping from words to IDs and the IDs map
to an embedding layer-- embedding.
And then after that, you'll typically
have a training test split.
And after this process, you'll output to a file format
that TensorFlow understands.
And why might you want to do this?
So, the input file format they use with TensorFlow
actually matters quite a bit.
So, one example is, suppose you're training inception
on P100 GPUs.
You want to keep the P100 saturated.
Actually, one of the most important things
you can do to make that work well
is work on your input pipeline.
And that involves working with cues.
There are tutorials on the TensorFlow website.
But specifically in this case, I'll talk about file formats.
Some file formats are faster than others.
We have specifically optimized the path
for working with tf.Example and tf.SequenceExample protocol
buffers.
SequenceExample is typically used
when you're training with RNNs and tf.Records
is a TensorFlow specific file format
that we use with TensorFlow.
And this is by far the fastest.
We use this internally and we've optimized this path
pretty well.
Additionally, TensorFlow also natively
has ops to support reading from various file formats like CSV,
JSON, fixed record length files.
We have upcoming support for Avro.
So you can use those.
Those are a little slower, but since they're native ops,
they actually work pretty well.
More commonly, people, if you, like,
read examples of TensorFlow code on the web,
people will be feeding data directly in Python.
And this is by far the most common.
It's the easiest to experiment with.
It also has the most flexibility,
meaning that you can use any file format that Python
natively can understand and then convert it to numpy arrays,
which can be fed as tensors in TensorFlow.
It's also useful for settings like reinforcement learning.
But it's also the slowest option.
So if you're running experiments, this makes sense.
But if you're trying to productionize a pipeline
and you care about training speed,
then you might want to consider switching to the first option.
OK, I'd like to mention actually why it's a bit slower.
There are two things that are involved--
reading from specific file formats in Python
itself may not be too fast.
So you have that part.
And then when converting from new pi arrays to TensorFlow
tensors, there is a copy involved
and that induces additional overhead.
So I've recommended tf.RecordSupport.
Hopefully at this point, you're convinced.
And how well does it work?
So, Apache beam, as of a few weeks ago,
actually has native support for this file format.
But a lot of you aren't running Apache Beam
and you'll be running Hadoop MapReduce and SPARC
jobs already.
And you just want to use that.
So if you go to the ecosystem repository in the TensorFlow
GitHub organization, we have code to work with TF records
there.
Note that I didn't mention protocol buffers at all.
Protocol buffers actually has native support
for a lot of languages you would be using.
OK, so now that you've exported this training data,
you want to go to training.
There are two common ways of doing training.
One is say you have a GPU on your desktop or laptop.
You want to do local training.
This works well for the specific cases of debugging or working
with small order data sets.
But as Megan mentioned in the intro talk,
distributed training-- like, 64 GPUs,
you actually get a 58x speed up in images processed per second
if you run on distributed starting
with 64 GPUs versus one GPU.
So you train much faster.
And for a lot of people in your organization,
this could mean the difference of taking a month
to train a model versus taking a day.
And the feedback loop is pretty important
to make machine learning work well in your organization.
So, there are two essential components
if you want to start with distributed
training in your organization.
Derek was talking about cluster managers.
We know organizations that have been using all of these
to make a distributed TensorFlow work well.
So we'd recommend any of these.
By Hadoop here, I mean Hadoop Yarn.
And so we actually have configuration examples
for Kubernetes and Mesos specifically
in the ecosystem repository.
So if you're already running that, then you can get
started pretty quickly that way.
But in addition to a Cluster Manager,
you should also be running distributed storage.
Derek was talking about this a little bit,
but these are the ones that we work with.
We added HDFS support in TensorFlow 11.
TensorFlow also has native support
for reading from Google Cloud Storage,
and if you're running on Amazon's cloud service,
then users have reported it working well with Amazon EFS.
So, the reason you would want to run distributed storage
is mostly for two reasons--
for your input data so each worker
can read from a single source and for output for your model--
so, for checkpoints in the exported model itself.
So, containers are completely optional
but highly recommended.
So, containers isolate your workers from their environment.
And one reason this is useful is because certain jobs
might have certain version dependencies that
might differ from other jobs.
And a specific example here is eventually,
we'll release TensorFlow 2.0.
Senseful uses semantic versioning,
and at 2.0, we may make some backwards incompatible changes.
And suppose you want to upgrade to TensorFlow
2.0 in your organization.
Then the containers make it much easier
to do gradual updates because you don't have
to update every job at once.
You can just update jobs.
You can pin the jobs to a particular TensorFlow version
and update each one one by one.
So, getting started with containers makes sense.
Docker is supported in both Mesos and Kubernetes
and we recommend that.
So, as a refresh for what Derek was talking about,
I'm going to show you what distributed trading code looks
like as a refresh of what Derek mentioned
and what configuration involves.
So, there are essentially two types of jobs that you have--
parameter servers and workers.
And Derek mentioned that the most common setup
and the setup that we recommend is between graph replication.
And in that case, workers operate completely
independently.
So the workers don't necessarily talk to each other
or even know about each other.
They only communicate with the parameter servers.
And for that, Derek actually showed this but
across multiple slides.
This is the code involved if you're using core TensorFlow.
With the higher level APIs, a lot of this
goes away and becomes much fewer lines of code.
But in essence, your Cluster Manager
specifies all the other parameter servers and workers
that this worker must communicate with.
You start a server.
If you're a parameter server, you just stop there.
But if you're a worker, then you need
to assign variables to parameter servers
and set the worker device so that this graph belongs
on this particular worker.
So, configuration really depends on which cluster manager
and which distributed storage system you chose.
We have examples, again, in that repository dimension,
that ecosystem repository.
And we're adding more and feel free
to contribute any that you know that work well in your existing
infrastructure.
Additionally, I'd like to mention
that Yahoo! recently open sourced code to run on SPARC.
And why is this useful?
Because you might think that SPARC
is like a data processing framework.
Why would it be useful to run on SPARC rather than a Cluster
Manager?
So, SPARC actually has support for running on Mesos, Yarn
or in standalone mode, and a lot of you
will already be running SPARC in your organization.
And this actually might be the easiest way to get started.
And for sure, it's difficult to run distributed TensorFlow
on Hadoop Yarn.
And for sure, it's much easier to get
started if you're running SPARC on top of Yarn
than to try to run distributed TensorFlow on Yarn directly.
So I'd recommend this in specific cases
where you're already running SPARC.
And Dandelion talked about TensorBoard.
You might wonder, how do you use this with distributed training?
So, with distributed training, typically, you
have output which are summaries and events that
go to a directory that you specified
on distributed storage.
All you need to do to run TensorBoard
is run the TensorBoard command and point it
to that distributed storage directory.
It will start in HTTP server locally
that you can just look at.
You can look at your losses.
You can go through the embedding visualizer
and how that stuff works.
So, like, I've skipped a lot of steps here.
Like, suppose you need to do hyper
parameter tuning and so on.
But suppose you're already done with all of that
and you're happy with your model.
You want to export it and serve it in production.
So currently, we have two common ways of doing that,
but in the long term, it will be one.
We highly recommend you go with SaveModel.
It's going to be the standard file format for TensorFlow
going forward.
And the advantage to SaveModel over everything else
is that it includes all the assets
you need to serve your model in production.
I mentioned that vocabularies are
pretty common for embeddings and you use a specific vocabulary
at training time.
You want to make sure to use the exact same vocabulary
and serving time.
It includes all those files for you.
So the other advantage to SaveModel
is it's a bunch of graphs and you
can specify which one you want to use for a particular use
case.
GraphDef right now is pretty common in particular uses,
particularly on mobile.
So, in the mobile example, you typically
freeze your variables into constants
in your TensorFlow graph.
The advantage to GraphDef right now
is that GraphDef is actually a protocol buffer which
accesses like a single file.
So you only have to distribute a single file.
That's much easier to deploy.
But that advantage is going away.
As of recently, you can also freeze a SaveModel.
So hopefully, you've exploited the SaveModel.
You want to serve it.
Noah-- the subsequent talk is actually going go much more
into TensorFlow serving and I'm only going to talk about
the top two here--
TensorFlow serving and in process TensorFlow.
So, these are both options for serving your TensorFlow
model after you've exploited it, but they are quite different.
TensorFlow serving, in its typical mode,
runs as a separate service that you make RPCs to.
So, why might you choose one over the other?
So, Noah's going to go much more in depth here,
but serving a model actually has a lot of hidden nuance
that you might not be aware of when you just think about it.
So, if you're just applying a model as a one off,
then it might make sense to just use in-process TensorFlow.
Flow
But if you think about--
like, suppose you have a job that
trains every day on new data and you want
to serve a new model every day.
How might you start serving that new model?
You can think about it.
There's like two basic ways that might happen.
In your server, you can unload the completely unload
the old model and then wait for that to finish,
wait for all requests to finish, and load your new model.
This is memory efficient because you only
have one model loaded at a single time.
Or, you can simultaneously load the new model
while it's running and double your memory usage.
tf.Serving offers that as a configuration option.
Additionally, for efficiency, particularly
if you're serving on GPUs, you want to batch your inputs.
This also has some benefit on CPUs,
but particularly if you're serving on GPUs,
you want a batch to fully utilize your GPUs.
As you can imagine, there's some nuance
to how you want to batch, particularly
in low latency settings.
So, tf.Serving handles that.
Also, if you're serving multiple models,
isolation between those models--
they're going to contend for hardware resources
and there's some difficulties in isolating between them.
It's particularly intensiveful.
You have to think about which threads--
like, how to size thread pools and so on.
You don't have to think about that if you're
on TensorFlow serving at all.
So, that's the next talk.
Now that I've explained all the benefits, you're like, OK,
then why not just use that all the time?
Why should I use in process TensorFlow at all?
So, in process TensorFlow is actually the standard mode
for certain things, specifically if you're
running on a mobile device.
Pete actually went over this.
Basically, you have a TensorFlow session and you run it.
If you're running batch inference-- like,
if you already have a TensorFlow model, you have a ton of data
and you want to run inference over that data using Beam
or SPARC, then it's much better to just use
in process TensorFlow than to start another tf.Serving
service.
There's the specific case of very strict latency
requirements where you don't want
to do the RPCs back and forth with TensorFlow serving,
but this is a very rare case.
Usually, the latency involved there is very small.
Operationally speaking, it's much easier
to run one service than to run two.
So if you want to simplify your deployments,
that's also a reason to do this.
But regardless of which one you choose,
the first two steps between them is practically the same.
You want to export your state model, make it accessible,
and then you have to write the code to do inference.
And I'll show you what that looks like in a moment.
For in process TensorFlow, you have the additional step
of linking to the TensorFlow shared library, which
we actually distribute.
And you can see that in our Go and Java instructions
and you can just download it right there,
or you can build it yourself.
So, here's the Go example of what inference looks like.
It's actually not too much code at all.
And actually, if you go to the Go repository on GitHub,
you can actually see real inception code
that works in Go.
So, what it involves is first loading your saved model.
You pass the directory that the save model is in.
As I mentioned before, save models involve
multiple TensorFlow graphs.
So, after you specify the directory,
there is a list of-- set of tags that you
can use to specify which TensorFlow
graph you want to load.
When you execute your model, all of you
are probably already familiar with TensorFlow sessions
if you've ever used TensorFlow at all.
It's essentially the same thing as the Python API
but in another language.
You take a tensor.
Here, we have a batch of images.
And you run your TensorFlow model using that tensor.
And as output, you get probabilities
for each classification for each image in the batch.
And you do something with it.
So I showed Go there.
Jeff mentioned that we support a ton of different languages.
These languages are all on our TensorFlow GitHub repository.
So you can actually just go to the TensorFlow GitHub
repository and start using any of these.
All of these languages can actually
be used to build and execute TensorFlow graphs,
But a common question is, can any language besides Python
be used for training?
And the answer is a bit nuanced, but the short of it is no.
And the reason is because a lot of the supporting libraries
like the training libraries, the high level APIs, optimizers,
the RNN library, all of those are only available in Python
at the moment.
So I would personally recommend that you only
do training in Python, but you can do inference in any
of these other languages.
The instructions for using the shared library
are in the readmes that are associated with each language.
So you can either download it or build it yourself.
I won't go into too much detail here.
So, as a summary--
so, these are the basic steps.
You're probably running infrastructure already
around it.
The point I want to emphasize here
is that, especially if you have a lot of data
in your organization, you really should
be thinking about how to make distributed trading work well
and what you should be running to do that.
As Megan said, there's like a 58x speed
up when running on 64 GPUs and you
can imagine what kind of effect that
would have on the engineers working with TensorFlow,
in training TensorFlow models.
And thank you.
[MUSIC PLAYING]