ADDISON HOWARD: Well, hello everyone
and welcome to today's Google Cloud Next Session
during Cloud AI Week.
My name is Addison, and I'm joined by my colleague Phil.
We're both members of the Kaggle competitions team,
and we're excited to speak with you today,
specifically about TPUs and TensorFlow on Kaggle.
But first, what's Kaggle?
Historically, Kaggle's become synonymous
with machine learning competitions,
but it's really much, much more than that.
Kaggle is a data science platform.
Over five million data scientists
from all over the world come to Kaggle not only
to participate in the machine learning competitions,
but to learn data science, build their skills,
polished their portfolios, and share data sets and code.
It's truly an incredibly exciting community.
Now my job at Kaggle is to run machine learning competitions,
and they run the gamut.
Right now we have 17 competitions running,
ranging from understanding bird sounds, [INAUDIBLE] lung
function decline, classifying melanoma, NLP challenges,
and even a bot-building simulation
competition, among many others.
We recently wrapped up a competition with a $1 million
prize that detected deep-fakes in videos.
We had a competition recently with both the NCAA and the NFL.
And even abstract reasoning challenge
with Francois Chollet.
And by the way, if you haven't seen his recent paper titled,
On the Measure of Intelligence, I mean, it's pretty amazing.
Really encourage you check that out after this session.
But basically, if you're interested in data or machine
learning in any form or fashion, we've
likely got a competition that will excite you.
And really, one of the most exciting elements of Kaggle
is that as the home of data science, professionals,
hobbyists, thought leaders, and academics
will come to Kaggle to test new theories, technologies,
and potential advancements in the state of the art of machine
learning.
Now we've been very fortunate to witness
the birth, or the spread and dissemination,
of many of the leading data science techniques,
from back when random [INAUDIBLE]
decision trees took over, to LightGBMs, XGBoost, and then
the introduction of libraries, like Harris.
So what better community to introduce great technologies,
like TensorFlow and TPUs, than through Kaggle competitions.
And the Kaggle community is filled with data scientists who
are eager to experiment and quickly
uncover the benefits and shortcomings
of newly-introduced products and methodologies.
And a Kaggle competition that can
provide a structured and engaging format
to test them out.
And specifically for TPUs and TensorFlow,
our competition platform allows us to host really large data
sets, and to offer both time and resource-constrained formats
that can help truly capture the power
and understand the benefits of these particular tools.
So if you'd like to check out, the latest TensorFlow edition,
we're calling it 2.x, as they continue to be
updated on the Kaggle platform.
Especially with GPU or TPU, Kaggle makes it easy.
You can see from the list on this slide
that we've eliminated a lot of the obstacles
to making TensorFlow 2.x build your models fast, with next
to no specialized coding.
No provisioning of the right kind of VM, no setup.
We already have data sets that are ready to go.
Our cloud notebook platform removes the burden
of needing to reconcile versions or libraries or packages
across collaborators, so sharing and iterating is a breeze.
And finally, we even provide a weekly allotment
to TPU and GPU time at no cost to our users
all over the world.
And the integration benefits afforded to TensorFlow
also apply to TPUs.
They're directly accessible from archival notebooks
that [INAUDIBLE] additional setup.
We have generous data sets that are
accessible via Google Cloud Storage that allow
for efficient data access.
30 hours of TPU use per week are granted to each user.
We've even hosted five competitions just
in the last six months aimed at these particular tools.
Most recently, we hosted a competition
that was TPU-centric with the Google Team, Jigsaw, that
was seeking to evaluate and classify
toxic comments on Wikipedia.
We also just launched a competition
called Contradictory, my dear Watson, which
seeks to use TPUs and natural language
processing to evaluate grammar.
But today we want to look specifically
at our first two competitions that introduced TPUs to over
five million data scientists.
First being TensorFlow 2.0 Question Answering,
the second being one on flower classification.
Now TensorFlow is already very heavily used
in our competitions.
It's currently being used in all of our running competitions,
but in the past few months, we've
run two TensorFlow competitions that I mentioned,
this one on top being a question answering NLP competition built
around the launch TensorFlow 2.0,
and a new competition for image classification
that introduced TPUs for the first time to our platform.
We already have tens of thousands
of people using TensorFlow 2.x right now to solve problems.
And three of the top solutions in the question answering
competition ended up using TensorFlow and TPUs.
And so next what I want to do is, let's dive
in to some of the code.
Let's look at some of the code right now
we'll build a model on Kaggle for our competition that
classify flowers on CPU, GPU, and TPU with no changes.
So Phil, I'm going to hand it over to you
to walk through some of the [INAUDIBLE] competitions.
PHIL CULLITON: Thanks, Addison.
We start off simple by importing TensorFlow 2 and the Kaggle
data sets library.
The rest of the code here does a bit of magic.
It figures out what kind of accelerator
is attached to your VM, and allows us to automatically
parallelize your model build.
This will determine an appropriate strategy,
whether you're using a V3 TPU, or a simple CPU VM.
Basically, this code asks for a TPU, and if it gets one,
it connects to it and sets it up.
If it can't get a TPU, it sets it up as well.
The rest of the code in these slides
is entirely accelerator agnostic.
Normally, outside of Kaggle and Colab,
you'd need to provision this TPU for usage
and do some other setup.
But on our platform, that's not necessary.
Just select the TPU from a dropdown and go.
Now we load up our competition data set.
It's provided and sharded TF record format for fast loading.
We use multiple files to provide enough data
streams to maximize the TPU's throughput.
First, we get the path to those TF record files
and pass it to TensorFlow, then set some parameters
for a training and validation sets.
Shuffling our training set, batching
both sets for optimal performance,
setting our training set to repeat so it'll loop back
around with the [INAUDIBLE].
We'll pass these into our model and TensorFlow
will access these TF record files
while the model is building.
The end result of this process is
that we have data sets ready to run
in our accelerator of choice.
This code does some heavy lifting.
Here, we load up VGG16 for some transfer learning,
and then define our model by adding
the global average pooling and output layer on top of that.
Note the with strategy.scope block.
That's basically saying, hey, whichever type
of acceleration strategy you're using here,
this is the process to parallelize.
Now our model will be compiled and trained
on the accelerator we've chosen.
Note again that this same code will run on CPU, GPU, or TPU.
As long as we identify our distribution strategy,
TensorFlow 2 automatically handles
putting this model in the right place and building.
After this, we compile our model with our choice of optimizer
and loss, pass the training and validation sets
from the previous slide into it, and start
the model-fitting process.
Making some flower predictions is easy after this.
We load up our test data set using the same type of code
that we used to load the training validation sets.
Once again, this results in data that
makes the best use of our TPU.
Then we run model.predict, transform our prediction
into the right form, and write our predictions out in a format
that Kaggle can read.
With a few tweaks and a few more epochs,
this model can actually score pretty well.
I'll finish the code section with some of the fun flowers
that our model has made predictions on.
You can see that some are right, they've got a little OK
above them, and some are wrong.
If you're interested, you're more than
welcome to drop by the Petals to the Metal competition
and beat these results.
That won't be hard, but it will probably be fun.
And now, Addison, back to you.
ADDISON HOWARD: Thanks, Phil.
As Phil demonstrated in his code review, using TPUs on Kaggle,
it's a super easy way to get used to TPUs.
In the flower competition-- flower classification
competition that Phil highlighted--
we had over 24,000 TPU-using sessions by Kaggle users,
and a very small amount needed any amount of help to get
started.
Thanks to some getting-started work
done by some members of team, over 80% users
needed no help to begin using TPUs.
As you can see in the screenshot of top Kaggle notebooks
in the flower classification competition,
users quickly created and shared their own TPU-based notebooks
and began to train each other with discussion topics
and highly upvoted notebooks.
Many of these top notebooks came from Kaggle's highest
ranked grandmaster users, and it's really exciting
to see the data science community very quickly
learning themselves so they can then quickly teach others.
So let's highlight some very specific outcomes
of this competition.
In the Kaggle notebook, users can
elect to use a CPU, GPU, or a TPU, like Phil mentioned.
So we picked two of the top-performing models using
each TPUs and GPUs that each to use effectively the same code.
The TPU model trained on 25,000 images in an hour and 15
minutes, and that same model would run out of memory when
they try to flip it over GPUs.
The comparative GPU model took five hours
to train those same 25,000 images,
but they had to be at a reduced size.
In the end, the accuracy still did prove competitive
with the TPU model.
Now, prior to this competition, there
had been some slight murmurings of TPUs,
but once the first competition was launched in February
of this year, you can see the large jump in discussion
forward mentions per month about TPUs, and it continued to rise.
Now as we introduce more competitions, including
a competition designed to help introduce TPUs
to the uninitiated, we're looking forward
to seeing how not just discussion
of TPUs and TensorFlow in the community increase,
but how the data science community continues to find
great insights in their spread.
So if you're interested in seeing TensorFlow 2 at work
on Kaggle, come check it out.
Try out the code that Phil just showed you,
compete in the flower competition,
or try it out with the data set of your choice.
Or try [INAUDIBLE] past competitions
or on our hosted notebook platform
where we give you some TPU hours play with.
In the coming months, we'll be launching
even more competitions so you can
leverage power TensorFlow and TPUs on Kaggle.
Now it's been a big group effort.
I want to give a special thanks to Martin Gorner, who
is the DevRel project manager who led the effort
to integrate TPUs, so thanks so much to him for his hard work.
It's pretty awesome to see how easy it is to use TensorFlow
and Kaggle right now.
So finally, thanks to you for joining us
with our session today.
Thank you for your time.
Please come visit us over at the Dory.
We'll be hanging out all week answering
any questions you may have about TPUs in TensorFlow on Kaggle.
And again, thanks so much.
Have a great day.