[MUSIC PLAYING]
YUFENG GUO: What does it take to build
a custom object recognition mobile app starting
from raw data?
We'll find out on this episode of "Cloud AI Adventures."
Today, we're joined by Sara Robinson, a developer
advocate on the team who also focuses on machine learning.
She created an app that recognizes
pictures of Taylor Swift.
And she joins me today in the studio
to discuss how she accomplished it.
Welcome, Sara.
Thanks for joining me.
SARA ROBINSON: Thanks for having me-- super excited
to be on "AI Adventures."
YUFENG GUO: So you wrote a sensational medium article
that I've got open here about your Swift Detector, which
kind of showcases end-to-end machine learning,
from gathering and pre-processing the data,
to training and serving from Cloud Machine Learning Engine.
And there's even Firebase and an iPhone app in the mix.
SARA ROBINSON: Lots of different tools.
YUFENG GUO: Yeah, definitely.
It's super amazing.
What inspired this project?
SARA ROBINSON: So I'd seen a couple
of other blog posts on the TensorFlow Object Detection
API.
And I'd been wanting to try it.
Out these blog post had things like detecting
pets, whether there is a cat or a dog in an image.
I saw another one about detecting raccoons.
YUFENG GUO: Oh, wow.
SARA ROBINSON: So then I thought,
I wanted to try it on my own with some training
that I found.
And the easiest thing I thought of
was to just look at my Google Photos
and find all the pictures of myself
to train a model to recognize me-- had lots of selfies.
I had enough training data.
YUFENG GUO: Yeah, and that search function
on Google Photos, super helpful.
SARA ROBINSON: It's pretty great.
So I trained that model locally.
It works pretty well.
But I wanted to build an end-to-end demo.
And I know a little bit of Swift.
So I thought would be cool if I built a iOS client to make
a prediction request against my model, which made me think,
why don't I built a model to detect Taylor Swift?
Because why not?
YUFENG GUO: Swift with Swift--
perfect.
So it looks like here you're using the TensorFlow Object
Detection API to find Taylor Swift in these pictures.
Can you talk a little bit about what that is,
and what was it like to use it?
SARA ROBINSON: Yeah, sure.
So the Object Detection API is some utilities
that make it easier to recognize objects in an image.
So rather than just classifying an image
as this image has a cat in it, it'll
return bounding boxes of where that object is in a image.
And to do that, it's using a technique
called transfer learning, which utilizes a model that's already
been trained on a similar classification
task using millions and millions of images.
And then what I do as a developer using Object
Detection API is I take that checkpoints
of the second to last layer of that
model that's been trained on millions of images.
And then I update it with my own data,
so that I don't need to start from scratch.
So I didn't have time to label millions
of images of Taylor Swift.
YUFENG GUO: Though I'm sure you wouldn't have minded, right?
SARA ROBINSON: I wouldn't have minded
YUFENG GUO: Going through all those pictures, yeah.
SARA ROBINSON: So I just needed to use a smaller
set of training data to update the final layer of that model.
YUFENG GUO: It's fantastic that you can kind of stand
on the shoulders of giants--
all the work and all the training that's
already been done on these models.
SARA ROBINSON: That's one of the great things about transfer
learning.
YUFENG GUO: Yeah.
So was the training kind of quicker, as a result,
because you didn't have to go through as many images?
SARA ROBINSON: Mm-hmm, much faster.
So instead of taking maybe days, it took just a couple of hours.
YUFENG GUO: Fantastic, awesome.
And then so actually doing the work that--
I guess you would have had to somehow get the images,
and do any pre-processing required to have the TensorFlow
Object Detection API kind of consume those images, what
formatting you needed to do, and then actually run the training.
SARA ROBINSON: Mm-hmm.
So the Object Detection API requires your images
be in what's called a TFRecord format--
YUFENG GUO: TFRecord, OK.
SARA ROBINSON: And to get them in that format,
I converted my images to what's called the Pascal VOC
format, which is just one format you can choose from.
It stands for Visual Object Classification.
And what that is is that's an XML file.
So I take all my images and I generate this XML file,
which gives me details on the bounding box of where
the object is.
YUFENG GUO: So that's what's in the XML.
It's just kind of coordinates here of the boxes themselves.
SARA ROBINSON: Mm-hmm.
So it has the x,y coordinates of where my object is is an image.
And then it just has the label associated
with that bounding box.
In this case, I just had one label, Taylor Swift.
YUFENG GUO: Only need one.
SARA ROBINSON: I only need one.
YUFENG GUO: Awesome.
So let's see.
We have our data.
It's labeled.
And I guess now there's only one thing to do, hit the training.
SARA ROBINSON: Yeah, so training is next.
YUFENG GUO: So with all these kind of steps,
can you just outline in how they all connect together?
SARA ROBINSON: Mm-hmm.
So I drew about bounding boxes around my images,
got them into the Pascal format.
And then once they're in that format,
I wrote a script to convert them a TFRecord, which I can then
use to feed into my model.
YUFENG GUO: Gotcha.
OK, so all this kind of steps, now they're all connected.
You've gone from the raw image to a--
well, not just a labeled image, but specifically, TFRecords.
It's even formatted correctly.
So now we're ready for our training, I guess, right?
And so in this case, we're going your training on ML Engine,
it looks like.
It's on your blog post here.
How is that process?
And what did you have to do to make that work?
SARA ROBINSON: So it was similar to the process
I used for running the training locally.
The main difference being I needed to put all of my data
in Google Cloud Storage and tell ML Engine where to find it.
So the first thing I needed to get in Google Cloud Storage
was the model checkpoints from the pre-trained model
I was using.
I used a model called MobileNets, which is optimized
for mobile, obviously.
So I put those checkpoints in my Cloud Storage bucket.
I also needed my training and test TFRecord files,
which I talked about before how I generated those.
And then I have a config file, which
tells the Object Detection API where my training and test
data is located.
And it supports ML Engine and Google Cloud Storage.
YUFENG GUO: Great, great.
And so that whole thing kind of comes together and, I guess,
just takes care of all training for you then?
SARA ROBINSON: Mm-hmm.
YUFENG GUO: Awesome.
Then how long did that training time take?
Everyone wants to know if the training takes
a long time, or a little time, and how long you
had to wait for that.
SARA ROBINSON: I think it took about 30 minutes.
YUFENG GUO: 30 minutes, OK, that's not too bad.
SARA ROBINSON: Not too long.
YUFENG GUO: Yeah, given what you're getting out of it,
that's quite nice.
SARA ROBINSON: Exactly.
YUFENG GUO: So once training is done,
what do you have kind of as your result?
What do you have in your hand?
SARA ROBINSON: So you have a model checkpoint,
which is just a binary file.
Similar to the pre-trained model checkpoint
that I used to start training, this
is now going to be the final model
checkpoint, which has been updated
with all of my training data.
YUFENG GUO: Gotcha, OK.
So that's very nice.
And then I guess we now kind of flip
it over the wall to the serving side.
SARA ROBINSON: So to serve it, I can use gcloud command
with ML Engine to deploy my model.
I just need to point it at where those checkpoint files are
of my final trained model.
And then in just a couple of minutes,
I have my model available for serving.
And I can use that to generate predictions against the model.
YUFENG GUO: Fantastic.
And so I guess this kind of brings us
to the next phase of this project.
Here we've gone from our data.
We've pre-processed it.
We've done the training.
And we have this kind of API that can serve up
predictions at scale.
SARA ROBINSON: Exactly.
YUFENG GUO: So now we're ready to call that endpoint.
And what did you end up kind of architecting here?
It looks like you have a couple of pieces.
SARA ROBINSON: Mm-hmm.
It's using a couple of different Firebase SDKs.
So the client is a Swift app.
It's an iOS app that's written in Swift.
And it's a pretty thin client.
So all the client is doing is it's uploading
an image to Firebase Storage.
YUFENG GUO: Oh, OK, so that's what that arrow is for.
SARA ROBINSON: Exactly.
And then we're using the Firebase SDK
for Cloud Functions.
So a function is triggered whenever an image is uploaded
to a specific storage bucket.
YUFENG GUO: Ah, very handy.
SARA ROBINSON: So the function is written in Node.js.
And that function does a bunch of different things.
It downloads the image.
Then it resizes it.
It base64 encodes it, which will prepare it for the ML Engine
online prediction request.
YUFENG GUO: OK.
SARA ROBINSON: So I make that prediction request.
If the confidence that I get back is greater than 70%--
that's just a threshold that I've chosen--
I'm going to draw a box around that image.
YUFENG GUO: OK, in the box drawing
is it looks like it's happening here in your Node code.
SARA ROBINSON: Yeah, I'm doing that using
ImageMagick, which is available in Google Cloud Functions.
YUFENG GUO: Oh, perfect, very convenient.
And so then, once the box is drawn,
how do you get that information back to your client then?
SARA ROBINSON: So once the box is drawn,
I write the image to Firebase Storage.
And then I also write some metadata on the image
to Cloud Firestore, which is a database.
So I'm writing the confidence of the detection
along with the path of the image in Firebase Storage.
YUFENG GUO: Gotcha.
So that's why you kind of have these two write operations
coming from your Cloud Function?
SARA ROBINSON: Exactly.
And the reason I'm using Firestore is, in my client,
I can create a listener on the ID of that image path.
And so then I get updates whenever
new data has been added to that collection in Firestore
so I don't have to keep pulling that to see if the detection is
finished.
So right when it's finished, my client will get an update.
And then I can then download the new image
with the box around it, along with the confidence detection
score.
YUFENG GUO: So that's really neat
that you've managed to use the kind of architecture
to signal in both directions.
You upload to Cloud Storage, and that
triggers the Cloud Function.
So on upload, something happens.
And it happens right away.
And then coming back, once the response from ML Engine
is received, you're using Firestore to basically give
that real time update to hit your phone right away, as well.
So you kind of keep that loop tight,
even though you managed to do that while all
the while maintaining this kind of thin client.
That's really nice, the way you've
kind of leveraged the existing technology.
You didn't have to write anything
to make sure everything happened nice and quick, but async.
SARA ROBINSON: Yeah, one thing that's great about Firebase
is I get those real time updates whenever
new data comes in that path.
YUFENG GUO: Fantastic.
So I guess, zooming out a little bit,
you've built the mobile app that builds a custom
machine learning model.
You've collected your own data.
You've annotated it.
You're training and you're serving at scale
to your mobile client.
What did you find to be kind of the most interesting part
of all of this?
SARA ROBINSON: I would say, definitely
learning to tie together all these different tools.
So there's a lot of different things
I used to build the app, from [INAUDIBLE],,
from TensorFlow Object Detection,
to Cloud Machine Learning Engine, which
has a couple of different components
that had training, serving and online prediction.
And then the client piece using some Swift SDKs and Firebase
SDK for Cloud Functions.
YUFENG GUO: Yeah.
And just along the way, you're hitting Python, Node, Swift,
Java--
well, JavaScript is there.
SARA ROBINSON: Even a little bit of bash
with the gcloud commands.
YUFENG GUO: That's right.
Outstanding.
And so, on the whole, what did you
find to be most challenging out of all of this?
SARA ROBINSON: The most challenging part
was getting all of the data into Google Cloud Storage.
Because there was a lot of different pieces
that need to be there, from the pre-training model checkpoints,
the TFRecords, and then that config file,
which tells Object Detection where
to find these different pieces.
YUFENG GUO: Right.
And these are all coming kind of from different places
and just getting it all arranged.
Yeah, I could see how that could be challenging.
And was there some fun parts, as well, hopefully, along the way?
SARA ROBINSON: It was.
It was pretty fun to build.
One thing that was kind of funny was
I label all the images myself.
So there was a day where I--
YUFENG GUO: Hard work.
SARA ROBINSON: I know.
It was a tough job.
So there was a day where I had pictures
of Taylor Swift on my computer.
People who walked by were probably
wondering what I was doing.
YUFENG GUO: Yeah, just perusing photos all day.
SARA ROBINSON: Exactly, generating
those bounding boxes.
And the most fun thing was definitely the final result
of seeing on the iOS app the prediction
with the bounding box come back in real time.
YUFENG GUO: Definitely, yeah, that's a long journey.
SARA ROBINSON: Mm-hmm.
It was.
YUFENG GUO: Awesome.
And as I understand, you've actually open-sourced your code
on GitHub.
SARA ROBINSON: Yeah, it's all available on my personal GitHub
repo.
And there's instructions in the readme
if you want to build the same model yourself.
YUFENG GUO: Awesome.
That's so great that our audience can go out and build
their own model.
And I want to thank you so much.
I really appreciate you coming in today
and chatting in the studio.
SARA ROBINSON: Thanks for having me.
Great to be here.
YUFENG GUO: Yeah, it's been great.
Thanks for watching this episode of "Cloud AI Adventures."
Today, we saw how to go from raw data,
to training, to a mobile client.
You can use Sara's GitHub repo to not only recreate
this, but extend it with your own data, a different model,
or a different client app.
For more details, we've included links
to all the resources we discussed in the description.
So check it out.
And if you've enjoyed this episode, please like it
and subscribe to get more episodes right
when they come out.
[MUSIC PLAYING]
[BEEP]
SARA ROBINSON: So the Object Detection API
is more of a series of--
sorry, I'm going to redo that.
[BEEP]
YUFENG GUO: What does it take to build a custom image
recognition-- no.
[BEEP]
SARA ROBINSON: Custom object recognition.
[BEEP]
YUFENG GUO: What does it take to build a custom recog--
[BEEP]
A custom object recognition mobile app.
[BEEP]
I don't know what to do next.
[LAUGHING]