DALE MARKOWITZ: So today we're building
this sort of DIY pet cam for tracking what
your pets do while you're gone.
Whether you want to know whether your dog's hanging out
on the couch, or your chickens are trespassing on the porch,
or your cat's just doing some casual software
development while you're gone, we'll
build an app that runs a machine learning model on your phone
to detect events, like dog on couch,
and then sends you a notification on Slack.
It'll also save snapshots of those moments to the cloud.
So let's get to it and build an ML powered pet cam that
runs on your smartphone.
[MUSIC PLAYING]
One of the cool things about this project
is that we're going to do the computer vision part entirely
in a web app that can run on a smartphone.
It'll be privacy conscious, so we'll only send a notification
or save a picture to the cloud when your pet does something
that you care about.
Otherwise, your video and your photos
will stay entirely on your local device.
I'm building this pet cam with my colleague, Jason Mayes,
a developer advocate and host of the YouTube show,
"Made with TensorFlow.js."
Hey, Jason.
JASON MAYES: Thanks for having me, Dale.
You know, we've got a ton of old devices
like smartphones just sitting around doing
absolutely nothing.
So I really wanted to think about how we can give those
a new life in a way that anyone can then replicate
and hopefully use to do something useful.
DALE MARKOWITZ: So why a pet cam?
JASON MAYES: Yeah, that's a great question.
So essentially, I saw a lot of people around me
these days buying kittens and puppies.
And I think it'd be quite fun to kind of track those
as they're growing up to make sure we're not
being too naughty when you're leaving the house
when they're alone.
And of course, you can always extend this
to other objects in the future too as well.
DALE MARKOWITZ: Kind of like a chicken.
JASON MAYES: Yeah, like a chicken as well.
Anything could be possible.
DALE MARKOWITZ: All right, let's do it.
OK, so our app's going to work like this.
First, we'll build a pet tracking camera
app that runs on your phone and detects
common pets like cats and dogs.
Then we'll make our app tracker even more powerful
by training a custom pet model on Google Cloud
to detect chickens.
[CLUCK]
And finally, we'll connect our front end
to a back-end built-in Firebase so
that we can store data and send messages to Slack.
JASON MAYES: OK, so let's talk about
the actual front-end application that tracks animals.
The first step is to enable the webcam access
and load a TensorFlow.js model for object detection.
Today we'll use the COCO-SSD model
that recognizes 80 common household
objects in any picture that you pass through it.
Now, importing TensorFlow.js and the machine learning model
is actually really simple.
First, we've got two script tag imports for TensorFlow.js
and the model itself.
Next, we can grab a reference to the video object
that we're rendering our webcam to.
And once we've got that, we can call COCO-SSD.load.
We can then call model.detect and pass it the video elements
on our web page to classify the current frame being
played from our webcam.
So once we can detect what's in a given video frame,
we then need to send an alert when the pet, in this example
a dog, gets close to an object that we care about,
such as a food bowl, couch, or a bed.
Here, we simply allow the user to choose
from the objects a model can recognize to make
their selection as shown.
And we ask the user to ensure the device is set up
in front of the object we want to monitor once enabled.
Then, when we've detected that the user has placed the device
in view of the object they want to monitor,
we can continuously check the video stream
to see if a chosen pet ever gets close to that object.
To do that, we calculated the distance
between their bounding boxes.
If a distance is less than some threshold,
we'll send an alert to our back end
to then send a notification.
Now, we also had to add some logic
to avoid spamming the user with too many notifications in case
the dog came in and out of frame continuously,
which you can learn all about over in our blog post.
So check the link in the description after.
And that's essentially how the front end works.
If you want to play with yourself and grab the code,
you can find it over on Glitch.com as shown.
DALE MARKOWITZ: So after Jason built the front end,
we can start testing the app.
But the problem was that Jason and I actually
don't own any pets, so we had to get kind of creative
with our testing setups.
JASON MAYES: And I also borrowed Benjy the corgie from a friend
to check couch and ball alerts for dogs,
which caught him every single time, as you can see.
It actually went better than expected.
DALE MARKOWITZ: But then, in the middle
of working on this project, I actually
did acquire some pets--
two baby chicks.
But despite how incredibly adorable they were,
our pre-built TensorFlow model didn't really recognize them,
so I had to figure out a way to adopt pet cam
to work for my chicks.
Which brings us to part two, training a custom chicken
detection model.
I did this using Vertex AI, Google's new machine
learning development platform.
Look at these cute little data points.
AutoML Vision is a tool within the Vertex AI platform
that lets you build a vision model without writing any code
and then export it to TensorFlow.js.
First, in the Vertex AI console, I
created a new image object detection data set.
Then I uploaded about 100 pictures of my new baby chicks
to Google Cloud Storage so they could
be imported into Vertex AI.
After the data imported, I opened an image
and annotated it by hand.
I decided to train the model to recognize Moira and Millie
separately because they're two different breeds of chicken
and I thought it would be super cool if the model could
tell them apart.
I also annotated the position of their food and water
bowl so that I could calculate when the chicks were
eating and drinking.
Then I just did this 100 more times,
and it took about 30 minutes.
When I was done labeling, I hit Train New Model
and then selected AutoML Edge to create a model optimized
for an Edge device.
I named my model, selected some configuration options,
configured it to train for five node hours, and then waited.
When my model was done training, I checked out the precision
and recall in the Evaluate tab, and this model
was pretty accurate.
To actually use it in my app, go to the Deploy and Test tab,
click TensorFlow.js, and then choose
a location in cloud storage to export the model.
From here, we can download it and use it
right in our app's JavaScript front end.
All right, now let's connect the back end.
To make this app actually useful,
we want to save these pet events when they occur
and then send an alert to the user over Slack.
For this, I of course turned to my favorite serverless tool
kit, Firebase.
When the front end app detects that a dog's on the couch,
it saves a snapshot and some metadata about the event
to Firestore.
Then I built a little app, a front end,
to pull data from that Firestore database
and then display it as a sort of diary
so that you can keep track of all the cute or mischievous
things that your pet is doing while you're gone.
To set up Slack notifications, I use a Firebase Cloud Function
to call the Slack API.
It's set up with a trigger so that when
data is written to the Firestore database,
it automatically triggers my Firebase Cloud Function
to run and then send an alert.
And that's it.
That's the whole backend.
If you want to learn how to build a pet cam yourself,
check out the code in the blog post linked below.
JASON MAYES: You can also check out
TensorFlow.js and other premade models
you can use on our websites.
And of course, if you want more inspiration,
check out #MadeWithTFJS hashtags on Twitter and LinkedIn.
DALE MARKOWITZ: By the way, if you
want to see how my colleague [? Marcue ?]
built his very own pet cam that runs on device
as well using Coral, then check out his take
on a very similar project here.
I guess great minds just think alike or something.
And if you do try this yourself, make
sure you let us know in the comments.
See you next time.
[MUSIC PLAYING]