[MUSIC PLAYING]
DALE MARKOWITZ: Father's Day is about a week away,
and like a lot of people, I'm not
going to be able to see my dad this year.
He's in New Jersey, and I'm in Texas.
So I decided I would try to build him something
that's easy to share digitally.
A machine learning-powered searchable video archive.
And at the end of this episode, I will share with him,
and we'll see what he thinks of it.
[SPINNING REELS]
My dad has definitely always been the primary video man.
My dad's first video was the day that I was born.
So he has his camera on me in the bassinet,
and I'm looking very red and gooey.
And then my mom has just given birth.
And it's amazing that she allowed
herself to be on camera.
And since then, he's just been taking tons and tons of video.
So a couple of years ago, my dad uploaded all of our family
footage to Google Drive.
So I decided to build my dad an AI-powered video archive,
which would allow us to actually search through all the videos
and find the most interesting bits.
Here's the searchable video archive
that I ended up building, that allows
you to search text and transcripts and objects
by term.
Let's take a look at how I built it.
The first step of this project was
to transfer all of my dad's videos from Google Drive
to cloud storage.
This way they'd be accessible to the rest of my GCP project.
If you're using files on your desktop,
you can just upload them directly from within the UI.
But if you're transferring files from Google Drive,
check out my blog to see the hack I used
to make uploading much faster.
Next, I analyzed all those videos
using the Video Intelligence API,
which is the main AI workhorse at the center of this project.
This tool takes a video as input,
and it spits out lots of different annotations,
like what's actually going on in an image.
Is it a wedding?
Is it fireworks?
A cute dog?
[BARK]
It uses computer vision to detect objects and events
in videos.
It also transcribes the audio track from videos
so that we can later search by what people say.
This is how I was able to search for my very first steps.
DALE'S DAD: That's Dale walking.
This is the first time she's taken
major steps on this floor.
DALE MARKOWITZ: It can also grab text shown on screen,
so you can search for things written on T-shirts or posters
or on signs.
Or on birthday cakes--
this is how I found my first birthday.
I also used this feature to grab the timestamps that
appeared on screen in some of my dad's older videos.
Once I extracted all of these searchable features,
I could use them to quickly search through those videos.
Now, using the Video Intelligence API
is pretty straightforward.
You just import the Video Intelligence library
and call the Analyze Video function,
passing a path to the location of your stored video in cloud
storage.
Now, because my video files were massive,
I tried to do all of this processing
exclusively in the cloud, because downloading things
onto my desktop would have been really slow.
I chose an architecture that looked like this.
First, like I showed you earlier,
I uploaded my videos to a cloud storage bucket, which
is like a folder in the cloud.
I used uploading a video to trigger a Cloud Function.
In that cloud function, I called the Video Intelligence API,
which analyzed the uploaded video
and wrote the data to Firestore.
Finally, I needed a way to quickly sort
through all of this data.
So for this, I used a handy non-Google tool built
by a company called Algolia.
It's a search tool that lets you upload JSON,
and then offers an API to search through it quickly.
It handles things like correcting for typos
and finding the most relevant results.
So that was pretty much how I built the back end.
To build a front end or our user interface,
I decided to learn how to use Flutter, which is a Google
framework for building apps.
And I built a Flutter web app, which looks like this.
Of course, there are tons of ways
you could build a front end without Flutter, using
JavaScript or for IOS or Android, your choice.
And that's it in a nutshell.
If you want to learn more about the technical details
of this project or to see the code,
make sure you check out my blog post linked in the description
below.
All right.
I guess it's time to show this app to a real user.
Let's show it to my dad.
So, Dad, I built something for you for Father's Day,
and I'm really excited and also really nervous to show you.
DALE'S DAD: Uh-oh.
DALE MARKOWITZ: It's probably the most high-effort Father's
Day gift I've ever worked on.
DALE'S DAD: Oh, look at all this.
I see Scooter and Monticello, so I think
I know where this is going.
Oh, wow, look at all this.
Wow.
I know all these pictures.
DALE MARKOWITZ: So I took all of your video in Drive,
and I analyzed it with machine learning so that, first of all,
I split all of your really long two-hour clips
into little chunks, and I made them searchable.
So in addition to these ones, Father's Day and Performance,
you can search by anything that you want.
DALE'S DAD: That's interesting.
So can I search for, let's say, we used to play ping pong.
You think the ping pong table might show up?
DALE MARKOWITZ: Yeah, try it.
DALE'S DAD: Oh, look at that!
There it is.
I remember when I--
when we bought your first bicycle, how excited you were.
So how about-- wait, this--
look it here.
This is what I was looking for.
This video right here.
But it brought all these other bicycles in, as well.
How neat.
Every one of these pictures has the bike in it.
But this is the specific one that I was looking at.
So, yeah, I would have definitely found it.
Very neat.
So you really could be very loose with your search terms
here.
You know, I guess the looser it is, the more likely
you are to find it, and then you can just zoom in.
I guess you can get specific, if there's a lot of occurrences.
This is very nice.
Thank you very much for the early Father's Day gift.
I can't wait to show Mom.
DALE MARKOWITZ: Of course.
That's all I wanted to show you.
Thanks for filming with us.
DALE'S DAD: OK.
So how do you need me to end this?
DALE MARKOWITZ: Oh, I have an idea, I have an idea.
Can you search for Fade Out?
DALE'S DAD: Oh, yeah, I said that all the time.
I think at the end of every video we said that.
Let's see, if I--
let's see how long this one is.
OK, I'm going to fade out, pussycat.
DALE MARKOWITZ: Can I go on the bike?
DALE'S DAD: We're not.
DALE MARKOWITZ: We're not.
DALE'S DAD: You sure?
You sure we're not going on the bike?