(upbeat music)
- Hello, and welcome to this AI/ML demo derby session.
What we’ve got for you today is three back-to-back demos
showing off interesting ways to use machine learning to solve problems.
I’m gonna be kicking things off
with a quirky project that will help my friend watch movies more safely.
Let's get straight into it.
The project I named Phobia Safe Videos with Machine Learning.
This sounds a bit weird, so let me give you a bit of backstory of what this is about.
A while ago, I was watching some movies with some friends
and I discovered that one of my friends actually has a very serious phobia of swans.
Now, it’s not actually swans, but I want them to watch this video & feel safe watching it.
So we were watching some classic movies,
so everyone else knew exactly when the swans were going to appear.
We would just tell our friend exactly when to look away
and then they wouldn't need to get super scared.
Now this worked pretty well
until sometimes we got it wrong and people got upset.
I figured I could potentially use machine learning to solve this problem.
There’s actually only a couple of tools that we need to solve this problem.
And one of them is called the Cloud Video Intelligence API.
This is a very powerful API
that allows you to detect what’s in a video using machine learning.
You don’t need to know anything about machine learning to use it.
You just feed it a video and it tells you what's inside the video.
It has a couple really interesting features that you can find out more about in the docs.
The next tool that could be useful for this problem is called the transcoder API.
Now, this is a very powerful video processing API
and a classic example of this is on YouTube
where you’ve got the option to watch a video in different levels of quality.
You’ve got standard definition, high definition, 4K.
This API allows you to feed in a raw video
and it will automatically output all the different formats
for a streaming over the web all for you. It has this,
and it has a couple other really interesting features
useful for video processing on the web.
So looking at these two tools, there is a couple of interesting features
that we can just plug together to potentially solve this problem.
We’ve got the label detection feature of the video intelligence API
that will tell you what’s in a video during what time segments.
For example, if we want to detect swans it’ll potentially tell us
from 10 seconds to 15 seconds, there are swans in the video.
We can use these times segments to then, in the transcoder API,
inject a full screen overlay to hide the swan throughout the video.
The inject overlays feature is classically used
for things like injecting watermarks.
But, I'm going to use it in this case to hide the entire screen.
So let's look at the demo in action.
All right, so we’re here in the Google Cloud platform.
This is my demo, and let’s just get into it and upload a scary video file.
Okay, so the video file is uploaded.
Let’s have a look at what it looks like.
So we can see it’s a nice, calming video.
There’s a plane taking off. Oh, there’s a guy with a banjo.
And oh my God, look at that. There's a scary swan.
So this is the kind of thing we don’t want. We want this to be fixed.
How’s that going to happen? Well, I have a cloud function
that it will trigger
based on the bucket that I just uploaded that video to.
And this cloud function, all it’s doing is taking the video
and running it through the cloud video intelligence API
and running label detection.
And it should output the labels to this next bucket.
And there they are.
So this is a JSON file with all of the things
that the video intelligence API spotted in our video,
along with the time segments that it found them for.
And now that this landed in this bucket, we have our final cloud function,
which is going to scan through all of those labels
and pick out the ones that feature the word swan,
and for the time segments that it detected those to appear,
it will feed that to the transcoder API to then hide the screen.
So let's see if it's done that.
And here it is.
So this should be our final safe-to-watch video.
Let's have a look.
We can see it starts off the same.
We have a relaxing plane taking off, and then a friendly man with a banjo.
And, oh, look at this.
Where I previously got startled by a swan, it has been hidden from the video view.
So that's it. That is the full demo video pipeline
that will make videos safe to watch based on anyone’s individual phobia.
Okay, so a quick recap of how things are plugged together.
I take the raw video,
and that gets fed through video intelligence to run label detection.
When I’ve got those labels, I then scan through for things I’m afraid of
and use the time segments to inject overlays using the transcoder API,
which then outputs a new safe-to-watch video
with all the scary things hidden.
A point to note is that you can’t teach the cloud video intelligence API new things
because it’s all pre-trained and static.
If you have something more specific that you’re afraid of
that video intelligence API doesn’t detect
you can train your own model using a tool like AutoML video classification.
For more details and code samples of how to do that,
check the description below.
I’ll hand off to Markuu, who will show you what he’s built.
-Thank you. -Hi, my name is Markku.
I'm a Solutions Architect at Google Cloud.
In this session,
I will show you a real life example of Edge AI.
A large car dealership has many visitors and employees
and wants to understand people and spaces but in a privacy centric manner.
Today we will show you
how the dealership was able to understand occupancy
and improve safety with Google Cloud, Coral,
and a technology partner called BlueZoo. Let's build this today.
Our goal is to count the number of employees or customers
entering or exiting the premises,
but to do so in a way that protects their privacy.
With Edge AI, we can feed the raw camera feed
straight into a machine learning model,
a custom machine learning model that we have trained to detect patterns.
One pattern could be an employee leaving through the door.
The output of this machine learning model
simply counters or numbers how many times these events happened.
The raw camera feed or images are never stored anywhere,
never transmitted out of the system.
And this is how we could build such a solution.
On the left we have a camera, which can be any camera,
and we have the Coral Development Board,
which has the Google Edge TPU Machine Learning Accelerator.
So first we will use the Corel development port
and a USB webcam to record some tests for it.
(light music)
Okay.
So I recorded myself walking in and out through the door several times.
And then in Vertex AI, I added them as a dataset.
So here I am walking in. That's good.
Now we can use the console
to then draw a bounding box around the object, like so,
and then we can choose which label this one is corresponding to.
This one is an event for person in.
Then what you need to do is click on Train New Model.
It's as simple as that.
I will select AutoML Edge and then the type of model I wish to train.
I wish to run it on Coral devices with the Google Edge TPU,
and I wish to have the higher accuracy version of the model.
Okay, let’s now export the model
and select the Edge TPU TensorFlow Lite compilation as the option.
Let me now process one of the example videos videos that I recorded of myself
walking in and out using this new model.
Okay, so we load the new model we trained. There’s me walking out.
You may have seen a short bleep there.
It said person out. it drew a bounding box around me.
There we go.
And there is an employee out event. So I labeled myself as an employee here.
Now I’m walking in. Let’s pay attention, okay?
Person in, and here we go. Employee in, one.
To collect these occupancy counters, we can now connect this Edge device
securely to Google Cloud using IoT Core.
And then we can stream the data payloads all the way to a table in big query.
Here’s the table schema holding the occupancy counters.
And if we query the table, we can see that there is no data.
So let’s now run this model through a real test
with testing footage taken at the dealership.
This footage was recorded with one employee and two customers entering the building.
There we go.
Transmitting inference summary to IoT Core. And what did it transmit?
One employee in and two customers in.
So when the devices transmit data, IoT Core will receive them
and drop them in this Pub/Sub topic.
And then we have a simple cloud function that is triggered by that same Pub/Sub topic.
The function is extremely simple. It will just take the JSON payload
and write it into BigQuery. And there we go.
The same data package was transmitted by the client with numerical values
telling us that one employee has entered and two customers have entered
during this 15-second time period here.
The visualization is built on Looker
using BigQuery as the data source, and in BigQuery, we have two tables.
The first one is from this system using machine learning inference.
And the second one is from BlueZoo using their occupancy sensors.
And then we merged and visualize that combined data.
Here we have the dashboard.
So in the map of Southern California, including San Diego.
The dealership has three BlueZoo wifi sensors.
Additionally, there are 21 Coral-based optical sensors
over each of the doorways for fine-grained occupancy detection,
both for visitors and employees.
Here the bar chart sourced the occupancy of most of the recent week.
The number of visitors in the service lounge
tells us when the lounge might exceed capacity.
And finally, if the sales closing rooms aren’t being occupied during much of the day,
we may have a closing problem.
I really hope that you enjoyed this session.
Thank you for watching and bye-bye.
- Hi, I’m Kaz, Developer Advocate from Google Cloud.
Today, I would like to show a tool called PDF to Audio Book.
The problem was I had too many downloaded PDF books.
Also, I wanted to lose my weight.
How can I solve these problems at the same time?
My solution was to build a tool
that converts the PDF books into audiobooks by using cloud AI,
so that I would listen to the audiobooks while I'm running or exercising.
So let’s take a look at how PDF to Audio Book works.
You can upload your PDF file on a Google Cloud storage bucket
and wait for 10 to 15 minutes. So you have the multiple MP3 files generated.
This is the audiobook content. No server management is required.
Cloud function is suitable for building a small automation tool
like PDF to Audio Book.
For implementing the work flow of PDF to Audio Book,
I’ve written a Python code like this.
I have defined a function called p2a_gcs_trigger.
That takes the file object or PDF file uploaded on Cloud straight
and passes it to the OCR function.
To register this to the cloud functions, you use Gcloud functions deploy command
by specifying the function name,
run-time name, and trigger bucket name and so on.
Once you upload the PDF file on the trigger bucket on cloud storage,
the whole process will begin.
It started with an OCR with vision API,
and then it extract features from the OCR result
and called AutoML Table to support extracting the body texts.
And finally, core text-to-speech API for generating the speech.
Let's take a look at the first step. Vision API.
This API has a wide variety of visual analytics features
and one of them is OCR, a text detection feature.
For example, if you upload a single page in a book to the API,
you will get the OCR results like this.
These texts are grouped in different paragraphs.
The API returns the result
as JSON format with various parameters of each paragraph
such as the paragraph text,
X and Y position of the paragraph, and et cetera.
To use Vision API from your application, you'll write a code like this.
At first, use space by the URI of the PDF file on cloud straight
and then switch find a URI for saving the result JSON file
and code the API.
So now you have converted the PDF book into text
but that's not the whole story.
As you can see,
there are many parts that should not be included in the audiobook,
such as page headers, diagram labels, sample code, and page numbers.
We needed a way to detect if each paragraph should be included or ignored.
For this classification I have used AutoML Tables.
This is a service allowing you to create your machine running model for tabular data
such as database tables or spreadsheets.
AutoML Tables, as its name says,
automates a major part of ML development process.
So all you have to do is upload in the training data,
start the training, and then wait for some hours.
Under the hood, AutoML Tables does the necessary tasks for data scientists,
such as feature engineering, model selection, and hyperparameter tuning.
For PDF to Audio Book,
I have corrected the features of the training data from the OCR result.
That includes the paragraph text,
character numbers, width and height,
x and y position of the paragraph, and so on.
So that we expect the equated ML model will classify
whether a paragraph is a body text or use it as garbage text.
Let’s take a look at how AutoML table works.
You can start creating the data set at first
and choose your tabular dataset and create it.
And you can upload your local CSV file by specifying the file name of the CSV file.
And press the continue button.
And then you can generate statistics of the uploaded features,
so that you can check out if the features
are going to be uploaded with the expected distributions,
and start training the model,
specifying the budget for training hours and wait for four hours.
After the training is finished, you will get an evaluation report like this.
It looks like it gets pretty high accuracy.
The report also includes a confusion matrix
where you can find the accuracy for detecting body text is 95%,
and the accuracy for the garbage text is 97%.
This isn't so bad.
If you take a look at the feature importance,
you can learn paragraph text is the most important feature
for this ML model.
Finally, I have used text-to-speech API for generic in-speech data.
I have defined a function like this
where you are adding SSML tags to each paragraph,
so that, for example you’ll hear longer break at each section header.
The reason they’re able to easily understand that there's a new section starting.
So that was the quick view of PDF to Audio Book.
With this solution,
I was able to listen to the majority of our PDF books.
Unfortunately, the technology doesn't seem to be working
for my weight loss.
If you’re interested in this, please take a look at the documentations for each product.
Thank you so much.
(upbeat music)