- Hi, my name is Markku. I’m a solutions architect at Google Cloud.
In this session, I will show you a real-life example of Edge AI.
A large car dealership has many visitors and employees
and wants to understand people and spaces but in a privacy-centric manner.
Doing so in the past likely required counting humans manually.
This is both error-prone and expensive.
Today, we will show you how the dealership was able to understand occupancy
and improve safety with Google Cloud, Coral, and a technology partner called BlueZoo.
Let’s build this today.
Our goal is to count the number of employees or customers entering/exiting the premises,
but to do so in a way that protects their privacy.
With Edge AI, we can feed the raw camera feed straight into a machine learning model.
A custom machine learning model that we have trained to detect patterns.
One pattern could be an employee leaving through the door,
and then the output of this machine learning model
are simply counters or numbers how many times these events happened.
The raw camera feed or images are never stored anywhere,
never transmitted out of the system.
All we get out of the machine learning model are these numbers,
how many times these events happened.
And then when we transmit these numbers,
we are transmitting highly actionable insights of semantic data.
And also the bandwidth needs at the site are greatly reduced
because we never transmit megabytes of camera feed.
We simply transmit bytes of actionable insights,
and this is how we could build such a solution.
On the left, we have a camera which can be any camera,
and then we have the Coral Development Board,
which has the Google Edge TPU machine learning accelerator.
To train the custom machine learning model, we can use Vertex AI’s AutoML service.
And then when we run inference at the edge next to the camera,
we can transmit the actionable insights
or inferenced results to Google Cloud using IoT Core service.
IoT Core will pass the data to cloud Pub/Sub, which is our message queue,
and then those payloads can trigger Cloud Functions
which can write the data in streaming mode to BigQuery, the data warehouse.
And then finally, we can visualize the results using Looker.
Okay, let's build this step by step.
First, we need to record example training videos and label them.
So first, we will use the Coral Development Board
and a USB webcam to record some tests for it
and then label the example dataset in Vertex AI.
Okay, so I recorded myself walking in and out through the door several times,
and I then uploaded these as training videos here to Google Cloud Storage.
And then, in Vertex AI, I added them as a dataset.
Here you can see previews of those five videos.
Then what we need to do next is to help the system identify the events
by labeling the interesting objects.
So, first I want to try labeling the events for person walking in and person walking out.
So here I am walking in. That's good.
Now we can use the console to then draw a bounding box
around the object like so,
and then we can choose which label this one is corresponding to.
This one is an event for person in. We select that, and there we go.
We have now added one label.
So we should do this for all the labels at least 100 times or 100 examples per label.
Now that we have labeled examples, we can simply train a custom model
that recognizes these types of events, somebody entering or exiting.
Once you have labeled a lot of examples, you can click on the Analyze tab
to see how many labels you have.
And if you have enough, at least a 100 per label,
then what you need to do is click on Train New Model.
It's as simple as that.
We will select the type of model we wish to train.
I will select AutoML Edge
because we want to run it at the edge on our device.
We give it a name, and we can either randomly assign
the training and testing data or manually file by file.
I will choose the randomly assigned one
and then the type of model I wish to train, I wish to run it on Coral,
devices with the Google Edge TPU,
and I wish to have the higher accuracy version of the model.
Then all we need to do is click Start Training,
which starts the training procedure and the service in the backend,
and now is a good time to go for lunch or dinner.
Okay, we have received some new email from Vertex AI.
The training has been completed. Okay, here we have the trained model.
It took about two and a half hours to train in real time,
but 32 hours of machine time because it scaled it out.
Okay, let’s now export the model and select the Edge TPU TensorFlow Lite,
Compilation as the option.
We will need to select the destination bucket and folder and click Export.
And we will get the labels file as well as the TensorFlow Lite model file as an export.
Then let's now test what really happens with this new model.
Let me now process one of the example videos that I recorded of myself
walking in and out using this new model.
And here in this development and testing mode,
I will use the HDMI output of the development board
to see what the model sees.
But in production, we will disable the camera view
and only get the numerical counters from the model.
But let's first see if this model works.
Okay, so we load the new model we trained. There's me walking out.
You may have seen a short bleep there.
It said person out. It drew a bounding box around me.
Let’s wait a few seconds to get the numerical counters,
which we will then be sending to Google Cloud later.
There we go. And there is an employee out event.
So I labeled myself as an employee here. Now I’m walking in. Let’s pay attention.
Okay, person in.
And I mapped this person in to an employee in output.
And here we go, employee in, one. Okay, so this seems to work.
It seemed to work in our internal testing. Now let's put it to the real test
at the car dealership.
And also, let’s start streaming the machine learning inference results to the Cloud.
To collect these occupancy counters, we can now connect this edge device securely
to Google Cloud using IoT Core, and then we can stream the data payloads
all the way to a table in BigQuery.
Since our internal testing worked, we trained a new model
that is able to classify between employees or customers entering or leaving
based on the uniform that the employees wear.
So this new model has four numerical outputs:
counters of employees in and out and customers in and out.
Then let’s take a quick look at the destination BigQuery table.
Here's the table schema holding the occupancy counters.
And if we query the table, we can see that there is no data.
There are no rows in the table at the moment.
So let’s now run this model through a real test:
with testing footage taken at the dealership.
I will start the client upon the Coral Board.
This footage was recorded with one employee and two customers entering the building.
So let’s wait a bit for the events to happen,
and then every 15 seconds, the client will send summaries.
And I also add it now the IoT flag here
so that the client will transmit the results to IoT Core.
There we go.
Transmitting inference summary to IoT Core, and what did it transmit?
One employee in and two customers in.
All right, that looks good.
So where did those data go?
Well, first they were received by Cloud IoT Core.
Here we have the device, Coral AI Cam, and it’s part of the registry,
which has a Pub/Sub topic, so when the devices transmit data,
IoT Core will receive them and drop them in this Pub/Sub topic, which is here.
And then we have a simple Cloud Function that is triggered by that same Pub/Sub topic.
The function is extremely simple; it will just take the JSON payload
and write it into BigQuery.
So let's now see if this Cloud Function was actually triggered.
So let’s refresh, the Cloud Function looks, and find the latest entry.
Our client was transmitting results just now. Let’s see what happened in Cloud Functions.
There we go. This is the latest entry from just now.
We have the same data JSON that was output
by the machine learning inference here on the client side,
received by the Cloud Function from Pub/Sub,
and then it was written hopefully to BigQuery.
So let's now go to BigQuery and run again the select statement
to see if we have any data in the table.
And there we go.
So the same data package that was transmitted by the client with numerical values
telling us that one employee has entered and two customers have entered
during this 15-second time period here.
So now the occupancy counters are in the data warehouse ready for analytics.
As a final step, we can then merge
this machine learning inference data in BigQuery
with our technology partner BlueZoo’s data, which is based on occupancy sensors
and visualize those.
The visualization is built on Looker using BigQuery as the data source.
And in BigQuery, we have two tables.
The first one is from this system using machine learning inference,
and the second one is from BlueZoo using their occupancy sensors.
And then we merge and visualize that combined data.
So here we have the dashboard.
So in the map of Southern California, including San Diego.
Let’s zoom in to see the car dealership.
And then we can switch to the floor plan to see the sensor placement here.
Here we have the different areas of the car dealership,
including the service area, the parts area,
used car sales, new car sales and showroom,
the used car sales parking lot, and the new car parking lot.
The dealership has three BlueZoo WiFi sensors
for coarse-grain occupancy detection, including one at the used car display lot
and one at the new car display lot.
Additionally, there are 21 Coral-based optical sensors
over each of the doorways for fine-grained occupancy detection
both for visitors and employees.
Here's entrance one, and here's the AI camera for entrance two,
and here is the service entrance.
Now let's scroll down to take a look at the occupancy graphs.
Here, the bar chart shows the occupancy of most of the recent week.
By default, the chart shows the occupancy of the previous week with the green line.
Now, Friday looks a bit strange with the search later in the day.
Let's take a look at the peak occupancy hours.
Hm, the current week total Friday visitors is 322 but previous week was 501.
We can also show the previous four weeks
to get an idea of the variability of the occupancy.
Now let's take a look at the dwell time of the visitors.
The two drop-downs here let us see first the wide area sensors
and then the threshold sensors, which are the optical ones.
So here we have the WiFi sensors, and here are the AI camera optical sensors.
So let's drill into one of the areas, the new car show room.
Not surprisingly, the peak visit day is Saturday.
The number of visitors in the sales show room
is obviously important for measuring the impact of promotional campaigns
and for scheduling sales staff.
The number of visitors in the service lounge
tells us when the lounge might exceed capacity.
And finally, if the sales closing rooms
are not being occupied during much of the day,
we may have a closing problem.
That was a quick look at how you can use Edge AI to understand occupancy
and to improve safety in insured spaces.
I really hope that you enjoyed this session. Thank you for watching, and bye-bye.