[MUSIC PLAYING]
PAUL RUIZ: Hey all.
I'm Paul Ruiz, an engineer with the Developer Relations
Team on Google ML.
And this is the first video in the ML
on Raspberry Pi with MediaPipe series, where you will learn
about the basics of Machine Learning,
along with how you can use Google's newest
On-Device Machine Learning tool, MediaPipe, to add
useful features to your own Raspberry Pi apps.
This series will aim to introduce some general concepts
and vocabulary around Machine Learning, how
to create a simple model, and how to implement some common ML
tasks, such as hand gesture recognition,
audio classification, and object detection
all on the Raspberry Pi.
By the end of this series, you should
have enough of a foundation in Machine Learning with MediaPipe
to continue learning and making great apps
without getting too far off track
with all of the math or underlying details.
If you are interested in a lot more
of the details behind Machine Learning,
I've also included a link in the video description
to our TensorFlow Zero to Hero video
series, which is similar enough to MediaPipe for our purposes.
So let's go ahead and start by talking about what
Machine Learning even is.
With traditional programming languages,
you provide an input and a function,
then the program produces an output.
So, for example, you might have a function that
adds 2 to any input number.
So 1 outputs 3, 8 outputs 10, and 40 outputs 42.
With Machine Learning, you provide a set of known inputs
and a set of known outputs, and then the computer
tries to figure out what the function is
that connects those two.
This process is called Training, and it's
one of the core steps in Machine Learning.
Without getting too far into the details,
Training involves solving a set of complex problems,
including optimizing models to work
on relatively restricted hardware,
using specific device hardware for acceleration
and ensuring your models work across multiple platforms,
which is why we've created MediaPipe
as a way to make these challenges a whole lot easier
for our developers.
Once Training is done and your model is ready to go,
your next step is implementing that model on your Raspberry
Pi.
Though, we'll cover that in a lot more detail
both later in this video and in later videos
through this series.
So, just a few moments ago, I mentioned
that MediaPipe is a tool for On-Device Machine Learning,
but what does that even mean?
As the name implies, On-Device Machine Learning
is a subfield of Machine Learning
and is a way to run Machine Learning inference
rather than Training directly on an edge device.
This includes mobile phones and tablets, IoT devices,
and even web pages.
Because of the single attribute, developers
gain three specific advantages.
First, Machine Learning is able to run quickly
using models that have been specifically designed
for edge devices.
Next, your apps will be able to use Machine Learning
features regardless of whether or not
the device has connectivity.
For example, when a user or a device is in a tunnel
or in a rural area.
Lastly, because data is never leaving the device,
that data is more secure and private.
This has opened the door for a variety of new products
and features for developers, especially when
combined with all of the options and flexibility
that you get with IoT hardware, like the Raspberry Pi.
While discussing these new ML features
throughout this series, I'm going to divide them
into four separate categories.
Natural language processing, or essentially text,
which are tasks that you might use to translate text,
provide answers from a contextual paragraph,
or understand some dialogue.
Vision, which involves images or videos to recognize content,
track objects, or even provide a modified version
of visual inputs.
These tend to be the most commonly used features
and a few IoT use cases in this domain
include barcode scanning, item quality inspection, and object
sorting.
Audio files that can accept input
from a microphone or audio file in order
to classify sounds, input spoken language,
or check the similarity between two separate audio files.
And finally, Other for those one-off tasks.
These could be tasks for content recommendation, tabular
classification, or generative content.
This list on the screen is also just a subset
of what we either currently support or plan
to support with MediaPipe.
So be sure to check out the official documentation
at developers.google.com/MediaPipe
to keep up with the latest as we continue to improve.
Great.
So what makes MediaPipe unique in the On-Device Machine
Learning ecosystem?
MediaPipe handles a lot of the monotonous tasks
that go along with Machine Learning, like flow control.
Meaning you don't need to worry about your app
sending data faster than inference can happen,
handling the underlying data structures so that you can work
with regular objects instead of a series of matrices,
or chaining multiple models together
to form even larger jobs.
MediaPipe also consists of a selection of solutions,
such as MediaPipe Tasks, which lets the developers easily
deploy ML features in their apps,
MediaPipe Model Maker, which helps you create custom models,
and the MediaPipe Studio, which lets you try out tasks
directly from a browser without needing to write any code.
This video series will primarily focus on MediaPipe Tasks,
as it's the most relevant to IoT development.
Though, it's worth mentioning that MediaPipe Tasks also works
with Android, web, and iOS.
I will include links to content focused on MediaPipe Model
Maker and MediaPipe Studio in the video description for those
of you that are interested in learning more about those two
tools.
Plus, there will be a video later
in the series covering both.
All right, now that you know a little more about On-Device
Machine Learning, as well as what MediaPipe is,
let's go into something a little more exciting.
In this case, we're going to make a simple Python
program that performs image classification.
And I'm going to use a custom model that can
detect a small set of animals.
For a bit of background, I have a few chickens in my yard,
but one thing to know about chickens
is that a lot of other animals would love to eat them.
In this example, I've created a new model using a set of images
that I pulled from Kaggle, which you can find
in this video's description.
And I'm using that model to detect things
like raccoons, foxes, or cats.
If I were using this model in a full project,
then I might use that detected information to notify me
of anything coming around.
That way, I can know if I need to fix any parts of my chicken
coop or add any additional fencing.
So before you get started, you will need
to set up your Raspberry Pi.
The most important thing to know is
that you must run a 64-bit operating system on the device.
In this case, and throughout this series,
I'm going to use the standard 64-bit Raspberry Pi
OS on a Raspberry Pi 4B.
You will also need a camera hooked up to your Raspberry Pi
for this example, as well as any other computer vision examples.
Though, you can use either the Pi Ribbon camera
or a USB webcam.
The next thing you will need to do
is make sure that you have the MediaPipe dependency installed
on your Raspberry Pi.
You can do that with this command from the terminal.
And since this is On-Device Machine Learning,
you are going to need your Machine Learning
model stored on your device.
There's a variety of ways that you
can do this, including using the wget command from the terminal
to get a stock model that has been tested already.
Though, I'm going to simply place
my custom model in the same directory as my Python script.
If you do want to use a tested default model,
we provide a few options from our developer documentation,
which you can find in the video description.
Now that the housekeeping is done,
it's time to create your new Python script to do
the actual classification.
While I'm only going to cover the most important parts here,
you can find an entire example on GitHub,
which I'll also link to below.
First, there's a couple imports that you'll need
at the top of your program.
After you have those, you can create a configuration options
object for your image classifier.
Some of the available properties for these option objects
includes the max number of results
that you want to receive from the classifier,
the minimum confidence score that
must be met before a result is returned,
and a callback that will receive the results from image
classification.
In this case, you will use the live stream running mode
because this example is going to use the camera
to constantly classify against a video stream.
Once you have your configuration object made,
you can create the image classifier
object that will do the majority of the ML work for you.
Next, you'll get an image frame from the camera
and do some pre-processing on it,
including converting that image to the red, green, blue format
that the Machine Learning model requires,
as well as create a new MediaPipe image object that
will be used for inference.
With that all out of the way, you
can call the classify async method
on the classifier with that MediaPipe image
and a time stamp, which will return the classification
result to your callback function.
One nice thing about this is that because MediaPipe already
handles flow control for you, you
don't need to worry about sending camera frames faster
than they can be processed.
If you do happen to send more data than MediaPipe can infer
at the moment, MediaPipe will handle
dropping the oldest image to keep your processing time down.
Finally, this callback will receive a result object,
which you can use to do whatever it is that your goals are
with your new Raspberry Pi app.
The important part to know is the structure
for image classification results that you
can see here on the screen.
This is what you'll use to navigate through the result
object to find the information that you need for your app.
In this case, I'm just going to display the results
on the screen for the most likely seen item that
meets the score threshold that I already specified.
Here you can see that I'm able to point
the camera at a picture of a raccoon
and see that it's classified correctly with a 70% to 80%
confidence.
We're excited to see all the cool things
that you make with MediaPipe on the Raspberry Pi.
So let us know in the comments what you've made
or what you want to make.
And I'll see you in the next video.
Thanks.
[MUSIC PLAYING]