[MUSIC PLAYING]
PAUL RUIZ: Hey, y'all.
I'm Paul Ruiz, a developer advocate with Google.
And this is the ML on Android with MediaPipe series
where I'll introduce some machine learning concepts
and their application on Android with a MediaPipe framework.
In this video, I'm going to focus on audio and how
you can perform audio classification in your Android
apps with MediaPipe tasks.
Audio classification can be used for quite
a few different things, including determining a music
style, different animal sounds, or specific spoken words
to name just a handful.
Before we really dig into how to use audio classification
in an Android app, we should go over
how audio data works so you can have a better
understanding of what's happening
when writing your Android code.
Two of the more common ways to visualize audio data
are waveforms and spectrograms.
Waveforms show sound pressure or the difference in pressure
from a sound compared to the ambient area pressure,
though really for our purposes, you
can just think of this like volume over time.
While this is kind of cool, it doesn't really
give us enough of an understanding of the data
to do what we want to do with it.
Two similar sounds may have waveforms that look
almost identical, such as a siren,
and a musical instrument that only plays one drawn-out note.
Spectrograms, however, add frequency to the mix,
giving us just enough information
to be able to tell the difference between two
unique sounds.
When you're looking at a spectrogram,
the x-axis still represents time,
but the y-axis now represents frequency, or pitch,
while the color represents the sound pressure.
Once you have this visual representation,
it turns out that audio classification
happens using a lot of the same techniques
that image classification uses.
So let's take a look at a couple of examples using
the 3D spectrogram Chrome experiment, which
I'll link in the description for this video.
[HARP SOUND]
If you select the harp sound, you
can see how the spectrogram-active areas
descend as the instrument plays lower and lower notes.
Whereas if you say the word right,
you'll get a sound with two immediate peaks
in the low and mid frequency ranges.
As you say "rye," then you'll notice
a flick of high-frequency sound when you add the "t" sound.
What's cool about this is that even if you have
different people say the same word,
you will still get a similar enough pattern
to be able to classify the word.
Now that you have a better understanding
of how to represent audio data in a visual way,
let's think about how a generic audio classification model
would approach the problem.
The model we will use in this video is called YAMNet.
And it attempts to classify audio
in segments that are 0.975 seconds in length.
As audio comes in, you will accept a segment,
attempt to classify it, return the results,
and then move on to the next segment that has been recorded.
But what if the sound we're trying to classify
appears across multiple segments,
rather than cleanly within one?
This is actually what we see on the left side of this image.
That's where a concept called overlap comes in.
Basically, you have two separate classification streams
happening at once.
But they're offset from each other
to try and catch those edge cases.
For example, you could have 50% overlap,
meaning the second set of segments
will start from the middle of the first set.
And they'll have a better chance at finding
sounds that are missed by the first classification stream.
With that out of the way, it's time
to take a look at how you can do this in your Android apps
with MediaPipe tasks.
We'll start out by importing the necessary library
through your build.gradle file.
You can find the current version in the official documentation
that's listed in this video's description.
If you've been following along with this series,
then creating the audio classifier
should look fairly similar to what you've
learned about in a past video.
You will start with a base options builder
where you can set the model path.
Following that, you can create the classifier-specific
options, which are where you will
set the minimum acceptable score threshold that must be passed
before a result is returned to the rest of the app, the max
number of results that you can expect back,
and the running mode.
In this example, you will learn about live streaming
classification, though you can also accept an audio file
from the user's device.
And since live stream classification
happens asynchronously, you will need
to associate a set of listeners with the builder
before creating the final options and classifier object.
This next step is where things start to diverge from the image
classification example.
To perform audio classification, you
will need a way to record audio.
For this example, you can use Android's audio record object.
Since there's a variety of different options here,
we should also take a moment to try and understand
what they're doing and why we need them.
The first parameter sets the default audio input hardware
for the device.
The second is sampling rate.
This represents the number of data
points an audio clip should contain.
So for this example, you will use
a value of 16,000, which means that each second of audio
should consist of 16,000 data points that can be analyzed.
The channel config value is set to use the default microphone
from the device.
Finally, you will need to define the byte buffer size.
Say that you know the sample rate for one second,
but the model only accepts clips in 0.975 seconds in length.
You will need to multiply those together and convert
them to bytes to initialize the audio buffer.
You might also notice that there's a buffer size
factor included in here.
This increases the buffer size to account
for any potential errors in recording.
And for this example, you can sample use a factor of 2.
With that created, you can start the recording
and kick off a runnable that will pull data
from the recorder buffer to classify it.
One thing worth noticing here is that you
can set the reasonable interval to account for overlap by using
the length of an audio record and modifying it
based on the percentage of overlap that you want to use.
After all of that, this classification
is done by creating a new audio data object,
loading the audio recording buffer into it,
and then the audio classifier that you created
earlier has a classify async function that
will do the inference for you.
At this point, you just need to accept the results
in the callback you created earlier and then do
something with them.
For our sample on GitHub, we display classified sounds
and their confidence like you can see here.
In the next video of the series, you
will learn about natural language processing
and how it can be used for classification, or finding out
how similar two blocks of text are,
and how you can use both of those features in your Android
apps.
So I'll see you there.
[MUSIC PLAYING]