Welcome back to the OpenVINO channel In all the videos so far, we have talked about
image and video processing.
But OpenVINO can help you accelerate audio, NLP and many other usages.
This video is an ultra fast introduction to audio processing.
There are many better and more comprehensive full tutorials, this is really just to get
you started..
Before talking about Audio, a quick reminder on image processing
Here are 2 pixels of an image,
These pixels cannot help us much with understanding the full image.
Even if we have all the pixels, it is not enough
Because if we move this image by one pixel to any side, the way the computer represent
this image is totally difference.
And if the image is scaled, bigger or smaller
rotated, or gone thru any such simple transformation, It’s a totally different image as represented
in memory
What we need is invariance to all these transformations.
So the first layers of the neural networks we usually use are doing exactly this
We are taking the original image
And the first layer or layers usually implement convolutions and other operations
to extract features from the image The first layer of features will sometimes look like edges,
The second layer will extract higher level features, maybe shapes like circles, corners
and so on.
So every layer of the first layers is extracting more and more features of the image,
Hopefully these features are invariant to all kind of scaling, rotating and so on.
And we apply filters with multiple sizes and basically extract features at few scales
or rotations
And now the fully connected layers can determine if this is a flower based on the combination
of all the features that were extracted
And this is AI for vision in a nutshell
The same apply for audio signals, Just one tone of the audio is not going to
help us a lot, and not even the full signal in the time domain..
We need, just like for the image processing, to extract audio invariant features.
And only then try to actually detect or classify the audio
So let’s talk about extracting audio features This is a 1 KHz simple sine wave.
And this is how it sounds..
Audio is a signal in the time domain.
The signal is sampled and quantized..
We are working with digital signals..
This is a 2KHz simple sine wave and this is how it sounds
Our ears are sensitive to audio signals in the time and frequency domains..
Processing audio in time domain alone is difficult.
So the frequency domain representation of a signal reveals several characteristics of
the signal that are either not easy to see, or not visible
at all at the time domain.
The Fourier transform give us the frequency domain representation of the synthetic signals
I created
A spike at 1KHz and a spike at 2KHz..
I have created a third wave by just adding the first 2
And another wave, which you can immediately see that is different but is also the combination
of the first 2..
A different combination
And the problem is that in the Frequency domain, we are getting almost the same representation
of them both..
Even though we see, and can easily hear that they are totally different
So there are advantages in analyzing the signal at the time domain, and advantages to the
frequency domain..
What we need is a representation that combines both the time and frequency information..
And we can get it using Short Time Fourier Transform (STFT)
If we apply several FFTs at different time intervals..
the result will be a spectrogram You can think of it as many FFTs arranged
next to each other, Each FFT is done for a small window of time..
The spectrogram you see here, is the representation of a synthetic audio signal I created that
sounds like that The colors indicate the Amplitude or strength
of each frequency.
The brighter the color the higher the energy And now we can clearly see how the frequency
of the signal changes over time..
So both time and frequency representation.
And this is the MFCC of that signal.
I will not get into too many details, but you can see that it is not very different
from the spectrogram representation, It’s a different scale on the vertical axis,
this is not a linear frequency scale but rather a MEL scale
-a scale of pitches that sounds to the human ear to be equal in distance from one another.
The intensity scale is also non linear… and also adjusted to the way our ears perceive
audio intensity Lots of materials on the internet if you’d
like to understand more,
The point is, that now we have a representation that basically looks like an image.
And this is something we can work with and feed into a neural network.
So now we can extract the Spectrogram out of the audio signal
And feed it into a CNN to generate a feature map, that could be used, for example by a
classifier to classify audio signal
Video 77, suond classification using the DL-streamer or by a python sample is doing exactly that.
Just youTube openvino 77,
OpenVINO also have speech recognition demo, look for that video too.
If you look at the demos and sample code you can find libraries and functions that you
can use to extract audio features, to efficiently extract MFCC and feed them
into neural networks.
This is it in a nutshell, there is a full world of knowledge out there
I don’t usually do that, but I do recommend “the sound of AI” channel of Valerio Velardo,
he is great
Audio processing with OpenVINO, Go make same noises