Welcome back to the OpenVINO channel In all the videos so far, we have talked about

image and video processing.

But OpenVINO can help you accelerate audio, NLP and many other usages.

This video is an ultra fast introduction to audio processing.

There are many better and more comprehensive full tutorials, this is really just to get

you started..

Before talking about Audio, a quick reminder on image processing

Here are 2 pixels of an image,

These pixels cannot help us much with understanding the full image.

Even if we have all the pixels, it is not enough

Because if we move this image by one pixel to any side, the way the computer represent

this image is totally difference.

rotated, or gone thru any such simple transformation, It’s a totally different image as represented

in memory

What we need is invariance to all these transformations.

So the first layers of the neural networks we usually use are doing exactly this

We are taking the original image

And the first layer or layers usually implement convolutions and other operations

to extract features from the image The first layer of features will sometimes look like edges,

The second layer will extract higher level features, maybe shapes like circles, corners

and so on.

So every layer of the first layers is extracting more and more features of the image,

Hopefully these features are invariant to all kind of scaling, rotating and so on.

And we apply filters with multiple sizes and basically extract features at few scales

or rotations

And now the fully connected layers can determine if this is a flower based on the combination

of all the features that were extracted

And this is AI for vision in a nutshell

The same apply for audio signals, Just one tone of the audio is not going to

help us a lot, and not even the full signal in the time domain..

We need, just like for the image processing, to extract audio invariant features.

And only then try to actually detect or classify the audio

So let’s talk about extracting audio features This is a 1 KHz simple sine wave.

And this is how it sounds..

Audio is a signal in the time domain.

The signal is sampled and quantized..

We are working with digital signals..

This is a 2KHz simple sine wave and this is how it sounds

Our ears are sensitive to audio signals in the time and frequency domains..

Processing audio in time domain alone is difficult.

So the frequency domain representation of a signal reveals several characteristics of

the signal that are either not easy to see, or not visible

at all at the time domain.

The Fourier transform give us the frequency domain representation of the synthetic signals

I created

A spike at 1KHz and a spike at 2KHz..

I have created a third wave by just adding the first 2

And another wave, which you can immediately see that is different but is also the combination

of the first 2..

A different combination

And the problem is that in the Frequency domain, we are getting almost the same representation

of them both..

Even though we see, and can easily hear that they are totally different

So there are advantages in analyzing the signal at the time domain, and advantages to the

frequency domain..

What we need is a representation that combines both the time and frequency information..

And we can get it using Short Time Fourier Transform (STFT)

If we apply several FFTs at different time intervals..

the result will be a spectrogram You can think of it as many FFTs arranged

next to each other, Each FFT is done for a small window of time..

The spectrogram you see here, is the representation of a synthetic audio signal I created that

sounds like that The colors indicate the Amplitude or strength

of each frequency.

The brighter the color the higher the energy And now we can clearly see how the frequency

of the signal changes over time..

So both time and frequency representation.

And this is the MFCC of that signal.

I will not get into too many details, but you can see that it is not very different

from the spectrogram representation, It’s a different scale on the vertical axis,

this is not a linear frequency scale but rather a MEL scale

-a scale of pitches that sounds to the human ear to be equal in distance from one another.

The intensity scale is also non linear… and also adjusted to the way our ears perceive

audio intensity Lots of materials on the internet if you’d

like to understand more,

The point is, that now we have a representation that basically looks like an image.

And this is something we can work with and feed into a neural network.

So now we can extract the Spectrogram out of the audio signal

And feed it into a CNN to generate a feature map, that could be used, for example by a

classifier to classify audio signal

Video 77, suond classification using the DL-streamer or by a python sample is doing exactly that.

Just youTube openvino 77,

OpenVINO also have speech recognition demo, look for that video too.

If you look at the demos and sample code you can find libraries and functions that you

can use to extract audio features, to efficiently extract MFCC and feed them

into neural networks.

This is it in a nutshell, there is a full world of knowledge out there

I don’t usually do that, but I do recommend “the sound of AI” channel of Valerio Velardo,

he is great

Audio processing with OpenVINO, Go make same noises