Welcome back to the OpenVINO channel
We have talked about the full Video-analytics pipeline in previous videos.
The first stage in the pipeline is to uncompress or "decode" the input video stream.
Just to explain why do you even need this stage..
A 1 minute of 1080p HD video in youTube format Weigh about 70MB..
The same 1 minute of video, uncompressed , in raw format can take almost 5GB of space
1 Frame of such Video will take, depend on the format about 6 MB
After the pre-processing, the inference takes as an input much smaller images
maybe 2 or, 300x300 pixels
Pretty easy to see how the size of the data is funneled
and why it make sense, if we want to save network bandwidth and compute we need to move more and more compute to the edge
But back to the first stage..
We see here a compression rate of about 50x..
Depending on the application, the activity, codec and more parameters, it can be even much higher
What we are basically trying to do it to take a video, compress it, or Encode it, so it will be much smaller
Because it is smaller, we can move it around faster and it will take less memory and bandwidth
But then when we want to process it again we need to uncompressed it or Decode it
Hopefully without loosing too much of the original information
The codec which is a combination of the words code and decoder
is basically the set of rules or algorithms used to compress and uncompress back the video and the audio
There are hundreds of different codecs, the most common I use these days is h.264 or AVC
The next one is h.265 or HEVC with compression that is almost double the h.264,
but is more complex..
Usually requires 3x the resources to encode
some terminology: what we usually refer to as "video file" is really a "container" holding many other files
For example .mp4 file is really a container file and it holds the video stream to display which could be h.264 for example
It also hold the audio stream to play which could be mp3 for example
and the meta-data which can hold all kind of information like bit-rate, resolution and many more..
One of the most important pieces of information is the codec to use..
How does compression work?
If I need to explain it really in 2 minutes, I would say that we are exploiting some of the redundancies that a natural video has
in order to present it with smaller amount of bits
One example is "temporal redundancy" Look a this video sequence,
the biker position is changing from one frame to the other, But a big portion of the image, the background, is totally static..
So basically I could say..
That for these frames, no need to store the right hand third of the image.. It is the same.. (for the 3 frames)
Another kind of redundancy, "Spatial redundancy"
The rectangle you can see here in the sky is about the same color.
In some cases, I could say that all of these pixels, could be represented by one, with an average color
There are so many other techniques..
Some of them are really complex,
usually there is a tradeoff between the compute required to the compression rate..
The more complex it is, you can compress more but it will take more time or more compute resources to encode and decode..
So it is obvious that some frames has unique new information, and we need to have all the pixels from these frames.
These frames are called I- frames
In between the I-frames, I could represent only the delta..
The changes from the previous frame..
These frames are called P-frames and basically hold instructions on how to generate the frame out of the previous I-frame..
A P-frame will typically have half the data of an I-frame
In between there are B-frame and they are interpolation between the frames and can have quarter the data of an I-frame..
So, I frames is a lot of data, P and B frames will have less,
Since we are broadcasting the video in a bit stream, it means that some frames take longer to accumulate on the receiving side and some less
So the bit-rate is changing and we need to synchronize it
Basically we are maintaining buffers to accumulate the bits as they fly in,
and once we have a full frame, we can lock it, hand it to the next buffer as a ready frame for analysis.
This also means that we need to keep the I-frame for a long time, because the next differential frames relay on it..
And when we are displaying the video after the re-encode we need to synchronize the frames
back so the video stream will look smooth again..
So a very big mechanism, just to get you your youTube videos
Intel integrated GPU has all of these operations accelerate on Hardware.
As I mentioned in video 18, we have fixed functions in the integrated GPU
to accelerate the decode, encode and even the image processing
We call it Intel Quick Sync And by using it, you can move all of these heavy operations from the CPU to the GPU, and do it more efficiently. On the dedicated HW..
And of course process more streams by your application..
You will need to have the right drivers and if you have installed OpenVINO you already have them all
Basically, On top of the GPU fixed functions we have the driver, on top we have a VAAPI Video accelerating API, also referred as LibVA,
You can access it directly with the right plugin from Gstreamer or ffmpeg or using our Media-SDK..
So we have seen what is video compression in few minutes..
On the next video, I'll show you how, with one line of code, you could take advantage of our decode HW acceleration using OpenCV
We'll see how to call Media-SDK or Gstreamer from OpenCV
Subscribe to our channel to get more videos like this..
Thank you