Hi. I am Adrian.
I am an AI evangelist and inference maker from Intel.
Today I'm going to show you how to build an intelligent queue management
application by leveraging OpenVINO
and our Edge AI reference kit. By the end of this video,
you will understand how this application is created
and be able to build your own.
And this application can be broken down into two main parts.
First, convert and optimize the YOLOv8 model
and second, build and then run the queue management application.
For each part, we have created a Jupyter notebook
and I will walk you through in detail.
So let's go to the code.
I already cloned the repository and started Jupyter Lab environment.
So on the left you see the full repository we have here.
There are many scripts, sample video, for example, zones,
definition file, but let's enter the docs directory
where we have two different notebooks
and the first one is convert and optimize.
So we will focus on this first and the goal of this notebook
is to convert YOLOv8 model to OpenVINO and to quantize it.
So let's scroll down to the first cell.
So first let's load some sample picture.
And this picture is with people, people standing in the queue.
So this will be for some sanity checks
to be sure that OpenVINO model, YOLOv8 model
they are able to detect people
and so let's load the YOLOv8 model from ultralytics package
and we decided to use
YOLOv8 ‘M’, ‘M’ means medium
because it's a good trade off between accuracy and performance.
If you need more performance select for example, small or another version,
if you need more accuracy, large or extra large.
Ok, the model is loaded.
So let's run the inference for the first time and see the results.
Good.
We see that the YOLO model is able to detect people, right?
So it would be useful in our application.
That's great.
Good. Let's scroll down.
So next we need to
convert the model from Pytorch format to OpenVINO format, and the function export
is already available in the ultralytics package.
So we don't have to do much here and it's already exported.
That's good.
So we can go to the directory called model
and we see that we have here YOLOv8
OpenVINO model.
And so we could stop here
and use that model in our application.
However, we would like to achieve even better performance.
So that's why we decided to go with quantization.
And the quantization is a process to decreasing the precision of the model
from FP32 to INT8 to improve the performance.
Right.
Of course we should expect a little accuracy drop,
but talking about my experience,
I have never seen more drop than 1%.
So I think we can sacrifice just 1% to have much better performance.
How much, we will see in the while.
Okay, so let's go with the quantization. To quantize the network
We need to provide a representative dataset
sometimes called calibration dataset or quantization dataset.
And so we need to provide some data
and what data we are going to provide here.
Well, the YOLOv8 model was trained on COCO dataset,
so we would like to use the same dataset, the validation subset in that case.
So let's run the code here.
And of course we need to wait
a little bit, right, because the validation subset from COCO
has more than 700 megabytes.
So it would take some time to download it.
97, 98,
99 and 100.
And of course, we need to download also labels.
When it’s download we need to extract them
from .zip archives.
So it should be just a second.
Yes, it's done. Good.
So let's create a COCO dataset, COCO detection dataset
and let's create a data loader we we needed for quantization process.
Of course.
Then let's load our OpenVINO model
we already exported to OpenVINO format right?
It's loaded, good.
Then let's create the dataset and NNCF format
and let's provide some transformation function
when we get rid of labels
because we care about
images only
then let's quantize.
And here we specify some ignored scope.
So we don't want to quantize these operations
like multiplication, subtraction and sigmoid.
And of course some also add operations.
Why that, because we know that ignoring and not quantizing
these operations would give us great performance preserving the accuracy
so if you would like to quantize all operations, it's okay,
but you can expect more accuracy drop down
ignoring some specific operations in that case
multiplication, subtraction and sigmoid.
Okay.
It's quantizing here in this function.
So we provided OpenVINO model here, we provided OpenVINO model
quantization, calibration, representative dataset
and we added some preset that we would have
mixed precision quantization to preserve the accuracy and
we specify above ignore the scope here
and it's collecting statistics
so from them
300 images from COCO dataset.
So we need to wait a little bit
still 20 seconds
and then we will have the stage of biases correction.
It should be rather fast and after that we are able to serialize
the model to disk.
Serialize means save, save the disk.
When we save the model to disk, we are able to open it
in another application and to use just to use
Good, statistic collection stage is done.
So let's wait for the next stage
Biases correction
Done. Done.
Everything is done.
However, I have some bonus for you.
I would like to show you what the difference in performance
between FP32 model and INT8.
So let's go to the last section of that notebook.
We will use CPU
device in that case
and let's benchmark the first model.
I mean OpenVINO FP32 model
on CPU with hint latency
because our queue management system
is latency system. Right?
We would like to have the result as soon as possible.
So that's why we care about latency and of course we will
benchmark for 30 seconds and let's see the results.
Let me scroll down
to the results.
We see that the average latency
is 148 milliseconds.
So it gives us 6.7 frames per second.
So it's not very slow, but it's also not very fast.
Right.
So let's compare these results to the INT8 model,
which is just one cell below.
And so we are doing exactly the same.
We are just changing the model to quantized the one.
And then let's scroll down to the results.
And now we have 47 milliseconds in average.
Which gives us 21 frames per second, which is three times faster, right?
Just quantizing our network.
So that's nice.
I like it.
Perfect. Good.
So we already converted the model from Pytorch format to OpenVINO
We now we quantized that model to INT8 precision
and now it's time to run the application itself.
Good.
Let's go back to the docs directory.
Then we have there we have one more
notebook on the application notebook and here
we would like to run the full application
seeing how we can detect people, how we can count people
and how to manage queues intelligently
so let's import some packages
maybe.
Yeah, I will run this.
So right now we are importing OpenVINO of course we are importing ultralytics.
We are importing also supervision because we we use some additional packages
not to write boilerplate code and so we prefer to use libraries.
Okay.
Letterboxes are part of preprocessing steps, so we are
we need to preprocess image to
to have the same ratio but to be smaller to to go into the network
And we have also preprocess function which
which is calling letterbox function
this one above and of course we convert
the image type to precision float32. We also normalized
the image by dividing by 255.
We changed the order of dimensions and we create a batch
with one image.
Good.
Then we define the post-processing function
because the results from the network are really, really low.
So we need to interpret them somehow.
So we do non maximal suppression, for example.
Then we scale boxes back to the original resolution
and we create
detections in supervision format.
And supervision is a nice library which allows us to detect people
only in specific area like for example, specific queue.
And then we need to initialize OpenVINO
We know we need to read the model
from disk and in this case it will be a quantized model.
But of course we can use also model in floating point 32 format
compile model for some device.
And if we select auto here it means that the auto and OpenVINO itself
will select the best device for us.
So for example, if CPU is available, CPU is selected.
If GPU is available, it will be selected
as probably the most performant device.
And we specify that we care about latency.
Yes, because we would like to have the results
as soon as possible to be very responsive
and then the function to load zones and what is zone in that case,
we have that file called zones.json
and then we specify polygons for our queues.
So the feel of the first queue consists of 14 points,
the second queue has eight points
and the third queue has polygon with eight points also.
So we log these zones
right here in this cell and then we specify annotators.
Annotator is like a GUI, it’s a graphical user interface.
These are these polygons,
these are these boxes around people and so on.
So this is just for
showing purposes.
Then we specify also function for drawing text
on the image we would like to show the performance we have.
We would like also to inform people
that the store assistant is required in some specific queue, and then we define
the main processing function where we load our model, OpenVINO model.
Then we create a video player,
video player will stream us some input.
It could be webcam stream, it could it can be video file.
So the video player will deliver the frames.
Then of course we create these zones for queues.
So these areas
annotators, graphical user interface, right?
And then we specify people counter.
We will count people for all queues
and processing time
Yes, we would like to measure
an average time
because that is a better approach.
Of course, we can use a video writer to
to write some outputs
to the external file to see the results
after the running of this application and we start our video stream
in endless loop we get frame by frame from the player
and then we resize that frame to feed neural network input size
and we measure the time here for three steps
pre-processing, model inference, and post-processing here.
And when we have our results, we,
we annotate our zones, right?
So we process these results according to each zone.
So for example, we filter detections
from the full image just to consider only detections
which are inside our zone and zone means specific queue.
Then we of course throw the boxes around people in that specific zone.
We also count how many people we have.
we would like to have stable algorithm right?
So we don't want to flicker like two, five people because someone
is entering queue and leaving immediately or something like that.
So we would like to be sure that in long term we have three or four
or a six people in the queue to inform the store assistant
that someone else is required at this queue.
Okay.
Then we of course
and draw the information of the image
that store assistant is required on some specific cash desk.
We calculate main processing time
from a few seconds
from a few last frames, draw text on the image
about inference time and write this to the external file
showing on the end on the screen at the same time.
Okay.
So we set the customer limits for two.
So if there are more than two people in the queue,
we will see the information that the store assistant is required in that queue.
So let's run all cells here and let's see the results.
Okay, so it's working.
We can see three queues.
The first one they seen the red, the red polygon and the red
boxes around people in the first queue,
the second in green and the first, of course, in yellow.
And in the top left corner we see that store assistant is required
on desk three because we see that there are four people in this queue
and we set the limit for two.
So it works, right?
It just works. In the top right corner we see the inference time,
which is 59 milliseconds, which gives us 17 frames per second,
of course is a little slower right now because I am recording my screen.
So, you know, it's a heavy task for CPU.
When I tested this without the screen recording,
I got about 28 or 30 frames per second.
So you should expect something like that on your computer.
However, of course it depends on the hardware you have.
If you have older hardware, it would be slower.
If you have newer hardware than me.
It should be faster,
but of course it should be faster than the running in PyTorch.
In addition to the Jupyter notebooks, we have also created separate Python
files, including app, main, and utils,
which can be used to deploy the solution more easily.
Thanks for watching.
If you have any questions or things you would like to share,
join our discussions over on GitHub or our Intel community support channel.