Welcome back to the OpenVINO channel
OK.
My inference looks good, I can build a full pipeline of video analytic or other AI based
application.
Where is the money?
How do I actually create a commercial application out of it?
Probably not easy.
Well, it is easier than you think !!
OpenVINO as you all already know is not just an inference-API
With OpenVINO you are getting all the tools you need, resources, samples, models, and
more
And at the end of this flow, when you got your algorithm and models ready
We have easy ways to deploy your application
Your first option is to integrate the inference-engine code into your application
Use one of the samples code supplied with OpenVINO for reference
VA-Serving allow you to build not only an inference server but a full DL-streamer based
pipeline as a service..
We explain and demonstrate it in another video
Or you could use Azure to build an OpenVINO model server
You can find all the instructions online,
And the last option, is to use the OpenVINO Model server
OpenVINO Model Server, is a scalable, high-performance solution for serving machine learning models
That QR code will lead you to the full documentation.
The model server is basically taking the inference engine runtime and packages it up into a network
service You can choose the network protocol to be
gRPC or REST
OpenVINO code is highly portable, as you already know.
So the actual serving inference could be done on multiple devices,
if my server has a Core or Xeon CPU, an integrated or discrete GPU from Intel or even a VPU..
I can use almost the same code for all
So we basically have here a package of the inference engine code and your model
And of course it could be multiple models, not only one..
And we are adding a network communication layer
And it can all be run in a docker container..
You can deploy it on a Kubernetes cluster or any local machine you have
And potentially serve any number of clients..
Which can now be very simple, low power devices, as the inference is done by the server..
This is the high level architecture of the server,
You can find many more detail on the documentation page and on github..
I’d like to show you how it works.
I’ll be following the quick start guide on this link to implement a face detection
model-server..
It is simple to follow and will take you minutes.
Let’s clean up first .. and make sure I have no images or containers in my system.
Do I have any images?
Yes, one Used by any containers?
Yes, I have 2 of them Let’s remove this one
And the other one..
Opps it is running, So let’s stop it first..
Now remove it
And I have no containers, Let’s remove the images
All set..
And let’s pull the model-server docker And I’m ready to go..
First I will download the face detection model.
Here are the IR files,, And I’d like to place them in the right
directory structure This is my model 1.
I can have many of them And you can see how they are arranged under
model/1 This is all..
Now I can run the server I’m using the model-server docker
I’m using the face-detection model And important to ask for auto-shape as the
input images from the clients might arrive in different sizes..
And the server is running.
Moving to the client, I’ll run the client on the same machine,
but it could be any other machine..
I’m downloading all the client utilities and the face-detection script
Let’s look at the face-detection script Scrolling down,
Here we are taking all the input images, And we are sending an inference request
You can see that we use the TensorFlow serving api.
For the specific model we want And now we are taking the results..
You can see the grpc request
And here is the output Basically the detected faces bounding boxes..
Which I draw over the original image..
Let’s try it out I’m downloading some sample images
They are in the “images” directory I’ve actually placed many more images there,
not only this one..
And I install the client requirements Prepare a directory for the results.
And I’m all set
Let’s run the client I’m sending a request to the server, to
take the images from the images directory and place the results in the results directory
I’ve opened here a TOP view on the CPU
Sending the request You can see that the CPU utilization is up.
And immediately down..
I got the full report, but let’s look at the results..
You can see that all the faces are detected and the bounding boxes drawn on all the images..
This is very cool
So, take your models and pack them into a model server.
Take your application code and create a client that can use this server for inference.
Don’t forget to subscribe to the channel And tell us how it works..