Welcome back ​ One of the greatest things I find in Hugging-face
is the simplicity.​ Just get me a full working pipeline,
Here is my input, get me the inference results..​ Or get me a model, get me a tokenizer..
Combine them into a pipeline, go.​ The usual problem with abstraction and simplicity
is that sometimes you lose performance.​ How about enjoying everything hugging-face
has to offer ​ AND getting all the benefits that OpenVINO
can provide like performance boost, ​ the ability to run on multiple devices, CPUs
GPUs and more ?​
​The power OpenVINO adds to hugging face for almost
no re-code is really great.​ let me show you in a minute
So using OpenVINO in hugging-face is pretty simple.
But maybe the simplest way for me to explain it
is using a simple example.​ I will just start from scratch.​
And I’ll leave you all the instructions and the code snippets below.​
Look for OpenVINO installation instructions.​ I’ll open a new virtual environment.
I’ll call it OpenVINO-optimum​ I’ll activate it​
You can see that I’m inside the environment.​ Update pip..​
And now install OpenVINO.​ I’ll install onnx and pytorch​
A great way to check if OpenVINO installed, is to run the benchmark_app –h​
And you see that my system has a CPU and 2 GPUs,​
Intel integrated GPU .. GPU.0 and Intel discrete GPU .. GPU-1​
Install transformers​ And install Optimum intel, with diffusers​
And we are all set..​
Let me show you the simplest possible example.
I’m taking a pre-trained model​ And the corresponding Tokenizer​
This is a pipeline for text classification using this model and tokenizer​
And I’ll classify this sentence​
I’m already in my virtual environment​ Running, ​
And yes, the model is pretty sure that this is a negative sentence.​
If you go to the hugging face documentation, I’ll leave you the exact link​
You can see how tasks are mapped to OpenVINO Optimum auto classes..​
I’ll take the text classification one..​
And use it here..
Instead of the regular Auto-model..​
I’ll use the OpenVINO class to get the model from pre-trained..​
And since the model format for OpenVINO is different,​
Export=True will automatically convert the model for me.​
Yea.
That’s all.
Run it and we’re getting the same results.​
If you do not want to load and convert the model each time you use it.​
Just save it..
I'll save it to a directory by the same name ​
I’ll just add that it’s OV format FP32​
Next time you’d like to use this model​ Just load it from the same directory​
BTW, If you search for OV in the hugging face models, you will find quite a few models already
in OV format.​ So you can load them directly..​
And then, of course you do not need to use Export​
​Now that you have OpenVINO as a backend to Hugging-face​
You can do all the good stuff that OV allows you,​
Using the optimization and Quantization tools and much mode.​
We’ll leave this to another video..​ But Let me show you just one example​
Here is the simplest stable-diffusion code.​ I’m using the OpenVINO stable-diffusion
pipeline​
I found a stable diffusion model ​ I load it from pre-trained and export, converting
to OpenVINO format..​
This is my prompt, looking for a Rembrandt style ship in a storm​
Run the pipeline and save the result​
And I’ll just measure the time it took​
So this is the code,​ And I’ll just run it​
You can see that the CPU is used for the initial processing, ​
But really fast it becomes a memory bottleneck and it remains like that​
The 16GB of memory in my machine is definitely not enough…​
I won’t even wait for it to finish.​
And now let’s make it faster​ First I found a model in the hugging-face
hub already in OV format.​ So bring it over, no need to convert, and
do not compile yet.​
A nice feature of OpenVINO is that if your input sizes are different than what the model
expect,​ you can re-shape the model's input to fit
your input sizes, and by that save scaling time during inference .​
Here I’ll reshape the model in advance.​
And I’ll use FP16 instead of FP32.
just half the model weights.​ It’s most useful if you want to save the
model, as OpenVINO will half it anyway automatically.​
And I’ll run it on my integrated GPU, GPU.0 in this case​
​And I’ll keep the same prompt and performance counters..​
So this is the code​ Let’s run it.​
you can see that pretty fast the GPU is fully utilized.​
And it took us only 23 seconds..​ And here is the result image.​
Please see my other video on the discrete ARC GPU A770 ​
where I run stable diffusion in 7 seconds.​ So, yes, the best of all worlds.​
Hugging Face, augmented with OpenVINO capabilities, try it out​
Subscribe, enjoy, thank you.​