Welcome back
Generative AI, capture our eyes, ​ We said that machines are
getting intelligent, ​ but this looks like creativity​
How does it work? ​ How can you run it in detail step by step?​
OpenVINO has few ready notebooks that can help you run stable diffusion right away,​
Let me show you in a minute.
At the basis of our ability to perform generative AI​
and manipulate or generate images is our ability to represent an image in
its features or Latent space and able to convert it back to the pixels space​
The first layers of neural networks like VGG16 and
others are built to extract the features in the image​
And by that create a representation of the image in the feature or latent space..
And here is nice example,
to get some intuition.. ​ I took a very small data-set.. Shirts…
And shoes…​ And trained a model to extract their features, ​
There is a nice video explaining this, I’ll leave you the link,
you can see here for simplicity just 2 dimensional latent space, on the left​
Using this 2 “features” or dimensions,
I can represent and of course distinguish between the shirts and the shows..​
Each blue point on the left represent one of the data-set images.​
For example, this point correspond to this red image.​
This point to this black shirt. ​ There is no data-set point, no image in between
them, but if re render this point in the middle​ We’ll get something between these two shirts..​
the total opposite side of the latent space represent a white shoe​
And the more we go up and to the right, we’ll get images closer to shirts and darker..
So we can extract the features of objects,
And a combination of these features, what we usually choose using a
feature-map, is that object..​ For example a cat and a dog both
has the feature “4 legs” for example​ And the feature “a tail” and so on.
So they will be located close to each other on our features or latent space..
A chair also has 4 legs, so it will be closer to them than a boat..​
Again just to demonstrate the concept.
On the other hand is the text, ​ We break it into pieces we call tokens.​
Here is the token for a cat
And here is the token for a dog. And we map these tokens to an
embedding space, the latent space for words if you like​
And here too, I envision that a dog will be closer to a cat than to a chair..
If we can map the embedding space to the image latent space,
we basically know how to represent the text in the latent space
So If you tell me to draw a cat, I know how the latent space of a cat looks like.​
all I have to do is go back to the pixel space, and you have a cat.
And how about a cat sitting on a chair? ​ Can we just add the features of the cat
and the features of the chair?​
What operations are legal at the latent space..​ And how can we get a smooth image out of this?
Well, this is the magic ​ We start with a noisy image..​
And we use an iterative process of removing the noise, ​
We use a U-net model for that.​ Basically down-sampling which smooth the image ​
and up-sampling again.. You’ll find many more details online..​
And we use this model to predict the noise​ The noise conditioned or associated
with out prompt embedding.​ And then remove this specific noise.. ​
This is all done in the latent space, so faster execution..
And when done, we can use the autoencoder, to get our image result in the pixels space..
To see that in action.​ Just navigate to the openvino notebooks,
there are several related notebooks ​ Please note that you will need a
machine with large memory !
If you just want to run Stable diffusion,​
You can use Notebook 236 ​ This is probably the easiest way​
And probably the minimal required code.​ + it’s optimized using Optimum Intel​
Let’s download the models ​ Save them and compile..​
Prompt. Run. ​ And I can paint like Van Goch.​
And there are many other related notebooks for image inpainting and more.
Notebook 225 is an opportunity to get more details.​
Make sure you have enough memory and the right SW versions.​
First thing to do is to get the pipeline and the 3 models from hugging face.​
The text encode.. From prompt to Embedding​
The U-net model, for the noise reduction and the diffusion process​
And the AutoEncoder to bring us back from the latent space to the pixels.​
And now we’ll convert each of the models to OpenVINO format.​
The nice thing is that you can find these models here in the files and explore them.​
And now OpenVINO core object, ​ load and compile the 3 models.​
And here is the OpenVINO stable diffusion pipeline.​
With all the models and the components that we already have.​
And now we can look inside, ​ For example check how the
tokens for Cat looks like​ Or the tokens for any prompt..​
And we can even see the embedding space, ​ It doesn’t tell me much, but there are
some ways to visualize them if you want..​
And let ‘s run..​ Text to image..​
Choose the prompt and other parameters..​ Run the pipeline, generate the image,
and present it​ Pretty cool​
And you can also run the image to image generation,
And that’s all, Text to
image or image to image generation ​ Step by step with OpenVINO notebooks.​
Enjoy, subscribe​ Thank you