Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
These amazing images were all made by an AI called DALL-E 2.
This AI is endowed with a diffusion-based model, which means that when we ask it something,
it starts out from noise, and over time, it iteratively refines this image to match our
description better.
And, over time, magically, an absolutely incredible image emerges.
This is a neural network that was given a ton of images, and a piece of text that says
what is in this image.
That is one image-caption pair.
DALL-E 2 is given millions and millions of these pairs.
So, what can it do with all this knowledge?
The key takeaway here is that it does no, or very little copying from this training
data, but it truly comes up with novel images.
How?
Well, after it had seen a bunch of images of koalas, and separately, a bunch of images
of motorcycles, it starts to understand the concept of both, and it will be able to combine
the two together into a completely new image.
Previously, we also compared these works to what real human artists are capable of.
And, in my opinion, this tool should not be framed at all as Team Humanity versus Team
AI.
I believe this should be framed as Team Humanity, supercharged by Team AI.
Here’s an example of that.
And the results are so good, and it is so easy to use, I said, on several occasions
that this is going to change the world of art creation as we know it.
And, you are an experienced Fellow Scholar, so you may ask the question, okay, but how?
Here are 5 beautiful examples of how artists are already using it.
One, texture synthesis.
Oh yes.
Whenever we see these beautiful animation movies or video games, we see a lot of real-looking
content in there.
The roads are realistic, the buildings and hills are also realistic.
We can get a great deal of help in making them realistic by…taking photographs of
real objects and importing them into our virtual world.
But there is a problem.
What is the problem?
Well, let’s see together.
Let’s take a photo.
There is a stain on the wall, is that the problem?
No, that’s not the problem, we can cut that out.
Now, we need to extend this texture, but since this is just a small snippet, and we have
long walls and big buildings to fill in a video game.
So, what do we do?
We start tiling it.
And, uh-oh.
Now that is the problem.
It is quite visible what happened to this texture, it has been extended, but at the
cost of seams and discontinuities appearing in the image.
So, how do we get a seamless texture from this?
Well, DALL-E 2 hopefully understands that this is a rock wall, and, can it fix this?
Now, hold on to your papers, and let’s see together.
Yes it can!
Amazing!
Now let’s have a look at it attached to a real virtual object and see how it looks
in practice.
Oh my.
This is gorgeous.
I love it.
Here is an even more difficult example with paint peeling off the wall.
This is the real-world phone photo.
And the tiled version with the seams.
And now, DALL-E 2.
Wow!
I could never tell that this image is not a real photo.
And, have a look at it added to a video game.
So good.
By the way, these are Stan Brown’s works, who used DALL-E 2 to great effect here.
Make sure to check out his work, for instance, he also made these amazing materials for one
of my favorite games of all time, Path Of Exile.
The link to his work is available in the video description.
Just imagine using the free 3D modeling tool, Blender and this plugin to create roads, and
you can make it an asphalt road, dirt road, or make it look like pavement in just a couple
clicks with DALL-E 2.
Two, DALL-E 2 is now able to generate real human faces too, and sometimes they come out
really well.
And sometimes, not so much.
No matter, in these cases, we can use a face restoration program to fix some of the bigger
flaws and run it through a super-resolution program to add more details, and bam.
There we go.
Fantastic.
And, if we have a portrait that works for us, we can plug it into an amazing tool called
Deep Nostalgia to create a virtual animated version of this person.
How cool is that.
What a time to be alive!
Three, this one speaks for itself.
The Cosmopolitan magazine made their first cover from an image that was made by an AI.
No tricks, no frills, just a good old image straight out of DALL-E 2.
And the kicker is that some of them are already good enough to become magazine covers.
That is incredible.
Talk about the age of AI image creation!
Four, this tool can even help us create new product designs.
And I mean not this one, although this toilet car might have some utility if you need to
go when you are on the go, but I meant this one instead!
And, finally, five.
We can also combine DALL-E 2 with other tools that take a 2D image and try to build an understanding
as if it were a 3D scene.
And the results are truly stunning, we can get so much more out of these images.
And one of the important lessons here is that not only AI-generated art is improving at
an incredible pace, but an additional lesson is that everything is connected.
So many of these tools can be made even more powerful if we use our imagination and combine
them correctly.
This is huge.
For instance, NVIDIA is already building virtual copies of the real world around us.
Why is that?
Well, because this way we can reenact real situations for their self-driving cars, and
even create arbitrarily complex new situations, and these AIs will be able to learn from them
in a safe environment.
Also, self-driving can also be combined with a number of other technologies that make it
an even more amazing experience.
See the assistant there?
It sees and identifies the passenger, and understands natural language.
It also understands which building is which and what shows are played in them today, so
we might jump into the car not even knowing where we wish to go, and it would not only
help us find a good show, but it would also drive us there.
And, +1.
Remember the shoe design example?
How about trying on pieces of clothing, but not one by one, as we do in real life, but
trying on…all of the clothes, at the same time.
Yes, these are all AI-generated images, and it is not on just one still image, but video
footage where a camera moves around.
This is spectacular.
Imagine what buying clothes will look like when we have algorithms like this!
Wow.
Or, if an actor did not have the appropriate attire for a scene in a movie?
Not to worry, just change it in post with the AI.
How cool is that?
You see, once again, everything is connected.
So, does this get your mind going?
What would you combine with DALL-E 2?
Thanks for watching and for your generous support, and I'll see
you next time!