Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to look at NVIDIA’s spectacular new AI that can generate beautiful images for us.
But, this is not image generation of any kind,
no-no! This is different. Let’s see how it is different.
For instance, we can write a description, and the appropriate image comes out. Snowy
mountains, pink cloudy sky. Checkmark. Okay, we can give it art direction.
Make no mistake, this is fantastic, but this has been done before, nothing new here. Yet.
Or, we can create a segmentation map, this tells the AI what things are.
The sea is down there, mountains, and sky up there. Looking great, but
this has been done before too. For instance, NVIDIA’s previous GauGAN paper
could do this too. Nothing new here. Yet. Or, we can tell the AI where things are
by sketching. This one also works, but this has been done too.
So, is there nothing new in this paper? Well, of course there is! And now, hold on to your papers,
and watch as we fuse all of these descriptions together.
We tell it where things are and what things are. But I wonder if we can make
this mountain snowy and a pink cloudy sky on top of all things….yes we can! Oh wow,
I love it. The sea could have a little more detail, but the rest of the image is spectacular.
So, with this new technique, we can tell an AI where things are,
what things are, and on top of it, we can also give it art direction.
And here is the key, all of these can be done, in any combination!
Now, did you see it? Curiously, an image popped up at the start of the video when we
unchecked all the boxes. Why is that? Is that a bug? I’ll tell you in a bit what that is.
So far, these were great examples, but let’s try to push this to its limits and see what it can do.
For instance, how quickly can we iterate with it? How quick is it to correct mistakes or
improve the work? Oh boy. Super quick! When giving art direction to the AI,
we can update the text, and the output refreshes almost as quickly as we can type.
The sketch to image feature is also a great tool by itself. Of course,
there is not only one way to satisfy these labels, there are many pictures that this could describe,
so, how do we control what we get? Well, it can even generate variants for us.
With this new work, we can even draw a piece of rock within a within the sea,
and the rock will indeed appear. Not only that, but it understands that the waves
have to go around it too. An understanding of physics. That is insanity. My goodness.
Or, better, if we know in advance that we are looking for tall trees and autumn leaves, we can
start with the art direction. And then, when we add our labels, they will be satisfied,
we can have our river, but the trees and the leaves will always be there. Finally,
we can sketch on top of this to have additional control over the hills and clouds.
And, get this - we can even edit real images.
So, how does this black magic work? Well, we have four neural networks, four experts if you
will. And the new technique describes how to fuse their expertise together into one amazing package.
And the result is a technique that outperforms previous techniques in most of the tested cases.
So, are these some ancient methods from many years ago that are outperformed, or are they
cutting edge? And here comes the best part. If you have been holding on to your papers so far,
now, squeeze that paper, because these techniques are not some ancient methods.
Not at all. Both of these methods are from the same year as this technique. The same year.
Such improvement in less than a year. That is outstanding. What a time to be alive!
Now, I made a promise to you early in the video. And the promise is explaining this. Yes,
the technique generates, even when not given a lot of instruction. Yes,
this was the early example when we unchecked all the boxes. What quality images can we expect then?
Well, here are some uncurated examples, this means that the authors did not cherrypick here,
just dumped a bunch of results here. And…oh my goodness, these are really good! The details are
really there, the resolution of the image could be improved, but we saw with the previous GauGAN
paper that this can improve a great deal in just a couple years. Or, with this paper, in less than a
year. I would like to send a huge congratulations to the scientists at NVIDIA - bravo!
And, you Fellow Scholars just saw how much of an improvement we can get just one more paper down
the line, so imagine what the next paper could bring. If you are curious, make sure to subscribe
and hit the bell icon to not miss it, when the followup paper appears on Two Minute Papers.
Thanks for watching and for your generous support, and I'll see you next time!