Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
I know for a fact that some of you remember our first video on image translation, which
was approximately 3 years and 250 episodes ago.
This was a technique where we took an input painting, and a labeling of this image that
shows what kind of objects are depicted, and then, we could start editing this labeling,
and out came a pretty neat image that satisfies these labels.
Then came pix2pix, another image translation technique which in some cases, only required
a labeling, a source photo was not required because these features were learned from a
large amount of training samples.
And it could perform really cool things, like translating a landscape into a map, or sketches
to photos, and more.
Both of these works were absolutely amazing, and I always say, two more papers down the
line, and we are going to have much higher resolution images.
So, this time, here is the paper that is, in fact, two more papers down the line.
So let’s see what it can do!
I advise you that you hold on to your papers for this one.
The input is again, a labeling which we can draw ourselves, and the output is a hopefully
photorealistic image that adheres to these labels.
I like how first, only the silhouette of the rock is drawn, so we have this hollow thing
on the right that is not very realistic, and then, it is now filled in with the bucket
tool, and, there you go.
It looks amazing.
It synthesizes a relatively high-resolution image and we finally have some detail in there
too.
But, of course, there are many possible images that correspond to this input labeling.
How do we control the algorithm to follow our artistic goals?
Well, you remember from the first work I’ve shown you where we could do that by adding
an additional image as an input style.
Well, look at that!
We don’t even need to engage in that, because here, we can choose from a set of input styles
that are built into the algorithm and we can switch between them almost immediately.
I think the results speak for themselves, but note that not only the visual fidelity,
but the alignment with the input labels is also superior to previous approaches.
Of course, to perform this, we need a large amount of training data where the inputs are
labels, and the outputs are the photorealistic images.
So how do we generate such a dataset?
Drawing a bunch of labels and asking artists to fill them in sounds like a crude and expensive
idea.
Well, of course, we can do it for free by thinking the other way around!
Let’s take a set of photorealistic images, and use already existing algorithms to create
the labeling for them.
If we can do that, we’ll have as many training samples as many images we have, in other words,
more than enough to train an amazing neural network.
Also, the main part of the magic in this new work is using a new kind of layer for normalizing
information within this neural network that adapts better to our input data than the previously
used batch normalization layers.
This is what makes the outputs more crisp and does not let semantic information be washed
away in these images.
If you have a closer look at the paper in the video description, you will also find
a nice evaluation section with plenty of comparisons to previous algorithms and according to the
authors, the source code will be released soon as well.
As soon as it comes out, everyone will be able to dream up beautiful photorealistic
images and get them out almost instantly.
What a time to be alive!
If you have enjoyed this episode and would like to support us, please click one of the
Amazon affiliate links in the video description and buy something that you were looking to
buy on Amazon anyway.
You don’t lose anything, and this way, we get a small kickback which is a great way
to support the series so we can make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time!