Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today, we will see how a small change to an already existing learning-based technique
can result in a huge difference in its results.
This is StyleGAN2, is a technique that appeared in December of 2019.
It is a neural network-based learning algorithm that is capable of synthesizing these eye-poppingly
detailed images of human beings that don’t even exist.
This is all synthetic.
It also supports a cool feature where we can give it a photo, then, it embeds this image
into a latent space, and in this space, we can easily apply modifications to it.
Okay…but what is this latent space thing?
A latent space is a made-up place where we are trying to organize data in a way that
similar things are close to each other.
In our earlier work, we were looking to generate hundreds of variants of a material model to
populate this scene.
In this latent space, we can concoct all of these really cool digital material models.
A link to this work is available in the video description.
StyleGAN uses walks in a similar latent space to create these human faces and animate them.
So, let’s see that.
When we take a walk in the internal latent space of this technique, we can generate animations.
Let’s see how StyleGAN2 does this.
It is a true miracle that a computer can create images like this.
However, wait a second.
Look closely…Did you notice it?
Something is not right here.
Don’t despair if not, it is hard to pin down what the exact problem is, but it is
easy to see that there is some sort of flickering going on.
So, what is the issue?
Well, the issue is that there are landmarks, for instance, the beard, which don’t really,
or just barely move, and essentially, the face is being generated under it with these
constraints.
The authors refer to this problem as texture sticking.
The AI suffers from a sticky beard if you will.
Imagine saying that 20 years ago to someone, you would end up in a madhouse.
Now, this new paper from scientists at NVIDIA promises a tiny but important architectural
change.
And we will see if this issue, which seems like quite a limitation, can be solved with
it, or not.
And now, hold on to your papers, and let’s see the new method.
Holy Mother of Papers.
Do you see what I see here?
The sticky beards are a thing of the past, and facial landmarks are allowed to fly about
freely.
And not only that, but the results are much smoother and more consistent, to the point
that it can not only generate photorealistic images of virtual humans.
Come on, that is so 2020.
This generates photorealistic videos of virtual humans!
So, I wonder, did the new technique also inherit the generality of StyleGAN2?
Let’s see.
We know that it works on real humans, and now, paintings and art pieces, yes, excellent,
and of course, cats, and other animals as well.
The small change that creates these beautiful results is what we call an equivariant filter
design, essentially this ensures that finer details move together in the inner thinking
of the neural network.
This is an excellent lesson on how a small and carefully designed architectural change
can have a huge effect on the results.
If we look under the hood, we see that the inner representation of the new method is
completely different from its predecessor.
You see, the features are indeed allowed to fly about, and the new method even seems to
have invented a coordinate system of sorts to be able to move these things around.
What an incredible idea.
These learning algorithms are getting better and better with every published paper.
Now, good news!
It is only marginally more expensive to train and run than StyleGAN2, and less good news
is that training these huge neural networks still requires a great deal of computation.
The silver lining is that if it has been trained once, it can be run inexpensively for as long
as we wish.
So, images of virtual humans might soon become a thing of the past, because from now on,
we can generate photorealistic videos of them.
Absolutely amazing.
Thanks for watching and for your generous support, and I'll see you next time!