Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Recently, we have experienced an abundance of papers on facial reenactment in machine
learning research.
We talked about a technique by the name Face2Face back in 2016, approximately 300 videos ago.
It was able to take a video of us and transfer our gestures to a target subject.
This was kind of possible at the time with specialized depth cameras, until Face2Face
appeared and took the world by storm as it was able to perform what you see here with
a regular consumer camera.
However, it only transferred gestures, so of course, scientists were quite excited about
the possibility of transferring more than just that.
But, that would require solving so many more problems - for instance, if we wish to turn
the head of the target subject, we may need to visualize regions that we haven’t seen
in these videos, which also requires an intuitive understanding of hair, the human face and
more.
This is quite challenging.
So, can this be really done?
Well, have a look at this amazing new paper!
You see here the left image, this is the source person, the video on the right is the target
video, and our task is to transfer not just the gestures, but the pose, gestures and appearance
of the face on the left to the video on the right.
And, this new method works like magic.
Look!
It not only works like magic, but pulls it off on a surprisingly large variety of cases,
many of which I haven’t expected at all.
Now, hold on to your papers, because this technique was not trained on these subjects,
which means that this is the first time it is seeing these people.
It has been trained on plenty of people, but not these people.
Now, before we look at this example, you are probably saying, well, the occlusions from
the microphone will surely throw the algorithm off, right?
Well, let’s have a look.
Nope, no issues at all.
Absolutely amazing, love it!
So how does this wizardry work exactly?
Well, it requires careful coordination between no less than four neural networks, where each
of which specializes for a different task.
The first two is a reenactment generator that produces a first estimation of the reenacted
face, and a segmentation generator network that creates this colorful image that shows
which region in the image corresponds to which facial landmark.
These two are then handed over to the third network, the inpainting generator, which fills
the rest of the image, and since we have overlapping information, in comes the fourth, blending
generator to the rescue to combine all this information into our final image.
The paper contains a detailed description of each of these networks, so make sure to
have a look!
And if you do, you will also find that there are plenty of comparisons against previous
works, of course, Face2Face is one of them, which was already amazing, and you can see
how far we’ve come in only three years.
Now, when we try to evaluate such a research work, we are curious as to how much these
individual puzzle pieces, in this case, the generator networks contribute to the final
results.
Are really all of them needed?
What if we remove some of them?
Well, this is a good paper, so we can find the answer in Table 2, where all of these
components are tested in isolation.
The downward and upward arrows show which measure is subject to minimization and maximization,
and if we look at this column, it is quite clear that all of them indeed improve the
situation, and contribute to the final results.
And remember, all this from just one image of the source person.
Insanity.
Thanks for watching and for your generous support, and I'll see you next time!