Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
This work is about creating facial animation from speech in real time.
This means that after recording the audio footage of us speaking, we give it to a learning
algorithm, which creates a high-quality animation depicting our digital characters uttering
these words.
This learning algorithm is a Convolutional Neural Network, which was trained on as little
as 3 to 5 minutes of footage per actor, and was able to generalize its knowledge from
this training data to a variety of real-world expressions and words.
And if you think you've seen everything, you should watch until the end of the video as
it gets better than that because of two reasons.
Reason number one, it not only takes audio input, but we can also specify an emotional
state that the character should express when uttering these words.
Number two, and this is the best part, we can also combine this together with DeepMind's
WaveNet, which synthesizes audio from our text input.
It basically synthesizes a believable human voice and says whatever text we write down.
And then that sound clip can be used with this technique to make a digital character
say what we've written.
So we can go from text to speech with WaveNet, and put the speech onto a virtual actor with this work.
This way, we get a whole pipeline that works by learning and does everything for us in
the most convenient way.
No actors needed for voiceovers.
No motion capture for animations.
This is truly incredible.
And if you look at the left, side, you can see that in their video, there is some Two
Minute Papers action going on.
How cool is that?
Make sure to have a look at the paper to see the three-way loss function the authors came
up with to make sure that the results work correctly for longer animations.
And of course, in research, we have to prove that our results are better than previous
techniques.
To accomplish this, there are plenty of comparisons in the supplementary video.
But we need more than that.
Since these results cannot be boiled down to a mathematical theorem that we need to
prove, we have to do it some other way.
And the ultimate goal is that a human being would judge these videos as being real with
a higher chance than one made with a previous technique.
This is the core idea behind the user study carried out in the paper.
We bring in a bunch of people, present them with a video of the old and new technique
without knowing which is which, and ask them which one they feel to be more natural.
And the result was not even close - the new method is not only better overall, but I haven't
found a single case, scenario or language where it didn't come out ahead.
And that's extremely rare in research.
Typically, in a maturing field, new techniques introduce a different kind of tradeoff, for
instance, less execution time but at the cost of higher memory consumption is a classical
case.
But here, it's just simply better in every regard. Excellent.
If you enjoyed this episode, and would like to help us make better videos in the future,
please consider supporting us on Patreon.
You can pick up cool perks like watching these episodes in early access.
Details are available in the video description.
Beyond telling these important research stories, we're also using part of these funds to empower
other research projects.
I just made a small write-up about this which is available on our Patreon page.
That link is in the video description, make sure to have a look.
Thanks for watching and for your generous support, and I'll see you next time!