Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Earlier we talked about AI-based techniques that can learn to clone your voice, and then,
we can perform text to speech. So, it would learn this. And then, the AI would generate this. Yes,
this means that we can write something, and the AI says it in our voice. So that
is text to speech. But get this, what about speech to gesture? What is that?
Well, scientists at Ubisoft had a crazy idea. They said, let’s create a dataset
of characters where the AI can see gestures for agreement, disagreement, being relaxed, neutral,
scared, and more, learn from it, and apply it to new virtual characters intelligently.
Sounds good, right? But, not so fast. Ubisoft is not the first to try this idea, here is a
technique from just two years ago. Look. There is something here, but we are humans, at least,
most of us anyway, and our eyes are highly attuned to the gestures of other humans, which means that
if even the smallest part of the animation is off, we will immediately notice. And I
think that every single one of you Fellow Scholars indeed noticed that something is off here. And
based on the quality of the animation that is required today to keep the illusion up, I am not
sure if this is coming to fruition anytime soon. Just look at these examples. We are so far away.
But in any case, let’s have a look at the new technique together. The concept is the same,
first, in goes a speech sample, and it will be able to generate long sequences using a style of
our choosing. Like this. That looks incredible. Let’s compare it to the previous method.
Yes, my goodness! A night and day difference! Such incredible progress in just two years. Yes,
this is from only two years ago. And that would already be pretty cool,
but, it gets crazier than that. Much crazier.
For instance, we can plug in a new speaker the AI hasn’t heard about yet,
and this happens. This is really good. But that’s still nothing. It generalized to not
only new speakers, but hold on to your papers, to new languages too. Listen.
Wow, and the list of features just keeps on going. For instance, we can also exert some
artistic control here. If we feel that the hands are too high up, we can lower them.
Or, we can even ask for more or less hip movement during the monologue.
And I have to say, it does a lot of things really well from the lowest energy level gestures, up to
the highest. And it still doesn’t stop there, it is not only better than the previous technique,
is not only more controllable than the previous technique, but is it also about 7 times faster,
and even better, it is done with a neural network that is significantly simpler than that one.
How many hours of these gestures did the neural network get access to to
learn it? How big was the training set size? Actually, it requires very little information.
All it was given was about 2 hours of these gestures, and it is not copy-pasting, but,
it can continue these gestures for a long-long monologue seamlessly. And as a cherry on top,
even some creative control is allowed. So, these amazing virtual worlds are going to
be even more full of lifelike characters, and with this work,
this process is going to be even easier and more accessible for all of us. What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!