Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Imagine that you are a film critic and you are recording a video review of a movie, but
unfortunately you are not the best kind of movie critic, and you record it before watching
the movie.
But here is the problem - you don’t really know if it’s going to be any good.
So you record this.
So far so good.
Nothing too crazy going on here.
However, you go in, watch the movie, and it turns out to be amazing.
So, what do we do if we don’t have time to re-record the video?
Well, we grab this AI, type in the new text, and it will give us this.
Whoa!
What just happened?
What kind of black magic is this?
Well, let’s look behind the person, on the blackboard you see some delicious partial
derivatives, and I am starting to think that this person is not a movie critic.
And of course, of course he isn’t, because this is Yoshua Bengio, legendary machine learning
researcher.
And this was an introduction video where he says this.
And, what happened is that it has been repurposed by this new DeepFake generator AI, where we
can type in anything we wish, and out comes a near-perfect result.
It synthesizes both the video and audio content for us.
But we are not quite done yet.
Something is missing.
If the movie gets an A+, the gestures of the subject also have reflect that this is a favorable
review.
So, what do we do?
Maybe add a smile there.
Is that possible?
Oh yes, there we go!
Amazing.
Let’s have a closer look at one more example where we see how easily we can drop in new
text with this editor.
Now, this is not the first method performing this task - previous techniques typically
required hours and hours of video from a target subject.
So how much training data does this require to perform all this?
Well, let’s have a look together!
Look, this is not the same footage copy-pasted three times, this is a synthesized video output
if we have 10 minutes of video data from the test subject.
This looks nearly as good, has fewer sharp details, but, in return, this only requires
2.5 minutes.
And here comes the best part.
If you look here, you may be able to see the difference.
And if you have been holding on to your papers so far, now, squeeze that paper because synthesizing
this only required 30 seconds of video footage of the target subject.
My goodness.
But we are not nearly done yet, it can do more!
For instance, it can tone up or down the intensity of gestures to match the tone of what is being
said.
Look.
So how does this wizardry happen?
Well, this new technique improves two things really well, one is that it can search for
phonemes and other units better.
Here is an example, we crossed out the word “spider” and wish to use the word “fox”
instead, and it tries to assemble this word from previous occurrences of individual sounds.
For instance, the “ox” part is available when the test subject utters the word “box”.
And two, it can then stitch them together better than previous methods.
And surely this means that since it needs less data, the synthesis must take a great
deal longer, right?
No, not at all, the synthesis part only takes 40 seconds.
And even if it couldn’t this so quickly, the performance control aspect where we can
tone the gestures up or down, or add a smile would still be an amazing selling point in
and of itself.
But no, it does all of these things quickly, and with high quality at the same time.
Wow.
I now invite you to look at the results carefully, and give them a hard time.
Did you find anything out of ordinary?
Did you find this believable?
Let me know in the comments below.
The authors of the paper also conducted a user study with a 110 participants, who were
asked to look at 25 videos and say which one they felt was real.
The results showed that the new method outperforms previous techniques, even if they have access
to 12 times more training data.
Which is absolutely absolutely amazing, but what is even better, the longer the video
clips were, the better this method fared.
What a time to be alive!
Now, of course, beyond the many amazing use-cases of DeepFakes in reviving deceased actors,
creating beautiful visual art, redubbing movies, and more, we have to be vigilant about the
fact that they can also be used for nefarious purposes.
The goal of this video is to let you, and the public know that these DeepFakes can now
be created quickly and inexpensively, and they don’t require a trained scientist anymore.
If this can be done, it is of utmost importance that we all know about it!
And beyond that, whenever they invite me, I inform key political and military decision
makers about the existence and details of these techniques to make sure that they also
know about these and using that knowledge, they can make better decisions for us.
You can see me doing that here.
Note that these talks and consultations all happen free of charge, and if they keep inviting
me, I’ll keep showing up to help with this in the future as a service
to the public.
Thanks for watching and for your generous support, and I'll see you next time!