Dear Fellow Scholars, hold on to your papers, because we have an emergency situation.
Everybody can make deepfakes now by recording a voice sample, such as this one, and the
lips of the target subject will move as if they themselves were saying this.
Not bad huh?
And now to see what else this new method can do, first, let’s watch this short clip of
a speech, and make sure to pay attention to the fact that the louder voice is the English
translator, and if you pay attention, you can hear the chancellor’s original voice
in the background too.
So what is the problem here?
Strictly speaking, there is no problem here, this is just the way the speech was recorded,
however, what if we could recreate this video in a way that the chancellor’s lips would
be synced not to her own voice, but to the voice of the English interpreter?
This would give an impression as if the speech was given in English, and the video content
would follow what we hear.
Now that sounds like something straight out of a science fiction movie, perhaps even with
today’s advanced machine learning techniques, but let’s see if it’s possible.
This is a state of the art technique from last year that attempts to perform this.
Hmm…there are extraneous lip movements which are the remnants of the original video, so
much so that she seems to be giving two speeches at the same time.
Not too convincing.
So, is this not possible to pull off?
Well, now, hold on to your papers, let’s see how this new paper does at the same problem.
Wow, now that’s significantly better!
The remnants of the previous speech are still there, but the footage is much, much more
convincing.
What’s even better is that the previous technique was published just one year ago
by the same research group.
Such a great leap in just one year, my goodness!
So apparently, this is possible.
But I would like to see another example, just to make sure.
Checkmark.
So far, this is an amazing leap, but believe it or not, this is just one of the easier
applications of the new model, so let’s see what else it can do!
For instance, many of us are sitting at home, yearning for some learning materials, but
the vast majority of these were recorded in only one language.
What if we could redub famous lectures into many other languages?
Look at that!
Any lecture could be available in any language and look as if they were originally recorded
in these foreign languages as long as someone says the words.
Which can also be kind of automated through speech synthesis these days.
So, it clearly works well on real characters…but.
Are you thinking what I am thinking?
Three, what about lip syncing animated characters?
Imagine if a line has to be changed in a Disney movie, can we synthesize new video footage
without calling in the animators for a yet another all-nighter?
Let’s give it a try!
Indeed we can!
Loving it!
Let’s do one more.
Four, of course, we have a lot of these meme gifs on the internet.
What about redubbing those with an arbitrary line of our choice?
Yup, that is indeed also possible.
Well done!
And imagine that this is such a leap just one more work down the line from the 2019
paper, I can only imagine what results we will see one more paper down the line.
It not only does what it does better, but it can also be applied to a multitude of problems.
What a time to be alive!
When we look under the hood, we see that the two key components that enable this wizardry
are here and here.
So what does this mean exactly?
It means that we jointly improve the quality of the lip syncing, and the visual quality
of the video.
These two modules curate the results offered by the main generator neural network, and
reject solutions that don’t have enough detail or don’t match the speech that we
hear, and thereby they steer it towards much higher-quality solutions.
If we continue this training process for 29 hours for the lip-sync discriminator, we get
these incredible results.
Now, let’s have a quick look at the user study, and, humans appear to almost never
prefer the older method compared to this one.
I tend to agree.
If you consider these forgeries to be deepfakes, then… there you go!
Useful deepfakes that can potentially help people around the world stranded at home to
study and improve themselves.
Imagine what good this could do!
Well done!
Thanks for watching and for your generous support, and I'll see you next time!