The other day I saw an article on the Verge talking about a lip sync deepfake technique
that anyone could do using the wave2lip algorithm on GitHub.
This technique uses a Google Collab notebook and all of the computing happens in the cloud.
Often when I see a tutorial about deep learning or artificial intelligence image manipulation
I get excited and want to try it only to be foiled by its complexity.
I usually find that I am missing code dependencies and have to install various python libraries.
Once I get those libraries installed I then find out that I don't have a powerful enough
GPU or other necessary computing hardware.
This technique uses the Collab notebook provided by the repository creator and allows anyone
to have fun creating lip-synced deepfaked videos.
I think it's important for as many people as possible to understand this technology
because it will become more and more prevalent and we need to have media literacy to know
what could potentially be real and what is not.
We need an informed public that can debate about future legislation or needed regulation
about these technologies as they become more powerful.
For now, I think that deep learning has lots of potential creative uses for animation and
other visual communication.
The first thing you need to do is go to Google Drive make sure you're at the My Drive level
of Google Drive.
You need to create two folders.
The first folder you will create is Wav2lip with a capital W in a lowercase l.
Then create a folder called Wav2Lip with a capital W and capital L. Once these two folders
are created, go to this Wav2lip GitHub repository.
It has all the needed code and pre-trained models that you need to make your lip sync
work.
It also has a live interactive demo and instructions on how to use the code on your own computer.
First you need to download the pre-built model so you do not have to train a model yourself.
Go to this link to download the file.
Click download and save it on your computer.
While that's downloading go back to the GitHub page.
At the GitHub page, click on the Google Collab notebook.
This will allow you to run all this code and its dependencies in the cloud.
All of the commands are already filled out for you.
Thanks internet.
You need to click these little arrows in order to run the code that's already been typed
in.
Click the first one.
Make sure you allow the notebook to run because it's not hosted by Google.
Then click the second one.
You need to give the notebook access to your Google Drive.
Click on this link, then select the Google account you want to use.
Click accept to give it the permissions it needs.
Then a code will display on the screen.
Copy the code.
Go back to the Collab notebook and paste it into the box and press enter.
Now you can continue clicking all of the arrows.
When you get to this section, you need to make sure that you have the pre-trained model
in the Wav2lip folder with the lowercase l.
Copy the file from your computer to that folder before clicking the arrow.
Click each of the arrows in order.
There is one other spot that might trip you up.
When it is uninstalling the tensorflow GPU you need to say yes.
So just press the letter Y and then press enter once.
It will look like you need to press it again but once you've pressed Y and enter it's fine.
It takes a little bit of time for everything to load.
I've sped this up considerably for you.
The entire process in this video from blank a Google Drive to a finished lip synced video
took about 10 minutes.
Once everything above “Let’s Try” is loaded you need to get your video and audio
files into Google Drive into the Wav2Lip folder with two capital letters.
Drag your MP4 video file and your wav file into the folder.
Make sure there are no spaces in the file names.
Back in the Google Collab notebook, notice that it has default files input_vid.mp4 and
input_audio.wav.
You could use these file names but you can also change them to the names of your videos.
Once you have lots of videos in your Google drive folder it's best to be able to change
this name in the notebook.
Remember to change it in both places otherwise it won't work.
Once both of the file names are changed, you can click the first arrow and then click the
second arrow.
This will run the lip-syncing algorithm on your video and audio files.
It takes a bit of time and you can see the progress.
If you get an error make sure that all of your video frames have a face in it.
If you have one blank frame at the end of your video it won't work, so make sure you
trim your video so it has a face and every frame.
Once the process is complete go to the folder icon on the left.
In the Wav2Lip folder there is a results folder that will have your lip synced video.
Download it to your computer.
Ask not what artificial intelligence can do for you, ask what you can do for artificial
intelligence.
There's also some other options in the notebook.
For example, if you scroll down you'll see that you can try adding padding to animate
more of the chin of the face during the lip sync.
There is also an option to reduce the size of the video to get a better result sometimes
this works because the model was trained on low-resolution video and if you use high resolution
video it may make mistakes.
You can also combine these by putting both of the parameters in the same line.
Remember that you have to change the file names so change both the file name of the
video and the audio file and you can click run and you will have another video result.
Notice that it does not make a new file here, it just overwrites the old one so make sure
after you run it each time you download the new video to your computer so you're not overwriting
all your work.
Your computer will increment the filenames but the collab notebook does not increment
the filenames it just overwrites them.
You can also rename it in the notebook but it's easier to just download it.
You can also take a still image and put it into an mp4 file by stretching it out in a
video editing program.
You can then lip sync this drawing or painting.
Make sure you have changed the file name and then run the lip sync algorithm.
When it's done, download the video.
Now you have two lip sync videos saying the same thing.
I think this technique of lip syncing drawings has a lot of potential for animation.
Good luck, have fun and use this responsibly.
Although there are lots of potentially bad uses for this technology there are just as
many creative uses for making animated drawings and other creative visual content.
Be good.