[Music] welcome to the code mental channel
in today's episode i'm going to show you an ai model that
is able to do lip syncing and i'm going to combine that air model
with some code i've created in the past to use the translation apis from gcp
to translate videos with the code i created before
i was able to translate videos with gcp apis
and using also text to speech and speech to text apis and
now i'm going to be able to go one step further and actually
do lip syncing so without further ado let's get started
okay so first thing i want to show you is the wave to it
ai model that can do uh basically take a video and do lip syncing
it's very cool it works with normal videos so you need someone to be
in the video itself facing forward um it also can work with the
cartoons as you can see here on the right it also works with static images
so you can convert a static image to a video and then it will be able to work
with that move the lips not just the lips and also
the chin so it will look quite natural so i think before we
start doing any gcp work let's just do a bit of an
experiment to see how this model behaves and also let's
try the google collab notebook the google collab notebook is available
from that page click here and you get there so i'm
going to take a video i've created in the past so
here we have a video hello everyone today i will show you how you can
speak more than 33 languages with the google cloud
platform so this is the video i'm going to
change so basically this ai model allows me to
basically provide a video file and then an audio and it will take that
audio and it's going to uh sync the video to that audio of course
you need to have a person there or something with a face
it can be a cartoon it can even be a static drawing
anything really and it will actually sync it so
and this demo here the google cloud demo which i'm using is actually very cool
because anyone can try it and the idea is it uses a gpu that google
makes available for this and uh you know i highly recommend
let's give it a go so this uses nvidia an nvidia gpu and that's why it's
checking here the first thing it does in this collab
notebook is checking for the presence of nvidia the kuda compiler
and we can see here let's run this again this is a safe note
notebook but run it at your own risk okay i am connected to a
nvidia because you can see here so i have a gpu
but if you're not sure there's another way to check it i can go to change
runtime type i can i can already see that i have
gpu fine so second step i need to basically
download a file for the model it's a checkpoint file and that's why
it's trying to mount your google drive so you'll have to basically download the
file which you'll find on the website itself so you'll have to
download this model here and then you have to place it in a
folder inside your google drive so i'll copy the code and now just place
it here and then press enter and it will do that
now it's time to call in the the project itself from github okay
so the production is cloned i already have this
downloaded okay wave to it so that's where i put mine so you need to make
sure that wherever you place your file you
need to figure out the path first so i know my
path is this one so make sure the forest flash is there
and then this is going to copy the model file
with the checkpoints into the the right folder this is already trained so it's
about 400 megabytes okay so this step here
i found it will hang forever because it will ask
actually ask for a yes or no so and i found that i don't really need to run
this right now so i'm not going to run that
so one of the points i have i'm going to actually take this notebook
and i'm going to extend it and i'm going to create a new one which i will
basically add some extra code to do the automatic translation
something to bear in mind is that gcp translates
english to many languages quite nicely but one thing that doesn't always work
very well is the transcription so whenever i do this i'm going to give
you the option of just doing the automatic translation via
the apis or if you just want to provide whatever
text it should say and it will be in english and then you
can translate it to any other language then
we'll do that as well so it'll be useful right if i can i will try to create a
drop down here in gcp so we can specify the language and
it would be quite interactive so the way i'm going to do this i'm
going to upload some sample data here while this is
running i might as well do that okay so i'm going to use the on the
phone so i pull that and i'll pull down the file as well so
two original files so you already installed the libraries they needed
so i'm going to run the next step downloading a face detection
ai model which will allow it to recognize the face whenever
i give it a video the last thing i need to upload is
an audio file which i already have prepared and i have a few audio files
i'm going to experiment with yeah i have like german arabic japanese
and chinese okay so i'm ready to basically try this ai model but the
first thing i need to do is i need to change these paths so i
don't need to to run this actually because i already
put what i needed inside the folder so i i don't need this
yeah i'm going to skip it but now i need to basically
take the file name i want to use so i'm going to use this on the phone quiet
first then the audio let's start with arabic
i'm providing mp3 you can do that as well or it can be a wav file
actually this in the background is using ffmpeg
which is fantastic it can take almost anything you throw at it
if you compile it with all the different options available
and that's actually something that i used in the other project i mentioned
just now so it's an error on the phone quiet
just made a mistake in a path i'm going to download the file first
before i run the other model again because if i don't then it's going to be
overwritten so you make sure before you run another
video or another sound file you make sure
that you download whatever it's inside here so this doesn't take too long
okay it's finished so now i can download it and take a look
let's give it a go and see how good it is
so
so i don't know if you speak arabic but if you do you can let me know in the
comment section if this is good enough um by the way i know i'm aware it's a
bit slow because when i did the original uh sound with the google cloud
i decided to slow it down just to keep up
with my pace you know so now i'm going to try with another
audio um but instead of giving it to the arabic i'm going to give it
the chinese we will see a difference in the video itself but
the lip syncing i think it works as you can see it works pretty well
it's done so let's see how i sound like in chinese
okay you ready for it
it's a bit you can notice my chin moving quite well
okay so one more done so i'm going to try this variation here
uh as the last one i do so this one they actually do a they resize the video
12 hour resolution and then we'll see how
how much better it is i think it's pretty good to be honest you do notice
that my lips change right like when i when you look at the actual
original video itself so you see my lips here are much bigger
thicker lips when you look at the video itself look
at the difference yeah so you see there's a subtle
difference here in the lips
some of you might be a bit disturbed by whatever you're saying so
i apologize this is only used for good but yeah i know what to
you might be feeling right now which is where is technology going today because
i literally can in five minutes create a video basically where i'm saying
something i wasn't even talking and you know what uh with when voice
technology becomes even better you won't even be able to
recognize whether it's me or not these research is obviously uh they are
brilliant right if they can do this so easily imagine what the government
might be able to do with a little more budget don't trust
what you see anymore that's something i can say because
you can't you know don't trust it you have to be careful
always double check cross-reference right next one
so i'm going to try to lower the resolution of this video
which i honestly don't think it's going to make a huge difference because the
video itself is not very high resolution okay and i'm
going to swap to japanese which personally i think i
sound cooler right so let's see if this is any good
okay so you can see it's smaller
it's good okay so it's a it's a comparable result but to be honest i
prefer the higher resolution because i think the high resolution is already
pretty good