there i hope you liked the demo in that demo
i actually demonstrated two very cool ai technologies
the first ai technology i want to highlight
and it's actually the main subject of this video is actually wave tweet
wave tweet is like uh revolutionary you know we basically can
take any video of someone actually speaking and you can feed it
like any other you like with someone else speaking
or even any sound and you can make this uh video
or whoever is in that video the face uh the lips will actually sync to the sound
that you give it it's not 100 perfect but it's uh way
very close to that with a little bit more work
uh i think this can be used in in many different scenarios like the one
i've used today first thing i want you to do is to visit
this website the github repository for wave to lip or you'll
find all the source code and you'll find some demos there and
also some information about authors of of this new uh ai model
and they are like four or five uh researchers uh
one of them is based in uh in cambridge i think
uh or in oxford and uh the other four uh they are basically
in india they are their work is amazing so that's the only thing i can say now
i'm going to go into more detail of the actual demo they've created
uh in a separate video uh what i'm going to do today
is actually show you how i've combined this uh
wavetrip ai technology with another technology
a very cool ai as well which is from google and it's
uh basically to generate synthetic voices because you heard me in a video
i was speaking what korean spanish french and you know i could have gone on
and on really i could have spoken like 33
language if i wanted to but obviously nobody would have that much
patience for for me so i just did the sample
right but you could see those uh voices in my opinion are very close to
uh to the point where you might not even be able to recognize that these are
syntactic voices and if you're not expecting it you might
not even notice so uh the cool thing about wavenet is that
it can generate a natural sounding speech and it can do it
very fast and another very cool thing about it is
that you can actually use this technology today
using a google cloud api and i go into so much detail on how you can use these
apis but the purpose of this video is
actually just to give you a like a teaser you know of everything you
can do and i'm going to be releasing which are
going to go into more detail so the first video i will be releasing
is the wave to whip video where i'm just going
to focus on wave to it and i'm going to uh
try the demo they have then i will also have a video on how
i basically use the the google cloud apis
for wavenet to generate synthetic voices and finally i'll show you how i put
everything together where i actually i took this google
collab notebook which was originally developed by the
researcher team that did wavetlip and they've kindly
given it you know for you to try you can actually try their website or you can
also go to google app so just bear in mind if you try the demo
in the website uh there is some limitations here you
can't really give very large videos if you give very large videos like it
would not work in fact i struggled to actually use this
interactive demo website so if i were you i would go straight to
the demo in google cloud so google cloud if you
don't know it's actually a basically a notebook a python notebook
which is hosted by google and gives you access to all their
servers with a gpu or with a tpu so i highly recommend for today for this
video i'm just going to show you the actual notebook i've created which
is based on the notebook creator that you can
find here the link is here this is the original google cloud notebook so it's a
wave to it quick trial so this is the original google
cloud notebook which you can try it's easy but i've actually modified
this notebook and i added some additional
functionality just to give you an idea you have here
ability of calling uh the gcp apis which you do have to enable first and you do
need to uh create like a secret key so you can actually
use it in google speak that's called a service account
and then you need to download the json file
and then you need to upload it here anyway if you do that then you'll be
able to use the google translation api the google the google text to speech api
the google speech to text api and so on another
cool thing i've added and i think that will be um that will be
useful uh i've added actually a small script with ffmpeg
ffmpeg is basically a video editing tool it's basically a script
that you can use to do all kinds of stuff with video
with this tool i actually can create a small video out of an image
and you can see that we actually can animate like a cartoon like
image and that's very cool and i'll show you how i've done that
for one example so today it's all about showing the
end result right but if you like this you will be able to go into more detail
because i'm going to release videos on and i will show exactly how to develop
this in fact i've recorded all of this it's just being edited right now and
you'll be able to see all of it and i think it's quite
interesting so you might learn a lot more
out of that than just watching this video so check it out
i'm going to be releasing it uh the week after i release this video
so if you want to actually uh be up to date
definitely consider subscribing because you never know when i'm going to release
it it'll be uh sometime during the week so without further ado let's get started
right so the first thing i do so i have this link here
which i'm going to put in the description so all you have to do
is basically click on the link which will be in the description of the video
and you basically load this website and i'm doing it again so you can see it
from scratch okay next you need to click connect yeah
so it's trying to go connect so what's going to happen google is going to give
you a machine that you can use with a gpu
but just double check that you have a gpu there's a couple
ways you can double check you have a gpu the first way you can double check
is just check many sessions that's one way and you'll see here gpu
okay but don't take my word for it the actual developers of the original
notebook also put here a check for you so this will tell me if
it's nvidia gpu because this notebook is only
compatible with the nvidia gpu so if you run this
ignore the warning it will tell you that this is a nvidia gpu so if this fails
in any way and in shape or form then you know you are in the wrong server so you
need to disconnect and and try again and the way to make sure
you are on a gpu you basically just let me go here you go
oh so you click change runtime type and then you can
change whatever you want here you don't need the tpu you can just have a gp a
tpu is basically uh like a tensor processing unit that
has some bespoke basically that google
design okay so we've done that another thing
you will have to do you're going to have to upload
a file into your google drive i'm not going to show you here how you're going
to do that um because i'm going to it's already covered in the next video
you'll see so i'm going to assume you have it there
and i'm going to upload a service account for google and
again i'm assuming you already know how to do that because i've showed
i'm showing that in a separate video right so i have my json file here
keep it safe i'm putting in a server because it's going to be recycled
afterwards so you don't have to worry about it and
when you create this service account just give you enough permissions
okay so now we have the service account we just have to go through these
steps here install all the dependencies all right so
it has run this so you need to now click restart runtime you only need to do it
once even though it asks you a couple of times
and then you can go to the next step so you can skip this step here
because we don't need it just yet this is for another
type of video that i'm going to show you how you can create later
oh i shouldn't have clicked here so here you need to make sure
that your service account path is correct so based on the folder i upload
uploaded here you'll be able to copy the path and then you place it here
yeah then it will modify it so you do it on the right hand side you don't need to
do it on the left hand side so next thing you need to do this is for
you to basically specify the language that you want to
output to so right now it's korean so it's if i let it be like this then it's
going to uh generate the video with me
speaking korean or someone else speaking korean so i'm going to leave it for now
you can change with two different languages no problem
i have to upload the video so i'm going to upload the video
okay so i'm going to use a video clip where i speak english
so it's uploading and while it's uploading i'm just going to show you the
video okay so this is the video i've uploaded
today i will show you how you can speak more than 33 languages with the
google cloud platform so the video is already
uploaded i'm just going to copy the path click play to make sure that it records
all these variables so the first demo i'm going to show you
will be to translate this video into caring
so the next step is to basically mount your g drive basically
you have to click on the link and you have to basically select the account
you want to use and you have to click allow to uh basically
trust all the permissions they need to give then you have to copy the code
and and then you place the code here so now it's mounted i can go to the next
step basically i'm calling the the wave to
whip repository getting out the source code
now check my g drive in my g drive i i've already
created a wave to it folder with this basically this model
file which has basically has already been
trained so you click on this link and you
download it and then you upload it to your g
drive on the path that is mentioned here yeah so you need to
create a folder inside your g drive i've already mounted the drive so i'm just
going to copy the file okay so now i need to just run through
these steps to install other the packages it will
take a bit of time so i'm going to fast forward this
so it has finished installing all the prerequisites and now we need to click
on the next step which is basically downloads a face detection model which
is used by the wave to leap algorithm to detect faces
in a video just be mindful that if you actually use
a video that has a section without the face most likely you'll have
an error when it tries to do the lip syncing and now
so i've added all these functions here these functions are used to interact
with the with the apis in the google cloud okay
first of all i'm actually at this function here
is to extract audio from a video we have a method here to upload
a file to cloud storage which we don't need actually a function to
convert speech to text to translate to do a text
speech which uses in this case webnet we have even a method to list all the
voices available so we can pick a voice that we like okay so you need to make
sure you load this so all the methods are loaded into
memory so next thing we can do taking into account that original video
is uh i've already shown you right uh it's i'm
speaking english so we need to extract the audio
from that video we need to basically transcribe it from
audio to text it's done this is the translation and
i'm going to probably have to modify it a little
bit bear in mind that the transcription is not 100 correct so you need to make
some changes yourself you can also add the delay um let me
just check the delay i have here so i i've added a very simple delay of
three seconds so it doesn't start straight away so it has a pause you can
use ssml ssml is a way to describe like
speech using text you can add pauses you can do many other things
i'm just going to check what voices i have available for korean
i'm just going to try one of them first so these are once i run this it's going
to translate this transcript here in english
to korean and i can listen to it now
right so continuing uh so we can see already the sound is working but this is
a female and obviously i'm not a female so let's see if i can do another voice
and see if it's any better
one thing that you you should bear in mind is that
this text speech actually will obey any full stops or the comments that you have
so you add some pauses so now that i have d i can basically go
to the next step this is good so now i'm going to
basically take this sound here and the original video i
created and i'm going to replace the original
sound with with a new sound in korean it takes a bit of time so you have to be
patient in the meantime i'm going to look for
a picture that i already took and that will be
the next demo i'm going to do so wavenet plus wave to whip
that's
yeah this is good so the sections where i'm talking
and the new audio is also uh talking uh then it's okay it will actually
change my lips to match the the new audio but when
there is any silence if the original video itself has any uh
any lips movement right it will still be the original lips
movement it doesn't necessarily make me quiet
okay so we've done this this is the main demo i wanted to show
you and now i wanted to add an extra here
just for fun you can actually use this algorithm
with cartoons and i've created the two cartoons of myself so one and two
so here i'm going to use these two images
to create two videos right where i'll be speaking
something in either japanese or or english i'll just change the language
for the sake of it and you'll see how it looks so
without further ado let's do that so i'm going to just upload the images
i've uploaded the two images so i'm just going to go back to that
cell which i skipped initially the generate video from image so you just
basically have to change the path just copy one of
the images i'm going to copy the first path so i'm
just going to create a small video you can specify how long you want
the video for so i'm specifying 15 seconds so you can see
it's quite it's very static there's nothing happening but you can
see that this is how it looks like original let me copy the path
here and i'm going to place it here to
replace the other video click play so i'm going to change the
language to japanese click play again to make sure
it records then i'm just going to change
to do to rerun this to translate again oh wait
before i do that i need to make sure i get all the voices available
so i get because i'm changing to japanese now i need to change it to
another wavenet so i'm going to try and guess what the
male voice is let me try this one it's quite quick
okay so this is the sound now let's try and create a video out of this
okay so it's done so let's see how it looks
my first
okay if i wanted to go any further i would try this one as well
i know it works i've already tried it so okay so this is it
this is the demo i wanted to give you i i didn't go into much detail i didn't go
into any code but you know this is going to be the
next episodes that you uh you can look forward to so i'll
be going detail i'll basically be coding
all these in real time so you might as well you'll
be able to understand more or less the apis the gcp apis i
used how i put everything together and i
think you will get a lot out of it yeah that's it thank you so much for uh
watching this video and i hope you got exactly
what you wanted out of it and i promise you if you watch my next
videos on this topic you'll get a lot more detail and you're
going to be able to reproduce this yourself in fact you can already
reproduces yourself because i'm going to give you the link to this
google collab notebook and i'll put that into the description and
if you already know how to work with the gcp you'll be able to do it if not then
just wait for my next video i will basically go
show you how to set that up alright thank you so much
see you again soon happy coding