you've most certainly seen movies like
the recent captain marvel or gemini man
where samuel l jackson and will smith
appear to look like they were much
younger this requires hundreds if not
thousands of hours of work from
professionals manually editing the
scenes he appeared in instead you could
use a simple ai and do it within a few
minutes indeed many techniques allow you
to add smiles make you look younger or
older all automatically using ai-based
algorithms they are mostly applied to
images since it's much easier but the
same techniques with small tweaks can be
applied on videos which as you may
suspect is quite promising for the film
industry and by the way the results
you've been seeing were all made using
the technique i will discuss in this
video the main problem is that currently
these generated older versions edited
images not only seem weird but when used
in a video will have glitches and
artifacts you surely do not want in a
million dollar movie this is because
it's much harder to get videos of people
than pictures making it even harder to
train such ai models that require so
many different examples to understand
what to do this strong data dependency
is one of the reasons why current ai is
far from human intelligence this is why
researchers like rotem saban and
collaborators from tel aviv university
work hard to improve the quality of
automatic ai video editing without
requiring so many video examples or more
precisely improve ai based face
manipulations in high quality talking
head videos using models trained with
images it doesn't require anything
except the single video you want to edit
and you can add a smile make you look
younger or even older it even works with
animated videos this is so cool but
what's even better is how they achieve
that but before doing so let me talk
about the sponsor of this video um
there are no sponsors for this video so
if you could just take a second to give
it a thumbs up and leave a comment about
what you think of the model or how you'd
apply it after watching the video or
even how you feel today that will be
amazing and i can promise you i will
answer within 12 minutes you can time it
so how does it work of course it uses
cans or generative adversarial networks
i won't go into the inner workings of
guns since i already covered it in a
video that you can watch right here and
linked below but we will see how this is
different from a basic gun architecture
if you are not familiar with guns just
take a minute to watch the video and
come back i'll still be there waiting
for you and i'm not exaggerating the
video literally takes one minute to get
a simple overview of what gans are we
will just refresh the part where you
have a generative model that takes an
image or rather an encoded version of
the image and changes this code to
generate a new version of the image
modifying specific aspects if possible
controlling the generation is the
challenging part as it has so many
parameters and it's really hard to find
which parameters are in charge for what
and disentangle everything to only edit
what you want so it uses any generative
based architecture such as style gun in
this case which is simply a powerful gan
architecture for images of faces
published by nvidia a few years ago with
still very impressive results and newer
versions but the generative model itself
isn't that important as it can work with
any powerful gan architecture you can
find and yes even if these models are
all trained with images they will use
them to perform video editing assuming
that the video you will send is
realistic and already consistent they
will simply focus on maintaining realism
rather than creating a real consistent
video as we have to do in video
synthesis work where we create new
videos out of the blue so each image
will be processed individually instead
of sending a whole video and expecting a
new video in return this assumption
makes the task way simpler but there are
more challenges to face like maintaining
such a realistic video where each frame
fluently goes to the next without
apparent glitches here they will take
each frame of the video as an input
extract only the face and alloying it
for consistency which is an essential
step as we will see then they will use
their pre-trained encoder and generator
to encode the frames and produce new
versions for each unfortunately this
wouldn't fix the realism problem where
the new faces may look weird or out of
place when going from one frame to
another as well as weird lighting bugs
and background differences that may
appear to fix that they will further
train the initial generator and use it
to help make the generations across all
frames more similar and globally
coherent they also introduce two other
steps an editing step and a new
operation that they call stitching
tuning the editing step will simply take
the encoded version of the image and
change it just a bit this is the part
where it will learn to change it just
enough to make the person look older in
this case so the model will be trained
to understand which parameters to move
and how much to modify the right
features of the image to make the person
look older like adding some gray hair
adding wrinkles etc then this stitching
tuning model will take the encoded image
you see here and will be trained to
generate the image from the edited code
that will best fit the background and
other frames it will achieve that by
taking the newly generated image
comparing it with the original image and
finding the best way to replace only the
face using a mask and keep the rest of
the cropped image unchanged
finally we paste the modified face back
on the frame this process is quite
clever and allows for the production of
really high quality videos since you
only need the cropped and aligned face
in the model incredibly reducing the
computation needs and complexity of the
task so even if the face needs to be
small let's say 200 pixels square if
it's only a fifth of the image as you
can see here you can keep a pretty high
resolution video and voila this is how
these great researchers perform high
quality face manipulation in videos i
hope you enjoyed this video let me know
how you feel about this one if you liked
it or not any feedback will be amazing
this is the last opportunity you have to
make mighty by clicking the like button
and leaving a comment before you go of
course the link to the paper and code
are in the video's description note that
the code will only be released on
february 14th as per the author's github
thank you for watching
[Music]