Today I'm gonna show you how to convert footage shot
at a standard frame rate into a high-speed frame rate
using deep learning. And if you've thought
some of my other videos were nerdy when it comes to
talking about camera stuff... Buckle up.
Let's get undone.
[offbeat music]
♪ Gerald Undone ♪
♪ He's crazy ♪
What's happening, everybody, I'm Gerald Undone, and this...
is the final countdown. ["Final Countdown" opening]
So, NVIDIA has developed this deep learning interpolation
technique called Super SloMo that will convert videos
shot at standard frame rates of 24 or 30 fps
into slow-motion-capable frame rates like 120 or even 240 fps.
And shout-out to a subscriber, Noel B,
for suggesting that I cover this topic.
Now, I'll warn you right now that this process
is a bit more complicated than your typical tutorial.
But even if you don't think that this will be applicable
to you, I recommend that you stick around and watch
this whole video to see the results, because some of them
are amazing. Just so you know there's no funny business
going on, I'm gonna be using stock footage for the test,
so the results can be repeated by anyone.
And that stock footage is brought to us by
today's sponsor, Storyblocks, which is my go-to source
for not just stock footage, but also backgrounds,
overlays, and even After Effects templates.
The interface is clean and the clips are easy to find,
and it's a one-time fee at a fair price that gives you access
to unlimited downloads including the 4K footage.
And the clips are royalty free for personal and commercial use
so you can use them as much as you want, wherever you want.
So, if you're in the need for a solid library
of stock footage and effects, head on over to Storyblocks
using the link in the description below.
Alright, so I've downloaded several different clips
from Storyblocks already of various subjects that we can see
how it performs in different scenarios.
But first, let me show you how to do it.
Now, I'm gonna put detailed written instructions in
the description below as well as show you here visually,
but this is gonna require two different things
to do it yourself. First, you'll need to carry out
some Python commands, which means dealing with
a little bit of code and installing some software
and repositories to go along with that.
And two: you'll want a modern NVIDIA graphics card.
This technique was built on NVIDIA's CUDA platform
just like you'd use to edit video if you have an NVIDIA GPU.
Now, you can do it on your CPU alone
if you don't have an NVIDIA graphics card,
but it's so much slower that it's probably not worth it.
For example, I converted a 17-second clip
in 12:51 using CUDA,
but when I try to convert that same clip without using
my graphics card, it took six and a half hours.
Alright, so the first step is to install a Python distribution
and for this I recommend Anaconda, which is available
for Windows and Mac OS, and works really well.
Then you'll need to download
the actual Super SloMo repository,
which you'll find a link for that as well as everything else
that we talk about, in the description below,
that goes along with each corresponding step.
Now it's best to start building a common folder
that you can put all of your slow motion
required assets inside. For me, I'm just using
my Gerald personal folder on my computer.
But inside that, if I wanted, I could put a Super SloMo folder
and then store everything in there.
But either way, make sure that the folder
and files that you name that use a short file name,
that you don't put any spaces in there,
because later on when we start referencing these things,
spaces can cause problems and you don't want it to be
so long that you're trying to figure out
what exactly you named that folder.
So, just keep it simple and no spaces.
So, go ahead and extract that Super SloMo zip
that you would have downloaded, the entire repository,
and extract it into a folder that you've aptly named.
Maybe just "SuperSloMo" with no spaces.
Now at this stage, you're supposed to train the model
which basically means you let it run through a ton of clips
so that it learns, you know, what slow motion looks like
and what kind of shots you take
and how to interpolate all that,
but that can take quite a long time.
And if you want to skip that step you can download
a pre-trained model that's already been fed
a bunch of Adobe 240p footage,
so it has a rough idea of how to do things.
And for all the tests that I'm doing here,
I'm using that same pre-trained model. So, I'll put a link
to that pre-trained model in the description.
Just download that and pop it in the same folder
that you just extracted the Super SloMo repository to,
put it right inside that folder.
Now, you're also gonna need FFmpeg
which is what will be handling the video stream.
I'll put a link to that in description.
Just download that and unzip it again into a simple folder.
Just call it ffmpeg. Not too long. No spaces.
Keep everything short and sweet and keep them all
in a similar parent directory.
Okay, so when you have all those things in place,
we're ready to launch Anaconda and get started.
So, we're gonna wanna use the Anaconda command prompt
by clicking on Anaconda, and then Anaconda prompt,
and here's where we're gonna put in our commands.
But we still need to install one more thing
which is the relationship between CUDA and TorchVision,
which is what's gonna be running this interpolation.
So, there's another link in the description that if you go to
it's for finding the correct installation string
that you're gonna need based on your computer.
So, I've got mine up here. The first thing, just choose
the stable build, and then choose your operating system,
whether it's Linux, Mac or Windows.
And then we're gonna choose Conda for the package.
And then the language that we're using
is probably gonna be Python 3.7
because if you downloaded Anaconda now
you probably used 3.7 link.
And then the CUDA version we want is 9.0.
And then we're gonna have this command here
and you can just copy this string
and then paste that into the Anaconda prompt here
and press enter and it will download and install
the necessary files and runtime environments
that you need in order to carry out
the actual interpolation that will follow this.
Now at this stage I recommend making
an input and output folder in that
slow motion directory that you're working on.
The input folder's where we're gonna put the files
that we want to convert into a high-speed frame rate
and the output folder is where we're going to designate
for the completed files to end up.
Now we're ready to process some clips.
So, the first thing you gotta do is get Anaconda to the directory
where your Super SloMotion folder is stored
and for that you're gonna Use the "cd" command.
So, type in cd, and then type in the folder directory
where all of your files that you extracted
from that repository are located.
And now you can execute your conversion command.
Now, I've included a sample command in the description below
but you're gonna have to take that
and change some of the variables
and I recommend doing this in a text editor first
and then copying the completed command
into Anaconda and running from there.
That we have to change it within the command prompt.
So, there's a few variables that you're gonna need to change.
The first one is you're gonna wanna put
the input folder, which, like we said,
we just created an input folder so it should be relatively easy.
So, you put that in there and then also put the file name
of the file that you want to convert.
Again, it's easy if you make these files small.
You don't want it to be some long string
that your camera will generate. So, say that it's a video
of an umbrella; just call it umbrella.mp4 in that folder,
and that'll be what converts.
So, we can just put it in input/umbrella.mp4.
then we move on to the next thing here,
we've got the SF argument. Now, this one is gonna be
the multiplier of how many frames--
how much the frame rate is being increased.
So, let's say that you started with 30 fps,
and you wanted to go up to 120 fps.
Well, that's four times more.
So, in this case, SF is gonna be "4".
If you use 24fps, and you wanted it to be 240fps,
then the SF would be 10 and so on.
The next thing is the directory for the location
of that pre-trained model that we downloaded
and stored in that folder.
So, you tell it where that pre-trained model is.
And then we have our output fps.
So, this one's obviously gonna correspond to the multiplier
that we just put in. So, like I was saying,
if you have 30 fps, you set an SF of 4, then the output
frames per second is gonna be 120 fps.
Just make sure that all the math works out there.
But this one now is the target frame rate
that you want your video to have when it's done.
And then you put the output file name and location.
Same thing: I would just keep everything the same
but now make it the output folder.
What I like to do is keep the filename the same but then
just add a little bit to let me know what the frame rate was.
So, in the case of that umbrella video,
it would now be umbrella120.mp4.
So, I know that's the converted one that's 120 fps
And then lastly we have the batch size.
Now this part is optional. You don't need to include it.
But if you have a graphics card with a lot of VRAM,
you can increase the batch size
in order to speed up the process.
This depends on what resolution you were working in.
So, for instance, with 4K, I can barely pull off
a batch size of 1, which is the smallest size.
But if I'm doing 1080p conversions,
I can do a batch size of 2, which is faster
than if I would've done a batch size of 1. OK,
so once you have that string with all the arguments complete,
copy it, paste it into Anaconda and press enter.
And once it's done you'll now have a video file
that has a much higher frame rate,
which means you can import it into your video editing software
and slow it down accordingly.
Now, I've run a bunch of these interpolation
processes already, so let's jump over to Premiere Pro
and I'll show you some of my results.
Okay, so let's start off by showing you
one that worked really well. This first clip
is of these lanterns here, and it's at 30p natively
and it should look something like this.
You can see the swinging back and forth.
It's pretty smooth for the most part
and we're just kinda lurching forward as we go.
I've got this clip over here, which was converted to 120p,
so that means we'll be able to slow it down to 25%.
And let's take a look how it performs.
So, for this one, I feel like it worked really, really well
and it's probably because everything was in a pattern
things were logical, the lines were congruent,
the lanterns were repeated
and I feel like the motion worked really well.
This is probably slower than you would need for this movement.
But I think that this is an example
of one that worked extremely well.
Now let me give you an example of one
that didn't really work so well.
So, here's a clip. It's only, like, 3 seconds long
of me just unzipping a bag.
If we take that one and we slow it down
and you see how my hand kind of turns into
this weird water, fluid-y kind of thing?
That's an example of it not working.
And my guess as to why this happens is because...
it's like one object over another object
and it moves kind of unpredictably.
There's not much of a flow or pattern to it.
My hand's just kind of interrupting the background.
It doesn't really work so well in that.
And I thought maybe it's because of 120p,
but it was trained on 240p.
So, I have a version here that we can slow down to 12.5%
This is a 240p conversion, but you can see that
it's the exact same problem, it's just slower now.
Now it does get the motion nice and smooth,
because you can see the way that the bag opens
in the middle, it does open with all the frames.
It is smooth, it just glitched right out.
And if you wanna have an idea what would happen if you chose
and use the footage-- so we'll use the original clip--
and slowed it down to 12.5% without using this technique.
So, this is what would normally happen.
You just don't have enough frames.
So, it only has 1/8 of the frames that's required.
So, there's a lot of stuttering action.
So, when we compare that even to the flawed one,
we can see that it is a lot more fluid
and it did a good job, especially with the tags,
of giving it a lot more fluid motion.
It just kind of got monkeyed up on my fingers.
And this happened in a couple other clips as well.
This one here is difficult for it because, again,
of the sporadic movement and the unpredicted nature.
You can see the sparks flying as it's doing the laser cutting.
And when we try to slow that one down--
now, you'll see that same sort of weird watery
kind of bubbling effect that happens.
It's kind of cool and trippy looking in this case,
but it's the same idea when you have
sort of sporadic unpredicted movement
it doesn't really work that well.
Alright, so now what about people?
So, this next example has a person
just walking down the road.
So, this one shot at 48p. It's 47.95-- whatever.
So, I quadrupled it, so it's 190 1.81.
It's 192fps. I thought that'd be an interesting test,
to see if it can handle, you know, bizarre frame rates.
So, we're [indistinct] slowed down to 25% speed,
and let's see how it did.
So, other than that one little glimpse
when it crosses in front of the sun
and the flare comes out in that kind of water effect,
most of the walking steps, especially if you look around
the shoulders and up-- and even the hair toss--
I thought were done extremely well! I was really impressed--
I was, like, "Oh, wow, that's pretty great."
And if you stare at it close enough, you can see
little bit of glitches in the dress,
but even still look at how well the dress
is performing considering the way that it's moving.
This was a complete success.
As soon as you get past that first bit of flare,
I thought it was great. I was really amazed by this result.
I've got a clip here at 30fps
and it's a person snowboarding.
And I'm curious to see what would happen with a person
and also what would happen to the snow and the trees
and everything-- It's a really action-intense dynamic shot,
but we're following a subject smoothly
just like we do with that person down the road.
And let's see what happens here.
So, right away you'll notice that the snow
is rippling like waves, which is hilarious.
It looks really cool and trippy,
but it's obviously not what you're going for.
but the person is being tracked really well
and their fluidity is fine, and I see even the snowboard
and a lot of the spray from the snow is working out.
Just the rest of the terrain is like-- [laughs]
wobbling like water, and it's crazy.
But again the trees did really well as well.
So, I think that's something to know the road--
if the path that the person is walking on is flat,
then it's OK. But if the path is moving like that,
it's got a lot of texture, it seems to have issues with it.
It seems to want to treat everything like water.
So, what happens if we give it some water
and this is where I find that this
interpolation is the most impressive.
And I also did one that was already in slow motion to see
what if you took a clip that was slow motion--
Can you slow it down even further?
And so here's a, you know, half-speed clip
that I downloaded from Storyblocks
of some water droplets hitting.
And it's already got that really flowy aspect to it.
The original clip here was at 29.97,
obviously slowed down from probably 60p already.
So, I took that clip and I bumped it back up
to 60p, so that we could slow it down another 50%.
So, let's take it and put it back down
to another 50% now, and we'll see how it does.
And as you can see in my opinion,
it's pretty much flawless.
This technique seems to just love water.
It treats everything like water apparently, even a snow hill.
But yeah, I don't see any issues at all.
And with the big splash that I think is coming down now
when you see the droplets come up, it's--
-[clicking] -it's pretty much perfect.
So, you can still apply it to footage
that was already slowed down. That's no problem.
And if you apply it to anything with water,
this would make for the best tech demo for it
doing water because you can't tell
when it does that weird ripple-y effect that it does over motion.
It just seems right because you're using it on water.
So, overall I'm really impressed by this.
Sure it didn't work with every clip that we put into it,
but considering that it's essentially free,
as long as you're willing to put the time in
and in some cases worked extremely well,
I'd say that this is a great little tool to have
for when you need it, As long as you have
a graphics card that's up to the task.
So, you want to turn your Clips into super slow motion?
It's really easy. All you have to do is install
a Python distribution. I recommend Anaconda.
And you'll need to download Super SloMo repository.
Start building a common folder-- slow-motion required assets
Super SloMo folder, to where everything in there-- no spaces.
And extract that Super SloMo zip train the model.
Also gonna need FFmpeg. Unzip it again.
Launch Anaconda and get started.
Use the Anaconda command prompt
CUDA and TorchVision interpolation.
Conda Python 3.7-- runtime environments--
input and output folder-- high-speed frame rate.
Now we're ready to process some clips.
Get Anaconda to the directory; type in cd.
Extract it from that repository. Now, you can execute
your conversion command-- change some of the variables.
umbrella.mp4-- multiplier 30fps.
120fps-- four times more,
and just make sure that all the math works out there.
Graphics card with a lot of VRAM--
copy it, paste it in Anaconda and press enter.
But that's gonna be it for me. I hope you found
this video helpful or at least entertaining.
And if you did, make sure you leave the ol' thumbs-up
and consider subscribing if you haven't already.
But if you did not find this video helpful or entertaining
feel free to hit the dislike button twice.
Alright... I'm done.