Hi, I’m Philip, and this is me getting ready for DLSS 3.
Let me explain.
DLSS 3 doesn’t upscale the image any further than DLSS 2 does- instead, it sort of ‘upscales’
the framerate by creating new frames between existing ones.
So if you were getting 30 fps, it could potentially transform that into 60.
Sounds too good to be true.
BUT IS IT?!
So my major worry was that it would introduce latency, since it needs to wait until the
next frame has been rendered before it generates the frame for you’d see BEFORE that one.
Digital Foundry got early access to the techology and revealed some comparison figures, and
sure enough, it’s more delayed than were the framerate interpolation disabled, but
it’s probably not as bad as you might have expected; the delay is measured in the hundredths
of a second, rather than tenths.
At best, it’s not much slower than DLSS 2 is, and at worst, it’s more comparable
to gaming at native resolution without DLSS on at all.
But at like, double the framerate.
So just to rephrase that: the worst of the 3 examples shown puts DLSS 3 as being the
same latency as native, but at double the framerate.
Competitive gamers will want to disable it out of principle and just to use the delay-reducing
‘reflex’ portion of the technology, but for the rest of us who may be using DLSS to
improve image quality, a little bit of extra latency might be worth it if we get up to
double the framerate.
But this video investigates the OTHER trade-off: and that’s the artifacts it might bring
to the image.
When drawing NEW frames, the computer has to make approximations and guesses because
it’s trying to draw things that are occurring, chronologically, half way between existing
frames- and it will inevitably get it wrong from time to time, because it’s sometimes
trying to draw something that can’t be seen.
I created 5 clips of varying difficulty to test image interpolation on.
The first is a flight simulation, which I think should be fairly easy for image interpolation
to handle.
The next 2 are me walking across the garden at different speeds, to see how well it can
separate me from the grass behind.
And then I have a fast-paced clip of CS:GO as a demanding example.
And then, finally, a NIGHTMARE example where I run through some grass, which I expect it
to really struggle with cos there’s so much moving that I don’t expect even the BEST
interpolation software to do a good job with this one.
I don’t have access to Nvidia’s interpolation tech -and I’m not sure it can be used on
videos yet any way- so I instead used an existing, high-quality AI-based interpolation program
called RIFE to try and simulate what sort of glitches we might see, and when.
Nvidia’s methods will likely be different, but still similar enough that you’ll know
roughly what to expect from what you see in this video.
We’ll start with a ridiculously low 5 FPS version of those videos I showed you earlier-
this is just to serve as an ultra-extreme example of image interpolation, far more challenging
than you’ll likely encounter in practice.
Just to get you used to what these artifacts will look like.
Now, I’ll use RIFE to try and return it to 60 again.
The AI is actually trying to draw 11 different frames between each existing one, and because
that’s such a massive task, the resulting issues from this are obvious.
So much of the image is disappearing off the side every frame, it doesn’t know what to
fill in the gap left behind with, so makes the whole area morph and shift about, really
distractingly; sometimes even moving in the wrong direction entirely!
Motion vectors will help to minimise this by knowing which direction every bit of the
screen is moving and by how much, which is something that Nvidia’s solution may have
better access to than RIFE does in this example, since it’s analysing a pre-existing video.
Or in this case, slide show.
The next example is me walking across the screen with lots of wavy bits everywhere to
see how well the AI could distinguish between me and the background.
And despite being just 5 FPS, it did a great job of NOT warping the grass around me!
Instead, it made me almost invisible, but… maybe that’s the best outcome we could hope
for here.
Next, I walked back across the screen at about half the speed- and you can see it does a
much better job when there’s effectively half the motion between every frame.
I don’t go invisible anywhere near so much, and it’s an excellent example of how image
interpolation attempts to smear stuff around to mimic motion.
In this instance, with limited success.
Next is some high-speed footage of me trying to look cool playing CS:GO.
A very tough situation for the AI to try and interpolate and to make look cool- especially
from just 5 FPS- you can see that in some cases, it doesn’t even try because the individual
frames are so different that it thinks it’s a different scene entirely.
But this is a GOOD thing- because you don’t want it trying to smoothly transition between
completely different images.
Take this shot, for instance- the camera cuts from the girl with the wooden acting to Fred
Williamson.
It’s MEANT to look like this; an abrupt transition.
It's NOT meant to look like this!
Oh my God.
So yeah, you don’t want this same effect happening in games when you’re going from
gameplay to a cutscene, or from the game to a menu.
Luckily, this should be fairly easy to remedy in-game, thanks to all the extra information
that DLSS 3 will have access to, to inform it when it’s a brand new scene.
As expected, at just 5 FPS, RIFE had no idea what to do with the final example.
It’s okay with the background imagery, which isn’t changing too much with every frame.
But the foreground is a distorted, blobby mess.
But as the starting framerate gets higher, framerate interpolation will get better for
several reasons, which I will attempt to show in this diagram here.
I say attempt, because it’s a bad diagram.
I know it is by the amount of resistance that Excel put up as I was trying to bend it to
my will!
But humour me here, and imagine that what you’re seeing here, is a second of 60 FPS
gameplay, running from left to right.
Each of these white bars represents a single frame on your screen.
And there’s 60 of them!
…Or are there?
Yes there are.
60 of them in a second equals 60 FPS.
They’re white, meaning they’re REAL frames, and not AI-generated ones.
Until you enable DLSS 3 for the first time, every frame of gameplay you’ve ever played,
is currently like this.
THIS, is what 5 frames a second, looks like, when interpolated up to 60.
There are 2 problems here: firstly, a lot of that second is spent showing AI-generated
frames, which is a LONG time for visual bugs to be present on-screen for.
And second, because the real frames are chronologically so far apart, it means they all look very
different from the previous one, meaning that AI has a really, really hard job trying to
generate middle-frames without there being, like, big, noticeable problems with them.
So how can we improve things?
Well, if we increase the real frames per second up to 15, it’ll help in two ways: firstly
because there are now more frequent real frames, so the differences between each of them are
much smaller.
And believe me, this goes a LONG WAY towards helping AI to generate convincing mid-frames
between the two closest real frames.
And it also helps, because now more of the second is spent showing real frames, and therefore,
less of the second is spent showing AI-generated frames where there might be problems.
In short: 15 fps interpolated will look EVEN MORE better than you might expect.
What’s youtubing done to me
And the flight example is much better.
The edges of the screen are still being sucked away, but the effect is far less noticeable
because there’s now less movement between every frame, and elsewhere in the image, I
think it holds the motion of the buildings moving past each other pretty well!
The illusion only really falls apart as we get close to the towers, where it fails to
correctly shift the textures and lines that are on them.
But we’re getting there.
Getting onto the first grass example, I no longer go invisible between every real frame.
There’s a sort-of judder to the movement.
Makes it look kind of like a cartoon!
And it’s struggling with the flailing snowman still.
I suppose it’s not really fair to compare it to 60 FPS- we should instead be comparing
it to 15 FPS without interpolation, and which one of these we think looks better.
It certainly looks smoother.
You can tell that something iffy’s going on with the framerate but it’s kind of mesmerising
to watch, isn’t it.
I’m sort of disappointed that the Counter-Strike example is still a mess.
You can see what’s going on, but the textures all warp and shimmer very distractingly.
This is admittedly a difficult example, with lots of close-up, fast-paced camera motion
and repetitive textures where mistakes will be obvious, but it’s still a far cry from
being acceptable.
And this last example is also a mess- though perhaps not in ways that I was expecting.
No matter when you pause it, the grass looks like grass.
But in motion, its warping is very distracting.
With 15 FPS out of the way, we get onto more relevant examples.
Because something magical happens as we approach and surpass 24 FPS; we enter the realm of
movies and TV shows, which we’re all used to watching at these framerates for years
already, and which I’ve spent years interpolating to 60.
Because of this, I am well versed in what to expect from framerate conversion from 24
fps.
I expect a shimmer around fast-moving objects, but for the rest of the scene to look fine.
And panning shots look SO much better.
There is a lot of controversy over whether higher framerates look better in movies- I
think it’s obvious that more is better, but traditions die hard.
Let’s try converting 30 FPS to 60.
And this is the first relevant example to DLSS 3 because, from the benchmarks shown,
it doesn’t look like it’s creating more than 1 AI frame between each real one.
Even though that would be feasible.
But being 30 real frames a second, interpolated TO 60, I think these upcoming examples will
represent the minimum sort of framerates that DLSS 3 should be used with.
At this point, the flight example looks 99% perfect.
If you look in the bottom corners, you’ll still spot tiny amounts of warping, but at
this framerate it’s updated so quickly to the next real frame that it’s easy to miss.
And it’s the same affair as we pass between the skyscrapers.
If we slow this down then the problems become more obvious, but at 100% speed…
they are still very hard to spot.
Both of my walking examples look perfect, much to my dismay cos I spent an awful lot
of money getting a camera that could capture proper 60 FPS.
Turns out I could have just interpolated it from 30 instead!
Okay, there is something a little bit fishy going on with my hair motion, and with the
flailing tail of the snowman.
What’s going on with it though is only really noticeable if I slow the footage down- but
had this just been a clip from a random video, I don’t think anybody would have looked
at this and noticed these things- let alone to think that they’re errors from framerate
interpolation!
The difficult CS:GO examples still don’t look right.
It’s not so easy to ‘say’ what’s going wrong, though- you can just see it.
Like the textures in the level are vibrating about in your peripheral vision.
I’d say the result is passable, but definitely not ideal.
Iin fast-paced shooting games you’ll want 60 FPS in the first place.
And it’s the same story with the final example, where it’s still distracting enough that
most gamers will take objection to the effect.
In conclusion: framerate conversion from 30 to 60 is probably fine for slower-paced titles,
at a push maybe even RACING ones, since the artifacts will typically remain around the
edges of your screen.
But for first-person shooters, you’ll still be wanting more.
Which is why I will now attempt to convert these Counter-Strike examples from 60 to 120
FPS.
Problem is, Youtube is limited to 60 fps, so it’s kind of hard to show you how this
looks.
I could combine real and fake frames together into one, like I’m doing here… or, I could
slow it down to half speed, like I’m doing now.
Neither really shows how it looks in practice, though.
Maybe if you double youtube’s playback speed it’ll do it.
But in case you’re unable to see 120 FPS for whatever reason, I’ll just reveal my
findings- and it’s here that I got the biggest surprise of all: I am a big believer in framerate
conversion.
I am consistently impressed with the results of converting 30 to 60 and similar.
But I WASN’T impressed with this 60 to 120 fps.
I thought it would continue to hone in towards perfection, but if anything, it simply made
the mistakes it made, more distracting, even if I could only ‘feel’ them, rather than
to properly see what was going wrong.
In conclusion: obviously, I will reserve judgement on DLSS 3 until I try it, but RIFE’s framerate
interpolation disappointed me with this final 120 fps example.
Because… let’s look at what you want from higher framerates.
In games like CS:GO, you’d want 120 FPS rather than 60 because it FEELS more responsive
than 60 does.
But DLSS 3’s framerate boost doesn’t make it feel more responsive in the same way- it
will only make on-screen motion LOOK more fluid.
And that’s only worth it, in my opinion, if it doesn’t come with noticeable artifacts.
Of course, a few things to note here: I’m testing RIFE and not DLSS 3, so chances are
it could be different, or better designed to look less distracting at super-high framerates.
And also: I’ve only been converting video files.
I still haven’t felt how framerate conversion of gameplay feels in practice.
I can see many fringe-case uses for DLSS 3- like in games where they CAN’T run faster,
through no fault of the GPU.
Like in CPU-limited games like Flight Simulator.
While the GPU’s just sat there waiting for the CPU to pump out the next frame, the GPU
might as well generate more frames while it’s waiting.
Jonathan pointed out that it could help bypass games with a fixed framerate cap in place.
Imagine older games, capped at 60 FPS.
Or Skyrim, where running 120 FPS for real, results in physics bugs and stuff.
These games could still benefit from framerate conversion, where the alternative would be
to be stuck at a fixed, lower framerate instead.
Of course, this would require the games support DLSS in the first place, unless DLSS 3 can
work its magic without having direct access to a game’s depth buffers and motion vectors.
DLSS 3 could also help when it comes to raytracing, especially in applications like Blender, where
maybe you’d rather have half the framerate, but double the number of rays fired every
frame.
Like, in situations where you don’t mind a few framerate conversion artifacts, if it
means that you get better looking raytracing from the first frame where it’s displayed.
I suspect this was one of the main reasons DLSS 3 was designed for in the first place.
So yeah, it’s definitely an interesting topic to keep an eye on.
And I suppose you’ve got to remember that DLSS 3 running at 120 FPS doesn’t NEED to
look as good as native 120 FPS does- it just has to be better than whatever framerate it
would be running at, were framerate conversion disabled.
‘That’s just to satisfy your curiosity.’