DLSS 3 is announced! And I admit, it took me a bit by surprise. The idea of creating new frames from
existing ones isn’t a new concept- it’s something that programs like RIFE can do pretty well,
and which SVP can already do in real-time with videos played on your PC. A few years ago,
I used this sort of technology to double the framerate of youtube’s oldest video.
And Door Stuck. Check it out for examples of how framerate interpolation looks in practice.
‘I BEG YOU!!!’
I suppose the two surprising things for me were: why hasn’t this been done for games already,
and why has it been featured under the ‘DLSS’ name? And while I don’t have any knowledge about
DLSS 3 that you don’t also have, I’ve got some thoughts on it that I’d love to share with you.
Firstly: DLSS 2’s techniques are not going anywhere.
It still looks to be the method DLSS 3 uses to ‘upscale’ images to higher resolutions.
It’s just that DLSS 3 then gets that result, and combines it with the new, generated frames
to give the illusion of there being more frames per second than there actually are.
With every trick, there’s a trade-off, and with these new frames being generated,
I can think of several issues that Nvidia will have to address… or at least, to minimise.
The first is when it makes a mistake. When generating new frames, it inevitably has to
guess at what’s behind moving stuff, because it’s a new frame, from a unique perspective and so on.
You can see a bad example of framerate interpolation getting it wrong just here.
Very distracting. And it’s particularly difficult to resolve motion that’s behind
complicated features such as meshes or fences. But this problem has sort of been addressed by
DLSS 3 by ‘when’ this new frame is generated. They could have had these newly generated frames come
after the latest rendered frame, but this would have introduced more image artifacts, and horrible
distracting stutters when it doesn’t quite predict your mouse movements accurately enough- because
using the previous frames can only get you so far. It doesn’t look like Nvidia has gone down this
route with DLSS 3, though- instead, it gets the last 2 frames rendered, and ‘draws’ one between
them. This will greatly reduce image artifacts because it can call on detail that’s present
before AND after the frame is meant to take place- but all this comes at one, terrible price:
and that’s to the input lag, since the image you’re seeing on-screen is always delayed by
1 or 2 frames. Which could get quite distracting at lower framerates. I think they know this, which
is why they also mention Nvidia REFLEX, which is there to minimise frame delay… but in this case,
it will simply lessen the ‘lag’ that DLSS 3’s generated frames introduce. In other words,
it doesn’t make the game MORE responsive, but limits how much LESS responsive it will feel.
Say there aren’t any visual errors with the framerate interpolation.
It can still make the motion look artificial, if boosted too much. For instance, if you have a 10
fps video and convert it to 60… it isn’t really 60. All of the actions look too ‘smooth’ and
‘angled’ because it’s trying to generate 5 quite different-looking frames between every real one,
and it assumes the same amount of motion between each of those. So unfortunately,
the best way to use framerate boosting is when you already have a reasonably high framerate.
I still notice these sorts of image problems when converting 30 to 60, but I think starting at 60
provides enough frames in the first place, all similar-looking enough, for the artificial look
of newly generated frames between each of these to effectively have their limitations hidden.
Now onto my second point: if it’s not ‘upscaling’, then why is it being put under the ‘DLSS’ umbrella
and not being marketed as its own separate feature? And I can actually think of reasons
for why it should be DLSS, beyond the usual marketing reasons. Right now,
DLSS has different presets ranging from ultra performance to ultra quality. And it could be
that DLSS 3 could help to extend this out further in both directions. If I’m honest,
I can see its usage being most important with low framerates, either to help low-powered systems
to hit 60 FPS instead of 30, or to allow for ultra-high quality raytracing or resolutions
on powerful systems… but at a more consistent framerate. These lower framerates are, of course,
where input lag will be most noticeable, but I can imagine it would be suitable for
slower-paced experiences, like for beautiful adventure games or for turn-based strategy.
They showcase it in Flight Simulator which I think would be a perfect thing to use it in.
But I imagine that most people will use it to boost a 60 FPS game to 120 FPS, just to feel
good about themselves, and this technology will of course confuse normal benchmarking
methods in some way. Reviewers will HATE it! I’ll have to test it before I know for sure,
but since it’s only a superficial change to the framerate and doesn’t come with the improved
responsiveness you’d expect from 120 fps- in fact, it actually lessens responsiveness-
I’m not sure how much I’d use it for this reason. I know already that highly competitive players
will want to disable the frame generation bit of DLSS 3 IF it adds any sort of delay to the image,
but they will still probably want to use the upscaling bit of the technology because
that does genuinely boost framerates. And obviously, VR users will absolutely love
DLSS 3’s ability to boost image quality and framerate. Maybe it could be configured to
only generate a fake frame when it detects a ‘stutter’ in the original framerate? Nah.
So yes, DLSS 3 is exciting, and I think shows the route Nvidia is taking the technology down.
It no longer refers simply to upscaling the resolution, but instead to the whole thing.
We’re quickly distancing ourselves from the conventional way of rendering stuff.
A pixel is no longer a pixel, a frame is no longer a frame.
It’s becoming increasingly abstract and …botched, and while with DLSS 3
I can still understand how it works by drawing parallels between it and existing techniques,
I feel with future iterations it’ll leave all that behind and venture into some scary new territory.
Beyond gaming, Nvidia are doing all sorts of amazing experiments with AI,
many of which already look like magic to me! I expect these to gradually be integrated into
DLSS as and when they’re ready to do so. They may eventually generate images from abstract
‘hints’ and ‘sketches’, allowing them to totally change the style of existing content to look
however they want it to look. I don’t know when it will get there, but eventually, every pixel
and every frame will be filtered and transformed by AI first before it even reaches your screen.