DLSS has an image problem. And for that, Nvidia have only themselves to blame! It’s a shame
it had to be like this, because actually, the idea behind Nvidia’s DLSS technology
is a good one. To use AI to improve image quality!
The hope was that this would achieve the best of both worlds: the image quality of a higher
resolution, but with the performance impact of a lower one. Effectively resulting in higher
framerates for free! BUT, as you probably know already, in practice,
it didn’t work. In games such as Battlefield 5 and Metro Exodus, the result was arguably
worse than simply playing the game at a lower than native resolution. The effect was beaten
by a simple sharpening filter by AMD! Also, DLSS was horribly marketed. First of
all, associating it with ‘super-sampling’ instead of being humble and settling just
for ‘Anti-aliasing’. But also, for another reason. Back when it was first announced,
there were to be 2 different types: one like the one we got, which upscaled and smoothed
the image. But there was meant to be another that kept the original resolution, but then
used AI to give it the finest quality AA possible! We never got that one. I think that tells
us it didn’t work as planned. Instead, we only got the version of DLSS that rendered
at a lower resolution and then upscaled to, say, 4K. The problem was, this was misleading:
it would be called 4K DLSS, even though it wasn’t actually rendering at 4K! I think
Nvidia’s hope for this was that their cards at 4K with DLSS would be compared against
rival’s cards running native 4K, since this would make Nvidia’s cards appear much faster.
Just look at the difference DLSS on / off makes to the same cards in these benchmarks
from overclock3d.net! Obviously, this deception is all just speculation
on my part, but with 4K in the name and with just enough truth to warrant it, it does make
it difficult to explain why that’s not the same as full 4K to some people, and more importantly,
why the difference matters. Luckily for me, I never had to explain this too much. DLSS
was called out for blatantly lowering the resolution and it became widely known that
it was inferior to true 4K, or whatever the resolution it was trying to pass for. AMD
took advantage of this to launch their own superior sharpening filter and right now,
I think people see it as superior to Nvidia’s DLSS.
When… that’s not the full truth. But its early implementations were terrible. It needed
training for each game it featured in, had its resolutions restricted for certain graphics
cards, and was so flawed there was little to no reason to use it. AMD’s simple sharpening
tool on the other hand could run on any card, at any resolution, and with very little performance
impact. There was little to no reason NOT to use it!
As much as everybody likes to smugly dismiss Nvidia’s DLSS as a failure, with more than
a touch of schadenfreude, I’m sad that it didn’t work. Because even if I disagree
with its marketing and execution, I like what Nvidia were trying to achieve with AI upscaling.
And I do think there’s value in this approach! DLSS doesn’t need to look as good as the
native resolution to have value- it simply needs to look better than if it’s not used.
If 1440p with DLSS looks better than 1440p without it, then DLSS has value.
Intelligent upscaling can certainly be done. The PS4 Pro and Xbox One X are both there
to run this generation of console games at higher ‘resolutions’ than the base consoles
do. They do this partly by having more powerful graphics hardware, but a large part of it
is thanks to checkerboard rendering… where it only renders every other pixel, and then
tries to fill in the rest intelligently. For years I wished that PCs could have something
similar- and honestly, that’s kind of what DLSS is.
The more I’ve looked into it, the more I question the term ‘upscaling’. Simply
showing a low resolution on a high resolution monitor is ‘upscaling’- but that has no
added value. Checkerboard rendering and DLSS may operate in different ways, but I believe
they strive to achieve the same thing: that is, to maximise picture quality with the fewest
pixels possible. I guess with that definition, AMD’s sharpening tool falls under that category
as well! This kind of ‘smart upscaling’ is the
solution to a lot of the problems we’re only just starting to face. Raytracing quickly
becomes more demanding to run as resolutions increase, so upscaling could really help with
it! The way our eyes work means we only need high quality graphics at the centre of our
vision- like say, for VR. The bit around that can be run at a lower resolution and we wouldn’t
notice! Again, I feel DLSS could be of value in VR. And as screen resolutions increase,
the value of each individual pixel decreases. Again, AI upscaling becomes a more attractive
option. And I suspect the next generation of consoles, if they support raytracing, will
have to rely heavily on upscaling to produce an acceptable image at a good resolution.
One day we won’t have to compromise. But until then, if we want demanding effects at
high resolutions, tricks and shortcuts will be required. As well as upscaling, there’s
also Variable Rate Shading, which renders clumps of similar pixels as though they’re
just one. You can see it in this 3Dmark demonstration here- it just looks like it’s more pixelated
because it’s doing it to the whole screen, but if it’s used only in areas of similar
colour, we wouldn’t notice. And that’s how it’ll be used.
And while the first games with DLSS were poor, it’s showing signs of improvement. Control,
for example. Nvidia published a nice little blog post about it which I certainly recommend
you check out. They point out the problems with image scaling, acknowledge the issues
with DLSS, and then discussed ways in which it could be improved- some of which can be
seen in Control. Honestly, as a game I find it a bit dull.
Lots of running about, then occasional, rather limited shooting mechanics. And the protagonist
reacts to stuff WRONG! I don’t like her. But it’s not like I got the game for the
gameplay. And no, I didn’t get it for the raytracing or DLSS either. If you must know
I got it for the really pretty table destruction physics. Don’t judge me. It’s right up
there with Red Faction! And look at that- whatever’s in front of
the camera gets shot, which is why the light behind me reacts whenever it’s in the way
of my crosshair. I’m sure that’s a pretty major exploit right there.
I initially planned to spread my DLSS analysis across 2 separate videos, then decided it
would be neater to combine them into just 1, then as I realised I had a LOT of stuff
about Control to talk about- a lot of which you might not be interested in- I split it
back into 2 again. So if you want the details, check out the
other video that’s dedicated to Control. But to summarise: with DLSS disabled in this
game, the game still appears to use anti-aliasing, which I assume is temporal. So I wasn’t
able to compare DLSS to there being no anti-aliasing at all. But in a way I think this is more
interesting, since DLSS has typically been compared to temporal in the past as well.
And right now, with their implementations in Control, I don’t think it’s a case
of one being BETTER than the other. DLSS simply approaches the problem DIFFERENTLY. In some
ways it’s better, like how it’s less blurry and almost always anti-aliases surfaces while
temporal can sometimes miss them. But in other ways, it’s worse, like how it can cause
visible shimmering on some surfaces, or pixellation on others when certain graphical features
are enabled. And in other ways, it’s down to personal preference- it smooths things
in the foreground at the expense of stuff in the background, and it runs a frame or
two slower than having it disabled. The strengths and weaknesses both become less
noticeable if you use 1080p or 1440p as the base resolution and ultimately I think it
boils down more to personal preference which one is better. It’s a nice alternative to
Temporal, and at least now it tells you what resolution it’s actually rendering at, which
I consider to be the biggest improvement with the feature.
At first I thought DLSS was just an excuse to do something with Nvidia’s AI cores until
they found other uses for them. But wouldn’t it make more sense if it was a technique developed
by Nvidia to COMPLEMENT their raytracing effects? Rendering at a lower than native resolution
is awfully convenient, given how intensive raytracing gets as the pixel-count increases.
But more than that- DLSS’s major selling point is that it eliminates stray pixels and
averages the result over a large number of frames- both of which I think have the potential
to mask current problems with real-time raytracing when the ray-count is low! The result in Control
is mixed, and while there are situations where I prefer Temporal, I do see potential in DLSS’s
approach. And if Nvidia are pushing for raytraced effects in games, it makes sense that they’d
also want an anti-aliasing method to complement it nicely and to mask its problems. It makes
sense that it would be DLSS, a technique they have full control over on a per-game basis.
I’m still not sure why each game needs to be trained individually. Although each game
has its own colour-palette and style, since there are still infinite combinations of images
within a single game, you’d imagine that if it produces good results in one game then
that effect could be successfully carried across to others. But whatever. I’m sure
Nvidia knows more about it than I do. In their blog post they admit to current shortcomings
with their techniques. They say that the sharper an image, the more you can see shimmering
and noise- which is something I picked up on in Control, both with and without DLSS.
But they talk about how they discovered certain ‘bugs’ which they could actually use to
fill in details that would otherwise be lost. There are some comparison videos as well-
the first showing how DLSS in Control reduces upscaling shimmer when moving about. The second
shat all over DLSS by showing it hiding lots of small fire embers that should be clearly
visible at the native resolution. And the third was the most interesting, showing a
much-improved AI research model that didn’t have the weaknesses.
Although left a little ambiguous, I think they’re saying that although Control is
their ‘best’ iteration of DLSS yet, they’ve discovered ways of improving it further by
using deep learning to train the AI… to make better AI. It gets a little bit confusing.
But the last paragraph makes it clear that Nvidia believes they can improve it further,
both in terms of image quality and in framerate. I would not dismiss AI upscaling, nor Nvidia’s
DLSS implementation of it. Of course, call it out when it fails. Criticise its marketing.
And praise a simple sharpening filter if it’s better. But I still think that AI upscaling
has a part to play in our future, for good or evil. A bit like Gollum!
Features such as DLSS are not going to go away. Whether its name remains that, or whether
it’s replaced with ‘checkerboard rendering’ or ‘AI upscaling’ remains to be seen.
Its early implementations were underwhelming, but you can’t lose sight of what it hopes
to achieve, and the progress it’s making towards that.