October of 2022 sees the release of NVIDIA's new GeForce RTX 40 series of graphical processor
units or GPUs that are designed to reset the state of the art for 3D rendering in games
and high-performance computation.
Perhaps the biggest and most exciting element is the introduction of DLSS 3 - an AI-powered
rendering technique which promises to not only generate outstanding image quality at
4K resolution but achieve up to 4x improved graphical performance in a given game when
compared to traditional 3D rendering by not just increasing the resolution, but also the
games frame rate.
I'm Tommy Thompson, welcome to AI and Games, and in this video - with the help of the folks
at NVIDIA - I'm going to do a deep dive into explaining how DLSS 3 actually works, and
showcase how this new technology performs when running games on an NVIDIA RTX-4090 graphics
card.
The big innovation on DLSS 3 is that it doesn't just uprez the frames rendered from the game
to higher resolutions, it uses a brand new component in the RTX 40 series GPUs to generate
entire frames of the game by itself, without consulting the game engine.
It's actually increasing the framerate of the game, by effectively letting the AI render
new frames in between the game engine's output, and to great effect.
It's pretty crazy, to be honest with you, but after testing it quite extensively I'm
happy to report that it works fairly well.
Naturally, this is far from a simple and straightforward process, so strap in folks, we've gotta lotta
ground to cover!
For those new to the DLSS, or you're new to this channel, I've already made an entire
video exploring how the original version of this technique works in detail, and I recommend
you check that video out first if you have absolutely no idea what I'm talking about.
Right now I'm going to quickly summarise how the previous versions have worked, and then
get into the nitty gritty on this third iteration of the technology.
Since 2018 the revolutionary element of the NVIDIA GPU's is the adoption of AI techonology
in order to yield increased graphical performance.
The GeForce 20 series released in 2018 introduced the Turing GPU architecture - named after
the British mathematician Alan Turing.
These graphics cards were now comprised of general purpose compute or CUDA cores, RT
cores that enabled for realtime ray tracing, and lastly what are known as Tensor cores.
Tensor cores are derived from existing technology used for for training deep learning AI systems,
and are built to enable fast and effective matrix mathematics which is a common requirement
for both training and executing deep neural networks.
This enabled for the introduction of Deep Learning Super Sampling or DLSS - a superresolution
technique that runs natively on the graphics card.
In which the graphics card first generates a frame from the game engine at a lower resolution
than desired, and then using DLSS - powered by the tensor cores - it uprezzes that image
to be of higher quality, but does so using far less power and memory than the GPU would
have done if it did it in-game.
So in essence, you could get a given game to generate an image at 1080p, only for DLSS
to uprez it to 4K and it would be less expensive, than getting the game itself to render in
4K.
This process has continued to evolve since then, with the GeForce 30 series of GPUs running
an updated version of the same architecture called Ampere, named after the 19th-century
French mathematician André-Marie Ampère.
The main difference between versions 1 and 2 of DLSS, was that the original required
bespoke deep neural networks to be trained for each game to work at higher resolutions,
and when the drivers had to be updated frequently to provide the pre-trained deep neural nets
for each game.
However, with version 2, the process changed to rely on a generic superresolution network
pre-trained using NVIDIA's supercomputer technology.
The other big change in version 2 was the introduction of motion vectors: whereby the
system gathers information from the game engine on how pixels are moving from one frame to
the next, in order to make it generate smoother images for fast-moving elements.
This is an idea that actually comes from the world of video encoding, and will prove critical
for this big step into DLSS3.
So let's discuss how DLSS3 works in summary, and then I'll get into the details: DLSS 3
builds on the existing infrastructure of DLSS2, but it also adds in something entirely new:
a concept known as Optical Multi Frame Generation.
This is an idea that addresses a limitation of the earlier versions of DLSS, namely that
while upscaling worked nicely, the boosts in framerate were often quite limited.
DLSS 1 and 2 were good at allowing you to increase the resolution of your games without
pushing the graphics card to its absolute limit.
This meant you could potentially increase the rendering frame rate of your game as well,
given the GPU could handle that increased workload.
Since it's cheaper to get the graphics card to generate frames at a lower resolution and
upscale them with DLSS than try to render them outright in the higher resolution without
the AI feature enabled.
So for example, instead of trying to run the likes of DOOM Eternal natively at 4K with
ray tracing enabled, and struggling to hit 30 frames per second, you could instead render
it at 1080p, and with DLSS upscaling to 4K you could possibly push that framerate closer
to something that's more fun to play on without killing the GPU.
This is of course quite valuable for a lot of people given you're all buying nice 4K
monitors that can run at 120 or 144Hz, where you want to push the framerate the game renders
at to increasingly higher limits.
But of course, the need to render more frames is creating a separate pressure on the graphics
card to do a lot of work.
And in some games, it's difficult for existing GPUs to hit that limit, and that's where DLSS3
comes in.
What DLSS 3 is doing is really quite interesting: it still increases the resolution as before,
but now it can also increase the frame rate of a game by using deep learning AI to create
new frames of the game in between the ones it receives from the game engine.
The Optical Frame Generation I mentioned earlier, is looking at the frames being upscaled, and
in a process that combines information from the game engine, alongside its own analysis
of the upscaled frames, it effectively guesses what the intermediate frame of the game should
look like and presents that to you on screen.
This is kinda bonkers, as as you can see already from the footage in this video, we're sometimes
seeing significant increases in frame rate, without any loss of visual fidelity.
And that's in part due to how all of this expands on the existing DLSS 2 technology.
Now let's dig a little deeper in order to explain how DLSS3 works and critically how
it differs from the previous version.
DLSS3 is essentially three different technologies NVIDIA put together:
- The first is the DLSS Super Resolution, essentially what we've already seen from DLSS
2.
- The second is Optical Frame Generation that is injecting new frames.
- And lastly, there is NVIDIA Reflex, a programming API that is designed to reduce latency.
DLSS3 is possible thanks to the GPU receiving a revamp, with the new 3rd generation architecture
having a significant increase in Tensor cores.
This architecture - named after Ada Lovelace, the woman cited as being the world's first
programmer - comes with a significant increase to AI processing power.
The RTX-40 series of graphics cards carries the new 4th generation tensor cores, with
not only an increase of 512 cores compared to only 336 of the 3rd gen versions in the
RTX-3090 Ti, but these new cores achieve up to 5x the throughput of their predecessors.
But perhaps more importantly, as we'll see in a second, the RTX-4090 now has 305 of the
Optical Flow Accelerator or OFA cores in the RTX 4090, compared to 126 in the RTX-3090
Ti.
This is a huge boost to the GPUs processing power not just because of the increased number,
but also - much like the tensor cores - the OFA cores received a revamp for the 400-series.
As we'll see in a second, the OFA is critical to enabling frame generation.
So as stated, DLSS3 essentially builds on the existing implementation, so let's dig
a little deeper into how DLSS2 originally worked.
Previously with DLSS 2, the process was relatively straightforward: the game engine would render
out the current scene of the game to an image at say 1080p resolution, and it would then
attempt to upscale that to 4K using the super-resolution deep neural net loaded into the tensor cores
on the GPU.
To do this, it requires the motion vectors that it receives from the game engine.
As mentioned already motion vectors actually derive from video encoding, and it's designed
to make it easier to render videos in a cheap and effective way.
A motion vector essentially tells the renderer that for a given piece of pixel data in the
image, where it expects that pixel to move on the next frame.
To produce motion vectors for the likes of video playback, you run a motion estimation
algorithm, which is analysing the direction and magnitude of motion for a given pixel,
relative to the previous frames of rendering.
This can lead to significant gains in video compression, given it's more cost-efficient
to render a new frame by shifting the existing one around, rather than drawing it from scratch.
But there is an inherent problem with this, and it comes down to how the motion vectors
are generated.
In the video, we run all of this through an encoder, which takes a little while to do,
but afterwards, the results are typically fantastic.
But for games, it needs to be much more responsive.
So as mentioned, in this instance the game engine helps provide the GPU with the motion
vectors for the super-resolution.
And this is where the problem lies, because while the game engine knows a lot of critical
information for the motion vector - like where the player is and in-world geometry - it doesn't
always know have 100% accurate information needed for every pixel, given they might be
influenced by a particular process that the game engine doesn't cache at that level of
fidelity internally.
So for example, when running realtime dynamic lighting, or having smoke or particle effects,
these are reliant on computations that will run in the next frame.
A shadow is the result of the position of nearby lighting in the game world, and if
we're moving around our character, they may be hit by multiple light sources at once,
and the game engine might not anticipate that.
And that's where DLSS3 takes this in a new direction.
As mentioned already, DLSS3 starts with the basics from DLSS2, and that it adds on the
Optical Frame Generation.
This is how it not only boost frames, but ultimately improves the current DLSS output,
given NVIDIA have essentially trained an AI to run the motion estimation algorithm in
real-time on the GPU.
To do this, it looks at the last two frames it's uprezzed into 4K from the game engine
and passes them to the Optical Flow Accelerator, which runs on the OFA cores.
The Optical Flow Accelerator looks at the frames, and generates what is called an Optical
Flow Field.
This is essentially a motion vector, but it's catching all of the information that in-game
motion vector can't calculate, because it's looking at the actual rendered frames from
the game itself and observing how things change.
It's worth mentioning that while the RTX-30 series cards had some of the OFA cores on
the chip, the 40-series is now running twice as powerful, and that enables for this to
run in realtime as it is here.
With this approach, the RTX-40 cards are able to determine in realtime how things like shadows,
reflections and particles have changed between frames.
And this is how the frame generation is made possible.
After running the Optical Flow Accelerator, the GPU runs a deep neural network that is
responsible for Multi Frame Generation network in the tensor cores.
It's now being passed in three pieces of information: - The uprezzed 4K frames generated by the
super resolution network from DLSS2.
- The motion vectors from the game engine.
- And the optical flow field developed by the Optical Flow Accelerator.
From this, it now has information on what the the last two game engine frames look like:
how the game engine expects objects to move around, and how the optical flow field imagines
more dynamic elements will change as well.
It's essentially got information on not just what those two frames look like, but how the
pixels shifted in between them, and from that, it can then generate a new frame of the game
to render as an intermediate frame.
And this is ultimately how DLSS3 increases frame rates: for every two frames rendered
in the game engine (both of which are subsequently upscaled before the user sees them), the GPU
is generating a third frame, which is what it has calculated to be what the game state
would look like in between.
What this means, is that using DLSS 3, the bulk of the rendering of a given game is actually
being pulled away from the game engine, with the AI on the CPU taking on more and more
responsibility.
If you consider the super-resolution, which is upscaling from 1080p to 4K, it is essentially
drawing 3/4 of that frame with the AI, with the original 1/4 handled by the rendering
from the engine.
You then combine that with the Frame Generation, and if you double a game's framerate from
say 60 to 120fps, then the AI is generating every 2nd frame by itself.
This means that ultimately the AI is now generating 7/8th of the pixel data across two frames.
This approach could look to continued gains for consumers as neural graphical rendering
continues to improve much faster than that seen in traditional 3D rendering.
One part I briefly mentioned earlier, that is worth coming back to is the new NVIDIA
Reflex API.
As it was explained to me by the folks at NVIDIA, Reflex is a tool that is critical
to the function of DLSS3, in that it's designed to minimise latency in the game itself.
Reflex is an additional workload for developers to integrate, but it isn't exactly new.
Reflex was released alongside the RTX 30 series GPUs back in 2020, and is already integrated
in a lot of online competitive shooters such as Fortnite, Apex Legends and Call of Duty:
Warzone.
The idea is to minimise the latency of the game, meaning that the time taken for a player
to press a button or move a mouse of stick, to then seeing some sort of response in the
game is reduced as much as possible.
While some latency is as the result of thee peripheral itself, or the TV or monitor you're
using, a good chunk of it is in the processing of the game state, and the rendering of said
state.
While a higher FPS can reduce latency, you're still at the mercy of intense GPU workloads,
since the game is generating a large amount of frames that are then pushed into the render
queue.
Reflex is designed to better synchronise the CPU processing of the game input, with the
GPU rendering the output by minimising the render queue and keeping the time from state
processing to rendering as little as possible.
Given that DLSS3 is now also taking on more GPU workloads, it makes sense that Reflex
is critical to the process, given you want to ensure that the GPU output and the CPU
workload are being synchronised as best as possible.
Particularly given the Optical Frame Generation means that the GPU is rendering a frame without
consulting the game state, so trying to capture player input as quickly as possible seems
like a fairly critical element to the process.
NVIDIA's 40-series GPUs are continuing to expand into the realm of high-fidelity graphics
both by pushing further in to the realms of 4K resolutions and real-time ray tracing,
but in order to keep up with the pressures of modern games, we're seeing AI being used
to help lighten the load.
Within the space of a few years, the DLSS technology in the GeForce GPUs has achieved
something that was still a fantasy even 5 years ago.
Naturally my focus in this video has been on explaining DLSS 3 to you all, but the footage
largely speaks for itself.
All of the games I tested on that were DLSS 3 enabled saw significant gains in frame rate,
and often looked just as good if not better than the native 4K versions.
It's a significant upgrade, and I'm keen to see how this evolves in the coming years,
and at the very least future episodes should be looking a lot crisper.
The new RTX 40 series launches on October 12th, and while I only had a chance to try
out a handful of the upcoming DLSS3 integrations such as Microsoft Flight Simulator, F1 22,
A Plague Tale: Requiem and Cyberpunk 2077, there are over 35 titles announced that will
be compatible with DLSS 3.
Ranging from already released games like Hitman 3, The Witcher 3, Dying Light 2 and Spider-Man
Remastered, to upcoming releases such as STALKER 2, Lord of the Rings: Gollum and of course
the re-release of Portal with RTX enabled.
Which y'know, any excuse to go in and give Portal another go huh?
Thanks for watching this video on DLSS3, and a special thanks to NVIDIA for providing access
to the demos recorded for this episode, not to mention of course a chance to try out the
RTX 4090 to boot!
We're seeing a lot of work in this space nowadays as AI continues to tackle areas outside of
traditional non-player character design and development, and you can be sure I'll have
something to say on it in the future.
In the meantime, go do the likes, subscribes and I'll catch you all in the next one!