Image upscaling features like DLSS and FSR have gone from being flawed and often hated
features into being ones that I think most of us can agree are… good. And worth using.
The new framerate interpolation seen in DLSS 3 is still in the hated stage,
even by those who haven’t used it yet. But I HAVE used it. And I think it’s already got its uses,
but that it is also greatly flawed, but having spent so much time thinking about it and using it,
I realise it has opened up some really exciting future possibilities which I now think are
totally do-able, bar a few teething issues. You see, a vision has been forming in my mind,
of what the future of game rendering could look like. It would combine the best bits
of upscaling with some features from VR. So the general idea behind this future
vision I have, is to have 2 different framerates: a slower one for the gameplay,
which may be as low as 30 or 60 fps… and for the OTHER framerate to be your monitor’s refresh rate,
and this higher framerate will be the one that your mouse movements will be tracked at. This
idea was borne out of my hatred for input lag, which DLSS 3 comes with as standard.
And I don’t think this delay is ever going to go away, not without the quality of the framerate
interpolation greatly suffering. So I say: KEEP this delay, keep the good quality framerate
interpolation visual quality, but make it so that I can no longer notice the lag by separating
my mouse inputs from the game’s framerate. This is what we all need, isn’t it. Immediate
mouse inputs, no matter how low or stutter-y the game’s actual framerate might be. I don’t think
I’d even notice if the game came with a bit of delay, as DLSS 3 currently does, so long as my
MOUSE still feels immediate and responsive. And I’ve seen this framerate disconnect with the mouse
cursor before, but never with the gameplay itself. I google this then I see results dating back
years, suggesting I’m nowhere near as original with this idea as I thought I was! But also,
OBVIOUSLY this is the next step to take, because with DLSS 3 hinting at the promise of doubling
current framerates, it would be REALLY NICE if that didn’t have to come with such a laggy
downside. And I believe that separating mouse movements from the rest of it will do this.
This is similar to how VR games work already! VR doesn’t work well if there are judders and
framerate stutters. When your vision is directly strapped to a computer screen,
it turns out that you require constantly updated visuals to keep you from feeling motion sick. So
what it can do is to update your vision even if there isn’t a new frame to display, just in case
your PC isn’t fast enough to generate a new one in time or whatever. What it does instead is to
simply update the direction that you’re looking, even if this generated frame may no longer fully
line up with the image that’s currently being rendered. Because it turns out that doing that
is still better than doing nothing. …so I ask: why not do the same with
mouse-controlled games? IMPOSSIBLE! I hear you cry. No, this isn’t, either- cos look:
here’s a prototype made by Comrade Stinger that’s already doing this! And it feels AMAZING. Because
it turns out it’s not really a delay in the visuals that I notice and hate so much- it’s
a delay to my inputs. And by doing away with that delay, it masks any delay with framerate
conversion enough so much so that I no longer care, nor or even notice. I still don’t think
framerate interpolation will ever be a good thing for competitive twitch-based gameplay, simply out
of principle, but I do think separating your mouse movements from the game’s framerate could get it
to the point where framerate interpolation is good enough for any other type of genre.
Using Comrade Stinger’s prototype game, I’m going to set its framerate really low, right down to 15,
just to prove a point and to make it easy to see for this video’s sake. So while the game is still
being updated at 15 fps, my mouse’s movements are disconnected from all this, and are still being
updated with the responsiveness of my monitor’s 144 frames a second. And to me, this is enough
to do away with that laggy, unresponsive feeling that normally comes with 15 fps, even if the game
itself is still only running at that! So this is a HUGE, OBVIOUS benefit to my experience,
but it does come with 2 additional downsides: the first is that movement through the game’s world
will still feel sluggish at 15 frames a second. And the second downside is that, whenever I move
my mouse before a new frame is displayed, I’ll see black in the direction I’m turning… because
nothing’s currently being rendered there. At least, not until the next ‘real’ frame is created.
So these are definitely problems, but possible solutions to both these things exist already.
So let’s start by trying to address that blackness that we see around the edges of
the screen when we move the view to somewhere that the frame isn’t currently rendering.
Comrade Stinger’s game already has something in place to remedy this,
called ‘Stretch Timewarp Borders’ which pretty much stretches the image out beyond what’s drawn.
This does a great job of making the effect less noticeable. And sure, at super low framerates,
too much movement will still greatly mess this up. But it could be improved further still by
borrowing from VR gaming, which provides a BETTER solution: known as foveated rendering! Your eye
can see lots of detail in the middle, but not so much around the outside. VR gaming will end
up tracking your eye’s movements to allow it to render where you’re looking at high resolution,
and the bit around it at a lower one. In a way that you won’t notice. So, in a similar vein…
why not render the area BEYOND your screen’s borders, just at a super low resolution,
simply to provide enough information should your mouse move to reveal these regions?
This won’t come for free: it will require stuff to be rendered that’s not currently
visible on-screen. But provided the framerate impact of doing this isn’t too great, it again,
would be beneficial as it would essentially mask any issue with the revealed imagery that’s
currently being blacked out around the edges of the frames. And I feel like this low resolution
rendering is something that features like FSR 2 and DLSS 2 are already great at handling.
With the edges of the screen now fixed, we need to look at the generally low framerate
that we see elsewhere, especially when running through the world. IF ONLY there
was some sort of magical technology that could convert low framerates into high ones,
without slowing our computers down too much!
HMM!
Yes, I’m of course referring to DLSS 3 here, which is designed for this exact use-case! And while it
adds a bit of delay to what we see, now that our mouse movements are no longer tied to this lag,
I don’t care so much. Obviously, Comrade Stinger’s game doesn’t feature DLSS 3. So
here’s some footage of Cyberpunk, which does. The top has DLSS 3 on, the bottom doesn’t. Everything
else in the game is the same, so… see if you can spot any flaws with the newly generated frames,
which I’ll say right now is very difficult to do. It really does a good job of concealing
any shortcomings in its image quality! Right now, DLSS3’s framerate interpolation can only
generate 1 frame between every real one, so 30 frames a second could become 60, 15 frames a
second could become 30 and so on. That’s still a massive improvement, but I do imagine over time,
as hardware and software advances, it will get better and will be able to boost the framerate
more. I wouldn’t see the point in this if DLSS 3’s generated frames were distractingly buggy,
but since they’re not, I’d be interested to see how the technology holds up when it’s generating
more than one frame between every real one. Comrade Stinger’s demo features his own form
of… framerate smoothing. It works by taking a sort-of 3D screenshot of the world from your
current perspective, and then by skewing that about until the next real frame is posted. This
effect quickly breaks, but that’s okay cos it only has to make do for a couple of frames.
And it goes a HUGE way towards making 30 and even 15 frames a second feel much smoother.
The difference it makes is most noticeable if I toggle it on, and then off.
On, off.
On. Nice one, Comrade Stinger. So it is nice to know that my dream
of disconnecting mouse movements from the game’s framerate is possible, and Comrade
Stinger’s little demo has done an impressive job of masking its shortcomings without even
requiring powerful image upscaling or DLSS 3-style framerate conversion. At the end of the day, in
an ideal situation you wouldn’t need any of these tricks. But until the hardware reaches that stage,
they WILL enable less powerful hardware to perform acceptably at lower than normal framerates,
and to deliver better-looking visuals than we are perhaps used to seeing from that
level of hardware. Hopefully. I mean… what do I know?! I didn’t even make this demo!
But that is at least the direction I see today’s technologies heading- be it DLSS 4,
FSR 3.5 or whatever it ends up being called. And that to me is VERY exciting indeed.