Nvidia has just announced RTX Video Super Resolution, which is ‘4K AI Upscaling’,
and it’s shown running on a hypothetical youtube video.
This one teaser video for it contains several ‘comparisons’.
Some of these are useless but I think I get the idea.
So: what is it going to be like?
To summarise: don’t get your hopes up.
I don’t expect this to be the DLSS of video upscaling.
Because DLSS 2, and FSR 2, are powerful game upscalers which really do make things look
a higher quality and resolution than they actually are.
Instead of those, this super-resolution thingy looks to be more like FSR 1, or Nvidia Image
Scaling, which are less powerful upscalers, but they exist because they’re a lot more
compatible and can upscale virtually any game.
I would expect this video ‘upscaling’ to be nothing more than an edge-sharpener,
and rather than adding detail to surfaces, I’d expect them to receive a more contrasted,
slightly artificial and plastic-y look to them.
But that’s if it’s being used to upscale videos which already look kind of okay.
But there’s another kind of use-case that I think would be more interesting, and that’s
if it was designed to improve low quality videos, like on twitch, or with youtube’s
lower quality settings.
And improving this kind of content would require a different kind of upscaling.
You might think a video is just a video, and that upscaling is just upscaling but NO, it
isn’t!
So before we can reach conclusions about this upscaling thingy, we need to know what kind
of video it’s intended for.
Time for a bit of detective work.
They show the original video’s settings that they upscaled from, but this only raises
more questions than answers.
8 mbps is quite high.
That’s a MB a second!
Almost double what twitch recommends for this kind of 1080p 30 fps video.
So maybe this kind of quality isn’t representative of what we can expect from upscaled streams.
But THIS is the other curious bit- they’re using H264.
Now don’t get me wrong- this is still the most ‘popular’ video standard -according
to piratebay at least- but most youtube content is using newer and better standards, like
vp9 and av1.
So it surprises me that, despite all this, and despite Nvidia’s own recent push for
AV1 for its various improvements, has still opted to use h264 as the basis of this upscaling
showcase.
I mean, good on them- a lower quality video will challenge the upscaler more than the
newer standards would have done.
But I don’t see the logic to using it here.
But here’s the other thing!
Just cos we know it’s 8 mbps, it still doesn’t tell us how good that video looked.
You can use a graphics card, or a processor, to make a video file of the same size.
In recent years the gap in quality has been closed, but I would still trust a processor
to do a better result.
It just takes longer than a graphics card does.
So, assuming they’ve made the original 1080p video look as good as it can for 8 mbps, I
set out to do the same.
I remade it by entering the upscaled video into handbrake, and by getting it to make
it into a 1080p, h264, 8 mbps video, using the best quality settings on my processor.
Obviously, youtube is going to hide how good this looks, so take it from me- it looks really
good!
So good, in fact, that I’d argue that of all the 1080p videos out there, it’s among
the least in need of upscaling.
So it remains a mystery to me what Video Super Resolution’s meant to be for.
If it’s best at upscaling already-decent looking videos then it’ll be nothing more
than a bit of a gimmick to keep your graphics card busy when you’re browsing dank memes.
HOWEVER, were it designed to enhance BAD quality videos and streams, then it could have real
value- especially for people on bad internet connections or with limited bandwidth to get
a more acceptable video quality while using lower settings.
Nvidia’s blog says that it will remove blocky compression artifacts and to upscale the video
resolution, which is a bit vague about how much of each it will do… but it sounds hopeful.
Even if I doubt this showcase video contained much blocking in the first place.
HERE's what a low bitrate, blocky video stream could look like.
And here it is, after I’ve used the Topaz video AI thingy to de-block and to upscale
it.
The result isn’t perfect- or even very good.
But then, the original was terrible so there’s that.
So I will be fascinated to see what Nvidia’s thingy can do with something like this, and
I expect it will operate in a similar sort of way.
But I’m not expecting it to be a miracle tool that would actually be able to transform
low quality videos into 4K masterpieces.
But we’ll find out for sure once it’s released next month.
And I’d LOVE to be wrong for once.
Firstly because it would be a refreshing change, and also because I’d benefit from GOOD video
upscaling as much as anybody would.
Video upscaling is NOT the same challenge as game upscaling is.
I would say that game upscaling is easier, because upscalers like FSR 2 or DLSS 2 can
get a lot more information from a live game to help with the upscaling process; they can
get access to the depth buffers and to the raw scene they’re attempting to improve
upon.
And all these behind-the-scenes tricks greatly aid with the upscaling process, which is why
stuff like DLSS 2 and FSR 2 look so good.
Essentially extra detail for free.
Compared with that, a video is already a ‘finished form’.
That’s it!
There’s no secret, hidden information the upscaler can extract information from.
The video you see is all the visual detail the upscaler gets to work with, and it’s
already had its quality degraded to some degree- in the case of streams and low bitrate youtube
videos, rather substantially.
Which means the upscaling ends up upscaling all the video’s blemishes and imperfections
as well.
It gets to a point where, if a video’s bad enough, it would be better just to try and
hide the imperfections, than to actually try and upscale the video itself.
So because of that, I don’t think RTX Video Super Resolution is intended to add REAL detail
to a video- which would be a sort of holy grail for video upscaling.
Instead, its focus will likely be on removing that smooshy look of bad video quality, which
it will do by de-jpegging the image and by enhancing the sharpness of the edges, and
the ‘clarity’ of surfaces.
So Nvidia have released one video containing a number of comparisons before and after upscaling.
I wouldn’t trust the comparison they give at the beginning, because the blurry 1080p
image they’re comparing it to is NOT how 1080p looks.
Not even close!
Clearly, it’s had some sort of artificial blur applied to it.
And looking at these stairs, there’s NO way that upscaling would be able to decipher
these as being individual stairs, from what’s essentially a blob with 2 lines across it.
So, for whatever reason, Nvidia has decided to artificially nerf the before image, which
makes the comparison redundant, because I don’t believe for one second that their
upscaled result was based on this ‘before’, blurry-looking input.
So why have they done this?
Other than to deceive people about the upscaling’s effectiveness, the only other reason I can
see for them doing this is just to get the point of the tech across more clearly to people
who don’t know much about it, and who don’t want to do pixel-peeping to understand what’s
changed.
Essentially reducing this comparison down to a simple: look, it makes blurry videos
look sharper again!
It's just unfortunate that by doing a fake comparison, it sucks for anybody who DOES
know a bit about upscaling and who wants to understand more about what Nvidia’s implementation
is actually doing, because this fake comparison masks the information that we need.
So while I don’t trust the ‘before’ image, I do think that we can trust the ‘after’
image as representing how a REAL 1080p video COULD look after being upscaled.
You can still see hints to its lower resolution origins in the distant trees, which lack all
the holes and speckling that I’d expect from a higher-quality, 4K video.
And the details on the rocks over here look a bit plastic-y and artificial, which again
is to be expected from trying to upscale and to sharpen textures from a lower resolution.
As I said at the beginning, the source file was probably already very good in quality,
and going by my own experience with this kind of upscaling, it looks like whatever video
Nvidia upscaled from -ignoring all that fake blur, of course- was super high in quality,
as it seems to have far more small details than I’d expect for a low-bitrate stream
to have.
I think that Nvidia must be showcasing a best-case scenario for the technology here, upscaling
a video that was almost perfect in the first place.
So I really am interested to see how the upscaling holds up when trying to upscale that twitch
video somebody accidentally output at a block-o-vision-ing-ly low bitrate.
This comparison confirms what I expected from the
upscaling- it isn’t adding ‘new’ details to the image; it’s simply hiding the blobby
look of low-resolution video, and then it’s sharpening the edges of stuff.
To the point where it may as well be a higher resolution around the edges of things.
So distant structures should look sharper, as should blades of grass like these ones
here.
But I don’t expect surfaces like THESE to receive the same degree of improvement, because
improving the quality of textures requires more than just sharpening the edges of stuff!
Because edges almost always benefit from sharpening.
But details might not.
Notice, for instance, how this example that they use lacks any sort of intricate texture-work.
I would say that’s deliberate, as the best we can hope for on the surfaces of scenery
will be some kind of sharpening filter to give the impression that there’s more definition
than there really is.
It’s why the rocks over here look a bit plasticy and overly-smoothed, post-upscale.
Nvidia’s Freestyle tool, which has been around for a few years now, has a filter you
can turn on which gives the impression of there being more detail on surfaces, but it’s
really just upping the contrast to bring out the texture more clearly.
In moderation, I kind of like the look of this, but turn it up too much and it looks
like everything’s been blasted with sand.
And I hate sand.
I wouldn’t be surprised to see it also being used in this video upscaling tech.
But they’ve got to be careful!
For while it could improve the look of already-detailed surfaces, if applied to a pixelated, disgusting
smoosh of a surface it could risk making it look even smooshier.
And the third example in Nvidia’s video shows the game in motion, and it returns the
before image to that artificially blurred looking one again.
The only thing of value to this is if we view the ‘after’ image and look for telltale
signs of shimmering or warping, which can happen if a video has been overly upscaled
or sharpened.
You’d especially notice it on stuff like this hedge here.
But…
I can’t see any, which again suggests it isn’t attempting SUPER amounts of upscaling,
it’s merely adding a bit of extra sharpening and contrast.
Which I think is the right way to go about it!
Here’s a video I made earlier.
Do you notice anything about it?
I’ll tell you: it’s only 720p!
Yet it still looks a lot clearer than Nvidia’s 1080p example, doesn’t it!
Anyway, I used Topaz Video AI to upscale it to 4K just to see how it would do.
And I think it managed a similar result to the Nvidia thingy.
It sharpens the edges of stuff, removes pixellation, but doesn’t try to add TOO much detail to
the image, because the more it deviates from the source, the more shimmering there will
be when a new frame is displayed with different details.
So this is something to look out for with videos being upscaled- and like I said, the
more it attempts to upscale it, the more obvious this kind of artifacting will become.
However, I don’t think this upscaling example of mine is the kind of video the Nvidia upscaler
is designed to upscale.
Because this one I’m using is way too good to begin with.
So here’s ANOTHER version of the video, this time using a very low bitrate to simulate
the blocky, blobby look you’d find on youtube or on twitch, at lower quality settings.
It’s a pretty horrible viewing experience with lots of distracting blob and blur everywhere.
THIS is the kind of video where a ‘good’ upscaler could be a game-changer.
Not to make it look good, but to stop it from looking bad.
Here’s what Topaz manages with it.
It’s not… great, it’s not exactly high-res.
But we’ve entered a half-way house of sorts, where we’ve swapped out the blocky, blobby
look for a much sharper, more artificial one instead.
I don’t know how Nvidia’s upscaler will fare in this sort of instance, but I wouldn’t
be surprised if it displayed the same sort of artificial and sharpened look to the image.
Because there’s only so much you can do to improve bad video quality!
The question is: which do you prefer, before upscaling, or after?
What is the holy grail of video upscaling?
I suppose it will reach a point where an AI will be able to identify all of the individual
things that make up a video clip, which will allow it then to be able to upscale all their
materials and shapes more convincingly.
We already have very powerful face upscaling- you’ve just got to try and imagine that,
but for everything in the scene, and not just on faces.
But before we reach that stage, I don’t hold out much hope for real-time video upscaling
getting much better.
But in videos where it can pre-upscale the thing before you watch it, I would expect
upscalers to decorate motion vectors with all the details present in a particular video
clip.
For instance: over the course of about 10 seconds, a video clip can collect a lot of
information about something that it’s filming- it just all isn’t shown at the same time.
But: if there’s a way to pick up details from earlier and later on in the clip, and
to apply to other points in the scene, it could essentially build up a more detailed
image than any one given moment in the video could.
Unfortunately, this gets harder to do, the lower the bitrate, and also because this kind
of motion vector trickery is kind of how videos work already…
so I’m not sure how much extra detail can be extracted from this process.
In short: I don’t expect miracles from video upscaling, and the result you get is very
much dependant on the quality of the input.
You have been warned!
Set your expectations low and… hopefully, you won’t be disappointed!