Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Let’s talk about video super resolution.
The problem statement is simple, in goes a coarse video, the technique analyzes it, guesses
what’s missing, and out comes a detailed video.
You know, the CSI thing.
Enhance!
However, of course, reliably solving this problem is anything but simple.
This previous method is called TecoGAN, and it is able to give us results that are often
very close to reality.
It is truly amazing how much this technique understands the world around us from just
this training set of low and high-resolution videos.
However, as amazing as super resolution is, it is not the most reliable way of delivering
high-quality images in many real-time applications, for instance, video games.
Note that in these applications, we typically have more data at our disposal, but in return,
the requirements are also higher: we need high-quality images at least 60 times per
second, temporal coherence is a necessity, or in other words, no jarring jumps and flickering
is permitted.
And hence, one of the techniques often used in these cases is called supersampling.
At the risk of simplifying the term, supersampling means that we split every pixel into multiple
pixels to compute a more detailed image, and then, display that to the user.
Does this work?
Yes it does, it works wonderfully, but it requires a lot more memory and computation,
therefore it is generally quite expensive.
So our question today is, is it possible to use these amazing learning-based algorithms
to do it a little smarter?
Let’s have a look at some results from a recent paper that uses a neural network to
perform supersampling at a more reasonable computational cost.
Now, in goes the low-resolution input, and, my goodness, like magic, out comes this wonderful,
much more detailed result.
And here is the reference, which is the true, higher-resolution image.
Of course, the closer the neural supersampling is to this, the better.
And as you see, this is indeed really close, and much better than the pixelated inputs.
Let’s do one more.
Wow.
This is so close I feel that we are just a couple papers away from the result being indistinguishable
from the real reference image.
Now, we noted that this method has access to more information than the previously showcased
super resolution method.
It looks at not just one frame, but a few previous frames as well, can use an estimation
of the motion of each pixel over time, and also gets depth information.
These can be typically produced inexpensively with any major game engine.
So, how much of this data is required to train this neural network?
Hold on to your papers, because only 80 videos were used, and the training took approximately
1.5 days on one TITAN V, which is an expensive, but commercially available graphics card.
And no matter, because this step only has to be done once, and depending on how high
the resolution of the output should be, with the faster version of the technique, the upsampling
step takes from 8 to 18 milliseconds, so this runs easily in real time!
Now, of course, this is not the only modern supersampling method, this topic is subject
to a great deal of research, so let’s see how it compares to others.
Here you see the results with TAAU, the temporal upsampling technique used in Unreal Engine,
an industry-standard game engine.
And, look!
This neural supersampler is significantly better at antialiasing, or in other words,
smoothing these jagged edges, and not only that, but it also resolves many more of the
intricate details of the image.
Temporal coherence has also improved a great deal, as you see, the video output is much
smoother for the new method.
The paper also contains a ton more comparisons against recent methods, so make sure to have
a look!
Like many of us, I would love to see a comparison against NVIDIA’s DLSS solution, but I haven’t
been able to find a published paper on the later versions of this method.
I remain very excited about seeing that too.
And for now, with techniques like this, the future of video game visuals and other real-time
graphics applications is looking as exciting as it’s ever been.
What
a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!