Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Let’s talk about video super resolution.
The problem statement is simple, in goes a coarse video, the technique analyzes it, guesses
what’s missing, and out comes a detailed video.
However, of course, reliably solving this problem is anything but simple.
When learning-based algorithms were not nearly as good as they are today, this problem was
mainly handled by handcrafted techniques, but they had their limits - after all, if
we don’t see something too well, how could we tell what’s there?
And this is where new learning-based methods, especially this one, come into play.
This is a hard enough problem for even a still image, yet this technique is able to do it
really well even for videos.
Let’s have a look.
The eye color for this character is blurry, but we see that it likely has a green-ish,
blue-ish color.
And if we gave this problem to a human, this human would know that we are talking about
the eye of another human, and we know roughly what this should look like in reality.
A human would also know that this must a bridge, and finish the picture.
What about computers?
The key is that if we have a learning algorithm that looks at the coarse and fine version
of the same video, it will hopefully learn what it takes to create a detailed video when
given a poor one, which is exactly what happened here.
As you see, we can give it very little information, and it was able to add a stunning amount of
detail to it.
Now of course, super resolution is a highly studied field these days, therefore it is
a requirement for a good paper to compare to quite a few previous works.
Let’s see how it stacks up against those!
Here, we are given given a blocky image of this garment, and this is the reference image
that was coarsened to create this input.
The reference was carefully hidden from the algorithms, and only we have it.
Previous works could add some details, but the results were nowhere near as good as the
reference.
So what about the new method?
My goodness, it is very close to the real deal!
Previous methods also had trouble resolving the details of this region, where the new
method, again, very close to reality.
It is truly amazing how much this technique understands the world around us from just
this training set of low and high-resolution videos.
Now, if you have a closer look at the author list, you see that Nils Thuerey is also there.
He is a fluid and smoke person, so I thought there had to be an angle here for smoke simulations.
And, yup, there we go.
To even have a fighting chance of understanding the importance of this sequence, let’s go
back to one of Nils’s earlier works, which is one the best papers ever written, Wavelet
Turbulence.
That’s a paper from twelve years ago.
Now, some of the more seasoned Fellow Scholars among you know that I bring this paper up
every chance I get, but especially now that it connects to this work we’re looking at.
You see, Wavelet Turbulence was an algorithm that could take a coarse smoke simulation
after it has been created, and added fine details to it.
In fact, so many fine details that creating the equivalently high resolution simulation
would have been near impossible at the time.
However, it did not work with images, it required knowledge about the inner workings of the
simulator.
For instance, it would need to know about velocities and pressures at different points
in this simulation.
Now, this new method can do something very similar, and all it does is just look at the
image itself, and improves it, without even looking into the simulation data!
Even though the flaws in the output are quite clear, the fact that it can add fine details
to a rapidly moving smoke plume is still an incredible feat!
If you look at the comparison against CycleGAN, a technique from just 3 years ago, this is
just a few more papers down the line, and you see that this has improved significantly.
And the new one is also more careful with temporal coherence, or in other words, there
is no flickering arising from solving the adjacent frames in the video differently.
Very good.
And if we look a few more papers further down the line, we may just get a learning-based
algorithm that does so well at this task, that we would be able to rewatch any old footage
in super high quality.
What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!