Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Super resolution is a research field with a ton of published papers every year, where
the simplest problem formulation is that we have a low-resolution, coarse image as an
input and we wish to enhance it to get a crisper, higher resolution image.
You know, the thing that can always be done immediately and perfectly in many of these
detective TV series.
And yes, sure, the whole idea of super resolution sounds a little like…science fiction.
How could I possibly get more content onto an image that’s not already there?
How would an algorithm know what a blurry text means if it’s unreadable?
It can’t just guess what somebody wrote there, can it?
Well, let’s see.
This paper provides an interesting take on this topic, because it rejects the idea of
having just one image as an input.
You see, in this day and age, we have powerful mobile processors in our phones, and when
we point our phone camera and take an image, it doesn’t just take one, but a series of
images.
Most people don’t know that some of these images are even taken as soon as we open our
camera app without even pushing the shoot button.
Working with a batch of images is also the basis of the iPhone’s beloved live photo
feature.
So as a result, this method builds on this raw burst input with multiple images, and
doesn’t need idealized conditions to work properly, which means that it can process
footage that we shoot with our shaky hands.
In fact, it forges an advantage out of this imperfection, because it can first, align
these photos, and then, we have not one image, but a bunch of images with slight changes
in viewpoint.
This means that we have more information that we can extract from these several images,
which can be stitched together into one, higher-quality output image.
Now that’s an amazing idea if I’ve ever seen one - it not only acknowledges the limitations
of real-world usage, but even takes advantage of it.
Brilliant.
You see throughout this video that the results look heavenly.
However, not every kind of motion is desirable.
If we have a more complex motion, such as the one you see here as we move away from
the scene, this can lead to unwanted artifacts in the reconstruction.
Luckily, the method is able to detect these cases by building a robustness mask that highlights
which are the regions that will likely lead to these unwanted artifacts.
Whatever is deemed to be low-quality information in this mask is ultimately rejected, leading
to high-quality outputs even in the presence of weird motions.
And now, hold on to your papers, because this method does not use neural networks or any
learning techniques, and is orders of magnitude faster than those while providing higher-quality
images.
As a result, the entirety of the process takes only a 100 milliseconds to process a really
detailed, 12 megapixel image, which means that it can do it 10 times every second.
These are interactive framerates, and it seems that doing this in real time is going to be
possible within the near future.
Huge congratulations to Bart and his team at Google for outmuscling the neural networks.
Luckily, higher-quality ground truth data can also be easily produced for this project,
creating a nice baseline to compare the results to.
Here you see that this new method is much closer to this ground truth than previous
techniques.
As an additional corollary of this solution, the more of these jerky frames we can collect,
the better it can reconstruct images in poor lighting conditions, which is typically one
of the more desirable features in today’s smartphones.
In fact, get this, this is the method behind Google’s magical Night Sight and Super-Res
Zoom features that you can access by using their Pixel 3 flagship phones.
When this feature first came out, I remember that phone reviewers and everyone unaware
of the rate of progress in computer graphics research were absolutely floored by the results
and could hardly believe their eyes when they first tried it.
And I don’t blame them, this is a truly incredible piece of work.
Make sure to have a look at the paper that contains a ton of comparisons against other
methods, and it also shows the relation between the number of collected burst frames and the
output quality we can expect as a result, and more!
Thanks for watching and for your generous support, and I'll see you next time!