Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
This work is about performing useful image restoration tasks with a convolutional neural
network with an additional twist.
Its main use cases are as follows: One, in the case of JPEG artifact removal,
the input is this image with many blocky artifacts that materialized during compression, and
the output is a restored version of this image.
Two, image inpainting where some regions of the input image are missing and are to be
filled with useful and hopefully plausible information.
Three, super resolution where the input image is intact, but is very coarse and has low
resolution, and the output should be a more detailed, higher resolution version of the
same image.
This is the classic enhance scenario from these CSI TV series.
It is typically hard to do because there is a stupendously large number of possible high
resolution image solutions that we could come up with as an output.
Four, image denoising is also a possibility.
The standard way of doing these is that we train such a network on a large database of
images so that they can learn the concept of many object classes, such as humans, animals
and more, and also the typical features and motifs that are used to construct such images.
These networks have some sort of understanding of these images and hence, can perform these
operations better than most handcrafted algorithms.
So let's have a look at some comparisons.
Do you see these bold lettered labels that classify these algorithms as trained or untrained?
The bicubic interpolation is a classical untrained algorithm that almost naively tries to guess
the pixel colors by averaging its neighbors.
This is clearly untrained because it does not take a database of images to learn on.
Understandably, the fact that the results are lackluster is to show that non-learning-based
algorithms are not great at this.
The SRResNet is a state of the art learning-based technique for super resolution that was trained
on a large database of input images.
It is clearly doing way better than bicubic interpolation.
And look, here we have this Deep Prior algorithm that performs comparably well, but is labeled
to be untrained.
So what is going on here?
And here comes the twist - this Convolutional Neural Network is actually untrained.
This means that the neuron weights are randomly initialized, which generally leads to completely
useless results on most problems.
So no aspect of this network works through the data it has learned on, all the required
information is contained within the structure of the network itself.
We all know that the structure of these neural networks matter a great deal, but in this
case, it is shown that it is at least as important as the training data itself.
A very interesting and esoteric idea indeed, please make sure to have a look at the paper
for details as there are many details to be understood to get a more complete view of
this conclusion.
In the comparisons, beyond the images, researchers often publish this PSNR number you see for
each image.
This is the Peak Signal To Noise ratio, which means how close the output image is to the
ground truth, and this number is, of course, always subject to maximization.
Remarkably, this untrained network performs well both on images with natural patterns
and man-made objects.
Reconstruction from a pair of flash and no-flash photography images is also a possibility and
the algorithm does not contain the light leaks produced by a highly competitive handcrafted
algorithm, a joint bilateral filter.
Quite remarkable indeed.
The supplementary materials and the project website contain a ton of comparisons against
competing techniques, so make sure to have a look at that if you would like to know more.
The source code of this project is available under the permissive Apache 2.0 license.
If you have enjoyed this episode and you feel that 8 of these videos a month is worth a
dollar, please consider supporting us on Patreon.
One dollar is almost nothing, but it keeps the papers coming.
Recently, we have also added the possibility of one-time payments through Paypal and cryptocurrencies.
I was stunned to see how generous our crypto-loving Fellow Scholars are.
Since most of these crypto donations are anonymous and it is not possible to say thank you to
everyone individually, I would like to say a huge thanks to everyone who supports the
series.
And this applies to everyone regardless of contribution - just watching the series and
spreading the word is already a great deal of help for us.
Thanks for watching and for your generous support, and I'll see you next time!