Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
In this series, we often discuss a class of techniques by the name image inpainting.
Image inpainting methods are capable of filling in missing details from a mostly intact image.
You see the legendary PatchMatch algorithm at work here, which is more than 10 years
old, and it is a good old computer graphics method with no machine learning in sight,
and after so much time, 10 years is an eternity in research years, it still punches way above
its weight.
However, with the ascendancy of neural network-based learning methods, I am often wondering whether
it would be possible to take a more difficult problem, for instance, inpainting not just
still images, but movies as well.
For instance, let’s take and old old black and white movie that suffers from missing
data, flickering, blurriness, and interestingly, even the contrast of the footage has changed
as it faded over time.
Well, hold on to your papers, because this learning-based approach fixes all of these,
and even more!
Step number one is restoration, which takes care of all of these artifacts and contrast
issues.
You can not only see how much better the restored version is, but it is also reported what the
technique did exactly.
However, it does more.
What more could we possibly ask for?
Well, colorization!
What it does is that it looks at only 6 colorized reference images that we have to provide,
and uses this as art direction and propagate it to the remainder of the frames.
And it does an absolutely amazing work at that.
It even tells us which reference image it is looking at when colorizing some of these
frames, so if something does not come out favorably, we know which image to recolor.
The architecture of the neural network that is used for all this also has to follow the
requirements appropriately.
For instance, beyond the standard spatial convolution layers, it also makes ample use
of these blue temporal convolution layers, which helps “smearing out” the colorization
information from one reference image to multiple frames.
However, in research, a technique is rarely the very first at doing something, and sure
enough, this is not the first technique that does this kind of restoration and colorization.
So how does it compare to previously published methods?
Well, quite favorably.
With previous methods, in some cases, the colorization just appears and disappears over
time, while it is much more stable here.
Also, fewer artifacts make it to the final footage, and since cleaning these up is one
of the main objectives of these methods, that’s also great news.
If we look at some quantitative results, or in other words, numbers that describe the
difference, you see here that we get a 3-4 decibels cleaner image, which is outstanding.
Note that the decibel scale is not linear, but a logarithmic scale, therefore if you
read 28 instead of 24, it does not mean that it’s just approximately 15% better.
It is a much, much more pronounced difference than that.
I think these results are approaching a state where they are becoming close to good enough
so that we can revive some of these old masterpiece movies and give them a much-deserved facelift.
What a time to be alive!
This episode has been supported by Weights & Biases.
Weights & Biases provides tools to track your experiments in your deep learning projects.
It can save you a ton of time and money in these projects and is being used by OpenAI,
Toyota Research, Stanford and Berkeley.
They also wrote a guide on the fundamentals of neural networks where they explain in simple
terms how to train a neural network properly, what are the most common errors you can make,
and how to fix them.
It is really great, you got to have a look.
So make sure to visit them through wandb.com/papers or just click the link in the video description
and you can get a free demo today.
Our thanks to Weights & Biases for helping us make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time!