Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Have you heard of the Ken Burns effect?
If you have been watching this channel, you have probably seen examples where a still
image is shown, and a zooming and panning effect is added to it.
It looks something like this.
Familiar, right?
The fact that there is some motion is indeed pleasing for the eye, but something is missing.
Since we are doing this with 2D images, all the depth information is lost, so we are missing
out on the motion parallax effect that we would see in real life when moving around
a camera.
So, this is only 2D.
Can this be done in 3D?
Well, to find out, have a look at this.
Wow, I love it.
Much better, right?
Well, if we would try to perform something like this without this paper, we’ll be met
with bad news.
And that bad news is that we have to buy an RGBD camera.
This kind of camera endows the 2D image with depth information, which is specialized hardware
that is likely not available in our phones as of the making of this video.
Now, since depth estimation from these simple, monocular 2D images without depth data is
a research field of its own, the first step sounds simple enough: take one of those neural
networks then, ask it to try to guess the depth of each pixel.
Does this work?
Well, let’s have a look!
As we move our imaginary camera around, uh-oh.
This is not looking good.
Do you see what the problems are here?
Problem number one is the presence geometric distortions, you see it if you look here.
Problem number two is referred to as semantic distortion in the paper, or in other words,
we now have missing data.
Not only that, but this poor tiny human’s hand is also…ouch.
Let’s look at something else instead.
If we start zooming in into images, which is a hallmark of the Ken Burns effect, it
gets even worse.
Artifacts.
So how does this new paper address these issues?
After creating the first, coarse depth map, an additional step is taken to alleviate the
semantic distortion issue, and then, this depth information is upsampled to make sure
that we have enough fine details to perform the 3D Ken Burns effect.
Let’s do that!
Unfortunately, we are still nowhere near done yet.
Previously occluded parts of the background suddenly become visible, and, we have no information
about those.
So, how can we address that?
Do you remember image inpainting?
I hope so, but if not, no matter, I’ll quickly explain what it is.
Both learning-based, and traditional handcrafted algorithms exist to try to fill in this missing
information in images with sensible data by looking at its surroundings.
This is also not as trivial as it might seem first, for instance, just filling in sensible
data is not enough, because this time around, we are synthesizing videos, it has to be temporally
coherent, which means that there mustn’t be too much of a change from one frame to
another, or else we’ll get a flickering effect.
As a result, we finally have these results that are not only absolutely beautiful, but
the user study in the paper shows that they stack up against handcrafted results made
by real artists.
How cool is that!
It also opens up really cool workflows that would normally be very difficult, if not impossible
to perform.
For instance, here you see that we can freeze this lightning bolt in time, zoom around and
marvel at the entire landscape.
Love it.
Of course, limitations still apply.
If we have really thin objects, such as this flagpole, it might be missing entirely from
the depth map, or there are also cases where the image inpainter cannot fill in useful
information.
I cannot wait to see how this work evolves a couple papers down the line.
One more interesting tidbit.
If you have a look at the paper, make sure to open it in Adobe Reader you will likely
be very surprised to see that many of these things that you think are still images…are
actually animations.
Papers are not only getting more mind-blowing by the day, but also more informative, and
beautiful as well.
What a time to be alive!
This video has been supported by you on Patreon.
If you wish to support the series and also pick up cool perks in return, like early access
to these episodes, or getting your name immortalized in the video description, make sure to visit
us through Patreon.com/TwoMinutePapers.
Thanks for watching and for your generous support, and I'll see you next time!