Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
If we are to write a sophisticated light simulation program, and we write a list of features that
we really wish to have, we should definitely keep an eye on defocus effects.
This is what it looks like, and in order to do that, our simulation program has to take
into consideration the geometry and thickness of the lenses within our virtual camera, and
even though it looks absolutely amazing, it is very costly to simulate that properly.
This particular technique attempts to do this in real time, and, for specialized display
types, typically ones that are found in head-mounted displays for Virtual Reality applications.
So here we go, due to popular request, a little VR in Two Minute Papers.
In virtual reality, defocus effects are especially important because they mimic how the human
visual system works.
Only a tiny region that we’re focusing on looks sharp, and everything else should be
blurry, but not any kind of blurry, it has to look physically plausible.
If we can pull this off just right, we’ll get a great and immersive VR experience.
The heart of this problem is looking at a 2D image and being able to estimate how far
away different objects are from the camera lens.
This is a task that is relatively easy for humans because we have an intuitive understanding
of depth and geometry, but of course, this is no easy task for a machine.
To accomplish this, here, a convolutional neural network is used, and our seasoned Fellow
Scholars know that this means that we need a ton of training data.
The input should be a bunch of images, and their corresponding depth maps for the neural
network to learn from.
The authors implemented this with a random scene generator, which creates a bunch of
these crazy scenes with a lot of occlusions and computes via simulation the appropriate
depth map for them.
On the right, you see these depth maps, or in other words, images that describe to the
computer how far away these objects are.
The incredible thing is that the neural network was able to learn the concept of occlusions,
and was able to create super high quality defocus effects.
Not only that, but this technique can also be reconfigured to fit different use cases:
if we are okay with spending up to 50 milliseconds to render an image, which is 20 frames per
second, we can get super high-quality images, or, if we only have a budget of 5 milliseconds
per image, which is 200 frames per second, we can do that and the quality of the outputs
degrades just a tiny bit.
While we are talking about image quality, let’s have a closer look at the paper, where
we see a ton of comparisons against previous works and of course, against the baseline
ground truth knowledge.
You see two metrics here: PSNR, which is the peak signal to noise ratio, and SSIM, the
structural similarity metric.
In this case, both are used to measure how close the output of these techniques is to
the ground truth footage.
Both are subject to maximization.
For instance, here you see that the second best technique has a peak signal to noise
ratio of around 40, and this new method scores 45.
Well, some may think that that’s just a 12-ish percent difference, right?
… No.
Note that PSNR works on a logarithmic scale, which means that even a tiny difference in
numbers translates to a huge difference in terms of visuals.
You can see in the closeups that the output of this new method is close to indistinguishable
from the ground truth.
A neural network that successfully learned the concept of occlusions and depth by looking
at random scenes.
Bravo.
As virtual reality applications are on the rise these days, this technique will be useful
to provide a more immersive experience for the users.
And to make sure that this method sees more widespread use, the authors also made the
source code and the training datasets available for everyone, free of charge, so make sure
to have a look at that and run your own experiments if you’re interested.
I'll be doing that in the meantime.
Thanks for watching and for your generous support, and I'll see you next time!