Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
As humans, when looking at the world, our eyes and brain does not process the entirety
of the image we have in front of us, but plays an interesting trick on us. We can only see
fine details in a tiny, tiny foveated region that we are gazing at, while our peripheral
or indirect vision only sees a sparse, blurry version of the image, and the rest of the
information is filled in by our brain. This is a very efficient system, because our vision
system only has to process a tiny fraction of the visual data that is in front of us,
and it still enables us to interact with the world around us. So what if we would take
a learning algorithm that does something similar for digital videos? Imagine that we would
need to render a sparse video only every tenth pixel filled with information, and some kind
of neural network-based technique would be able to reconstruct the full image similarly
to what our brain does. Yes, but that is very little information to reconstruct an image
from. So, is it possible?
Well, hold on to your papers, because this new work can reconstruct a near-perfect image
by looking at less than 10% of the input pixels. So we have this as an input, and we get this.
Wow. What is happening here is called a neural reconstruction of foveated rendering data,
or you are welcome to refer to it as foveated reconstruction in short during your conversations
over dinner. The scrambled text part here is quite interesting, one might think that,
well, it could be better, however, given the fact that if you look at the appropriate place
in the sparse image, I not only cannot read the text, I am not even sure if I see anything
that indicates that there is text there at all!
So far, the example assumed that we are looking at a particular point in the middle of the
screen, and the ultimate question is, how does this deal with a real-life case where
the user is looking around? Let’s see! This is the input….and the reconstruction. Witchcraft.
Let’s have a look at some more results. Note that this method is developed for head-mounted
displays, where we have information on where the user is looking over time, and this can
make all the difference in terms of optimization.
You see a comparison here against a method labeled as “Multiresolution”, this is
from a paper by the name “Foveated 3D Graphics”, and you can see that the difference in the
quality of the reconstruction is truly remarkable.
Additionally, it has been trained on 350 thousand short natural video sequences, and the whole
thing runs in real time! Also, note that we often discuss image inpainting methods in
this series, for instance, what you see here is the legendary PatchMatch algorithm that
is one of these, and it is able to fill in missing parts of an image. However, in image
inpainting, most of the image is intact, with smaller regions that are missing. This is
even more difficult than image inpainting, because the vast majority of the image is
completely missing. The fact that we can now do this with learning-based methods is absolutely
incredible.
The first author of the paper is Anton Kaplanyan, who is a brilliant and very rigorous mathematician,
so of course, the results are evaluated in detail, both in terms of mathematics, and
with a user study. Make sure to have a look at the paper for more on that!
We got to know each other with Anton during the days when all we did was light transport
simulations, all day, every day, and were always speculating about potential projects,
and to my great sadness, somehow, unfortunately we never managed to work together for a full
project. Again, congratulations Anton! Stunning, beautiful work. What a time to be alive!
This episode has been supported by Linode. Linode is the world’s largest independent
cloud computing provider. They offer affordable GPU instances featuring the Quadro RTX 6000
which is tailor-made for AI, scientific computing and computer graphics projects. Exactly the
kind of works you see here in this series.
If you feel inspired by these works and you wish to run your experiments or deploy your
already existing works through a simple and reliable hosting service, make sure to join
over 800,000 other happy customers and choose Linode. To spin up your own GPU instance and
receive a $20 free credit, visit linode.com/papers or click the link in the description and use
the promo code “papers20” during signup. Give it a try today! Our thanks to Linode
for supporting the series and helping us make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time!