Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Approximately two years ago, we covered a work where a learning-based algorithm was
able to read the wifi signals in a room to not only locate a person in a building, but
even estimate their pose.
An additional property of this method was that, as you see here, it does not look at
images, but radio signals, which also traverse in the dark, and therefore, this pose estimation
also works well in poor lighting conditions.
Today’s paper offers a learning-based method for a different, more practical data completion
problem, and it mainly works on image sequences.
You see, we can give this one a short image sequence with obstructions, for instance,
the fence here.
And it is able to find and remove this obstruction, and not only that, but it can also show us
what is exactly behind the fence!
How is that even possible?
Well, note that we mentioned that the input is not an image, but an image sequence, a
short video if you will.
This contains the scene from different viewpoints, and is one of the typical cases where if we
would give all this data to a human, this human would take a long-long time, but would
be able to reconstruct what is exactly behind the fence, because this data was visible from
other viewpoints.
But of course, clearly, this approach would be prohibitively slow and expensive.
The cool thing here is that this learning-based method is capable of doing this, automatically!
But it does not stop there.
I was really surprised to find out that it even works for video outputs as well, so if
you did not have a clear sight of that tiger in the zoo, do not despair!
Just use this method, and there you go.
When looking at the results of techniques like this, I always try to only look at the
output, and try to guess where the fence was obstructing it.
With many simpler, image inpainting techniques, this is easy to tell if you look for it, but
here, I can’t see a trace.
Can you?
Let me know in the comments.
Admittedly, the resolution of this video is not very high, but the results look very reassuring.
It can also perform reflection removal, and some of the input images are highly contaminated
by these reflected objects.
Let’s have a look at some results!
You can see here how the technique decomposes the input into two images, one with the reflection,
and one without.
The results are clearly not perfect, but they are easily good enough to make my brain focus
on the real background without being distracted by the reflections.
This was not the case with the input at all.
Bravo!
This use case can also be extended for videos, and I wonder how much temporal coherence I
can expect in the output.
In other words, if the technique solves the adjacent frames too differently, flickering
is introduced to the video, and this effect is the bane of many techniques that are otherwise
really good on still images.
Let’s have a look!
There is a tiny bit of flickering, but the results are surprisingly consistent.
It also does quite well when compared to previous methods, especially when we are able to provide
multiple images as an input.
Now note that it says “ours, without online optimization”.
What could that mean?
This online optimization step is a computationally expensive way to further improve separation
in the outputs, and with that, the authors propose a quicker and a slower version of
the technique.
The one without the on-line optimization step runs in just a few seconds, and if we add
this step, we will have to wait approximately 15 minutes.
I had to read the table several times, because researchers typically bring the best version
of their technique to the comparisons, and it is not the case here.
Even the quicker version smokes the competition!
Loving it.
Note if you have a look at the paper in the video description, there are, of course, more
detailed comparisons against other methods as well.
If these AR glasses that we hear so much about come to fruition in the next few years, having
an algorithm for real-time glare, reflection and obstruction removal would be beyond amazing.
We truly live in a science fiction world.
What a time
to be alive!
Thanks for watching and for your generous support, and I'll see you next time!