Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
When shooting feature-length movies, or just trying to hold meetings from home through
Zoom or Skype, we can make our appearance a little more professional by hiding the mess
we have in the background by changing it to something more pleasing.
Of course, this can only happen if we have an algorithm at hand that can detect what
the foreground and background is, which, typically, is easiest when we have a green screen behind
us that is easy to filter for even the simpler algorithms out there.
However, of course, not everyone has a green screen at home, and even for people who do,
may need to hold meetings out there in the wilderness.
Unfortunately, this would mean that the problem statement is the exact opposite of what we’ve
said, or in other words, the background is almost anything else but a green screen.
So, it is possible to apply some of these newer neural-network based learning methods
to tackle this problem?
Well, this technique promises to make this problem much, much easier to solve.
All we need to do is take two photographs, one with, and one without the test subject,
and it will automatically predict an alpha matte that isolates the test subject from
the background.
If you have a closer look, you’ll see the first part of why this problem is difficult.
This matte is not binary, so the final compositing process is not given as only foreground or
only background for every pixel in the image, but there are parts, typically around the
silhouettes and hair that need to be blended together.
This blending information is contained in the grey parts of the image, and are especially
difficult to predict.
Let’s have a look at some results!
You see the captured background here, and the input video below, and you see that it
is truly a sight to behold.
It seems that this person is really just casually hanging out in front of a place that is definitely
not a whiteboard.
It even works in cases where the background or the camera itself is slightly in motion.
Very cool!
It really is much, much better than these previous techniques, where you see that temporal
coherence is typically a problem.
This is the flickering that you see here, which arises from the vastly different predictions
for the alpha matte between neighboring frames in the video.
Opposed to previous methods, this new technique shows very little of that.
Excellent!
Now, we noted that a little movement in the background is permissible, but it really means
just a little…if things get too crazy back there, the outputs are also going to break
down.
This wizardry all works through a generative adversarial network, in which one neural network
generates the output results.
This, by itself, didn’t work all that well, because images used to train this neural network
can differ greatly from the backgrounds that we record out there in the wild.
In this work, the authors bridged the gap by introducing a detector network, that tries
to find faults in the output and tell the generator if it has failed to fool it.
As the two neural networks duke it out, they work and improve together, yielding these
incredible results.
Note that there are plenty of more contributions in the paper, so please make sure to have
a look for more details!
What a time to be alive!
What you see here is an instrumentation of this exact paper we have talked about, which
was made by Weights and Biases.
I think organizing these experiments really showcases the usability of their system.
This episode has been supported by Weights & Biases.
Here, they show you how you can use Sweeps, their tool to search through high-dimensional
parameter spaces and find the best performing model.
Weights & Biases provides tools to track your experiments in your deep learning projects.
Their system is designed to save you a ton of time and money, and it is actively used
in projects at prestigious labs, such as OpenAI, Toyota Research, GitHub, and more.
And, the best part is that if you are an academic or have an open source project, you can use
their tools for free.
It really is as good as it gets.
Make sure to visit them through wandb.com/papers or just click the link in the video description
and you can get a free demo today.
Our thanks to Weights & Biases for their long-standing support and for helping us make better videos
for you.
Thanks for watching and for your generous support, and I'll see you next time!