Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
When we look at the cover page of a magazine, we often see lots of well-made, but also idealized
photos of people.
Idealized here means that the photographer made them in a studio, where they can add
or remove light sources and move them around to bring out the best from their models.
But most photos are not made in the studio, they are made out there in the wild where
the lighting is what it is, and we can’t control it too much.
So, with that, today, our question is what if we could change the lighting after the
photo has been made?
This work proposes a cool technique to perform exactly that by enabling us to edit the shadows
on a portrait photo that we would normally think of deleting.
Many of these have to do with the presence of shadows, and you can see here that we can
really edit these after the photo has been taken.
However, before we start taking a closer look at the editing process, we have to note that
there are different kinds of shadows.
One, there are shadows cast on us by external objects, let’s call those foreign shadows,
and there is self-shadowing, which comes from the model’s own facial features.
Let’s call those facial shadows.
So why divide them into two classes?
Simple, because we typically seek to remove foreign shadows, and edit facial shadows.
The removal part can be done with a learning algorithm, provided that we can teach it with
a lot of training data.
Let’s think about ways to synthesize such a large dataset!
Let’s start with the foreign shadows.
We need image pairs of test subjects with and without shadows to have the neural network
learn about their relations.
Since removing shadows is difficult without further interfering with the image, the authors
opted to do it the other way around.
In other words, they take a clean photo of the subject, that’s the one without the
shadows, and then, and add shadows to it algorithmically.
Very cool!
And, the results are not bad at all, and get this, they even accounted for subsurface scattering,
which is the scattering of light under our skin.
That makes a great deal of a difference.
This is a reference from a paper we wrote with scientists at the University of Zaragoza
and the Activision Blizzard company to add this beautiful effect to their games.
Here is a shadow edge without subsurface scattering, quite dark, and with subsurface scattering,
you see this beautiful glowing effect.
Subsurface scattering indeed makes a great deal of difference around hard shadow edges,
so huge thumbs up for the authors for including an approximation of that.
However, the synthesized photos are still a little suspect.
We can still tell that they are synthesized.
And that is kind of the point.
Our question is “can the neural network still learn the difference between a clean
and a shadowy photo” despite all this?
As you see, the problem is not easy - previous methods did not do too well on these examples
when you compare them to the reference solution.
And let’s see this new method.
Wow, I can hardly believe my eyes.
Nearly perfect.
And it did learn all this on not real, but synthetic images.
And believe it or not, this was only the simpler part.
Now comes the hard part.
Let’s look at how well it performs at editing the facial shadows!
We can pretend to edit both the size and the intensity of these light sources.
The goal is to have a little more control over the shadows in these photos, but, whatever
we do with them, the outputs still have to remain realistic.
Here are the before and after results.
The facial shadows have been weakened, and depending on our artistic choices, we can
also soften the image a great deal.
Absolutely amazing.
As a result, we now have a two-step algorithm, that first, removes foreign shadows, and is
able to soften the remainder of the facial shadows, creating much more usable portrait
photos of our friends, and all this after the photo has been made.
What a time to be alive!
Now, of course, even though this technique convincingly beats previous works, it is still
not perfect.
The algorithm may fail to remove some highly detailed shadows, you can see how the shadow
of the hair remains in the output.
In this other output, the hair shadows are handled a little better, there is some dampening,
but the symmetric nature of the facial shadows here put the output results in an interesting
no man’s land where the opacity of the shadow has been decreased, but the result looks unnatural.
I can’t wait to see how this method will be improved two more papers down the line.
I will be here to report on it to you, so make sure to subscribe and hit the bell icon
to not miss out
on that.
Thanks for watching and for your generous support, and I'll see you next time!