Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today, we are going to take a light simulation algorithm that doesn’t work too well,
and infuse it with a powerful neural network. And what happens, bam! This happens. Wow.
So, what was all this about? Well, when we fire up a classical light simulation program,
first, we start out from…yes, absolutely nothing. And, look, over time, as we simulate the path of
more and more light rays, we get to know something about the scene. And slowly, over time, the image
cleans up. But there is a problem. What is the problem? Well, look.
The image indeed cleans up over time, but no one said that this process would be quick. In fact,
this can take from minutes, in the case of this scene, to get this, even days for this scene.
In our earlier paper, we rendered this beautiful, but otherwise,
sinfully difficult scene and it took approximately
three weeks to finish. And it also took several computers running at the same time.
However, these days, neural network-based
learning methods are already capable of doing light transport. Why not use them?
Well, yes they are, but they are not perfect. And we don’t want an imperfect result.
So, scientists at NVIDIA said that this might be the perfect opportunity to use
control variates. What are those and why is this the perfect opportunity?
Control variates are a way to inject our knowledge of a scene into the light transport algorithm.
And here is the key. Any knowledge is useful, as long as it gives us a headstart,
even if this knowledge is imperfect. Here is an illustration of that. What we do is
that we start out using that knowledge, and make up for the differences over time.
Okay, so, how much of a headstart can we get with this? Normally, we start out from a black image,
and then, a very noisy image. Now hold on to your papers, and let’s see what this can do to help us.
Wow. Look at that. This is incredible! This is not the final rendering, this is what the algorithm
knows, and instead of the blackness, we can start out from this. Goodness, it turns out that
this might not have been an illustration, but the actual knowledge the AI has of the scene. So cool!
Let’s look at another example. In this bedroom, this will be our starting point. My goodness,
is that really possible? The bed and the floor are almost completely done. The curtain
and the refractive objects are noisy, but do not worry about those for a second.
This is just a starting point, and it is still way better than starting out from a black image.
To be able to visualize what the algorithm has learned with a little more finesse, we can also
pick a point in space, and learn what the world looks like from this point and how much light
scatters around it. And we can even visualize what happens when we move this point around.
Man, this technique has a ton of knowledge about these scenes.
So, once again, just to make sure. We start out not from a black image, but from a learned image,
and now, we don’t have to compute all the light transport in the scene,
we just need to correct the differences. This is so much faster! But actually, is it?
Let’s look at how this helps us in practice. And that means, of course, equal time comparisons
against previous light transport simulation techniques. And, when it comes to comparisons,
you know what I want. Yes! I want Eric Veach’s legendary scene. See? This is an insane scene
where all the light is coming from not here, but the neighboring room through a door that is just
slightly ajar. And, the path tracer, the reference light transport simulation technique behaves as
expected. Yes, we get a very noisy image because very few of the simulated rays make it to the next
room. Thus, most of our computation is going to waste. This is why we get this noisy image.
And, let’s have a look what the new method can do in the same amount of time. Wow!
Are you seeing what I am seeing? This is completely out of this world. My goodness.
Okay, but this was a pathological scene designed to challenge our light transport
simulation algorithms. What about a more typical outdoors scene? Add tons of incoming light from
every direction and my favorite, water caustics. Do we get any advantages here? The path tracer
is quite noisy, this will take quite a bit of time to clean up. Whereas the new technique…oh
my, that is a clean image. How close is it to the fully converged reference image? Well, you tell
me, because you are already looking at that. Yes, now, we are flicking between the new technique
and the reference image, and I can barely tell the difference. There are some, for instance, here,
but that seems to be about it. Can you tell the difference? Let me know in the comments below.
Now, let’s try a bathroom. Lots of shiny surfaces and specular light transport. And the results are…
look at that! There is no contest here. A huge improvement across the whole scene.
And, believe it or not, you still haven’t seen the best results yet. Don’t believe it? Now, if
you have been holding on to your papers, squeeze that paper, and look at the art room example here.
This is an almost unusable image with classical light transport. And, are you ready? Well, look at
this. What in the world! This is where I fell off the chair when I was reading this paper. Absolute
madness. Look. While the previous technique is barely making a dent into the problem,
the new method is already just a few fireflies away from the reference. I can’t believe my eyes.
And this result is not just an anomaly. We can try a kitchen scene
and draw similar conclusions. Let’s see. Now we’re talking! I am out of words.
Now, despite all these amazing results, of course, not even this technique is perfect.
This torus was put inside a glass container, and is a nightmare scenario for any kind of light
transport simulation. The new method successfully harnesses our knowledge about the scene,
and accelerates the process a great deal, but, once again, we get fireflies.
These are going to be difficult to get rid of and will still take a fair bit of time. But
my goodness, if this is supposed to be a failure case…then yes, sign me up, right now!
Once again, the twist here is not just to use control variates,
the initial knowledge thing, because in and of itself,
it is not new. I, like many others have been experimenting with this method back in 2013,
almost ten years ago, and back then, it was nowhere near as good as this one. So, what is the
twist then? The twist is to use control variates, and infuse them with a modern neural network.
Infusing previous techniques with powerful learning-based methods is a
fantastic area of research these days. For instance, here you see an earlier result,
an ancient light transport technique called radiosity. This is what it was capable of
back in the day. And here is the neural network-infused version. Way better. I
think this area of research shows a ton of promise and am so excited to see more in this direction.
So, what do you think? What would you use this for? I’d love to hear your thoughts,
please let me know in the comments below.
And when watching all these beautiful results, if you feel that this light transport thing is pretty
cool, and you would like to learn more about it, I held a Master-level course on this topic at the
Technical University of Vienna. Since I was always teaching it to a handful of motivated students,
I thought that the teachings shouldn’t only be available for the privileged few
who can afford a college education, but the teachings should be available for everyone.
Free education for everyone, that’s what I want. So, the course is available free of charge for
everyone, no strings attached, so make sure to click the link in the video description to get
started. We write a full light simulation program from scratch there, and learn about physics,
the world around us, and more. If you watch it, you will see the world differently.
Thanks for watching and for your generous support, and I'll see you next time!