Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to have a look at this insane
light transport simulation paper and get our minds blown in three chapters.
Chapter 1. Radiosity. Radiosity is an old old light transport algorithm that
can simulate the flow of light within a scene. And it can generate a scene of this quality.
Well, not the best, clearly, but this technique is from decades ago. Back in the day,
this was a simple and intuitive attempt at creating a light simulation program,
and quite frankly, the best we could do. It goes something like this: slice up the scene
into tiny surfaces, and compute the light transport between these tiny surfaces.
This worked reasonably well for diffuse light transport, matte objects if you will.
But, it was not great at rendering shiny objects. And it gets even worse, look. You see these blocky
artifacts here, these come from the fact that the scene has been subdivided into these surfaces,
and the surfaces are not fine enough for these boundaries to disappear.
But, yes, I hear you asking, Károly,
why talk about this ancient technique? I’ll tell you in a moment. You’ll see, I promise. So, yes,
radiosity is old, some professors still teach it to their students, it makes an interesting
history lesson, but I haven’t seen any use of this technique in the industry in decades now.
If radiosity would be a vehicle, it would be a horse carriage in the age of high-tech Tesla cars.
So, I know what you’re thinking. Yes, let’s have a look at those Teslas.
Chapter 2. Neural rendering. Many modern light simulation programs
can now simulate proper light transport with shiny objects, none of these blocky artifacts,
they are, as you see, absolutely amazing. They can render all kinds of material models,
detailed geometry, caustics, color bleeding, you name it. However, they seem to start
out from a noisy image, and as we compute the path of more and more light rays, this noisy image
clears up over time. But, this still takes a while. How long? Well, from minutes to days.
And then, neural rendering entered the fray. Here you see our earlier paper
that replaced the whole light simulation program with a neural network that learned
how to do this. And it can create these images so quickly that it easily runs not in minutes
or days, but as fast as you see here. Yes, in real time on a commodity graphics card.
Now note that this neural renderer is limited to this particular scene.
With this, I hope that it is easy to see that we are now
so far beyond radiosity that it sounds like a distant memory of the olden times. So, once again,
why talk about radiosity? Well, check this out.
Chapter 3. Neural Radiosity. Excuse me, what? Yes, you heard it right.
This paper is about neural radiosity. This work elevates the old old radiosity algorithm
by using a similar formulation to the original technique, but also, infusing it with a powerful
neural network. It is using the same horse carriage, but strapping it onto a rocket,
if you will. Now you have my attention. So, let’s see what this can do together.
Look. Hmm! Yes, it can render really intense specular highlights. And now,
hold on to your papers, and wow. Once again, the results look like a nearly pixel-perfect
copy of the reference simulation. And, we now understand the limitations of the old radiosity,
so, let’s strike where it hurts the most. Yes, perfectly specular, mirror-like surfaces.
Let’s see what happens here. Well, I can hardly believe what I am seeing here!
No issues whatsoever. Still, close to pixel perfect. This new paper
truly elevates the good old radiosity to the next level. So good! Loving it.
But wait a second. I hear you asking, yes, Károly, this is all well and good, but
if we have the reference simulation, why not just use that? Good question. Well,
that's right, the reference is great, but that one takes up to several hours to compute, and
the new technique can be done super quickly. Yet, they look almost exactly the same. My goodness.
In fact, let’s look at an equal-time comparison against one of the Tesla techniques, path tracing.
We give the two techniques the same amount of time, and see what they can produce. Let’s see.
Now that is no contest. Look. This is Eric Veach’s legendary scene where the light only comes from
the neighboring room through a door that is only slightly ajar. Notoriously difficult for any kind
of light transport algorithm. And yet, look at how good this new one is in tackling it.
Now, note that not even this technique is perfect. It has two caveats. One, the training takes place
per-scene. This means that we need to give these scenes to the neural network in advance,
so it can learn how light bounces off of this place, and this can take from minutes to hours.
But, for instance, if we have a video game with a limited set of places that you can go,
we can train our neural network on all of them in advance, and deploy it to the players,
who can then enjoy it for as long as they wish. No more training is required after that. But,
the technique would still need to be a little faster than it currently is.
And two, we also need quite a bit of memory to perform all this.
And, yes! I think this is an excellent place to invoke the First Law Of Papers,
which says that research is a process. Do not look at where we are,
look at where we will be two more papers down the line. And two more papers down the line,
who knows, maybe we get this in real time and with a much friendlier memory consumption.
Now, there is one more area that I think would be an excellent direction for future work. And
that is about the per-scene training of the neural network. In this work,
it has to get a feel of the scene before the light simulation happens. So how does the knowledge
learned on one scene transfer to others? I imagine that it should be possible to create a more
general version of this that does not need to look at a new scene before the simulation takes place.
And in summary. I absolutely love this paper. I takes an old old algorithm,
blows the dust off of it, and completely reinvigorates it
by infusing it with a modern learning-based technique. What a time to be alive!
So, what do you think? What would you use this for? I’d love to hear your thoughts,
please let me know in the comments below.
And when watching all these beautiful results, if you feel that this light transport thing is pretty
cool, and you would like to learn more about it, I held a Master-level course on this topic at the
Technical University of Vienna. Since I was always teaching it to a handful of motivated students,
I thought that the teachings shouldn’t only be available for the privileged few
who can afford a college education, but the teachings should be available for everyone.
Free education for everyone, that’s what I want. So, the course is available free of charge for
everyone, no strings attached, so make sure to click the link in the video description to get
started. We write a full light simulation program from scratch there, and learn about physics,
the world around us, and more. If you watch it, you will see the world differently.
Thanks for watching and for your generous support, and I'll see you next time!