Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today I am very happy. Do you know why? Of course, because we
are going to talk about an incredible AI light transport simulation paper.
You see, these simulations normally take place by creating a little virtual scene,
and we would like to see what this virtual scene would look like in reality. We can
do this by using a technique called ray tracing, which simulates the path
of millions and millions of light rays to create a beautiful image or even video. This
technology is also used for simulating the path of soundwaves in some modern speakers,
and even in Apple’s new Vision Pro headset. So, millions of rays, right? Sounds great!
However, look. This is anything but a beautiful scene. Why is that? Well, before
we simulate enough light rays, which may take from minutes to hours, we will get a
noisy image. This image will get less and less noisy over time, however, we don’t have hours
of time per image. In a video game or any kind of virtual world, we prefer real time results.
So, what do we do? Well, we start noise filtering. These are techniques where
we can show a noisy image. And, magically, they clear up this
noise by imagining what this image could depict. Some of them are incredibly good,
and many of these AI-based techniques are so fast that they even work for video. Loving it.
However, not so fast. Unfortunately, if we add volumetric effects like haze and smoke,
things get a lot more difficult. Look. This is the noisy input with smoke. And a previous technique
does…this. Whoa. That is not great. Well, do not worry, because this technique doesn’t
yet understand smoke, just hard surfaces. So let’s see what this previous technique
with additional training on volumes can do. Ouch. Well, smoke looks a bit more like smoke now, but
unfortunately, the whole scene got destroyed. So, is that it? We can perform incredible ray tracing
on these beautiful scenes, but only without haze smoke and similar volumetric effects?
Well, don’t despair quite yet, hold on to your papers Fellow Scholars,
and have a look at the new technique. Wow!
This is way better than the previous technique. Actually, it is not perfect, but so much closer to
the true simulation result. This is the one that would have taken up to several hours to compute,
and this is what we now get almost immediately from the new AI. My goodness! And once again,
this is compared to the previous method that was trained on volumes too. And this
was not just by chance, when comparing other cases, the new method also comes out ahead.
So, if the new one is so good, this previous one must be from super long ago, right? Some ancient
paper. Well, is it some ancient paper? Nope. Not at all, this is the Neural Temporal Adaptive
Sampling and Denoising paper from 2020. Such incredible improvement in just one paper. Wow!
And hold on to your papers Fellow Scholars, because here comes the best part:
all this runs now not in hours, and not even in minutes. What you see here runs in real
time. It can really take these incredibly noisy sequence of images, and my goodness,
it reconstructs these beautiful videos from it that are closer to reality than ever before. I
absolutely love this. It almost feels like we are living in a science fiction movie.
And can that really be? Yes, it gets even better. The technique decomposes these
images into surface and volume components, so it can separate the opaque objects in the scene
from the smoke itself. Is that good? Well, to say that is good news would be an understatement,
because look! It can even reimagine the scene as if the lighting was different,
or we can even play with the smoke plumes themselves. So, from now on, we will likely
be able to enjoy these beautiful games with simulated smoke. Bravo! What a time to be alive!
Also, training this neural network took only 24 hours on a single NVIDIA graphics card,
that is insanity, and that just needs to happen once, and then, everyone in the world can enjoy
it for as long as they wish. And it didn’t even take looking at millions and millions
of noisy and clean image pairs to train this network, it only took a bit more than
10,000 images. That is not a lot. Proper learning from very little data. I am out of words.
And when watching all these beautiful results, if you feel that this ray tracing thing is pretty
cool, and you would like to learn more about it, I held a Master-level course on this topic at the
Technical University of Vienna. Since I was always teaching it to a handful of motivated students,
I thought that it’s not right to have the teachings only be available for the privileged
few who can afford a college education. No. The teachings should be available for everyone. Free
education for everyone, that’s what I want. So, the course is available free of charge for
everyone, no strings attached, so make sure to click the link in the video description to get
started. We write a full light simulation program from scratch there, and learn about physics,
the world around us, and more. If you watch it, you will see the world differently.
Thanks for watching and for your generous support, and I'll see you next time!