Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Finally, this is my happy episode. Well, of course, I am happy in every episode,
but this, this is going to be my happy happy episode, if you will. Why is that?
Well, buckle up, because today, we are going to use OpenAI’s DALL-E 2, a text to image AI and
we will see what it is made of. Can it create beautiful light transport effects or not? We
will see through 4 beautiful experiments. For instance, this may sound like science fiction,
but today, we will also see if it can recreate this scene from a true master of digital 3D art.
So, what is this light thing I keep talking about? A light transport simulation means a
computer program that is able to compute the path of light rays to create beautiful images
like this and this and this. And our key problem is that initially, we only get
noisy images and it can take quite a long time for the simulator to eliminate this noise. So,
can DALL-E 2 help with that? Well, how? For instance, it can perform variant generation,
where in goes one image, and the AI synthesizes other, similar images. This is really cool, as it
means that the AI has a good understanding of what it sees and can create different variations of it.
And, wait a minute. Are you thinking what I am thinking? Oh yes, experiment number one,
denoising! Let’s give this noisy input image from a light transport simulator,
give it to the variant generator, and see if it is able to recreate the essence of the image,
without the noise. Let’s see…well, that is super interesting. It did not denoise the
image, but it did something else…it tried to understand what the noise is in this context,
and found it to be some sort of gold powder. How cool is that?
Based on the insights gained here, let’s try again with a little less noise. Oh yes,
this difficult scene would normally take even up to days to compute correctly. Do you see these
light streaks here? We would need to clean those up. So, variant generation, it’s your turn again!
And, look at that! Wow. We get a noise-free image that captured the essence of our input.
I cannot believe it. So good! Interestingly, it did not ignore the the light streaks,
but it thought that this is the texture of the object, and synthesized the new ones
accordingly. This actually means that DALL-E 2 does what it is supposed to be doing,
faithfully reproducing the scene and putting a different spin on it. So cool!
And, I think this concept could be supercharged by generating such a noisy input quickly,
then, denoising it with one of those handcrafted techniques for these images,
these are typically not perfect, but they may be just good enough to kickstart the
variant generator. I would love to see some more detailed experiments in this direction.
Now, what else can this do? Well, Experiment number two, my favorite, caustics. Oh yes! These
are beautiful patterns of reflected light that we see a lot of in real life, and they produce
some of the most beautiful images any light transport simulation can offer. Yes, that’s right,
with such a simulation, we can compute this too, how cool is that! So now, let’s ask DALL-E 2 to
create some of these for us. And the results are truly sublime. Regular caustics, checkmark. What
about those fun, heart-shaped caustics when we put a ring in the middle of an open book?
My goodness, the AI understands that and it really works! Loving it!
However, if you look at these beautiful volumetric caustics,
when running variant generation on that, it only kind of works. There are some rays of hope here,
but otherwise, I feel that the AI thinks that this is some sort of laser experiment instead.
And also, don’t forget about Danielle Baskin’s amazing results who created these drinks,
but wait, we are light transport researchers here,
so, we don’t look at the drink, what do we look at? Yes, of course, the caustics. Beautiful.
And, if we are looking at beautiful things, time for experiment number three. Subsurface
scattering. What is that? Oh boy, subsurface scattering is the beautiful effect of light
penetrating our skin, milk, and other materials and bouncing inside before coming out again.
The lack of this effect is why the skin looks a little plasticky in older video games. However,
light transport simulation researchers took care of that too, this is from our
earlier paper with the Activision Blizzard game development company, this is the same phenomenon,
a simulation without subsurface scattering, and this one is with simulating this effect. And,
in real time! Beautiful. You can find a link to this paper in the video description.
So, can an AI pull this off today? That’s impossible, right? Well, it seems so. If
I plainly ask for subsurface scattering from DALL-E 2, I did not get any of that. However,
when prompting a text to image AI, we have to know not only what we wish to see,
but how to get it out of the algorithm. But, if we ask for translucent objects with
strong backlighting, bingo! DALL-E 2 can do this too, so good. Loving it.
And now, hold on to your papers, because now is the time for our final experiment. Experiment
number four, reproducing the work of true master. If the previous one was nearly impossible, I don’t
know what this is. Here is a beautiful little virtual world from Reynante Martinez, and it
really speaks for itself. Now, let’s put it into the variant generator and see what DALL-E 2 is
made of. Wow! Look at that! These are incredibly good. Not as good as the master himself,
but I think the First Law of Papers should be invoked here. Wait, what is that? The First
Law of Papers says that research is a process - do not look at where we are, look at where we
will be two more papers down the line. And, two more papers down the line, I have to say I can
imagine that we will get comparable images. I also love how it thinks that fingerprints are part of
the liquid. It is a bit of a limitation, but a really beautiful one. What a time to be alive!
And, we haven’t even talked about indirect illumination, dispersion and many other
amazing light transport effects. I really hope we will see some more experiments, perhaps from
you Fellow Scholars in this direction too! By the way, I have a light transport simulation
course for all of you, free of charge, no strings attached, and we write a beautiful
little simulator that can create this image and more. The link is in the video description.
Thanks for watching and for your generous support, and I'll see you next time!