Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to learn how to take an object in the real world, make a digital copy of it,
and place it into our virtual world.
This can be done through something that we call differentiable rendering. What is that?
Well, simple, we take a photograph, and have an AI find a photorealistic material model
that we can put into our light simulation program that matches it. What this means is that now,
we can essentially put this real material into a virtual world. This earlier work
did very well with materials, but it did not capture the geometry.
Later, this followup work was looking to improve the geometry side of things. Here,
let’s try to reproduce a 3D model from a bunch of triangles. Let’s see….and. Look.
This is lovely. Seeing these images gradually morph into the right solution is an absolutely
beautiful sight. But, of course, in this case, the materials are gone.
And, if we wish to focus more on materials, and we don’t care about geometry at all,
we can use one of our previous papers for that. This one uses a learning algorithm to find out
about your artistic vision and recommends really cool materials.
Or, with this other work, you can whip up a fake material in photoshop, and it will magically find
a digital photorealistic material that matches it. But once again, no geometry. Materials only.
But wait a minute. Am I reading this right? With previous method number one
we get the materials, but no geometry. Or, previous method number two. No materials.
But, the geometry is much more detailed. Or, previous method number three. Excellent
materials, no geometry. We can’t have it all, can we?
So, is that it? We can’t really take a real object and make a virtual copy of it? Is the dream dead?
Well, don’t despair quite yet! This new technique might just be what we are looking for. Although I
am not so sure, this problem is super challenging. Let’s try this chair and see what happens.
Here are a bunch of images of it. Now, your turn, little algorithm!
And, we start out from a sphere. Well, I will believe this when I see it! Good luck! And, hmm.
The shape is starting to change, and…wait! Are you seeing what I am seeing?
Yes! Yes! Yes! The material is also slowly starting to change. We are getting
not only geometry, but materials too! Oh boy! Now, based on the previous works, our expectation
is that the result will be extremely coarse. Well, hold on to your papers, and let’s see
how much detail we end up with. Oh my. That is so much better. Loving it! Actually, let’s check.
This was the previous technique, and this is the new one. My goodness,
I have to say there is no contest here. The new one is so much more detailed! So good!
And, it works on a bunch of other examples too! These are not perfect by any means, but
this kind of improvement, just one more paper down the line. That is absolutely
incredible. The pace of progress in computer graphics research is nothing short of amazing,
especially when we look at all the magic scientists are doing in Wenzel Jakob’s lab.
And, the whole process took less than an hour. That is a considerable amount of time, but
an artist can leave a bunch of these computations to cook overnight,
and by the morning, a bunch of already pretty good quality virtual objects will appear
that we can place in our virtual worlds. How cool is that? What a time to be alive!
Make sure to check out the whole paper, it has crystal-clear mathematics, and
tons of gorgeous images. The link is available in the video description. I
believe this will become an excellent tool in democratizing asset creation for games,
animation movies, and all kinds of virtual worlds.
So, what would you use this for? Do you have some cool ideas? Let me know in the comments below!
Thanks for watching and for your generous support, and I'll see you next time!