Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to see how NVIDIA’s new AI can do 3 seemingly impossible things
with just one elegant technique.
For reference, here is a previous method that can take a collection of photos like these,
and magically, create a video where we can fly through these photos. This is what we call a
NERF-based technique, and these are truly amazing. Essentially, photos go in, and reality comes out.
So, I know you are thinking, Károly, this looks like science fiction. Can even this be topped?
And the answer is yes. Yes it can! Now, you see, this new technique can also
look at a small collection of photos, and, be it people or cats, and it learns to create a
continuous video of them. These look fantastic. And remember, most of the
information that you see here is synthetic. Which means, it is created by the AI. So good!
But, wait, hold on to your papers,
because there is a twist! It is often the case for some techniques that they think
in terms of photos. While other techniques think in terms of volumes. And get this,
this is a hybrid technique that thinks in terms of both! Okay…so what does that mean? It means this!
Yes, it also learned to now only generate these photos, but also, the 3D geometry of these models
at the same time. And this quality for the results is truly something else. Look at how previous
techniques struggle with the same task. Wow. They are completely different than the input model.
And you might think that of course they are not so good. They are probably very old methods. Well,
not quite! Look, these are not some ancient techniques, for instance, GIRAFFE is from
the end of 2020 to the end of 2021 depending on which variant they used. And, let’s see
what the new method does on the same data…wow. My goodness. Now that is something. Such improvement
in so little time. The pace of progress in AI research is nothing short of amazing.
And not only that, but everything it produces is multi-view consistent. This means that we don’t
see a significant amount of flickering as we rotate these models. There is a tiny bit on the
fur of the cats, but other than that, very little. That is a super important usability feature.
But wait, it does even more. Two, it can also perform one of our favorites, super resolution.
What is super resolution? Simple, a coarse, pixelated image goes in, and what comes out?
Of course, a beautiful, detailed image.
And here comes number three. It projects these images into a latent space. What does that mean?
A latent space is a made-up place where we are trying to organize data
in a way that similar things are close to each other. In our earlier work,
we were looking to generate hundreds of variants of a material model to populate this scene. In
this latent space, we can concoct all of these really cool digital material models.
A link to this work is available in the video description.
Now, let's see. Yes. When we take a walk in the internal latent space of this technique,
we can pick a starting point, a human face, and generate these animations as this face
morphs into other possible human faces. In short, this can generate a variety of different people.
Very cool. I love it! Now, of course, not even this technique is perfect, I see
some flickering around the teeth, but otherwise,
this will be a fantastic tool for creating virtual people. And remember, not only photos of virtual
people, we get the 3D geometry for their heads too. With this, we are one step closer
to democratizing the creation of virtual humans in our virtual worlds. What a time to be alive!
And, if you have been holding on to your papers so far, now, squeeze that paper! Because get this,
you can do all of this in real time. And all of these applications can be done with just one
elegant AI technique. Once again, scientists at NVIDIA knocked it out of the park with this one.
Bravo. So, what about you? If all this can be done in real time, what would you use this for?
I’d love to know - please let me know in the comments below.
Thanks for watching and for your generous support, and I'll see you next time!