Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to use an AI to take a collection of photos,
and hopefully create a digital human out of them.
However, not so fast! For instance, have a look at NVIDIA’s amazing face generator AI. As you see,
it can generate beautiful results, and they even show us the illusion
of these characters moving around, however, this does not have a strong concept of 3D information.
These are 2D images, and 3D consistency is not that great. What is that?
Well, look. This person just one frame away is not exactly the same person. For instance, the
hair moves around. This means that these images are not art directable, or at least, not easily.
As you see here, it has improved a great deal over two years and
one more research paper down the line, it’s still not quite there.
So, fine details, checkmark, 3D consistency, not quite there.
So let’s have a look at solutions that have more 3D consistency. To be able to do that,
we can take a collection of photos like these, and magically, create a video
where we can fly through these photos. It is really crazy, because this is possible today,
for instance, here is NVIDIAs method that can be trained to perform this in a matter of seconds.
This is remarkable, especially that the input is only a handful of photos.
Some information is given about the scene, but this is really not much.
So, can we have a solution with 3D consistency? Yes, indeed, for these, 3D consistency,
checkmark. However, the amount of fine details in these outputs is not that great.
Do you see the pattern here?
Depending on which previous method we use, we either get tons of detail, but no 3D consistency,
or we can get our highly coveted 3D consistency, but then, the details are gone.
So, is the dream dead? No digital humans for us?
Well, don’t despair, and have a look at this new technique that promises one megapixel images that
are also consistent. Details and consistency at the same time! When I first saw this paper,
I said I will believe it when I see it, so, let’s have a look together.
This is StyleNERF, a previous technique, and we see the problems that we now expect - the hair
is not consistent, the earring is flickering a great deal, and there are other issues too.
So, are the authors of the new paper claiming that they can solve all this? Well,
now, hold on to your papers, and have a look at this. This is the new technique.
Oh my goodness. The earring, facial features and the hair stay still as we rotate them,
and it indeed shows the new angles correctly. I love it!
Let’s examine this phenomenon
in a little more detail. Look at the hair here with StyleNERF. It looks decent,
but it’s not quite there. And I really wonder why that is. Let’s zoom in and have a closer look.
Oh yes, upon closer inspection, this hair is a bit of a mess. And, with the new technique, now
that is what I call consistency across the video frames. Smooth hair strands everywhere. So good!
And what I absolutely loved about this new work is that it outperforms this previous technique,
which is from how many years ago? Well, if you have been holding on to your papers, now
squeeze that paper, because this work is from not even one year ago, just from 8
months ago. And you already see a meaningful improvement over that. That is insanity.
Now, not even this technique is perfect. I don’t know for sure if the hair consistency
is perfect here and we are dealing with video compression artifacts,
or whether this is still not a 100% there, but this truly is a great step forward.
Also, here, you see that the images are still not as detailed as the photos,
but this seems to be a roadblock to me that is much easier to solve than 3D consistency. My
impression is that with this, the most difficult part of the task is already done, and just one or
two papers down the line, and I am sure we will be seeing even more realistic virtual humans.
So, can we enter into a virtual world as a digital human from just a collection of photos?
Oh yes! Time to meet our beloved ones from afar, or meet new people, and play some games together.
What a time to be alive! So, does this get your mind going?
What would you use this for? Let me know in the comments below!
Thanks for watching and for your generous support, and I'll see you next time!