This episode has been supported by Lambda Labs.
Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
In our earlier paper, Gaussian Material Synthesis, we made a neural renderer, and what this neural
renderer was able to do is reproduce the results of a light transport simulation within 4 to
6 milliseconds in a way that is almost pixel perfect.
It took a fixed camera and scene, and we were able to come up with a ton of different materials,
and it was always able to guess what the output would look like if we changed the physical
properties of a material.
This is a perfect setup for material synthesis where these restrictions are not too limiting.
Trying to perform high-quality neural rendering has been a really important research problem
lately, and everyone is asking the question, can we do more with this?
Can we move around with the camera and have a neural network predict what the scene would
look like?
Can we do this with animations?
Well, have a look at this new paper which is a collaboration between researchers at
the Technical University of Munich and Stanford University, where all we need is some video
footage of a person or object.
It takes a close look at this kind of information, and can offer three killer applications.
One, it can synthesize the object from new viewpoints.
Two, it can create a video of this scene and imagine what it would look like if we reorganized
it, or can even add more objects to it.
And three, perhaps everyone’s favorite, performing facial reenactment from a source
to a target actor.
As with many other methods, these neural textures are stored on top of the 3D objects, however,
a more detailed, high-dimensional description is also stored and learned by this algorithm,
which enables it to have a deeper understanding of intricate light transport effects create
these new views.
For instance, it is particularly good at reproducing specular highlights, which typically change
rapidly as we change our viewpoint for the object.
One of the main challenges was building a learning algorithm that can deal with this
kind of complexity.
The synthesis of mouth movements and teeth was always the achilles heel of these methods,
so have a look at how well this one does with it!
You can also see with the comparisons here that in general, this new technique smokes
the competition.
So how much training data do we need to achieve this?
I would imagine that this would take hours and hours of video footage, right?
No, not at all.
This is what the results look like as a function of the amount of training data.
On the left, you see that is already kinda works with 125 images, but contains artifacts,
but if we can supply a 1000 images, we’re good.
Note that a 1000 images sounds like a lot, but it really isn’t, it’s just half a
minute worth of video.
How crazy is that?
Some limitations still apply, you see one failure case right here, and the neural network
typically needs to be retrained if we wish to use it on new objects, but this work finally
generalizes to multiple viewpoints, animation, scene editing, lots of different materials
and geometries, and I can only imagine what we’ll get two more papers down the line.
Respect to Justus for accomplishing this, and in general, make sure to have a look at
Matthias Niessner’s lab, who just got tenured as a full professor and he’s only 32 years
old.
Congratulations!
If you have AI-related ideas, and you would like to try them, but not do it in the cloud
because you wish to own your own hardware, look no further than Lambda Labs.
Lambda Labs offers sleek, beautifully designed laptops, workstations and servers that come
pre-installed with every major learning framework and updates them for you, taking care of all
the dependencies.
Look at those beautiful and powerful machines!
This way, you can spend more of your time with your ideas, and don’t have to deal
with all the software maintenance work.
They have an amazing roster of customers that include Apple, Microsoft, Amazon, MIT and
more.
These folks really know what they are doing.
Make sure to go to lambdalabs.com/papers or click their link in the video description
and look around, and if you have any questions, you can even even call them for advice.
Big thanks to Lambda Labs for supporting this video and helping us make better videos for
you.
Thanks for watching and for your generous support, and I'll see you next time!