Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to make some crazy synthetic music videos.
In machine learning research, view synthesis papers are on the rise these days.
These techniques are also referred to as NERF variants, which is a learning-based algorithm
tries that to reproduce real-world scenes from only a few views.
It is very challenging, look!
In go a bunch of photos of a scene, and the method has to be able to synthesize new photorealistic
images between these photos.
But this is not the only paper in this area, researchers are very aware of the potential
here, and thus, a great number of NERF variants are appearing every month.
For instance, here is a recent one that extends the original technique to handle shiny and
reflective objects better.
So, what else is there to do here?
Well, look here.
This new one demands not a bunch of photos from just one camera, but from 16 different
cameras.
That’s a big ask.
But, in return, the method now has tons of information about the geometry and the movement
of these test subjects, so, is it intelligent enough to make something useful out of it?
Now, believe it or not, this, in return, can not only help us look around in the scene,
but even edit it in three new ways.
For instance, one, we can change the scale of these subjects, add and remove them from
the scene, and even copy-paste them.
Excellent for creating music videos.
Well, talking about music videos.
Do you know what is even more excellent for those?
Retiming movements…that is also possible.
This can, for instance, improve an okay dancing performance into an excellent one.
And three, because now we are in charge of the final footage, if the original footage
is shaky, well, we can choose to eliminate that camera shake.
Game changer.
Still, it’s not quite the hardware requirement where you just whip out your smartphone and
start nerfing and editing, but for what it can do, it really does not ask for a lot.
Look, if we wish to, we can even remove some of those cameras and still expect reasonable
results.
We lose roughly a decibel of signal per camera.
Here is what that looks like.
Not too shabby!
And all this progress just one more paper down the line.
And I like the idea behind this paper a great deal because typically what we are looking
for in a followup paper is trying to achieve similar results while asking for less data
from the user.
This paper goes into the exact other direction, and asks what amazing things could be done
if we had more data instead.
Loving it.
And with that, not only neural view synthesis, but neural scene editing is also possible.
What a time
to be alive!
Thanks for watching and for your generous support, and I'll see you next time!