Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to spice up this family video with a new, stylized environment.
Then, we make a video of a boat trip much more memorable, up the authenticity of a parkour
video, decorate a swan, enhance our clothing a little, and conveniently forget about a
speeding motorcycle.
Only for scientific purposes, of course.
So, these are all very challenging tasks to perform, and of course, none of these should
be possible.
And this is a new AI-based solution that can pull of all of these!
But…
how?
Well, in goes an input video, and this AI decomposes it into a foreground and background,
but in a way that it understands that this is just a 2D video that represents a 3D world.
Clearly, humans understand this, but does Adobe’s new AI do it too?
And I wonder how much it understands about that?
Well, let’s give it a try together.
Put those flowers on that dress, and…what?
The flowers look like they are really there as they wrinkle as the dress wrinkles, and
it catches shadows just as the dress catches shadows too.
That is absolutely incredible.
It supports these high-frequency movements so well that we can even stylize our kite
sailing videos with it, where there is tons of tiny water droplets flying about.
No problems at all.
We can also draw on this dog, and remember, we mentioned that it understands the difference
between foreground and background, and…look, the scribble correctly travels behind objects.
Aha, and this is also the reason why we can easily remove a speeding motorbike from this
security footage.
Just cut out the foreground layer.
Nothing to see here.
But I wonder, can we go a little more extreme here?
And it turns out, these are really nothing compared to what this new AI can pull off.
Look.
We can not only decorate this swan, but, here is the key…and…oh yes!
The swan is fine, yes, but the reflection of the swan is also computed correctly.
Now this feels like black magic, and we are not even done yet, now hold on to your papers,
because here come my two favorite examples!
Example number one.
Biking in wonderland.
I love this.
Now, you see that not even this technique is perfect, if you look behind the spokes
of the wheel, you see a fair amount of warping.
I still think this is a miracle that it can be pulled off at all.
Example number two, a picturesque trip.
Here, not only the background has been changed, but even the water has changed as chunks of
ice have also been added.
And with that, I wonder how easy it is to do this?
I’ll tell you in a moment, after we look at this.
Yes.
There is also one issue here with the warping of the water in the wake of the boat, but
this is an excellent point for us to invoke the First Law of Papers, which says, do not
think of where we are, look at where we will be two more papers down the line.
So, how much work do we have to put in to pull all this off?
A day of work maybe?
Nope, not even close!
If you have been holding on to your papers so far, now, squeeze that paper, and look
at this.
We can unwrap the background of the footage, and stylize it as we please.
All this takes is just editing one image.
This goes in, this comes out.
Wow.
Absolutely anyone can do this, so this is a huge step in democratizing video stylization
and bringing it to everyone.
What a time to
be alive!
Thanks for watching and for your generous support, and I'll see you next time!