Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to see how NVIDIA’s crazy new AI is able to understand basically any
human movement.
So, how is this even possible?
And even more importantly, what is pose estimation?
Well, simple, a video of people goes in, and the posture they are taking comes out.
Now, you see here that previous techniques can already do this quite well, what’s more,
if we allow an AI to read the wifi-signals bouncing around in a room, it can perform
pose estimation, even through walls.
Kind of.
Now, you may be wondering what is pose estimation good for?
By the end of this video, you will see that this can help us move around in virtual worlds,
the metaverse, if you will.
But let’s not rush there yet, because still, there are several key challenges here.
One, we need to be super accurate to even put a dent into this problem.
Why?
Well, because if we have a video, and we are off by just by a tiny bit from frame to frame,
this kind of flickering may happen.
That’s a challenge.
What else is challenging here?
Two, foot sliding.
Yes, you heard it right.
Yes, previous methods suffer from this phenomenon, you can see it in action here.
And also, here too.
So, why does this happen?
It happens because the technique has no knowledge of the physics of real human movements.
So, scientists at NVIDIA, the University of Toronto and the Vector Institute fired up
a collaboration, and when I first heard about their concept, I thought you are doing what?
But, check this out.
First, they perform a regular pose estimation.
Of course, this is no good, as it has the dreaded temporal inconsistency, or in other
words, flickering.
And in other cases, often foot sliding too.
Now, hold on to your papers, because here comes the magic!
Now, they transfer the motion to a video game character, and embed that character in a physics
simulation.
In this virtual world, the motion can be corrected to make sure they are physically correct.
Now, remember, foot sliding happens because of the lack of knowledge in physics, so, perhaps
this idea is not that crazy after all!
Let’s have a look!
Now this will be quite a challenge.
Explosive sprinting motions.
And…whoa.
This is amazing.
This, dear Fellow Scholars, is superb pose estimation and tracking.
How about this?
A good tennis serve includes lots of dynamic motion, and just look at how beautifully it
reconstructs this move.
Apparently, physics works.
Now, the output also needn’t be stickmen, we can retarget these to proper textured virtual
characters built from a triangle mesh.
And, that’s just one step away from us being able to appear in a metaverse.
No head mounted displays are required, no expensive studio, no motion capture equipment
is required.
So, what IS required?
Actually, nothing!
Just a raw input video of us.
That is insanity.
And all this produces physically correct motion.
So, that crazy idea about taking people and transforming them into video game characters
is not so crazy after all.
So now, we are one step closer to be able to work, and even have some coffee together
in a virtual world.
What a
time
to be alive!
Thanks for watching and for your generous support, and I'll see you next time!