Before we explain the motion-captured mouse who I’ve named Moby Fitzsimmons,
Let’s go to basics.
We’ve all seen some photo like this one, with some poor actor in a weird outfit, covered in dots
whether it’s on “The Hulk” Mark Ruffalo’s instagram
or Star Wars’ C3PO’s Anthony Daniels on his account.
But where does that motion…go?
By focusing less on the actors and more on how it works
you can understand the real artistry behind motion capture.
There are a few main ways to capture motion.
This is Rokoko’s Smartsuit Pro and it uses inertial sensors.
These broadcast the location of devices embedded inside the suit
kinda like how your phone knows which way it’s turned.
More common in high end video games and movies, you’ll see optical tracking of markers
in which a camera is learning where parts of a person’s body are by looking
for high contrast areas.
Jimmy Corvan directed business development for
motion capture studio, House of Moves for about ten years.
It worked on everything from the Injustice series
to Mortal Kombat to Marvel movies to Barbie's blog.
"Comment, like, and subscribe so you can get the latest and greatest Barbie vlogs and more!"
Is it more common in high end stuff like what y'all are doing?
Is that pretty much always going to be optical thing, rather than a sensor?
For the foreseeable future
and I'm sure there's somebody at one of the inertial companies that will correct me on this
but in all of our testing and everything that we've found...
optical motion capture is sub-millimeter accurate.
These suits don’t have any sensors at all.
They’re basically fashion, made to be seen really clearly.
I will always get the question: How hard are they?
How do you get the technology in the balls?
And I'm like, well, you can squeeze them.
They're just retro reflective tape on the outside of little squishy balls.
The footage shot with the suits on is fed through software that interprets
what the camera sees, before artists review it.
If you look at Mark Ruffalo’s tweet of him, Tom Holland,
and Don Cheadle on Avenger’s Endgame,
you can see how additional symbols allow software
to know more about the location of the suit on the performer.
Even if one symbol is dark or out of focus, the software
can figure out what’s going on thanks to the pattern.
The first step in the in the cleanup process is somebody goes through
and makes sure that you can see every one of those dots in every frame of the shot.
It sounds tedious because it is.
So all the markers have a different name
and somebody's job is going through making sure you can see each marker in every single frame
and that each marker is properly named.
That process is called tracking and labeling.
This can be augmented with facial capture like this
a range of motion capture gloves
and even the ability to label and motion track objects.
For hobbyists, there’s stuff like Moby Fitzsimmons.
His form comes from a free to download mocap-ready website.
I captured his movements using AI called Plask that just figured out the motion
without any dots or sensors.
These methods can vary wildly in cost and quality but the output is actually similar.
It’s bones.
This structure is the basic output of most motion captures
no cloth, no muscle, no hair
just what the software calculates as the skeleton.
Each dot here is a joint, while the longer shapes are bones.
In this example, they’re... my bones. Soon to be Moby’s.
These bones are given a hierarchy
so if the top of your arm rotates, the lower part usually will too.
These subtle variations can convey a lot of movement.
These numbers represent the rotation of a single elbow joint.
Watch how they change as the elbow moves during the animation.
We are trying to figure out exactly where the skeleton exists
because that's what we're really capturing, is your skeleton moving.
We aren't capturing the flesh on top.
We've got very, very close, but it is not perfect.
If the bone lengths are just slightly shorter or just slightly longer
then even something as simple as clapping
like if I were to do this, it might end up that they go through each other like this
because the bone lengths are different.
That's where animators come in.
There's a lot of that fixing and you end up seeing it in postures.
There's a very popular phrase in the mocap space called monkey butt.
And that is the the hips kind of jut back a little bit
and it kind of looks like they have a monkey butt.
And so an animator will need to go through and push the
hips back forward, kind of undo that work.
This outfit provides a clue to why motion capture is harder than it looks after the data is cleaned.
The Hulk is significantly larger than Mark Ruffalo.
VFX artists take the time to make Taika Waititi’s capture
fit the body of Korg, but you can see what goes wrong
without extra work.
See how Moby’s hands run into his face here?
That’s not a problem when I do it because I don’t have a head like Moby’s.
The same goes for position in space.
See how his feet wobble around?
This is part of why you don’t see tracking images
on the actor modeling for Groot, a tall talking tree
or actor Sean Gunn, who stands in for Rocket Raccoon.
These figures, and even motion captured figures, are too difficult to graft on human motion.
So the people just used for reference.
Even if you're not using all of the data, specifically
having the timing created by the actor or performer
is really, really helpful.
A lot of people, like myself included
I talk with my hands, and if an animator tried to animate me talking with my hands
I'd get the timing wrong.
Look at this side by side of Hulk and Mark Ruffalo.
Ruffalo gives a great performance that you can really see.
But look at Hulk’s shoulder size, and the cloth and the space his hands take up.
You can even see, in this clip, how his right hand has been tweaked completely after capture.
The way that motion capture tends to get covered is...
look Andy Serkis and look at Gollum.
They do a wonderful job of performing, but there is a huge gap in between.
That is months and months and months and months of work.
It's portrayed is just like there is an animate button.
Look at all the changes within this short clip.
That level of detail would take a long time for people to animate.
But the limited structure of the final format
means there’s a ton of animation after the capture.
Mocap depends on the quality of the capture and the work put into it.
That’s what makes animation feel real.
Without that, it’s just joints and bones.
Space Jam 2, legitimately we set up a system around
a professional sized basketball court.
The court that is in the movie is the court that we shot on
and they wrapped the entire thing in a giant green screen.
The whole thing.
So you have this self-contained volume that is holding all the air in itself
because it's just all the way to the ceiling, green screened all the way to the floor
360 degrees around.
And the director wanted to, instead of adding in fog effects in post
he pumped a bunch of fog into this volume
and I don't expect people to know what that does to
optical motion capture
but to say you can see fog and our cameras need to be able to see.
The way they work is the camera shoots light from a strobe around the camera at those little markers
and that little markers bounces the light right back to the camera
and fog introduces a bunch of little water molecules into the air
and water molecules scatter light.
So, we are now shooting in a volume that is filled with water
and just bouncing light all over the place.