Hi everyone, this is Jackie Yang from Stanford University Today, on behalf of my coauthors
Fang Qin and Tuochao Chen, as well as my advisors Monica Lam and James Landay.
I’m here to present HybridTrak Adding Full-Body Tracking to VR Using an Off-the-Shelf Webcam
So.
Full-body tracking is very important for VR.
It improves immersion in social VR apps and It is essential for functionalities in sport
VR apps However, Common full-body tracking systems need multiple anchored sensors which
are complex to setup More convenient VR systems use egocentric cameras.
They are growing in popularity, but they have high distortion and low accuracy on the legs.
So, let’s take a look at why this is the case.
On the left, we have an egocentric camera view As you can see, it has a better view
of the user’s hands but is not so good with the user's legs.
The user’s body commonly occludes their own legs.
On the right, we have a webcam anchored to the world.
This gives a clear view of the user’s legs at almost all times, but the user’s hands
can be occluded by their body.
That’s why commonly, full-body tracking systems need multiple sensors anchored to
the world These systems have very specific constraints on where the cameras are located.
This is so that the user’s hands are always visible to at least one of the sensors.
What if we can create a hybrid tracking system that uses both cameras to get less occlusion?
Then, we have much fewer constraints on where we place the cameras And that’s what we’ve
done with HybridTrak.
It works by processing 2D poses from a single webcam and 3D poses from an egocentric upper-body
VR tracking camera.
We use a pose conversion network To generate VR poses that can be used in any full-body
tracking app supported by SteamVR.
To make HybridTrak work, we used data augmentation and a synthetic dataset.
Specifically, we solved two problems: One is that 2D pose tracking does not work well
with a VR headset on the user’s face The second is the lack of data available to train
the pose conversion network For the 2D pose tracking, we used pose_resnet trained on the
COCO dataset.
However, people’s faces captured in the original dataset are almost always visible.
While using this with VR, the model sometimes thinks that the user is facing backward because
of the lack of direct sight of the user’s face.
This causes the user’s leg tracking points to be flipped in the wrong direction.
We fixed this by generating synthetic images by overlaying a 3D model of a VR headset on
top of the faces in existing images in the COCO dataset.
With this augmentation, our model generates much better 2D tracking results.
Another problem is how to train the pose conversion network.
We can ask many people to wear VR full-body tracking sensors to collect a new dataset,
but that would be very expensive.
Or, we can instead use an existing dataset like Human3.6m But this data set has very
different tracking points.
VR tracking points are usually 6 points with position and orientation, while the Human3.6m
dataset is composed of 17 points with position only.
So we created this conversion, that interpolates the 17 positions and calculates the transforms
to generate a similar 6 points format with both position and orientation for each point.
With this, we can create a new model without needing to collect a new dataset OK.
Let’s see the system in action.
On the left, we have the 2d pose tracking result.
On the right, we have the predicted six 3D tracking points with associated positions
and orientations.
Specifically, we have the user’s head, hands, waist, and feet tracking points similar to
those provided by a commercial full-body tracking system.
it looks pretty good when rendered on this flat 2D screen What about in actual VR apps?
To see how well this system works for actual users in a social VR app, We compared HybridTrak
to both a depth camera-based tracking system (KinectToVR) and to an upper-body-only tracking.
In this study, we invited participants from public chat rooms in VRChat to observe an
experimenter performing multiple poses in VR captured by the THREE different systems.
We found that participants rated the poses generated by HybridTrak to be significantly
more natural than both the depth camera condition and the upper-body only tracking condition.
They also rated the transition between poses by HybridTrak to be more natural than the
depth camera condition Here are some examples of poses that can be supported by HybridTrak.
These poses are well supported by both HybridTrak and KinectToVR For these poses, KinectToVR
starts to show some problems tracking the user’s legs, while HybridTrak still performs
pretty well.
For these poses, neither system generates satisfactory results.
We think this is because the posture the user is performing is not part of the 3D training
set, while the underlying 2D pose tracker continued to perform well.
Essentially, we only need to improve the pose conversion network.
It’s possible that we can leverage humanoid animations commonly present in VR games to
further fine-tune the model towards a specific game.
This will ensure that all the poses used in a particular game are supported by HybridTrak.
So HybridTrak shows a promising future where every VR user can have their full-body represented
in the virtual world, by adding just a single off-the-shelf webcam.
Thanks for watching