Thanks for attending my talk everyone, I am Karan Ahuja, a fifth year PhD student
at the Human Computer Interaction Institute of Carnegie Mellon University. Today I am going to
talk to you about ControllerPose - Inside-Out Body Capture with VR Controller Cameras.
This work was done with my collaborators Vivian, Cathy, Nathan, Andy and Chris.
Today’s consumer grade virtual reality systems generally only
track the position of a user’s head and hands.
To capture the rest of the body, previous systems have opted to use accessory worn sensors.
Or external tracking infrastructure such as the Oculus Rift.
Recently researchers have looked into more mobile systems by making use of headset-mounted cameras,
but these offer an oblique and often incomplete view of the user’s lower body.
We present a new and practical method for capturing user body pose in VR:
integrating cameras into the controllers, where batteries,
computation and wireless communication already exist.
By virtue of the hands operating in front of the user during many VR interactions,
our controller bone cameras can capture a superior view of the body for digitization.
More specifically our pipeline composites together both cameras on each controller
to create two minimally occluded 185 degree views of the user.
We can then estimate the 2D body pose in each view.
We use the resulting key points to create two more views
that are torso normal aligned and compute two more body poses.
We take this multi-view data and combine it with head and hand data reported by the headset and
VR controllers. This fused sensor information is fed to a multi-modal neural network which outputs
a full body 3D pose estimate to help reject outlier poses and provide continuous animation
during brief periods of tracking loss, we passed a raw 3D pose estimate
through an inverse kinematic solver which controls an avatar.
Our model was trained on data from nine participants performing different poses
while in VR. To evaluate our model we recruited eight test participants
and collected around 34,000 instances of data across a variety of different poses.
There was no overlap between test and train participants.
When benchmark against connect capture ground truth,
ControllerPose had a mean 3D euclidean joint error of 6.98 centimeters.
The distribution of the error across the 17 body joints can be seen here.
We showcase the utility of ControllerPose through a variety of example applications.
For example, we created this human tetris game, where users must assume
the correct full-body pose to avoid obstacles and progress in the game.
As another game example, we created “Feet Saber”,
where not only the hands must hit blocks, but also the feet.
Similarly, this hockey goalie demo allows users to block shots with any part of their body.
Uses in health and fitness are possible too, such as this balance exercise app,
which could be part of a rehabilitation regime.
We can also extend the repertoire of exercises
that VR fitness apps track, such as rep counting squats and lunges.
While we believe ControllerPose offers a new and practical way to digitize the
body – especially the lower body – it nonetheless has pros and cons like any technical approach.
First are limitations of a computer vision approach operating from a handheld controller.
ControllerPose cannot resolve a good view of the body if the hands are too close to the user,
and certainly if the arms are resting by the side of the body. In these cases,
it is impossible to get a full view of the body with our current camera placement.
However, if an additional camera was added to capture the legs,
it seems possible to estimate body pose even from these disadvantaged views.
Furthermore, the framerate, latency and other performance characteristics
are not comparable to that of a commercially engineered and optimized implementation.
ControllerPose’s full pipeline runs at roughly 7 fps with a 297 ms latency. Put simply,
our performance stats should be considered the floor of performance, not the ceiling.
Lastly, ControllerPose already fuses computer-vison derived body keypoints
with IMU data from the headset and controllers. However,
there are other sources of data future systems could leverage
to further improve accuracy and robustness, such as the cameras and IMU found in the VR headsets.