Thanks for attending my talk everyone, I am Karan Ahuja, a fourth-year PhD student at
the Human-Computer Interaction Institute of Carnegie Mellon University.
Today I am going to talk to you about PoseOnTheGo.
This work was done with my collaborators Sven Mayer, Mayank Goel and Chris Harrison.
PoseOnTheGo is a full-body, yet zero cost, motion capture system that uses sensors already
found in today’s smartphones, thus requiring no extra or external hardware.
To achieve this, it uses an extreme sensor fusion approach.
Using the front-facing camera of the smartphone, we can capture the relative spatial position
between the head and hand holding the phone.
As we do not have data for joints – such as the wrist, elbow and shoulder – we use
Inverse Kinematics to estimate and animate probable body poses.
Simultaneously, the IMU allows us to orient the hand holding the phone.
The hand that is not holding the phone generally cannot be seen or tracked, except when the
user touches the screen, which then allows us to estimate the finger and arm position.
By using the phone’s front-facing depth camera, we can compute a chest normal, which
allows us to orient a user’s torso independently from the head.
We can also animate the legs, despite having no direct sensor data below the torso.
First, we use the phone’s locomotion mode prediction to load the appropriate kinematic
scheme, such as standing, running or walking.
Then, by using the IMU to integrate absolute body movement, we coarsely animate the avatar’s
legs at the appropriate speed.
Finally, using the phone’s rear camera and visual odometry software, we can estimate
the elevation of the phone with respect to the ground plane, which, in turn, gives us
the elevation of the user’s arm and head.
We feed this height data to the Inverse Kinematic solver, which allows our pose model to stand,
step onto objects, and sit down.
We performed a series of studies across 8 participants, benchmarking our approach against
a professional-grade Vicon optical tracking system.
PoseOnTheGo’s error for estimating the head, torso and wrist rotation is about 7.5, 16
and 10 degrees respectively.
In terms of euclidean error, the mean error across all joints is roughly 21cm.
As the root node for all the poses is the chest, the error increases as we move away
from it.
The absolute world tracking error is around 24 cm with a head height tracking error of
10 cm.
Although Pose-on-the-Go is, unsurprisingly, less spatially accurate when benchmarked with
Vicon, the estimated pose is more than sufficient to power a wide variety of new and interesting
applications.
For example, it could enable new full-body, mobile gaming experiences, such as this fantasy
game.
In this third-person shooter, the phone is a laser, used to aim and shoot, and users
can freely walk around and duck behind cover.
Pose-on-the-Go could also be valuable for health and wellness applications.
For instance, phones could automatically count exercise repetitions and track performance
quality.
In this demo app, reaching exercises could be part of a physical therapy rehabilitation
regimen.
Similarly, today’s face-centric Animoji could be extended to full-body animations,
allowing for more expressive communication.
It is important to note that Pose-on-the-Go is only an estimation of the full-body pose,
and it is not meant to compete with high accuracy tracking systems such as Vicon.
It has incomplete data about the body, most notably the hand not holding the phone, which
is almost never digitized except when the user touches the screen.
In future, PoseOnTheGo can give a more comprehensive picture of the whole body by using the complementary
information from the smartwatch to capture the hand not holding the phone.
Also, it can use the camera to not only track the body pose but also the facial expression
and appearance of the user.