Greetings everyone, I am Karan Ahuja a fifth year PhD student
at the Human-Computer Interaction Institute of Carnegie Mellon University.
Today I'm going to talk to you about my work on CoolMoves - user
motion accentuation in Virtual Reality. This work was done with my collaborators
Eyal Ofek, Christian Holz , Mar Gonzalez and Andy Wilson at Microsoft Research.
Virtual reality is a wonderful technology that enables a user to disconnect from the real
environment and transform themselves into new and imaginative worlds.
Here, the appearance of the user's avatar can be completely different from the real world.
However in this virtual world our motions remain tightly coupled to our body.
In this game notice the difference between the avatar controlled by the user versus
the computer-generated one. The computer aided one is stylized and has a full range of motion
whereas the user in VR has only two motions: blocking and punching.
This is quite different from the motions we are accustomed to in non-VR settings.
In current PC and console game environment avatars have a characteristic style associated
with them. In FIFA, Ronaldo runs and kicks the ball in his unique character style.
Similarly in tennis the motion is stylized for each character.
However current VR systems are bereft of such stylization and
embellishment of the user's motion.
Let's dive into why that is.
The first reason is that unlike keyboards and controllers that are discrete button
presses mapped to character motions the motion in VR is continuous in nature.
This continuous data stream is coming in at at least 60 Hz
and the systems need to process it that fast to keep the VR experience real time.
It is also paramount that the user should feel a sense of agency
-that is control over their avatar, and a sense of embodiment - which is akin to body
ownership. Thus synchronization between the user and the avatar's motion profile is key.
Furthermore current consumer-grade VR systems track input from the
user's headset and handheld controllers only.
This severely limits the sense of embodiment of the user as they
cannot see themselves represented in the VR world.
When full body avatars are introduced without keeping the human motion
dynamics in mind it can lead to results that break the immersion as seen here.
To alleviate this inverse kinematics has been used to generate the untracked joints.
This inherently limits the expressiveness and elegance of the avatar's range of motions
to the track joints and the user's ability and skill.
Many times we do not want the original motion.
Users might want to feel like the avatar they are imbuing.
Another key advantage of virtual reality is that it allows the user to manipulate large
amount of objects in their vicinity. However in real life our physical spaces are constrained.
In such limited spaces we want our virtual avatars to expand our motion footprint.
We also want to avoid fatigue over long VR sessions.
Thus it is beneficial to move away from this one-to-one mapping.
A common approach that enables us to do this while maintaining embodiment and agency is to trick our
proprioception. Prior research has found that the offset between the user's real and virtual hand
can reach 14 cm before body ownership is reduced and 21 cm before task performance degrades.
This has enabled researchers to delve into haptic retargeting, remapping of the user's
hands in VR and scaling them as well. Building on such systems we propose
CoolMoves a virtual reality system that processes motions from the headset and handheld controllers
to synthesize expressive and elegant full-body motions for the user's avatar in real time
resembling those produced by expert performers while closely following the user's own input.
Note that our task is different from post reconstruction as you
can see in the external view here the user's feet are stationary
but the ones produced by CoolMoves are accentuated based on the user's context.
Utilizing existing motion capture datasets CoolMoves generates the corresponding full body
motion based on the user's input by comparing similar spatial temporal motions with the
nearest neighbor search and interpolating them using a weighted distance metric.
Joint prediction probability is then used to temporarily smooth the synthesized motion
using human motion dynamics as a priori.
Here is an example of the system in action when the user is dribbling a ball
is walking
or is climbing a ladder
From a very sparse search space CoolMoves can map out motions that are spatially coherent and
rotationally invariant even though the exact motion may not even be in the search space.
They are also temporarily adaptive to the user's speed and motion direction
despite limited motion velocities in the search space.
Here are what different motions will look like based on the activity context.
Notice how the foot placement is different for different motions.
This enables a variety of use cases.
For example, using CoolMoves we can generate full body accentuated avatars
or social avatars that encode the affect and emotion of the user in their motions.
CoolMoves can also help combat motion fatigue by scaling the user's motion to expand their
overall footprint while preserving immersion and embodiment. Apart from scaling cool moves can help
in the remapping of limbs. For example the user can make use of their hands to run while sitting
on a chair. This capability can enable interesting accessibility applications for VR in the future.
This wraps up my presentation. Thanks for
watching. Please refer to the paper for full details