Hello everyone, thank you so much for taking the time today to come to my talk. I really
appreciate it. Today I’ll be presenting RotoWrist, our
work on continuous wrist angle tracking for mobile device input. First, I want to thank
my amazing coauthors on the paper Wolf, Eric, Aakar, and Benko.
New computing platforms, such as head-mounted displays for virtual and augmented reality
have the potential to change the way we work, play, and communicate.
More and more, our computing surfaces are not limited to the smartphone screen in our
hands but are placed ubiquitously throughout the environment.
Wearables are great solution for these input devices as they are continuously available,
but most are focused on detecting events, like taps, flicks or other gesture. These
are useful but we need to complement this gestures with precise continuous tracking
for common tasks such as pointing or drawing.
To enable precise tracking, we built RotowWrist. It’s a high precision infrared light-based
tracking technique that goes beyond discrete events and tracks 2-degree-of-freedom relative
angle of the wrist with respect to the forearm using a wristband in real time. Here we show
RotoWrist which can be used to reconstruct hand pose.
Now, I want to focus on the related work on inside-out wrist tracking. There has been
many work in this space, and I have split them into two main categorizes.
tracking wrist movement by leveraging the internal anatomy of the wrist and
tracking using the external contours of the hand.
So in this space, we have work that use EMG signals as shown in Empress or CTRL labs that
can track wrist and finger movements. Other methods include air pressure sensors, EIT
and pressure sensors. IR light has also been used in SenseIR to detect gestures using a
wristband. Other methods include capacitive sensors, ultrasound and bio-acoustic signals.
Now let’s look at work that use the external contours of the hand to track hand movements.
In this space, we have camera-based methods like digits, proximity sensors like WristWhirl,
sonar like FingerIo, and more recently BeamBand showed that ultrasound can be used to track
wrist movement in one dimension. Tactual has also showed that they can use
capacitive sensing to no only track wrist angle but also finger movement. And lastly,
millimeter wave can be used track wrist or hand movements. Now the reason I brough all
the related work was do conduct a comparison between them and show which one might be useful
for the goals that we initially put forward. Setting aside that previous work has not demonstrated
that some of these methods can do continuous tracking, these systems are often very sensitive
to the positioning and potential slippage of the device and therefore often require
per-session training. The question mark here means that the authors of the previous work
did not talk about it. On the other side, external methods typically are more immune
to slippage. Among them, we picked IR light which has been shown to have good formfactor
and power consumption but was not shown that it can continuously track with minimal calibration
between users. Now let’s talk about how RotoWrist works.
The system consists of sensing wristband that incorporated eight time of flight IR modules
and a controller arm band that handles power and communication. Each low power sensor measures
the absolute distance from the sensors to hand.The conical fields of view of each individual
time of flight sensors overlap significantly. As the user moves their wrist, the relative
distance between the hand and each sensor changes which RotoWrist uses to measure the
2 DoF wrist orientation. Our goal here is to enable interaction in a manner that is
subtle, portable, precise and does not require user calibration.
We use a simple tracking algorithm that reliably tracks wrist orientations across different
users. As expected, we observe that the topmost (𝑆2,𝑆3,𝑆4) and bottommost (𝑆6,𝑆7,𝑆8)
sensors have the most linear effect when there is a flexion/extension activity or pitch and
the sensors on the left (𝑆8,𝑆1,𝑆2) and right (𝑆4,𝑆5,𝑆6) sides are most
responsive to radial/ulnar deviation which we call yaw).
We model these observations using these two equations where 𝐷𝑆𝑖 refers to the
distance value that is returned by the sensor 𝑆𝑖 at a certain point in time.
For each of the equations, there is a weight term associated with each of the sensors 𝑊𝑝𝑠𝑖
and 𝑊𝑦𝑠𝑖 which defines the individual weight given to a particular sensor and 𝐵𝑝
and 𝐵𝑦 refer to a bias offset value. We learned these 14 parameters in a user study
and we formulated the problem as a non-linear optimization problem which we solved using
a Levenberg-Marquardt algorithm. To evaluate our system, we recruited 15 participants
and ask them to perform 2, 4-minute sessions. The users were asked to move the wrist in
a random orientation while keeping a fist. Data was captured by both an OptiTrack system
and Roto Wrist. In the beginning of each session, we asked the user to performs 3 up/down and
3/left and right movement with their wrist. We evaluate the tracking accuracy of RotoWrist
in three different categories: 1) cross-user where no calibration is needed, 2) cross-session
where we train on the 1st session of a user and test on the 2nd session from the same
user, and 3) per-session where we use the initial up and down wrist movement of the
user to train our device. Figure on the right, shows the CDF of orientation
accuracy between all three evaluations discussed. The ability to track one’s wrist orientation
reliably and precisely without per-session training enables a wide range of applications.
To highlight RotoWrist’s ability to perform fine-grained continuous tracking (beyond gestural
swipes), we implemented a drawing application driven solely by the user’s wrist orientation.
This free-form drawing can be used to draw shapes for gestures recognition or can be
used for handwriting reconstruction. Here we show a user that used RotoWrist to write
"hello" in the mid air. By combing RotoWrist with arm or wrist tracking
solutions, pointing can be done in absolute coordinate. To demonstrate this, we attached
a Rift-S handheld controller to the top of the controller armband. During a simple calibration
phase, we determine the transformation from the pose of the Rift-S controller to the pose
of the hand. At runtime, we add the wrist angles estimated from RotoWrist to the transformed
pose of the controller to accurately capture the 6-DoF pose of the hand.
To demonstrate the potential of RotoWrist as a pointing device, we demonstrate how it
can be used to play Beat Saber, one of the most popular VR experiences.
In this video, we purposefully had a user play the game using a Rift-S handheld controller
on the right hand while having our device on their left hand so that we can compare
the two input devices. Now you can see even with larger hand movements
and wrist rotations which is a more natural way of playing, our device still performs
well.
RotoWrist is also robust to
most changes in hand pose, for example a user can change their wrist pose from a fist to
open palm. They can also move their individual fingers without effecting the wrist pose estimation.
And with that I would like to thank you for listening and would be happy to take any questions.