Hi, I’m Jacqui Fashimpaur and I’m a researcher at Reality Labs Research.
Today I’ll be presenting our work investigating wrist deflection scrolling techniques for
extended reality, which is an umbrella term for virtual, augmented, and mixed reality.
Heads-up displays like AR glasses have the potential to assist with our chores, work,
and personal lives by making relevant information instantly accessible.
We should be able to provide input to these devices in a way that is similarly low-effort
and seamless with our other tasks, but this may not be possible with existing technology.
Today, the vast majority of XR interactions use either hand-held controllers or mid-air
gestures detected by hand-tracking using cameras in the headset.
These techniques present a trade-off between two potential problems.
First, if you use controllers you encumber the user’s hands in a way that may not be
practical if they need to perform a task with their hands or take their XR device on the go.
But if you want to free the user’s hands by using mid-air gestures, you constrain their
arm posture to somewhere the headset’s cameras can see, usually out in front of them.
This arm-outstretched pose in particular can be fatiguing and socially undesirable, but
every posture is sometimes inaccessible.
For example, you may not be able to hold out your arm in a crowd or point your arm down
while seated.
The friction introduced by these methods undermines some of the potential value of wearing an
always-available display.
Imagine instead that you could instantly and subtly interact with virtual interfaces at
any time with minimal interruption of your primary task.
To realize this vision, there is a need for interaction techniques that are both freehand
and independent of arm posture.
One promising form of input that would have these properties is wrist deflection, the
angle between one’s hand and one’s forearm.
Past research like WristWhirl and RotoWrist has shown that optical sensors on a wristband
are a practical way to measure wrist deflection without encumbering the user’s hands, and
because this value is local to the wrist it should be flexible across arm positions.
This makes it an attractive solution for convenient XR input.
Our paper investigates wrist deflection applied to scrolling in extended reality.
Scrolling is a fundamental interaction primitive that’s received little attention in the
context of XR.
But think about how often you scroll.
Whether it’s navigating a checklist, a document, a newsfeed, a schedule, or an app menu, there
are many day-to-day use cases in which 1-dimensional scrolling and selection are the primary method
of interaction.
So it’s a good place to start evaluating this new input modality.
We had two high-level research questions.
First, “Is wrist deflection effective for XR scrolling interactions?”
and second “What is the best design for wrist deflection scrolling?”
We approach these questions through an in-depth user study examining two proposed wrist deflection
scrolling techniques.
Both of our techniques use wrist extension and flexion as their primary input, a thumb-to-forefinger
pinch gesture to activate scrolling, and a double-pinch to select.
The main difference between them is in the way that the wrist angle maps to scroll movement.
One technique uses something called rate control, where the user’s input dictates the scroll
speed like they would with a joystick; and the other uses position control, where the
user directly shifts the vertical offset like with a scroll wheel or touch screen.
Let’s see them in action.
The rate control technique is called Wrist Joystick.
The user pinches and holds to begin scrolling and releases to stop.
While scrolling, the angle of their wrist determines the speed at which the content
scrolls, with a neutral angle meaning no movement and more extreme angles causing faster movement
in their respective direction.
Our position control technique is called Wrist Drag.
The user pinches to grab the content, and angles their wrist up and down to conceptually
drag the content back and forth.
They can drop it by releasing their pinch.
If they release while their wrist is moving, they can flick the content and let it travel
a greater distance before slowing to a stop with simulated friction.
To learn more about these two techniques, we evaluated them in a 24-person user study
alongside two state-of-the-art VR scrolling techniques.
The first of these is called Controller Joystick, a rate-control technique that uses the deflection
of a joystick on a Meta Touch controller to determine the rate at which the content scrolls,
using the same mapping function that we do with Wrist Joystick.
Then we have Arm Drag, a position-control technique that uses the Quest’s default
hand-tracking raycast to point at the content and drag it when the user pinches, using the
same sliding and friction behavior as Wrist Drag.
We selected these techniques not just because they represent very common approaches to scrolling
in extended reality, but also because they’re close analogues to Wrist Joystick and
Wrist Drag, respectively.
This allowed us to compare not just which techniques performed better, but how users
behaved with each one, and to therefore see the impact of wrist deflection in particular.
Our study included two different kinds of tasks designed to evoke different scrolling
behaviors.
The first task was called Reciprocal Selection, a variant of the Fitts’ reciprocal tapping task.
Participants had to scroll back and forth between two targets in a numbered list as
quickly and accurately as possible.
For each trial we varied both the distance and height of the targets for six total size-distance
pairs.
Here we’re showing the scroll time results for this task.
We’ve split this graph up by the three different target distances.
As you can see, Controller Joystick - shown in red - had the fastest completion times,
at least at the medium and longest distances, while Wrist Joystick was consistently the slowest.
Arm Drag and Wrist Drag were in the middle and pretty comparable to each other.
That’s a really promising result, because it shows Wrist Drag as competitive in the
space of freehand techniques, if not quite at the level of a physical controller.
One possible explanation for this performance comes from the users’ behavior with Arm
Drag and Wrist Drag, which was very similar to each other: quick drag and flick gestures
that didn’t necessarily take advantage of the additional range of movement supported
by Arm Drag.
In contrast, the two Joystick techniques had very different user behavior from each other:
participants using Wrist Joystick smoothly shifted between scroll speeds, gradually accelerating
and decelerating, but with Controller Joystick they quickly jumped to very high speeds and
back again, possibly aided by the physical joystick’s elastic recentering.
So to our research questions, these results suggest that yes, Wrist Drag could be an effective
freehand scrolling technique, and that position control might be a better design for wrist
deflection than rate control, at least in this task.
But that’s not the whole story.
While the Reciprocal Selection task was all about quickly scrolling to a target in a known
location, we also wanted to see how the techniques would perform in a situation more like casual
browsing, in which the user needs to mentally process the content as they scroll through it.
This brings us to the second task in our study, the Counting task, in which participants were
given a grid of 99 photos from the Oxford-IIIT Pet Dataset.
I’m just showing a basic grid here, but imagine each cell has a photo in it.
Most were pictures of dogs, but there were also some cats randomly mixed in.
Participants were asked to scroll to the bottom of the list and count the number of cats as
quickly and accurately as possible.
Here’s how the four techniques performed on the Counting Task in terms of total scroll
time.
Once again the Controller Joystick did very well, but in a reversal from the previous
task it was not Wrist Joystick but Wrist Drag that performed the worst.
It was significantly slower than both joystick techniques.
Meanwhile Wrist Joystick was competitive with both state of the art techniques, indicating
it’s a very promising approach for tasks like this.
And that’s consistent with the user behavior we observed.
Participants using either Controller Joystick or Wrist Joystick followed basically the same
strategy: find the angle that maps to a suitable scroll speed, and then just hold still there
while the images go past.
Meanwhile with the two drag techniques, users had to perform repeated dragging movements
without the added efficiency of flicking the list.
And that’s where Arm Drag’s extra input range may have given it an advantage over
Wrist Drag, since each motion with the former could be larger.
Then, returning to our research questions, these results indicate that Wrist Joystick
could be an effective scrolling technique, and between the two wrist deflection designs,
rate control might be better than position control for this task.
Now you might have noticed at this point that none of our wrist deflection techniques are
outperforming the state-of-the-art.
That’s expected, because their added value is in subtlety, convenience, and flexibility.
But for that reason we wanted to confirm in our study that their good performance wasn’t
limited to an arm-in-front posture.
So, while all four techniques were evaluated first in a seated position with the user’s
elbow on an armrest, we also added one block of each task for just the wrist techniques
that would be performed in a standing position with the user’s arm hanging by their side.
And what we found is that the change in posture had no noticeable effect on behavior or performance.
If we look at the scroll time graphs from before next to the arm-at-side times, we can
see that across both tasks, each wrist technique had very similar scroll times before and after
the posture change, which is a good sign that they are flexible with respect to arm position.
So for that first research question– “Is wrist deflection effective for XR scrolling
interactions?”--it looks like the answer is “yes.”
At least in situations requiring freehand interaction, wrist deflection may be a desirable
alternative to vision-based hand tracking.
In both of our study’s tasks, one of the wrist techniques performed comparably to Arm
Drag in an arm-in-front posture, and both techniques retained their performance in an
arm-at-side posture that Arm Drag can’t support.
This added flexibility could be key to enabling quick selections or extended browsing without
the social or ergonomic costs associated with midair gestures.
Our second question –“What is the best design for wrist deflection scrolling?”– is a little
more complicated.
It looks like context makes a big difference.
Each of our proposed techniques was able to perform one task well, with behavior very
similar to its state-of-the-art counterpart.
But they also each performed relatively poorly on the other task, possibly due to limitations
of wrist deflection input.
And though I couldn’t get into it much here, there was also room for ergonomic improvement
of both techniques.
I do want to highlight one ergonomic figure, though.
These are density plots for each users’ 2D wrist angles over time, across all wrist
techniques, tasks, and arm postures.
As you can see, many participants demonstrated titled or curved paths instead of just flexion
and extension.
But what’s really interesting is how consistent many of those shapes are across different
contexts.
This suggests an opportunity for future work to personalize each user’s input to capture
a larger and more natural range of wrist angles.
Other future work could explore ways to mitigate the issues we observed with each technique,
such as increasing Wrist Drag’s input range, adding haptic feedback to Wrist Joystick,
or designing a hybrid rate control-position control technique to capture the best aspects
of both.
I’d like to thank all my co-authors for their support.
There’s a lot more analysis that I couldn’t cover here, especially about ergonomics, so
be sure to check out our paper if you’re interested, and reach out if you have any questions.
Thank you.