Hi, my name is Eric Gonzalez and today I'll be presenting our work,
Sensorimotor Simulation of Redirected Reaching,
using Stochastic Optimal Feedback Control.
With VR, we have this unique opportunity to control how and
where users see their hands.
Since vision please such a major role in our perception,
offsetting their virtual hands can actually influence where they feel
the real hands are in space.
This of course influences then how they reach and move about.
What's really happening here is where it's affecting the users' sensorimotor loop,
which is the process in which we perceive sensory signals, generate movement,
and then generate new sensory signals and so on.
From an interaction perspective, this strategy of sensorimotor remapping
has been used to create phantom sensations of weight, improve interaction ergonomics,
and allow passive props to be dynamically repurposed.
A fundamental technique common to many of these applications is reach redirection,
which is the process of gradually offsetting the user's virtual hand as they perform a reach.
Previous redirection research has mostly focused on exploring applications like the ones we saw before,
as well as evaluating how it impacts user experience,
things like perceptibility, task performance, and comfort.
These provide important guidance for how redirection can be used effectively
to enable new experiences, but don't emphasize the sensorimotor remapping
underlying the whole process.
Last year, we published work that began to look at this,
which involved a simple sensorimotor model of redirection.
Because this model was used in a real-time framework though,
it was optimized for simplicity.
We ignored important characteristics of the human system, such as noise,
and made strong assumptions like assuming reach time was known.
Following this, though, we asked what would be a more biologically plausible redirection model
and what insights might it afford us.
Here, we were interested in capturing things like the integration of vision and proprioception
and natural movement variability.
Why is this simulation like this value, though?
Well, currently, the state of the art for knowing how redirection will affect movement
is by trying it out on real users.
The ability to simulate realistic redirected movement, though,
would help us by tightening the loop between development and evaluation,
as well as supporting new computational methods and reducing the burden of user testing.
The core contribution of this project is thus the simulation of plausible,
realistic, redirected reaches, which can bring us closer to these goals.
We do this by leveraging an optimal feedback control or OFC model,
which researchers have shown is a good approximation for human sensorimotor control.
In the HCI community, there's also been recent interest in using OFC for modeling things
like mouse pointing and gesturing.
Recently, Abtahi et al. highlighted the value of a sensorimotor perspective
for illusory and beyond real VR interactions,
but we haven't yet seen too many detailed implementations embodying this approach.
So with this in mind, in this project, we explore a different model per redirection
capable of factoring in these considerations that we mentioned previously,
specifically by using an infinite horizon,
stochastic optimal feedback control model.
This basically means that we're not specifying an end time for the reach
and we're accounting for sensorimotor uncertainties and noise.
In 2018, Li et al presented a single model of this form
that accurately captured reaching to static and moving targets.
In this work, our objective was to adapt this model to simulate redirected reaching,
validate it against real world data,
and use it to explore how different sensory conditions might affect redirection.
In the context of reaching, we can break down an OFC model into a few key blocks.
There's the dynamics, which tells us how the user's hand would physically move
in response to some motor commands.
Sensory measurement, which tells us how the user is getting information
about their own hands state through things like vision and proprioception.
Within our model of the user's brain, there's an estimator,
which takes that sensory information and generates their best guess of where their hand is.
And a controller, which determines what the right motor command is,
based on that best guess, as well as what their goal or task is.
Overall, OFC just formalizes each of these blocks with some math
and finds the optimal controller and estimator for a given goal.
Following Li's model, we use the position velocity and delayed muscle force
at the hand as the state, commanded force as control,
and vision and proprioception as sensory measurements.
To account for redirection, during simulation, we shift the visual feedback of the hand.
The hand dynamics are modeled as a linear mass damper with proportional motor noise.
Measurement comes from the delayed and noisy signals for a vision and proprioception,
which are then integrated into an optimal estimator.
The cost function here penalizes effort and deviation from the target
without requiring a reach and time.
While this model has many parameters, the relevant free parameters include the motor
and sensory noise scales and the displacement, velocity, and effort penalties of the cost function.
The optimal controller and estimator are found through an iterative algorithm
presented by Li, and we capture redirection by augmenting visual feedback during simulation.
For normal reach, this model produces reaches that match what we'd expect
in reality in terms of its smoothness and variability from reach to reach.
These plots show the individual and average state and control values over 20 simulations.
By checking when the hand is slowing down near the target,
I would get a measure of when the reach ends,
and because we're producing a variety of different reaches,
the simulation produces a range of reach times just as we'd expect from real users.
To validate this model, we collected a dataset of redirected reaches from 12 participants.
In 60% of trials, there was no redirection, so users would generally be reaching like normal.
Importantly, we fit only one model per subject using only data from their non- redirected reaches.
We tested four redirection conditions, horizontal, vertical, gain, and step offsets,
each at the different levels shown here.
Here's what one particular subject's reach trajectories look like for the largest levels in each condition.
Taking a closer look at how these trajectories change over time for this particular subject,
we see the same S-shaped position and Vell-shaped speed curves that we'd expect.
And here are the results of our simulation of the same 10 reaches.
If we overlay these, we see that in terms of the shape and variability of these trajectories,
the model produces consistently good results.
The main difference here is we can see an underestimation of peak hand speed,
which we think could be a result of us using standard values for some of the hand dynamics parameters,
like mass and damping.
We see this kind of agreement across different conditions and participants.
One of the conditions we tested though really highlights the benefits of using an infinite horizon model,
and that's gain redirection.
Gain redirection affects users' reach time, with scaled up movement leading to faster reaches
and scaled down movement leading to slower reaches.
To understand why, let's consider two simple examples.
First, let's consider a user reaching normally.
Based on where they see the target, they plan and execute a movement.
We tend to reach faster for targets that are farther away,
and slower for targets that are closer.
So in general, in normal conditions, we'd expect peak speed to increase with distance.
Now, let's consider reaches to those same targets, except under gain redirection.
So the virtual target always appears the same distance away.
Without redirection, we of course expect to see the same reach profile.
Now, if instead we scale down virtual hand movement during the reach,
initially, the user will move in the same way, since they have no reason to think that anything's
different. But as the reach continues, they have to subconsciously correct for the fact that
they need to reach farther than expected, creating this longer tail on the speed profile.
On the other hand, scaling up virtual hand movement results in faster reach time,
since basically the virtual hand is approaching the target faster than they expected,
so they need to slow down more abruptly creating this shorter tail.
In reality, this is exactly what we saw in the data.
Looking at the averaged trajectories across all participants for the different gain scaling
factors that we tested, we see that the speed profiles tend to follow the same curve until about
halfway through the reach, which is when the difference is emerge. Compared to no redirection,
the scale down hand movement led to these wider profiles of speed while scaled up hand movements
led to shorter tails. If we look at our average simulation results, we see these same trends.
And directly comparing these reach times from the real and simulated data, they have a similar
range of about 0.75 seconds and follow roughly a linear trend. Importantly, this is really the first
time the effects of redirection on reach time have been simulated. Across all redirection types,
simulation tended to produce results that mirrored real world data in terms of their trajectories,
which are shown here. Here, the top row is showing the average reach trajectories across all participants
for each redirection type and level, while the bottom row shows the corresponding simulated trajectories.
Traditional results comparing reach duration and reach accuracy for all conditions,
in details about how we computed model fit, please refer to our paper. Beyond replicating real
world data, one of the biggest values of a simulation is the ability to gain insights about your
system without having to run real world experiments. In this case, we were curious about how visual
noise might affect redirection. And we found that when simulating redirection, more noise tended to
cause the hand to be redirected less, as well as lead to slower reaches. If you think about
closing your eyes as having infinitely noisy vision, this actually makes a lot of sense. No matter
what the visuals are, you would just be reaching forward based on proprioception. And you'd also
likely move slower since you're less certain about the world. One reason this is interesting is
because noise could be a proxy for visual attention. Objects in your peripheral vision are less
precise, so if you're reaching for something while looking elsewhere, our simulations suggest
that redirection might not be as effective. To see if these insights hold in experimental data,
we performed a follow-up study with nine additional participants with a set up of tasks was
identical to the first study, but we also had users visually focused on our target,
those offset from where they were reaching to. To simulate this effect, we introduced a scaling
factor lambda for visual noise in our model. This representative example shows the effective
increasing visual uncertainty on redirection. As the user's focus shifts further and further
from the reach target, we can see that trajectory variance increases, and the virtual hand path is
less likely to reach the target. When we look at the reach trajectories averaged across all participants,
we see this same trend of redirection becoming less effective as the user's gaze shifts further
and further from the target. Additional results and analysis can be found on our paper. Limitations of
our approach include that model optimization takes a few seconds, which makes it difficult to
optimize in real time, like what you might need for a model predictive control. Once you do have a
model though, generating simulated trajectories is much faster. As we showed before, the current model
also tends to underpredict peak hand speed compared to real data. Additionally, by definition,
infinite horizon OFC only optimizes over the steady state, which may be an oversimplification in
this case. And lastly, we're not considering here the non-linearities of the full arms kinematics
and biomechanics in this model. To sum up, in this work, we show that a single SOFC model with
parameters fit only to non-redirected reaches, reproduced reach trajectories across different
redirection and attention conditions. We also showed how these models can give us insight into
the effects of practical sensory uncertainties. In the future, it'd be interesting to look at how a
similar model could capture adaptation to redirection over time, as well as integrate into a broader
computation on redirection framework.