Hello, everyone.
My name is Nisal Menuka Gamage, and today I will present,
so predictable., Continuous 3D hand trajectory Prediction in virtual reality.
This work was done together with the Deepana ishtaweera, Martin Weigel
and Anusha Withana.
We are from University of Sydney, Australia, Honda Research Institute,
Europe and University of Moratuwa, Sri Lanka. In this work
we present a kinematic-based regression model for ballistic hand
movements in VR., which outperforms classical kinematics models.
The model is both user and activity independent,
which makes it usable without any additional training.
And to validate our model we also conducted a follow-up study
with new users and activities. Have u experienced lagging in applications?
It is also a concern in VR, which is our primary motivation
For highly immersive VR experiences, Latency remains a major challenge. When
there is a significant delay in feedback, users
can notice the asynchrony which leads to negative VR experiences.
Our work, builds upon previous work which has explored
various predictive models for reducing latency in VR.
Most predictive models are often limited to end point prediction However,
not only the end of the movement, but predicting a cts movement is important.
For example, Consider a VR user playing boxing with a virtual character. The
collision between the two boxers is not at the end point,
but along the trajectory of the movement.
By anticipating the trajectory, these events can be predicted
and it enables novel applications such as prerendering graphics
and pre-emptive haptics. Now I will move on to describe
how we built our model. Our goal was to develop a formula
to estimate hand location for a future time t,
while minimizing the error between the predicted and actual location
To build our model, We collected data from 20 participants
utilizing the optitrack motion capture software to record the movements
of the upper body of the participants.
We collected data via two tasks.
The structured task contained reaching and pointing
movements; the gaming task included three third party games,
which imitated movements in swordplay, boxing and table tennis.
We first tested our data with classical kinematics models As
seen in this graph, with the classical kinematics models
prediction accuracy decrease rapidly after time
exceeded 160ms. With 15% of the data as training data,,
we applied regression to build personalized models.
The regression model out performed classical models; but having prediction
models for each user, activity and prediction time is not practical
as for our user study only, we needed to maintain over 40000 models.
So we iteratively built a generalized model,
by making the regression user, activity, axis independent,
which did not significantly reduced the performance.
This was our final model,
as this simple equation;
this is similar to classical kinematics models,
the difference being time dependent variable alpha.
And we empirically derived the values for these Coefficients. When compared
to other approaches, our model is easy to compute and transparent.
For more details on how we built
the model, please refer to our paper.
Next I will discuss
how we evaluated the model. For the testing data,
If we predict the hand location
300ms ahead, our RMSE is 3.15cm" And for prediction times
less than 200ms, RMSE is lower than 1cm,
We also note 340ms
as the upper limit for our model,
where model’s 𝑅2 score
decreases below 0.9. And visually we can see
how our prediction model performed.
We further tested
our model for pointing task
only, which showed us that
our model had an angular error 4.0°
from the landing position for 50%
the way through the movement. Similarly, for reaching tasks, it achieved
error of 0.41cm from the landing position for 50% of the way.
Our model is not invincible of large errors,
Large errors can especially happened
for high prediction intervals as 300ms.
However, less than 90% of the errors
for is still under 4cm.
Now I will move on to
how we validated our model
We conducted a followup study with 3users and 2 new tasks;
which included free form sweeping arm movements and a dancing task
where our model showed similar results
party motion capture database which included dancing movements,
Our model can be used to provide timely multimodal feedback, For example,
by predicting future collisions, we can render haptics beforehand.
We also show that our system can be used to reconstruct missing
trajectory segments due to intermittent loss of trackers. Our approach
is not fundamentally limited to VR ,
we can render haptics beforehand.
We also show that our system
can be used to reconstruct
missing trajectory segments due to intermittent loss of trackers. Our
approach is not fundamentally limited
to VR , Maybe in future, using this model, we may be able
stop a rotating blade before you touch it“ To sum up, we
presented a kinematics-based regression model for ballistic hand movements in VR.
which outperformed classical kinematics models We showed Our model is both user &
activity independent, which makes it usable without additional training.
And we validated our model by
conducting a follow-up study with new users and activities. Thank you.