I'm presenting on behalf of Michael because he became
a father recently, so therefore, he cannot be here.
But I'm happy to present for you.
So this work was done with my colleagues
from Ulm University in Germany, and we
tried to combine pseudo-haptics with muscle input.
So let's jump right into it.
So when we look at virtual reality like it is today,
we see that we have very high-resolution displays that
provide lots of details.
We have high virtual fidelity, which
enables convincing 3D graphics.
And we have very sophisticated hardware
in terms of head-mounted displays and controllers,
which are robustly tracked with 360 degrees of freedom
in space.
So having this hardware allows us to--
for novel ways of directly interacting
with virtual objects and virtual environments.
But one of the problems that still
exists in consumer virtual reality
is the haptic feedback is mostly limited to the vibration
of controllers.
So to illustrate the problem, I have a little video.
When we lift virtual objects, it does not
matter if they are light or heavy.
Therefore, grabbing a cup, a table, or even a whole wall
just all feels the same.
So of course, we are not the first to tackle this problem.
We've seen a lot of nice work during the last days
and also in the past.
And all of those approaches can be
categorized into the following three categories.
We have hardware-based feedback--
for example, pulley systems, robots, or exoskeletons
that can provide really nice feedback.
But one of the problems with them
is that they are often special purpose.
They are large and complex, and they are not
really applicable to consumer virtual reality
as we have it right now.
So then we have passive haptic feedback,
which makes use of existing passive objects
and their surroundings, like the project [INAUDIBLE] unit, which
uses the objects inside a room and mapping them
to a virtual environment.
So it has also brought good feedback,
but it's constrained by the available physical properties.
Third, we have pseudo-haptic feedback,
which is very low-cost and flexible because it only
uses virtual displacement and no new hardware.
And that is used to convincingly communicate
friction, stiffness, or direction of forces,
such as the approach that we presented in CHI last year.
So a problem with this redirection
is that it leads to a loss of control.
So what we wanted to create with this project
as a system that does not need any complex hardware
or is constrained by the physical properties, but also
provides more control to the user.
So therefore, when we combined the previously presented to you
haptic feedback approach and added muscle tension
as an additional input channel.
So let's first explain the pseudo-haptic feedback
approach.
So usually, when we grab objects in VR,
they lose their physical properties, have no weight,
and are bound to the track control of the system.
Now, in our approach, the virtual object
retains its weight, so gravitational forces apply,
and object is pulled downwards.
The now existing offset between the object and controller
position is now used to calculate
an offset force, which is pulling
from the object towards the controller.
So this means the heavier the virtual object is,
the larger the offset where both forces are equal get.
Or, in other words, to lift a heavier
object to the same height, we have to lift our arms higher.
So when we now add muscle tension as an input,
we introduce an additional muscle force,
which is pulling also in the direction of the offset force.
And when the muscle tension increases,
the muscle force is scaled accordingly,
effectively reducing the offset and moving the virtual object
towards the controller.
So this approach does not only work for weight.
It's also applicable for resistance of objects.
So when reaching through virtual objects with relaxed muscles,
only the offset force applies.
But when we now tense the muscles, the offset is reduced,
and we are able to apply higher forces to that object.
To implement that concept, we used standard consumer
hardware, namely the Oculus Rift as the virtual reality
headset and the MYO armband from Thalmic Labs
with its EMG sensors.
So this was worn on the upper arm of users measuring
the biceps and triceps of them.
We used to only do the largest factor of the eight
measurements provided and filtered them
for a smooth signal.
And to calculated the applied muscle force,
we used this formula.
Let's take, first, a look at the inner term, where
m is our current measurement.
And min and max are values that we retrieved
during two-point calibration of relaxed and fully tensed
muscles before the experiment.
So this gives us a value between 0 and 1,
which is then multiplied by 110 Newtons, which
is approximately what a human can hold with one arm.
And the offset force, O, which was
introduced by the pseudo-haptic feedbacks approach.
Of course, we wanted to test that system.
And so we conducted a study in a virtual environment
to test our system.
And we choose within-subjects brought with the three
conditions-- state-of-the-art, so no haptic feedback,
pseudo-haptics approach alone and pseudo-haptics combined
with the muscle input.
We had 21 participants, belonging to a pretty much
standard university sample.
And as measurements, we used the E
squared I questionnaire for enjoyment and immersion
and five single item questions.
So within each condition of the experiment,
five tasks had to be fulfilled by the participants.
So the first half was to push a wooden structure that it
creates a bridge for them.
The second task was to crank a big wheel to clear
the path in front of them.
Third task was to push open a double-linked wooden door,
followed by lifting the lid of a heavy chest,
and finally, lifting a golden dragon statue out of the chest.
So let's get to the results.
For every year condition, we had high enjoyment ratings.
And the pseudo-haptic were high-rated as the baseline,
but pseudo-haptics combined with muscle input
was highest ranked.
And the differences between them are significant.
So interesting here is that we also
see the lowest ratings for the pseudo-haptic muscle approach,
and I will come back to that later.
So in terms of immersion, we see no strong variations,
but a tendency for pseudo-haptics
combined with muscle forces to be rated the highest.
So therefore, we argue that this approach
is able to increase immersion.
Now, for the single items questions,
we found that the realism of physical constraints
was influenced most by the introduction of pseudo-haptics.
So both conditions that used pseudo-haptics
are significantly higher rated as the baseline condition
and achieve the same median score.
For the feeling of manipulating real objects, it varied more.
And we found the difference to be significant for all pairs,
with the baseline rated lowest, followed
by pseudo-haptics and pseudo-haptics
combined with muscle force.
The biggest variations that we observed
were for the statement, "I could feel a resistance."
So obviously, we have very low ratings
for the baseline condition that does not
provide any haptic feedback.
But also, the condition with muscle input
is rated significantly higher than pseudo-haptics alone.
So for a statement, "I could influence the behavior
of objects," the median ratings were
equal for the baseline condition and
the pseudo-haptics condition.
But muscle input provides more control to the users,
and therefore, compared to the other two approaches,
it is not completely passive.
So that's probably why it was rated higher here.
So last but not least, we let participants rate
how much they liked the three approaches.
And pseudo-haptics with muscle input was not rated
is higher as pseudo-haptics alone.
And this can be explained because the baseline is
basically having superpowers.
And adding that approach is an additional challenge,
so it hinders users to achieve their goals
as fast as possible.
So beside those quantitative results,
we expect to gain further insights
based on the informal feedback of participants
during the experiment.
So first, tensing the muscles without
a real-world counterpart is something
that not everybody is accustomed to.
And it can feel strange at first.
So before deploying such a method,
you should provide a training for the users.
And also, users did not really have
to lift heavy objects during our study.
Tensing the muscles is still exhausting.
And therefore, it should be used with caution and probably not
for every single interaction but only
when it's helpful and appropriate.
Muscle input introduces an additional challenge.
So while normal reality feels like super powers,
this challenge can be a burden.
And if the user just wants to achieve the goal as fast
as possible-- for example, in the gaming setting--
they won't like it that much.
And this was reflected in the enjoyment and liking
statements.
So in terms of applications, this approach
can be either be defined very realistic
by using a more complex calibration
and an exact mapping of real weight and virtual weights
or real forces and virtual forces.
Or it can be used as a game mechanic,
with different strength levels according to the character
state or character development.
So to conclude, our approach doesn't need complex hardware.
And obviously, it cannot substitute real haptic
feedback.
But using muscle tension as an additional input
is a simple and effective solution
to enhance pseudo-haptic feedback approaches.
We showed that it can increase immersion and enjoyment,
and it was rated more realistic and better
conveys physical resistance of virtual objects.
So muscle tension makes interaction more natural,
because users have to really make an effort
and feel the weight of objects.
So ultimately, it improves pseudo-haptic feedback
by giving users more control within the virtual world.
So this brings me to the end of my talk.
I was very fast, so maybe I skipped something.
But because of that, I'm very happy to take the questions
that still remain.
Thank you.
[APPLAUSE]
So thank you very much.
We have plenty of time for questions.
Hi, I'm Harrison Allen from the MIT.
I was wondering if you've used this in conjunction
with inverse kinematics, because I
know that can also increase immersion
of these kinds of experiences.
Sorry, can you repeat the first?
Yes, so have you used this for maybe a full-armor, full-body,
as well?
Oh, yeah, sure.
Yeah, so now we only used it for the biceps and triceps,
which is good for pushing and lifting objects.
But of course, you can increase to the other muscles.
But then again, it gets more complex.
You need some full-body suits for user.
So what we wanted to do here is really use simple hardware,
a simple solution that can be applied
to consumer virtual reality.
Yeah, thanks.
So I'll ask a question.
You found kind of some mixed results in terms
of people's appreciation.
And I think that some of that was probably due to--
well, at least two factors strike me.
And one that you mentioned-- trying
to get the job done fast.
Yeah.
And so there might have been a speed
slowdown as a result of this.
And the other one was the particular kind of application
and what the goals that the user might have.
And the goals might not have been to use their muscles.
Yeah.
So have you thought more about both the potential
of speeding up the process-- like,
what is making it slow or slow enough for people to notice it?
And then, also, are there other applications
you're looking at where the tensing of the muscles
would be for a purpose other than immersion and enjoyment?
Like an end into itself?
OK, so for the first part, I think
it's not only about speed.
It's also because they really had to make a physical effort.
So it just gets more exhausting over time.
So it's not just being slower, it's
also having to make a greater effort.
And the second part was?
Was if there's other applications where
the point might be to make physical effort so
that the muscle exertion that is maybe
problematic in this application would actually
be one of the objectives, one of the intrinsic goals
of the user--
a workout application would be the obvious one.
Yeah, sure.
Yeah, it just helps to feel the weight of objects
so you actually have to do something to lift objects.
So a workout application would be a nice addition.
Wasn't sure if there was one that
was less obvious beyond that.
[APPLAUSE]
All right.
Any other-- OK, oh there's a couple more.
How about right here and then in the back.
Did you try different placement for muscle measurement?
I mean, now you're using arm, right?
Which kind of displacement?
Placement.
I mean maybe you can place--
measure muscle around hand or--
Yeah, so because we started thinking
about the weight of objects, we started at the biceps
and stayed there.
So we did not look at other muscles.
OK.
So I'm wondering about experiences, you
or the people who try it out for a longer term.
So usually when we have a paper, we have to short-term study.
We get people, and they use it, they go out.
But I think if you look at children, how long it takes
them to sort of lift a cup without sort of spilling,
until they get sort of the understanding of the feedback--
is there something what you as researchers,
by trying it out probably many times before you really
went to the study, is there a learning effect?
Just to speculate on it, is it sort of something
which we should look at not just sort of this one-hour
where people try it, but sort of the longer thing that we may,
if we stay longer, and we have a completely different approach
to haptics than we do if we do the one-hour study?
[MICROPHONE CREAKS]
Sorry.
Yes, I'm not an expert on EMG measurements.
But I think there is a learning effect that you can maybe
try to trick the sensor--
but spending a lot of time in our--
I'm not so sure about the second part of your question.
So sorry.
[CHUCKLES]