- I'm Jackie Yang from Stanford University.
and this is the work I've been done last year
while I was the intern at HP.
So, on behalf of my co-authors, Hiroshi Horli,
Alexander Thayer and Chico Ballagas
who should be here today.
And I present, VR Grabbers, using ungrounded haptic
retargeting by passive haptic feedback for grabbing tools.
Okay, so everyone here, so we all believe virtual reality
is really important, it's good for
training and simulation purposes.
And we all believe that to provide a realistic experience
we need haptic feedback for it.
And however, traditionally, providing haptic feedback
requires hardware with precise actuators.
They add complexity to the system, basically.
And of course, illusion based technique like
haptic retargeting, shows great promises by offering
haptic feedback without adding actuators.
One common strategy is using passive haptic prop
for like in this case there's only one
real cube on the real desk.
But the system tricked user to to think that they
are manipulating multiple cubes in virtual reality.
However, this does not apply to tool and requires
many different haptic props for object
of different shapes and sizes.
So, our problem is, how do we apply haptic illusion
based technique to grabbing tools?
So, in this paper we focus on a specific type
of grabbing tools, which is, sorry, some specific
type of handheld tools called grabbing tools,
Those kind of tools usually have two handles
and then is used to grab object.
Including tweezers, pilers and chopsticks.
Of course, some other tools they are not grabbing tools
but they share some similar characteristics
as grabbing tools.
For like, scissors, snips and wirestrippers.
They also have two handles and the majority of
force feedback while operating a tool is
between two of the handles.
So therefore, ungrounded.
And out technique, also be used to represent
part of the force field act for them.
Okay, so we introduce this controller,
called VR grabbers, which is, sorry
(background chatter)
(laughter)
So this VR grabber has a similar shape while with
training chopsticks but is pure passive and
to provide a happy feedback for VR grabbers we
created this technique called
ungrounded haptic retargeting.
It averages the visual dominance, our visual perception,
to create a haptic illusion where a
virtual object feels physically present.
So, as you can see in the video, in virtual reality it
appear that the controller arm is
converging on a virtual object.
But in real life, the controller just
converging with each other.
So, this technique averages the ungrounded feedback
adhering to the grabbing tools.
The user attribute, the force of the
opposing arm to the force that they
are extracting from the virtual object
in between their virtual bar,
the virtual object is in between the
virtual controller in the virtual environment.
So, we basically achieve this by dynamically
adjusting the converging rate of the
controller arm in virtual reality.
So, now we can go into details about the system.
So, what's so special about his ungrounded
haptic retargeting technology?
So, the first thing is our technology can accommodate
our object with different shape and sizes.
As you can see, the user can grab an object,
sometimes a cubical object, sometimes a spherical object.
They have different shape and sizes, but the
technique always accommodated for all of them.
Also, it can adapt to user change on intents.
The user my close their controller completely sometimes
and they also may want to grab on some object sometimes.
And the extreme cases,
the user also may close their controller
halfway through on an object, then they change their mind
and then they go somewhere else and then to
close their controller completely.
And none of those reaction is scripted.
So we don't have to ask the user
to pick up a specific controller.
They can use this controller to explore
interactive environment.
So to solve those few characteristic
of ungrounded haptic retargeting.
We introduced this algorithm called key state mapping.
Where we defined three concept.
The first concept called state, which is
the opening angle of the controller
in both physical world and virtual world.
In this case, as you can see,
this is the, where the, where it looks like
in the virtual world this is the exact
opening end of the controller where as in
the real world it's closed 50%.
And key state, is the fuel that's defined
by the tool controller that is necessary
to maintain the illusion.
For like, in this case, two key state are defined,
one is closed, the other is open.
As here in the name, open means when the controller
is completely open and the controller should be both
completely open in virtual reality and in reality
to maintain the illusion.
Where as when the controller is completely
closed in the real life, it should be
just on the edge of object in virtual reality to
provide those kind of forced feedback.
So, then this is the, kind of an interesting part
where we basically generate all the other state,
which we call transfer state,
according to the current state
and the key state.
For like, in this case you see if
the user close their controller 25% more,
then they should be reached at this
specific transfer state.
And if they open their controller 25% more in real life
it should it should reach this transfer state.
This is, we said, if the controller, if the object
becomes larger or smaller what happens just that the
right users controller converge in the virtual reality
will change to represent force feedback.
So no haptics, no acute haptics is required,
but we can emulate different object shape and sizes.
So, now lets talk about change of intent.
What if the user then move away and then
the object's no longer in between their controller?
So we can do the same thing, whichever the key state
and note that because there's no longer object,
the closed key state change from the
end of the object to completely closed.
So, with this we can just generate a transfer state,
similar as before and what we will have is basically,
if the user open the chopstick you will
behave exactly the same as before.
But only when the user is closing their chopsticks,
their virtual chopsticks will move a little bit
faster in real life, so that when they
completely close it, it's also
completely closed in the real life.
Sorry, it's also completely closed in the virtual world.
To represent the forced feedback of
completely closed controller.
For more details, please see our paper.
Okay, let's talk about implementation of this.
So, this is the VR grabber controller,
which we are passing around.
I didn't attach the VR tripod today,
but that is used to track the position orientation
of the controller in virtual reality and in real life.
A potentiometer is attached to track the
opening angle of the tool in real world.
So that we can represent some, some transferred version
of it in the real life, in the virtual world.
And as all the grabbing tools, we have
two handles and one joint.
On the right, we show the dimension of this controller,
so basically, the handle is around 11 centimeters,
while the maximum opening angle is around 38 degrees.
Note that we have this special
(spoken off mic)
(indistinct speaking off mic)
we put it there to limit the minimum opening angle so that
we closed the controller we can adjust
within a five degree polarity.
We can make the controller close one degree,
five. six degree and none degree and so on.
Okay, so the first state we conducted is to
explore the limit of the illusion,
basically how much we slow down and speed up the handle.
We basically used that haptic block and then we have
basically five positions off that haptic block.
And we put that haptic block in a specific position
in both virtual reality and in reality,
because we have five position we have
like, total of, number of 25 configuration.
Each user tried those 25 configuration in a random order
and in each one for the trial we ask user
whether they noticed there's any retargeting
which means like, whether do they feel like
they are holding a different controller
in real life than in virtual reality?
Yeah, so this is what a user is doing.
So a user is basically,
close their controller multiple times,
then we asked question, whether they feel like
there is any retargeting happening.
And the result is, we, so we used this detection
threshold metrics, which essentially means
50% of the people think there is some retargeting happening.
And the result is that, we can change the control
display ratio in the range of zero point seven one
to one point seven seven times.
What does that mean?
That basically means that, if this is the controller
if the true state here is basically the
completely closed state and the completely open state.
If the user moves halfway,
from the open state to the closed state,
we can render the virtual chopsticks,
either in the zero point seven axis motion or
one point seven seven axis times
the original motion, like shown in those red boxes.
Okay, so then we think, then we think a little bit more,
we don't necessarily have to make the tool
completely close in the real life
to represent the closed chopsticks.
Right?
Our goal is, as you can see in the previous slides,
this technique works both in the passive state,
we can speed up users closing of the chopsticks
or we can slow down users closing of chopsticks.
So, we think, we do have to make the user completely
close the chopsticks, we can put the block in a specific
position where we can emulate the largest
object possible when the user close their chopsticks,
they can still feel like their chopsticks closed.
So we decided to put the haptic block in position too,
where we can emulate virtual objects size less than,
almost less than three centimeters.
And for the next study we measure task performance
and gather user's feedback.
So, one problem is we need to even mend
a VR grabber without ungrounded haptic retargeting.
The way we're using that is inspired by primal work
and common approaches in state of the art VR games.
We do that by make the virtual opening angle
and the real opening angle always the same.
Until they actually grab on the object.
After they grab on the object, the opening angle
no longer changes so that the handle stays on the
surface of the object.
Where at this state,
the tool controller hasn't converged yet.
So you can still move your controller around while the
virtual angle will no longer change.
So a designers task, where the user first align
their controller, the virtual controller to start the timer,
and then they pick up a moveable cube.
They align this moveable cube to a target cube and
once they release in the correct position and orientation,
that's when the task end.
We ask the user to try those two configuration
in a balanced order with these objects.
For each configuration we ask user to try
36 performance task, as I have said before
and the object will be rendered in different sizes.
We will ask them to do different movements.
And after each configuration, we ask user to fill in
this presence questionnaire.
So the result of this study basically shows that
the controller with ungrounded haptic retargeting
can reduce the task completion time,
basically increase the performance by almost two seconds.
And in the presence questionnaire we
also show higher realism.
Okay, so what's the limitation of this technique?
So, currently we only implement
ungrounded haptic retargeting
with the single controller VG grabber.
And the other limitation is, currently the controller
can only work for objects smaller than
two point nine five centimeters.
So, in the future we could either
apply this similar technique to other controllers,
other than just VR grabbers.
We can of course apply them to grabbing tools,
we're pretty confident about that.
We can even apply those to like,
this specific controller which is called
a low array by Choi et al
which is basically, they have an active break,
each one for your finger,
so that if you grab a virtual object,
you will stop your finger at where your finger touches
the virtual object.
We could potentially implement a passive version of that,
basically, your finger always stop at the same position
and we use ungrounded haptic retargeting to make you feel
like you are holding object of different shape and sizes.
On the right we can, in the other future work could be
we can use some active mechanism to increase
the working range of this tool.
So, for like on the right, we show one of the
designs mentioned in tweezers device.
This design basically is really active, really cheap,
but the only problem is I can only
stop at discrete point.
And also it has a lot of breaking force,
which is ideal for simulating hard object.
And because it can only stop at discrete point
we can potentially use ungrounded haptic retargeting
to fill in the gaps, basically the controller stop
at the nearest point and then we use
ungrounded haptic retargeting to emulate the differences.
So that basically, we can emulate object with larger sizes.
So what's the take away of this talk?
So we think in the future,
every consumer do not need to wear
expensive haptic equipment or prepare a variety of
haptic props for, to have a realistic haptic experience.
With ungrounded haptic retargeting everyone can
enjoy quality haptics at a lower cost.
Thanks.
Happy to take any questions.
(audience clapping)
- [Announcer] Thank you Jackie. Any questions?
Alright, it's me again.
So, during your summer internship with us
at Microsoft Research, you got a chance to try our
various controllers, normal touch the CLAW and so on,
which allow you to grab virtual objects
and of course we do this actively
by breaking your fingers down, then breaking them right?
- Right
- [Announcer] And acquiring the objects.
And I'm wondering did you iterate through
the various controllers or did you
start with chopsticks right away?
Because one component you didn't really talk about is
sort of the extension of chopsticks right?
How long are they?
How close are they to your finger?
I mean you touched a little bit on that in the
future work part, talk about Wolverine right?
- Right
- [Announcer] Could you integrate this totally
in your fingers and just do that
or is there an effect of the length of the chopsticks?
- So, the reason why we focus on chopsticks initially
is we want to implement a controller that can
basically manipulate virtual object
easily in virtual reality.
With, what we consider about pilers and in tweezers
that usually they have a very limited working range,
where chopsticks usually have
a larger opening angle than them.
As for extending for like maybe longer handle chopsticks,
or shorter handle chopsticks, we try to,
when we measured our limitation which
always try to measure the control display ratio.
Which we believe is like a little bit more generalizable.
So you can imagine if we have longer handle chopsticks,
we can still apply the similar control display ratio,
like this measurement, for like if you are
completely open chopsticks,
right now it's around five centimeters.
If the completely open chopsticks is ten centimeters
you could potentially emulate a six centimeter object.
That's what we imagine.
- [Announcer] Okay. Cool. One more question?
If not, we're gonna thank Jackie again.
(audience clapping)
- [Announcer] That was a perfect Segway...