- My name is Mauricio,
and I will be presenting a new technique
to improve perception of pointing gestures,
for collaborative mixed reality environments.
We call this technique Warping Deixis.
I will start by discussing Deixis,
and then I will talk about warping.
So, Deixis, or deictic gestures,
are a natural way to communicate
objects, locations and directions, that is.
Deixis is the gesture of pointing,
to communicate the targets to others,
and can be followed by words or not.
For example, the one over there.
Pointing gestures are very important in communication,
because they promote sharing of information;
with a simple gesture we can identify objects
without going into long descriptions.
And this is why this kind of nonverbal communication
is the subject of study by the collaboration community.
Being able to do pointing gestures improves collaboration,
and has a great impact on task performance,
when it is necessary to quickly identify objects,
that are close.
However, there are two types of pointing gestures.
Pointing to objects that are close to us,
proximal pointing, there is no issue here,
because we can almost touch the targeted objects.
That is not the case for distant objects,
the gesture of pointing to distant objects,
is hard to interpret.
Also, it can cause misunderstandings.
A recent study by Herbort and Kunde,
suggests that people systematically misunderstand
pointing gestures.
Because people performing pointing gestures,
and interpret them in different ways.
Pointers seem to place the index finger on, or close
to a line connecting the eye, and the reference.
People do this to have a greater precision
indicating the target.
Just like using a handgun.
However, to interpret the gesture, and determine the target,
observers interpret pointing gestures
by extrapolating the target, using a factor
defined by the arm and the index finger,
which at the distance, will be something
completely different.
So, if someone wants to communicate the target A,
it's a little bit annoying that the other person
realizes B as being the target.
Actually, Herbort and Kunde have determined
that people are not very good
at doing linear extrapolations.
And so, people usually perceive something
between target B, or C.
And this is still not the target,
this is not the original destination
that the pointer was intended to.
This position is also possible to calculate,
because Herbort and Kunde contributed the mathematical model
to determine with some certainty,
the place people think the target is.
So please see the paper for more details about
this predictive model.
So, the implication of this is as follows.
If this confusing stuff happens in the real world,
this interpretation problem is aggravated in
virtual and augmented reality collaborative environments,
that should benefit from nonverbal communication,
because virtual presentations of people
try to simulate real life behavior.
In this work, we focus on improving pointing perception
in mixed reality environments.
In virtual environments, peoples' representations
can be avatars, or 3D models,
emulated by someone's movements.
Or complete 3D captures using point clouds.
All these representations allow for good communication
of nonverbal cues.
Yet, pointing gestures remain ambiguous.
For this reason, previous approaches relied on
visual indicators such as laser beams, spotlights,
stretching the pointing arm,
even highlighting an entire target object.
But it's also known that these approaches
can clutter the environment with artifacts,
that can cause occlusions.
They also create confusion about who owns the laser,
or the spotlight.
And this is why the main objective of this work
is to go a step back, and improve the perception
of pointing gestures.
But in a natural and transparent way,
attacking first the root of the problem.
So, the approach that we propose is warping, that is,
deforming the arm that's doing the pointing gesture,
to the benefit of the person interpreting that gesture.
Our thesis is that manipulating the pointing arm,
perception can be improved in
mixed reality collaborative environments.
So, if without any manipulation,
people do not realize that this guy is pointing
to that target, because they are looking at
a completely different location.
We just move the arm to the right place,
for the people to observe the correct target.
Now, I'm going into more details.
First, Warping Deixis consists of,
identifying the target
that the pointer intends to communicate,
then determine what observers may interpret,
not using a straight line but a curve,
given by Herbort and Kunde's predictive model.
And then move the pointing arm
not towards the final target A,
but in direction of another target, the target D,
that its interpretation will make people realize
the original target A.
So, for this work, we draw inspiration
from the haptic retargeting body warping technique,
from Azmandian and colleagues,
whereby manipulating the representation of an arm,
a single physical prop can provide the semantics
for multiple virtual objects.
So, to detail our body warping approach,
we need to go to the next step,
which is deforming the arm of the pointing person,
in that credible fashion.
And to do this, we propose a rotation of the arm
around the shoulder.
So for the skeleton base of the representations,
like avatars,
we already know which points need to be updated.
However, in point cloud based virtual representations,
and intermediary step must be made,
consisting of interpellating a series of points
in the middle of the skeleton joints,
and using the radius around those points,
we can determine the volume containing all points
of the arm's surface.
So now we know all the points representing an arm,
and we can manipulate them at will.
So, to evaluate our approach,
we conducted an evaluation with 18 people,
who participated in pairs in a virtual reality environment.
For this, we referenced in virtual reality,
the original Herbort and Kunde's methodology,
used to determine how people interpret targets,
on a numbered pole, with three meters of height,
and at three different distances,
one, two, and three meters.
We employed the same task,
in the virtual reality environment,
and replicated the numbered pole with targets.
Each target was a square measuring eight centimeters,
and people were side by side at the distance of two meters.
The experimental setup was comprised
of two connected depth cameras.
To capture the entire body of the participant,
point clouds, point cloud representations
in skeleton models.
We also used two Oculus Rift goggles
for participants to be immersed in the virtual environment.
The prototype was built on top of our CreepyTracker
toolkit.
(applause)
That expedites the prototyping of these kinds of experiences
using skeleton tracking with multiple depth cameras,
and also facilitates the capturing of virtual
representations of people.
Participants alternatively took on the role
of a pointing person, and an interpreter, the observer,
and they have different views of what was going on
on the pole.
The pointing person could point normally,
and was not able to see the effects of
the deformation on his arm,
and he was presented with the pole with blank squares,
and one green target.
The job of the observer was to identify that target
that the person was pointing was trying to communicate.
It was the same pole, but observers would see
numbered squares, with the numbers randomly jumbled.
Similarly to Herbort and Kunde, we evaluated this
interpretation task with the pole
at one, two and three meters,
in two different conditions.
We evaluated Warping Deixis against normal pointing,
without any manipulation whatsoever,
and we evaluated the two conditions
in regards to task performance, and user preferences.
In terms of task performance, the interpreted targets
were reported out loud by the observers,
and were recorded by the evaluation moderator.
We also used a user preferences questionnaire
to determine if participants noticed any difference
between the two conditions.
We then calculated the error distance
between the correct targets, and the targets reported
by the observers, and we obtained the mean distance
of the errors for each condition
on the three pole distances.
Results from the Two-Way Repeated Measures ANOVA
showed that all effects were statistically significant.
Overall, the main effect suggested that our approach
was superior in task performance,
although performances in both conditions
declined with distance.
A Paired T-Test showed significant difference
between one and two meters, one and three meters,
and two and three meters.
The interaction effect between distance and technique
was also significant.
Knowing that the Two-Way ANOVA
demonstrated significant differences,
we applied the Post-hoc Paired T-Test for each distance,
and we found that at the distance of one meter,
the two conditions were essentially the same, that is,
the target is so close that there is little ambiguity
and warping had no effect.
Yet, our approach demonstrated a significant
error decrease.
At two meters, a decrease of approximately one square,
and at three meters too.
We can then conclude that our approach
reduces ambiguity, and improves the interpretation
of the gesture.
In regards to user preferences,
we asked the participants to rate their level of presence,
and their colleague's level of presence,
on a 6-point Likert Scale.
We also asked them if they felt that they were pointing
to where they wanted to point,
and if it was easy to see where their partner was pointing.
Results from the Wilcoxon-Signed Ranks Test
showed no statistical differences,
suggesting that the participants felt no difference
from one condition to another.
We also asked an open question to see if anyone had noticed
anything strange with their partner's representation.
From this question, we had no concrete response
related to our manipulations,
and therefore we can conclude that in this evaluation,
warping was not perceivable.
So, to summarize the results,
our approach reduced perception errors,
and it was not noticeable.
We consider Warping Deixis as a starting point
to improve perception of pointing gestures,
in collaborative environments,
but our approach also contributes to other areas,
in particular, in collaboration between robots and humans.
An example of this is our humanoid guiding robots
that can help people,
and even non-player characters in entertainment
virtual environments.
So, concluding, people are not very good
at interpreting pointing gestures.
But pointing is really important,
when we don't want to waste too much time
on long descriptions.
That's why we propose Warping Deixis,
to reduce perception misunderstandings.
As a side note, I'm finishing my PhD,
and I'm actively looking for a postdoc position,
so if you find this work interesting,
please feel free to contact me.
With that, I conclude my talk,
thank you very much,
I hope I am just right to have some minutes for questions,
thank you.
(applause)
- [Announcer] So, we have a speaker over there,
if you have any questions, please go ahead,
and please say your name and operations
before you ask the question.
- Nice talk, thank you for, um--
- [Karthik] So, my question,
I'm Karthik Ramani from Purdue,
so my question is related to,
sort of maybe having a virtual pointer on your hand,
so the ambiguity is directly removed in terms of
feedback to the other participants.
Did you think of that approach,
because here you're sort of using a lot of geometry to,
sort of extrapolate and figure out and so on,
but if you had a virtual pointer, or something like that,
there's no ambiguity, in terms of what you have to resolve,
and the user will change his arm position
to get it right.
- There's no ambiguity, but pointers--
The focus was to go a step further,
and first correct the arm.
This is not incompatible with pointers.
With those kinds of gestures, those pointers
following the eye index factor,
it's weird coming out of the end.
More than that, there's occlusion issues,
that I've talked about,
and with environments with multiple people,
it's difficult to understand who's the owner
of those kinds of pointers.
- [Karthik] I was sort of thinking of,
emanating from your hand itself, and going there.
- Okay, yeah.
- [Announcer] Okay, we may have time for one more question.
- [Audience Member] Okay, so actually I have a question,
so, have you considered that the moving target,
particularly pointing to a moving target,
will that make some difference?
- Yes, I've considered moving targets, yes,
but we've pushed that to future work.
- [Audience Member] Great, okay, thank you!
- Thank you.