Hi, my name is Sven Mayer I'm gonna present this paper which was accepted to CHI 2020
and the title of the paper is Improving Humans' Ability to Interpret Deictic Gestures and Virtual Reality
and I present this paper on behalf of my co-authors Jens, Robin, Brighten, Valentin, Katri, and Niels
and all together we are from the Carnegie Mellon University, the University of Stuttgart
Hamburg University of Applied Science, Macalester College, and the University of Regensburg.
I started this research topic in a bit different domain than deictic gestures.
and my background here on this topic is mid-air pointing
and over the course of this talk, I'm going to reference back to prior work in
mid-air pointing and how this now relates to deictic gestures.
So first of, midair pointing is used to interact with computing devices already since the 1980s
were Richard Boll proposed a system "put that there"
where a human can interact with appointing armed pose and a voice command with a display
or any other computing device.
So this means that midair pointing is a form of the user interacting with a computing device
but on the other hand when we looking at deictic pointing the system method
how we humans interact with humans.
In detail, deictic gestures are a form of nonverbal communication between humans.
They can either be a substitute or an addition to a verbal explanation
so the main idea with deictic pointing is to communicate an object or place within
the vicinity of conversation partners.
For example, expressions like "look over there"
is mostly followed by a pointing gesture in the direction where the actual place or object is.
This nonverbal expression of deictic gestures can be very powerful
as it can really reduce the words needed to express our reserves in a meaningful matter
However, on the other hand, from the middle pointing domain, we know
that it can be very hard for systems to estimate which place or object the pointer actually meant.
So in a conversation, they might follow-up questions
if it was not clearly conveyed which object of place one actually meant
So now bringing in the idea of virtual reality, we may can help and support conversation partners
by for instance giving them visual clues to convey the actual point
and here one of the early works that actually embrace virtual reality to enhance conversations
is Wong and Gutwin in 2014 presented in CSCW where they analyzed which visual feedback method
is the best to convey these deictic directions expressed by deictic gestures
which then finally resulted in a better human-human interaction
as the conversation partner could understand the referenced place faster and better.
In the latest paper embracing virtual reality, for deictic gestures was
presented last year CHI with the paper type of Warping Deixis
and the idea of the paper is to actually bend the users or the pointers farm in a way that
it better fits the mental model of the observer
which then will result in a better approximation of the direction of the pointer
and you can see this here in this little clip
where from an observer perspective you see the pointer pointing at different targets on the scale
and your task is to estimate where the pointer is actually pointing towards
so estimate the number of on that scale.
But looking closely, here the authors in this paper basically reduced pointing down to
a linear task going on the up and down on the scale in a linear matte.
However, when we compared this task to tasks used in the midair pointing domain
and how these result in error correction functions
we know that the underlying systematic error it's not only up/down but also left/right
We argue, in this paper, that we can support observer even better
when we warp arm not only in one dimension
but also include the horizontal plane warping the arm also left and right of the observer
So to be more precise, in the slide before, I presented how in the deictic gesture domain
linear tasks on an up-and-down scale have been used to study the effects
and to improve the observers' performance.
However, in the midair pointing domain, systematic errors have been shown to occur in both dimensions
up down as well as left and right.
And the clip you can see here is from a task in which we analyzed the underlying systematic errors
when performing mid-air pointing gestures
and I presented these results at CHI 2018 in which we followed up on an earlier paper
showcasing that actually, those systematics errors did not only occur in the real world
but also, more importantly for this paper, they also occur in virtual reality.
Now one might argue that all systematic errors only occur because
a system is actually evaluating where the pointer is pointing towards
and commonly there are different types of recasting techniques
used to estimate the direction of such a mid a pointing gesture
However, when taking a closer look on those results, we see that in fact
all sorts of raycasting technique all follow an underlying systematic error
So, we think it's worth investigating if those systematic errors also would occur in deictic gestures
and if we can correct for those systematic errors in both dimensions and thus improve upon prior work
So to study this in-depth we set up an experiment in virtual reality
where we had two participants in virtual reality at the same time
and the observer was able to see the body posture
and thus, was able to estimate where the pointer is actually aiming towards.
So to include the left-right effect we placed participants in a cylinder with a radius of 4 meters.
and then we displayed targets on the wall of the cylinder only visible to the pointer.
Moreover, if they varied the height in which the targets are displayed on the cylinder walls
as well as the distance between participants
as both our main impact factors on how well the observer can estimate the pointers direction
The ray cast of the pointer as indicated here by a blue dashed line was not to the pointer norther observer
The pointers got no feedback on their performance to not influence their pointing behavior.
However, the raycast of the observer indicated here as a green line was visible to the observer;
however, not to the pointer.
This was done in order for the observer to be precise in his or her estimation
but again feedback on the quality of the estimation was not given to not influence the observer.
So in summary, pointer and observer took turns on aiming and estimating the target's position
over the course of the study we varied the distance between pointer no server and three different setps
as well as the rotation of the target around the pointer in the full 360 degrees
as well as the height positions of the target on the cylinder walls
While this was a within-subject design pointer an observer kept their rolls over the course of the study.
and totally we recruited 14 pairs
with 19 male participants 9 female participants and an average age of 24.5 years.
So now looking at the results for the pointer,
we want to ensure that they're similar to prior work from the mir air pointing domain
as we hypothesized that a pointer in a deictic pointing scenario would
perform gestures in a similar matter as a midair pointing task
and in fact, for the eye index finger raycast we see the exact same of all behavior
that the intersect between the raycast and the wall is a bit to the lower right of the target across all targets.
Now let's look on the results of the observer which are important to improve the observers' performance
when estimating deictic gestures.
First we want to make sure that estimated deictic pointing gestures is in fact two-dimensional
looking at the scatter plot we can clearly say that
estimates are not only shifted on the height axis but also on the rotation axes around the observer
giving us a clear indication that when including both dimensions we will end up with a better estimation.
now that we know we have systematic errors in both dimensions when estimating the pointers directions
we look at the midair pointing domain and how they corrected for systematic errors
and here function which worked quite well for us in our CHI 15 and CHI 18 in paper
to correct those systematic errors is a polynomial function
However, for deictic gestures, we have the rotation around the pointer involved.
We found that rotations cannot be corrected for using a polynomial function.
This is why in this paper rather than fitting a polynomial function we fit actually two sine waves to the data.
One sine wave here is responsible to correct for the rotation error
and the other sine wave is responsible to correct for the height effect
and as a result we know how to warp the arm not only in the up/down direction
but also in the left/right direction to then fit the mental model of the observer better.
so let me run you through how this works in detail
and how this then improves the observers performance
so first here the pointer in blue aims at the target and based on that the observer makes an estimate
This is the uncorrect at version but then we have the corrected version
in which the observer is getting presented that warped arm version of the pointe.
We hypothesized that the observer can estimate the position better than before
and now to confirm this hypothesis and actually validate our model we run a second study
in which we compare the none corrected not warp arm against the Warped arm version
So, as already said the main factor here will be no correction vs with correction
and the second study is split into two parts the first part is a pure validation study
where we only reduced the parameters in half to keep the study lengths manageable
and in the second part we have a more free task which is more game-like
where we ask the pointer to aim at an apple and observer to pick the right apple
validating in a more natural environment.
and for the validation study, we recruited 19 male participants and 5 female participants
with an average age of 23.9 years.
When looking at the results of our validation study and the scatterplot of the
distance between the target and the intersect
or in other words the estimated position of the observer.
We seen an overall trend that with correction performs better than without correction.
Purely speaking in terms of error here in this bar chart and by running an ANOVA here
we can confirm that we have a statistically significant reduction in error with correction
Thus, we can confirm that our model works as expected
and thus this model will help to support our observer in estimating where the pointer actually aims towards.
So, to conclude this talk,
we first discussed how deictic gestures are essential for human-human conversations
we have hypothesized that deictic gestures are in fact two-dimensional
and we showed this in our first study.
Based on the first study, we also developed a model
that can improve the observers' performance on estimating deictic gestures
and with our second study we were able to show that our model works
and we can in fact reduce the error of the observer.
While we position this paper in the context of a human-human interaction,
we can also see how technique can help when interacting with NPC players in a game environment.