- Thank you Michael, for the nice introduction.
So as said so, my name is Teresa,
I'm from Ulm University and I'm going to talk about
our design space for gaze interaction
on head-mounted displays.
Future AR and VR HMDs are expected
to have eye-tracking as an integral
part of the device.
As you can see on the slide there are already
a few examples of devices that have eye-tracking integrated.
And the question is, why do we wanna have
eye-tracking with HMDs?
What can we use it for and what are the benefits?
So in our literature review we've found
two basic areas where eye-tracking is applied for HMDs,
and the first one is eye tracking
as an additional explicit interaction modality.
So we can use eye tracking additionally
to a different interaction modality,
like a controller or gesture.
And one example was presented at
at last year's CHI by Kyto et al.
where gaze was used to indicate where
the user wants to interact, and then the action
was refined using the controller.
The second part, or the second area
where eye tracking is useful for HMDs
is that eye tracking has the potential
to address unique challenges of HMDs.
The most prominent example for this is
foveated rendering.
Meaning that we adapt the content or display
to the visual perception.
So we only render the part actually where
the user is actually looking at, and blur out the rest.
So one example here was presented by Patney et al.
And you can see the part where, where the yellow circle is,
where the user's looking at
and the part around that is blurred out.
So what we ask ourself, is that all?
And these two examples I just showed point towards
the beneficial combination of watching HMDs
and the solutions that have been proposed
are more or less being classifiable into those two
examples I just presented.
We found that these two examples are only
examples of a much bigger design space
which hasn't yet been explored.
So, the question is, why hasn't it yet been explored?
Why haven't we found more solutions?
And our assumption is that we assume
that it's related to the viewpoint that eye-tracking
researchers or eye-based interaction researchers
had or have on games.
Most probably, this relates to our main interaction devices
which are still the smartphones, smart watch,
and ambient displays.
And in this case, gaze is treated as a 2D construction,
a two dimensional screen.
However, with these new devices, with these new HMDs,
we are able to create whole new three dimensional works
or augment the existing 3D world with information
at different depth levels.
So, we should consider gaze also as a 3D construct,
somewhere between the physical and the virtual worlds
we create with those devices.
So how should we think about gaze interaction
on HMDs to explore further possible solutions?
And a design space has been shown to be a suitable
tool in HCI to systematically analyze the
combination of new technologies,
and in our case we therefore created a design space
to analyze the combination of HMDs and gaze interaction.
And our design space is intentionally directed to its
non expert content creators, so we want to give
a basic framework on how to think about
those two technologies,
how to think about gaze interaction and HMDs.
For everybody, so to say, so you don't have to have
an expert background in HMD technology or in eye-tracking.
So in the following, I'm going to present the design space.
It was created using a morphological analysis,
however, I will skip the method part,
like the more detailed parts, so please have a look
at the paper if you're interested in the method
we came up with.
And basically, our design space is made up by
two dimensions which are technical properties of HMDs
and properties of human depth perception,
which are the two components that establish the
interactive system, right?
So we have the HMD on the one hand side
and then we have the human part on the other hand.
Starting with the first dimension,
which is technical properties of HMDs.
We, so this is basically what kind of parameters
can we find in literature to describe HMDs?
And we found three main parameters,
which are display type, device type, and world knowledge,
which we found by literature review of classifications
and taxonomies of HMDs.
So, we have device type, which is split into AR and VR.
Then we have display type, which is split
into monoscopic and stereoscopic displays.
And we have world knowledge, which is split into
full and none world knowledge.
And the third parameter is based on Milgram et al.'s,
"A Definition of the Extent of World Knowledge",
which is the continuum.
However, we reduced it here due to reduced complexity
and to make the design space more applicable
to the only two values.
So, in our case none of course means not having
any information about the world,
and full world knowledge means in VR
that we have always, in VR we have
always full world knowledge
because we created the world we designed.
And in AR, full world knowledge refers to having
a 3D mesh of the environment and information
about the augmented data which is put on top to it.
So, if we now combine all the values of the parameters,
we get eight rows for the first dimension.
The next dimension, or the second dimension
is properties of human depth perception.
And here we were interested in how we can actually
measure where a person's looking at in space.
So, what's the physical response our eyes
actually do if we look at something in space,
and how can we measure that?
And basically, we have two ocular motor depth cues.
Which are vergence and accommodation.
Vergence hereby describes the simultaneous inward
rotation of the eyes to locate a certain object
of interest on the fovea, and accommodation
refers to the alteration of the lens to keep
that object of interest on the fovea.
When we measure that, what do we get?
On the left hand side we get, for vergence,
we get a 3D gaze point somewhere in space.
Basically, this is is intersecting the two gaze phrase.
And for this we need both eyes, so this is binocular
gaze cue, um, depth cue.
And for accommodation, it's enough to measure
only one eye, therefore it's a monocular depth cue,
because for healthy eyes, the lens in both eyes
alternate equally if you focus or accommodate
to something in depth.
So, on the left side we get a 3D gaze point
and on the right side we get a gaze depth plane.
Taken together or taking those, putting those
into parameters, we have oculomotor depth cue
and ocularity.
Oculomotor depth cue is split into vergence
and accommodation and ocularity is split into
monocular and binocular.
Again, if we now combine all the values we get four columns
which make up the second dimension.
Now, if we take everything together we get this
for dimension one and dimension two,
and yes, as you can see it's already a bit complex,
so what we did is we reduced the parameters
to binary values.
We mainly, just because we wanted to make the design space
applicable in practices,
but if you want to use the design space
for your research you are very welcome
to define your own parameters or additional parameters
or additional values, whatever you might be interested in.
So, our main contribution is actually
to offer the approach and the guideline
to how can we think about this problem,
and not the concrete content.
Okay, so if we combine now everything together
we get a four times eight matrix.
And now the next step of our morphological analysis
is to delete the cells that don't make sense
or are not relevant for our definition of the problem space.
And in our case, monoscopic VR headset for example
doesn't make sense in this problem space
because then you would have like half virtual world
and half real world.
So if you delete those cells we get
a three times five matrix which is the final design space.
And now to the more practical part,
so how do we actually use this design space?
What can we do with it?
And we propose two usage techniques,
where the first one, the classification technique
is the top down approach and the second one,
the ideation tool is a bottom up approach.
The first one focuses on the content
that can be put into the space.
It can be classified into, into the space.
And in the following I'm gonna show some examples.
Starting with the whole lens, with an additional
binocular eye tracking add on.
So we could put this,
on the human part we could put this
into binocular vergence measurement
and then in the HMD part we can put this
in the cell of a stereoscopic AR display
having full world knowledge, in our definition.
We can also put other prototypes in there,
for example, EyeVR which was presented by Geiselhart et al.
This is a VR headset with a monocular eye tracker.
And then we could also, for example, put Koulieris et al.'s
work into the design space where they attached
an autorefractor to the VR,
which means its a device that can measure accommodation.
So what I wanna show here is we can of course
put a lot of examples in here, and we can also
put different kinds of content in there.
Here, I only presented devices, but we could also use
applications or interaction possibilities.
And what I wanna show is that we can use the design space
to indicate promising families of solutions
by heavily populated cells,
but also we can also identify possible lack of solutions
by empty cells or a cluster of empty cells.
So this was the first technique.
The second technique is a more bottom up approach.
It is, we called it ideation tool.
And here again, to remind you, we treat gaze
as a 3D construct between the physical and virtual world.
So what I'm going to show you is, are some examples,
and on the left side I'm going to show the design space,
on the right side I'm going to show the concept
I'm talking about.
Starting with vergence.
So how can we actually measure 3D gaze?
As I said before, we could for example
use binocular eye tracking and then we would get
3D gaze point somewhere in space.
So now you can see on the right side,
if we would use 2D eye tracking then the upper
and the image on the bottom would result in the same
two dimensional gaze point on the plane.
But, in space these are really different.
So binocular vergence measurement adds depth value
to the gaze point.
Now if we take the left side now into consideration
and in this example we took a stereoscopic AR display
with full world knowledge, again, which could be
the HoloLens, for example.
What we get is we get information about the physical
and the virtual content now.
So now if we look at the intersection of both the column
and the row, what we get is that we can differ
whether the person is looking at virtual content
or looking at physical content, for example.
And one example of what you could do with it is,
we can fade out for example the virtual content
once we realize the person is looking at something physical
and then if we realize that a person wants to interact
with the virtual content again, we display it
and we fade it in again.
So we can switch between those two,
and obviously the idea is that the augmented content
is attached to a wall or floating in between
but not really adapted to the point where
the user's actually looking at.
Another example could be that we could not only
display virtual content in front of a physical object
but also behind a physical object,
like a wall for example.
And one example we implemented is,
what we ask ourself is, could we not display
something behind or something the user is not able to see
in the real world, so can we create a system
that allows the user kind of to exceed physical limitations
and to kind of look through walls,
just glance through the wall?
And of course, per se, this is not possible.
Because it's also not possible to look at something
you cannot see, so you cannot focus on empty space
or behind something.
But what we did is we displayed some virtual content,
some gaze pattern, gaze guiding pattern
to enable a user to actually perform eye movements
they wouldn't be able without this pattern.
And in fact we can guide the user to actually look
behind the wall, so I'm going to show you a video.
We call this application x-ray vision.
So now the person is looking at the wall and you see
the pattern.
Now, if the person focuses behind the wall
you can see content that lies behind the wall.
In our case it was a beach.
And then when a person converges the eyes again,
the content disappears and the person sees the wall,
and now the content behind the wall again.
So, what you saw here, these gaze patterns
which lie both in front of and behind the wall,
and we are able to display this because we're using
a stereoscopic AR display.
So we can now kind of allow the user to do something
to exceed physical boundaries.
Okay, the example I showed was vergence space,
but we could also imagine that at some point
we could measure accommodation.
I mean, we can already measure accommodation
but not in this place.
So what we would get is now the same
but with a depth layer and not a point in space.
So now we can add again information about
physical or virtual content.
If you look at the intersection, we can again differ
between real and virtual content.
And now what our design space does
is now we can actually see alternative solutions
for the same interaction possibility.
So of course I've only shown examples now,
so please refer to the paper.
There are a lot of more examples in it.
What we did, we provided a design space
intended to serve as a basic framework
to guide people into how to think
about gaze interaction HMDs.
I presented two techniques here,
one to classify content, one to derive new ideas
and new combinations.
In the paper there's a little more use,
about the usage techniques and also about
different content types and different perspectives,
so, thank you very much.
Happy to take your questions now.
(audience applauding)
- [Moderator] So the same procedure as before.
We have microphones down here, one, two, three, four.
I'm going to ask you one.
- Mm-hm?
- [Moderator] I was attending the workshop
on social head-mounted displays and missing you there
in the sense that it would have
been great to have you there.
- (laughing) Thank you.
- [Moderator] Because we proposed the one thing
that you eliminated from your design space,
which is a very eye prototype of a head-mounted display
that only has monocular vision.
And as an expert in eye-tracking, I was like,
we haven't fully explored that concept,
but you really radically erased it from the design space
and you said to me today that this doesn't make any sense.
So, how would you approach that, like?
- Did you have, so are you referring to VR?
- [Moderator] So, we took a cardboard.
- Oh, okay.
- [Moderator] Cut it in half.
- Uh-huh.
- [Moderator] Put the smartphone this way
and experimented that way, because we wanna be able
to interact in a more immersed view than AR
and also have access, quick access to the real world.
- Mm-hm, actually that sounds interesting
and I'm curious how it felt.
Because I think one problem you could get
is you that you don't get one gaze point,
but somehow you get
one for the right eye in a virtual world and then one
for the left eye maybe in the real world.
And I cannot really imagine how you would perceive that.
So what do we actually measure?
I don't know, gaze is defined to have one gaze point
where the two points intersect, right?
So, this might be difficult to--
- [Moderator] Yeah, it was tiring very quickly, but.
- Yeah.
- [Moderator] Thanks, all right,
so we have a second question here?
- [Audience Member] Hi, very nice work.
The example you showed of the, using vergence
to allow the user to peer through the wall,
that's really cool, but it's hard to imagine.
Is it difficult for users, for people to do that?
If there's no, to peer past a solid object?
- Yes, so I think, no one in our lab at least
could perform this act without any guiding patterns.
So what we did is we used a HoloLens and displayed
a grid of points at different depth levels
and some of them, like the half of them
were before the wall,
in front of the wall, and half of them displayed
behind the wall, and then you're actually able to
perform this movement.
And then you're able to perform this divergent movement.
Because, for example, convergent movement is a little bit
easier to perform, you can look through like this.
But to diverge your eyes intentionally, that's really hard.
So we needed this pattern to actually do it.
- [Audience Member] Thank you.
- [Audience Member] Thank you, emcee, over here.
Speaking of the divergence and convergence.
For some people they have issues with this,
there are phorias, A phoria, D phoria.
And I'm wondering, were you able to pre check
for these issues before you did the exploration
of how people responded,
just to see if that was having an effect?
- We tried it out with, I think with four or five people,
and they didn't have any issues, so we didn't measure
that yet, but that's a good point, maybe in the future.
- [Audience Member] Yeah, isophoria, exophoria.
Really easy to check for and it would be awesome
to actually see if this was a way of helping people
back from that, too.
- Yeah definitely, so what I'm really interested in
is how can we adapt this, how can we use HMDs
to actually help users improve their vision,
or maybe improve not their vision, but their visual
limitations, like you said.
Yeah, thanks.
- [Moderator] All right.
(audience applauding)