Thanks, Fraser.
So OK.
So I did an internship at Microsoft Research,
and we explored a possible use for eye gaze input.
This is one of the applications we developed.
So we will have this user and jigsaw VR
experience to solve the jigsaw.
It's not as-- can you see anything of it in this picture?
Let me replay this.
Anything off?
So we just change this puzzle piece so as
to have the user solve the jigsaw
to make the jigsaw easier to solve.
We do this with respect to the eye gaze input,
so we only change stuff that is outside the [INAUDIBLE] you
see here.
So for example, we changed now the piece on the lower right.
Yeah.
So we do this a bunch of times, and we also
apply other techniques, such as the visual masking.
The cat is moving, for example.
And we also pay respect to the user's
internalized spacial relationship
between the objects, which are the puzzle pieces here.
So why are we doing this?
The overarching question we want to answer
is how do we make users not notice something in VR.
And I've given you one example and throughout this talk,
I will give you a couple more.
And the answer to this is mise en scene,
which is the creation or arrangement
of a set of scenery.
And you have this in different sorts of media productions
such as theater or film, where you see a certain thing
because the director controls the feed of you,
and you don't want to see other stuff, such as the boom
operator, the camera person, et cetera.
And this approach doesn't work with VR productions,
because the feed of view is under user control.
So we see this phenomenon here where in VR, the feed of view
is increased, eventually even to the full 180
degrees of human vision.
And with head tracking, there's even more error we cannot
control.
So our idea is kind of simple.
We just use eye gaze, eye tracking to secretly update
content in VR.
So we decrease the area we cannot control to the foveal
vision of five degrees.
Or in other words, we increase the range for updates
to 355 degrees.
There's a bunch of related work on this.
Two works I find interesting here, which I picked out
is [INAUDIBLE] proxy and dynamics of category direction.
We use eye gaze and measured [INAUDIBLE]
to reposition the user or change the motion mapping to enable
in the first case passive haptics
and in the second real walking.
There's other stuff such as auto pager, where we automatically
page turn when the user has reached the end of the page.
All of this work is somehow fuzzily routed
in this phenomenon of changed blindness
or inattention of blindness.
And you probably know this work on the right here,
where the watcher of the video cannot see the gorilla walking
through the scene when counting the ball throwers of the white
team.
So we can somehow rephrase this question now I asked initially.
So how do we make viewers not notice something in VR
as the same as how do we apply this cognitive illusions in VR?
And there's two people who can help us
with answering that question.
10 years ago, Steve and Susannah were keynote speakers
at this very conference in Victoria.
One year before, they published a magazine article saying
cognitive delusions are not sensory in nature,
rather they involve high level functions such as attention,
memory, and causal inference.
So attention, memory, and causal inference.
And this is what mise-unseen delivers.
It delivers computation models of those very three things.
With the computation model of causal inference,
we can prevent anticipation of a change that is about to happen.
With the second we can prevent observation of the change,
and with the third we can prevent
recall of the change that has happened,
and we use gaze to process--
we process gaze in different forms all of these layers
to do exactly that.
And I will give you three examples
like the forge and the gallery and the jigsaw you've
seen earlier, which all use a different subsets
of those layers.
So let's look at the gallery and the model of current attention.
This is kind of the naive approach
you would initially do.
So the user is looking at the modern art piece
painting on the right.
We change the painting on the left also to modern art
so as to individualize the content.
So we do it like this.
We'll get the gaze point, the current gaze point.
We compute the complex hollowed object
we want to change, compute the distance.
We have all-- and this distance is above our full view
vision of five degrees I mentioned earlier,
then we apply the change.
There you go.
We can get a little bit more complex than that.
There's this notion of covert attention, so where you look
is not necessarily where you attend to.
For example, you can look at me and not look at the slide
but attend to the slide.
So look at me, attend to the slide.
You might be able to see the gorilla there.
Same thing.
This is called covert attention and you can only
consciously apply it 300 milliseconds
after you've done a second.
We can measure seconds, so we measure the second
with timing of 300 milliseconds where we can apply the change
and that's it.
So there's another layer.
Let's talk about the forge and the layer of spatial memory.
So this relates to haptics.
So we see two weapons on the table in front of the user
but physically, you have only one prop,
which is on the desk on the lower right.
So which weapon do you match the physical prop?
From the user's case, we infer that the user's interest
is more on axes than hammers, so we match the axle to the prop
but we also move the hammer so we maintain
the special relationship between the objects the user has
internalized.
All right.
So how do you do this?
Similar to before, we compute the complex parts
of the objects and we implement it directed right
at graph between those objects and we have the physical prop.
So now the user looks throughout the scene
and we adapt the weights depending
on where the user looks.
So if the user looks at the axe and then the hammer,
we'll change the weight between them, et cetera.
So now when we decide to move the axe,
we move the hammer with respect to the weight
between the hammer and the axe.
w
All right.
So we can also use this model of spatial memory
to trigger the change, to prevent
anticipation of the change.
So what we got is basically a right vector.
We've got a scripted right vector and we can compare them,
and if this falls below a certain threshold,
then we trigger the change.
So the scripted weight somehow corresponds
to our notion of the user deciding to pick up
the axe instead of the hammer.
Let's talk about the jigsaw.
So the jigsaw application uses random triggers.
So we change the puzzle pieces randomly,
but there's also tons of other stuff like vision masking
techniques.
So the cat moves as a callback function to the mise unseen,
triggering the change in the scene.
There's also low contrast we use with the puzzle pieces,
crossfire, and stuff like that.
We evaluated these three applications and more,
and we found that-- also, on the left side
there's [INAUDIBLE] noticeability ratings
of the participants.
So we noticed that mise unseen is indeed
able to induce those covert changes
but there's an important difference here,
which we dug into.
So we did another psycho physics study
on the visual masking techniques.
So we had the baseline where we pump up this little dot,
which the user needs to identify right in front of the user.
Then we have the [INAUDIBLE] condition
where we have the ball pop up somewhere in the periphery,
so representing the importance of eye tracking.
Then we have the add cross fade.
Th ball on the upper left like fades
in with the two second time window
and then we have other stuff such as low contrast.
If we add the task of the user needing to read the text and we
have motion detectors.
And what we see is that the detection rate decreases
with each visual masking technique we apply.
I want you to pay attention to the second one,
to the [INAUDIBLE] condition.
So the off over condition is better
than baseline, which represents the importance of eye tracking
but with each masking technique, this gets better.
So it's both.
It require eye tracking and we need visual masking techniques.
In other words, there's some sort of signal to noise ratio.
We need to decrease the noise, we induce them into the scene
but we also can add--
reduce the second version, but we can also
add noise to mask the signal.
So we implement a bunch of other applications
where this might be useful.
So on the upper right for example are
our [INAUDIBLE] riddle.
The user needs to find the right call sequence here.
But the user cannot guess.
When the user guesses, the solution correctly,
we changed the [INAUDIBLE].
The user needs to understand.
So user has figured out the hint on the lower
left, which we can track through [INAUDIBLE] tracking.
Then we allow for the correct solution.
On the upper left, there's a real working scenario,
which is similar to the passive haptic scenario.
So whenever the user is not looking,
we can accommodate for less walking space
by changing here the [INAUDIBLE]..
There's other stuff.
For example, on the lower left, we
trigger this low fidelity explosion
depending on user space.
So this relates to [INAUDIBLE] rendering.
On the lower right, this is our idea of virtual locomotion
and teleportation where we basically
reduce the update rate in the periphery to 1 hertz
and keep the 60 or 90 hertz in the middle
to reduce motion sickness.
So to sum up, we develop mise-unseen,
which is a software system that induces convert changes
into virtual reality scenes by having,
by preventing anticipation, observation,
and recall of a change.
These models and masking techniques
reduce noticeability of those cover changes to 0.
We've shown a bunch of different uses
for those cockpit changes in virtual reality.
This is basically an [INAUDIBLE] to using
eye tracking for-- yeah, hiding changes in virtual reality
scenes.
This work was supervised with Andy.
We want to thank Christian, Mar, and Al,
So that was mise-unseen.
My name is Sebastian [INAUDIBLE]..
Thank you for your attention.
[APPLAUSE]
Thank you, Sebastian.
And thank you, everybody, for your patience with the noise.
I'm hoping that's taken care of now.
There's lots of time for questions for Sebastian.
[INAUDIBLE] here.
Very cool talk.
Thank you very much.
I'm wondering if you could comment on kind
of how resilient the effect is.
I guess, for instance, when you switch the hammer and the ax
you maintain the spatial relationship.
And I imagine if you swapped out paintings in a room
with only two paintings people would start to catch on to it
eventually.
Could you comment maybe on how resilient the effect is
and how much you can change in that?
All right, so I talked about those different layers, right?
You need to prevent the observation of the thing?
And it's correct that in the gallery, users
comment a lot on-- and not as a [INAUDIBLE]
because I looked at the painting before.
And no one noticed it because I haven't
used optimized model of spatial memory in that application.
So if you do that, you can then just not trigger the change.
The user has looked at this before,
so by having implementing all three stages to kind of prevent
that.
And, of course, it sets the limitation, right?
So if the user really has looked at everything,
and everything is kind of internalized,
what can you change, right?
So this is an obvious limitation to the thing.
Yeah.
[INAUDIBLE] [? Berryman, ?] Arizona State University.
This was a really cool talk, Sebastian.
I was very fascinated.
This reminds me of redirected walking in ways.
I was curious for the examples you've shown,
it looks like you're fixed.
It's not look-- you're sitting down, basically.
You're not freeform moving around physically
in the virtual space.
Is that correct?
With the passive haptic sample, or which one?
With this these scenes in particular.
So as I look around, it's a matter of my orientation.
I'm curious how my physical movement would affect this,
as if whether or not you guys thought about--
so we haven't factored this in.
Of course, we couldn't factor in a lot of other stuff,
such as physical movement.
We also can do-- what I want to do
is include some video processing to have [INAUDIBLE],,
have another model of visual saliency to factor all this in
and get a better model.
True, we only focus specifically on eye tracking in this work.
So do you see this being capable of evolving into something
that compliments physical walking
in virtual environments?
Say that again?
Do you see this system that you built
being capable of complementing physical navigation
in virtual spaces?
Yeah.
So one of the examples on the upper left
is basically a sketch, right?
So that's the same notion as the passive haptics example where
we accommodate or change the virtual world
of the layout with respect to what the user has decided.
And we can do that for passive haptics,
all right, with the viscosity of physical props, all right?
But we can also do that with viscosity of physical walking
space, all right?
The change is just bigger, and it is harder
to implement because it covers more of your view.
Yeah.
Hi.
I am [INAUDIBLE] from [INAUDIBLE]..
And so I was wondering, the people
who noticed the effect, how did they react to the changes?
Well, all right.
Every time you cheat, you break immersion, right?
So if something is not believable,
it's not good for the application.
And I guess it's a probabilistic thing,
same as the change blindness stuff,
same as the user studies on [INAUDIBLE] from [INAUDIBLE]
from the '90s, at the same thing, where 20%--
I don't know-- I'm guessing here--
of participants still notice the thing.
And I'm unsure of what we can achieve here,
if it's 80%, or 90%, or 20%.
This is super application specific.
But some people do notice that's true.
But this is what we want to prevent with this.
Yeah.
Still time for questions.
And we have more beads.
If not, the other two, the other side
is running a little bit behind.
So I'll ask a prickly question that I never want to get asked.
There's potentially ethical implications
to this where you're having people look at something.
And then they look away.
And now it's not what they saw before.
Have you thought about how this might be used for evil,
or is it some other part where humans
start to doubt themselves in what they've seen?
Interesting.
We could latter brainstorm on that.
Yeah.
Let's try something out later on.
Yeah, definitely.
Maybe we can get higher up on Facebook or something
and then iterate on that.
Thanks.
All right, sounds good.
So maybe we'll get the next people to start setting up.
And everybody [INAUDIBLE].