Thank you, Robert.
So this work has been done at AIT lab at ETH,
together with Anna Feit, and Otmar Hilliges.
So this video only plays every second time
I start the presentation.
So let's try it again.
See?
[LAUGHTER]
So mixed reality allows to display personalized contents
anywhere and anytime and essentially with any arbitrary
amount of information.
And it complements current technologies, such as desktop
or smartphone computing.
What we are arguing is that mixed reality is
inherently context-sensitive.
So the interface changes every time
the user changes their task, or moves
to a different environment.
So manually designing for all those different situations,
or their combinations, is infeasible.
So context sensitivity should avoid two main problems.
The first one is constant manual adaptation.
So here, the user is switching tasks,
from reading a paper to taking a break.
And if the interface is distributed in 3D,
to have to adjust all the applications to fit
their needs.
For example, that they can see when their next train goes,
or to watch a YouTube video, or when their food
comes, or a [? news. ?] So this they
have to do every single time they
switch tasks, which is tedious and cumbersome.
The second problem that happens is information overload.
If you would go for a trivial solution, where
all the information is displayed constantly
all the time, that would be, in the end, pretty annoying,
I believe.
So the question for our research is really how can we
automatically adapt virtual contents
to avoid constant manual adjustment
and to not overload users?
And here's an example of our approach.
So here, the user writes down notes in their office,
and experiences medium cognitive load,
as indicated in the lower left.
The visibility of the virtual elements--
their placement and level of detail--
are automatically decided by our system.
They can be adjusted by the user,
but they don't necessarily have to.
So prior work has mostly focused on layouting
of virtual elements to avoid visual clutter--
for example, they work by [INAUDIBLE] [? colleagues ?]
and [? flair, ?] or they work by [INAUDIBLE] [? Grasset. ?]
[INAUDIBLE] [? Julier ?] both adjusted the amount
of displayed information to avoid
unnecessary visual complexity, or they filtered information.
Also, tomorrow, in the morning, I
recommend going into the session where my colleague, Christoph
Gebhardt, is going to present the reinforcement learning
approach for information filtering and mixed reality.
So most of these works really focused on visual complexity.
What we wanted to do is that we wanted
to jointly optimize an interface for user's current context.
And with context, here I mean cognitive load, their task,
and their environment.
So let me explain what we did and how we did it.
So, based on user's current context,
we want to automatically decide on the visibility, placement,
and level of detail of applications.
So, visibility is pretty straightforward.
We want to decide, should an element
be displayed or shouldn't it be displayed?
For placement, our approach decides
if an element should be anchored in the view or in the world.
So with view anchored, [INAUDIBLE]
displayed as heads-up item.
So that's another one of these videos.
Beautiful.
So we decided on this more simplistic model of placement,
since it allows users to exploit their spatial memory,
but it could be potentially extended
with an adaptive placement method.
So here, the user's adjusting the position
of the items in the world, and we decide on
if it's in the world or as heads-up display.
Thirdly, we want to decide on the level of detail
of an application.
So here you see an example with the email application
with five levels of detail, and all
with increasing information, from an icon to your last three
emails.
And we do this with a mix of rule-based decision-making
and combinatorial optimization.
So in essence, what we want to do
is if we have an increase in cognitive load,
we want to show less virtual elements based
on the task and environment.
To do so, we have two main inputs--
ones that are provided by content creators, or designers,
and the other ones that are measured live by the system.
So the content creator provides the applications.
So, for example, the email application,
and for each visually designed levels of detail.
And then they also provide a cognitive cost of an element.
So how cognitively expensive is it
in order to display something.
And then they provide a set of tasks.
And for each task, they provide the utility of an element--
so how useful is it, and their frequency of use.
So how often is this used.
Currently, this is input with a set of simple sliders,
but it's easily feasible to actually
have a data-driven approach to replace
this manual approach here.
The second part is measured by the system.
So for one, we just geometrically
measure are elements actually visible to users.
And the second part is our cognitive load estimation.
So here we build on the work by Duchowski
and colleagues on the index of pupillary activity.
The Index of Pupillary Activity, or IPA,
is essentially the frequency of meaningful pupil dilations.
And this is correlated-- or it should be correlated,
at least--
with cognitive load.
So with higher frequency of meaningful pupil dilation,
we have a higher cognitive load.
And the same holds for low cognitive load.
In our approach, this could also be replaced by other methods
if they are found to be more reliable.
We have a three-step process in order to solve that problem,
where we first decide if an element should
be world-anchored or view-anchored.
Secondly, we decide on its visibility and level of detail;
and third, on if it's decided that it should
be visible and view-anchored we decide
where it should be displayed in the heads-up display.
So to decide if world- or view-anchored,
we use our knowledge of the environment.
So as you see later, we have a depth camera mounted in front,
or if you would apply this approach in VR,
you have all the information you need anyways.
And an element is displayed world-anchored-- so,
anchored in space if it's visible to users.
So it's in the field of view of users,
and it's not occluded by geometry.
The second part, and what I think is also
the core of the method is where we
decide on the visibility and the level of detail.
So in a nutshell, what we want to do
is we want to display as many elements with the highest
level of detail while not exceeding
the cognitive load of users.
We use integer linear programming to solve that.
So what we take is the input from the content creators,
where we take the utility of an element for a specific task
and its frequency of use.
We then use combinatorial optimization
that allows us to find out which elements to display
and with which level of detail.
So note that the trivial solution here
would be to always display everything with the highest
level of detail, which does not solve our second problem
with information overload.
So therefore we introduce two main constraints.
The first one is a cognitive load constraint.
So we take the current cognitive load of users,
and the cognitive load of the displayed virtual elements,
and we say that this should not exceed a certain threshold
minus a small alpha to not max out cognitive capacity of users
constantly.
The second constraint is a level of detail constraint, which
essentially says only display each application
with a single level of detail.
So we don't have duplicates in our virtual world.
The third part is then that we place view-anchored elements.
So if an element should be visible and view-anchored,
we greedily place it in the best slots.
So what do I mean by best slots?
So here is an example, for example,
how the Skype icon gets placed.
So the best slots are essentially
a function over the user's field of view
where we say the for-real area is of low quality,
so that you don't have the virtual elements constantly
in front of you.
Sorry, [INAUDIBLE] area.
The mid-peripheral area is of high quality, where
we say this is a good balance of visibility,
but also it doesn't disturb you.
And the far-peripheral area is of low quality, mostly because
of constraints of current headsets.
And once we've decided where to place it, we place them.
Or if an element has been previously assigned,
we put it into the same slot again
to exploit spatial memory.
So this is our three-step process
to automatically decide when, where,
and how to display virtual elements.
In terms of implementation, we implemented our prototype
in unity with SteamVR, and we used
the Gurobi Solver for the linear program;
for the integer program.
This runs in real time at roughly 7 milliseconds
for 12 elements with five levels of detail.
In the paper you can find the details
that we checked on the scalability of the approach,
and we found that it scales linearly
with the number of elements and their levels of detail.
So for example, if we were to have 200 applications with four
levels of detail each, this takes roughly 40
millisecond for the solver to solve.
Note that while this is not real time,
I do not think that this is a process that should happen
at 30 frames a second, but really only when there is
a meaningful change in context or task or cognitive load,
for example.
In terms of hardware, we use a VR headset in HTC Vive Pro,
with [? a set ?] mini-camera mounted
in front and the integrated Pupil Labs eye tracker
for the cognitive load estimation.
We decided on that approach, since it gave us
a larger field of view compared to available AR headsets,
like the HoloLens 1 or the Magic Leap.
I'm going to show you three scenarios which we implemented.
So for the first scenario, the user
is just doodling, which I would argue
is a low cognitive load task, and which we also
measured here.
And there's more information available.
A lot of information is at hand.
So the user has all the information.
And because they are-- they can handle it
in terms of cognitive load.
Second is brainstorming, which was measured
as a medium cognitive load activity, where
we show less information that is more
task relevant-- for example inspirational images,
or the current tasks.
The last one is where we read a difficult scientific paper,
where we only show very minimal information to not overload
users.
We also evaluated our approach to find out about the usability
and applicability.
So we had 12 users that followed a dual-task paradigm.
And they completed three different primary tasks
with different cognitive loads.
So counting backwards in steps of 17,
which is for many people, or for most people,
a high cognitive load task; counting backwards
in steps of 2, and icon search.
As a secondary task, they were asked
to answer questions verbally.
So the system with text-to-speech
asked them questions, and they needed a couple of applications
that were in the virtual room in order
to answer those questions.
As dependent variables, we measured
primary and secondary task performance,
as well as the number of interactions needed in order
to answer the questions.
We performed the study in VR, because the setup
I introduced earlier introduces a lag
of roughly 60 milliseconds, which
is OK for kind of testing.
But it introduces a bit of motion sickness, which
we are avoiding, and also it provides an even larger field
of view.
We tested two main conditions, where the first
is the manual condition.
So here the user is counting backwards in steps of 17,
and then gets asked a question-- for example, who
is your second Skype contact?
They answer the question verbally,
which gets logged by the experimenter,
and then they move back to the task.
In the optimized condition, our system
has a rough knowledge of the tasks
that the user tries to achieve, and then suggests
the placement-- so that when, where,
and how of the virtual elements-- and users
can still adapt it in order to fulfill their secondary task.
In terms of results, we found no difference
in task performance between manual and optimize.
This is not unexpected.
The performance is really dominated by the primary task
here.
But our system did not disturb user
in fulfilling the task, which is,
for an adaptive system, already kind of the first goal,
definitely.
What we also found is that our method actually
decreased the number of necessary interactions
by 36% compared to manual, which makes us believe that this is
definitely beneficial in the long term,
and also makes it applicable if you really have this context
adaptation.
So in conclusion, we are arguing that manual design
of mixed reality interfaces is not
feasible because of the constantly changing context.
What we are proposing is to jointly optimize
mixed realities based on this context.
And we provide a first implementation
based on rule-based decision making
and combinatorial optimization.
There's still a set of open challenges.
So for one, cognitive load estimation.
The IPA introduces a lag of roughly 30 milliseconds.
And I think that a sensor fusion approach-- for example,
with GSR--
would be a way to go that might be more reliable and also
faster.
Also, predictability is super important for any user
interface, as probably all of you know.
And also, we have mostly tested applications
that were inspired by desktop computers,
and not more deeper augmentations--
like, for example, what happens if I actually
changed a color of a physical object through HMD,
how would the user react to that.
And also we did not yet investigate larger changes
in context--
for example, changing the context
from the home to the office to outside.
I think this will yield a set of new and interesting challenges
that will require computational approaches to make
mixed reality feasible.
So to sum it up, I think that context
where mixed reality really balances
virtual contents and the physical world
to avoid overloading users, and to avoid
the constant manual adjustments of virtual elements.
So the last thing I want to say before Robert cuts me off
is that we published our source code
even with a small 2D playground for you
to get started with integer programming,
if you're interested in that.
And everything is published [INAUDIBLE]..
So thank you very much for your attention,
and I'm happy to answer your questions.
[APPLAUSE]
Thank you, very much.
Let's see if there's any questions in the audience.
All right.
So I have a question.
Actually, this is about the cognitive load estimation.
So you said there was a 30-second delay
on the cognitive load estimation.
The values were shifting during the actual videos you showed.
What's that from?
So, I mean, we do constantly do the eye checking
and the calculation of the IPA.
But we need roughly 30 seconds of data
in order to compute kind of the current value.
So this is why this delay.
So we have a sliding window of 30 milliseconds,
and an event that has happened 15 seconds ago
will only kind of show up in the cognitive load
estimation with this delay.
We are currently investigating kind
of if you can do it actually with 15 seconds or 10 seconds.
But I think especially with that method,
it tends to be more noisy the shorter your window is.
We're talking about a frequency of two changes a second,
or something like that.
So if you only have five seconds,
you have a very weak signal-to-noise ratio here.
I guess the values that you show here,
those are all based entirely on the pupil dilation numbers.
Yes.
That's cool.
So I think really kind of something
where you combine it with GSR, with stress level, for example,
would really be an interesting approach, which
is more immediate.
And then you can find out if that actually
works better or not.
Thank you.