- Yeah, thanks a lot.
Just in time.
Thank you computer.
So I'm Max.
My co-authors are Brian Hall and Michael Nebeling.
And his work has been conducted
at the University of Michigan School of Information.
And I'm just going to take that away.
However, I'm with CNA.
As a User Experience Manager.
So I got back into industry.
Why did we do this research?
So we talked to people.
We talked to friends.
And we talked to students.
We talked to colleagues.
And fellow researchers about Mixed Reality.
And it seemed like everybody
had kind of a different understanding
of what Mixed Reality is.
And then also some devices popped up and told us
well, we not all Augmented Reality.
We're Mixed Reality.
That's totally different.
So we got confused.
And we asked ourselves
what is Mixed Reality then?
It seemed like an unsolved question.
And I can tell you that we did not find
a one sized fits all ultimate definition of mixed reality.
Rather our work has two contributions.
So first of all we gonna present
six working definitions of Mixed Reality.
And from these six working definitions
we gonna derive a conception framework
to classify a Mixed Reality experiences
to make them better comparable.
Better understandable.
And to make it easier for you, actually
to put them into context.
The working definitions we found
results first presentation
is Mixed Reality according to continuum
that many of you might know.
Mixed Reality as a synonym for AR.
Mixed Reality as a combination of AR and VR.
Mixed Reality as a stronger version of Augmented Reality.
As a type of collaboration.
And as alignment of environments.
I will get into the details later, obviously.
And then the framework we've derived has seven dimensions.
The number of environments.
The number of users.
The level of immersion.
The level of virtuality.
The degree of interaction.
Input.
And output.
Now, how did we get there?
First we interviewed 10 experts from academia industry.
To inquire into what they think.
And how they perceive the different concept.
And after that we conducted a lit review with 68 sources
that were related to a Mixed Reality research.
And also specifically to the difference
between Mixed Reality and Augmented Reality.
Step one
for the expert interviews
we designed a questionnaire
that was intended to really inquire into
how would experts define different concepts.
And we were also trying to push the boundaries.
And see what is enough to constitute Mixed Reality.
That's why we also inquired
into different aspects of reality
that are relevant beyond the purely visual.
So for example we asked them
is listening to music
is that Mixed Reality?
Is Tilt Brush Mixed Reality?
Where you have this fully virtual view
but you have a motion that's translated
from the real world into the virtual.
And also Super Mario Brothers
where you push a button in the real world
and somebody jumps in a virtual environment.
Might that be Mixed Reality?
Additionally we also asked them
will these concepts still exist in the future?
Will they be different?
Might they just disappear?
And also whether a single definition of Mixed Reality
would actually be useful?
Also especially for the exact community?
As I said we had 10 experts.
Five of them were from academia.
Five of them were from industry.
And they included professors
the head of an AR lab
a CEO of an AR company
the CTO of a company
an R&D executive and similar.
All of them had at least two years
of experience in the field.
And eight had more then eight years of experience.
And what we found was that
Augmented and Mixed Reality were much more difficult
to define for the experts than Virtual Reality.
So VR was always pretty clear.
It was like, you have a fully virtual view.
And it was also mostly concerned with specific devices.
Say head were in displace.
But for Mixed Reality we had at least found that
it seems that according to the experts
special registration is necessary to constitute MR.
And also that seeing at least part
of the environment is necessary.
That's why Tilt Brush even though we had this motion
that was translated from the real into the virtual
was mostly seen as VR.
Because you are fully immersed in the virtual view.
They also stated that simple input is not necessary.
Is not enough.
Not enough to constitute MR.
Again, the example Tilt Brush.
So the motion capture was perceived as simple input.
And they said that's not enough to constitute MR.
In general they said a single definition
would actually be useful.
But they also agreed that this might never be happen.
Or at least, it's highly unlikely.
And also from these interviews we derived
four initial definitions of Mixed Reality.
The first one
Mixed Reality according to the continuum.
That was first presented in 1994 by Milgram and Kishino.
Which basically just means
Mixed Reality is anything in between
completely real and completely virtual.
And now the experts couldn't really agree
on whether the extreme account into Mixed Reality or not.
So it's both possible.
The second working definition.
Mixed Reality as a synonym for Augmented Reality.
So some experts said Mixed Reality is just like
a new marketing term for Augmented Reality.
But others than that there is no difference.
So not a lot to add here.
Mixed Reality as a combination of AR and VR.
This is more interesting.
So one specific example would be Pokemon Go.
Where you have a fully virtual overworld.
Which could be VR.
And you catch Pokemon in an Augmented Reality.
So you have one system that combines both.
Some experts said that's Mixed Reality.
And then finally the fourth one
we do some interviews.
Mixed Reality as stronger version of AR.
Where they said that while you are just an observer
in simple AR
where you just view a scene
through the camera of your smartphone
in Mixed Reality you can really walk into the scene.
And also the devices have a better understanding
of the environment than simple AR devices.
So one example that was given here a lot was HoloLens.
Then in step two we did the lit review
to confirm these things we found in the interviews.
And for this we selected papers
from CHI, CHI PLAY, UIST and IMSMAR.
Between 2014 and 2018.
That included Mixed Reality in their title or abstract.
And then we added four other sources
that we knew were dealing
with the specific differences between AR and MR.
So we looked at all of the papers.
And in the papers we tried to find
the points of which the authors defined
their understanding of Mixed Reality.
And we also looked at which papers were cited
as definitions of Mixed Reality.
So this is the citation graph
for the first round of lit review.
And as you can see
for example in Milgram and Kishino, 94
cited quite a lot.
And in general on a high level
the findings we derived from the lit review
was that different conferences seem to use
different notions of Mixed Reality.
So for example
while CHI seems to be about collaboration
CHI PLAY seems to be more about aligning environments.
Overall they were 22 sources that were cited
to define or explain Mixed Reality.
And Milgram and Kishino, 1994
the continuum is indeed the single most cited source
for Mixed Reality.
But it was used by much less then 50%
of the papers we reviewed.
So it is the single most popular notion.
But it's not universal.
And also finally
we found besides confirming the four notions
we already found
we found two additional working definitions
of Mixed Reality.
So the fifth one was Mixed Reality as collaboration.
Where an AR and or VR system is used to facilitate
the collaboration between more than one user.
So in this specific example that we reviewed
we have an AR user and a remote VR user.
And the environment of the AR user
is recreated in Virtual Reality for the remote collaborator.
And then finally
the sixth working definition we found is
Mixed Reality as an alignment of environments.
Where we have two distinct environments
that are synced through some sensory tracking or input.
So this specific example is an Earthquake table
where we have building blocks on a UF card table.
And a projection.
And whatever happens to the building blocks
is synchronized with the works review in the projections.
So the environments are synced.
So that gives us a total of six working definitions
of Mixed Reality that we found to be in use in the wild.
According to the continuum as a synonym for AR.
As a combination of AR and VR.
Strong AR.
MR as collaboration.
And as an alignment of environments.
Now we took all these working definitions
and we wanted to make this conceptual framework out of them.
So we tried to find the dimensions that would
uniquely identify the specific differences
between the different notions.
So that each one of them could be classified unambiguously.
And what we found
is that the number of environments is important.
The number of users is important.
Especially for the collaboration aspect.
And the level of immersion.
So how immersed you feel of Virtual Reality.
Which is independent of the immersion.
But it tells how much virtual content
you actually experiencing.
And its also very important to distinguish between
implicit and explicit interaction
in all of these experiences.
But more to that.
We also inquired into the relevant aspects of reality
that might be considered for MR beyond the purely visual.
Both in the expert interviews and the lit review.
And what we found is that audio might be relevant for MR.
Motion, haptics, taste, flavor.
Some people have started looking into smell.
And geolocation can be an input or output.
Temperature is important.
And basically anything sensors can track
can be an input into Mixed Reality.
And any any stimuli and sensory modalities
that you can feel and experience
can be be an output to MR systems.
So we took and put an output
and we also added it to the framework.
And now these are
very important on an per experience basis
to define certain experiences.
But for the working definitions
input and output can just be anything.
So its independent of the notion.
But very important for the specific implications.
So to get back to the example with the Earthquake table.
Here we have many environments.
We have one to many users.
Because obviously more than one person
can play with the Earthquake table.
The whole experience is not to partly immersive
because we have this Earthquake table
that's just a purely real environment.
And we have a projection
where you can argue how immersive it is.
And then its also both not and fully virtual.
So Earthquake table was projection.
We have implicit and explicit interactions
because you can just walk around the whole
Mixed Reality scene.
But you can also explicitly interact
with the building blocks and just push them down.
And the environments will be aligned.
So therefore the input is motion.
And the output is visual.
And this is how we would classify
this very specific Mixed Reality experience
according to the conceptual framework we just came up with.
So to conclude
then what is Mixed Reality?
According to one of our interviewees
it's all marketing mumbo-jumbo at this point in time.
However we believe that as of now
Mixed Reality is one of six things.
And all of these six things valid.
But it just depends on the background
and the specific experience of which of these six things
constitute the type of Mixed Reality you're describing.
Also it's an evolving concept.
So some of these working definitions might disappear.
There might be new ones coming up.
Also the conceptual framework that we came up as of now
I think we have a pretty complete picture.
But also this one might change in the future.
But most importantly
Mixed Reality is something we need a common vocabulary for.
First of all, so that we can put our work into context.
And so that different works based of Mixed Reality
are better comparable.
Better understandable.
And also just to avoid misunderstandings
when talking about Mixed Reality in the HCI community.
But also in the research community in general.
And yeah.
We were hoping that we can provide
very good basis for that.
And thank you for your attention.
(audience clapping)
- [Announcer] So that's great.
(murmurs) questions?
- [Oliver Shaler] Stand?
Oliver Shaler, Waterloo.
So the experts said that it usually involves sight.
We often think of VR
Virtual Reality as having headsets or screens.
But what would your impression be
for people with sensory impairment?
For example people who are blind?
Or visually impaired?
Even in your classification for input output
there's all sort of possibilities there.
But you said in addition to visual feedback.
- No no no.
What I meant was beyond visual.
- [Oliver Shaler] Beyond it?
- Yeah beyond visual.
So none of this is restricted to visual output.
- [Oliver Shaler] So...
- Could be anything.
If you were talking about impaired
like visually impaired users
you could still have some spacially registered audio
and that would be Mixed Reality
according to this conceptual framework
for the visually impaired user.
- [Oliver Shaler] Okay.
- So.
Visual is not a necessity here.
- [Oliver Shaler] Okay great.
Do you have any ideas of how
this might inform work into...
Because I still think we're very headset and screen focused
with any star reality.
Right?
- Yup yup.
- [Oliver Shaler] So any insight into how we might
how this framework might help us
when we look to people who
for example might have these impairments?
- We did not specifically look into that.
I mean we're hoping that it will facilitate
work into this direction.
And probably help thinking outside
this whole headset box that you're describing.
But we've not asked specific question about that.
We'll just see what happens, right?
So we're actually hoping to contribute
to getting outside that box.
Yeah.
- [Dan Cosley] Hey, Dan Cosley.
Cornell University.
And actually it's being a little more
in my NSF role right now.
So first of all, I really appreciate the work
that tried to defined this.
Cause when I see proposals on Mixed Reality come in
and often it's totally unclear what's happening.
And it's like the rest of the KII community is saying
cyber bullying or crowdsourcing.
Meaning everything from Wikipedia to Uber.
Okay.
So I really appreciate this.
I like the work to move toward definitions.
And I'm just curious if you have any sort of preferred
personal definitions that you would like
or terms for certain classes of this
that you'd like to do to carve up this Mixed Reality term
a little bit.
To help create that vocabulary.
Because you get to be prescriptive a little bit
when you're doing an analysis like this.
So tell what you think.
- I have a least favorite working definition.
Which is simple using Mixed Reality
as a synonym for Augmented Reality.
That is kind of lame, right?
So, but I mean it's out there.
People are using it.
So we have to consider it.
But it's also kind of odd
to have two different terms for the same thing.
Which is like acknowledging that people are doing this.
So working definition number two
is definitely my least favorite.
And I think that one should maybe go away at some point
for less confusion.
But I think all the others are absolutely valid.
Because they just describe...
They are kind of competing and also kind of overlapping.
But they're all valid definitions.
No favorite there.
It's just very important to be clear
about what you're talking about.
And that's why we did this conceptual framework thing.
- [Julan] Thanks for the talk.
I'm Julan from University of Konstanz Germany.
So your paper just answers one of the question from my...
Several months ago I asked this question
to one of my colleague.
Because during my Master
I only know AR.
I work on the mobile AR devices applications.
But after the HoloLens came out
I start to know the concept MR.
A SKU I don't see the difference.
But now for many people I ask them
what kind of devices do you think is AR?
Or VR?
So from this point of view
have you asked your participants
which kind of device they think is AR or VR?
For example, for HoloLens for me it VR and AR.
So I can't tell the difference.
It's hapmonted AR.
- Yeah.
Before I forget it.
I want one more thing.
Michael says hi to his mom.
Hi Michael's mom.
To get back to your questions.
We did not specifically ask about devices.
But in the case of this strong AR definition
of Mixed Reality
some of the experts exclusively said
this is probably bound to a specific device.
Which would be HoloLens in this case.
Other than that Haponted displace
where you have really a fully immersive view
which is usually seen as VR.
So the only specific Mixed Reality
was mentioned was the HoloLens.
Anything else Hapmont.