Thank you, Jeff, for that introduction.
We were fortunate enough this past summer
to have two really great undergrads.
Kendra and Nikhil with us in the lab, and so they were--
they did an amazing job really of getting
a lot done in a short period working on this.
So I'm going to first talk about the motivations.
So a couple motivating factors that influenced this research
that we did.
I'm going to go over some related work.
Then we'll get into the fun stuff
with the experiment design and the implementation
and how he did stuff.
And then we'll go over some results at the end.
So this work was largely motivated
by two common limitations and current AR
[INAUDIBLE] displays.
So the first of these is that the augmented field of view
isn't really the same as the user's normal physical field
of view.
And in fact it's generally much smaller.
Then well typical error devices can provide
visual and audio stimuli--
additional sensory channels, they
have the potential to be pretty powerful.
So in particular, we were interested in expanding
the sensory stimulation to include
some form of haptic feedback in addition
to the visual and audio.
And for both of these things we're
particularly interested in their effects
on the user sense of co-presence in AR.
So what do we mean by co-presence in AR?
The term 'presence' has been around
for a while in virtual reality, and it
may have lots of nuances.
Different meanings for different people.
But historically, it's been thought
of as referring to this one sense of being
there in a virtual environment.
So you feel like you're present.
In AR the concept of co-presence might be described instead
as how one perceives the sense of being there--
sorry, in the sense of being together
with virtual entities that are in that augmented or mixed
reality environment.
And Slater introduced a related concept
of plausibility illusions, Psi, originally it's a VR concept,
but it seems potentially relevant to AR entities.
In particular, with the plausibility
of the interactions and sensory stimuli, or lack thereof, in AR
interactions.
And some of the plausibility stuff
will make sense when I go over the rest of the experiment.
So looking at some related work--
related to field of view.
Previous research has shown that restricting
a user's peripheral view can have behavioral effects.
So such as causing the users to move their head around more,
and it can affect navigation tasks.
It can affect search tasks in VR.
It can affect search tasks in AR.
And it may be related to distance underestimation in VR.
Some of our previous work has also
showed that restricting the peripheral vision,
while wearing HoloLens for example,
can results in similar behavioral changes.
So head motion and some proximate behavior.
In particular, with interactions with real virtual humans.
Unlike that work, this current-- this
work that I'm presenting here does involve virtual humans.
It involves other entities, and it was a very different task
than the AR search task.
Historically, there have been somewhat of a lack of research
looking at haptic feedback and AR and VR.
A lot of that is because of the complexities and limitations
of the technology.
That said, there has been work that's done--
that's been done, and it's generally pretty promising.
Tying haptic feedback to let's say co-presence and task
performance as well as presence, social presence, more
natural interaction behaviors.
Recent work has also shown positive effects
of indirect haptic feedback in AR scenarios.
So that is rather than getting haptic sensations directly
from a virtual entity, you instead
get those sensations through a shared physical object.
So let's say a floor that you're standing on
with a virtual entity or a table that you
share with a virtual entity.
So this is a concept that we'll use again in this work.
So let's get to the details about the experiment itself.
We had 22 total participants.
It was a within-subjects design.
13 male and 9 female.
Ages 18 to 41.
Many participant spent around an hour doing-- going
through all the conditions.
The task itself was a search task.
It involved a physical set of shelves in cardboard boxes.
The boxes then were augmented with virtual labels,
and they could be opened virtually.
So the experiment starts with a virtual human, in this case
Barbara, asking the participant for help.
She's lost her cat in one of these boxes,
and she needs the participants help in finding that cat.
She briefly explains how to open the boxes using
the clicker for the HoloLens.
And then, of course, the search task
was to find the cat in those boxes before time--
timer ran out.
So they had one minute to complete the search task.
In those boxes there's one of these five things.
Either the cat they're trying to find or one
of these four less pleasant critters that may
leap out and fly out at them.
So I have a short video here so you can see.
This is at the introduction showing
the user opening the boxes and various critters
that would fly out or leap out at them while their opening
them.
You'll see if they go and open a box they've already
opened it just remains empty, because they've already
searched there.
Whatever was there has already come out.
And you'll notice that the critters very quickly
move towards the participants.
And in the end, this person was able to successfully find
the cat inside.
Hooray for them.
So as part of this physical setup
we have a way of introducing haptic feedback from the user.
We do this through a transducer that
is attached to a rigid platform on which the participant would
stand.
In our particular setup we used a butt kicker transducer.
And if you recall how the critters would
jump towards the participant from the boxes.
The way that the vibrotactile feedback
was designed so that it would mimic the sense of the critter
hitting the floor at their feet, and then scurrying
across the floor.
So it felt very convincing.
And as you may have noticed in the video,
the critters jumped quite fast towards the participant meaning
that they very quickly disappear below the visual field of view.
In this setup, the vibrotactile platform
is actually making up for some of that visual-- the lack
of visual stimuli.
Because you can feel the critter landing at your feet
and moving even though it's moved out
of the bottom of the augmented field of view visually.
So speaking about field the views,
on the left here you see a view from inside a HoloLens.
This is the backside of a HoloLens.
In this case, unobstructed and, of course,
if you're wearing a HoloLens like this
you can pretty much see the entire physical set up there.
The shelves, the platform, everything.
It's pretty clear.
In contrast, we could restrict the field
of view of the participant by masking around
everything but the augmented field of view.
So in this sense, what their physical field view
was essentially equivalent as the augmented field of view.
And you can see what they would then see there on the right.
Very limited, but everything in that field of view
would be augmented.
So we can achieve a similar effect by restricting--
instead of restricting the field of view,
by de-emphasizing stuff in their peripheral content.
So for example, by not having it illuminated.
So one of the things that we could
vary in our experimental setup was the lighting.
So one of the conditions involved only
having the center of the area lit.
So the main area where the shelves
are in the boxes as part of the task.
The rest of the periphery was darkened.
In particular, you'll notice that the floor
area, so around where the platform and their feet were,
was quite dark.
So you wouldn't expect to be able to see
objects that were down there.
Similar to masking out the physical field of view.
So in all we had three independent variables
that we could control.
Whether the participant experiences
vibrotactile feedback.
Whether participant's field of view is restricted.
And whether the periphery, including the floor area--
floor area by the the participants feet, is darkened.
And as we mentioned previously, we used a full factorial
within the subject design.
So given those three independent variables,
we have a total of eight experimental conditions.
So here you can see on the top full lighting versus only
that central lighting.
In this case with the unrestricted field of view.
On the bottom you see each of those with the restricted field
of view.
And then finally, all four of those we
did with and without vibrotactile feedback.
So for subjective measures we adjusted
some standard questionnaires that
have been used in other previous work.
All four questionnaires were relatively short.
In the range of five to nine questions.
And all the questions were on a seven point scale.
Such that we could get a final average
score for each particular subjective dimension
on that one to seven scale.
The four dimensions that we looked
at where spatial presence, social presence, engagement,
and then gaming experience.
We had a few hypotheses going into the experiment.
The first of these was that we expected the participants would
have higher co-presence with the addition
of the vibrotactile feedback.
The second hypotheses we had was that the central lighting
condition, where the periphery was dark,
would also produce higher co-presence.
And similar to that we expected that we would see higher
co-presence when the augmented field of view was restricted
to match the--
sorry, when the physical field of view
was restricted to match the augmented field of view.
So let's talk about some of the interesting results
that we saw.
I'm going to show you some of the results
from the subjective questionnaires,
but first let me quickly describe how are
things organized and the plots.
So on the y-axis, that's the specific dimension
of the subjective questionnaires.
So in this case, the spatial presence questionnaire.
And remember that it's on a scale from 1 to 7.
And then the data is divided by all the conditions.
So you have central lighting.
You have normal lighting.
And then we further divide to unrestricted
field of view, restricted field of view,
and finally the conditions with no vibrotactile feedback
and those that had haptic feedback on.
So with the vibrotactile feedback enabled,
we indeed saw a significant effect
in spatial presence dimension.
Likewise, we saw a significant upward difference
in social presence with the vibrotactile feedback
turned on.
And also in conditions with haptic feedback
enabled we saw significant increase
in the engagement dimension.
So all three of those were significant effects
with vibrotactile feedback turned on.
For engagement we also found that users reported
significantly higher engagement in the central lighting
condition as opposed to the full lighting condition.
So when the periphery was not lit.
This could have been related to the specific task
we were doing.
Sort of the creepiness of the fact
that there's these dark critters that are jumping out.
So it may have seemed more realistic and more appropriate
for this particular task, there was some indication
of that from some of the anecdotal responses
that participants gave us.
And speaking of anecdotal responses,
we also noted some interesting reactions.
So we looked for a few specific avoidance behaviors
during the trials.
So we saw that in some of the trials
participants would pull their heads [INAUDIBLE] back.
So they'd move backwards as a critter was jumping out
at them.
More people than that we actually saw moved to the side
to try to avoid these virtual critters that are coming out.
So moving one side the other side.
And we looked for people that would raise their hands trying
to protect themselves.
We didn't see anyone do it.
At least not during the particular trials
for this study.
We saw it maybe in other cases demo-ing people but--
So we saw some anecdotal comments
from users talking in particular about an increased
sense of immersion when the lighting condition
was dark in the periphery.
So for example, one user said that the darkness
made the experience feel more immersive.
And another comment was that compared
to the previous condition having floor vibrations with lights
off felt more immersive than having floor
vibrations with the lights on.
And in fact we received multiple similar comments
from participants that suggested that the effects
of the vibrotactile feedback was actually
stronger in the conditions with central lighting
even though the vibrations themselves
would have been the same across all the conditions.
The central lighting actually enhanced
the effect of having that vibrotactile feedback.
We also got several comments from participants
that indicated a phenomenon that's sometimes referred
to as attention funneling.
So this is where they were almost exclusively
paying attention to the augmented region
in the field of view.
And this could have been related to the specifics of this task
because they were focused on searching those boxes that
were directly in front of them.
And generally they were facing where the augmented field
of view would cover anyway.
So reviewing our hypotheses.
In the end we found support for our first hypothesis.
An important takeaway from this is really
that adding vibrotactile feedback
can be a really good, relatively easy way, of potentially making
the AR experience significantly more effective.
The second hypothesis.
We didn't see really evidence of increased co-presence related
to the central lighting condition,
but we did see significant effect in engagement.
As well as some pretty interesting anecdotal feedback
talking about the enhancement that the central lighting
condition can have.
It's perhaps-- it perhaps warrants
some additional investigation or consideration particularly
depending on the specifics of your AR scenario or task.
And we were actually surprised that we
didn't see any support for our third hypothesis
in contrast to the lighting variations
or restricting the deal the view to only be the augmented field
view.
This was interesting because previous work
that we had done and seen suggested that we would see
some different effects by this.
But again, this may have been related to the fact
that there was this attention funnelling going on,
and the search task was very limited to objects
that were generally directly in front of the users.
So to conclude-- to go over things again.
The vibrotactile feedback can be a very powerful tool for AR.
So consider kicking some butts or using whatever device you
might want to use.
The controlled lighting also has potential
for enhancing an AR experience perhaps
in combination with other factors.
One exciting aspect of this work is
that it's related to technology that's really starting
to expand a lot more now.
So we're really excited to see what the effects are
of dramatically increasing the augmented field of view
that AR devices have.
And all sorts of great new haptic feedback devices
that may come along with those.
So the other photo that's on here
is a demo that we set up of a very similar setup
at a large conference, I/ITSEC where attendees could actually
experience the AR with vibrotactile feedback.
And we did get a lot of great feedback
and some early comments that helped
guide what we did in the study.
It was a lot of fun so thank you.
[APPLAUSE]
If there are any questions?
[INAUDIBLE] research.
[INAUDIBLE] Very static environment [INAUDIBLE]
flat background [INAUDIBLE] this might
be in a more dynamic or dimensional environment
[INAUDIBLE]
Yeah, so I mean a lot of that, I guess,
was related to the specifics of this task.
There wasn't really a need for them
to pay attention to a lot of the things in the periphery.
As you said, there wasn't anything dynamic there
that they had to pay attention to.
It could be that if we repeated this experiment with something
that was going on the periphery that we may see
more significant differences.
For example, for restricting the physical field of
view to match the augmented field of view.
You may see more of that head motion
and behavioral things like that.
So, yeah, that is a good point.
Very inspiring.
[INAUDIBLE]
[CHUCKLING]
Was really [INAUDIBLE] audio to help enhance the idea
of human presences [INAUDIBLE]
Yeah, so that's a really good question.
In this particular study there wasn't really
any special audio.
So there was audio as you heard in the video.
And the critters made different sounds,
but it really wasn't spatially localized audio.
It was either they heard it or they didn't hear it.
I don't know how good the audio spatialization would be
on a HoloLens device like that.
You could certainly repeat it with spatialized audio
especially if you had a maybe slightly larger area where
the critters kept making noise as they moved.
We didn't look at it in this case
but it could be a very interesting thing to look at.
Any other questions?
Let's thank this speaker again.
[APPLAUSE]