- All right, thank you.
- Good afternoon everyone.
I'm Isaac Wang from the University of Florida,
and I'm here to present our work on
exploring virtual agents for augmented reality.
Just to start off, when I talk about augmented reality
or AR, I'm referring to being able
to see virtual content superimposed
onto the real world like a hologram.
AR headsets, such as the Microsoft HoloLens here,
enable users to manipulate and interact with
this virtual content in a natural and hands-free manner.
One common way right now to provide a hands-free interface
is through the use of a virtual agent
like the digital assistants we currently have
on our smartphones.
Now, what we could do is simply port
these agents over to AR headsets
and thus enabling natural and hands-free interaction there,
but these agents weren't really made for AR
and don't take full advantage of its capabilities.
For the most part, these agents don't have any visuals
or rather embodiment to them.
Based on prior work, we know that there are many benefits
to embodiment such as helping with rapport, likeability
and being able to enrich interaction
through nonverbal communication such as gestures.
Similarly from human-robot interaction,
we know that having a physical presence would have an impact
on learning and overall bring more influence to the table
based on just an agent or robot being there with the user.
In the context of AR, giving an agent
some sort of body would make sense
as AR is designed to give things
a virtual presence in the real world.
But then, that raises the question,
would embodiment translate well to AR,
and is that what users really want?
So, our goal was to explore
how different agent representations in augmented reality
would affect how a user perceives the agent
and how users behave differently if at all.
For example, what about having a tiny person on your desk,
or maybe a virtual smart device to interact with,
or could that be too distracting?
We wanted to understand what users would prefer to work with
in an AR environment, and how they would react and respond
to different visual representations for virtual agents.
To study this, we created a scenario which was inspired
by a prior study by Geven et al.
What we did is we had people where an AR headset and search
for hidden objects in images while interacting
with a virtual agent that would accompany them in AR.
We wanted a task that would keep users engaged
while also having an agent alongside them
that could give hints and help complete the task,
ultimately acting like an assistant.
For these tasks, what we did is we created
a hidden object game that ran on a touchscreen PC,
and we borrowed images from the I Spy series
of children's books to use as the puzzles.
How things would work is that a user would need
to find one object hidden in each image,
for example the cat shown here.
A user would need to interact with the agent
in order to learn what they are looking for,
and ask it for hints if they got stuck.
Then once the user found the object,
they would just tap on it to move on.
Our agents supported the four different hints
that would describe the object
which were context, color, shape and location.
We designed four different agents
for users to interact with.
The first, Ava, was a voice-only agent,
and so users did not see anything,
and so they just heard a voice through the headset.
The second agent, Zee, was made to mimic a smart home device
and stood on the desk in front of them.
The third agent, Mikey, was a miniature human
that also stood on the desk,
and our fourth agent, Jake, was a full-scale human agent
that sat at the desk across from them.
The goal was that these would give a sampling of the range
of different visuals for an agent,
two agents that mimicked existing devices,
and two agents that were embodied.
For our study, we recruited 24 participants
to interact with the different agents and find the objects.
Participants wore the Microsoft HoloLens one headset
and sat in front of an HP Sprout touchscreen computer
which had the hidden object game running on it.
We used the within-subjects design,
so each participant got to interact with all four agents,
and the order of agents was counterbalanced
with a Latin square.
Each participant would complete five different puzzles
with each agent, and they would fill out
a questionnaire after each set,
that asked them to rate their experiences with the agent.
To put it all together, here's an example of a user
searching for an object in an image,
while interacting with our agent, Mikey.
- Hi Mikey, what am I Looking for?
- Try and find the baseball player.
Let me know if you want a hint.
- Can I get a hint?
- Sure, what kind of hint do you want?
Context, color, shape, or location?
- Location.
- The baseball player is in the second keyhole
from the right.
- What's its color?
- The player is white and yellow.
- Okay.
During each trial, we measured how long it took for
a participant to find an object in an image,
and we also recorded their utterances,
so what they said an how often.
We also kept track of how often participants
gazed at the agent, which, due to the field of view
of the headset, we estimated by seeing if the agents
faced or if the agent was visible on screen.
Again, we asked participants to fill out a questionnaire
after each agent, and we asked them to explain
what they liked most and least about each agent.
And also at the end of the study,
to state which one out of the four
they liked the most and least, and why.
We also asked them to rate each agent along eight different
scales, which were adapted from previous studies
on agents, as well as AR/VR.
For some of these metrics, we would expect the results
to be the same across agents, such as task completion time,
or helpfulness, because the agents shared
the same intelligence.
Instead, they just had different visual representations.
For purposes of this talk, I'm only gonna highlight
the results that we found were significant,
starting with gaze.
For gaze interaction, we found an effect of agent
on gaze behavior, both in terms of the number of gazes,
as well as gaze duration.
For this, we only looked at the three agents
that could be seen, which were Zee, Jake, and Mikey.
Post hoc test with pairwise t tests
with Bonferroni correction showed that
our participants gazed more at the two human-like agents
versus the non-human agent, Zee.
And likewise we found that, overall each gaze was longer
in terms of duration as well.
Quite often how this behavior would play out
is that a participant would look at an agent,
ask a question, and then wait for the agent's response
before looking away.
Next, I'll talk about the questionnaire ratings.
Overall, for these we found differences
for three of the scales, which were
presence, relatability, and realism.
For presence, we asked participants to rate
on a visual analog scale, how much they agree
with the statement,
I felt like the virtual agent was in the room with me.
Statistical analysis and post hoc tests here
showed that the miniature agent, Mikey,
was rated as having a higher level of presence
compared to the Zee device,
which we found was kind of interesting,
given that they were both visually apparent,
and the same size.
For relatability, participants rated each agent
against the statement,
I could relate to the virtual agent
on a personal level.
Here, we found that Mikey was rated higher
than both Ava and Zee, in terms of relatability.
For realism, we have the statement,
I felt like I was interacting with a real person.
In this case, we found that both our human-like agents,
Jake and Mikey, were rated significantly higher than Zee.
Also note how the average rating for Ava
fell somewhat in between, but not significantly
different from the others, which I will discuss later on.
Going back, at the end of the study,
we also asked participants to rate which agent
of the four they like the most and least.
Based on their answers, we tallied up the number of people
who liked each agent the most,
and some people couldn't decide on one,
so their votes were split.
However, these don't tell the full story here.
We also had to take into account
the number of people who liked each agent the least,
in the form of dislikes.
And then we subtracted the dislikes from the likes,
to get a net score, which allowed us to see
the agents that were most preferred.
And then you can see here, Mikey was liked the most overall,
with Ava in second, but more polarizing
with an almost equal number of likes and dislikes,
and then Zee, followed by Jake,
which were both really close,
with more participants disliking them than liking.
Why did our participants
like the smaller Mikey agent the most,
and what does this mean for the future of agents in AR?
In order to answer that, we analyzed the open response
questions to get a deeper insight into participants'
perceptions of each agent.
According to one participant,
they liked Mikey because,
"he felt like a little helper that as in the room with me,
"wasn't too distracting, and was amusing because
"he was so small."
From our other results, recall that we also found that Mikey
had higher presence and relatability ratings
than Ava or Zee.
Based on these results, and participant comments,
this is somewhat related to a novelty effect
that was reported in a prior work,
where embodied agents were rated
as more engaging and novel compared to others.
Additionally, we also found that participants
gazed more at the human-like agents, Jake and Mikey,
compared to the smart device, Zee,
and most likely this is because
people tend to gaze at faces,
or just that there was more visual stimulus
with those two agents.
The take aways here is that this is good news
for having embodiments in AR.
One of the key benefits is being able to enrich speech,
through the use of nonverbal gestures and expressions.
What we saw here is that people
will gaze at the human-like agents,
which is kind of a necessary first step
for utilizing nonverbal communication.
Moving forward, we also saw that Mikey
was liked more than Jake by participants.
They had the same design.
Just one was smaller than the other.
Going to the comments, we found that
although participants felt Jake was more realistic,
he was also considered off putting and uncanny.
One participant wrote,
"Having a huge animated guy
"right next to you was a bit much."
Unfortunately, we concluded that Jake
fell in to the uncanny valley,
where he was still fairly realistic or human-like,
but not quite enough to be really acceptable.
But recall that Jake and Mikey both had the same design,
just different in size.
So, we saw that size really does have an impact here.
A tiny person, of course, does not exist,
so it would be considered less human on this scale,
but more likable, as it avoids the uncanny valley.
There could be other characteristics that one could explore,
for example, the design of agents or other factors,
such as appearance and movement.
In the end though, uncanniness is still an issue
that virtual humans face, even in AR,
but it looks like size can help mitigate that a bit.
And last, I want to talk about something we noticed
in the participant comments.
Zee, the smart device, had a lot of comments
about how robotic it felt, for example,
"Since it wasn't a person but an inanimate object
"it felt less like I was interacting with somebody."
That makes sense for the two human agents
based on the realism ratings that we looked at earlier,
where the human-like agents were considered
more real than Zee,
but what about the voice only agent, Ava,
which wasn't rated significantly lower than the other two
human-like agents, but instead, sat somewhat in between.
What made Zee come across as more robotic here?
Reviewing the literature, we came across the notion
of an agent having what's called a default image.
So, in summary, people would consider an agent to be human,
unless they had evidence to the contrary.
In this case, seeing really is believing.
And according to our participants,
this is most likely what was happening.
Here's one quote that really epitomized this concept,
"I did not see Zee's face as it helped me.
"The first assistant, Ava, had only a voice,
"which allowed me to imagine that it had a face and thoughts
"that were just not shown."
So, my take away here is that any non-human appearance
could be potentially worse than having no appearance,
unless the goal really is to make the agent
feel more robotic, or thus embody a different role.
In conclusion, do embodiments work for agents in AR
and should we have them?
We would say, yes, based on people preferring the smaller
embodied agent overall, and how the embodied agents
had increased gaze behavior from users,
as well as higher ratings
for realism, relatability, and presence.
Overall, we also showed how the visual representation
an agent has can affect how users perceive an agent,
such as uncanniness in terms of size,
and perception as humanist, as we just saw.
So, what's next?
In terms of future work, we're looking at exploring
different sorts of tasks, like real world scenarios
for AR to see how that could influence user perceptions.
You can think of using an AR headset with an agent
in everyday life or in industry,
or maybe like while driving.
We're also looking at how we can really take advantage
of this gaze behavior and the embodied agents,
to enable nonverbal communication in AR with agents,
possibly through behaviors such as mirroring.
Ultimately, all these things will help us reach our goal
of understanding how to create better virtual agents
for augmented reality.
And with that, I'd like to give special thanks
to my co-authors, Jessie Smith and Jamie Ruiz,
and thank all of you for listening.
(applause)
- [Host] Thank you, Isaac.
Any questions in the audience?
- [Audience Participant] Yeah, hi.
I guess I have two questions.
The first is more of a comment.
Between your scenarios, you varied at least two things.
You varied the embodiment and the voice,
so I have a problem with accepting
conclusions drawn to the embodiment
if you also changed the voice.
Maybe you can comment on this.
And the other thing, which is more of a question is,
I think it would be fun to have a little guy on a desk,
or like a little figure on the desk,
but I wonder what happens if you use this agent
in real world tasks.
I think at some point, you just want the thing
to do what you want, and you don't really care if
there's an embodied figure,
so I wonder if you thought about this?
If someone longterm uses an agent like this,
do they really care about the embodiment,
or do they just want to get their stuff done?
- Right, that's a really good question.
I did talk about this sort of novelty effect
that emerged, and again, we'd need more research
to see if that wears off eventually,
and you can definitely think about
how tasks would potentially affect
which agent the would prefer.
Again, we looked at distraction as one possible factor.
Having a little guy running around might be a little bit
distracting for something critical, like driving,
for instance, so definitely, that's an avenue
that we're looking at in future work.
That would definitely be interesting to explore.
- [Audience Participant] Hi, I'm Bjorn,
I'm from Uber.
I'm really happy to hear you comment and speculate
about some of these other tasks,
in particular driving, for obvious reasons.
I'm curious to hear about this more,
because it seems like the aspects of the agent
that would appeal most to the user
would depend a lot on the user's goals
in a given scenario, so this is sort of like a game.
Say you have a small amusing little helper,
it's kind of consistent.
I'm wondering if you could speculate a little bit more
about what kind of traits you think would be good or bad
in a driving context, or in something else,
like a typical Siri or Google Assistant task,
like just information retrieval.
- Sorry, could your repeat that last part?
- [Audience Participant] I'm interested to hear you
speculate about what specific traits might appeal
or not appeal to users in like a driving task
in an agent, but then also then maybe a more generic
information retrieval task,
like how many pounds in a kilogram kind of thing?
- I think for, at least for driving tasks,
we might expect one to favor that disembodied voice
a little bit more, based on not wanting to have anything
distract them from the road.
But possibly for those kinds of scenarios, you could have,
in terms of traits, I would think,
possibly having the agent wear clothes,
more casual clothing, or something that would fit
the scene of driving, for instance.
Maybe have them in a tiny little car?
Something like that.
Really, there's so much in terms of the design space
of just the visual appearance.
In our study we focused on these different classes,
big human, small human, smart device,
and disembodied voice, but really there's a lot more
research that needs to go into this,
and this is exciting 'cause we're just staring off
with AR headsets, and we'll move forward.
- [Audience Participant] Hi, Ferris Bolis, Volkswagen.
Less a question, more a suggestion.
On the scale of embodiment, you went from
torso to human quite fast, so maybe something
in between, something like an organic pet-like
block thing might be interesting to have the benefit of the
range-ability, but not the off side of distractions
or a lot of fixations of the human, so maybe
something in between might be interesting.
- Yeah, definitely.
I know in prior work, they've kind of looked at,
even gaze analogs, so a box with a nose could
be an anog for a head, for example.
And they've found promising results with that.
Something between, maybe a humanoid looking robot
may be interesting to explore.
Something between and that could definitely evoke
different sorts of feelings, and we'd need to see
what that's like.
- [Host] Let's give another round of applause for Isaac.
- Thank you.
(applause)