Hi.
I'm Zahra Ashktorab.
And I'm presenting the work we did on Human-AI Collaboration
in a cooperative game setting--
measuring social perception and outcomes.
HCI researchers have done a lot of work
on determining what can lead to successful collaborations,
and outcomes, and human-AI collaborative settings.
One factor that has been explored
are social perceptions between collaborators.
As we move into a world of human-AI collaboration,
these perceptions on how they impact
the outcomes of the collaboration
are not yet well understood.
In this work, we study the differences between human-AI
and perceived human-human social perceptions
in a collaborative interactive setting.
One opportunity for studying user perceptions of AI agents
are cooperative partially observable games.
Cooperative partially observable games
require players to cooperate with one another,
especially since there are aspects of the game that
are only partially observable.
These games require users to coordinate, cooperate,
and communicate with an AI agent in ways
that their actions are deemed interpretable.
One example of a cooperative game
that has been studied by AI researchers
is Hanabi, as pictured here.
We created a cooperative partially observable game,
and we studied the collaboration between humans and AI
in the context of this real-time interactive game.
Due to the cooperative nature of the game
and the clear notion of success in the setting,
the game afforded us a unique opportunity to study human-AI
collaboration.
And specifically, we were able to study
how social perceptions differ, when
players think they are playing with an AI, versus when
they think they're playing with a human.
Our work is not interested in the intentional
anthropomorphizing of a system but
the effect of identity disclosure,
and the effect of that identity on outcomes,
and social perceptions of the AI or human partner
in the setting.
So here, we can see an example of gameplay, in which the AI is
providing the clue laboratory.
The first guess is science, and then
stimulus is provided by the AI, and the word was experiments.
So you have several research questions
in which we are investigating.
First of which, how does a partner's
belief in whether they are interacting
with an AI or a human affect social perceptions
of their partner?
How does a partner's belief in whether they are interacting
with an AI or a human affect game performance/outcome
of the collaboration?
How does a partner's confidence level
affect social perceptions of their partner?
And how does the partner's confidence level
impact game performance/outcome of the collaboration?
Each participant played five rounds
with five different words.
For each round, participants were allowed
a maximum of 10 guesses.
If they did not guess the target word correctly
after 10 attempts, they lost a round
and moved on to the next target word.
We looked at how the following factors impacted
user perceptions of their opponent,
whether participants were told the partner was
an AI agent, a human, or undisclosed, and
their partner's confidence-- consistently high,
consistently low, none displayed, or mixed.
After playing the games, users took
a survey in which they reported on their perceived
intelligence, perceived rapport, perceived likeability,
and creativity, as well as an AI score,
in which we measured the degree to which they
felt that their partner was a human or an AI.
Due to time constraints, we will focus on our results
around perceived intelligence.
A multiple regression was calculated
to predict intelligence based on AI
score of the partner, the assigned role,
and the confidence of the partner,
as well as the demographic variables.
In the conditions in which partners
were told their partners were human and in conditions
in which participants were not told definitively
whether their partners were an AI agent or a human,
participants found their partners
to be less intelligent.
Conversely, we found a significant main effect
for AI score, showing that the more a participant believed
that their partner was a human, the more intelligent
the partner was perceived.
Interaction effects show that when individuals believe
that they are interacting with a human
and we disclosed that they are interacting with a human,
or did not disclose at all, they found their partners
to be more intelligent.
There was no significant effects for gameplay outcome.
When participants believe they are interacting with a human,
they are more likely to find it to be more intelligent,
likeable, and have more rapport, and to find it more creative.
When participants are told they are interacting with a human,
and they believe they are interacting with a human,
they are likely to find their partner more intelligent
and have more rapport with their partner.
Gameplay outcome was not affected given the perceptions
of the AI identity.
If we take a closer look here, we
can see that, when the AI agent is perceived as a human,
it is more likeable, it has more rapport,
and it is more intelligent.
We also asked participants reasons
they believe they were interacting with the human-AI.
And the reasons for human included high quality clues,
low quality clues, feeling of understanding,
adapting to guesses, and speed.
When they perceived the partner to be an AI,
there was indirect clues, lack of adaptation, and a speed.
For example, one user who perceived their partner
to be a human, said, that he was able to leap into correct
answers and seemed to adopt his hints
to match my guesses somewhat.
There is also, those who perceived their partners
to be an AI.
I don't think a human's first association to plastic, which
was a target word, would be a metal.
Thank you for your time.
I'm looking forward to the discussion following
this presentation.