work in hci and ai based explanation
systems
has uncovered an interesting
contradiction
people prefer and learn best from why
explanations
but expert esports commentators
primarily answer what questions
when explaining complex behavior in
real-time strategy games
in this work we find this is likely
because why explanations are cognitively
demanding
process during a live match our story
begins with explainable ai
which is the ability for artificial
intelligence systems
to explain behavior to external
observers
to curate explanation strategies for
explainable ai systems
recent work by dodge and colleagues at
kai
studied how expert esports commentators
called shoutcasters
forged for information synthesize
observations
and present explanations to their
audience during real-time strategy game
broadcasts
they began their inquiry with existing
work
on intelligibility types these types
categorize approaches to explaining the
operation
of complex systems based on different
types of questions
an end user might ask foundational work
on intelligibility types
show that users learn best from and
prefer why
explanations however dodge and
colleagues find that human shoutcasters
overwhelmingly answer what questions and
why type explanations
are their least common so
why is this the case there are several
possible answers
first it could be that well-informed
audiences
who tune into esports events don't need
y explanations
because they can already predict expert
level play
second it could be that why explanations
are beneficial but require too many
cognitive resources
for the audience to process in the
context of a live match
finally it could be that audiences can
process why
explanations but they are underserved by
human shoutcasters who do not have the
resources to generate why
explanations in real time in this work
we empirically explore these
possibilities and find
evidence that supports possibility too
that why answers are cognitively demanding
for even expert audiences to process
during a live match
for the experiment we built an online
platform to stream a professional
dota 2 match to our study participants
dota 2
is a popular online multiplayer strategy
game with a large
professional player base participants
access our online platform through their
web browser
to test our three possibilities we split
our participants
into three groups group one watched a 10
minute recording of a professional dota
2 match
group 2 watched the same 10 minute
recording but were also presented with a
map of the dota 2
play area on this map icons appear
during 12
important events that happen during the
clip
when an icon is clicked a simple what
style explanation
of a recent event is given in text
this explanation simply summarizes the
event providing no further analysis
group three also receives the minimap
and icons but their text contains
specially curated why
explanations to generate why explanations
we devised a curation task and recruited
three
expert dota 2 players dota 2 rates skill
and matches players together based on a
metric called mmr
which is similar to the elo system used
to rate chess players
here is a distribution of nearly 4
million players
across mmr tiers in season 4 of dota 2.
the average
mmr of our three experts was 6540 which
places them in the immortal tier this
tier comprises the top
0.78 of dota 2 players
we told our experts that a why explanation
connects two or more
actions or states in order to explain a
causal sequence
we then asked them to create why
explanations for each of our 12 events
after collecting responses we coded each
sentence
and accepted full responses with the
most common features between our experts
[Music]
to collect participants we used the
prolific crowdsourcing platform
our participants needed three qualities
to be considered for our study
recent experience playing dota 2 english
proficiency
in a desktop or laptop computer to use
the experiment platform
after finishing their session
participants fill out a series of
questionnaires
first they are asked to rate the mental
effort they expended while watching the
match on a nine point scale
second we had a simple quality check
about what teams participated in the
match
and how long the video was finally we
asked each participant to answer seven
multiple
questions about events in the video each
one
is a what question about gameplay events
and the answer is always
a particular player the choice options
for each question
are the list of 10 players that
participated in the match
given our three possibilities and three
groups we expect to observe
one of three outcomes
if possibility 1 is the case we expect
either no difference in recall
or higher performance in group 1. this
would be because
explanations do not help or possibly
even distract
informed viewers if explanations
distract viewers
we would expect higher reports of
cognitive effort from groups two and
three
if possibility two is the case we expect
group two to outperform the other two
groups one the recall task we also
expect higher cognitive effort from
group
three if viewers benefit from
explanations but why explanations are
harder to process in real time than we
expect the group receiving what
explanations
to score the highest and the group with
y explanations would report higher
cognitive load
finally if possibility 3 is the case we
expect
group 3 to outperform the other two
groups on the recall task
if this is the case viewers are being
underserved by shoutcaster and can
benefit from real-time why
explanations
first we analyzed the recall scores we
performed a one-way between subject's
anova and found a significant effect for
the three conditions
a post-hoc test showed that the what
explanation group performed
significantly higher than the no
explanation group
the why explanation group lays somewhere
in between neither being significantly
higher than the no explanation group or
being significantly lower in the what
explanation group
next we analyze the cognitive effort
reports we performed a second
one way between subjects anova and found
a significant
effect for the three conditions a
post-hoc test
showed that the why explanation group
reported significantly higher effort
than the no
explanation group the what explanation
group lays somewhere in between neither
being significantly higher than the no
explanation group or significantly lower
than the why
explanation group
together these results support
possibility too that explanations are
beneficial to informed audiences
but processing why explanations in real
time is a difficult task
the most exciting aspect of this work is
that it opens up new possibilities for
ai systems that dynamically explain
complex events this study was undertaken
by the weaver project whose goal is to
provide
automated personalized stories to
audiences during a live esports match
the results of this study are exciting
because they show that by simply
directing a viewer's attention to
important events and providing short
natural language summaries
of game events we can add value to a
viewer's experience
future work could test this effect in a
large scale production environment or
test
other aspects of audience psychology for
use in automated real-time explanation
generation
my name is justice robertson and i want
to thank you for attending
from myself my colleagues at the
university of york
and at weaver