hi everyone thanks for attending my talk
my name is andrew anderson i'll be
presenting our work mental models of
mere mortals with explanations of
reinforcement learning
so in the title i mention these mere
mortals
it's fair enough to ask what do i mean
by that um
i'd like to introduce you to alex who's
a domain expert
so they're capable of doing things like
flying planes calculating taxes
in our specific case they're real-time
strategy game players
it's important to note that they are
not ai experts
so they don't have any formal knowledge
in ai or how it makes its decisions
which got us wondering which of two
explanation types might work for
such people would it be a saliency
explanation so where the ai is looking
when it's making its decisions
or a reward explanation some calculated
future score
for an action it takes just now for
those not familiar with real-time
strategy games
this is starcraft 2. it's a very busy
environment and there's a lot of
opportunities to miss
information we wanted to make sure that
our participants didn't miss any
information
so we designed our own game in this game
the ai makes decisions
those diamonds along the bottom and
for those decisions the ai is
controlling this kite-shaped tank in the
center of the map
and it has to decide which of four
objects or four quadrants it wants to
attack
so in this case would it attack its
allied objects those white objects on
the right
or its enemies those objects on the left
so once it goes through and makes that
decision it has to destroy that object
or be destroyed
and if it destroys the object it
continues
to make decisions until it's destroyed
so what about the the treatments for our
study
we had four treatments in an ablation
style study
where the control just had access to the
game map
and the saliency treatment had access to
both the game map
and the saliency maps which you can see
in the middle the rewards treatment had
access to the game map
and the reward bars which you can see on
the right and
for the everything treatment we gave
them the game map and both
explanation types our research questions
were as follows
RQ-Describe describe which treatment is
better and how it enabling people to
describe how the system works
so they're forming a rule book in
their head of in the form of a mental
model
RQ-Predict it is another
measurement of a mental model
which treatment is better and how it
enabling people to predict what the
system will do
and then when they're predicting how are
they doing so what information about the
agent or ai
do people draw upon to make their
prediction
for the purpose of this talk i'll be
dropping RQ-Describe
so let's get started with RQ-Predict
at each decision our participants would
see the map
without any explanation and they would
have to select the object that they
think the ai would attack
if they didn't get it correct we gave
them zero points
and if they did get it correct we would
give them one point
so from this from this graph all i
really want you to see
is that the there was a lot of variation
in people's accuracies
across the 14 decision points so
from this we had to derive patterns
such as these three decision points
where the control those green bars on
the left were actually the most
accurate compared to any of the other
treatments
so why was this so we started looking at
some of their responses
and this is an everything participant so
those blue bars on the right
and they're juggling a lot of
information they're saying you know i
think it considers this thing
and this thing and then it does this
thing
so it's a very detailed
attempt at understanding what's going on
whereas the control participant here
is using a more intuitive sense you know
this object has the lowest health of all
the enemy objects
this was actually something that some of
the participants bemoaned at the end of
the
the task you know it was confusing all
around to figure out the main factor
for movement using the maps and bars in
this participant's case
which is understandable because in the
bottom left
that is the interface that they saw
so the next attempt we had to try and
gain understanding is from all the
decision points we had to find
patterns in these three cases
the presence of the enemy
kite-shaped tank is the common theme
throughout them
and what we actually saw was
that the participants who were
who saw reward bars
which are these bars here on the
right
they are more accurate at predicting
what the agent will do in these three
cases
than the participants who didn't see
reward bars so these bars here on the
left
so let's do a more formal understanding
of
you know how are they making their
predictions so this is an example of the
interface
that we that the participants saw
before they saw any explanations and
they had to predict
so perhaps they're just using
information that's currently available
some again that intuitive sense of
what's going on
perhaps they're bringing in
information from prior
game states or maybe they're even trying
to predict what the explanation would
look like
before they see it so i want to focus on
this
last point first for predicting the
explanation
so this is the number of times
that participants used
the this kind of prediction technique
to try and understand what the agent
would do
and all i want you to see from this is
that
the rewards and everything treatments
are using it more often than the
saliency
participants so this is
the box plots of those that same
information
and what we actually see is that when
we run
welch's anova the the p value is very
very small
so the saliency participants down here
are actually using it significantly
less than the rewards or the everything
to participants
so this is an example of how they're
predicting these explanations
this rewards participant is trying to
say okay this object will provide more
damage rewards and most destruction
rewards which were two
types of rewards in our domain
but andrew i hear you ask you know and
the everything
participants saw both explanations so
which are they trying to predict
the truthful answer is some of them
did both you know that
it will look at its friend or enemy
then destroy the one with the highest
value
so the look is the salient in the
highest value is the rewards
but 91 of all their predictions were
just about the rewards without
mentioning saliency maps
so that was interesting and what about
using the current game state
what we actually noticed when we when
we
looked at this table was that the
rewards and the everything participants
were always
using the current game state the least
of the four treatments
so again we chart this
and we start seeing okay this is
oddly lower than the others
and when we run the ANOVA
we get ap value less than .05
but it doesn't tell us which one exactly
so once we remove the rewards
participants and we run the same test
because the significance disappears that
means
that the rewards participants are using
the current game state significantly
less
than the other three treatments
so throughout this paper here are some
of the things we found
we found that counter-intuitive
situations are in fact difficult
but the participants with reward bars
did show some advantages
we found that one size does not fit all
participants which we saw with
larger error bars or situations again
that very behavior
what we did see was that diverse people
had
diverse reasoning for their predictions
and that warrants further investigation
and we also need an arsenal of empirical
methods both quantitative and
qualitative to hope to explain
reinforcement learning to mere mortals
thank you again for donning my talk i'll
be happy to take your questions