My name is Katy Gero and I'm excited to present our work on mental models of
AI agents. This is work that I did while interning at IBM last summer and I was
part of a really big and amazing team so I just want to take a moment to
thank everyone who contributed to this work and just point out that I'm
presenting on behalf of a lot of different people. So we were really
interested in mental models of AI agents. Mental models are essentially the way
that individuals think systems work. So they're often studied in science
education where the student has a mental model of the thing that they're studying
for instance the atom and this mental model is unique and individual to every
student and it might have some things that are wrong, right, as we're learning
we might misunderstand for instance how the electrons exactly are rotating
around the nucleus and in science education they distinguish between the
mental model which is the student's individual understanding of how
something works and the conceptual model which is the scientific consensus of how
the atom actually works and so in education we're obviously
interested in helping students develop their individual mental models to become
closer and closer to the scientifically accepted conceptual model. But mental
models are also studied in design, perhaps most famously by Don Norman who
did a series of very seminal experiments on how people thought calculators worked
and so individuals who are using the calculator had their individual personal
mental model of how the calculator worked and I for one can attest that
when using calculators I often didn't quite understand what the Clear button
was meant to do or how it worked with caching and so oftentimes when I was
gonna start a new calculation I would hit the Clear button three four maybe
even five times to make sure that I had fully flushed out the memory. Of course
this betrays a lack of understanding of how the Clear button actually works on
calculators and again in design we distinguish the mental model which is
the individual's understanding of how something works from the conceptual
model which in this case comes from the designer or engineer who has a really
good understanding of exactly how the calculator does work. Don Norman in his
research found that mental models are incomplete and limited that they're
unstable and unscientific, we're often superstitious about things, but they're
also parsimonious and they lack firm boundaries so we're always drawing on our
experiences of similar devices. Now this kind of paints a bad picture of
mental models but actually they're incredibly useful they're the way that
we interact with an incredibly complex environment and mental models value
utility over accuracy. As a calculator user I don't have to perfectly
understand how our calculator works in order to use it in a way that's
extremely functional. Now in our work we're interested in mental models of AI
agents. We think studying mental models of AI agents is incredibly important
especially as AI agents and AI systems are starting to be used in really high
stakes environments like in medical decision making or policing or hiring
decisions and of course the people who are interacting with these systems have
mental models of the systems they're interacting with and as like all mental
models they probably have some inaccuracies about them so understanding
how people develop these mental models and where they fall short is a really
important part of figuring out how AI systems are going to be used in the
world. Now we put the image of the AI agent in the middle of the science
mental models and the design mental models because there are some distinct
differences. The conceptual models in science come from discovery right they
come from research and actually they can change over time so our conceptual model
of how the atom works has changed over time. That's a little bit different than
in design where we're creating the artifacts that we're studying and so as
a designer or an engineer we probably have a very solid understanding of how
calculator works from the fact that we built it. AI agents are somewhere in
between we argue because on one hand yes we build the AI agents ourselves but
there's also a little bit of discovery involved in understanding how it
actually behaves and we'll get back to this in a second. Okay so we wanted to
study mental models and if we want to study mental models of AI agents we're going to
need an AI agent that we're gonna study so we decided to use an AI agent that
would play a game. Games are a long-standing testbed for AI systems
and the particular game we picked has some really nice features so let me go
over how this game works. It's a two-player word guessing game one player
has a secret word in their head that they're trying to get the other player
to guess and they give a series of one-word hints so for instance they
might give you the hint salad you think okay maybe they're trying to get me to
guess the word lettuce, so if that's not right they just they can give another
hint so now their hint might be red and you go okay salad red maybe they meet radish
if it's still incorrect we get more hints now they give the hint fruit and
you go oh I get it salad red fruit they must be thinking of
tomato and then if you win a dialogue box pops up and says you win yes the
target word was tomato. One thing that's important to note is this is a
collaborative game so winning is the same for both players they both have the
same goals which means we don't have to deal with any of the deception issues
that come with other kinds of games but another important point is this is
really a game about developing mental models it's really about understanding
what your partner is thinking so we thought it was a really good testbed to
start to probe how people develop mental models. So this is how the gameplay works
next I want to show you how the AI agent that we developed works to play
this game. Okay so essentially our AI agent is a trained neural network we
call it wordbot we don't really have time to go into all the specific details
of the architecture but suffice to say it has two main inputs it has the game
state which is all the information about the game so far so it would include the
hidden word if it's giving hints the previous guesses for this particular
game and any previous hints for this particular game and then it also takes
as input a bunch of information from a knowledge base we use conceptnet which
is essentially a large knowledge base of common-sense information, based on these
inputs the AI agent outputs what it thinks the next hint should be. This AI
system is trained in a reinforcement learning context so actually technically
we have one AI agent the plays as the hint giver and one that plays as the guesser
but for the rest of this talk we're just going to talk about the AI agent as the
hint giver. Now going back to this idea of how exactly do we develop conceptual
models of AI agents we developed this model so we had a
really in-depth understanding of exactly how it worked
how was trained the data it was trained on etc but still we have to do a bunch of
analysis to really understand its behavior because sometimes we couldn't
predict how it would behave purely from the design of the system so to give you
an example of that I want to talk about a subset of words that the AI agent tended to
get pretty bad hints on which was city names and so if we look at the example
of Paris it gave pretty bad hints but you might think that this is because it just
doesn't have enough information about Paris maybe it's not available the
knowledge base or in the training data but actually it does have a lot of really
good information about Paris this is just a small subset but it knows that it
has the Eiffel Tower and the Louvre that has lots of cafes that it's in France so
this is plenty of good information for lots of good hints however if we look at the
hints given they're pretty terrible there are best misleading and at worst
they're they're very confusing so what we want to point out here is that in
order to study mental models we have to first understand the conceptual model
and in AI agents this takes a little bit of the designer approach where we have
to understand how it was built but also a little bit of the scientific approach
we have to discover how it behaves through analysis. So once we developed a
really good conceptual model of our AI agent we ran two studies to try to
understand how people are developing mental models of how it worked. The first
study was a small-scale think-aloud study where we brought participants into
the lab and we watched them play a number of games with the AI agent asking
them to think aloud through their process and then we did semi-structured
interviews with them so we collected a lot of qualitative data about how people
were thinking about the AI agent. We used all this qualitative data to do a
thematic analysis which resulted in ten emergent themes that we coded all the
transcripts with. We don't have time to go into all ten themes in this talk we
encourage you to read the paper but we'll just talk about one of the most
prevalent ones which had to do with anomalies and distress. This was when the
AI agent did something unexpected and the participant remarked upon the fact
that something unexpected happened and how they responded to it. There seem to
be two main types of responses: on one hand some people would respond to
something unexpected by saying, Wow I'm so stupid because I don't understand why
the AI agent did this thing for instance give a hint that didn't make sense, on
the other hand some people responded by saying the AI was so stupid that it
might be broken or making a mistake because it did something that didn't
seem to make any sense. What's interesting is that sometimes both of
these responses were incorrect, sometimes the people who felt like they were
stupid were actually experiencing the AI agent doing something that didn't make
any sense, on the other hand sometimes people who were quick to blame the AI
agent for making a mistake actually weren't understanding the logic it was
using to provide a hint for instance sometimes the AI agent provided antonym
hints which at first seemed confusing but upon reflection was an OK strategy
to take. And it was in the face of these anomalies that people were most likely
to reflect on and sometimes revise their mental models, and so one of the findings of
this study was that people are most likely to change their mental models
when something unexpected happens. The other thing that came out of this study
was a framework for how people generally were thinking about how the AI agent
worked. This framework has three main components, the first is global behavior
for instance people wondered if the AI agent was learning from their past games
and customizing it to them personally, the second part is knowledge
distribution people seem to be aware that it's possible does the AI agent
would know more about food than pop culture or more about certain places
than others, the third part is local behavior which had to do with how it
made an individual decision, why it gave a single hint. Now these three categories
have very specific interpretations for our particular AI agent but we actually
think it's a really useful framework for all kinds of different AI systems for
instance let's say you have a sentence classification tool, global behavior
might be whether or not it's continually learning from new examples, knowledge
distribution might be if it's equally accurate on all types of sentence inputs
and local behavior might have to do with what features of the sentence it's most
paying attention to when it makes a single classification. Using this
framework we were able to develop a survey that could
probe people's mental models of the AI system and this allowed us to do a large-scale
online study where we could have people play the game online and fill out this
survey so we didn't have to bring a ton of participants into the lab. For this
study we recruited over a hundred participants from Amazon Mechanical Turk
and collected a bunch of demographic data to make sure that none of our
findings were confounded by anything like age or education level. We looked at
two main variables in this study, the first was the number of games played, we
hypothesized that more if they had played more games they would have more time to
develop a better mental model of the system, the second variable was win rate
we suspected that people who won the game more often would have more accurate
mental models and that this would allow us to probe what was getting those
people to those accurate mental models.
When we looked at the results we didn't
find any significant differences between the number of games played, this is
perhaps because the difference simply wasn't large enough and in future work
we're interested in looking at far more games played for instance 30 50 or even
100 games played perhaps over many days where people could really start to
adjust and learn from the gameplay. However we did see some significant
differences when we looked at the win rate so to look at the win rate
we wanted to compare people who won the game most often to those who lost the
game most often so we split everyone up into two categories the winners who won
more often and the losers who lost more often so in this table I'm going to go
over just some of our survey questions and each survey question posed a question
about what they thought the AI agent knew or could do and then they answered
it on a Likert scale so we're also showing here what we think a correct
answer would be based on a strong understanding or the conceptual model of
the AI agent. So first I want to show the results for knowledge distribution as
you can see here there actually weren't any significant differences between
winners and losers about over the knowledge distribution this is likely
because there in the number of games that they played they simply didn't have
enough exposure to all the different kinds of categories of words but we do
see that people were sensitive to the overall trend that the AI agent did know
more about some things than others so this just goes to show that people
really were sensitive to the things that were going on with the AI agent and it
wasn't just kind of random or people were totally making up their answers.
However when we looked at global behavior we did see some differences so
these are two questions we posed about attributes or abilities of the AI agent
that the AI agent actually did not have and we found that the winners were more
likely to say that the AI agent didn't have or wasn't able to do those things
than the losers so the losers were overestimating the AI
agents abilities but actually in both cases people were overestimating the
abilities the winners were just more likely to be closer to the real answer.
We did some follow-up surveys to try to understand exactly why this was and
losers were likely to say to actually acknowledge that they didn't have
evidence for their claims so they would say, You know I didn't see any evidence
that it adjusted its hints but I believe that that's something it should be
able to do. So this was a pretty interesting result even though we didn't
exactly see what was encouraging accurate mental models we did start to
see the seeds of why people had inaccuracies. So to quickly summarize we
found that people were most likely to develop and improve their mental models
when responding to something unexpected that the AI agent did and we saw that
people tended to overestimate their the AI agents abilities even in the face of
no evidence. I want to thank everyone for listening to this talk, there's a demo of
the system online and I want to say thanks again to the big amazing team
that worked on this with me.