Hi there. My name is Mahsan Nourani, and on behalf 2 0:00:06.000,0:00:10.640 3 and User Reliance in Explainable AI Systems, 4 Metaphorically --> machine learning models are referred to as black-boxes due to their
lack of transparency. These models 6 0:00:20.880,0:00:26.320 7 predictions, Following the same metaphor --> 8 researchers propose opening the box and making the model’s decision-making more transparent
by providing explanations. With the Explainable 10 0:00:38.320,0:00:42.640 11 how their actions affect the model outcomes, 12 This is also known as human’s mental model of how the algorithm works. In this paper,
we focus on user mental models in XAI systems. To 14 0:00:55.920,0:01:02.640 15 and strengths as they use the system, However --> 16 in our previous research --> we observed that the order of encountering errors can influence
impression formations and trust. In that regard, 18 0:01:14.720,0:01:20.880 19 had problems calibrating their trust over time, 20 On the other hand --> with positive first impressions that were caused by seeing
model strengths early on, users tended to have 22 0:01:32.560,0:01:37.120 23 impressions on mental model formations --> 24 we formulated two high-level research questions: How do positive and negative first impressions
affect user mental models? Can explanations 26 0:01:53.040,0:01:57.280 27 the video activity recognition domain --> 28 which has many real-world applications such as security surveillance.
Specifically, we focused on activity recognition 30 0:02:07.280,0:02:12.880 31 each activity as a combination of actions --> 32 objects --> and locations, For example, here the person is moving pineapples to a plate, with
move as the action, pineapple as the object, and 34 0:02:26.720,0:02:31.840 35 whether an activity takes place in a video, 36 The model is able to answer the query and provide explanations to justify its rationale.
Here is an overview of our XAI system interface. 38 0:02:45.200,0:02:50.560 39 actions --> objects, and locations from a list 40 of defined vocabulary, When the user searches for an activity, the the system would categorize the
available videos based on whether the activity 42 0:03:02.560,0:03:08.640 43 model found cutting cucumber is taking place --> 44 while the videos on the bottom are those the model did not find the activity in.
Upon clicking on each thumbnail, a more 46 0:03:18.800,0:03:23.680 47 model’s explanations for its prediction, 48 For the explanations --> the model shows the segments of the video used to find the prediction. For each
of these segments, the model shows it’s top-3 50 0:03:37.040,0:03:43.680 51 designed a hypothetical scenario where the kitchen 52 staff had to follow a set of kitchen policies and guidelines, and user’s task was to understand
whether each of these policies are being followed. 54 0:03:56.320,0:04:01.920 55 4 policies to reflect system’s strengths --> 56 and 4 policies to reflect system weaknesses, We altered the order of the policies to control
the order of observing system capabilities 58 0:04:13.280,0:04:20.240 59 variables, The first variable is policy order --> 60 where half of the users observed weakness policies at the top of the list while the other half
observed the strength policies at the top. We also 62 0:04:32.480,0:04:38.240 63 policy verification task to build a mental model 64 of the system --> where we also measured usage and human-machine task performance. After completion
of the main task, we measured participant’s mental 66 0:04:50.800,0:04:56.560 67 predict model’s behavior on different inputs, 68 Our results from the main task indicate that while participants with positive first impressions
used the system significantly more, they 70 0:05:08.720,0:05:15.040 71 the users tended to rely heavily on the system --> 72 even when the model was wrong, On the other hand --> with negative first impressions, participants
reported they used the system significantly less 74 0:05:26.880,0:05:33.840 75 this study --> we used 2 mental model questions, 76 For the first question --> we selected 8 activity components that could be actions/objects/
or locations, with half of them being model 78 0:05:45.200,0:05:49.600 79 model’s detection accuracy for each of 80 the 8 components, Also --> we asked users to rate their confidence in their estimation.
Our results from the first mental model 82 0:06:01.520,0:06:06.560 83 strengths while being more confident about their 84 estimations with positive first impressions, Their counterparts, on the other hand, significantly
underestimated the model capabilities while 86 0:06:18.960,0:06:23.680 87 impressions were significantly more skeptical 88 of the model capabilities while not being so confident about their skepticism.
In the second mental model questions, participants 90 0:06:36.560,0:06:40.320 91 model would match the frame to the query, 92 They were also asked to rate their confidence, The user’s aggregated predictions showed an average of
around 59%, indicating that the estimations 94 0:06:54.400,0:07:00.320 95 showed significant differences, First --> without 96 explanations --> those who had a positive first impression were significantly more confident
about their estimations. For those with negative 98 0:07:12.240,0:07:18.320 99 results --> guessing with high confidence can be an 100 indication of over confidence, This overconfidence took place in two situations: first, with positive
first impressions and without explanations and 102 0:07:31.440,0:07:38.080 103 positive first impressions or explanations can 104 lead to user’s over confidence in their mental model. So the main take ways from the study are:
Addition of explanations was not enough to 106 0:07:53.200,0:07:58.160 107 understand how the model works when --> in fact, 108 they do not, We demonstrated that anchoring bias can affect over confidence, user over or under
reliance, and formation of mental models. 110 0:08:11.120,0:08:18.080 111 overconfidence and mis-reliance can be dangerous, 112 Thus --> future research efforts can propose techniques to mitigate the human biases through:
1) Making users aware of their biases, 2) Preventing 114 0:08:33.040,0:08:40.080 115 For more information and details about the system --> 116 study --> and the, Results, please refer to our paper. With that, I would like to take
a moment to thank all my co-authors from both 118