Hello, my name is Jeremy Block and I'm presenting this work on behalf of my collaborator Sina
Mohseni and our advisor Eric D Ragan. In this work we propose a novel evaluation methodology
for artificial intelligence explanations that balances the trade-offs between
running subjective user studies and standard computational evaluations. Let me explain:
machine learning makes incredible predictions and provides excellent support in numerous fields
these techniques take data and communicate the patterns that they recognize often these outputs
are provided without context leaving the user to either fully trust the magical black box,
or justify why the system made a mistake. In decision-making tasks, like detecting cancer,
it's easy to imagine how these automated techniques benefit when they are more transparent
and can explain themselves. So, researchers have suggested numerous ways a model could explain how
it determines a classification. Machine learning is applied in a variety of ways and there's no
standard way to evaluate or compare the goodness of one technique to another. One fairly common
approach uses saliency maps to show what parts of the data are most important in a classification.
This technique is fairly popular and generally evaluated against using some hand-annotated region
of interest. This kind of single-layer ground truth pixel-by-pixel evaluation is computationally
straightforward but fails to accommodate human factors like user satisfaction their trust or
understanding too. On the other hand, often the only way to capture these human factors is to run
large-scale user studies where you ask users about their satisfaction, trust, and more
while they work with your explanation technique. This helps capture descriptive interpretations but
is challenging to generalize beyond the specific explanation type and scenario.
So, there is a trade-off between these two approaches: computational methods are
considered objective because they evaluate the model against some baseline ground
truth and have repeatable processes. Yet, they generally fail to quantify human
interpretations. On the other hand, when we ask human subjects to review a model's explanation,
we may get more descriptive interpretations but this can be costly,
imprecise, and subjective to the task; not to mention there are learning effects when
a subject is exposed to an explanation, so repeat evaluations are not really feasible.
Still, we want to find a reliable methodology that can provide both simple computational methods
with the intricate, detailed subjective interpretations.
Therefore in this paper, we propose an alternative approach in which multiple human annotators select
the most descriptive parts of a classification and the intersection of those selections helps
to generate a gradated ground truth that we can then use to evaluate our model explanations.
This gradated map becomes a baseline and helps with evaluation because we can now
see the areas people pay the most attention to with higher granularity than the typical
single layer ground truth baseline. And since multiple users review the same image
this image becomes a metric we can repeatably test against with future explanation techniques;
thus results are reproducible. But, as is the case with any new methodology, how can we verify that
it's better than previous techniques? We attempt to compile some evidence. Our paper considers two
user studies: one to demonstrate the technique, and the second to evaluate the performance. First,
to compare how our technique applies in different domains we use samples from two popular image and
text data sets. We selected only a handful of samples to show how our methodology does not
require many annotations to effectively evaluate a new explanation technique. The images are nicer
to view in a presentation so we focus on them here but the same techniques apply to textual data too.
Let me explain the steps we took to create our benchmark. We started by collecting annotated
data from 10 Amazon Mechanical Turkers in both text and image domains people were
given a series of images or text documents and were asked to lasso or select words that they
felt were descriptive of the target class. We then aggregate and normalize the annotations
to arrive at a gradated mapping of the areas, or words, people expect an explanation to cover.
This is meant to demonstrate where the most attention is paid by human evaluators,
so we refer to these as human attention maps. This is a simple technique and the foundation of
our methodological contribution. To evaluate how these multi-layer human attention maps compare to
the usual single layer techniques popularly used in other benchmarks, we run a second user study.
In this case, we asked 200 different Turkers to score images from 1 to 10, based on the saliency
map's goodness. The explanation, in this case, was another popular multi-class saliency map technique
called Grad-CAM. Grad-CAM generates a heat map that represents the areas most important in
determining a target classification. Comparing the judgment scores for the Grad-CAM technique
to the scores we would expect for the human attention maps should reveal some interesting
differences about how users behave and what they expect in machine learning explanation.
So, let's talk about the results of using this technique. First, we examine how our
human attention map compares with the standard single-layer ground truth map without considering
judgment scores. In this case, each point is an image with the Grad-CAM saliency map overlay and
the position is determined by the mean absolute error between the pixels covered by the saliency
map and either baseline. Here, with all the points hovering around the ideal regression (signified
by the black dashed line) it appears that the amount of error between the grad cam technique
and the multi-layer human attention baseline and the error between the grad cam technique and the
single-layer baseline is similar. This shows that both baselines have a high degree of overlap, yet
our regression test for homogeneity with the ideal line fails: meaning that they don't necessarily
contain the same exact information. With a slope just under 1, we have evidence that the gradated
human attention maps capture different information than the single-layer ground truth methods.
This is good news for our approach because it appears to have less error than the single-layer
baseline when using the Grad-Cam technique. But we also want to know how these techniques compare in
terms of the subjective ratings they received. In this case, we are comparing the Grad-CAM technique
(the blue data series) with the human attention multi-layer overlays (orange data series).
the human judgment scores are averaged across the number of raters and normalized to a scale
from 0 to 1. Again, the mean absolute error is calculated, but this time the error is
only with respect to the single-layer ground truth masks. There are a few points of interest in this
graph. Perhaps most obvious is the lack of ratings below 0.03. It appears that there is a floor
effect where just having a heat map on an image encourages users to rate an image higher than the
absolute minimum available. Also, the data is much more distributed than the previous graph.
Inspecting points at the extremes reveals that these images tend to have multiple occurrences
of the target object, like the cat we've been reviewing. In the Grad-CAM segmentation map,
we can see that the back of the top cat is highly activated, whereas the cat to
the lower left is almost unnecessary to make the classification. This kind of segmentation mapping
deviates from the single-layer ground truth significantly while likely being subjectively
rated around a six or seven. The effects of multiple occurrences encouraged us to look
for additional patterns or unintentional biases in the subjective ratings the 200 participants gave.
For example, to evaluate the appearance of the segmentation mask,
we asked users to judge the output of the same model but using the LIME post-hoc technique.
These images are more chunky due to the way the algorithm selects superpixels and it shows in the
data. The bias factor for the LIME technique (in green) has a higher mean absolute error,
likely because it incorporates so many unimportant areas of the image in its segmentation. This kind
of response led us to examine how participants react to false positive and false negative pixels.
First, we divided the reviewed samples into two groups, images with high false-positive
explanation errors (when the model is selecting those background pixels) and images with high
false-negative explanation errors (when the model is missing target pixels). If the users evaluated
both saliency error types the same way we would have expected to see the same regression slopes,
however, this is not the case. Instead, it indicates that users pay less attention to
false-positive errors and in turn, are more critical of false-negative errors.
Ultimately, by aggregating the wisdom of the crowds, patterns emerge that contain
different information from the typical ground truth baseline. We show how the technique can
be applied to both text and image domains and discuss how the methodology is simple in theory
but powerful in practice. Our benchmark demonstrates a quantitative evaluation
methodology for machine learning explanations that bridges the trade-off between objectivity
and subjectivity of different evaluation methods. We offer this methodology to the community as
a potential equilibrium that contains both human expectations and the objective true class features
as an appropriate evaluation technique. You can find the human attention samples collected in our
GitHub repository and invite others to apply this technique when evaluating explanations.
Thank you for your attention and you can find more details in our paper, thank you.