Hi, I’m Angie Boggust and this is Shared Interest,
a method to measure human-AI alignment
and identify recurring patterns in model behavior.
To begin,
let’s imagine you’ve just trained an image classifier.
It achieves high accuracy,
but you want to know if it’s making its predictions
for the right reasons.
So you select an image from your dataset to evaluate.
Your model correctly classifies this image as a snowmobile,
and its explanation highlights the snowmobile.
So it's correct and focused on the correct region.
This is promising,
but you want to explore some additional examples.
So you sample two more images from your dataset,
and see a similar pattern.
Both are correctly classified,
and the explanation highlights the snowmobile,
so your confidence in your model is growing.
But what if you were to sample again and get this image?
Now, you might not feel quite as confident.
It's correctly classified,
but its explanation highlights the helmet
as opposed to the snowmobile.
While helmets and snowmobiles are definitely correlated,
you would not want your model
to have learned a rigid dependence between the two.
So the take away here is that
selecting the right inputs to evaluate is hard.
We have large datasets,
so we can't evaluate everything.
And depending on how we subsample,
we could miss the most informative datapoints.
This was the problem that led us to develop Shared Interest ---
a method to explore model behavior at scale
In Shared Interest,
we start with a dataset to analyze our model’s behavior on.
We quantify human decision making
via ground truth annotations.
And we explain model decision making
via saliency.
Here, we use LIME
applied to ImageNet images
and their object-level bounding boxes.
However,
Shared Interest is agnostic
to data modality and saliency method.
For each image,
we quantify the relationship
between the saliency and the ground truth
via three coverage metrics:
IoU Coverage,
Saliency Coverage,
and Ground Truth Coverage.
IoU Coverage
measures the alignment
between the saliency and the ground truth.
Low IoU Coverage indicates that those regions are disjoint,
like our snowmobile image from before.
High IoU Coverage
indicates the saliency and ground truth are almost identical.
Saliency Coverage
measures the proportion of saliency features
that are in the ground truth.
Low Saliency Coverage
suggests that few of the features salient to the model
are also ground truth features,
like in this image
where the train tracks are salient as opposed to the train.
High Saliency Coverage
indicates that almost all of the saliency features
are ground truth,
like in this example,
where the saliency highlights a subset of the dog.
Finally,
Ground Truth Coverage
measures the proportion of ground truth features
that are salient to the model’s decision.
Low Ground Truth Coverage
means that few ground truth features are salient,
like in this example,
where the saliency focuses on the biker as opposed to the bike.
High Ground Truth Coverage
indicates all of the ground truth features are salient,
like in this image,
where the saliency highlights the cab and the street behind it.
Using these scores,
each image in the dataset is quantified,
allowing us to systematically sort and explore them
to understand model behavior.
Applying Shared Interest to a variety of models
uncovered eight common cases of model behavior
that persisted across modality, task, and saliency method.
Each case is defined as a function of the model’s output
and the Shared Interest scores.
For example,
Sufficient Subset instances have
high Saliency Coverage,
low Ground Truth Coverage,
and are correctly classified.
This case uncovers instances where
a subset of the ground truth is salient to the model,
such as the tractor tire to the tractor prediction.
On the other hand,
the distractor case contains incorrectly classified instances
with low IoU Coverage.
This case uncovers instances where
the model is “distracted” by secondary features,
like the negative words “nothing awesome”
elsewhere in the review.
Now let’s apply Shared Interest to a real world use case,
model assisted dermatology.
In this example,
we have a computer vision model
trained to predict melanoma
from images of skin lesions.
The model achieves high accuracy,
but is it making decisions for the right reasons?
Let’s see.
We’ve populated an interface with the images in our dataset
and scored them using Shared Interest.
Using the interface,
we can filter for high and low Shared Interest scores,
select a specific case,
and view the score distributions.
Let’s begin by analyzing images
the model gets right for the right reasons.
To do so,
we filter for correctly classified images
with high IoU Coverage.
These images exhibit ideal model behavior
because the features of the lesion
are salient to the model’s correct prediction.
However, not many images have high IoU Coverage
meaning not many images fall into this category.
So let’s explore some other model behaviors.
Context Dependent images
are correctly classified
with salient features within
and surrounding the ground truth region.
Malignant examples are especially interesting
because cancerous cells can lie outside of the pigmented area,
so the model could be identifying subtle features.
But, the model should primarily rely on the lesion
just like a dermatologist would.
So this may warrant further analysis into lesion metadata.
Finally,
let’s look at the Sufficient Context case.
In these examples
background features are most important to the model’s decision.
Focusing on the benign images,
we see our model has latched onto medical artifacts.
In this dataset,
artifacts only occur in benign images,
so it is sufficient for the model to use them
to make its prediction.
But this is a dangerous flaw
because the model might say
a cancerous lesion is nothing to worry about
if an artifact exists in that image.
Overall,
Shared Interest helped us rapidly analyze the model’s behavior,
uncovering dangerous pitfalls
and generating hypothesis to improve the model going forward.
We can also use Shared Interest to query model behavior,
enabling deeper analysis using a single instance.
In this setting,
we pre-compute saliency for all classes a model is trained on
and interactively specify a ground truth region.
Then, using the Shared Interest metrics,
we uncover the classes with the highest alignment to that region.
Here for example,
we set the ground truth to be the dog’s face,
and the interface returns the classes
with the highest IoU coverage to that region.
The returned classes are types of dogs,
and the saliency for each class
is focused on the dog,
indicating the model has learned
generalizable visual features of dogs.
Next, we select the man’s hat.
Even though this image is labeled otterhound,
the classes with the highest Shared Interest to the hat region
are other types of hats,
suggesting the model has also learned
about secondary objects.
Finally, we highlight the man’s hand.
Hand is not a label in the dataset,
yet returned classes are objects
that would occur alongside a hand,
like a notebook.
It is promising that the model has learned
correlations about unlabeled objects;
however, this may warrant further analysis
because the model should not be dependent
on the presence of a hand
to classify a notebook.
Overall,
using Shared Interest in this style of what-if analysis,
we were able to rapidly test hypotheses
about what the model has learned
and where it might fail.
In summary,
Shared Interest enables
efficient large-scale analysis of model behavior,
so machine learning models can be properly evaluated
and safely deployed.
Thank you!