Hi!
I am Forough and I am going to talk about our joint work with my colleagues at Microsoft
on manipulating and measuring model interpretability.
Machine learning is increasingly used to aid decision making in high-stakes domains, such
as medical diagnosis and credit risk assessment.
Often in these settings, decision making is a collaboration between people and models,
where models make predictions and people can choose whether to follow these predictions
or to override them.
While in theory, following a model’s predictions should enable people to make faster and more
consistent decisions, in practice, there are scenarios where people may be resistant to
using models for various reasons, including difficulty in understanding how they work.
In response to this concern, a great amount of effort has been invested on a relatively
new line of research for interpretable machine learning.
But, despite this progress, there has been little to no consensus about how to define or
measure the interpretability of a machine learning model.
We take the perspective that the lack of consensus around defining and measuring interpretability
stems from the fact that interpretability is a latent property that can be influenced
by various manipulable factors such as the number of features that the model uses, the
transparency of the model, and even the user interface, and these factors, in turn, can
impact several measurable outcomes such as the extent to which people follow the model’s
predictions, or their ability to debug the model.
While the manipulable factors that are known to impact interpretability are properties
of the machine learning system, the measurable outcomes that are often the end goal are properties
of human behavior.
Different factors may influence different outcomes in different ways.
Therefore, it is necessary to directly manipulate different factors in the machine learning
system and measure their impact on people’s behavior.
What is or is not interpretable must be defined by people’s behavior and their ability in
achieving their intended goals, not by what appeals to intuition.
Drawing on this perspective, we designed and conducted a sequence of human-subject experiments
in the domain of real-estate valuation.
We asked our participants to predict the selling price of properties with the help of a linear
regression model.
In addition to a baseline condition where users did not have access to a model, we considered
four primary experimental conditions in a 2-by-2 between-subject design, as shown here.
The models at the top use two features to predict the price, while the models at the
bottom use eight features to predict the price.
The models on the left are completely clear -- meaning that participants can see all the
regression coefficients and how the model prediction has been calculated,
while the models on the right are a black box.
All participants saw the same set of apartments -- meaning, the same feature values -- and crucially,
the same predicted selling price for each apartment.
This is critical in letting us attribute any observed differences in participants’ behavior
entirely to the presentation of the model.
Here are some key findings from our experiments: First, as expected, we found that transparent
models with few features were easiest for participants to simulate.
This is inline with our hypothesis and common intuition that people better understand how
simple, transparent models work.
Surprisingly though, we found no significant difference in the extent to which participants
followed a model’s predictions when it was beneficial for them to do so on typical examples
for which the model’s prediction was generally accurate.
And finally, we found that transparency hampered participants’ ability in detecting model’s
sizable mistakes and correcting them, and this was seemingly due to information overload.
In general, our experiments and results emphasize the importance of taking a human-centered
approach in the field of interpretable machine learning and prioritizing testing with people,
rather than relying on what appeals to intuition.
Thank you for listening.
If you have any questions, feel free to contact me or any of the other authors.