Welcome!
My name is Sophia Hadash and today I will present a joint work of me and my supervisors
Martijn Willemsen, Chris Snijders, and Wijnand IJsselsteijn on improving the understandability
of interpretability tools.
There are a lot of black-box machine learning models and if you want to understand how these
work we can make use of so-called interpretability tools.
These tools use proximity sampling to learn about the behaviour of the actual model in
the neighborhood of a data point and build a proxy model.
The proxy model typically produces a list of feature-importance values that can then
be visualized.
The feature-importance list describes how important each feature is to the decision-class,
using numeric positive and negative values, for example "checking status" contributes
negatively towards ineligibility for a loan could be a feature-importance.
Sometimes these explanations are difficult to interpret, especially the interpreter has
an interest in the decision-classes.
In the case of loan-applications "eligibility" may be the desired outcome.
We will show this with some examples.
These are some visualizations as generated by two popular tools: LIME and Shap using
loan-application dataset.
And in cases where the prediction is loan-inapplicable, positive bars indicate contribution towards
not getting a loan.
This may be counter-intuitive when users value getting a loan positively.
From a technical standpoint it is logical to frame feature contributions with respect
to the decision-class.
Data scientists have come accustomed to this framing as well.
However, from a cognitive point of view we know that people are better in handling positive
information and it may make sense to frame contributions with respect to the positive
class instead.
This would look as shown on the right, where we frame the contributions such that positive
contributions imply positive outcomes.
If we frame it like this we see a small positive contribution and two strong negative contributions
towards eligibility so it becomes clear that you're not getting the loan.
Another possible improvement is to add semantic labeling to the visualizations.
Currently the readers need to understand what positive and negative feature contributions
imply from the context such as the decision-class and the framing that was used.
And sometimes this can become complicated, especially when negative contributions contribute
negatively toward not getting a loan.
That sentence in itself sounds complicated.
The semantic labels directly explain what the feature contribution implies and we hypothesize
that this improves understandability.
These improvements as well as other undiscussed improvements are available in python and javascript
packages.
To test the effectiveness of the two proposed improvements we designed an empirical user
study where we showed participants a prediction and an explanation of the prediction with
varying framing and semantic labeling conditions.
When we used semantic labels we used both positive framed labels such as "+5% eligibility"
and negatively framed labels such as "+5% ineligibility".
After each explanation and after each condition we asked participants how well they understood
the explanation and this wholle study was done across two domains with a large sample
size.
What we found, is that positive framing leads to higher understandability.
On both the left and the right graphs we can see that positive framing leads to higher
understandability than negative framing, which is especially noteworthy on the right where
the decision-class is negative and still the positive framing class works better.
Current interpretability tools would use negative framing in this case when the prediction is
negative and we thus find that when a prediction is negative it's actually better to use positive
framing to improve understandability.
Next, we look at the results for semantic labeling.
When we used negatively framed semantic labels we did not find an improved understandability.
Not even when positive framing was used as well.
However, positively framed semantic labels led to improved understandability irrespective
of the framing that we used, and this suggests that the semantic labels does make the explanations
simpler, since the used framing is no longer relevant for understanding the explanations.
We would like to give two takeaways from our studies.
First, it seems that positive framing always improves understandability over current decision-class
framing when this is possible.
Secondly, using positive semantic labels would also improve understandability and may be
desirable because this reduces framing effects altogether.
This does require that the decision-classes lie on an ordinal scale and thus cannot be
used for classifying cats and dogs, at least depending on who you ask.
I invite you to look at our paper for all the details of our study and our findings.
Thank you for your time.