>> Today on this special build edition of the AI Show,
we get to hear from Marco Ribeiro,
Senior Researcher from the Microsoft Research Team.
When you build machine learning models,
it's important to know what they're doing.
Marco, the creator of LIME or
Local Interpretable Model-Agnostic Explanations,
will talk about the research,
how it works and why it's important. Make sure you tune in.
[MUSIC]
>> My name is Marco, I'm
a Senior Researcher at Microsoft Research.
Focus on my research is helping humans
interact with machine learning models meaningfully,
that includes interpretability, for example,
I wrote the LIME paper and a bunch of others.
I'll talk a little bit about LIME in this video,
but before that I thought a little history would be interesting.
So LIME was the first paper I wrote in my PhD and at the time,
interpretability wasn't as hot as it is now,
there were almost no papers coming out on this subject,
and this is how I got into it.
I was doing an internship at
a large tech company trying to use machine learning for a task,
and I trained the model and it worked
really well in cross-validation.
But whenever we sent it to user testing,
it sucked and it took me forever to figure out what it was.
Through a long time of debugging and trying to understand,
I figured out that my model had
learned to distinguish between how we had collected
positive and negative data rather than picking up
on what it should be picking up in the real tests.
So it learned a shortcut,
it detected how we collected the data
and that didn't translate into the real-world use case.
Now, this experience, there are
two things that really bothered me.
So the first one was
that cross-validation accuracy didn't
translate into real-world performance.
Maybe this is obvious for people who are doing
machine learning in the real world,
but as a researcher,
if you take a machine learning course,
you learn that cross-validation is a good way to evaluate models.
The model has not looked at
that data and it works well, so therefore it's good.
But models are really good at
picking up quirks or spurious correlations and
datasets and that can really mess up
performance when your real data does not have those quirks.
But the thing that bothered me most was
that I could not figure out what my model was doing.
So here I was, PhD student,
supposed to be an expert and it took me
the longest time to understand
how my model was making predictions.
So I came back from my internship,
and at the time I was working on a completely different topic,
distributed systems and machine learning,
but I decided that this
bothered me enough that I wanted to work on it.
So I looked at the literature to see what people had done in terms
of understanding how models were making predictions.
Then I saw a bit of a gap,
so I decided to change topics and work on that,
and that's how I got into interpretability in general.
So why should we bother with interpretability at all?
Interpretability has many uses,
sometimes they're even forced to do it due to regulation.
But the use case that's dearest to
my heart is the one I was just talking about.
You're training your model and you
want to figure out, does it work?
Is it doing something sensible?
How can I improve it?
I think that understanding what the model is
doing really helps all of that.
It helps you avoid putting
a really bad model into production because
you catch things that don't make sense ahead of time.
Machine-learning models are really good at
a particular kind of over-fitting that I was just mentioning,
picking up on spurious correlations,
picking up how we get the data,
picking up on things that will not generalize.
If you understand it, if you can see those things happening,
usually it's obvious when you see it,
but even when you don't have situations like that,
if you understand why mistakes are being made by a model,
it's easier to figure out how to fix them.
So it's easier to figure out if you need to get more data,
if you need to change model architecture,
how to change it, etc.
We had some experiments in the LIME paper where we had people with
no machine-learning expertise picking
up or trying to decide between two models,
which one is going to generalize better and even people with
no ML expertise could do really good model selection.
The could also do some feature engineering,
so they could look at what the model was doing and say,
the model should not be doing this,
and click out on words.
In this case, it was text,
words that the model should not have been using.
So even people who have
no machine-learning expertise at all can already do something.
How much more people who are actually
training these models and applying them?
So the actual developers of models,
I think, can gain a lot.
So understanding what's going wrong,
it already gets you almost half of the way into fixing them,
and I think interpretability is really useful for that,
among many other uses.
So I've been talking about LIME,
but I didn't even tell you what it is.
I'll just give you the key ideas behind
the technique that we use in
the paper and the key ideas are as follows.
First idea is, we're not going to try
to explain everything at once.
If you have a model that's really complicated,
it's a very hard task to explain everything it's doing.
Let's say I'm trying to predict risk of defaulting on a loan,
it's a complicated problem.
So maybe you think like, oh,
if credit score is super-high or super-low, it's easy,
just look at credit score,
but maybe in the middle it's complicated and then you realize,
oh, it interacts with time.
So if your credit score is slow because you're
an immigrant and haven't had time to open up accounts,
but you work at Microsoft and this is actually a problem that I
face as an immigrant getting a credit card approved,
but maybe that should be taken into account.
So you don't just look at credit score.
So if you try to explain,
this is just a small example,
if you try to explain all of that at once for a good model,
you're going to get into trouble.
You're going to hide too much or
definitions can be super complicated.
So what LIME is going to do is going to try to
explain a single prediction.
So let's say that the model thinks that I, Marco, have low-risk,
we want to understand why that is,
and not try to explain every single person in my dataset at once.
Then the other idea is that we're going to treat the model
as a black box, and there's a trade-off.
If you treat the model as a black box,
you cannot exploit its internals and you need to do other things.
But there's also a gain that you can explain any model,
and we made that decision of explaining the model as a black box.
But if the model is a black box,
how can you understand it?
How can you explain?
I think the only way you can do it is by
perturbing the example in seeing what the black box model does.
Now, there's a lot of ways you could perturb the example.
There's a lot of things you could do,
but we did a very simple thing with LIME, and this is what we did.
Let's say we want to understand why I'm low-risk,
what you do is change different features.
So you change, for example,
you say Marco doesn't work at Microsoft,
he's unemployed and then you see,
does the risk prediction change?
If it changes to high risk,
that's a good indication that
the model is using the fact that I work at Microsoft,
but you don't want to change only a feature at a time,
like remember the case where maybe credit score was important,
but you have to take into account
whether a person is an immigrant or
whether their accounts are new and so on.
So maybe I change my job and my credit history a bit,
and maybe I change a lot of other things at once and you perturb
a bunch of different times and see what the model does.
After many perturbations, you learn a simple explanation,
a linear model that says something like,
Microsoft lowers the risk by 0.2,
being an immigrant raises it by 0.05 and etc.
So this explanation would be describing
the model for people who are close to me,
people who are similar to me.
The explanation has to be really
faithful or really good for people who look like
me and maybe for people who look
like me and looking at zip code doesn't matter.
So the explanation is not saying that
zip code is irrelevant for the model, it's just saying,
for people who are in the neighborhood of Marco,
what matters is where you work,
being an immigrant, and so on.
Anyway, combining these two,
the main idea is this,
treat the models as black box,
perturb the example you want to
explain and then learn an explanation
that really describes the model in that region,
in that neighborhood, instead of trying to
explain everything at once.
That's a very simplified version of what LIME is and does.
I hope I've given you a little bit of the background
and convinced you to at least consider interpretability,
consider, can it help me
to understand why my model is making predictions?
I think it can,
and LIME can be a useful tool in
your tool belt of interpretability.
There's many others as well,
but signing off. Thanks.
[MUSIC]