SARA ROBINSON: Welcome back to the last video in our What-if
Tool miniseries.
In the first video, James and I gave you an overview
of the What-if Tool.
And in the second video, we did a deep dive
of the Datapoint Editor.
Here, we'll be focusing on the Performance & Fairness tab
using the same housing data set from the last two videos.
This model predicts whether a house is worth more or less
than $160,000.
Let's get started.
[MUSIC PLAYING]
JAMES WEXLER: The first thing we see on the Performance
& Fairness tab are various performance metrics
for our model.
The confusion matrix tells us the percentage
of each class our model classified
correctly and incorrectly.
If we add the numbers on the top left diagonal,
it will equal our overall model accuracy-- in this case, 97%.
This shows us that 50 out of 51 data points from our test set
were correctly classified as over $160,000,
and 47 out of 49 data points from our test set
were correctly classified as less than $160,000.
The model had very few false negatives and false positives.
So far, we haven't changed the threshold slider.
Sara, do you want to explain what we can do with it?
SARA ROBINSON: Sure.
The threshold indicates your model's decision point
between classes.
In this case, any prediction above 0.5
will be classified as more than $160,000.
Changing the threshold is one technique
for optimizing your model's performance.
To change it in the What-if Tool, we can adjust the slider.
When we do this, you'll notice that our metrics change.
In this case, increasing the threshold
decreases overall accuracy but also decreases
our false positives.
JAMES WEXLER: If we select a single threshold from
the optimization strategies, the What-if Tool will choose
the threshold that yields the highest overall accuracy
on the loaded data points--
so the lowest total of false negatives and false positives.
For this model, it's still a 0.5.
This view shows us how our model is behaving across all data
points.
But what if we want to see how it's
behaving across different slices of our data?
That will help us understand if there's
areas where it's treating subsets of data points
differently.
Sara, do you want to show us how to slice by different features?
SARA ROBINSON: Sure.
Let's try slicing by garage type attached,
which indicates whether the garage is
attached to the house.
Notice that houses with an attached garage
have a higher likelihood that our model will value them
at more than $160,000.
Now let's imagine that we wanted our model to price houses
with attached and unattached garages in the same way.
In this example, we care most about having
the same percentage of positive classifications across slices
while still achieving the highest possible accuracy
within that constraint.
For this, we should select demographic parity
as our optimization strategy from the fairness section
on the bottom left.
Notice that our threshold sliders and accuracy
metrics change when we set this strategy.
JAMES WEXLER: What do all these changes mean?
If we don't want the garage placement
to influence our model's price, we
need to use different thresholds for houses
depending on whether the garage is attached.
With these updated thresholds, the model
will predict the house to be worth more than $160,000
when the prediction score is 0.99 or higher.
Alternatively, a house without an attached garage
should be classified as over $160,000
if the model predicts 0.52 or higher.
What about other optimization strategies?
SARA ROBINSON: We can use the equal opportunity strategy
to achieve a similar percentage of correct predictions
across each slice for the houses in our test data
set that are worth more than $160,000.
And equal accuracy will strive for similar accuracy
across both slices.
Finally, one more performance and fairness feature
we should look at is changing the cost ratio.
JAMES WEXLER: So far, all of our optimization strategies
have tried to pick the best thresholds,
given that we care equally about false positives
and false negatives.
But this may not be the case for many models.
For a model used in cancer screening, for example,
you might want the model to avoid false negatives, meaning
misdiagnoses, at the cost of having
more false positives, which could then be screened out
at the next level of testing.
SARA ROBINSON: To update thresholds
based on how you want your model to handle false positives
and negatives, you can change the cost ratio.
For example, setting the cost ratio to 4
means that we want to consider false negatives four times
as costly as false positives.
Then we can find the optimal threshold, given this new cost
ratio.
JAMES WEXLER: We've covered a lot in this video.
To recap, we looked at a few metrics
for evaluating the performance of a model,
including the confusion matrix, and true and false
positive rates.
Then we looked at modifying the prediction threshold
and analyzed how our model behaved across different slices
of our data.
SARA ROBINSON: Keep in mind that while this
is a data set of houses, we'd want
to do a similar analysis for other types
of data, especially human-centered data.
It's also important to note that we've
shown just one strategy for improving the fairness of an ML
model.
There are many other methods available for this,
such as augmenting training data and designing
fairer cost functions for use at training time.
JAMES WEXLER: For more resources on everything
we showed, check out the links in the video description.
Thanks for watching.
[MUSIC PLAYING]