[MUSIC PLAYING]
YUFENG GUO: What if you could inspect
a machine learning model with minimal code required?
What if you could even see if your model was being fair
or not?
We'll cover all that and more on today's episode
of "AI Adventures."
[MUSIC PLAYING]
Welcome to "AI Adventures," where
we explore the art, science, and tools of machine learning.
My name is Yufeng Guo, and on this episode,
we're going to look at the What-If
Tool, an open-source project from the People and AI Research
team at Google.
It's designed to help you probe your models to better
understand its behavior.
Now, a few episodes ago, we looked
at Lucid, which was great for understanding
convolutional networks.
It's only fitting that we now look
at What-If Tool, which specializes in structured data
and text.
Let's take a look at what it looks like.
The What-If Tool, or WIT, can be used inside a Jupyter or Colab
notebook or embedded into the TensorBoard web application.
Today, we'll just look at one example
data set, our old friend, the US Census data set.
It's a nicely-sized data set for this type of exploration
and it has many of the flaws of a real-world data set,
so we can have some pretty interesting findings.
In this example, two models were trained
on the Census data set, a linear classifier
and a deep classifier.
So they'll be compared to each other as we use the tool.
Within the tool, they are referred
to as Model 1 and Model 2.
The first view of the What-If Tool
has Facets Dive embedded in it.
You can use it to slice and dice your data along the x
and y-axes, as well as see how your data
points are distributed.
Longtime viewers will recall an early episode
of "AI Adventures" about the Facets tool.
If you were wondering whatever happened to that, well,
here's your answer.
The Facets tool has been integrated into the What-If
Tool and updated with more features.
Facets Dive is incredibly flexible
and can create multiple interesting visualizations
through its ability to bucket, scatter, and color data points.
You can click on values in the chart
and edit them to see what the prediction would
be for your newly edited value.
Try doing this near the decision boundary
to get a better understanding of what features have a greater
effect on the outcome of your predictions.
Here, we change the age and hours per week of a data point
to see if that changes the outcome of the prediction
and by how much.
Notice that the predictions for both the linear model
and the deep model are shown here.
This view also features an ability
to show the nearest counterfactuals.
The nearest counterfactual is the most similar data point
that got a different classification.
Once you turn on the toggle, each time
you click a data point, its counterfactual
also gets highlighted.
Here, we see an example where these two data
points are very similar, but the occupations are different.
Now let's move over to the next tab.
Yeah, this tool's got two more tabs of information.
The next one is called Performance and Fairness,
which allows us to look at overall model performance
and ask questions about model performance across data slices.
Be sure to set that Ground Truth feature
in the upper left, which is the label that we'll
be predicting on, in order to get the most out of this panel.
We can try moving the prediction threshold around
for each of the models to see how it impacts that confusion
matrix.
We can also choose to slice by up to two fields
and then select various fairness optimization strategies
to see how that affects the threshold values in each model.
Now, the third tab is not to be left out.
Here, we see Facets Overview embedded into the What-If Tool.
You can use this to look at the distribution of your data
points across all the different features,
as well as some key information about each feature.
This is a great way to spot imbalanced data, as well.
There's a lot of features, a lot of use cases for the What-If
Tool, so we're not going to be able to cover them all today.
But if you're interested in learning more about it,
let me know in the comments below or on Twitter,
and I'll try to make another episode to dive
into the other features of this tool in more detail.
And if you can't wait for that to happen, go ahead and join
the What-If Tool community on our Google Group.
There, you'll find announcements of new features
and all sorts of lively discussions.
I've included a clickable link below in the description.
Thanks for watching this episode of Cloud "AI Adventures."
And if you enjoyed it, click the button
and be sure to subscribe to get all the latest episodes right
when they come out.
For now, check out the What-If Tool
and learn about how your model is operating.
[MUSIC PLAYING]