>> You're not going to want to miss this episode
of The AI Show where we talk
about the responsibility AI dashboard and scorecard,
a deep dive with my friend Minsoo. Make sure you tune in.
[MUSIC].
>> Hello and welcome to this wonderful episode of The AI Show.
We're going to do a deep dive into
the responsible AI dashboards
and scorecards with my friend Minsoo,
Minsoo, why don't you introduce yourself,
tell us who you are, what you do, my friend.
>> Hi everyone. My name is
Minsoo Thigpen and I am a Product Manager at
Microsoft working on responsible
AI tooling on Azure Machine Learning.
>> Fantastic. Let's dive in
Responsible AI put it
all together and set the stage. Can you do to that?
>> Yeah, of course, first of all,
Responsible AI is a big topic.
Part of what we do here is give people tooling,
people like you every day,
Data Scientists, ML Pros,
the tools to put an implement responsible AI in practice.
We started out with launching an open-source project called
responsible AI toolbox last December in 2021.
Now I'm here today to talk to you
about how you can actually implement
and scale this On-Cloud in Azure Machine Learning Platform.
>> Fantastic let's dig in.
>> Yeah, so there's nothing better than a good old example.
Let's go over to our Azure Machine Learning workspace,
where we're going to take a look at our data.
The scenario that we're looking at
today is actually a made-up scenario.
It's synthetic data that we created just for
this episode and for this demo and we're going to
be training a linear logistic regression model
on a list of programmers.
These programmers were predicting and giving a score
between 0 and 10 on whether or not they're going
to be allowed access to a GTP 2 model.
We have things like what style you're a programmer of,
do you use spaces,
do you use tabs, how many years of experience do you have,
what IDE you use,
programming language, where you live?
All of those really interesting things
that programmers we're looking at when we're
distributing opportunity across people
who are applying for access to this model.
>> Just to be clear,
this is all made up data.
It's not real. We just picking it a little bit?
>> Yeah.
>> Cool.
>> Exactly. We're going to have
our training pipeline job in Azure machine learning.
We've kicked out off, we've completed it,
and we actually also have kicked off and completed a job to
create the responsible AI dashboard with
all of our different responsible AI components,
which I'll go into in a minute.
Then also a responsible AI scorecard.
Today in this episode we're going to be actually talking about how
you're going to be able to find those
responsibly a dashboards in your studio space,
as well as a quick overview of
what all value you can get out of these specific insights.
>> That's cool, can we go back on because I have
one question is this is something
that because I know we've talked about this before.
There's actually two jobs that's going on.
The first job is training
the logistic regression model
to Score Programmers and give them access.
That's the first model. Then once this job is done,
you take this model and then you run
the dashboard job on and as well. Is this right?
>> Yes.
>> Okay.
>> That's correct.
>> Cool.
>> The cool thing about the Responsible AI dashboard and
Scorecard are they're flexible components.
You can construct the pipeline job, run it.
You can also do that from the CLI through YAML specifications.
You can also do it from the SDK with a Python.
Then you can also do it on the UI
itself within Azure Machine Learning Studio,
which guides you through a nice UI Wizard
that's going to help you create
the dashboard so you don't touch code at all.
>> I see. After I run my model,
if I register the model,
this UI that I'm looking,
I guess this way, this UI that I'm looking at,
you can just go to the model and be like
generate a dashboard for me as well. That's cool.
>> Exactly. Yes. As you can see in this screen,
we're looking at a list of dashboards that were generated.
You would go into the Model Registry,
which you can see here we're in
the Model Registry as well as the Breadcrumb up here.
You click into the model that you've registered,
which we've done here.
We've fetch are registered model after we train it.
Then you'd be able to click into
the Responsible AI tab up above
and see all the Response AI dashboards that you've generate.
Even cooler than that,
is maybe you're working and
collaborating across technical and non-technical teams.
In that case, you can actually
generate a responsible AI scorecard,
which is going to allow you to download a PDF of
all of your key insights and
share those outside of your workspace.
>> That's pretty fantastic
because I can say something like on a Friday,
I got to write this responsible report all day on Fridays.
>> Exactly.
>> All day and then literally
just run the thing and then the PDF comes out.
>> Exactly. This is a tool
that can really help you in something like,
let's say a compliance review
or you're getting audited for your models.
You can show that you've done your due diligence to assess
and help understand your model and build trust and confidence.
>> That's cool. Now can we look at
one of the dashboard because I've looked at it before.
Every time and I've done these demos before.
There's so many things in there.
I was hoping you could maybe help me understand
what each one of those things
are telling us and how we can look at it.
>> Absolutely. Let's say you've trained your model and you are in
the process of evaluating
your model before you deploy it into production.
It's going to be out in the wild.
You want to have a couple of different assessments to understand
how your model and your data work
together to give you your predictions.
In this view here you can actually
click on the Dashboard that you've generated.
I'm just going to switch over to the tab here,
and it opens up to a full page.
You're going to see all the tools and components
that you've created for your responsible AI dashboard.
There's a lot of stuff here and I'll very quickly go over them.
First you have Error Analysis,
which is showing you how error is
distributed across different cohorts in your dataset.
In model training, we oftentimes
have aggregate metrics that doesn't tell you the whole picture.
We have an intuitive,
tree-like map that shows you where exactly your errors live.
Then we also have a heatmap for showing you
across multiple features where your errors live.
You can evaluate errors in your model this way.
Secondly, we have the Model Overview,
which actually it's going to show you
some fairness metrics here as well.
Fairness metrics are essentially nothing more than
showing you some disparity or some other definition
of fairness as how
your model is treating different cohorts in your dataset.
Here you can see an overview
of the different cohorts in your data,
as well as fairness, difference,
and ratio of the metrics that you're looking at.
In this case, you have the mean absolute error,
squared error and prediction.
You also have a handy visualization tool at
the bottom to help you visualize this table form as well.
Thirdly, we have data analysis because
when you're looking at the performance and output of your model.
The data story is really important in understanding
over or under-representation that
might affect how you've trained your model.
We have Aggregate plots,
we have Individual plots,
Scatter plots and you can actually like,
really key this into
different cohorts of your data that you're interested in.
Maybe I'm only interested in seeing the distribution
of error across people who live in Antarctica.
Then we have your
Classic Feature Importance's which shows you overall and at
an individual level what
actually is impacting the prediction that your model mix.
In this case we see the style is the highest impacting feature for
your model predictions and
you can see a breakdown of those as well.
Then we have counterfactual what-ifs, which actually,
this is like a really cool tool that generates
these diverse set of data points that are going to show you what
are the minimal changes required to flip
the classification or get
your desired outcome within a certain range.
Then we also have
this awesome feature here where you can actually manually
perturb the specific features you care
about and see a real-time update in your prediction.
Then last but not least,
we have causal analysis,
which is going to show you what is
the causal effect of certain features that you're interested
in like number of GitHub repos someone has or
years of experience they have on their desired outcome,
which in this case is a programmer score.
The programmer score as well,
but enables them to get GTP to model.
You can see it for overall,
all of your data points for individual people,
what exactly needs to be changed as well
as a recommended treatment policy for each feature.
If the treatment feature we want to understand
how to have a policy around
increasing or decreasing this number
to have a real-world effect on the score.
Then we have this really awesome set of tools here
and visualizations that's going to show you how
to actually impact the real-world outcome of the score
through a recommended policy and how it impacts
the gain in your score as well as if you
don't have the resources top 10 people to target here.
That's really brief, but essentially,
there's a lot of stuff here to help you really
understand your model and
your data and develop it more responsibly.
>> This is really cool. Like every time I see this stuff,
I feel like I understand the error analysis,
I understand the data stuff.
I understand the what if and the counterfactuals.
But my sense is I've never gone into each one of them.
I'm going to propose up until you tell me what you think.
What if we do more of these shows,
but we only pick
one thing that we're like the error analysis, for example,
next time we do, because I feel like there's
so much to explore in each one of these,
then every time I do this demo for someone,
I'm just like, I am not getting all of it.
All the way. Minsoo, what do you think?
>> I think that's a great idea.
There's so much here, so much richness.
I would love to share it with you in
individual and demos of each individual component.
Because that way we have time to dig in deeper and
show you how it can actually provide value in your MR lifecycle.
>> I'm a huge fan of that.
For those that are getting started with RAI,
where can people go to find out a little bit more?
>> That's a great question.
We have a lot of content online.
We have the Microsoft Docs,
which shows you how to generate these response way,
a dashboard, what is even responsible AI,
and how to read these graphs and interpret them.
Then we also have a blog that's going to be able to show you,
like step-by-step walking through each of the component.
>> Yeah, this is awesome.
Like I said, I'm excited to learn a lot
more from you about each one of these things.
Thank you so much for spending some time with us, my friend.
>> Thank you for having me and I'll see you next time.
>> Awesome. Well, thank you so
much for watching and learning all about
the Responsible AI dashboard and Scorecard with more to come,
thanks for watching and hopefully
we'll see you next time. Take care.
[MUSIC]