>> You're not going to want to miss
this episode of the AI Show where Minsoo
takes us through the responsible AI Dashboard
and scorecard in Azure Machine Learning.
Make sure you tune in.
[MUSIC]
>> Hello and welcome to this episode of the AI Show.
I've got a special guest, Minsoo,
who is going to take us through
the Responsible AI Dashboard and
scorecard in Azure Machine Learning.
Minsoo it was so good to see you again after
you've helped me in the keynote.
>> I had to help you serve.
>> I needed a lot of help.
>> You needed a lot of help.
>> That is true. Just a reminder,
we built a the script called
GPT-2model that built some heroic marketing text.
We wanted to get it.
I hand-crafted some data and we built the model,
but it turns out that the model was super biased.
Minsoo, tell us, how do we fix that?
>> Well, once you had your model,
we ran some responsibly AI insights on it and saw
some things that we really needed
to fix in order to improve the model,
both in making sure that it's
allocating opportunity to all the developers
equally and also making sure that you're taking
into consideration other things when developing your model.
>> I'm guessing this process starts once
you have the model, what happens after that?
>> Yeah. Let's dive right in but just to preface it,
we are actually providing
three sets of ways to create irresponsibly AI insights.
The first set of way is actually
a code first experience with the CLAI.
You can create a YAML configuration file
that you can with one line of CLAI,
create a pipeline job and run
your responsible AI Insights to get your dashboard.
We also have the corresponding SDK experience
for that, and then finally,
the third way is totally no code just from
the Azure Machine Learning Studio UI end
to end guided wizard experience.
Let's get started with the SDK experience.
>> Let's do it. Let me bring up your screen.
>> This is actually built on
the exciting new announcement of the SDK,
V2 of Azure Machine Learning Python experience.
What you have essentially is you have the training pipeline,
you've already configured it,
you have your data sets,
you've already run that,
then you register the model,
then you have all these different components
that are going to be able to you can put them together,
customize them to generate
a responsibility AI dashboard with your insights.
For instance, we have
the registered model that we're getting for you,
then we have a constructor, an explanation,
all of these RAI components,
and then we gather them together.
A great visual way of seeing this is
actually something like this where you
have the model here and
then you have your training and test data set.
You put it all together into the constructor,
then from the constructor you can
specify which components that you want.
Maybe you just want air analysis,
maybe you just want explanations of anyway
you're able to create all those components and
plug them in and gather them up into
this one gather component to
generate your responsively AI dashboard.
>> Interesting. Basically that SDK stuff that you were
doing creates this pipeline with all of those components together.
>> Exactly. SDK for instance,
we have here there are
AI constructor component that's you can specify the title,
what the task type is regression because we have
a linear requester that
the categorical column names what's the target value,
and then this would be an example of an explanation component.
For instance, you can either add a comment
and you would have your output for causal,
you would specify which treatment features you care about.
Treatment features are the things that are the levers.
You want to understand the causal effects of
those treatment features on
your outcome which in our case is the score.
>> Interesting. this is the SDK where
you mentioned there was a different way?
>> Exactly. Once you are able to
set off the SDK CLAI and also the UI,
you can come back always to your model details
page in order to view all of your response AI Insights.
That is where we've located
the Create Responsible AI Dashboard Wizard.
If you go into your model that you've trained,
you can actually create the responsible AI insights right here,
takes you through this really nice guided UI experience.
You just specify your data sets,
you specify what modeling task type you have,
In our instance, it's a regression.
Essentially, it's just doing all the things
that you were doing in code,
but really streamlined in
a no code fashion experience for people who just want to
see specific things about create
their response AI dashboard without
having to get their hands dirty with SDK,
Python code and the CLAI.
In our instance we want to use the model debugging,
but we can also click into real life interventions to see causal.
Says see the causal analysis component.
Essentially here after you click into that,
you specify what your target feature was for your model and you
can see all the different components that are available.
We have your error analysis here.
You have a counterfactual,
what if examples here,
you have an explanation here,
and so here this is where you specify
everything in the code but right in the UI.
For instance, I'm going to say style and location
I want to generate heat map for that
or the counterfactual examples.
I want to have 10 counterfactuals per data point
and have them all be between seven and 10.
That's the range where we accepted the programmers.
Then once you specify those,
then you just configure your experiment and you click "Create".
>> That's awesome. The boxes from the step before are
the same components that you specified in code or in the UI.
>> Then once you hit ''Create'',
it's going to take you to your job,
your experiment, and it let you know when your job is done.
Let's hop right in to the response by a dashboard.
>> This is the output of the entire process.
This is what the output is of this entire job.
>> Exactly.
>> Got it.
>> You can actually see it if you go
back to your model details view.
You can see it right here in the responsible AI tab,
and you can see multiple dashboards
because you're able to configure,
and they're so flexible you can
create any amount of dashboards for your model.
That's going to really help with maybe
auditing compliance reviews,
and you can view these for any of your registered models.
If you click into one of these,
that is what takes you into here.
>> What are we looking at here?
>> We're looking at all of
our awesome tools on a single pane of glass.
There's a lot to see here,
but just to start,
I'll tell you about one of the awesome features we have,
which is an integrated compute resource.
Here we have one of my compute resources in the workspace running,
and that's because the responsible AI dashboard is really
a dynamic visualization dashboard that is
showing you some metrics and
some visualizations that are calculated on the fly.
For instance, I could retrain
this tree based on different features.
For example, if I just want to see everything
but the errors on your operating system.
But one of the coolest things that I think about
this dashboard is this component error analysis,
it gives you this great visualization,
a tree like view that shows you the prediction path
to the cohort with that's the worst performing.
Where does your error distribution lie across your dataset?
What are the underrepresented cohorts
that are showing you the most amount of errors?
Actually, as you click between these different ones,
you can actually see different insights on those.
For instance, we can actually see the temporary cohort here and in
all the other components are changed when we are
dynamically changing between nodes of this error tree.
>> That's why it has attached compute,
because these are calculated dynamically, is that right?
>> Yeah, that's true.
For instance, the counterfactual examples,
if you are creating a counterfactual example,
then what you're doing essentially is you're
taking all of these examples that we've created for you.
That's what the counterfactual example said.
But at the end of the day,
you can actually just create your own.
If you wanted to have someone who had
years of experience be 25 instead of 14,
you can see a new prediction value generated right here
for any of the features that you
perturb for this reference data point.
It really helps you to drill in and dig deep
into maybe Seth scored a five.
I really want to know how to get him to a seven.
>> That's really, really cool.
This is all just from running that job,
and all those components
together on that model, you get all of this.
>> Exactly. You get the error analysis.
You can get a model overview of your performance and
also your fairness metrics, your disparity metrics,
which is how your cohorts are actually performing
relative to each other in a maybe sensitive feature.
You can also see imbalances in your data.
We see a lot of people are from Antarctica.
It's like very skewed,
and you can also see that.
You can see like the explanations for why your model made
such a prediction and
your counterfactual that we just went
over as well as causal analysis,
which is going to tell you what are the causal effects of
your treatment features that you specified on your outcome.
How do I optimize my outcome,
and that really helps you make
real life interventions and policies for business decision makers.
>> That's really cool.
It feels like you've told us everything.
Am I missing anything else? Is there anything else?
>> Yes, there is one more thing.
There's the responsive AI scorecard.
For instance, if you are doing
a compliance review or you're doing an audit
and you want to share your model
and your model development with different stakeholders,
you definitely want to have
your responsibility insights compiled
and summarized in a way that is really easy to share.
For instance, you're going to be able to also say,
click on one of your responsibly dashboards
and be able to generate a scorecard for it,
and then this is the scorecard that you would see.
You would see the summary,
the target values that you would use
your domain expertise to be able to say
my threshold is I want every square to be below two.
If it's not below two,
then I want you to say it does not meet the target.
I'm able to see this really nice summary pdf
that you're able to just
download from the studio
UI and share with variety of stakeholders,
both technical and non-technical.
>> This is so cool and I feel like we need to have you back on
to just drill in to some of these components a little bit more.
Every time I have you on my new show and
it's like information overload.
Sorry about that. Where can
people go to find out more about this stuff?
>> Alongside all these awesome things
which were public preview that build,
we have awesome documentation outlining what
is responsible AI an overview of what that even means.
How are you going to implement it?
The demo that we actually reviewed today,
the sample notebook is also available for
anyone to view and run and there is
also a lot of information about how to get
started and how to generate these things on your own.
All of those information is out there.
Please take a look, and we're looking so
forward to everyone being able to use
these tools to implement
their machine learning and development more responsibly.
>> Well, Mensa, this has been amazing.
Well, I have to have you on again so we can go into
each component in a detailed manner.
Thank you so much for being with us.
We've been learning all about
the Responsible AI Dashboard and
scorecard in Azure Machine Learning.
Thank you so much for spending some time with
us and hopefully we'll see you next time. Take care.
[MUSIC]