>> Reproducibility in machine learning is super important.
In this special build edition of the AI Show,
we will hear from Dennis Eikelenboom,
Program Manager on the Azure Machine Learning team.
He's going to talk all about tracking in
Azure Machine Learning through
a very interesting demo. Make sure you tune in.
[MUSIC]
>> Hello, and welcome to
this session on Data to Machine Learning Tracking.
My name is Dennis, and I'm
a Program Manager on the Azure Machine Learning team.
Maintaining a digital audit trail for
your machine learning activities matters.
It enables experiment reproducibility and
improves the operability of your machine learning solutions.
Legislative reasons might also be
an important driver to keep track of your assets.
For this demo, we'll take a look at the analytics company,
Contoso, that provide services to
renters on the Amsterdam housing market.
Since the market is saturated and changing rapidly,
the company helps renters to find
the optimal price for their properties.
In their analysis, they make use of both static reference data,
as well as data that is coming in live from housing reviews.
There are two main drivers for Contoso why tracking is important.
First, because prices fluctuate,
they want to include the latest data in their models,
and hence retrain and redeploy often.
Each time they make a pricing recommendation,
they would like to understand what model version was used,
and which data was included.
Secondly, when customers sign up for the services of Contoso,
they agree that their data might be used for analytical purposes.
From a legal perspective,
they might sometimes receive a request to delete
the data that was used due to European law.
I'll now show you how
Azure Machine Learning enables Contoso to keep
track of the various assets that go
in and out of the experimentation process,
like code, data, and models, as a result.
Nowadays, those are used
more downstream in the various stages
of the machine learning lifecycle.
The data engineering and data science teams work closely together.
Contoso's data engineers collect
live data from random reviews that people post online,
and then the data science team use to data as
input to several of their machine learning experiments.
Let's have a closer look at one of their pipelines.
Data is captured via an Azure event hub,
and the team uses Spark structured streaming to add
static the reference data as the reviews stream in.
Since data comes in JSON format,
the team applies pre-processing logic
to set the scheme of the columns,
and then connect the results in a table
for easier and downstream processing.
You can see that rental reviews are posted continuously.
Browsing over them, we can see that reviews
contain information regarding the neighborhood,
specifications of the apartment,
its rental price, and the
satisfaction scores that people gave regarding their stays.
Contoso's data science team has
issued a survey with local guides in
the perceived sentiment of renting a place in
each of the different Amsterdam neighborhoods.
One of the metrics that the survey yields is
a score for the relative closeness
of the neighborhoods to the airports.
The team captured the survey results in a table.
The data engineering team uses the reference table with
the survey results to enrich each of the reviews that come in,
and then register into storage for later analysis.
The data science team builds in Azure Machine Learning to
run the machine learning experimentation and deployment.
Whenever the team makes a change to
the data that is used as input to their experiments,
they would define a new datasets version.
Using Azure Machine Learning datasets,
they define a reference to the data on storage,
allowing them to use it as
a version input to their machine learning pipeline.
In this notebook, the team defines
a new dataset version to include
the most recently collected data points.
Note that when creating a dataset with Azure Machine Learning,
we define the data reference, whereas,
the actual data remains in the source.
When registering the datasets,
we make it available for others to use.
One of the experiments that a data science team carries out
is the determination and advice for rental price of an apartment,
given its characteristics, ratings, and time of the year.
The team learnt that, to prevent the model from drifting quickly,
they would need to incorporate new data relatively frequent.
Hence, they would make
a new dataset version in the same way as we saw before,
and reference that dataset version for training.
In order to do so, they would simply make
a reference to the datasets in the training script,
using Azure Machine Learning SDK,
provide a version, and then pass the dataset along for training,
in this case, using an [inaudible].
As we add the random versions,
it is easy to lose track of which dataset version
was used as an input to what experiment run.
Therefore, when we submit a run using Azure Machine Learning,
several metadatas track for us.
This includes the datasets
that we passed along through our experiment,
get tracking information,
our particular experiment configuration in terms of parameters,
as well as any metrics that we
might have collected during training.
Next to the inputs to our experiment run,
Azure Machine Learning also helps keep track of outputs.
For this particular run,
you can see that one model was registered,
for example, when we finalized the run.
In Azure Machine Learning, we can register a model as
an asset that can be versioned and tracked for its users.
Similar to the tracking details that we
saw for the experiment run basis,
the model registry catches
all the metadata associated with your model,
including the experiment that was used for model training,
tags that we might add to the model,
as well, we can find
an overview of the endpoints
where a model is deployed for scoring.
To sum everything up,
from the inferencing endpoint displayed here,
Contoso is able to track
what model version was used for rental advice.
They can trace this version back to a particular experiment,
as well as through a particular version of their input datasets.
Then from the dataset definition,
they can linear us back the raw files that define the version.
In this session, we looked at
the tracking capabilities in Azure Machine Learning.
To recap, keeping well track of
your assets is important for a variety of reasons.
For most, the ability to reproduce
the same outcome during experimentation matters.
It enables us to replicate
specific results we obtained at the past,
as well, it allows others to build forward on existing work.
There might also be legal constraints
that your team needs to comply with.
Audits or legislative requirements, like GDPR,
have increased the importance with
teams to keep well track of their assets.
From an operational standpoint,
tracking helps you identify the impact of issues as the
might arise with data or models in production.
For example, you might find out that
the data quality issue caused
the model version to start producing correct outcomes.
Thanks to tracking, you might identify that, in practice,
multiple model versions were
impacted by the same corrupt datasets.
Whatever your motivation is to track,
keep in mind that Azure Machine Learning can help you to
simplify this job with its tracking capabilities.
[MUSIC]