Hello everyone, today we're going to be walking through how
you can build an image recognition model to recognize handwritten
numbers in Python.
I'm going to show you how you can do this
using notebooks in the Azure Machine Learning Studio and as
we build the model,
there's a couple of exciting notebook features that I want
to show you.
So first I'll walk you through how you can set
up your environment.
Then I'll show you some features that will help speed
up your coding productivity.
Then we're going to do some data exploration as well
as model training,
and once we finish this,
I'll show you a number of features that can help
you get your notebook to production.
Lastly, I'll show you how to use the Azure ML SDK
and what the SDK allows you to do is to
conduct your run remotely or you can use it to
register your assets to the workspace like your datasets,
your experiments, and your models so that you can keep
track of all the previous runs you've done and if
you'd like to refer back to them in the future,
you can do so easily.
So let's dive into the model.
Today we're going to be building a simple multi class
logistic regression using the MNIST dataset.
This data set consists of 70,000 grayscale images,
all representing pictures of handwritten numbers,
and each of these images consist of a an array
of 8 by 8 pixels.
So the input of our model is going to be
this array of 8 by 8 integers,
where each integer represents the intensity of that pixel and
the output of our model is going to be the
number between zero and nine that's written in that handwritten
image. So now that you know how this model is
going to work,
let's switch over to the Azure Machine Learning Studio now.
Before we begin, first you want to make sure that
you have a running compute and you can find that
in the top right hand corner here. If you don't
already have a compute setup,
you can easily create one using the new compute process.
And
from this window you can select a virtual machine with
the required hardware that you'll need.
On the left hand side,
here is our File Explorer,
so this is where all of our documents are stored.
You can see our .ipynb notebook file here.
This is also where you can find any supporting Python
files, yaml scripts,
or if you upload any data as CSV that can
also be found here as well.
I also want to point out that the Azure Machine
Learning workspace is a collaborative environment,
so if you have coworkers who have access to the
same workspace,
their files will also be stored here as well.
This makes for easy collaborative sharing,
so if you want to share your notebook file with
any coworkers,
all you have to do is link them the URL
to that notebook.
I also want to show the terminal and what the
terminal allows you to do is access your Linux machine
directly.
So from this window you can conduct any git actions
such as cloning in any existing repositories.
Or you can also push your code to any remote
repositories.
You can also do any pip actions here,
so if you're looking to update or install any packages,
this can also be done through the terminal.
So I'm going to enter Focus mode here just to
give us a bit more space to work with.
So before we actually check the notebook contents,
I also want to point out the table of contents,
which allows us to look at a high level overview
of the contents of this notebook,
so we know for today we're going to be doing
some data exploration,
some model training, and after that's done,
we'll output the model. And each of these sections in
the table of contents is designated by a markdown header.
So if you click into that section,
it will navigate you directly to that markdown header within
the notebook.
So let's jump back up to the top of our
notebook.
So the first two cells are for set up.
We're going to import some packages and then we're going
to set up some function definitions and then let's dive
into the data exploration section.
So the first thing we're going to do is download
our MNIST data set.
And then we're going to take that data set and
split it up into our X and Y testing and training sets.
And at this point we're probably wondering what exactly is
contained within these four variables and how we might usually
check is by printing either the variable itself or the
shape to see that it actually contains the contents that
we need.
So instead of actually printing out this code,
what you can do is use the variable explorer,
which provides a glimpse into the variables directly without needing
to write that code.
So the variable explorer gives information on all variables,
including the data type, the size of that variable,
as well as a peek into the value,
so we can see here for our X & Y testing and
training sets,
the testing sets both contain 10,000
elements in that array and the training sets contain
60,000, so you can see how those 70,000 images were
split up and you can see for the X variables,
this contains that array of 8 by 8 integers and
the Y variable contains that integer between 0 and 9.
So now let's randomly display a couple of images from
the X training set and you can see here that
we've selected 30 random handwritten images displayed here and directly
above is the corresponding number which we are going
to try to predict.
So now we're ready to begin training our logistic regression
model.
Let's first set our regularization rate to 0.5,
and then let's create a logistic regression class object and
set this to multiclass.
And here I want to point out a feature that's
useful if you're working with a library that you aren't
super familiar with.
In this example, we're using logistic regression from the
sklearn library,
and if you aren't familiar with parameters,
there's a notebook feature that we have integrated with that
provides help with things like auto completions and function help.
So if you start writing the name of the class
here,
we're typing logistic regression. Intellisense suggestions will pop up,
providing suggestions, and you can tab or enter to autocomplete.
And if you start typing ( this gives a popup with
hints as to what parameters are required to instantiate this
class.
So here you can see that some of these parameters
that we had in our logistic regression class are appearing.
So you can see multi class.
You can see random state as well as our C value.
And let's say you're reading a notebook that somebody else
has written,
and you're not planning to write any of the code
yourself,
but you still want to understand what's written here.
What you can do is hover over a class or
a method,
and this provides a full function documentation,
so you can see here
there is help on the parameters as well as some
attributes,
and examples.
And this works for libraries beyond sklearn.
So if we scroll up here and check out random
from numpy
this also provides the full function documentation,
so you can read this directly within the notebook.
And so let's run this,
and while it's training, you might notice that the cell
state for this particular cell has changed,
and what this bar here indicates is whether a cell
has been queued for running,
whether it's currently running, or whether it has successfully run.
And if a error occurs during the cell run
that will also be reflected in the cell status bar.
So from a previous run,
we've already generated a prediction,
so we're going to store this prediction using this logistic
regression model that we just created and store it in
a variable called y_hat,
and then we're going to take y_hat and compare it
to the actual values in
y_test to determine the accuracy.
So from our previous run,
we know that this model is able to achieve an
accuracy rate of 92%.
So we're quite happy with the results of our model.
The last thing we're going to do is output that
model.
So I'm going to insert some code here to help
us track our run and as part of this run
we're going to log 2
metrics: our regularization rate and our accuracy,
and once we have these two logged,
this will help us send these two metrics to our
AML workspace so that we can refer back to it
later.
The last thing I'm going to do is output our
model to a pickle file so that later we can
send this to our model in our asset section in
AML workspace.
So we've got our model.
Now, let's say we want to take this Python notebook
and convert it to a script so that in the
future we can send the script off to be run
remotely. First what we might want to do is take
our notebook and clean up all of the exploratory work
such as displaying these variables and displaying these images. We'll
probably want to take that out and then we want
to convert this to a Python script.
For the first part, I want to point you to
a feature named Gather and what Gather does is determine
all the code dependencies needed to generate a given output
cell and it will gather all that necessary code into
a new notebook.
So let's run gather on this final cell which contains
our model output and it should take only the essential
code.
So as you can see in this new notebook,
it only contains the required parts,
so all of that data exploration and image outputs that
we had generated earlier at the beginning of the notebook
has all been cleaned out and only the parts needed
to generate a model remain.
So at this point our code is still in this
.ipynb notebook format and we want to convert it
to a Python script.
We can do that easily using the export feature,
which you can find in this dropdown menu,
and if you select Python from the "Export as..." options,
this will compile all of your notebook code into a
Python script and download that locally.
And if you also want to share your notebook with
non notebook users,
you can also export as HTML or as Latex and
that just makes for easy collaboration.
So I have a copy of this gathered Python file
stored in my sklearn-mnist directory already,
so we're going to use this to send to a
remote run.
So I'll show you now how you can train this
script remotely as well as register your assets to the
AML workspace so that you can refer back to them
later here. I'm going to open up a new Python
notebook which contains new code and this notebook
we're going to use exclusively to explore the AML SDK.
So going back into focus mode,
let's dive into the code here.
So the first thing we're going to do is import
the AML SDK and we're just going to import azureml.
core.
And then we're going to specify the workspace that we're
working in,
and we can do this by reading in the contents
of our config.json file and sending this to
a workspace object here.
So if you're not familiar with the AML SDK,
we have some code snippets and what the code snippet
does is compile a list of common code excerpts from
the AML SDK so that you can insert that code without
having to remember what's required.
So for this particular action,
we're trying to configure Workspace.
If you type config, you can see here the second
option in this snippets menu is get workspace from config
so you can tab or enter to insert that code,
and this will import all of the code needed to
create this workspace object and read the contents of this
config
.json file.
So the next thing we're going to do is register
our data set to the workspace so that we can
access it later from here.
And then we're going to create an experiment object and
name this tutorial
-sklearn-mnist. And when we check our
experiments later,
we'll refer back to this name.
And then we're going to create our remote compute.
So here we have an option.
We can either train this script locally,
which is what we did earlier using our local compute,
or we can also take advantage of remote resources by
creating a cluster of Azure virtual machines.
So let's create a compute cluster.
And I'm going to run the cell and then create
a cluster named CPU cluster.
We're going to refer back to this later when we
set up our run configuration.
So now we can set up the configuration to train
on this remote cluster.
First we're going to create a new folder and this
is just the folder that contains our script as well
as any supporting files so that the run knows where
to refer to the documents.
And then we're going to configure the training job.
So let's create an environment first that contains all the
necessary package dependencies.
So we're going to specify here that we're using Scikit
learn,
and then we're going to set the package version.
And then we're going to create our script,
run config object, and here we're going to specify the
training script so the training script is the image classification
gathered Python script that we created, the compute target,
which is the compute cluster that we had created earlier
in the notebook,
as well as this environment which we have just specified.
And then let's submit our run to the cluster.
So while it's executing, I'm going to surface this run
widget and the widget is going to provide a real
time update on how the run's progress is going and
we can monitor this directly from the notebook.
So that's going to take a while to run.
These are the results from a previous run that I
had conducted.
And you can see here that we have logged a
regularization rate of 0.5 and achieved that 92%
accuracy.
So we like these results and the last thing we're
going to do is register this model to the workspace
so that we can refer back to it here.
So we've logged a number of different assets to the
workspace,
so now we can go and check to see what
those look like.
If we navigate to datasets,
this is where our mnist_opendataset is stored.
You can see that in the data sets tab,
under experiments. This gives an overview of all the experiments
that you have run as well as details for each
run.
So if we click into our tutorial-sklearn
-mnist,
you can see that we have queued one run which
is the experiment that we had just submitted and in
the past there have been 16 successful completions. So
If you click into a specific run,
it will provide metrics for each of those runs,
and if you have logged any custom metrics like we
had,
that will also be displayed with that run.
So if you had done any sort of exploration with
the parameters or changed anything up with the model,
you can refer back to the results in this experiment
section.
And then lastly, we had registered our model to the
workspace and here you can find that we have our
sklearn_mnist model.
So that concludes this tutorial on how you can use
the Azure Machine Learning notebook and workspace to build models
to classify images.
I hope you enjoy the session and thank you for
your time.