>> Okay.
So this is the second installment
of how to use Azure Machine Learning Services,
and we wanted to make
an installment in bite size so people could
get a feeling for what's going on
and they could jump back and forth and know where to go.
We just talked about the setup. You do that one time.
>> Yes.
>> Okay. Now let's talk about the actual experiment.
How do you go through the experiment?
Because you mentioned that
our services are really good at
the data prep and
the experimentation site. Let's get into that.
>> Sure. Perfect. So I have
the Azure Machine Learning work bench
installed and the first time you open this,
it will ask you the same experimentation account which
you created and it will just authenticate you.
So this is our way to find out
the actual user who is supposed to use that.
So once you do that,
you get this kind of welcome screen.
Now in this welcome screen,
we show you some starter videos and
then essentially there are
links to the examples and documents and things like that.
What you would require to
start with is "Create a Workspace."
Now, think of workspace as like
if there are multiple data scientists.
Each data scientists could have
their own workspace which is
associated to the same experimentation account,
and then each workspace would have
different projects which the data scientist can use.
So once you've created a workspace,
you create a new project inside it.
For that, again, you just
provide the name of the project,
you can write some description
if it's already in a GIT Repo,
then you can give that URL.
Now, you can start with
a blank project or we provide you with a lot of examples,
like different frameworks, like CNTK,
Tensor Flow, Classical Machine Learning.
So you want to do Regression, you Spark.
So it's really easy to start as well.
So we have provided that.
Now, because I'm going
to talk about the Income Prediction Application,
its the application which I've built,
so it's already pre-loaded over here,
I am going to start and we'll see how it goes.
>> Fantastic. So before we get there,
the experimentation service is creating a service where
there's workspaces and associated projects.
>> So the user creates it.
>> Okay. But they go into the experimentation?
>> So yeah, it gets associated, linked with it.
>> So when you're looking at that experimentation,
you can actually keep track of the various workspaces.
So let's just say, you have a workspace for
doing image classification for blah,
you have another one that's doing NLP for something else,
and you have another one that's doing we
want to be able to suggest products.
Those could be three separate workspaces
and then each one of them you might have
projects that either improve upon
other ones or that are just different aspects of that.
>> That's right.
>> And for collaboration everyone can
have access to specific workspaces and their projects.
>> That's right because we have [inaudible].
Also, essentially, you can provide that URL to
your co-worker then you can share it.
>> Fantastic. So now that we've got that straight away,
now we're in a specific project
and workspace with something you've already set up.
>> That's right.
>> All right. Let's take a look at it.
>> Cool. So when this workspace loads,
essentially, you would see a readme kind of a file.
In this case, I've already provided some information,
but it could be a blank if it's a blank project or based
on that template what you have.
The second step is to actually bring in data.
That is where on the left hand side you will see
this tab which is that the "Data Tab",
you click on this "Plus" and then create a data source.
We support a lot of data formats.
So it could be CSV,
Excel, or even a Sequel Database.
In this case, I have my data which is stored in a Blob.
So what I would do is select "CSV" format,
click "Next", "Blob", and then provide the "Blob" URL.
>> Okay.
>> So once that is done,
my data is loaded already.
So this is the data which we have.
Now, when you load the data,
you don't want to load the entire data.
So you bring in only a sample dataset.
>> Okay.
>> Because as long as
your dataset is a representative sample
of the whole dataset,
you are fine with it.
In this case, I'm loading just the first
10,000 rows and see what the data tells.
>> So for clarity, the actual data that you
are working with is in Azure Blob storage?
>> Yes.
>> So when you're tinkering locally,
you're just tinkering with about 10,000 rows?
>> Yes.
>> But when you're running the experiments,
is that happening on the entire dataset?
>> Yes. That's right. So the first idea
is just to get a feel of data
because that data tells your story.
So essentially, you try to figure that out.
Once you're happy with it, you can scale it out.
>> Fantastic. So I've heard that we have
some really cool data
prep things in
our particular application that we're looking at.
Why don't you describe why someone would use it?
Because, as you know,
if you're a data scientist, we
spend the majority of our time.
>> 80%.
>> We are literally data janitors.
We clean stuff and so we have
some nice things to help with that.
Why don't you show us what those are?
>> Sure. So in this case,
I loaded the first 10,000 rows.
It immediately tells me that
how many data rows are missing or it could be erroneous.
Now erroneous could be like it's
a numeric column but there is a string over there.
So we can figure it out.
The other thing is we can also look at the pattern of
the data because you want to
see like which are your outliers,
what kind of distribution you have,
what are mean, median,
and all those metrics which is there.
So we show you that part as well for all the fields.
>> And that's really nice because, in this particular,
dataset you might find out that if everyone is over 60,
they are going to be making over it.
This is another dataset.
You might just figure out from all of
your data that you just less than
symbol in order to do machine learning in
this particular case, but this is not the case here.
>> That's right. Yeah. Everyone would
have an income greater than 50K.
You are right.
The other cool thing which we
have is like transformations.
Now, a lot of data scientists would
be familiar with either R or Pandas,
Python, or things like that.
But what if you could do some of
those transformations without actually
writing a single line of code and,
intelligently, just show it visually and
then we generate all the code magically for you.
So in this case,
I have this age field which is there.
It's 1982 or something.
But I want to bin it for
example, for simplification purposes.
First, I have do a "Prepare"
basically and create a Data Preparation Package.
Now, the idea behind the Data Preparation Package is
that you do some transformations,
you do some joining whatever you
want to do, but you want to scale it out.
So we created as a separate component effect.
>> I see. So a Data Preparation Package in
this case is once you want to do cleaning,
it's not just happening on the 10,000 rows,
you've got to define a way to do it on
that huge dataset that's on the box.
>> So we can generate the Python code behind this.
Now, I already have
a Data Preparation Package which is available.
And right now, you see I
haven't done any transformations.
It's the same dataset as you saw over there.
What has changed though is now on the top side,
you see a new "Transforms Menu",
which was not visible when you were
just looking at the data.
So you select a particular column, in this case,
for example I select "Age" and you can see
all the transforms which are
possible on a numeric kind of a column.
So in this case again,
I don't have missing data but a lot of times,
we have to deal with missing data,
erroneous data, so you can do all those replacements,
and when you do those replacements,
we actually give you the option.
So it's not just one value,
but you can do
multiple columns and then replace with some mean,
median and things like that.
>> So when you're
looking at the tab before that's the raw data?
>> Yes.
>> You can look at like the distributions,
you can look at missing values, etc.
But then when you want to start
doing some transformations on
the data to make it palatable
for the machine learning algorithms,
then we would do this deep
prep stuff to actually prepare.
>> That's right. So for this,
I'm going to show you how to bin a particular age.
So I'm going to tell this that
"Derive Column by Example" and I'm going to
tell it that I want to bin say 90 as 90 to 100,
and I just show one example,
and you will see that it
filled all remaining rows by itself.
>> I see. And it's doing this locally on a 10,000
but we're defining a way to do
it on all of the data when we do training.
>> That's right. That's right.
And I just click "OK" and you
see there is new step which is created on the right side.
Now the good thing is if you want to edit it,
you can go back in history and "Edit" it.
And you can have more options so say for the example.
I want to give it a name
or I want to change 90 to 110 or something like that.
We can still do that.
So and you can of course "Delete" it.
So if you delete it, the dataset would be
back to where it was
and you can do it for any of the steps.
>> So eventually, you might have a set of steps
of transformations that then make it a little bit better.
For example, we have missing data,
we didn't replace it with something,
we can try and an experiment
replacing with the mean or the median or whatever.
We can also remove probably columns with missing data.
But this is where you do all of
the data wrangling type of work.
>> That's right. Now, for today's experiment,
I'm going to use notebooks.
So a lot of data scientists
these days use Jupyter notebooks.
So Jupyter is basically acronymous.
And then you can do R,
you can do Python, or you can do Julia.
So today, we are going to focus on Pandas.
So it's Phyton basically.
If you did transformations here,
you can use the same dataset in
a Jupyter notebook or you can start from fresh
and then have that entire experience
of reporting the data
and then doing the deep prep and things like that.
>> So the option you're presenting is I could use
the preparation stuff I did
before or I could just go directly to the data.
>> Perfect.
>> So what we're going to do then is we're
going to break it up here because
data prep is obviously a lot of stuff and we want
people to be able to find this about data prep.
So thank you so much. In the next episode,
we're going to be looking at how to actually
train and operationalize the models. Does that work?
>> Yeah.
>> All right. We'll see you then.
>> Yeah. Thank you.