>> You're not going to want to miss this episode
of the AI Show where we talk
about Azure Machine Learning Service datasets.
Imagine being able to know exactly where your model came from,
how it was built, and what data is in it. Make sure you tune in.
[MUSIC]
>> Hello and welcome to this episode
of the AI Show where we're going to
talk about something called a dataset.
I have with me my colleague.
May you tell us who you are and what you do, my friend.
>> My name is Mae. I'm a PM working in Microsoft
and the product I'm working on is Azure Machine Learning datasets.
>> Fantastic. So why don't you to tell us what datasets are.
>> Sure. Datasets is that assets in
your Machine Learning workspace where we help you connect to
the data and your storage service and
make the data available for your machine learning experiments.
>> So when you say asset,
it's basically are we copying data?
Does it cost more? What's going on there?
>> No. So by creating a dataset is you just create
a reference to the data in your storage service.
We do not copy your data.
So there's no storage cost incurred when we create our datasets.
>> So when you're using a dataset it's basically
a pointer to some other data
that's stored on storage. Did I get that right?
>> Exactly.
>> So why would you do that?
Why would you make a dataset
if you could just point it directly to storage?
>> Well, first we make it easy for you to access your data.
What you only need to register data once and then you can
reuse it across different experiments
or share it with your colleagues.
So again, we integrates well with our training products.
You can use datasets as
a direct input for your script [inaudible] ,
Azure Machine Learning pipelines
and also Bayesian machine learning.
Third, we help you check where data has been used.
For each experiments run,
you will be able to see this is
the input datasets I used for this experiment and you can
also even register in a dataset with
your models so that users will be able to know,
okay, this date, this model was traced by
these datasets for reproducibility and audit purpose.
>> So let me see if I understand this.
There was three reasons you said.
The first one is because it's
a shareable asset that anyone can use.
>> Yes.
>> The second one is because it's easy to
put into experimentation and
the last one was this thing of being able to
trace where the datasets have been used.
>> Exactly.
>> Okay. So let's start first because I
wanted to see how you actually use it.
So let's first start with how do you create a dataset?
Number one and number two, how to use it? Can show us?
>> Sure. Definitely.
>> All right. Let's do it.
>> So I'm in my Machine Learning workspace.
>> Okay.
>> We have datasets under assets.
You can create datasets for
either local files from your data storage or your web URLs.
>> So let me pause right there.
>> Sure.
>> So when you when you say let's go back
to that little dropdown here.
So when you say create dataset.
So from local files it's basically I can upload stuff?
>> Yes. Exactly.
>> Okay. From a data storage if I already
have a storage that I've mapped to
my Azure Machine Learning. I would say, is that right?
Okay. So far I'm two for three.
I'm not sure what it means by web files.
Why wouldn't you describe what that is.
>> Web files is just you can paste
a URLs for public datas, for example, right?
You can paste a URL that step off the data together.
>> I see and does it pull it down and put it somewhere for you?
>> No. We just record that
as a pointer and when you use the datasets,
we will help you load data from the web.
>> That's pretty cool. So in essence it's like
if there's a public Kaggle dataset or some other datasets,
you can have a web referenced
directly to that CSV file and I'm done with this.
>> Exactly.
>> Fantastic. All right. Sorry. I interrupted you. Let's make one.
>> Sure. I'll use from my datastore.
So these are all the storage container I have.
I'll use the workspace default
Azure Blob here and if I click "Browse",
it will show all the files available in
my Blob Storage and by creating this,
you can either point to a single CSV files or our full folders.
So which means multiple files.
So in this case, I'll just point to these folders.
>> I see. Each CSV file has the same tabular structure.
>> Exactly. Same schema.
>> Before you go to "Save",
so when you pointed it to datastores,
that's the mapping of the actual Azure Storage
to the machine learning workspace, is that right?
>> Yes.
>> Okay. Cool.
>> So I'll click "Save" and give my dataset a name.
When I click "Next," it will try to pass my data into
a tabular format and it would
help detect a file formats delimiters,
column headers all for you and
this order first few rows off my data,
and if I click "Next," this is the schema of my data and
I can choose to drop
a few columns that's not relevant for my experiments.
>> I see.
>> Or I can change the column type if I want.
Then after that, I will click "Done".
So then these datasets
will show up in my Machine Learning Workspace.
>> Got it and so now that asset is
shareable across anyone that's in the workspace?
>> Exactly.
>> That's awesome and this feels almost using Pandas,
but storing all this stuff in the actual workspace. Is that right?
>> Exactly.
>> Cool. All right. So how do you use it?
Well, you're going to show us something here.
>> So this is the overview page of
the dataset I just created as for how do I use it.
So there are two ways to consume your dataset.
The first way is to consume it directly.
So here are the sample code.
You can load, after you get your datasets by
name from the workspace object,
you can load it into common data frames like
Pandas or Spark and then data scientists can
continue their data provisional official engineering
using the libraries that they're very familiar with.
>> That's really cool because it actually puts
in your sub idea in resource group and everything.
So your workspaces ready. You could just literally
copy this code and put it into your notebooks, right?
>> Exactly. So that's the first way to consume it.
The second way is more for
deep learning scenarios where
data frame doesn't really make much sense.
>. Right.
>> So datasets help you mount your data
from your storage service to the remote compute.
>> Right.
>> I can show you a demo.
>> Let's do that because as you're getting the tab open,
I'm a computer vision guy.
I love computer vision and it is you don't
want to copy all of the images into the container.
>> Exactly.
>> You want to somehow mount it.
So I'm interested to see how that's done with datasets.
>> Sure. So here I'm
loading my dataset for Azure ML open datasets.
I'll use MNIST here and
I will register this MNIST dataset with my workspace.
By clicking that, this MNIST dataset
will actually show up in my workspace.
>> Obviously, you created a dataset is encode this time?
>> Yes.
>> Got it. Cool.
>> By registering in this job in the list.
So MNIST dataset here.
>> Cool.
>> Cool. So here I'm going to do some data exploration
and so what I will do is to mount
my MNIST datasets to my Notebook VMs.
So again we don't copy your data.
Your data stays in the Blob Storage and here's just to mount data
to the Notebook VM so that we can take
a look at a few sample files.
>> I see. So when you're calling MNIST dataset mount,
you're mounting it to a path in the container,
but it's not copying the files over?
>> No. It's just mounting.
>> Got it. Cool. So this order like simple images and labels,
it all looks good and now I'm going to
start configure and training.
>> Cool.
>> So I will first get my dataset back from the workspace by
name and dataset to
path shows you what other files available pointed by your dataset.
>> That's cool.
>> Then here I'm configuring my TensorFlow estimators
and the one thing I want to highlight here
is how you use datasets as the input.
So you can call it dataset.as named input. as mount.
This basically tells to
mount my MNIST dataset to this compute target.
>> Got it.
>> This is my training script.
It shows you how to get your data.
So as you can remember
we pass this as an augments and in your training script,
you will get this data folder back and it basically points
to the mounted path on your remote compute.
>> If I remember right, this is actually running
inside of a remote compute inside of a container.
>> Yes.
>> So basically when you say the data folder,
it thinks it's just a local file on the container when in reality,
the subsystem is mounted as a share.
>> Exactly.
>> Got it. Cool.
>> So that for data scientists,
didn't really need to change a single line of code in
their training script when they switch for
a local training to remote training.
>> That's cool.
>> Yeah. So you can just load your data as if local path.
>> That's cool.
>> So after all this configuration,
you can just submit the rounds and you will be
able to see the experiments
from your workspace and this
was the experiment I submitted just now,
and if you go to run detail page,
we have input datasets as saved as one properties.
So this is where we help you
chase which datasets was used for the run.
>> So that is what it's called data lineage.
For example, when you run something,
obviously a model is produced.
In the model you can point back to the experiment that made it and
then you can point back to the actual dataset you used as well.
Is that right?
>> Yes. So if you click this link,
it will link you back to the MNIST data sets.
>> Got it.
>> So this was the datasets that I used for this training.
As for questions about models,
you can actually register model with the dataset directly.
So I will just run the register model,
register my model with my training datasets and if I do that,
under MNIST dataset, you will see a model tab which
list all the models that has used this MNIST datasets.
>> That's really cool because
generally when I do machine learning,
I just run the thing and export the model.
But now if you do this in
a context of Azure Machine Learning Service,
you're able to trace the model
to the experiment to that dataset as well which is really cool.
>>Yes.
>> That's awesome. Well, where
can people go to find out more about this?
>> So you can find out more documentation about datasets in
our Azure Machine Learning documentation website
and we have lots of similar notebook in our GitHub.
>> Fantastic. We'll put some links
below so you can take a look at those.
Well, thank you so much for spending some time with us.
>> Thank you, Seth.
>> Thank you so much for watching.
We've learned all about how to use
datasets in Azure Machine Learning Service workspace.
Thanks for watching and we'll see you next time. Take care.
[MUSIC]