>> In this special build edition of the AI show,
we will hear from Vijay Khanan,
Senior Program Manager on the Azure Machine Learning Team.
You know what, your machine learning is only as good as your data.
VJ will show you how to increase your productivity with data
labeling using Azure Machine Learning. Make sure you tune in.
[MUSIC]
>> Good morning. Thank you for joining this session.
My name is Vijay Khanan.
I'm a Program Manager in Azure Machine Learning.
In this session, we'll be focusing on data labeling in
Azure ML and how ML
assisted labeling can help in the labeling process.
Talking about data labeling,
this is an important step before
you can train ML model for computer images.
You have images, you need to specify the expected output,
whether you are looking to classify the images
or you're looking for objects in images.
Next, we will talk about why
it is challenging to do data labeling.
First, it is about data.
If you have a huge amount of data,
you want to avoid moving the data
around and still make it available to do the labeling.
Second, you need your multi-user labeling tool that
will allow multiple people to
work in parallel to do data labeling.
Third, the workflow.
You need to distribute the task among many people
so that they can work in parallel on different data points.
Also, keep in mind, the data labeling can be iterated.
Depending on the volume of the data and the number of people
involved in labeling can become your project management problem.
There you need to optimize on cost, time, and quality.
Quality control is extremely important
because poor quality label means poor models.
Finally, you want to do
your data labeling without compromising on fundamentals,
that is privacy and security.
Many of you might be working on sensitive projects or solutions.
Azure ML data labeling integrates very well with the datasets.
It consumes the dataset and after labeling,
it produces a dataset.
You get all the benefits that comes with the dataset.
It comes with a pre-built browser-based multi-user labeling tool
so you don't have the rate or maintain your own tool.
On the work floor end,
it takes care of task distribution and
also allows you to iterate on your labeling project,
which is very powerful feature.
It has various capabilities for project management and, of course,
it is built on top of Azure Privacy and Security,
the labelers who are going to do the labeling are
authenticated using AAD or MSA.
Here is our demo flow.
We will use a dataset based on images that is kept in Blob store.
Then we will create a labeling project as a data scientist.
We will label few images ourselves,
then we will sign back into
the project management portal and check the quality of the labels.
Finally, we will export the labels as a dataset.
Next, we get into the very interesting portion,
which is the demo, demo of the AML data labeling tool.
What you see here is the Azure Machine Learning Studio.
In Azure Machine Learning Studio,
we have various assets like notebooks, auto ML,
datasets, experiments, pipelines, compute,
data store, and etc.
Towards the bottom, we have the data labeling capability.
Once you click on the "Data Labeling Capability",
it will load the projects list page as you see on the screen.
In order to create a new data labeling project,
you should click on "Add Project" button which is on the top left.
Creation of the project includes 4-5 steps.
First, you need to provide the project name.
Next, you need to choose the project type.
Today, we support three project types;
image classification with a multi-class,
image classification with multi-label, and object detection.
You need to choose the project from the drop-down.
For this demo, I have chosen
the project type as object identification bounding box.
Now, I click "Next".
The next step is selecting or creating a dataset.
I have already created a dataset
pointing to my Azure Blob Storage,
which I'm going to use it in this demo.
At this point, you create your new dataset altogether.
Now, I click on "Next".
The next step is about defining the labels.
Now, I will go ahead and create the labels.
Now, I click on "Next".
This is the labeling instruction page.
In this step, the project manager or
the data scientist can draft a set of instruction
or guidelines for his labeling team
which would help during the labeling process.
Now, I have typed in the instructions here.
Now, I will go ahead and click "Next".
The last step is the ML assisted labeling.
In this step, the project manager can seek
the help of the machine to do the automated pre-labeling.
The ML assist process will kick in after
receiving certain number of manually labeled images.
In order to turn it on,
we need to enable this check box,
Enable ML assisted labeling,
then we need to provide a compute.
Already I have created a GPU compute for this project,
which I'm going to use it.
If you don't have the compute at this point,
you can still go ahead and click "Create Compute".
After providing the compute,
you just need to click "Create Project".
The project creation process takes a few minutes.
Now, we have created our project.
I will go ahead and click on the Project.
Once the project page is loaded,
it shows the default dashboard page.
This page shows the progress of the project.
We have not labeled any image so far,
so the progress is shown as zero.
Now, I will go ahead and click on the Data tab.
The data tab has two sub-types,
Dataset preview and Labeled data.
Dataset preview tab shows
some sample images from the source dataset.
Now, I will go ahead and click on the Details tab.
The detail tab has various sub-types.
The first is the project details.
By clicking on the Project details,
you'll be able to see what is the project name,
what is the project type,
when it was created, and who created it.
Clicking on the Input dataset,
we'll be able to see what is
the dataset name and what is the data store.
Then we have the Intermediate Label Storage tab.
When we create a new Azure Machine Learning workspace,
internally, a workspace gets created.
We will leave it at this workspace
when we export the labeled dataset.
By clicking on the label classes tab,
you will be able to see what are
all the various classes which we
defined during the project creation.
During the execution of the project we will
also be able to add new labels.
Again, the instruction also
can be updated during the execution of the project.
Also, ML Assist processing can also be
turned on or off during the execution of the project.
Now, we have configured the project.
Let's get into the most interesting portion,
which is labeling data.
I'm going to go ahead and click on
the label data button on the top,
which is going to display their labeling experience.
When the labeling experience loads it,
it shows the instruction page.
In this case, I as
a project manager or a data scientist of this project,
I myself can label the data.
So I will go ahead and click "Start labeling".
In order to do manual data labeling,
first I need to identify the object which is of interest to me.
In this case it is cars.
First, I need to go ahead and click on car as my label.
Now, I will go ahead and draw
a rectangular bounding box for the gray car.
Here, I see a red car overlapping with the existing car.
In order to draw a bounding box within a bounding box,
it is easier to lock the existing bounding box.
In order to do it, first,
we need to go ahead and click the "Lock" function,
which will lock the existing bounding box.
Now, I'm good to create a new bounding box for the red car.
After creating the bounding box for the red car,
I will go ahead and click on "Unlock".
Now, I have a bounding box within an existing bounding box.
If things looks good,
I will go ahead and click "Submit" for my next image.
In this picture, my object of interest is the person.
So I will go ahead and click the "Label as a
person" and then draw a rectangular bounding box for this person.
Here, I have the option to zoom into this image or even zoom out.
I will be able to adjust the brightness of
the image as well by adjusting the slider control.
Also, I'll be able to adjust the contrast of this image as well.
At times we might get an image
which does not fit into any of this label category.
To handle it, we have the skip function as well.
If you want to skip this image,
we still can go ahead and click "Skip".
Here everything looks good.
I will go ahead and click "Submit".
In this case, the object of interest is the boat.
So I will go ahead and click on
boat and then I draw a bounding box.
Everything looks good. I will go ahead and click "Submit".
Now, we have three images labored.
Let's log back into the project and
see what is the progress on the project.
By logging back into the project in the dashboard page,
you will see the progress of the labeled dataset.
At this point as a project manager,
if you want to review the work of your team of labellers,
you can click on "Data" and then click the "Label Dataset" tab.
Here you can review the labels of the labeled data outputs.
At this time, if you are unhappy
with any of the labels on a specific image,
you can still go ahead and reject that image.
What this will do is it will put back
this image back into the labeller queue for re-labeling.
At any point during the execution of the project,
you can go ahead and export the data.
Today, we support two ways to export it.
One is export as COCO file.
The other option is export as Azure ML dataset.
I will go ahead and export as Azure ML dataset.
It will take a minute or two to how their dataset created.
Now, we have successfully
exported the labeled output as a dataset.
The labeled output dataset is available in the dataset tab.
Here you would notice the exported output.
Now, let's get back to the ML-assisted labeling presentation.
Now, let's talk about ML-assisted data labeling.
This is a feature we have released in
public preview for image classification.
For object detection, it in limited preview mode.
Please contact us if you want to use this capability.
Now, let's talk about a ML-assisted labeling data flow.
At high-level, this is how it works.
You have labeled data,
there is a task creator component
that create tasks for human labellers.
Human labellers pick their task and start
labeling so you can get the label data.
This thing continues, and of course
you are free to export the labels if you want.
However, we do something more interesting
once we have certain number of labeled data available.
We train a machine learning model in
the background and used the output of the model in various ways.
Initially, you can expect
the quality of the model will not be good,
but still it is usable.
You can use the output of the model
to cluster the data points together.
At this point, we start giving cluster tasks to
labeller are they start seeing
very similar images coming together.
The idea here is to reduce the cognitive load or
context switching for them to do a better and faster labeling job.
As the model quality exceeds the confidence threshold,
it will start producing the pre-labels for each image,
which would help their labeller to
perform labeling at a faster pace.
Now, let's look at the demo for ML-assisted data labeling.
I have created three projects for this ML Assist demo.
One for bounding box,
the another one for multi-label,
and the third one for a multi-class.
All these three projects have reached their relabeling state.
Let's go into the bounding box project and see
how the machine has generated the pre-labeling bounding boxes.
I'll go ahead and click "Bounding box".
I click "Label data", start labeling.
As the first image gets loads up,
I see already the bounding boxes have been marked by the mission.
I'm very happy how the bounding box shows up here.
So I go ahead and click "Submit " for my next image.
Here, already the mission has marked
the bounding boxes for various objects like bus, car.
I'm happy with this as well.
So I click "Go ahead" and click "Submit".
Now, let's look up how
pre-labeling shows up for the multi-label project.
So here is the multi-label project.
I will go ahead and click.
Here the mission has already marked the pre-labels.
As a labeller, I don't need to do
anything if everything looks good.
So I'm happy with how the pre-labeling is done here.
So I will go ahead and click "Submit".
Again, on the next image,
the mission has already mark the pre-labels.
I feel things looks accurate,
I'll go ahead and click "Submit".
Now, let us look at the example for multi-class.
Now, already the mission has marked the pre-label for this image.
So this is identified as a subway.
I'm happy with the pre-label.
I'll go ahead and click "Submit".
Even the next image is marked as subway which is correct.
So things looks good.
I will go ahead and click "Submit".
This is the demo for ML Assist.
Here are the session resources.
Feel free to try out our product and ML-assisted labeling.
If you have any questions,
feel free to reach out to us. Thank you.
[MUSIC]