>> In this special build edition of the AI Show,
we'll get to hear from Tracy Chen,
Senior Program Manager on the Azure Machine Learning Team.
Sometimes when we're doing batch inference,
there's just a lot of stuff to do.
At the same time,
Tracy will introduce you to something called ParallelRunStep and
how it makes it easier to do that workload. Make sure you tune in.
[MUSIC]
>> Hello everyone. I'm Tracy Chen,
Senior Program Manager with Azure Machine Learning.
Today, I'm going to introduce to you how to run
batch inference using Azure Machine Learning Pipeline,
ParallelRunStep.
First, let's talk about what is batch inference.
Actually, batch inference is everywhere.
Cyber security team analyze
hundreds of thousands of e-mails for security threats every hour.
A retailer has millions of customers and want to
segment the customers for various marketing campaigns.
Glossary want to forecast the sales of
thousands or products for each store,
banks that we saw that the ATM machines.
I want to predict your money refill amount and frequency.
These are all batch inference scenarios.
They are really large workloads.
You are dealing with terabytes of data.
I don't expect to get instance responses.
You can wait for hours or days to check the results.
We know the challenge to run batch inference is
not easy to scale up and out with such large workloads.
You need to partition the data structure
also unstructure terabytes of data.
You need to distribute and
coordinate to the workloads across a cluster of
machines and the consolidated results returned from each machine.
You need to manage as a machine cluster.
You need to handle failures and errors.
We have the solution to help you.
It's the ParallelRunStep.
It's fully compatible with Azure machine learning pipelines.
It's a manageable stack with out-of-the-box parallelism.
You don't need to worry about how to split the data,
how to distribute workloads,
and how to manage the cluster.
Just tell us where is the data,
what model you are going to use,
how many machines you want to run the job.
We do all the heavy lifting for you.
If attacker failed and we would try it,
if a machine dies,
we retry it to another one.
By default, we retry three times and
you can also set the retry count.
If the job partially failed,
we still return the partial successful results.
Let me show you an example of how to
use ParallelRunStep to do batch inference.
In this notebook, I'm using
a pre-trend digital identification model and
run batch inference MNIST test the dataset.
Firstly, you need to prepare some resources.
Connect it to any workspace,
create a new one, or connect it to any existing one.
You'll also need a computer cluster.
Here, I'm using a CPU cluster.
Then you need to configure the inputs and outputs.
ParallelRunStep accepts Azure
Machine Learning dataset as the inputs.
In this demo, we're using images,
so I'm create a FileDataset.
We are putting the MNIST test images on a public datastore,
so I'm creating a datastore,
and connecting to the public blob.
Then create a FileDataset from this datastore.
So in order to reduce the pipeline,
I'm creating a pipeline parameter
for this input datasets so that when I the resubmit the pipeline,
I can pass in a brand new dataset.
I will show you how to use this parameter later.
Now, you have the inputs ready,
you need to create the output to start the inference output.
I'm using a PipelineData for the output.
This section is just to prepare the model,
just the model model and then register it with your Workspace.
Now, let's build the ParallelRunStep pipeline.
It's important that you provide the inference group,
we call it the interest group.
Because you need to spread the data into multiple mini-batches.
The inference group tells us how
to do inference on each mini-batch.
So there are two major functions within interest group.
One is the init function.
So init function is easy to do common preparation
such as the laws of the model into a global object.
So run function told us how to do inference on the mini-batch.
So it's very small for loop to iterate on each of the mini-batch,
and it gets the prediction.
You also need to specify
the environment by the calling environment
to run your inferences group.
ParallelRunConfig is a major config that you're going to use
to wrap up all the configuration for ParallelRunStep.
There are few very important parameters.
Mini-batch size define the size of the mini-batch.
It'll be passed to [inaudible] and inference group.
It's the demo I'm using for this [inaudible].
It means a number of files per each mini-batch.
Error_threshold, it defines a number of
failures the entire job can tolerant.
[inaudible] is the number of
file failures and for tabulate is the number of record failures.
If the count that goes beyond this threshold,
so the entire job will be aborted.
You cannot have 32 minus 1 if you want treated all failures.
Output_action define how to organize output.
You can choose between two values,
summary only, or like append_rows that I'm using here.
Append_row means you will
combine all the outputs into one output file.
I'm also using append_row_file_name
to customize the output of file.
So this mnist_output.txt will
contain all the inference results for the entire job.
Process_count_per_node and the node_count
can be used to find here the computer resource.
There are also several other configurations
like I'm not using here,
like logging, Time auto setting,
and the retry count.
All these configures can be defined as
PipelineParameter so that when you resubmit runs,
you can find to these parameters.
Now, you have everything you need to create a ParallelRunsStep.
Passing the Config, the inputs and its output.
That's it, you are all set.
Submit to the pipeline_run now,
check the results later.
As I've defined the input dataset and
set a config says pipeline parameter,
I can now submit the pipeline_run with
a brand new data input and also find too those parameters.
Go to the Azure portal to check the results.
Here are several runs I submitted earlier,
go to the Run detail page to get the run ID,
and go to your configured output location to get the results.
Here are a few links for resources.
If you have any questions regarding
batch inference or ParallelRunStep,
reached out to me or my team. Thank you.
[MUSIC]