>> You're not going to want to miss this episode of the AI Show.
We'll look at batch inference with something called
a parallel run step and more next episode.
Don't miss it. We'll see you then.
[MUSIC]
>> Hello, and welcome to this episode of the AI Show.
We'll talk about batch inferencing.
I have my friend here, [inaudible]. How are you doing, bud?
>> Hey, I'm doing great. Thanks for having me.
>> Tell us who you are and what you do?
>> Yeah. I'm [inaudible] I'm Program Manager in Azure AI
focused on machine learning model inference,
and I'm here to talk about new batch
inferencing capability in Azure machine
learning that allows customers to
run inferences on large-scale datasets.
>> So let's get a little bit of context.
Because it feels like when you say inference,
what does that actually
mean for people that haven't heard that term?
>> Absolutely. So stepping back on machine learning,
you typically start with data,
with known answers to your problem.
>> Correct.
>> Then you build a model that
predicts those answers accurately to start with.
Once you have a trained model,
the next step is to put the model to real test,
against real-world data that it has not seen before.
>> I see.
>> That's when you do inference,
and depending on the scenario you're focusing on,
your looking at real-time inference
where you want the responses to
come back immediately where you've typically exposed,
rest endpoint and a request response.
Low-latency pattern.
Or for those scenarios where
instantaneous response is not mandatory,
you could process this offline asynchronously.
A good example of this could be like an offline fraud detection,
lead generation, demand forecasting,
there are numerous scenarios around this.
>> This is interesting because like foolishly I always say, hey,
I put this model R and I need to predict right
away on single things.
But when you look at the way these things are set up, of course,
there's going to be some scenarios where
not only do you want to it offline,
but you have to because there's
no other way because you have a lot of data.
>> Yes. You're talking about terabytes of data are
being processed offline to get inferences
in a highly salable performance and
cost effective way by fully leveraging the power of Cloud.
>> That's awesome. So now that we know
why batch inferencing is important,
can you tell us what it looks
like inside of Azure Machine Learning?
>> So we handle high volume basically using three steps.
First, we partition the data
using the power of Azure Machine Learning dataset.
Dataset is a resource for like exploring,
transforming, and managing data in Azure Machine Learning.
The next step is to delegate this inference
workloads to the auto-salable Azure Machine Learning compute.
Then once the results get done,
the output is automatically stored in user-specified storage.
>> I see. So let me see if I understand this right,
because this is sounding very similar to
the experimentation pattern except
now we're moving into inference.
First, we have datasets.
In the datasets, we firm it out like we used to do.
I'm spit-balling, you tell me where I'm wrong?
>> Yes.
>> The clusters will
scale up and down depending on how much you're doing,
and then it will put the answer somewhere where you tell them.
>> Absolutely.
>> Did I get it right?
>> Absolutely. Once the job gets done,
it has join the VM, you don't pay anything.
So you get the cost optimization aspect as well.
>> That's fantastic. So now that you
told me all of these glorious,
I feel like I need to see how it works.
>> Absolutely. So for enabling this capability,
just to set some context or using
the Azure Machine Learning pipeline,
and pipeline for starters as we all know is a set of
computational tasks that you've stitched
together to enable the machine learning workflow.
>> Correct.
>> We have added a new step called parallel run
step to do the processing,
which enables this auto parallelism
using the dataset as I mentioned,
or it enable scaling through the Azure Machine Learning compute.
The key thing about the parallelism is
that there is no use of orchestration here.
In the past, you had us to write the MPI code,
to do all this complex larger.
>> That makes sense. That's because when it comes to batch,
you don't really need to do it all in sequence.
You are like, take this and then output to the same place.
>> Absolutely. We've also got two datasets.
For both structured as well as unstructured dataset.
Structured dataset could be a CSV,
TSV or even like a party or a SQL query.
Unstructured could be a file or a blob.
>> Or like for me I do computer vision.
It could be a folder of files, for example.
>> Yeah.
>> That's fantastic. All right.
So let's see some code.
>> Absolutely. So for this demo,
I'm going to show a digit recognition model
that is already trained using an MNIST dataset.
For convenience, I have converted all the input into PNG files.
So we could show the file dataset in action.
>> We'll just go on. Because everyone knows you can
download MNIST in a CSV,
or it's a zip file of bits,
but this is actually more of a vision scenario.
>> Yeah. I'm going to walk through the Notebook here.
I'm going to skip through some of the generic steps.
For example, connecting to workspace.
Workspace is a container where
you have all the machine learning artifacts.
You connect to the workspace.
You assign machine learning compute here.
I've assigned a CPU cluster here but you could also do
a GPU if you want to.
>> For MNIST I feel like CPU is enough.
>> It's more than enough.
Here, you specify the input data store.
We have a blob, a container.
You've already copied all the files
and we're pointing with our data store.
This is the interesting piece,
where we have the file dataset to now manipulate the data,
and to reference the files here.
This is the MNIST data that I spoke about,
and this points to the PNG files.
Now, if you want to play with structured data,
you could use the tabular data set as well.
>> That's cool.
>> All right. So we download
the model and we register the model in the model registry.
>> Or you could use a model that's
already in the registry too, right?
>> Yes, absolutely.
Then you wrap your inference script here.
This is consistent with what we do for real-time inference.
>> I've already seen all of this.
>> You initialize the model and then you have
a run function that runs inferences in this case,
for a subset of models.
You run inference for a subset of files and
you take a parameter called mini-batch,
which controls the number of files that
get run for every run execution.
>> I see.
>> Corner dependencies.
Again, there's a standard step.
The crack starts here.
This is the parallel run step.
That is the configuration input
for a batch inference that you define the score.py.
You define the file input.
I mentioned about the mini-batch size.
You saw the number of files that get executed in
every single run execution.
Error threshold is interesting parameter.
This defines the number of files that get ignored for error.
So in this case, if it exceeds 10,
it's going to automatically stop the job.
So it saves you a lot of cost.
>> That's smart because, for example,
for some reason maybe someone submitted
an MNIST digit that was full color,
which has the wrong tensor shape, it'll break it.
It doesn't happen very often.
Or for example, the Alpha channel,
that's going to break it as well.
This is a way of being able to say,
yeah, that one, okay.
Let's go on to the next one. That's cool.
>> Then you create a pipeline step which has got the model
and the input and output and
then it's an experiment run from there on,
where you run the experiment.
Here is the reset I processed a few minutes ago,
which shows the first files that got recognized.
>> So as I'm looking at this,
I'm coming to an interesting rule.
Correct me if I'm wrong because I come to
this realization sometimes or in the videos as you know.
This feels like we're talking about batch inference,
but a parallel step can be used with anything, right?
>> Beautiful impression, right?
To think about it, it's like a paddle mop-up operation.
So you could use that not just with
a modulus input but also for actions related to inference,
for example, pre-processing, post-processing and so on.
For pre-processing, a good example is you want to reduce
the size of your images before setting them for
inference. Standard use case.
>> For example, tensorflow, you could
put it into a TF records set,
for example, which is really smart.
>> Yes. Or even post-processing,
you might want to put some bounding boxes on top of the images to
see what objects are detected from your detection model.
You could do all these operations.
It's a parallel map operation that you can take
advantage of this stuff for doing that as well.
>> Because in theory,
there's certain operations where it's like,
look, we just need to do a thing on all the things.
Well, let's scale up some machines and do the thing,
and all the things in separate little batches.
>> Absolutely. It boils down to how you leverage
the auto-parallelism and
the auto-scaling capability to your benefit.
Other is by using model as an input or without it,
for pre and post-processing it works for both scenarios.
>> That's amazing. So just to finish up,
can you summarize what we've seen
today and how can people learn more?
>> Yeah. We've looked at how we could
use the parallel run step in the pipeline
to execute your batch inference jobs in a highly scalable,
performant, and cost-effective way for large-scale dataset.
>> This is pretty cool. Because like I said,
there's a couple of vision scenarios
where I literally need to go through every file,
make sure it's the right size,
make sure I've converted into a batch of TF records.
That's not inference, but it's something I could do with this.
If I want to do inference I could do that.
I mean, there's ton of cool things here.
>> Absolutely.
>> Awesome.
Well, thank you so much for spending some time with us.
Thank you so much for watching.
We are learning about the parallel pipeline step.
Did I get it right?
>> Yes.
>> How it can do a number of things
including batch inference, which is of really good use.
>> It's called parallel run step.
>> Parallel run step. Thank you.
We want to get that right.
Thank you so much for being with us, friend.
>> Thanks for having me.
>> Thanks you so much for
watching and we'll see you next time. Take care.
[MUSIC]