>> You're not going to miss this episode of
the AI Show where we talk all about
pre-built Docker images for inference
in Azure Machine Learning with the amazing Shivani,
make sure you tune in.
[MUSIC]
>> Hello, and welcome to this episode of
the AI Show where we're talking all about
pre-built Docker images for inference in Azure Machine Learning.
Welcome Shivani, why don't
you tell us who you are and what you do?
>> Sure. Thanks, Seth for having me.
Hi, everyone. My name is Shivani.
I am PM on the Azure Machine Learning Team
and I work on the model deployment and serving area.
>> Fantastic. What are some of the challenges that people
face or what they tell you about when
they're working with machine learning?
>> Sure, absolutely.
We have interacted with many of
our customers and data scientists,
and what we know once you've trained a machine learning model,
getting that model to production is key,
especially having the right environments.
Environments encapsulates all your Python packages
and software setting on obvious scoring script,
and hence having the right dependency when you
deploy your machine learning model is very important.
>> Let's talk about the solution.
What is it that you've done to make this pernicious?
Because obviously, building a model and
handcrafting it and making it wonderful is nice,
but we need to actually have business value.
What's the solution that you're suggesting we should use?
>> As I mentioned, having the right environment
for a model deployment is very important.
During Microsoft Build 2021 we've
released pre-built Docker images for inferencing.
As the name suggests, these are pre-built Docker images
that come with popular machine learning framework
and Python packages.
Our customers, users, data scientists don't have to
start from scratch and it will
improve their getting started experience.
>> Let's go. If I'm understanding this right,
these pre-built Docker images have a set of
things already preinstalled. Am I getting it right?
>> Yeah, you're [inaudible] . We have
pre-built Docker images like TensorFlow,
PyTorch, X- Runtime,
and we're continuing to adding more.
>> What if I have
a special unicorn environment that
only I have and obviously I love PyTorch,
I'm using PyTorch, but then I'm using
something like spaCy because it is an NLP thing.
How do I make that work with those pre-built containers?
>> Definitely, we all know in
machine learning world models have a lot of
dependency and we definitely cannot
capture all of them in our pre-built Docker images.
But what we do and
what capability we provide to our customers is to
extend on top of our pre-built Docker images.
That is, if you have
dependencies that are not included in our images,
we provide you with a capability to add these Python packages.
We can do that in two ways.
Number 1 is the dynamic installation.
In this method, you would provide
us with the requirements.txt file.
This txt file would contain
the extra Python packages that are required for your use case.
During container boot time,
we need this Python packages in the
requirements.txt file and install them.
Now, this is very handy for rapid prototyping and death best Loop.
Now let's see, you have finalized the dependencies that
your model requires and you want to take this model to production.
The second method we provide is pre-installed mounting solution.
In this, you provide us with a folder which has
all your Python packages installed and
we mount to this filter onto the container.
Both this method do not trigger that image
built so it also improves a modern deployment latency.
>> That's really cool. For those
that are watching, they are wondering,
building a container takes a long time,
and this notion of being able to have the dependency is just
mounted directly, super-fast.
That's pretty amazing. I feel like I want to take
a look at how it works. Can you show us how it works?
>> Definitely. Now,
can you-all see my notebook that I have up here?
>> We got it.
>> In this particular example,
I'm going to deploy
a PyTorch model using our Azure Machine Learning Python SDK.
This particular model is trained to
classify chickens and turkeys. Very interesting.
Now first, if you can take a look at my folder structure,
I have my requirements.txt file in
the same folder as my scoring script. This is important.
Next, I'm going to register my PyTorch model that I have here,
and next step is to write my requirements.txt file.
Now, just to note,
even when you're prototyping,
we always recommend our customers to pin their package washers.
This make sure that you're scoring script does not
break and it does not cause
any failure during deployment or scaling.
Now once you have written your requirements.txt file,
I'm going to go ahead and define my environment.
As mentioned previously, this contains
all the Python packages and your software dependency,
you can deploy it for deployment.
You see my environment name is PyTorch deploy pre-built example.
I'm using one of our PyTorch pre-built Docker images here,
and using the environment variables
concept that we have in Azure Machine Learning,
I'm referring to my requirements.txt file.
Once you create this environment,
we have a amazing environment
CY in our Azure Machine Learning Studio,
and you can look at the custom environment list,
and you can see the one that I've created right now.
We also have curated environments.
Right now I'm filtering them by influence equal to true doc.
We can also take a look at all the environments that we provide.
Now heading back to our notebook.
Once you have created your environment,
you're going to go and create a scoring script.
Once you have your scoring script ready,
which I have right here,
what you're going to do is you're going to
deploy your model to an ACI container.
You can also deploy to Kubernetes,
managed endpoints in Azure machine learning.
But for right now,
I'm deploying to an ACI.
Once this deployment is completed,
it creates an endpoint for you.
I already had deployed this model
and I have an up and running endpoint.
As you can see, my status is healthy and my Container Instance,
and this is the model that I deployed.
Now next, you have a web service up and running.
Let's see if it actually works.
I'm going to pass this image
to our endpoint and see what it replies.
The results that we're getting,
that it's a turkey with probability of 0.93.
Not bad. As you can see,
I have my environment using dynamic installation methods.
>> This is really cool. Let's scroll back a little bit.
Let's go to the environments tab,
if you will, the second tab that you were looking at.
Basically what we're doing,
and this is the Kubernetes look,
I struggle with this all the time.
I have the environment and I have this.
It works on my machine problem with my dependencies.
These environments basically encapsulate
them as a noun inside of Azure Machine Learning, is that right?
>> Yes, exactly.
We have these beautiful frameworks, TensorFlow,
ByDraw, scikit-learn, Exebaz,
the ones that are very popular,
and we have curated them and we have them in EML.
>> Fantastic. Then the second thing and this was
super subtle and it went by really fast,
and I want to be able to understand this is,
you basically married that environment
with a scoring file and then what was it that produced?
>> Once you have your scoring file what we did is,
with our dynamic installation method,
you have this environment on top of that.
All the requirements.txt Python packages that I have mentioned,
it gets installed on top of your pre-built Docker image.
>> I see. Then once you do that,
what is this endpoint your tie?
Is it like a rest endpoint than people can call?
>> Yeah, exactly.
You have this resting point here which you
can go and take a look at our endpoint CY,
and then I have this ''Nice click to copy'' button,
and I can use that for all my inferencing requests.
>> It's basically you take in a model,
add an environment around it,
like the scoring script and we'll
take a look at a scoring script in a little bit,
and then push it out into
a rest endpoint that now is actually consumable by the business?
>> Yes, exactly.
>> That's fantastic. Can we take a look at
the scoring script for a second,
just so people can get a sense for what it is that that looks like?
Let me go full screen here.
It's just basically loading up
the model and then running the model?
>> That's it, yes. You're inner turn drawn function is
basically the logic that you would
require to run up your scoring scripts.
So you load your model and you get your predictions.
>> Well, this is amazing.
Where can people go to find out more, my friend?
>> Definitely you can see the link
for our pre-built Docker images.
We have documents that talk all about
the Docker images we provide and as I say,
we are continuing to add more,
and we also provide documents on how customers can
use our extensibility solutions that I mentioned earlier.
>> Let me put another link up here.
There's the extensibility solutions that you were talking about.
Then finally, I think there's
a blog post that people can take a look at.
Do you want to give us a preview on what that is?
>> Yes, exactly. The blog post talk about
improved environments experience both
for training and inferencing.
We have environment CY,
we have pre-built Docker images,
we have curated environments all to help
our customers to improve their machine learning models.
>> Well, this has been amazing.
Shivani, thank you so much.
We've been learning all about pre-built Docker images
for inference in Azure Machine Learning.
Thank you so much for watching and
hopefully we'll see you next time. Take care.
[MUSIC]