>> You're not going to want to miss this episode of the AI Show.
We talk about AML Compute,
how to save money,
how to have good enterprise features,
how to be a data scientists without being an IT pro,
all of this and more in the next episode. Make sure you tune it.
[MUSIC]
>> Hello and welcome to this episode of the AI show.
I've got a special guest.
Shank, why don't you introduce yourself?
>> Hey, Seth. I am Shank.
I work on the Azure Machine Learning Team
specifically on Azure Machine Learning Compute.
>> Fantastic. So I just want to throw this out there,
when it comes to machine learning,
it's the compute where all of the actual work happens.
>> Yes.
>> So I've heard this new term AML Compute.
Why don't you describe what it is and why it's so-called?
>> Yeah. So AML Compute is
the manage compute offering from Azure Machine Learning.
The advantage that we wanted to give this was a wrapper over
the underlying Azure infrastructure so
that you are able to spin up your compute,
are able to easily scale up clusters,
are able to just send your job,
and we take care of the rest which is managing your resources,
making sure that all the costs are taken care of,
and making sure that all of
your logs and metrics are being streamed back,
and all of that is just taken care of by AML Compute.
>> That's awesome because, basically,
there's a VM somewhere running your things?
>> Yeah.
>> I don't like doing VME kinds of things, right?
>> Yeah.
>> So when we're talking about AML compute,
it's a wrapper around the VMs, is that right?
>> Yeah.
>> So can I ask a couple questions about that?
So what kind of VMs can we wrap around?
>> So essentially, we bring you the entire power of Azure Cloud,
which means all of the CPUs and GPUs that Azure supports
is something that you can use AML Compute cluster out off.
In addition, we also allow you to be easily
able to get dedicated and low clarity clusters,
something that our customers really care about
because there are times when their jobs are not high priority,
and they are spending up low clarity cluster and saving costs.
>> So here's another question.
Because when I was hearing this, I was like,
"AML Compute represents a machine," but it could
also represent a cluster of machines, is that right?
>> Yeah. So there are a couple of ways to think about it.
If you're looking at a dev test workstation,
you use something like compute instance,
something that we recently announced with Ignite.
Then you could also be looking at
a training cluster which automatically scales
from one to say 100 nodes
and it can do distributed training for you.
All that you need to worry about is
how your training script is actually going
to use something like [inaudible]
to actually communicate between the nodes.
But then we are essentially going to go create a VM image,
we're going to go deploy the container,
and it will make sure that all of your data is inside
that container and spin off your job,
make sure that the communication is set up between those nodes,
and get the logs stream back into your Cloud storage.
>> So basically, I get to do all the fun things,
and you do all the housekeeping?
>> Yeah, that's it.
>> If I'm only interested in running
my Python training straight, right?
>> Yeah.
>> The other thing that I have a question about is,
whenever I'm talking to you about
standing up a VM and I'm in an enterprise,
there's always going to be someone that is going to be like,
"Wait a minute, you can't just start up a VM.
There's some rules."
What enterprisee-like features do we
have that helps with the IT pro side of stuff?
>> Yeah. AML Compute has been
working towards enterprise readiness,
and there are a few things that
we know that enterprises really care about.
There are things like control,
there's a thing like monitoring,
and then there's also things that are on
security that enterprises are passionate about.
Today, I wanted to just show you
some of those updates that we've made to
the product and see how it resonates with our enterprise.
>> All right. Let's take a look.
>> Yeah. So yes,
here I am on the Azure portal.
I have my "Compute" tab open.
As you can see that I can go create a new compute if I wanted to.
Let's go try that.
I'm just going to spin up a test cluster.
As I do that, it gives me
a bunch of options that I can choose from.
So in this case, I'm going to go for machine learning compute,
which is AML Compute, and you automatically
see a bunch of options that you get to choose from.
There is a CPU and GPU support that we have.
You could choose from any of these.
In addition, we also started introducing things like
VNets that you can not set up in your cluster itself.
That's something that is really interesting for enterprises.
In addition, if you wanted to start
debugging something on a cluster,
you can also set up an SSH account which you can
now parse during the provisioning.
I, as an admin, has access to this account so
I can go into the nodes and actually see what's happening.
But as the rest of the data scientist,
if I wanted to control them from having SSE access,
they don't really have access to this.
>> That's nice because, for example,
VNet support is super
important because maybe I want to bring it in
the workload onto my on-premises locally, right?
Then I want to be able to map that together,
but what are some other advantages?
Because, look, let me just be honest with you.
Whenever I create a virtual machine,
there is an IP address,
and I don't want people to get to that machine.
What are we doing to help with that?
>> Yeah. So good question now, Seth.
We have heard feedback that customers do not want
a public IP both on instance or on the cluster,
and that's something that we're going to introduce
towards the end of the year.
This was a recent announcement from
Azure Services started supporting no public IP policy,
and that's something that's going to come out.
So we're really excited to work
on that and deliver it to our customers.
>> Fantastic. Just keep going. Sorry, I interrupted you up.
>> Yeah. So in addition,
what we started doing was if you're
deploying your cluster inside of VNet,
we'll automatically close the SSH port for you.
You can still SSH into it inside the VNet, but essentially,
for an enterprise standpoint,
we are not going to keep an open port for the Internet.
Additional things that you can start
doing is around monitoring, for example.
If you go into the "Metrics" tab,
now you can see alerts that you can set up.
I'm looking at my resource, which is a workspace,
and I'm going to say the metric
that I'm interested in this Active Cores.
I can just change the duration to say the last seven days,
and you'll see how easily I can plot
this Active Cores graph across the entire workspace,
across the various clusters that were there.
So these are things that enterprises
can now start using to set alerts
on to be able to monitor exactly how much usage is happening.
Is there someone who was abusing the clusters?
Are clusters lying idly,
its capacity being paid for but not being used, things like that?
>> This is really cool because
as I've looked at it as a data scientist,
I just want to run the thing,
but as IT professionals inside the organization,
especially during enterprises, you want to manage costs.
You want to manage to make sure that the machines aren't insecure,
and now we're adding a lot of features to let people do that.
>> Absolutely.
>> That's so cool.
>> Yeah. Let me show you a couple of other things also.
So now I'm using
our new workspace which
is a Data Scientist workspace we announced Studio at Ignite.
As you go here, I am again on the same cluster,
and now I can see what are the various states
of the nodes in that cluster itself,
but one of the more interesting things you'll see here is that now
I can actually start listing down the nodes on the cluster.
This is a five-node cluster,
but say it was a 20-node cluster,
I can easily see what is the state of each node,
what is the public and private IP address
if I wanted to SSH into it.
You could do that now. You can
list the actual runs that are coming on
cluster so you know exactly how many nodes
are being used for each run, things like that.
>> That's cool.
>> You go to the "Runs" tab.
Now you have a nice view of all the runs that are
either queued or running on this particular cluster.
So you just go through these views and you
actually know these other various runs that are there,
how many number of nodes are these runs actually taking,
when were they started,
other tags that you've set up on those runs.
>> That's amazing.
>> Yeah, and this view helps not only an admin to
understand exactly how the clusters being used,
but also for data scientists to know
exactly which cluster should I use to go submit my job on.
Because I got five clusters.
But which one is being utilized, which one is not,
those are things that I need to make a decision on.
>> It might be usually because you might
have a cluster with an ANSI 6,
and some with ANSI 12,
and if you look at the resource utilization,
maybe you don't need to spin up
that huge machine because you're not even using the ANSI 6.
It's just a good way to figure it out.
>> Yes. One of the things that we are
also adding as both counters here
which means that you'll be able to see your CPU
and GPU utilization very soon,
and you can do it for a job.
You can see it for the entire cluster if you are an admin trying
to understand how much of a cluster has been utilized.
>> That's cool because I could just thinking about this.
For example, I may have set up my estimator wrong,
and it's using only CPU even though
I've stood up a bunch of huge GPU machines.
You can be like, "Hey,
we're not using the resources correctly.
You can go ahead and change that."
>> Yeah, and I'll put the icing on the cake.
We'll also be pushing all of this data
into Azure Monitor, the view that I just showed you.
>> That's cool.
>> Which means that you can now set up a load.
So as an admin, I can just rest and
assume that whenever somebody is not utilizing my GPU,
it's going to throw an alert, and I'll actually
know who's doing that.
>> That's awesome. So another question I have on
this is I've had this question
come up a lot is how do I grant access to a workspace?
I mean, I know that there's something called
role-based access security in Azure.
What does it look like here?
>> Cool. So when you go to
your workspace just like with any other Azure resource,
there's something called Identity and Access Management.
You click on this and you try to go give
access to someone on your workspace.
So in this case, I'm trying to make someone an owner.
Say, I want to make set an owner of this.
I'm not sure if I wanted to do that,
but just because we're doing this video, I'll try to do that.
So I'm trying to have Seth Juarez.
>> Let's do S-E.
>> S-E.
>> S-E-J-U-A.
>> J-U-A.
>> R-E.
>> Let's see if I'm in there.
Maybe I've been fired.
>> Yeah. I wasn't expecting that.
>> That is embarrassing.
>> Yeah, let me just search for my name.
Maybe we are both in the same boat.
So this could be our last video.
Let's see if I am able to find myself. I am not.
In my case, I have already assigned the role to myself.
What I can do here is if I go to this portal now,
you will see that I
actually do not have access to creating compute.
Somebody has actually applied a role where I don't see a plus new.
This is something that is so powerful
for enterprises because they can apply a rule.
Now going forward, they do not need to be concerned with
our customers or their data scientists
about creating a compute by mistake.
>> That's cool. Let's go back to the portal.
>> Yeah. Let's go back.
>> In the access control,
you can basically say,
"Here's a user and here's what
they can and can't do inside of the work space."
>> Exactly. Yeah. You can define custom roles.
You can say a user can do experiments,
but they cannot be doing deployments.
Those are all things that you can define
against a user in a work space,
so you can have a different identity between work spaces also.
But for a particular work space,
will now honor exactly what your role definition says and say,
prevent you from creating clusters.
>> That's awesome, because you don't want someone frivolously
spending up huge resources and then having to
be responsible for those particular costs,
which brings me to my next question.
These things can be quite expensive,
are we doing anything to help manage costs?
>> Yes. There are a few things that we're working towards.
One of the first things that we've already talked about
a little bit is that if
you are spending up an AML Compute cluster,
now I'll just switch to my notebook.
When you're spending up a cluster,
you actually are able to set in a min and a max node.
In this case, we just assume that the min node is zero.
So this auto-scaling automatically gives you cost savings.
As soon as the cluster has done its jobs,
it's automatically going to spend down.
We also allow you to
actually configure something called an idle timeout.
This is something that enterprises can use to say,
"Hey my data scientists actually do a lot of depth test."
Which means that they keep iterating on
the training script before they submit the final script.
They don't want the data scientist to pay the cost of,
"Hey the cluster went down, now it's scaling backup."
So give it that idle time mode where it will wait for
a new job for some time before it scales down.
In addition, we also have a parameter that you can use in
your training script itself which actually allows you to define.
I'm just trying to, there you go.
So you have a script run config,
which is a run configuration that gets used to define your run.
One of the parameters there is maximum duration seconds.
So you can say "Hey I want to
limit this run to not go beyond 1,800 seconds," for example.
This is something that will also be introduced as a policy soon.
So as an admin, I can say I don't want
10 day jobs being run on
my clusters because they might be in a mistake,
but I'm pretty sure my data scientists are
not doing that on purpose.
>> That's also cool because look,
as data scientists I want the thing to run forever.
But if I'm smart,
I'm also check pointing my model as I go through
each epoch and it just might be, look,
the policy is you can only go for 10 hours and
then I'll just pull the best model out
that happens after 10 hours, which is cool.
>> In fact there are customers that we've talked to who have
scenarios where they want to stop their model training in between,
look at the performance of the model, update the hyperparameters,
and then again continue with the model training
or tweak something in the training script.
That's something that you can easily do
by just having parameters such as this
which allow you to control the job instead of
having to manually go into the job,
inspect it at all times.
>> That's awesome.
>> Yeah. Some other things that I did want to
mention were also that we have support for low-priority.
So in this, you can also use a parameter called VM
priority and you can pass it as either dedicated or low-priority.
In the case of low-priority,
what it allows you to do is use Azure's discounted capacity,
so all the excess capacity that Azure has,
you can get it at a preemptible rate
which is almost at a 70 percent discount.
So it's a steel for GPUs especially.
You can pretty much use it whenever you
want if there is excess capacity.
If a dedicated workload comes into Azure,
say a big customer spins up, say 1,000 GPUs,
your jobs will get preempted.
But when there is excess capacity,
you are automatically getting a cost discount.
>> That's cool because maybe you want to run it over the weekend,
like a Wednesday at 2:00 and
then you'll be able to save a lot of money.
>> Yeah. Azure also has this functionality
of reserved instances where you
come into Azure for say using a 1,000 GPUs over three years.
When you do that, you automatically get
about 60-70 percent discount.
That's a dedicated capacity,
not even paying low-priority price experience,
but now you are trying to save a lot of money by
just dedicatedly spending on Azure and committing it.
>> This is awesome. So just to summarize,
we've talked about ML compute,
we've talked about some of the enterprise
features around spending up these kind of resources,
we talked about how we're saving costs.
Is there anything else you'd like to add?
>> Yeah, just one more thing that we tried to introduce
recently was being able to easily
share resources between different work spaces.
So one of the experiences that our admins wanted was,
"Hey I have these 20 work spaces.
I have data scientists spending up and
using clusters across these work spaces.
It's not easy for me to distribute this capacity.
Say I have a 1,000 cores. How do I fair share?
Or how do I give priority to a work space which is doing
research papers versus someone else who is
just doing some deftest experimentation?"
For something like that, we introduce this expedience where
you can essentially come into the usage in quota bit,
and you can actually start seeing
the different usages across VM family.
So in this case. I have a work space
and you can see which cluster is using how much.
One of the added advantages here is that now you
can also start setting limits on this.
So I can say, "Hey, this work space,
which is a flight delay demo,
is not supposed to go beyond a 50 core limit.
Once I do that, you will automatically be
limiting the work spaces user strict 50 cores.
This is an easy way for you to do something like capacity sharing,
governance, and cost management
and allocations across work spaces.
>> That's awesome. This is
the thing that I really like because there's
aspects of a data scientist's job that I just don't like doing.
I don't want to manage machine.
I don't want to worry how much this stuff costs.
I don't want to accidentally have a
$3,000 bill at the end of two months.
So it's really nice to be able to add this governance and
this enterprise flavor to these things.
>> Yeah. The whole point is that we want to take the IT out of
data science and leave it with the machine-learning compute.
What you focus on is actually
deftest experimentation and building your models.
>> Well, this has been awesome.
Thanks so much for spending some time with us.
>> Yeah. Thanks so much Seth, and we'll try
to find you back on sometimes.
>> Fantastic. We'll put some links
below on where you can find out more.
But again thanks for being with us. Thanks very much for watching.
We're learning all about AML Compute and all of
the amazing features that you
have now with Enterprise capabilities.
Thanks for watching. We'll see you next time. Take care.
>> Thanks for watching. See you.
[MUSIC]