>> HDInsight introduced a new feature called autoscaling.
It enables you to make the best of your budget,
and Alicia is here to tell us all about it today on Data Exposed.
[MUSIC]
Hi, and welcome to another episode of Data Exposed.
I'm excited to be here today with Alicia,
who is joining me from the HDInsight team.
So welcome to the show.
>> Thank you, Jeroen. Thank you everyone. I'm Alicia Li.
I'm the Product Manager from Azure HDInsight Product Group.
I've been working on the platform area.
So my goal is to make the platform most reliable and cost
efficient for our customer to run open source big data analytics.
>> Cool. So I'm excited to have you here.
It sounds like you have been working with
HDInsight for a little bit, right?
>> Yeah, correct.
>> So what is the main feedback that you've
been getting from our customers?
>> Yeah. Well, the feedback that we got from customer is
that they found it a bit challenging to control the cost.
We know for the big data analytics the per unit cost is low.
But when the customer is running a lot,
a lot I mean some customer they may process petabyte of data,
are running 1000 [inaudible] of hours,
you may end up get big bill by the end of the month.
>> Right, and it's what you would expect.
It's big data for a reason, right?
>> Yeah.
>> So it's going to be about processing a lot of data.
>> Yes, you are right. So how to control cost that would be, well,
the major requirement of our big data customers.
So today, well, let's consider that the world of static clusters.
So we might say the static cluster,
I mean the size of the cluster is fixed.
So why do customers need this long-running static clusters?
We have observed that our customers are
running this large long-running clusters.
When I say long-running cluster,
I mean this cluster running 24 by
7 every single day and the cluster size is fixed.
So why do customer need to do that?
So through the engagement with the customer,
we found that they have
very complex configuration for the clusters,
like [inaudible] , like they ruined your security policies.
Also, they will have a lot of
customer applications to install in the clusters.
So all of this makes it very complicated
for a customer to recreate a brand new cluster.
>> Right, it will take lot of time
to destroy and then build it all up.
>> You are right.
>> So that's why they built this long-running big static cluster.
>> Yeah. You think about that so the customer,
it may take them a lot to time to build this.
So it's more hard for them
to recreate the cluster several times a day.
So they have to keep this long-running.
You think about, when they're provision this cluster they
have to make sure they can meet their job performance requirement.
But sometime they have to end up provisioning for the peak usage.
But their load actually coming to this cluster is a variable.
So maybe more load during the working hour,
or less load or even no load in the non-working hour.
So this all it has very low utilization for the customer cluster,
which means they are the wasted there.
>> Right. But I get this.
If it's hard to built this cluster,
you want to do it once and then you'll want do
it good so you'll design useful peak.
>> Yes.
>> I get it. It doesn't sound very Cloud-like though.
>> You're right. That Cloud is designed to be elastic.
>> Right.
>> Right. So what can we do to help our customers?
>> You of course you have an answer for that.
>> Yeah, we've heard the pain and here is what we are doing.
So we introduced autoscale capability.
So where customers can let the service work for you.
So all the customer need to do is to set the configurations.
So for example, we can set
the minimum and the maximum number or no the file cluster,
or you can set the schedule at when you want
to scale up and scale down over your cluster.
After that, your job is done.
>> Awesome. So basically I can now just
tell whenever you need more resources,
take more, whenever you don't need them anymore,
just take less resources,
that what you're trying to do here?
>> You're right. The remaining,
please leaves to us, the service will take her that.
So service will monitor the CPU and
the memory usage to see
how much available resource right now I have,
and then I can pair with
the pending task and resource requirement.
If my available is less than the requirement,
then I will add more nodes,
which means scale up,
and I have extra resources then I will scale down.
One more important thing here or
the complexity that we are trying to hide from
our customer is that how to ensure
the job success rate during the cluster scaling down.
>> Right. Because, yeah, there's something happening.
The system is busy scaling
and you're still firing job. So how do you handle that?
>> Yeah. So during our scale down.
We will do a graceful scale down.
So we'll shift the data to that node that we decide to keep,
and we would like to node to complete the current task.
Then during the job queue and smoothly remove the node silently,
so that the job will continue to run until succeed.
>> So with this, basically, as a customer,
I get the best of both worlds, right?
I get the Cloud, I get
the big data scaling that I need when I need it.
>> You're correct.
>> But I don't overpay for just
scaling at the peak all the time as you
explained we had to do before?
>> Yes, in this way you can see the high utilization on
your cluster and you pay only for what you really need.
So therefore, there's no waste anymore.
>> So yeah.
>> So our customer have
been using this feature for several months.
So our customers told us they've saved up to 62 percent.
>> Wow.
>> We are very excited about this result.
>> Sixty-two percent saving,
that's amazing for just an innovation like this, that's awesome.
So can you show me how I use this?
>> Sure. Let me show you. Okay. So first let me show you how
to create HDInsight cluster with this autoscale capability.
You go to Azure portal and then you'll create the resource,
then let's find HDInsight.
>> Yeah.
>> Okay. Now, I want to create a new cluster.
So let's say I want to create this cluster
to do the business analytics.
When I say business analytics,
that I want to analyze the usage of my customer, my business,
the bills of my customer all of this,
and the same as the nominal step that that you will
select sufficient resource group and now let's give a name.
Let's say, BusinessAnalyticsAutoscale.
>> Let's see if that name has been taken yet
>> Is spinning.
>> I always have this problem that when I do this,
that the name is already taken.
>> It means you choose a popular name?
>> Yeah, probably.
>> I mean, now I got this name.
>> There we go, perfect.
>> I choose this spark cluster and then we give it
a password and then let's directly go to the configuration page.
Here, you'll see, this is a button,
you can say "Enable autoscale".
Here you will see two mode.
What is load based?
Load based you need to specify
the minimum that even when there is no load,
how many nodes we want to keep there.
You can have a maximum.
We have the peak node, how big
you want your cluster will scale up to.
>> Right. So the minimum node is kind of
keep the lights on idea, right?
>> Yes.
>> If there's nothing happening,
at least this amount will be reserved for us so we have something.
>> Yeah, just in case something come in you can work it directly.
>> Okay.
>> Here it will give the estimation with the price.
So you can see that with the minimum,
you only pay $2.99 per hour and the maximum you pay $23.
So think about without this feature customer
may have to learn how pay this much all the time.
>> Right.
>> That's a lot.
>> Yeah.
>> There's another mode I do want to touch
on that one is a schedule based.
So this one works better,
for example, if you have data scientists team.
You know they will come at 9:00
AM they start working and 6:00 PM they go back home.
So for this scenario,
you want to maybe set your cluster like this.
I want to select Monday to Friday,
let me give them 30 node and let's say
at 6:00 PM let me give them six node,
or even for the weekend,
yeah we can add another condition to the rate for the weekend.
>> You could say maybe three notes or whatever.
>> Okay. Yeah, you're right.
>> It might be a data scientist working in the weekend who knows.
>> It could be.
>> Right. Could be.
>> So leave three for them.
>> Yeah.
>> Then after this then you can
review the configuration and create your cluster as normal.
>> So this is very cool and I like the way it's very
clear to the customer where the setting is
and what the impact on cost will be.
>> Yeah, I only take them several second to set up,
then anything else it taken care of.
>> Wow, awesome. So that's how I start using this.
Now, let's imagine I have it, I'm running this.
How can I monitor what's happening in the cluster?
I mean, I want to be sure that
the settings that I have are correct, right?
>> Yeah. So I already have two cluster running.
So let me show you how I monitor it.
So let me start with this one.
So this is our HDInsight cluster overview page
and they use scroll down,
here there's "Monitoring" section,
you select "Metrics" and
here you may want to see the active work node.
So by default, the past 24 hours let me go to,
let's say the past three days.
>> Okay. Yeah.
>> So you can see when I have more load at peak time and to
go to 10 node and when there's no load it goes to three.
>> Yeah.
>> Okay.
>> Cool.
>> So these validating your assumptions.
Yeah interestingly it actually worked for me.
So let me quickly show you another one.
This is schedule based.
The same thing, you'll go to "Monitoring",
go to the "Metrics" base.
This is the something that we just discuss.
I want to set
time former data scientist team and see how it works.
So you can see it form pattern,
and this definitely is the weekend.
>> Yeah. It's a pretty pattern.
I see exactly the settings
that I make they show up in the diagram.
>> You're right. This is very easy to validate.
If after you evaluate this, you think,
well, maybe I can still save more,
I want to make some changes.
You can always go to here
the cluster size and here you can switch between
this mode and you can change
the configuration and you can enable or disable autoscale.
>> Right. So if for whatever reason, let's say,
I know there's a big project coming up like the next four days,
you could maybe even go back to
not autoscale you just peak for a little while,
and then I can switch back to autoscale whenever I want to?
>> Yeah, and also for this situation
you can change your configuration to
provision for the peak and for
the 24 hours and then after the load is gone,
the event is over, you can switch back.
You have the full flexibility.
>> That is very cool. This is so cool.
>> Thank you.
>> So yeah, I mean, I'm sure what customers will love this.
So is there any, I now relatively know how
it works and what it does and how I start using it,
there is any best practices you can share
for customers that want to get started?
>> Sure, I think one of the practice I want to share
that you may want to go back and forth to check,
to validate as I've just shown to validate actually
your cluster size match with your requirement.
So if you still see there's a waste,
you can always adjust it.
>> Okay.
>> So further tune your settings.
So that's the first piece of advice I want to give.
The second one that I want to give is that, for example,
if you are using this schedule based autoscale,
that if you have the pipeline coming on the fixed time,
let's say you know you already
scheduled pipeline at 9:00 AM coming,
you may want to give a buffer.
Because when the system trigger this scale up,
scale down it has some latency there,
so you my want to specify your load, sorry,
you may want to scheduled the number of your cluster to 8:30,
so that by 9:00 AM your load is
coming the cluster is ready to process.
>> So you're so you're saying them whenever you
expect a load to come in 30 minutes before that,
make sure you're ready start scaling?
>> Yeah, that's give some buffer,
maybe not 30 minutes, but at least give 10 to 20 minutes buffer.
>> Okay. Cool. Those are really good tips.
Thank you for that, and
thanks a lot for being here and thanks for sharing,
I find this very exciting.
>> Welcome.
>> I think you find it very exciting too.
So thanks for watching.
Please like, subscribe, and comment,
and I hope to see you next time on Data Exposed.
[MUSIC]