>> Good news, in this video,
we're going to get a chance to figure out what happened after
the last cliffhanger where we were talking about
reliability and I realized
that I wasn't going to get the reliability I needed.
I'm here with Sergio and Niels who are going to
help me with this. Please tune in.
[MUSIC]
>> Welcome to the Azure enablement
show where we will be discussing
the challenges you and
our other text-heavy customers have encountered.
Today we'll be talking with the experts to
find out how they think about these problems,
recommended tools and best practices and tips
they have learned from years of experience that you can use.
Today we're going to be discussing and diving
deeper into Azure workload reliability.
In order to do that, I want to go back
to where I was in the previous video,
where I was in the middle of being a customer that
just learned something a little bit unsettling.
Sergio, Niels last time we talked,
you were basically giving me the not-so-great news
that I had not architected in the way that I needed to,
to get the level of reliability that I was hoping for.
I'm not so happy about that.
What do I do now?
I guess is the big question I have for you.
>> That's right.
We had a look at the architecture last time
and we came to the conclusion that you ended
up with a 99.94 percent
availability where we were aiming for 99.99.
There are some things you can improve to
increase this availability, this composite SLA.
Why don't we go over a few
of the technical things. How does that sound?
>> Yes, please.
>> One thing we want to do to increase
the availability is to add some redundancy to your solution.
In this case, you have one Kubernetes cluster.
We would recommend in this case to have multiple,
because in that case,
it sounds obvious if one of them breaks,
the other one might be working.
To get them as far apart from each other as possible,
you could put them in multiple Azure regions.
For example, you could have one in East US,
one in South US,
one in West Europe, one in North Europe, for example.
The thing here is that they would have to be identical.
That means that the deployments of these clusters themselves,
as well as the containers that are on top of
them would needs to be deployed in an automated way.
You would need to have your infrastructure as code in order,
have all your codes in
repositories, build and release automatically to make
sure that the two clusters in different regions are in fact
identical because a request that comes
in might end up in either one of the regions.
One other thing to consider if you have
multiple regions is that you also
need to have your data layer available in both regions.
Because if the request can either
end up in region A or in region B,
the data that is available to
serve those requests must also be so.
Luckily, you're using CosmosDB,
which has a really nice capability where
you have multi-region writes, so multi-master database.
That means that any change you make to it in
one region is automatically synchronized to the other region.
By enabling this capability,
you also increase the availability of
CosmosDB itself so that it will
really help you achieve the numbers you're looking for.
>> It seems like,
if I'm getting this right,
I need to lean heavily into regions and I need to lean
heavily into fall tolerance the redundancy.
But does that actually get me where I want to be? Is that enough?
>> Well, we can run the calculation in a minute.
One other recommendation we have is since you're using
AKS for this solution to enable availability zones,
what that does is makes sure that's
the cluster nodes you have are spread
across different physical facilities
in the Azure region so that if there would be
a facility wide issue going on there then the chance
that's your cluster is impacted is going to be
less because your workload is running in either of the facilities.
>> That's really interesting, but Sergio,
can I take this conversation in another direction?
One of the things that was a little disturbing to me about
our outages is how we code and and how we dealt with it.
I'm wondering if you have any advice around that.
>> Absolutely. Let's take a look at the Azure portal itself
and work through an example of what
this looks like when we make it real.
Very briefly what I want to do is
take a quick look at where you are now,
where are we starting from,
and then how is that going to look different?
In today's example,
you essentially have your website running in a single region.
>> Right.
>> We're going to go ahead and use
Azure App Service as a standard for
AKS just so we can remove some of
the complexities of all the containers
that are running in your solution.
But the observability and
our reactions to failures is going to be exactly the same.
In today's world, you have your website running in one region.
In this case, we place this website in East US.
Now, in the event that something goes wrong,
we're simply going to stop this website.
Well, this is exactly what today's picture is for you
and today if you had a problem in AKS,
in this place we're simulating that,
this is the experience that your customers come to expect.
You don't have any redundancy
when there's a problem, you're just down.
So by moving it to multiple regions,
one of the things that we can do is we can say,
"Instead of running in just East US,
let's run the WebApp in East US and West US".
What we talked about in the calculation is in
order to make this seamless for the customer,
we need an additional component that you don't have today.
We suggested in this case,
the Front Door might fit the bill.
Front Door is a global load balancer
that is able to provide your customers
a single website address like they have
today and when they go to that address,
Front Door is going to make a decision about which of
those multiple website instances
that we have now makes sense to go to.
So In the very first panel,
we now have a single websites that we can visit.
In this next panel here, very briefly,
I want to show you where
those two different websites are showing up.
We have the concept of a backend pool,
and essentially this is all of the different regions.
In this case, we only did two regions,
just like in the calculation,
but here they are.
With these two websites now behind this one address,
if we now go to your company website,
then we find ourselves exactly where we started,
we're in East US.
However, now let's go back to the example
that we created for your single region deployment.
Let's go into one of your regions.
In this case, let's go into
East US and let's see what happens when we just stop it.
Obviously, we're forcing a worst-case scenario.
When we go to stop this,
the expectation is we should be in a better place.
We're now stopping the websites,
let's go back to where we were.
We reload the websites.
>> Oops, we sent out an error message.
Well, that's because even with something
like Front Door in front of two websites,
there's still a question of how long does it
take to detect that there is a problem?
How does it even detect if there's a problem?
Well, there are health probes that are hitting
the backend websites all of the time.
They're trying to determine, are you okay?
Are you okay? If you are okay,
then I'll send customers there.
Well, it takes a certain amount of time
to determine that a website has gone away.
If we now reload,
we now find that enough time has passed that it
realized that East US website was down in a catastrophic way.
We just stopped it. Right now,
West US is now available and so
your customer is now able to continue their experience
on your website because of now
both redundancy of your websites as well as Front Door,
that makes that transparent to your customers.
>> What do I do to know when something has gone wrong?
What do I do about it when something goes wrong?
>> Sure. At this point,
you're going to have several options.
All of the individual services have
their own login and their own metrics
that you can leverage to make those decisions.
Why don't we go ahead and just take a look at Front Door itself?
>> Okay.
>> Front Door itself keeps track of all of
those times when nodes were healthy and unhealthy.
Let's just take a graphical look at it.
All of this is available to be queried
and you can write complex queries around it.
But just graphically, we can see that we've
received some requests, no surprise there.
We can even break it down by backends.
We're seeing some requests are going to one region,
and this different color
indicates we're going to a different region.
But now we can also look at backend hell.
This is one visualization,
but what this should represent to you is that you
now have data that can be queried
and you can create alerts around this information to be able to
know and either be notified or even take
action when one of these regions drops out.
If you see this orange region,
this was region A or East US.
When we stopped it,
that's what it looked like.
Now you have visibility into when these outages are occurring.
>> Presumably, I shouldn't just stare at graphs all day long.
To find out when something's wrong,
having people with eyeballs on
graphs doesn't seem like a good use of people's time.
What can I do instead?
>> Absolutely. One of the things that you can do is
you can leverage built-in alerts for a given service.
Now again, we're sitting inside of Front Door,
which might just be one portion of your application.
But in here, what we can do is we can create a new alert,
and base it on any condition that we'd like.
In these conditions, I talked about the fact that we
have lots of different metrics,
lots of different activities to choose from.
We looked at backend health percentage as an example.
This is the same thing that we can
leverage now to build out reaction.
Here's that same graph to confirm what we're looking at.
There is some sophistication you can build into this.
If you're wanting to consider a window of
time versus just outage happened logic,
you can use this alert logic to implement that.
Once you've done that,
and you've specified a condition that
makes sense for what you're trying to track,
you have several options for actions.
For actions, you can do the more traditional alerting as it were.
I need to plug a value in here.
You can just alert somebody,
alert that human as you mentioned,
or you can take advantage of several other types of actions.
Action groups can include notifications,
but they can also include things
like executing a runbook or an Azure function.
If you have had enough experience with this solution to
know how to resolve a particular problem,
you can automatically resolve them with
actions that are specified in this action group and not
just have to send the page or send
the text to the technician who would have to go fix it themselves.
>> I'm a big fan of letting computers do what computers do,
and people do what we need people to do.
I think that that's really great.
I have one last question to ask you that's been nagging
me this whole time as you've been talking about this.
I really appreciated our chance to look
over value mode analysis and be able to figure
out what could fail and how
often and come up with composite SLAs and stuff like that.
But I just have this sense like
that's a really good way, and an important,
and the first thing we have to do is
figure out that which we know could go wrong,
and plan for it.
But I'm also, based on my recent experience with this outage,
really quite aware that there's stuff
that I know that I don't know.
What do I do about the things I don't know about,
the failure modes that I couldn't have anticipated?
>> You're absolutely right to identify that
every exercise that we've done so far
has relied on you anticipating a potential future.
But the reality is, as you sense,
this is really just the beginning of a long-term learning process.
There are several ways that we can learn as far
as continuing to evaluate the reliability of our solution.
One of them, and probably the least effective,
but unfortunately sometimes most common,
is that we can simply wait around.
We're going to wait for something else to happen,
another outage, another unexpected failure.
Not a great way to do it,
but it is something that will always be with you.
There will always be the unexpected,
but we can do better than that.
One of the things that you saw me demonstrate in
showing you the value of a multi-region deployment,
is we can simulate failures,
we can force failures.
I also encourage you to look at
the components that you are using and figure out,
like in the case of AKS,
can we just stop containers and see what happens?
See if our plans are meeting reality.
Are we detecting it as we planned we would?
Are we being able to react to it automatically
if that was something that we planned for?
Forcing failures is a great way to double-check the design work.
It doesn't have to stop there though.
One of the things you can also do
is take a more experimental model.
That is, something that looks a lot closer to
the scientific method where we form an idea,
a hypothesis, and then we just try it out.
Oftentimes, we try out
that hypothesis in our production environment.
Now before we go into anything like
that, from an experimentation perspective,
what becomes super critical is that we have put in place
the ability to observe our solution as fully impossible.
Because otherwise, if we're just experimenting in our environment,
we won't have any way to know what is happening,
and that's just causing chaos.
Chaos is a funny word because there is
actually a branch of engineering called chaos engineering,
where it's the idea that once we properly instrument our solution,
we can introduce real-world scenarios just at random,
just chaotically, and see how our solution organically response,
see how are processes that meet our solution responses,
see how those line up,
and so we can design experiments and go through and do that.
We also have things like game days as another type of exercise,
where again, we set up a scenario.
Oftentimes, we have teams in those scenarios
where maybe in a retail setting like yours,
we might have one team generating unpredictable load
or exercising your website in unusual ways.
Perhaps it's scale, perhaps automation is part of that as well.
Then another team that is essentially responding to that.
Are we seen the monitoring alerts come through?
Are we seeing the automatic actions?
We can really design these experiments
to basically validate that all the design work, and everything
we've tried to do is basically resulting
in meeting the targets that
we had specified at the beginning of this exercise.
>> That makes total sense to me.
I could talk about this stuff with
you-all day, this reliability stuff,
because this is really interesting,
and I really appreciate you taking the time to look over my stuff.
You and Niels have been super helpful,
I want to thank you for that.
I really appreciate. I think probably I have
to go back to host mode.
Well, that was really cool.
I'm glad to have the chance to talk with
Niels and Sergio about reliability.
But we have to end here. I hope you've enjoyed this.
I hope it's been useful to you.
Thank you for joining us and I hope
you'll watch some of the other videos in this series.
Take care and we'll talk to you a little bit later on.
[MUSIC]