>> Hey. Good news. We're here with
the second episode about Chaos Engineering.
At this time, we're actually going to get a chance to
see how this stuff really works.
Join us here on the Azure Enablement Show.
[MUSIC]
Welcome to the Azure Enablement Show where we'll be discussing
the challenges you and
our other tech-savvy customers have encountered.
Together, we'll be talking with experts to
find out how they think about these problems,
recommended tools and best practices,
and tips they've learned from years of
experience that you can use.
To tell you, we're going to pick up where we left off
on the other Chaos Engineering video,
and we're going to try to see how this gets done.
To see this, it would be better for me to see it as a customer,
so let me switch to customer mode.
Cool. Well, here I am back as a customer.
What I'd like to do is continue the conversation I was
having with John about
Chaos Engineering and a chance to get all the background here.
But now I want to see what it looks like we're doing the stuff.
To that effect, the good news is that I'm here with Shrunga,
who I think can give me a demo. Hey, Shrunga.
>> Hey, David.
>> John was kind enough to give
me what I need to know about how to get started,
and how to think about Chaos Engineering.
I'd love to talk to you about the nitty gritty,
about what I need to show,
and maybe if it's possible to see a demo of this stuff in action.
The question that I wanted to ask just as
the the question before we get into the demo land is,
clearly we do a bunch of experiments.
How do I know what the first experiment
I should be doing is when I want to get into Chaos Engineering?
>> The major recommendation that we
normally give to customers that are just
starting out is look at
your past life site incidents and
your outage scenarios for your service.
The two major areas to look there are:
the first one is like
your high impact but low frequency outage scenarios.
Incidents that happen pretty rarely,
but when they do happen,
they blow up a lot of things for your service.
The second area is
the low impact but high frequency incident scenarios.
They occur pretty frequently and are a drain on your service and
your team but they're not
widely impacting and your customers are
not heavily feeling these incidents.
Those are the two major areas and scenarios that we
suggest that you replicate using Chaos Studio.
Once you have those scenarios identified,
you can go about constructing your experiment
based off to be able to simulate those particular scenarios.
>> That makes a lot of sense.
I think I may already have in mind some of the
some of the tiny outages that we've had recently,
which I think would make superb experiments to mess with.
But now I'd like to know how to do this.
Is there any way I can see this,
this Chaos Studio you keep on mentioning, can you show it to me?
>> Yes. Let's go do the demo.
Basically what I'm looking at right now is the Azure Portal,
and this is the sample application
that I created for the purposes of the demo.
It's really simple.
There's a public IP address,
and three VMs that are serving for this application.
Demo 0, Demo 1, and Demo 2,
all behind a load balancer that is directing
the traffic evenly to all three VMs.
Another thing to note,
I think about this application and
the way it's set up for the purposes of this demo,
is that this application is supposedly,
and this is the hypothesis resilient to
availability zone down scenarios.
If a particular availability zone down or a zone goes down,
each VM is in a different availability zone.
This one's in Zone 1,
this is in Zone 2, Zone 3.
Theoretically, if the Zone 1 goes
down or Zone 2 goes down or both Zone 1 and 2,
or any combination of two zones goes down,
the applications just still have 100 percent or
close to 100 percent availability for
all the traffic that's coming through the the front end IP.
Where I also have setup is observability for that same hypothesis.
Like my colleague John mentioned,
it's always good to have your observability for your application
set up before you perform experiments so that you
can see what it's doing to your system,
and draw conclusions based off
of that regarding the resiliency of your system.
We can see the three VMs that are in the 50,000 range here,
and all of them have 100 percent availability.
Data path is looking good.
This is split and what we're looking at here is the aggregate of
basically this overall data path availability
for this load balancer and for the application.
That's the setup of what we're going to be doing Chaos on.
Since that's the application,
I'm back at the Azure Portal and
I already have Chaos Studio booked up here,
but you can also look for it in the taskbar
and you'll end up on the landing page.
The first thing to think about when
you're looking at Azure Chaos Studio,
and before you start causing your chaos creating experiments,
is we need to onboard resources as targets.
What I mean by that is we need to make sure that we are onboarding
any specific resource that you want to target with
fault injection as something called a target.
After that, we're able to inject fault into it.
How to do that?
Let's start on the left side here.
I have a filtered down list.
This is like all of the targets over here,
but what we care about for the purposes of this demo,
are these three VMs.
These three VMs are my target.
These are the ones that I want to inject fault.
I want to shut down the VMs for these three,
so I'm going to go ahead and enable these VM targets.
For purposes of demo,
I've already done so,
but to do it,
it's as simple as checking the box
on any particular target resource.
You can also do this in bulk and click Enable Targets,
so you'll be able to enable targets here.
Since it's already done,
it's grayed out right now.
After that's done, you can then go ahead
into actually creating your experiment.
The place to do this is via "Experiments" tab.
You can click here.
There's a few that you'll see that are already created here,
but what we're interested in is
this top bar right there which says "Create".
Once you have your targets onboarded,
you go ahead into the experiment creation wizard.
This is where you can pick your faults and target your targets.
I'm going to give it a simple name, experiment designer.
Here we get into what actually constitutes an experiment.
An experiment is composed of steps and branches.
Steps are executed in
sequence and branches are executed in parallel.
Basically if you have multiple steps,
they will occur in the order of Step 1,
Step 2, Step 3,
and if you have multiple branches within a step,
all of those branches will be occurring
simultaneously as this particular step is running.
What you can do in this view is basically give it a name.
The type of experiment that I'm trying
to create in this case is to be able
to successively shut down the VMs in each of availability zone.
For example, I could go about doing that.
I'll rename the branches with a couple
of pre-filled options there, but from history.
But basically you can name it so that
you it makes more sense to you
and then add either a fault or a delay.
In this case, I'm going to go ahead and add a fault.
You can go and see from our extensive fault library,
and this is what you can use to simulate any outage that you have.
I'm going to go ahead and pick a VM shut down
because that's what I'm trying to replicate.
If an availability zone goes down,
my VM is in effect shut down.
I'm going to go ahead and say "Abrupt shutdown".
Next is you pick your target resources.
These are the resources that you onboarded in the previous step.
For this very first step,
I'm just going to say,
shut down demo zero,
which is the VM in the first availability zones.
I'm going to go ahead and click Add.
For brevity sake, I have already gone through
and created a full experiment right over here.
Let's take a look at the fully created experiment.
This is the full version of the experiment.
Let me give you a brief rundown of
what I expect to happen in each of these stages.
Step one is we're targeting the VM in availability zone one.
What we're doing is for 10 minutes,
we're going to be shutting that VM down.
This is like basically out of the three VMs,
we're shutting one down.
The expectation is the other two will still stay up
and data path availability will still remain high.
In Step two, so the next 10 minutes we're going to take down
both VM like the VM in
availability zone one as well as the VM in availability zone two.
Both of these are true for the second 10-minute duration.
What this is replicating is both availability zone
one and availability zone two have gone down.
All that's remaining is one VM and we're depending
on the load balancer to redirect
all the traffic to VM number three.
The last thing, the last step here is basically we're taking
down all three VMs in all three separate availability zones.
What this last step in effect is doing is
simulating what would happen in the event of a full region outage.
To be able to say at this point,
none of the availability zones are available,
the VMs are all shut down just to
be able to see what your system does in that particular scenario.
What I expect in each of these steps is up
until we don't do the full regional shut down,
which is up until we don't get to step three,
that my availability still stays high.
We can go ahead and once you have that experiment created,
you can say Start,
is going to give you a little warning that reminds
you that this is not simulating a shutdown.
This will actually shut your computer down and to be
able to make the decision to use this wisely.
I'm going to say, Okay,
and it'll start showing you each of the steps pre-processing,
waiting to start and running as well as
the success or failure of the demonstration.
Let me skip ahead just a little bit because this is as you saw,
a 30-minute long experiment and
show you the results of something that I run an hour ago.
Basically similar to what
I was describing as the expectation before,
basically our data path availability
before the experiment started,
which the experiment started around
this mile marker is all three of our VMs were at
a 100 percent availability and
so the traffic was evenly distributed across the three.
What you're seeing here is one of them being shut down.
This is the execution of step number one,
which is we are taking down
one availability zone in this case, one VM.
Going forward, you are also able to observe
steps two and three being executed.
This is number three where all of our compute is shut down,
simulating a regional outage.
Some interesting observations that you can make out of here is
basically a dip in
broader availability for the application itself,
even though we do expect it to stay at 100 percent,
if we have two availability zones left,
there are some anomalies in the data path availability.
What I'm looking at here is the aggregation or
the average data path availability for
the load balancer during and after the experiment run.
What I expected originally is up until step number three,
our data path availability would have
stayed at 100 percent throughout up
until we killed all three VMs or all three availability zones.
But that is not the actual case here.
As soon as we killed the first one,
our availability and our load balancers' ability
to redirect traffic did actually go down.
This is an example of how you can
basically test your assumptions using Chaos Studio
and essentially validate whether or not your
hypotheses about the resiliency of
your system are actually true or not.
In this case, this is learning to see
check back in and figure out why this is
and how can we
build better resilient applications
that our customers are served better.
But yeah, that's all I have for you today, David.
Hope that was useful.
Let me know if you have any questions.
>> Yeah, well, I mean, I got to say that
the thing that I appreciated from the demo
is originally Chaos Engineering sounds like it would be hard,
but setting up something that's just three steps
that works in a progressive way is really interesting.
I was also interested by the fact that you
might have a step in which you take everything out.
It seems to me, in addition to simulating a regional failure,
it also shows you that you're shutting off the right things.
Like if in fact you have
an experiment that takes everything
down and you're still serving something,
then that notes that maybe you're
not targeting the right stuff and it
seems like that seems like a good pattern to make sure.
The thing you know for sure is true,
which is if you shut everything off,
things should stop working.
You validated that part and that seems
equally as important as the other parts,
because I know I've been in situations where I thought I was
testing one thing and I was really targeting something else.
It's really cool to see this.
It's also interesting to see a situation
where you're surprised by the results.
With all of that now the question is,
cool, I want to get on this.
How do I start? Like, where do I start if I want to get
in working with Chaos Studio to get going on this?
>> I think the best place to start,
especially if you're brand new to the area,
is follow our quick start guides.
That shows you from ground up how to spin up a few VMs,
to set up your resource groups to be able to like just
dip your toes in the water and start a few experiments.
As you get more comfortable with the space,
you can basically start small with your actual applications.
Pick some of the most risk-tolerant portions
of your application that can
stand chaos and having thoughts injected towards it.
They're not mission-critical for
your application and have
fewer dependencies and you can sort of build
your way into more robust experiments
for like your production environments, for example.
I would definitely say start small
and the best place to start is our quick start guides.
>> Okay, I dig that and I like the idea of setting
up just something for these experiments test.
Because the first thing I'm going to do after we're
done with this conversation is go
stand up exactly what you did and try to figure out what's going
on with that graph because that's like super interesting.
I want to thank you. This is super great.
I'm going to get some notes.
I think I'm to switch back over to host mode now.
Cool. Well, thank you,
Shrunga that was super cool to
watch and I want to thank you-all for watching this.
We really appreciate it.
Hope you'll join us again on
another episode of the Azure Enablement Show.
[MUSIC]