Hey, friends.
For mission-critical applications, resiliency is paramount.
Always available, resilient to failure, and fully operational.
On earlier episodes, we looked at Azure Chaos Studio and Azure Load Testing.
Now Martin and Heyko are here to show me how to combine these services
for deployment testing of mission critical apps, today on Azure
Friday.
Hey, friends.
I'm Scott Hanselman, and it's Azure Friday.
I'm here with Martin and Heyko.
How are we doing, gentlemen?
Hi. Happy to be here. Good. Very cool.
So this one's going to be an interesting one.
We're going to be talking about continuous validation.
We say continuous deployment.
We say continuous integration.
This is something that we talk about all the time.
But Martin, help me understand, what do we do?
Everything continuous now,
What does continuous validation do I need to be doing this right away?
You don't have to as a developer, but you certainly can.
And just to increase the confidence in the things you are building in Azure,
this we are talking about clouds, we are talking about Azure applications.
And in the context of what we do,
Heiko and I are talking about Mission Critical Azure applications.
And for those continuously validating everything is essential.
Interesting.
So Heiko, mission critical, like do I decide what's mission critical?
I have a blog, but I'm pretty sure it's not mission critical.
It's important to me though I could still do continuous validation.
That's that's right.
That's a really good question.
And in the end, it's you who decides if something is mission critical or not.
For our customers, it's it might differ what mission critical is.
And it really depends on their business, their use cases.
So the scenarios we are talking about are different.
But one thing have all of them in common, and that is
that we're usually talking about applications that never go down.
So we expect 100% availability or applications that are
available if possible.
And this is something that we try to ensure using continuous limitation.
Interesting. So they never go down.
So when I integrated continuous integration and continuous deployment
into my site, I did it so I wouldn't have fear of my site.
I wasn't sure what code was in in, in production at the moment.
So I needed continuous deployment.
I wasn't sure about the third party dependencies that I was integrating with.
So I added continuous integration to make sure that I was doing testing.
But I'm not sure if I'm validating
the correctness of this this application.
Help me understand where that fits into my pipeline.
Yeah, pretty good point.
So as part of the as our mission critical design methodology,
we define continuous validation as one of the key concepts.
And this is also reflected in our reference implementation.
So we are using as a pipeline to, as part of the deployment pipeline,
do Azure load testing and in parallel
we use Azure Chaos Studio to inject failure
that helps the developer to test
the workload, the changes, the new code under production like conditions
to have load to have failure
because failure is something that we have to expect in a cloud environment.
So this allows the developer to test, for example, if the retry logic works,
if the application can handle noisy neighbors
or if it, for example, is able to scale in and out as expected.
Interesting. Okay.
But Chaos Studio affects what's outside the application, right?
It's telling you that like this network has gone down or that neighbor was noisy
or puts strain on it, but it doesn't test my code being wrong.
So I'm still counting on my actual test suites and things like that.
This is a tool as a part of deployment
test plus load testing plus Chaos studio.
It's not going to replace my existing test pipeline.
Yes, that is correct.
As part of the pipeline, we run the regular test.
We have all the unit tests.
We are doing integration tests.
We also have smoke tests to validate that the application is live and responding.
But then we are adding to it by simulating failures
to verify that it can withstand those failures
which normally happen in the cloud, but instead of doing it manually.
And that's actually a fun exercise.
We did that once when we were just randomly destroying infrastructure
pieces in Azure and just observing how the application reacts to that.
And it was great because it helped us understand that, Oh,
this piece is not resilient and this database connection is not retrying properly.
But Chaos
Studio allows that allows us to automate all of that and use
a service to do it automatically every time we deploy
or every time we do a major change where we expect things like this change.
Well, this sounds too amazing.
Can it can can I see a demo?
Can we actually do this live?
Absolutely. Let's do it.
So before I started the demo, I will set the scene
a little bit because this slide is not random.
This actually represents what we have deployed in Azure.
So we'll see Azure pipelines
and then we will have Azure load test generating
artificial load to Azure Front Door, which has two backends,
two AKS clusters each in a different region.
And then we apply chaos to one of those regions and we expect Front Door
to catch this situation and direct traffic only to the healthy region.
So let's let's take a look
at Azure Mission critical.
We have all of this implemented.
It's not just guidance.
So you will see if you deploy it, you will see that there are three types
of pipelines, production integration and end to end
and to end is used by developers
to produce their own development environments on demand based on a branch.
And we definitely recommend all customers who are building mission
critical applications in Azure to have the ability for developers
to just deploy on demand their own environment,
which resembles production, but it's just downscaled.
But functionally it's the same.
So that's what we call end to end.
When I trigger this pipeline,
I can switch to my feature branch.
That's
a new feature
and as all of us do,
it's called Feature three.
It will read the YAML from the pipeline and then I'm presented
with three very important checkboxes and this is all defined in YAML.
This is not Azure DevOps doing this, it's actually the pipeline itself
being implemented in a way that I have the option to run chaos
and I have the option to run a load test.
Wow. Okay.
And I want to make sure I get this because as soon as that popped up,
I'm thinking to myself, Did they add a new feature to Azure DevOps?
No, you did.
You can add those checkboxes as booleans and then they provide input
into your YAML.
So you have upgraded DevOps to Azure Mission Critical.
Super important point we want to highlight here is that all
these artifacts are available, so we're not just talking about
theoretical things here.
We've implemented everything.
You can take it today
from our GitHub repository and apply it to your own environment.
I will uncheck destroy because that's what I don't want to do.
When the pipeline is finished.
This is used for dependency validations, things
like updated NuGet packages or NPM packages.
I just want to see if my app still works.
It will take a few seconds to trigger
and this
pipeline will deploy the whole infrastructure.
It'll build the application, it will run the tests we talk about.
It will do some smoke tests,
it will redirect traffic through the Front Door,
it will do tests again and finally it will upload a
JMeter JMX file to a little testing service.
It will also create a chaos experiment in Azure Care Studio
and run those in parallel.
We have to add a delay
so that the load test has enough time to start
so that we have some artificial load and then chaos kicks in.
And then this pipeline description here,
this is a visualization of what's going on in your YAML.
Yes, exactly.
All of this is YAML defined.
Just an example.
This is what the experiment we are using in Chaos studio.
So we are targeting AKS and we have an
experiment which is going to
disable all pods
within our workload namespace.
So we have a workload namespace and we're basically saying for 3 minutes
don't allow any pods from this workspace to work.
And we're targeting one of the two clusters
since this whole run
takes many minutes over the not to wait for it here,
I will show you a previous one we ran earlier
and this is a load test result page.
There's a lot happening.
So I will point out directly to the charts at the bottom,
because these are these in the best demonstrate what actually happened there.
You can see the ramp up time was a few seconds,
a few minutes, so that we get enough time for the application
to probably boot up and and work.
And then we started the actual chaos experiment here.
So you can see the number of errors actually growing
because one of the regions was not working at that time
and it was still getting requests.
That comes together with the increase of latency.
Suddenly this spike stops and the number of errors goes down.
But this is not 3 minutes yet, so it means that the chaos is still there,
pods are still being disabled, but the application is healing.
It is still working.
What happened is that Front Door realized, oh, this this region is not healthy.
I need to take it out of the rotation and I need to out of the load balancing
and I need to direct all my traffic to the one healthy region.
But what happened to that one healthy region?
It was like, oh, no, there's so much traffic going on here,
I need to scale out.
So we see the response time going up for a while,
but then as the scale out kicked in, it's actually going down again
as we have more pods, more nodes handling this traffic.
And finally, when the chaos experiment ended,
we see that it went back to normal.
Very cool.
Now, is it
is it up to us as the application owners and architects to decide
if this is appropriate or correct behavior?
Because we might say, well, that didn't take that took too long
and there were too many errors and it didn't recover quickly
or that's unacceptable.
Like we decide what the metrics are for success and failure.
Right, Exactly.
That is one of the parameters we provide.
When we create a load test.
We can say that the test criteria
for this test to be successful are these.
And for instance, in this particular case, you see that the test actually failed
because we were too optimistic about the response time
we set the threshold to 75 milliseconds,
but this is all defined in code.
We actually have two configurations for this.
We have baseline for chaos.
And if you have baseline for the normal operation
and the reason for this is that we realized that under chaos
we expect the latency to be a little higher.
So 300 milliseconds, even like a second, might be actually acceptable
as long as the application still responds to the user's request.
So that's why there are actually two different baselines.
We didn't use it in this case, but normally we recommend
having those set up specifically, and this is what it looks like.
It's just JSON file,
but definition of the metrics and you can see the 75 milliseconds here.
Do you think that people aren't using this
just because they don't even know it exists?
Because that's why I'm not using it.
That might be one reason.
But also it's not trivial to configure.
You really have to understand your application.
You really have to know what you're looking for and what you're testing.
And you need to synchronize all of this properly as part of the pipeline. One.
That's a great point.
Yeah, being intentional about it.
And also of course I can I can imagine your application startup times.
I know that a lot of people don't have a cold boot strategy for their apps,
so there's prepping the site, getting all of the, the data being out.
There's certainly
the back end database needs to be prepped with whatever test data that's in it.
So making sure that your site can completely deploy and, you know,
quote unquote cold boot and then as you say, synchronize
so that you're getting a true test of what the warm state of your application is.
And you're not seeing errors on the on the on the front end or the back end.
Mm hmm.
Very cool. This is this is really exciting.
So you've taken this information and you can make a judgment
based on your understanding of your application.
You set your thresholds about the appropriate number of of errors.
And this is telling me what's going on
on the on the you should on the back end is this looking at things
like insights into the container, can we get more than just
requests per second and response time.
So this is all provided by Azure Load Test.
So this is just one lens, just one point of view.
The other interesting point of view, for instance,
is looking at Front Door because that nicely demonstrates
what happens from, from Azure's point of view.
This is more the view of the end user,
but this is what Azure is actually doing and we can see that there are two backends
orange and blue, and the blue one is the one with the chaos.
So that's one of the regions which actually was not working.
And we see that Front Door actually stopped sending requests to it
and the other region took up those requests.
But again, it wasn't able to cope with it.
So it struggled for a while, but then it picked up and scaled out properly.
And in the end, when the chaos
ended, they converged again in something
like a 50-50 split of all the requests coming to the application.
So when evaluating all of this, you can use
any other part of Azure to actually see the effect
from one another.
Nice view and I don't have that prepared here,
but another quite nice view is Application Insights.
You can watch the life view and you can see exceptions popping up
as your application is being overloaded or as it's enduring the chaos,
enduring the chaos.
I love that
this must take a long time.
I mean,
I recognize that there's some experimenting in some scripting,
but I want people who are watching the show to be able to go
and learn how to do this themselves, because this is a very powerful tool.
Yeah, And this is one of the reasons why we made all this public.
So we not
only have the guidance on Microsoft learn, we also have all the artifacts.
So even without applying everything that we have in Mission Critical,
someone can just go and take, for example, the,
the parts that we've developed to implement continuous validation
takes the examples
to make it a bit easier to adopt the service in your own environment.
That's cool.
That's very generous of you to do that.
And then for my site and for folks that are listening and watching right
now, you know, there may be a little bit of experimenting and scripting.
I assume that's for, you know, for timing and getting a baseline
and really understanding your app.
Because if you don't understand your app, applying chaos
to, it's not going to necessarily help.
And that's the tricky part.
So we provide framework.
We provide here is a scaffold of how to do that.
But you are 100% right.
It's a learning, it's a it's a journey.
You have to
learn more about your application, what to expect, how to react to failure.
And then over time
adjust the baselines that Martin, just showed to improve them
because you have two potential outputs of a failed test, right?
You have either a baseline that needs to be adjusted
or you have something broken in your code that needs to be fixed.
Very cool.
So where can folks go to learn about Azure Mission Critical Online?
So one of the places we have is the GitHub repo.
So that's either GitHub dot
com slash Azure slash mission critical or mission critical online.
That's one of the specific implementations we have.
I recommend scrolling down a little bit
and going to the Getting Started guide, which is really a step by step set of
instructions which allow you to deploy it to your own Azure subscription.
It takes something around an hour to get completed, but then you will get
the whole thing, the whole mission critical in your subscription.
Or you can just browse the repo and find artifacts and copy
paste them into your solution and just play with them and experiment.
That's what we do as well.
So that's one part of it.
And the other one is on Microsoft Learn,
we have a set of opinionated guidance which actually contains
a lot of content with considerations and recommendations
on how to build mission critical applications on Azure.
Very, very cool.
Well, thank you very much, gentlemen, for hanging out with us today.
Thank you.
It was a pleasure. Thank you for having us. All right.
I am learning all about the importance of continuous validation
of mission critical applications today on Azure Friday.
Hey, thanks for watching this episode of Azure Friday.
Now I need you to like it.
Comment on it, tell your friends retweeted.
Watch more as Friday.