>> Good news. We're back with
another episode of Armchair Architects,
the Azure Enablement series that's
an ongoing conversation with Eric and Uli,
two of the smartest architects I know.
This episode is a particular
favorite of mine because we're going to
be talking about chaos engineering, so join us.
[MUSIC].
>> Hey. Welcome back to another episode of
Armchair Architects as part of the Azure Enablement Show.
This is the episode
you've been looking forward to about chaos engineering,
I know you might have been looking for it in the previous one,
but this is where we're going to get into that stuff.
My guests, Uli and Eric,
know that I have opinions on this topic as well and I think this
is going to be a really fun video because I want to get into it.
There's a question I'm dying to ask them
and I'm looking forward to asking them as part of that.
But I think before we do that,
because I feel even though in
the operation space and in
my space chaos engineering has been around for a while,
I'm getting the sense that it hasn't been in
the gestalt of our realm nearly as long.
How would either or both of you define what chaos engineering is?
Let's just get that onto the table and
then we can argue about it or not.
Raise your hand, go first, I want to hear it.
>> Eric has strong opinions about this one.
>> I do, but it's more along the lines of what architects
tend to do in terms of putting a solution into production,
but to me and I don't know if this is literarily correct or not,
but the way I think about chaos engineering is that it's
a practice that's part and parcel of
application development in which you try to excise uncertainty
about specific failure conditions that might
happen with the system or its components.
Today what I notice a lot of architects do is that
they guess what's going to happen if there's a failure condition.
In many instances, they don't even like to think about it.
I know I didn't. When I'm
designing this beautiful, elegant solution,
the last thing you want to think about is
what happens if somebody just goes and
kicks over a rack or unplugs
something or somebody doesn't an update assert.
How is your application going to respond?
But in today's world,
we need to think about these things
and we shouldn't be guessing about it and to me,
the practices of chaos engineering help us to excise guessing
and achieve certainty about what
failure looks like in our applications and the architecture.
>> I would go a step further.
We've always had the need for chaos engineering in
the sense of always expect the unexpected is
really the mantra that
chaos engineering is putting to the forefront and I
look at this as most people have been
saying," This will never happen until it does."
The Cloud, because you have two independent life cycles,
you have the Cloud lifecycle,
the services that move forward and bring new capability all the
time and you have your own application or system lifecycle,
which of course moves on as well.
Sometimes these two life clash were
incompatible is that either you
introduce or the service introduces,
make your application not work anymore.
Therefore in the Cloud it becomes more important
than on Prem where you have
more control over the entire lifecycle,
but I would argue on Prem it has been also really important
because you just don't know what you don't
know and introducing failure,
which is really what chaos engineering does into
your system in a way that you might or
might not have expected and then see what happens,
is really a great art form
and it's certainly good practice to do that,
the most extreme form is to do that in production.
I would say 99.9 percent of the people I work with,
the customers and partners are not
ready for that level of chaos engineering,
but introducing it in the development and test cycle,
maybe in the canary where you have very limited impact,
is a great starting point for doing that.
Maybe Eric you can share a little bit what chaos engineering
really is in terms of introducing failure. What does that mean?
>> To me, it's the opportunity for us as architects to not guess,
to actually acquire knowledge
before failures happens about what failure actually looks like.
To me, a principal component of
chaos engineering is fault injection
as part of the application development process.
Fault injection is something that I think everybody has heard of,
but we don't really know what it really means,
let alone included this part of our app development process.
To me, I think about it relatively simply.
Suspend services, suspend components,
pull out the network art infrastructure,
turn off the integration
between your components and see what happens.
But it's also a mindset that you actually have to embrace.
As this happens,
how gracefully do your components,
starting with the user experience all the way
down to the data layer or storage layer,
how do they handle a failure.
Is that failure that they had,
the way that they handled was it okay?
Or can it be optimized?
Then once you establish that baseline of failure,
fix your components, CI/CD redeploy them,
whether it's in the canary or
whether it's in a testing environment,
do the same types of failure again and see
how much better you got and compare that to your baseline.
But in each step of the way,
my advice is embrace the failure.
Again, it's smashing your beautiful glass house with a hammer,
but you're going to need to do that so that you
don't have this obscurity around.
What's going to happen in production if my assistance fail?
>> I think of this stuff also it's like you're
running scientific method like experiments.
We have a hypothesis and you have stuff you want to test
and you want to see what happens if you test these hypotheses.
I don't like the hitting the glass house with the hammer one,
because it's going to break.
Chaos testing is not like,
what if I turn off my database,
the app will fall over, we all know that's not
chaos testing from my perspective.
But I really like the idea of trying to push the own,
trying to understand the unknown.
What happens if I apply CPU pressure?
What happens if I apply memory pressure?
What happens if we run out of disk space?
Or another funny one that can be done is,
what if they set the clocks ahead by two hours?
What if I daylight savings time happens?
Which is our own little chaos testing that happens.
I get it that most people don't run in production really,
but I have heard statements saying,
if you don't chaos test your system in production,
the Internet will for you.
Do you agree with that statement?
>> I agree with the sentiment.
Again, a lot of systems are not Internet facing,
[inaudible] the running of
the Cloud and therefore it is less applicable,
although it's still applicable.
I agree that if you are mature organization,
you should do chaos testing in production.
That's great, but most companies don't
feel that they are mature enough to do that yet.
>> Again, the list of failures you
introduced are all fun ones, all good.
The network is always a good one, increasing latency,
reducing throughput, those things.
Having services disappear and reappear.
Sometimes reappearing is more dangerous than
disappearing because people plan for the failure of disappearance,
but they don't plan for the,
oh, the thing is, back up.
What do I do now and stuff like that.
I think Chaos Engineering is a great way of
introducing failure and failure modes,
which we talked about in season 1 quite a lot as part
of reliability engineering to really go and saying,
did I really understand the failures that could happen?
Can I deal with them?
If this failure happens, what do I do?
Again, I think a used,
in the last episodes,
I used the Helix architecture from
Netflix with their simple DNS trick,
which I still love,
where they say DNS is dead.
I would say a lot of applications would fail
because they always rely on name resolution and go for that.
That's the right thing to do.
But you should have an exception in there in
your code that says if the name does not resolve,
here's a cached file of IP addresses I at the last time and I go
and effectively connect or try to connect
using IP addresses that I got from the DNS server the last time.
I think that robustness is what people think about when they think
about Chaos Engineering and
Chaos Engineering forces you to go down this path.
Now, one thing that I always ask myself,
so how do you get to great chaos?
Meaning what makes sense?
Because all the things that you mentioned
obviously can happen and should be thought about.
Sometimes it does have impact,
sometimes it does not.
The other piece that I always think
is I have an incident database.
What happened to either because your Cloud provider had incidents
so test for those or you had incidents,
make sure that they are part of
your chaos repertoire so
that you can always introduce failures you've seen before.
They will be to failures you have never seen before
ans despite people's imagination,
there will always new things that happen because life
is just tough and
therefore I think really
tracking incidents and making
sure that they are part of your chaos,
testing, tooling suite, whatever it is, is, I think,
a good practice to make sure that your systems continue to
improve in reliability and robustness like Eric said before.
>> Yeah. Eric you have something to say and then
I have two questions to follow up.
>> Sure. I'd like
the chaos repertoire moniker because it's the Chaos Engineering,
I think, starts with just the obvious heavy handed favorites.
Pull out the infrastructure, disrupt the network,
take a service down,
and then it becomes more hypothesis based.
I think maybe, David, you mentioned this earlier.
It's not random.
It should be hypothesis based,
which is if I do this to the system,
I expect that this will happen.
That did or didn't happen and now
I need to make allowances for it.
But also evolving from that hypothesis methodology should
be correlated events that correspond to real world events.
If a region goes down,
these types of previous failures are going to happen.
Let's correlate these together and simulate that as part of
our Chaos Engineering repertoire into the application.
I think it's an evolution.
Start with the primitive things that
are then just to see what's going to happen,
evolve your failure hypotheses
and then begin to correlate a lot of
these particular events together so that you
can simulate real world actual events that might happen.
>> What I'd like to know in
a little bit of a challenging way really is,
I agree totally that it's good to have
past outages as examples of things you might run as experiments.
Because you want to know, did I learn from that outage?
Are we now resilient to whatever that was at
the point in which you put in the stuff to deal with that?
Is that still chaos testing or is that just like unit
or some other testing if I'm testing?
Are we still in chaos land or is that just
like another one of your stats as part of your regular testing?
>> It is fair that you could argue that this is a unit test,
but chaos testing is not only about the things you test,
you also when you test them.
Because if you think about it,
if I throw in an error at 2:00 AM in the morning,
the team is not there.
It is running in production,
let's say in Canary with reduced SLA,
but it's still an SLA.
Now all of a sudden the operation team has to deal with it.
Can they deal with it instead of just the people that
actually can put a piece of new code in,
patch it or whatever they need to do and go from there.
For me, chaos testing is also about timing when you throw it in.
When you do a unit test,
everybody know it's a unit test.
That's a great practice to do and everybody should do it.
It's part of our shift left
quality move that I think we have talked about before.
But for me, throwing the arrow in when
it's least expected is also part of
chaos testing and then see can
the operation team deal
with the problem instead of just the deaf team?
I think that's another level of robustness that you should
acquire as part of adopting a Chaos Engineering methodology.
>> I have a remaining question unless Eric you had
another response then I wanted to ask you the way it can work out.
>> We're just going to say it and I don't know if I,
but I'll plug it again if I already said it,
it should be part of the fabric of how you develop an app.
Fault injection as part of testing in app development,
is part of folding
Chaos Engineering into the creation
and deployment of your application?
I just wanted to say,
like Uli said, it's not how often you do it,
although it should be frequent,
but it's also when you do it just
to determine whether or not your ops team can handle it.
You always practice a fire drill and it shouldn't be
the first time you actually
need the fire extinguisher you don't know
where it is, metaphorically.
>> That's a good one, Eric. Actually, I like that one.
>> That's a good segue to my last question,
which is you use the word maturity,
Uli, and I don't think you meant like,
do you have a gray beard like I do?
You're not saying, has
your organization been around for a long time?
You were talking about are you
mature enough to test in production?
What does maturity mean?
I know that Martin Tyler was talking about you have to
be this tall to ride this ride and
so I'm just curious if you
can say something about what's maturity mean?
How do you know whether you're mature enough?
What do you mean by that word?
Let's just ask that as my last question.
>> Yeah. Let's take the idea of if you don't do unit tests,
don't do Chaos Engineering because you know you will fail.
You have to write a certain level of things like
unit tests, great all abilities,
which effectively is burn tests against your system and so forth,
so that before you go into
random testing of events that you have never thought about,
you should absolutely have maturity in terms of what are
your practices that you use and drive from there.
I've seen actually mainframe code that had
zero exceptions in it, no exception handling.
They just assumed that everything
was awesome and it would never fail.
Obviously, that's not the real world that we live in,
so if your coding practices don't
have the normal standards, things like,
I need to back off and retry,
I need to handle exceptions and
so forth and I have to have a general exception,
if I don't know those exceptions, what am I going to do?
Because I only know so many out of this real world assembly.
For me it all comes down to
what are your practices that you live every day?
Once you reach a certain level of maturity in the sense of yes,
I have all the exception handling in,
I have re-tries,
I have back-offs,
I have throttling for my services
and all these things that we talked about before,
then you can throw Chaos Engineering at it to fine tune what
you don't know and I
think that's the key part from my perspective.
>> That makes perfect sense to me.
Well, I think that's a super good place to stop.
I want to thank you,
Uli and Eric and I would like to thank all of you for watching
this episode from Armchair Architects
as part of the Azure Enablement Show.
Join us next time.
We'll take out some other really cool topics. See you then.
[MUSIC]