>> Good news, we're back with
another episode of Armchair Architects,
the Azure Enablement series that's
an ongoing conversation with Eric and Uli,
two of the smartest architects I know.
This episode we're going to talk
about architecting mission-critical apps.
How do you build things that are
mission-critical and stay running no matter what? So, join us.
[MUSIC].
Hey, welcome back to
the Azure Enablement show, Armchair Architects episodes.
Today we're in for
a really super cool one for me who's into this reliability world.
We're going to talk about, how does architecture intersect with
those applications and systems and
services that have to be always on,
that are, I guess the term is, mission-critical?
I'd like to welcome Uli and Eric
back and I want to talk to you and find out
why isn't architecture the
same for mission-critical apps as it is for any other app?
Don't we want everything to be running all the
time and when we designed it that way anyway?
>> My thought process is we'd like it to be
a consistent set of
architectural principles for everything
from the small to the large.
But in general, I think when
you're building mission-critical applications,
we need to think about things in terms of
those super important architectural patterns
and characteristics of these systems.
The way I think about mission-critical apps
is understanding that there's going to
be failure resident within
these applications at some componentry level.
The only difference here with
mission-critical applications, in my estimation,
is what happens when one of
those application components are under a system or an outage?
How graceful can your user experience be?
Can you employ some of the techniques we talked about
in Season 1 to increase reliability and
then potentially self-heal and self-recover such that
the failure is transient but
the users of that mission-critical application
never know the difference,
no data is lost, no transactions are sacrificed?
>> Uli, what do you think,
or do you think we need to treat this class of
applications differently from any other class of applications?
>> The story is very fairly simple.
On the one side, we talked about cost in
the first episode of Season 2.
If you have a mission-critical environment,
which simply means that your business life or
whatever is super important to
you is at risk if this application fails.
Let's take 911 in the US as
a calling system for emergency services.
If that is down, people are at risk
and therefore you have to make sure that the system is available.
If you are running a business,
we talked about the Black Friday example where shopping websites
need to be up and running because there is a lot of
business that runs through there and if your system is not up,
you might put the business that you are serving at risk.
For me, mission-critical is always
subjective in terms of what mission are you serving?
Sometimes it's as extreme as life at risk.
Sometimes it's the business is at risk.
But at the end of the day, it's a subjective decision.
But once you make that decision that
this application, the system solution,
whatever the word is you want to use,
is mission-critical, then you need to go and change gears,
which also includes a cost conversation because
a mission-critical system is more expensive to host,
operate, maintain and so forth than a system that has only,
let's say, taken two nines as an availability measure.
That might be good enough for a lot of stuff.
But for the key pieces that really run your business,
you want a higher level of availability,
robustness, and so forth.
>> But is it a matter of just throwing more money at the problem?
I know we were talking about cost before,
but it's cool,
I've just bought the largest tier you got
with lots and lots of GPUs or something.
Am I mission-critical now because I have a large bill?
>> As with many things, David,
I don't think it's that simple.
But I do think, the way to answer that question,
is to take the first step
in classifying the critical nature of the system.
If you can understand
the classification system or you have one already that exists,
the goal would be for you to say this system or
its components because they serve
multiple systems that might
have different level of mission criticality,
it's too big to fail.
It can't have more than five minutes of downtime a year.
Those types of classifications will help
you as you begin to take this to
your decision-makers to justify
the investment required if they
want that particular level of service.
It also has a backwards effect to determine
whether or not you make modifications to
the system architecture or you
actually have alternative scale units as we talked about
in the last session around how I actually scale
this system and make it
available or highly available based on the criticality.
>> I want to bring it back to
the architecture patterns that Eric mentioned in the beginning.
At the end of the day, mission-critical is
effectively a usage of different architecture patterns.
We have, for example, talked about active-active.
You don't do that with systems that don't need to
be available because it's really expensive.
Running two systems in completely different regions,
making sure that they are not balanced and
so forth, that's expensive.
You have to do that for a system which is critical to you,
the mission that you are serving.
Ultimately it all comes down to
those architecture patterns that you utilize
and let's say and these services that you utilize because
GPUs having more or less of them is obviously not the criteria.
But if you have a state management system
like a database or a storage,
you want to be able to say,
the storage can never really fail
and I can't lose the data, therefore,
I need to replicate it into
another region, potentially multiple regions,
so that if the real disaster happens,
the business is still able to serve again.
Again, 911 in the U.S is
the calling system for the emergency services.
When we supported the New York City state with this capability,
we ran the primary system on Azure and
the backup system on-premises.
That backup system was a hot standby
and not an active-active system,
but a hot standby that could take it over just in case
the Azure region that serves as
a primary had a problem or something like that.
That's really the difference
between non-mission-critical capabilities
and mission-critical capabilities.
>> Mission critical doesn't necessarily mean instantaneous.
It means things keep on going if there's a failure.
There are certainly some things like
the 911 example where you're not call back in an hour,
I'm sure things will be fine.
But there are also ones where,
if it doesn't immediately trigger
the order and it triggers it
five minutes later because I had a fail over to something,
that's okay as well.
It seems like mission critical is not just one big bucket of
must always work all the time
at the same speed it was working before.
>> I would agree with that.
Again, it goes back to that point
about criticality and classification.
If you can do that classification,
the role of an architect or solution designer would say, well,
what does that translate into in terms of reliability
and security and network connectivity
and how does that impact my health modeling and
my pen testing and
any operational procedures that
need to be in place to support this?
I think that those are the design areas in which you begin to
flush out based on
the categorization of the criticality of the system.
>> I want to put a little bit of
illumination on the concept Eric just pointed out.
Let's take an imaginary system
that has two components; two workloads.
One is a video stream of a sports event
and the other one is a social capability around that sports event.
If you are on the day of the sport event,
all of a sudden the video stream
becomes mission critical because people are
sitting in front of the TV or on
the Internet browser and watching the sporting event.
Obviously they want to see the event.
Otherwise, why would they watch this at this point in time?
Because the provider offer also video on demand later,
but people love to watch sport events in real time.
The social component while
important is actually not as mission critical.
People will forgive you if this is a
bit slow or it's not quite there.
They are not happy, they will be grumpy,
but they won't be absolutely mad at you.
If the video feed fails while the game is on,
you actually certainly are in trouble and
the situation most likely reverses once the game has been played.
Now people want to talk about the game.
Now the social component becomes far more
important than the video on demand where people might say,
I come back in an hour because all of a sudden this.
Think about workloads, which is really
the characteristics in terms of what
you're thinking about and then also timing.
It isn't necessarily black and white.
This is mission critical and this isn't.
It also can depend on the time of day,
the situation that you're in and so forth,
and being able to vary
the architecture patterns and be able to turn things on and off,
I think is going to be
the art form that people have to think about.
>> I love this example Uli
because in order to make those detections and to
understand when the video signal is degrading or
the social platform is showing signs of fatigue or intention,
you need robust health modeling.
You want a scenario in which
your mission critical application tells you as
the owner that there is a problem before it becomes widespread,
before it becomes noticeable by customers,
and certainly before they call
you or start tweeting that there is a problem.
Robust health monitoring,
which includes architecture in terms of observability,
patterns, pipelines, logs, metrics, all those things,
you need to have a way of emitting that telemetry,
but also aggregating that telemetry,
running inferences across it to say, hey,
it looks like people from this particular region
watching on these devices are showing network contention.
They've dropped down from 4K to 1080p to 720p.
Looks like there's going to be a problem
there. Here's what we do about it.
>> Gosh, I want to go that direction because
monitoring and observability is close to my heart,
but I find myself wondering whether
one of the hallmarks of mission critical architecture
is the capacity or the importance or
the ability to switch design patterns on the fly.
Is a key part of what
an architect does when they're designing these things
constructing things so that
different patterns swap in at the right time or
are mission critical situations
design pattern stable most of the time?
>> Personally I don't know. Uli,
I've never run into a circumstance in which you reach
a point in an application's lifecycle and you're quick,
let's switch to the Beta design pattern instead of the Alpha one.
What you said there, David,
maybe did think of a scenario in
which you are in a situation where you've anticipated,
thanks to your observability pipeline and your health modeling,
you've observed a problem state and you're like oh wow,
the application as it exists today in production,
we can't deal with this.
We didn't foresee it.
Quick, let's figure out what we need to do, let's scale test it,
let's deploy it as quickly as
possible into those specific regions or
across the entire implementation so
we can fix this before it becomes an issue.
There's almost an element of
deployment and testing in terms of CI/CD
and being able to fix things quickly so that
the system doesn't just come down, I think is really important.
>> Well, when I was thinking design patterns,
I was thinking about a previous conversation we
had about load shedding and I was thinking
that a switch in design pattern is load shutting and then, oh,
we're in trouble so now we go to
a load shedding instead of putting it out
everywhere where we switch to
a shedding model where we're in a different way of thinking.
I'm just wondering is that something we
can say abstractly is true in general that
you want to be able to have
a contingency pattern that
is already built into the system, ready to go?
Is that an accurate statement for how to deal with faults that
are latent in complex systems?
>> I think, David, the key that you're
pointing out is composition of
architecture patterns and solutions into an end-to-end capability.
Again, let's go back to this sport event capability.
Actually, there were three workloads we described.
We described a real time event,
we described a social media and video on demand,
which is a different solution.
It might reuse some of the capabilities
of the real time capability, but it's not the same.
Therefore, you can vary
the cost and reliability and availability patterns
accordingly because you have flexibility.
That means you also use then various patterns that are
effectively going on against this load balancing,
elasticity, load shedding or throttling.
All of those are fair game and
obviously something that is not just for mission critical systems.
Mission critical systems mostly deal
with availability and reliability,
meaning you cannot have the system fail if disaster strikes.
Let's say you are in a specific region,
that entire region is out.
That system has to still be available and
that system ideally is made out
of multiple parts not necessarily many parts,
but multiple parts so that you can
determine based upon requirements,
time of day, whatever it might be,
how to make this capability available in
a specific level of availability or robustness.
Because again, building super robust systems,
meaning making them available in multiple areas does cost money
and therefore it has to be planned out for
the right reasons and not just because you can.
>> Once again, you gentlemen have sucked me
into a direction whether I liked it or not
when we're talking about monitoring
observability and we're talking about availability and
understanding how to handle
situations there even if we haven't planned for them,
because that's a little bit of what
we were getting into is that question
about it's cool if you have
a priority knowledge as to how things are going to fail,
but it's a lot harder if you don't.
It occurs to me that there is
one thing that people are starting to do in
my world to try
to determine this and to try to get a sense of well,
what do I do about things that I don't know about yet?
That's the land of CASS Engineering.
This is something we promised we'd talk about
last season and we didn't get to it,
but I can wait no longer.
If you're up for it, I would really
like to talk about CASS Engineering because
that's going to be a fun conversation if you're up for it.
>> Can't wait.
>> Absolutely.
>> Cool. Well, I want to thank you-all for watching this.
I hope you'll join us on the next episode which
now you're watching it and now know ahead
of other people what it's going to be where we're going to talk
a little bit about CASS Engineering with Uli and Eric.
Please join us again on another episode of
Armchair Architects as part of the Azure Enablement Show.
[MUSIC]