>> Hey. Good news.
We're here today at the Azure Enablement Show talking to
Kunal about Operational Excellence. Join us.
[MUSIC]
>> Welcome to the Azure Enablement show.
We will be discussing the challenges you
and our other text-savvy customers have encountered.
Together, we are going to be talking with experts to find
out how they think about these problems,
recommended tools, and best practices,
and tips they've learned from
years of experience that you can use.
Today, we're going to be talking a little
bit about operational excellence.
I thought it'd be useful for me to approach this like a customer,
so let me switch to customer mode.
Now I'm back as a customer.
Now I'm going to get a chance to talk to
Kunal about my operational excellence problems.
Hey, Kunal. It's really great to see you.
>> Hey, David. Nice to see you too.
>> I could really use some help.
Things at work aren't going nearly as well as I'd like.
Let me tell you the things that are going on for me.
One of the things that's going on is that we have,
I guess you call them epic deployments, in which
we rub our software like once a quarter, and
everybody is all hands-on deck, and we all
have to stay there like over the weekend, in which we
try to go to the next version,
but then also stabilize, and you get it working.
Then the week after that is terrible, because we were
trying to clean up the pieces
that we didn't get during the weekend,
and then it starts all over again.
That's one of my problems.
The other problem that I'm having is that
I've got issues with my monitoring.
I don't have a good sense of how my systems are doing,
but I do know that there are paging me
every hour and I'm getting pages
at 02:00 AM at night about this stuff.
It doesn't seem sustainable.
I would really like your help.
Are these the things that you think you can help me with?
>> That's a great question actually, David.
I think we need to talk about Operational Excellence.
If it's okay, I will share my screen.
>> Yeah. Absolutely,
glad to hear what you have to say.
>> David, let's think about operational excellence as a function.
It's a function of disciplined execution.
It's about doing the right thing the right way most of the times,
and then positive outcomes.
This is where we're really asking,
are we solving the right problems?
Moving on from there, actually,
if we think about this function and plot it as a graph,
there are four quadrants we can look into.
Let's start with the bottom left quadrant.
This quadrant is what I call the quadrant of chaos,
it's common for anything where there
is no clear ownership or leadership.
When we think about apps, In the Zone,
these are the ones that are no longer actively
maintained, but are required to be kept running.
Teams that are usually maintaining them
lack knowledge of how these apps were built,
what needs to happen,
and those things as well.
Management and operations in this quadrant
is very ineffective and chaotic.
Moving on from there to the next quadrant,
this quadrant is a quadrant of high toil zone.
Your teams will spend a lot of
time in manual activities like release patching backups.
Based on what you are saying,
it feels like your release management process
falls under this quadrant.
>> Yeah, truly.
>> Again, these are the things that could be easily automated.
You're spending a lot of cycles on things that could be automated.
If you think about an example is your teams are
good or great in this quadrant in following procedures,
step-by-step processes, but might be able to think about how
I can take it to the next step.
Moving on from there, this is
the next quadrant that I call it unsustainable quadrant.
In this, the focus is usually on
treating the symptom and not much on underlying cause.
I know you mentioned you spent a lot of time over
the weekend to resolve a production issue.
What did you notice, David?
>> Well, I noticed that we seem to be
doing the same thing over and over again.
The same problems come up and
the same things have to get handled manually,
which seems really unpleasant.
>> I see. Was there
any specific reason for the downtime that you noticed?
>> Well, sometimes the downtime is caused by us upgrading stuff.
Sometimes the downtime is caused by something
where the application just decides to die on its own.
It's really hard, but it seems
to do it over and over the same way.
>> How did you solve this? What was the fix for that?
>> We started, we hit it with a hammer until it works again.
I can't say that we do anything.
I wish I knew the one magic way to do it.
>> I see. This is exactly the problem with this quadrant.
When you are working in this quadrant,
you're always firefighting.
You really want to solve the underlying problems.
For example, every time the problem
happens and you just might be scaling up your machine,
might be you want to think about, hey,
might be I want to look into what's happening in the application,
is it really scaling, as example there?
We move on to the next coordinate.
I think this is the desirable or the strategic quadrant,
is what I call it. Very strategic.
Before we go there, it does
require a considerable amount of
investment both from your team
as well as from the senior leadership.
If you think about it, it helps you to scale better.
In Azure systems, complexity grows,
or as the number of workloads you manage grows,
you are in a much better position to scale. Does that help, David?
>> It does. It makes me a little sad, because I think we're in
the wrong quadrants and we have to move towards them.
I would love to know like, where do I go?
How do I move towards that magic quadrants up into the right?
>> That's a great question,
might be we should look into
Microsoft's Azure Well-Architected
Operational Excellence guidance.
Let me quickly walk you through those.
If you think about
operational excellence as one
of the key pillars of Well-Architected,
and it offers you the guidelines to clear
desirable and sustainable applications.
Think about it, that top right
quadrant, what we're thinking about here.
There are five key principles for Operational Excellence.
I will take you through, but based on what
I'm hearing in your challenges,
probably we'll focus on first two.
>> Okay.
>> The first one is about optimizing
your build and release processes.
I would start with the question here, David.
What if your non-production environment,
let's say you have a big release coming up,
it might be just before the Black Friday, release is coming up,
you have a strict procedure,
you test everything in
non-production before it's released to production,
what if your non-production environment was deleted by accident,
how quickly can you bring that up?
>> The short answer is we would have to really work at it.
We might have to go back to our documentation,
like it was built manually.
I imagine we'd have to rebuild it
manually and figure out what we did, and hopefully get that right.
>> That's characteristics, that's a challenge.
You want to really think about everything
as score when you think about this area, this principle.
Not just infrastructure as score, which is again,
a very good practice and common practice these days,
but also configuration as score.
That's something you want to think about.
We move on from here to the next principle.
This is about monitoring
systems and understanding operational health.
When it comes to monitoring,
you can monitor a system for different things.
You can monitor it for availability,
for thresholds in all CPU memory,
might be abnormal performance degradations,
you can monitor for that, security
issues or it might be business processing packs.
However, what I see and hope so commonly is known,
the question of how do I monitor is relatively
easy to answer, but what do I monitor is not that easy.
That's something you want really to think about.
Maybe, work with your Dev teams,
operation teams, business teams to figure that out.
I will move on from here to the next principle, which is
use loosely a couple of architectures.
Think about this as the advantage
loosely coupled architecture gives
you, allows you to deploy systems independently.
You can deploy them, test them,
update them, patch them, all those good things.
What you're doing essentially is you're minimizing
their blast radius by having
the clear boundaries between
different systems. I will move from there,
so next principle is about rehearse recovery and practice.
This is all about testing your disaster recovery drills,
including failures scenarios and validating operational on books.
I've worked with hundreds of customers and the challenge
is I see everybody agrees when we
talk about this, but rarely does it happen.
This is again an area you want to investigate and
see how confident your teams are,
because failures are bound to happen,
but having the right procedures in
place and confidence in those procedures will help you.
Finally, embrace continuous operational improvement.
As business evolves, it is necessary to circle back and review
the procedures that are in
place to ensure that they're fully optimized.
>> That makes perfect sense. I get these five things.
I have to admit to wondering,
I can go at this many different ways,
but I would like to know,
how do I get started?
How do I know, where is the right place to dig into this stuff?
>> I'm so glad you asked that question, David.
If it's okay, I will share my screen and I will show
you a quick demo of this.
>> Great.
>> What you can do is you can do an Azure Well-Architected review.
If I click on this, I can start my new assessment.
When you start a new assessment,
you can fill in details about your app,
your e-commerce app,
Contoso app. Then you can sign in to
your subscription and pull in
recommendations like Azure Advisor
recommendations that we provide.
That can be provided.
Once you can sign in and provide those recommendations,
then you can go and select, hey,
I want to do an operation excellence review.
You can say, I want to do
operational excellence review for this solution.
It will present you with series of questions that will help
you understand your systems and you'll figure it out.
Once you've done that, you can fill in those questions.
Then once you've filled in those questions,
I would highly recommend this to the group activity.
Talk to your team,
ask them questions and work together, if you can do so.
Once you've done that, you can actually see the results.
It will tell you where you stand.
This will show you what your score is.
In this case, it's showing me my score is 28.
It will also show me
recommendations, what are the things I should be looking into.
Those are the details you can then work with your team.
David, that was my demo.
Again, if you are one of the managed accounts and you got
your Clouds Solution Architect working with you
or your Customer Success Account Manager working with you,
you can reach out to them to
actually help you with these assessments
and also remediations as well.
Do you got any questions for me, David?
>> No. I guess my only question is,
do you know anybody who could help
me with these first two things,
the release engineering aspect and
the monitoring aspects that I could talk to?
>> Yeah, I know two of my colleagues would be great
you could talk to. I will connect you with them and you
can definitely dive deeper on some of these topics.
>> That would be fabulous.
>> That's great, David. By the way,
I have a question for you.
That snapping thing you do,
does it work for me as well? Can I try it?
>> I have no idea.
>> Let me do it. Hey, that's pretty cool.
Maybe I can do it to you.
>> This is getting entirely out of hand.
I think what I want to do is thank you, Kunal
and I want to thank you for watching this.
Folks, I hope you'll join us for
another episode of the Azure Enablement Show where
we're going to dig deeper into some of
these topics that we've talked about in this video.
Thanks very much for watching.
We'll talk to you a little bit later on.
[MUSIC]