>> Good news, everybody. I'm here with Daria.
We're going to be talking today
about observability and monitoring.
I hope you'll join us on
this episode of the Azure Enablement Show.
[MUSIC]
Welcome to the Azure Enablement Show.
We will be discussing the challenges you
and our other tech-savvy customers have encountered.
Together, we're going to be talking with experts to
find out how they think about these problems,
recommended tools, and best practices,
and tips they've learned from
years of experience that you could use.
I'm going to go follow up on
Canal's recommendation to talk to an expert on observability.
But I think it would be better if I did that as in customer modes.
Let me switch to that right now.
Better. Now I'm in customer mode.
I think it's time to go talk to Daria. Hey, Daria.
>> Hi, David.
>> Hi. I was wondering if you could help me
with stuff that's going on at work.
Let me just lay it out for you.
One of the problems that I was talking to Canal about was
our situation around our monitoring and observability is terrible.
By terrible, I mean
like I don't think I have a good idea what's working,
what's not working, what's running,
and not only that,
I'm getting paged all the time.
In the mornings, at night,
late at night, stuff like that. It's just terrible.
Can you help me understand what I
need to do to start thinking about doing this right?
>> Of course, David. Frequently when we think about observability,
we think about collecting many metrics,
have logs for everything,
and have a lot of traces.
But that's not enough because as we evolve and
build more services in different modes, for example,
we're now in micro-services world,
the concept of how we approach observability
is morphing into what problems we want to solve,
what questions we want to have answered.
We frequently do a lot of
mistakes which are common across different engineers.
For example, the over instrument.
We have so much data and we don't really know what to do with it.
Or we have too many dashboards.
I knew teams who have 200 dashboards,
and when problem happens,
they start looking, "What dashboard should we look at now?
Which one will answer our questions?"
Also, there is a lot of tribal knowledge and, very common,
not enough documentation and a lot
of knowledge which is inside people's head.
That's common things. Where would you start, David?
>> I guess I want to say that
the reason why I was smiling is
that it almost sounds exactly like our setup.
We've tried to do the thing where you monitor
all the things. We've done that.
Everything, we got monitors.
Let's talk about where we're monitoring here.
I think I'm doing the right thing.
I'm monitoring all our components.
Like if there's a problem with CPU,
I know about it instantly,
if our disks fill up. Isn't the right thing
to monitor all the things?
>> Well, it depends, David.
Of course, monitoring everything could be a solution,
but you first need to
understand that you are monitoring the right thing.
Where I would start this to understand how
your users are using your services.
What matters to them.
You actually can answer a question
about what's their experience in using
your service and not about
how many CPUs are experiencing high CPU every day.
Because if your service is architected in a way that it
fails over when specific component is not functioning,
your customer experience will be great and you don't even
need to alert your team in the middle of the night.
You can just deal with it as
a problem that you just solve when the time comes.
I would always start with questions.
What are different scenarios that
your customers care about when they use your service?
Also, I wouldn't postpone
instrumentation of your service until it's already in production.
I would start doing it when you are
building your services because this is the right time to
think about what are different modalities and
end-to-end journeys of service usage.
Think about those user scenarios,
but also think about your dependencies.
If you are building your service
depending on not reliable services,
this will cause you a lot of problems.
Make sure that you instrument to
measure your dependencies and reliabilities,
and not just measuring your service reliability.
Because frequently we depend on different components
and different services. Does it help?
>> Yeah. That helps. I just
want to figure out the dependency thing.
You're saying that it's important not for me
to just measure my own components, stuff like that.
Obviously, I have to measure from user scenarios,
but I also should be paying attention to the things I
depend on and monitoring those as well, right?
>> Absolutely, yes.
Because if you just monitor your service
and you'll know that your service is performing absolutely well,
but your dependencies are down and
your customers are calling you and
saying, "Your service is not working."
You look at your metrics and say,
"No, everything totally fine."
But your key dependencies are down.
So you obviously need to monitor for those,
and you need to make sure that you also architect
your service in a way that
those dependencies don't impact your customer experience.
If one of your dependencies is down,
you need to make sure that you have a good failover strategy.
>> Daria, I have another question.
It seems like there's different data I could be collecting.
What things should I collect,
and what are they good for?
>> Before you think of kind of data you will be collecting,
think about different usages.
For example, you will need to have alerting in place.
You need to assure that you are collecting
fast signals about your service health
and that you are alerting on customer experience
and not on underlying components that may not
even reflect in customer experience
and it's just a temporary glitch which is not impacting anything.
You would need to think about
your alerting scenarios because you want
to know about your problems fast.
Then you also will need to collect metrics,
logs, and traces to troubleshoot issues.
When you received an alert and you know something is going wrong,
you need to then troubleshoot to answer questions,
what exactly is going wrong?
In what stage of
multi-step customer experience you have a problem?
This is where logs and traces will help you,
obviously, and metrics as well.
Also, you will not only interact with
your metrics and dashboards and logs when problems happen.
You would also use them for proactive problem detection.
You may just look for patterns and
slice and dice your data by region,
by service version,
by component, by subscription.
You will be looking for patterns in
your collective metrics, logs,
and traces, trying to understand,
is there any problems,
is there anything you can optimize and make better?
Is there any problems that you
would do then by looking at this data?
Think about those three scenarios.
We talk about alerts,
which need to be fast,
and alert on what matters.
You've talked about troubleshooting,
which also need to be fast and need to point you to the problem,
and you need to do proactive problem management
to understand where you can optimize your service.
>> I get that. I understand from what you just
said where metrics and logs come into play.
Where do traces come into play really?
>> Well, they also come into play during
troubleshooting and also during problem management.
Because you need to understand how
your service is processing multi-step journey.
This is where traces are extremely
valuable because you can look at every step and you
can understand one specific user experience end to end,
and traces are great in that.
>> Great. I have one last question for you.
Given my situation, I understand
the user scenario part and understand what this is about.
Where should I start?
Where do I start getting myself out of the hole
we've dug ourselves in with monitoring the wrong things?
>> Well, I would start by going and talk to your customers.
Understand what matters to them.
Identify top journeys or
customer scenarios of what matters to your costumers.
Does it matter to them that your service
is fast, that it's successful?
It may not matter that your service is fast if
your customer is processing
transactions which shouldn't be in real-time.
It just matters that they process at some stage of the date.
In this case, latency probably wouldn't matter to you.
Or if you are also seeing this transaction but your customer need
to have currency that it will be successful,
then you will just monitor what matters to the customer.
You will monitor success rate,
you'll implement re-try logic,
and makes sure that transaction
processes in the end, and latency wouldn't matter.
But at the same time, if your customer is
running a service that is all about speed,
it need to be very fast,
then obviously you will not only care about success rate,
but you will also care about latency.
Talk to your customer,
understand what dimensions of reliability which matters to them,
and what are those scenarios.
Then you will look at your service, obviously,
don't switch off all your existent monitoring
because you need to have visibility into your service health.
But start instrumenting those scenarios
that customers indicated as critical for them,
and slowly, you will be able to change
your alerting strategy to the new
one which will alert on customer experience.
>> That's super helpful.
I think I got a lot of work to go do.
I think I want to thank you.
Let me switch back to host mode.
Back to host mode.
I want to thank you, Daria,
and I'd like to thank everybody for watching this.
I think that I learned a lot of stuff and Daria gave
me some good URLs for people to go look at,
to start reading about this. I hope you do that.
I just want to thank everybody for joining us.
I hope you'll join us for
more episodes of the Azure Enablement Show.
[MUSIC]