>>Doctor, it's just not getting better.
The anxiety is still there, and I've enabled all the alerts.
I mean, all the alerts!
And we're talking, talking, talking, talking, talking,
but we're not communicating.
>>Patrick, I need you to disable all of your alerts.
>>All of the alerts?
No, no, no, no, those are my alerts, I need those alerts!
>>I'll give them back to you one at a time,
but only if they satisfy our criteria.
There are alert-worthy metrics
and there are trouble-shooting metrics.
>>Trouble shooting.
>>Google has published a series of four golden signals.
>>I like Google.
>>These four golden signals will be our criteria.
They are latency, errors, traffic, and saturation.
Only if your alerts satisfy these four criteria
can you have them back.
>>Well, okay, that could work.
>>Let's go to the clinic.
>>Is that the same rate?
>>Yeah.
So, welcome to the clinic.
>>Yeah, and you don't have diplomas all over the wall,
so it must be the clinic.
>>No diplomas, just helpful tips and tricks.
>>Okay, so we're talking about trouble-shooting metrics
and alert metrics, they're just not the same thing.
>>That's right.
We have a lot of metrics that we gather
in the SolarWinds products,
and some of them are alert worthy,
and some of them are just trouble-shooting worthy.
>>Do you find a lot of people end up
sort of focusing maybe more on the trouble-shooting metrics
because those are the ones that are connected
to default out-of-the-box alerts?
>>Yeah, let's take a look at a classic PerfStack setup
that we have.
As you can see, this one's full of CPU.
It's showing how my CPU is spiking up and down.
But in my opinion, this isn't where we need to focus.
What we need to focus on more
is the four golden metrics that I mentioned,
that Google writes about.
They are latency, errors, saturation, and traffic.
And to me, number one is latency.
But here in PerfStack, we're seeing somebody
who has set up a PerfStack chart that is a lot of CPUs.
They're watching their CPU, they're concerned about CPU.
>>And they've put that up here at the top,
and I'm wondering if really maybe that goes down lower.
Maybe that the sort of tracking metric
that a user care about should actually be
at the top of that stack.
>>Exactly.
Here we have latency down at the bottom of the PerfStack,
when in fact, I'd put it on top.
Because if my latency's fine,
and everything is going smoothly,
I don't really care about CPU.
Now if there's a spike in my latency,
then I'm gonna start looking to my trouble-shooting metrics
to see what's going on.
But until that latency spikes up, I don't really care.
>>Well, and the thing that's interesting here too is,
I mean, you've got some big sustained spikes up here
for CPU utilization.
But if I come down here, I look at the latency,
the actual users that are hitting it, it's fine.
This is successful to me, right,
because this means I'm fully utilizing my infrastructure
but everybody's happy with it.
>>Right, and that's why we buy infrastructure, right.
We buy CPUs to use it, we buy memory to use it.
So if those things are going up high, that's okay.
So as long as my application is responding quickly,
I should be fine with everything else being utilized.
>>It's part of Google's awareness of this that
you almost get free resources on Prim.
Maybe it's CapEx, you get what you get.
But when you go to the cloud,
all of a sudden you're paying, you know, either per opp,
or for the amount of memory or something else.
So are they just thinking more
in terms of that full utilization?
So they're looking for a metric
that actually is attached to what people are experiencing?
>>Exactly, that's what the cloud technology
is moving us towards, right,
is I wanna only be paying for or using what I need.
>>Right.
>>So, when you move to the cloud
or similar types of technologies,
you only pay for how much CPU am I using?
So if I'm not using CPU, why should I pay for it?
If I'm not using memory, why should I pay for it?
I don't need to buy these giant servers
if I'm not gonna fully utilize all of the resources on them.
So, in that sense, you don't really need to have
the infrastructure monitored so much as
the actual application that you're running it.
So again, points back to latency.
You know, how well is my application running
from the user's perspective?
And that's a lot of what the features
that we're writing at SolarWinds are geared towards.
Let's move over to WPM.
WPM is a great example of a product
that's focused on the user experience.
You're taking the WPM monitors,
and you're monitoring the actual HTTP traffic
so that we're able to gauge
what the user experience actually is.
>>Okay, so for cloud users
what are they looking at?
>>So cloud users, we should take a look
at the RUM feature on Pingdom.
The RUM feature's really great
because it's actually just a little piece of code
that you're putting on your website,
that's actually measuring the real user experience.
>>So, where there's a JavaScript inside the browser
and it's what a human being is experiencing.
>>Right, so now you know,
what are my users actually experiencing?
Are they seeing a lot of delay?
Are they having a great experience?
As long as your users are having a great experience
on your website, I don't care if you're using CPU,
I don't care if you're using memory,
that's what they're there for.
>>And then that area of all those sustained peaks
in the CPU that normally would be lighting up my dashboard
and setting off my phone that,
oh, we've exceeded a threshold.
They're fine!
This is the threshold you care about.
>>Exactly.
>>Okay, so that's latency.
What about errors?
>>Errors is my number two highest
because fast and wrong is not good.
So, we're monitoring our latency,
but we also wanna make sure that
we are monitoring our error rates as well.
>>They're not all co-equal set-off an alert
for every single one, or just throw 'em all in a folder?
>>No, no, not at all.
So, that's why in the Orion Suite of products,
we have the Log Viewer, or the new Log Analyzer.
That allows us to track
the rates of errors within our system.
So here we have a great example of a system
where we're tracking the number of errors
as it's coming through the log system.
And as we track these rates of errors,
we're actually able to watch at different spikes of errors.
So, here on this example, we have a certain spike of errors,
that's more concerning to me.
I expect errors all the time through my system,
because that's just a normal part of software,
but spikes in errors, that's a different story.
>>'Cause that means something's changing.
>>Exactly, something changed in my environment
so that the error rate is different.
So, Log Viewer inside of Orion.
We also have offer Loggly and Papertrail
for those of you on the cloud.
These products are more at tracking your logs.
And when you've got your logs coming through,
that's a perfect opportunity
to start tracking your error rates,
because logs have keywords like exception, error,
other things like that, that really allow us
to focus in and track those rates.
>>So it's really about turning errors into another metric,
a derived metric.
>>Right, and once we have that additional metric,
we're able to then bring it up to the top.
We're there to watch it.
So we're watching latency,
then we're watching our error rates to make sure that,
hey, not only are we being fast, but were fast and correct.
>>So alert on the rate, not every error.
>>Right, exactly.
You don't wanna look at every single error
because you'll never have enough room in your inbox
to deal with the amount of errors
that come through software logging.
>>Okay, so that's latency and errors.
So what are the last two?
>>Last two are traffic and saturation.
And let's take a look at them together.
Usually when we think of traffic,
we think immediately of network traffic.
And that's important,
but that's not really what we need to look at here,
we're more looking at the application traffic
and the application saturation.
>>Right, we make the joke, it's not the network,
but generally the network's pretty stable,
or at least that ends up being something
that changes a lot less than application definition does.
>>Right, because our networks are generally
highly available, redundant,
they've got plenty of bandwidth,
and that's not gonna be our problem.
What we're gonna be looking at is
the saturation of the traffic
that's going to our application.
>>Okay.
>>So, let's take a look at our classic networking screen.
Here we have the network saturation.
And this is important, but this is where we need to start
blending between what's report worthy
and what's actually alert worthy.
So as I'm looking at traffic and saturation,
I might think of things like network traffic,
and volume traffic, or volume saturation I should say,
and those sorts of things,
but those are things that I would run our reports on.
'Cause I wanna look at my overall trends to see,
am I going to be saturated on my physical infrastructure?
But that's not something that I necessarily need to be
woken up in the middle of the night for.
>>So, you're gonna still correlate that
with latency and errors,
but still not automatically add alerts.
>>Right, exactly.
So, if we take a look at our application monitor for IIS,
here we're looking at the actual saturation of,
am I serving up enough pages per second?
Am I serving up enough requests per second?
This is the saturation,
this is the traffic that I'm gonna alert on.
If I get to a moment where I know that on a normal day,
I can serve up this many pages per second,
but then I start to spike above that,
that's a saturation problem.
Which usually ultimately ends up showing up
as either errors or latency issues in the long run,
but these will be earlier tip-offs
that I'm going to be running into a problem.
So, I'll actually alert on
the traffic and saturation rates of my websites.
>>That's really cool.
And so again, this is Server and Application Monitor,
and I guess this view is app insight for IIS.
And that would be, if you were on Prim,
or maybe you were monitoring an Asure instance
with a whole lot of BMs.
If you're looking at this more from
a cloud native perspective,
you're probably looking at it in AppOptics.
So again here, you have the same metrics, right,
so we're looking at response time, requests per second,
and the rest of it, but if you are then correlating that
with the infrastructure metrics,
then that's gonna show up in the host metrics individually.
>>Right, and I love the first screen that you showed,
because this is something that we learn from our own SREs,
from our own DevOps team, is that latency is up on top.
We're watching that latency, number one,
and if that latency has a problem,
then we start to look at the bottom of our pages
where we have our error rates, our traffic and saturation,
but latency is still king on this page.
And so what we've learned
from monitoring our own infrastructure,
is that we need to watch that,
and when that latency goes up,
then the other ones are where we need to start looking next.
This is also strongly correlated to something
if you're running your own private internal cloud.
If you have your own infrastructure, you know,
you're not watching the VMs individually that spike,
but the overall saturation
of that cluster of virtualization.
Am I looking at, you know, am I running out of memory
or am I running out of CPU, as a whole?
And do I need to then add something onto my cluster?
And that's where the cloud's flexibility
really comes in again, is like,
I can just dynamically add that on.
And a non-premise cloud,
I might need to look at my overall traffic and saturation
to then get to the time to add on
an individual server into the cluster.
>>Don't let CPU metrics drive alerts
that then drive dynamic provisioning.
>>Exactly.
>>I really like where you're going with that.
And I don't know if it's part of the SRE handbook or not,
but that idea of make your actual alertable values,
put those on top,
and then push all the trouble-shooting metrics,
like CPU and some of the other DTL
that we incorrectly put alerts on,
below that just as a reminder.
I think a lot of people are already doing that
with PerfStack, where, you know,
here's a basic PerfStack view for a web application.
So, a lot of times what they'll do
is come in and start with something like a latency value.
Like, let's take this Average Response Time,
and we'll put this up here on top,
so that top line metric again.
This would be something I would alert on
to say users are being affected by this,
I wanna be able to do something.
But then you wanna add a trouble-shooting metric
that may be related to that
as part of that trouble shooting.
So in this case, like wait time
is gonna be probably part of that,
if this is bound by database wait time,
so maybe I wanna take a look at database wait time.
Now I could say if any of it goes over a certain amount,
I wanna create an alert, but that would be a bad habit,
you're saying.
>>Right.
>>So instead, I wanna put that below,
below my top line, so maybe I'm really interested in that.
But it's a visual reminder that
the thing that I should be alerting on
is that latency time at the top,
not the details that are specific to that data.
>>It's really great because
now I can focus on the top,
and when the top's okay, the noise below is just noise,
and I don't need to pay attention to it,
until I see an actual problem at the top.
>>Right, and when we get into events, for example,
in Database Performance Analyzer, you were saying,
then think about using machine learning
as a way of making that a little bit easier.
>>Right, exactly, and that was the key reason
that we added anomaly detection to DPA
was to give an easy visual indication
so that you're not only just seeing your problems,
but you're seeing them highlighted red
when there's an anomaly.
>>But we don't wanna generate alerts in advance
based on a predictive issue, or do you?
>>Not necessarily,
because what we really are concerned about
is the performance of our application.
So if there's anomalies within my database,
but again, my application is behaving fine,
not a big deal.
Sometimes anomalies happen,
and they might be bad, or they might not be.
But as long as my application is behaving, I'm fine.
>>Well, Karlo, thanks so much for the clinic.
This is great!
>>No problem.
Let's head back to the couch for one last joke.
So, Patrick, how are you feeling
about your new alerting and reporting setup?
>>You know, I feel free, liberated!
I can look at my email box without shame!
>>Excellent!
We have made wonderful progress today.
Now, can we finally start talking about
those voices in your head?
>>[sighs] Logs.