The Hybrid Cloud observability platform records tens of thousands if not hundreds of thousands of
metrics all across our infrastructure, and what's nice about that is it means you can troubleshoot
things. The problem is, it's kind of tough to figure out what you want to troubleshoot when you're
dealing with hundreds of thousands of metrics. Well that's where performance analysis or what we call
"PerfStack" comes in, and it's pretty easy to build one of these, so let's go ahead and get started.
start with my dashboards home performance analysis -
someone's complaining about a web application performance problem so
let's add our web server. Go ahead and go to add entities,
and I know the server name, so I'll search for West Web we're having problem number one today
we'll expand that out I'm going to select West Web. Now what I like to do is to pretty much eliminate
the server as a problem and for me that's normally the very basics, CPU and memory. So let's look at
those. We'll drag average CPU memory, we'll drag average CPU load, and average percent memory
and since they're both percentages we can dump them on the same graph.
All right, doesn't look like anything here is out of the ordinary - in fact, the CPU load is incredibly
low, so there's a problem somewhere else. Now I can hunt this down myself or I can let the system tell
me what's connected to it. So come over here to West Web 01 and I'll click add other entities
and this is where we can start eliminating some
things. So let's bring over the IAS application first
we have a breakdown of all the different web
services so I'm just going to add a problem with this
how much memory is being used, how much CPU is being used, also doesn't look like a problem and
we're also seeing the green checkbox over here, so probably not the problem but it's also connected
to SQL Server so let's go ahead and look at our SQL - whoa - we've got some problems there.
So let's look at the SQL Server
bring in a couple performance counters we can bring in our disk iOPs
read and write how many pages we're going through per second, and our page file usage,
okay, there are some jumps here, so that could be a problem - but maybe it is something else related to
the SQL server and we need more details on that. We can then click on show related entities yet again
so I want to see what's actually affecting this -
and because we know about all the storage behind that, here's where that information
comes in. So we can look at the virtual machine, looks fine - we can look at the host - also looks
fine, the LUN - well there's a problem there - if you look at the pool, storage array, database, but
I want to go even deeper on this so I'm going to go ahead and look at the database instance.
So here's the instance blocking locking dead locks all that stuff but what I really like is
database wait time, because if we're having a Slowdown not a problem it might be some
type of problem in our actual SQL. So what I like to do is I grab total instance wait time
and because I want that to be first and foremost I'll just drop it at the top.
So now if I grab this little segment surrounding our Peaks
I can see the actual SQL that's executing. Wait times 255 seconds, taking up 10% of the time.
This is probably our problem. This is causing some type of bottleneck and chewing up
10% of our entire wait time. So you can see with Performance Analysis, I was able to go
from the node to the application, to the actual database, and all the way down to
the code level to see what could be causing a problem. This is just another way that you can
help diagnose problems with the Hybrid Cloud Observability platform.