[upbeat electronic music]
>>So we're talking a little bit about the history
of the application monitoring products.
So if we go actually back a long ways,
we've had Pingdom for a while,
which is end user monitoring.
So it's kind of from the outside
looking into your application,
really from the end users' perspectives.
But then we also purchased a product called Librato,
which is infrastructure monitoring and custom metrics
and also kinda some cloud monitoring,
things like platform-as-a-service monitoring
in AWS and Azure.
Then we also bought a product called TraceView,
which gives you code-level visibility
into your applications.
And over time, TraceView and Librato became AppOptics,
so trying to kind of make it easier
for the users to get observability
into both their infrastructure and their applications.
And in this session, we're gonna talk about kind of
what we've done since then and where we're going.
So really what we're trying to achieve
with our different SAS products
and what our users are asking for is observability.
So what is observability?
Really, in a nutshell, it's being able to observe
or see into the different parts of your systems.
So if you think about modern web application today,
where user makes a request, it goes from the browser,
across the network, into a web tier application,
then maybe into something like Kubernetes
and so on and so forth,
it's important to be able to see
into all those different components,
the network, the application,
the infrastructure lying underneath,
so when something does go wrong,
you can understand what it is,
who's responsible for fixing it,
and get to resolution as fast as you can.
So now we're actually gonna do a bit of a demo,
show something new that we've done
to kinda help increase observability
and make these troubleshooting flows easier.
When we talked about the history of the applications,
we've also brought, we talked about bringing TraceView
and AppOptics together.
We've also brought now together
what was AppOptics and Loggly.
So a lot of people use infrastructure metrics,
application metrics, and logs
to get the observability into their applications
to really understand what's going wrong.
So in the screen that I'm showing you here,
on the left-hand side, we have things like dashboards.
We have services, which is the APM data from AppOptics.
We have infrastructure metrics,
so insight into containers
and the hosts that are running them.
And now we have logs as well.
So right in this kind of integrated experience
that we've brought these products together.
We've got logs, metrics,
infrastructure monitoring, cloud monitoring,
and applications all in one.
On the theme of observability
and kind of being able to understand what's wrong,
we've also created this new homepage.
What this homepage does and is really designed for
is helping you understand,
is there something broken right now?
And if there is, what's broken,
and who should I get to fix it?
So if you look at the screen here,
we've got top services by response time trend.
What this showing us is which services are slower right now
than they were a week ago.
And jumping out at us here
is we've got our web tier service.
It's running about 56% slower compared to a week ago.
We can also take a look at our services by error rate.
So the web tier is 97% slower than it was, sorry,
the error rate is 97% higher than it was a week ago,
basically twice as many errors as there was.
And you can also look at request volume.
There's something different there.
Or you're running some sort of sale on an e-commerce site.
Suddenly your traffic is exploding.
And if you come on the log side,
so we can see that, hey, there's something different.
Are we seeing increased log volume
or errors or anything like that
from our services or our hosts?
In this homepage that we're looking at here,
pretty clear we've got the web tier jumping out
in pretty much all the locations,
lower this week, and it's got more errors this week.
It's spewing out more log messages.
And kinda lastly on this page,
we're also calling out some of our infrastructure
that's running hot.
We can see that there's this one host, aodemo1,
that's one running really high CPU.
It's basically pinged about 99%.
So if we wanna troubleshoot this more,
it's pretty clear that there's something wrong
with the web tier.
It's certainly worse than it was a week ago.
I can click into the web tier
really in any of these locations,
but we're gonna take a look at the latency problem first.
Why is it slower this week?
So we clicked into the web tier,
and it gives us a pretty good summary of what's going on.
It's showing us the last six hours here.
Saying, hey, response time is about 56% higher
compared to a week ago.
And most of the time is being spent
waiting on remote calls to the booking service.
Kind of going back to that observability trend,
being able to understand what's going on
in the application, what might be the problem,
where might be the,
who should we get involved to fix this?
It kind of spelled it out for us, right?
We know the response time is higher in the web tier.
And we know roughly that it's being caused by
actually a downstream service called the booking service.
If we want to dig into this issue further
and really understand, okay,
what's going on in the booking service,
why is this truly slow?
All we have to do,
click on this View Traced Requests button here,
and this takes us to essentially a scatterplot
or a heat map of requests.
So if we wanna understand what's slow
during this big spike here
when users were having a poor experience,
and they were waiting for just 20 seconds
for a website to load,
we can actually just click and drag a box,
and this will filter down to those requests.
Then I can just click on one of these.
I can look at a request that took about 22 seconds.
And then when we open that up,
we're gonna get more information about why this was slow.
So this request took about 22 seconds.
We can see that 80% of the time was being spent
waiting on remote calls to the booking service.
And if we actually wanna see, okay, I get it,
it was waiting, spending a lot of time
making calls to the booking service.
If we wanna see why the booking service was slow,
we can just click once more.
So now we're kind of a couple layers deep, right?
We started out on a homepage,
and the web tier was slower this week than last week.
We clicked into the web tier.
We found a trace that said,
hey, most of the time is being actually spent
waiting on the booking service.
We clicked into one of the calls to the booking service,
and it was, of course, the database, right?
Most of the time was being spent
making these MongoDB queries.
We can see that, hey, there was about 500 of them,
over 500 of them in total.
We're basically getting some sort of N+1 query loop
that was inefficient.
So we went really high level down to the problem
with a specific query loop pretty quickly.
The next thing we can do,
you kinda wanna see this in a little bit more context.
We can look at the details,
see kind the control flow of how we went from web tier
down to a database calls.
And talking about the integration that we just built,
if we will also want to look at the logs
from this transaction, kind of increase our observability,
we've got a direct integration to what was Loggly,
now in this same interface.
And we can actually just click Search Logs.
We're still within this same APM integrated experience.
We've still got access to our services,
infrastructure, and logs.
All right, and what we've done actually here
is instead of going to a logging product and guessing
which logs might be related to this same request,
we can actually just click on that Search Logs button.
And these are the exact log messages
from that request that took 22 seconds.
So really, we're trying to kind of
make these workflows easier for our users,
basically the community,
and hopefully the people watching this session
just by kind of reducing the number of clicks,
surfacing this information,
and making it kind of easier to get to that root cause.
If you think about logs, metrics, and traces,
they're kind of just different perspectives
to the same application, or in this case, the same problem.
So we're making those workflows easier
by bringing that data together.
One of the things that you might be thinking,
I did mention at the beginning of this call
that Pingdom has the end user perspectives.
Pingdom is still part of our SAS portfolio
and our APM suite.
One of the next things we're working on
is trying to bring that into this same experience
to kinda give the full end-to-end visibility
from end user, your uptime checks, synthetic transactions,
and connecting it to the applications,
the traces, the metrics, and the logs.
And if you are a data monitoring engineer
or, let's say, an infrastructure engineer,
and you're not really responsible for these applications,
infrastructure is really kind of your domain,
and that's where you live,
building on this topic of observability
and seeing how are these things connected?
And if there's a problem in the infrastructure,
how does that relate back to an application?
We go to the infrastructure link on the left-hand side.
We can actually look at all of our infrastructure
that we're pulling, either from EC2
or from running an agent.
And we can actually group our infrastructure by service.
So if you remember, at the beginning of this call
when we were looking at the homepage where it said,
hey, you know what, the web tier was slow,
and there was one host called aodemo1
that had really high CPU,
when we group our infrastructure by service,
we can see that, hey, you know what?
There's one host running our web tier service.
Actually, right now it's got 5% CPU,
which isn't that bad, memory utilization at 42%,
pretty healthy metrics.
We can actually click on this
and see what else this host is doing.
So we can see, when we look back at six hours,
when we had that issue and we had our latency was higher,
we can see that, hey, you know what?
The CPU was spiked.
Our user CPU was through the roof.
System CPU basically took it to 100.
And actually, we can look at all the services
that are running on this host.
In this example, this is basically essentially a single host
running a bunch of Docker containers and services.
You might be running something like Kubernetes,
where there's a one-to-one relationship.
But in this case, we could see that,
hey, when the CPU was spiked,
we can see its effect on different applications, right?
We can see our web tier spiked up.
We can see, we also have,
we're running the booking service on here.
The booking service latency went up to 500 milliseconds.
It kind of gives us that context to understand,
hey, if I respond to an alert on just the infrastructure,
out of memory, disk errors with CPU,
how is that really kind of affecting the end user?
What's the context of that?
If I'm an SRE, or if I'm an infrastructure engineer,
I get an alert at two in the morning,
and it says CPU's at 100%, if I come in and see,
hey, this is just running some analytics service,
I really don't care if it's slowing it down right now.
But if I come in and see maybe we have customers
in a different time zone,
and it's really slowing down our web tier,
maybe causing errors or causing an issue,
that information can really give me the context of,
hey, can I snooze the alert and go back to sleep,
or is this something that I need to solve right now?
So we've got just a couple minutes left,
so we'll open it up for Q&A.
Let's take a look at the questions.
First, if you do have any questions,
put them in the chat, and we can answer them.
And if you're wondering kind of
how to get started with this,
if you wanna play around with any of these products,
you can go to appoptics.com,
sign up for a free trial, play around with it.
And also, if you have any feedback on this stuff,
I mean, the community feedback
is what drives these products.
It's what makes us make decisions to put them together.
If you ever wanna talk to the product team,
we're always willing to chat with you.
So reach out in Slack.
Both AppOptics and Loggly and Pingdom
are all in Slack right now.
But reach out to us anytime,
and we're happy to speak with you
either on a call or just in the forums.