SPEAKER: Hello, everyone.
Thank you for your interest in understanding tools that
help you identify root cause of system issues.
My name is Bernhard Suhm and I work in product
marketing for the Elastic Platform.
I've explained the concept of root cause analysis in a different video.
Follow the link below if you need to catch up.
This video will focus on applying root cause analysis to observability.
In other words, applying machine learning power tools to IT operations.
And that's become known as AIOps.
ELK says you can follow along the demos,
use the sample logs and APM data that's
available under Try sample data on
the home screen in your elastic cluster.
Let's get started.
But before we can get to the root cause,
you need to find the problems in your actual data.
ELK says anomaly detection to the rescue.
And the very first step is getting your log metric or
trace data from your applications loaded into elastic
and elastic agent is the perfect tool to orchestrate the data ingestion,
and it also supports open telemetry.
In our first clip today, my colleague Jeff Vestal demonstrates
elastic anomaly detection in the context of log
stream data and how you can start exploring the nature
of the issues with the Anomaly Explorer in Kibana.
JEFF VESTAL: So here I am under observability and the alert section
and you can see we have one alert that says there were 20,000
log entries in the last five minutes,
which was well above our threshold of 75.
So let's investigate.
When I click on View App,
it takes me over to the log stream since this is
a log source and from the log stream
you could do our fulltext searching.
You could scroll through here and try to kind
of hunt and peck and see what was going on.
But instead let's take advantage of elastic machine learning.
So when I click on Anomalies,
you're gonna see a chart here with lots of colors on it.
The anomaly detection engine is looking for unusual rates in your logs.
So here we have a range from 0 to 100.
Zero is everything normal 100 in your logs are very unusual spikes.
The red indicates the unusual areas here.
And while you can see there's a lot of activity in this area of time,
the first one that was having unusual amount of logs is Postgres.
So let's take a look at this.
I'm going to open this and go into the Anomaly Explorer.
Now from here you can get in-depth
and look at all the specific anomalies,
not only for this one, but in this time range.
SPEAKER: Alright.
So now we have pinpointed some possible issues in your data.
It's time to investigate and work towards understanding
the underlying issues and work towards the root cause.
In the next clip, Jeff continues,
how elastic helps you explain spikes in lock rates if your system
suddenly produces a lot more log entries than normal,
that may be indication something is going on.
JEFF VESTAL: Let's take advantage of Elastic AIOps' capabilities
and click on explain log rate spikes.
So this is a log source and I'm still in the time
range since that's where we saw this bike.
And sure enough, you can see the spike in my chart right here.
So all I need to do is click on one of these.
I can actually move it over to kind of narrow down the deviation.
So elastic it's gonna do, it's going to baseline
the key-value pairs that are in your log
messages and then identify which key-value
pairs contributed most to the deviation.
And when we look through here, we can see, sure enough, it's a database.
It is Postgres just like the anomaly detection identified.
We can see the docker container that it's running in Postgres.
So if you had a large deployment with lots of Kubernetes,
pods, and Docker containers,
you could identify perhaps if it's just one particular
pod or container that's contributing to it here.
So this is great.
We're able to confirm that it is Postgres.
It is Docker and some information about it.
But we also have a large number of log messages that we didn't
parse out ahead of time and are just full-text strings.
Let's see if we can find a little more detail about what's in those.
We click on logs again and we still see the similar spike.
What I wanna do here is tell it which field to identify to process.
We're gonna use the message field,
which again is a fully on structured message field.
And we can see that the Postgres logs are coming in.
So here's select Star from PG Stats database.
It's updating.
We have some drop tables with PG bench inserting into PG bench.
So it looks like perhaps a DBA ran some postgres benchmarking
tools to see what was going on Postgres.
So we should at this point go and talk to
them and see was this an intentional run?
What do they know about it?
Maybe there's a better time.
They can schedule it.
SPEAKER: We'll stay for a moment with analyzing log data.
Analyzing those messages can become difficult if there is many,
many different messages and you don't see the trees for the forest.
Here, our log categorization can help.
SPEAKER: Elastic uses the powerful categorized
text aggregation to accomplish this,
and it really makes exploring mountains of logs much easier.
SPEAKER: This time my colleague Hugo from the Observability team walks
you through applying what is called log categorization.
HUGO: From the messages that are displayed in there,
the categories so the most what Elasticsearch is going to do
here is that it is going to group the log messages, right?
So we are seeing that it has a fail actually in the card store.
It's a different service.
Maybe that's that redis was down for a while, right?
And the card service couldn't access.
SPEAKER: Let's switch to analyzing a different type of data
that's frequent in observability APM data.
The visualization of the various services with an occasion
which ones had anomalous behavior and triggered
alerts versus which ones were operating normally
may lead you towards inferring what's going on.
So in the next clip, we show our APM service map how
that visualization helps you accomplish this.
HUGO: You can actually check that.
In the service in the service map, which I believe I have not shown yet.
We have the service map in here in which we can
see how the service is actually communicate.
So we go over to the observability part and then APM service map.
You see in here we have the cart service, which is a dot net service.
The front end will access the cart service.
Ccart services will access this Redis instance in there you
can see the latency and throughput of all those services.
And it seems that at 9 am like that,
our anomaly was showing at 9;00 am there was
a problem in there and it's actually fake
transaction rating and 1.3% in there.
Interesting.
Let's dig deeper into analyzing ATM data for Root causes.
The next clip demonstrates our APM latency correlation feature.
The idea is you correlate observed latencies
or failures with different fields in
the highly correlated ones may indicate they
have something to do with the problem.
So that's how this helps you get to a root cause.
So this is the latency distribution of everything,
all the traces that were captured for the front end.
We can correlate latency.
See?
What fields are highly correlated with
the traces that are or with the transactions
that are taking longer or with the transactions that are failing.
If I click failed transactions for failed transaction correlations,
it will find correlations among all my transactions.
Those are the ones that are failing, right?
And there is a high correlation with that specific
product OLJ maybe might have the name in there.
But we have that product in here, right?
Yeah.
So the failed transactions at the frontend right so.
In the beginning of the transaction, right?
The transactions at the frontend are all related to this, right?
Alright.
I hope you found these demos useful and they get you started applying
it to your own data and accelerate finding root cause in your system,
monitoring, and observability use cases.
To learn more, check out our AIOps page at Elastic
Observability Aiops you'll find useful
how to and getting-started videos in our YouTube
channel in the observability section.
You can also engage with our community at discuss dot elastico
and if you need access to elastic, sign up for a free 14-day trial.
ELK says don't take my word that you should consider elastic.
Renowned analysts like Gartner and Forrester have confirmed
elastic leadership for the observability space.
SPEAKER: Companies like Dish Media are realizing ROI
by applying our machine learning and tools,
accelerating root cause analysis, and thus also time to resolution.
Thank you for your attention.
Have a good day.