SPEAKER: Welcome to the second part in our series about
implementing AIOPS using the elastic stack.
Doing a quick recap.
We are collecting logs, metrics, and traces from 11
microservices running in a Kubernetes cluster.
We have set up that in our latest video.
So, if you're familiar with all that,
you might as well land in this video.
So we know we are going to to to start actually
using this data to produce value to us.
In this video, it's all about reducing the noise.
We need to reduce the noise in order to extract
relevant information from all this data.
And we are going to see how much data this infrastructure,
this fairly straightforward kind of simple infrastructure,
just 11 boards in there, 11 containers in there
that are communicating with each other.
We are going to see how much data this has produced in
the last three days now that we have been running this.
So, let's check it out.
Let's head over to the bottom.
And you can see it in Discover.
If you're following along, you will have APM,
you have logs, and you have metrics.
I'm showing the last seven days in here.
We have 36 million documents in the last four days, actually.
So, four days, 36 million load lines.
You can actually show the front end from the demo,
which it will be right here.
This is the site for the open telemetry demo.
It's a e-commerce website.
Allows you to show products, change quantity,
add to cart, pay, you know, change currency.
Change the currency.
And then place order.
So, all these interactions producing events,
obviously those events will be stored in here.
And we have the messages in there.
You see fetch cart.
That's one message from a service.
I'm just selecting one log line in there.
And you have the service called product catalog service.
And the cart service is telling to fetch cart.
Product catalog service it's just saying product found, ID something,
and then National Park Foundation Explorascope.
So, one way to do noise reduction if you were
to look at those individual messages,
they probably won't make a lot of sense to you.
But one place in which the elastic stack
really shines is in the search part,
obviously, it can make (UNKNOWN) searches.
So, I just took an ID, product ID in there.
This search will be done in those 36 million documents and it will find,
obviously in the message and in other types of logs.
You are seeing here a different type of log you see here to (UNKNOWN).
It's a raw message from GetQuoteRequest.
This is a quote service probably.
So, I have filtered the noise down 36 million to 255,000
in just a single node,
by the way, I'm just using the bare minimum cluster.
I wanted to start really, really small,
and that's what you can do as well
during your trial if you want to or you can in install it on premise.
But during your trial mainly you can just creating
single node and just go ahead with it.
See what you can do with it.
You can obviously create larger clusters,
but you can start as small as possible.
That's what I'm trying to say.
So, that's one thing.
That's a classic way.
You don't really need machine learning to that.
But we do have something in here back,
we do have something in here that might be novel to you,
even if you are already a user of the elastic stack for the logs,
which is field statistics.
In the background when I click that,
in the background what that the Elastic
search is doing is sort of aggregating
in the types of fields I have just two selected in there,
which is service name and message.
It's going to give me a little bit of information
about what different types of services they have,
what different types of messages, and all.
You see, I have 10 distinct values for service dot name.
I have 11 microservices.
Maybe one of this microservice is not producing logs,
for instance, because I have just 10.
And those 10 they will show up here probably.
The service name is calculated from OK.
So, calculated from 35,000 sample records.
Still should have 11 in there.
I just have 10.
So, I can see that product catalog service response for the majority
of the log messages in my infrastructure and front end 21%.
Now, right out of the bad again.
Just filter for front end in order to focus
the logs on just this part, for instance.
Just on the front end.
It will filter my dataset in there.
Those are the messages from the front end.
You're in there.
The front end it seems it's not really verbose because
there is actually a failed condition in there.
But that I'm going to use I'm going to inspect that in our third video,
actually, which is issue detection.
Here is just noise reduction.
So, I'm just reducing noise in here.
So, those are the first two things I can search on my data,
obviously, and I can do field statistics.
By the way, searching on my data what I can
do is that I have a hot tier right now.
All those log messages are arriving at hot nodes,
which are powerful nodes.
But I could have less powerful nodes even...
If I want to store the same data across those nodes.
So, for instance, I will store the first
seven days on hot nodes and then
I will just move this data to lower tiers like warm tier or a cold tier.
I'll have a full video dedicated to that.
But my search, my search experience is the same.
I don't need to restore anything from anywhere.
If I have, for instance, a frozen tier which will store this,
will move those logs over to a bucket and I trigger
a search that will have this data participated in,
Elasticsearch will automatically take those
logs from a snapshot in that bucket,
put back in Elasticsearch and respond to my
search and it'll cache that data for later.
So, we have a seamless search on all data tiers.
So, the last thing I wanted to show regarding
noise reduction is the following.
Let's head over to Machine Learning in here.
And right here we're going to see a section for the AIOPS.
Let's click Explain Log Rate Spikes.
And we are going to select our logs data view.
Not put the last 24 hours.
Put the last hour.
I can filter this even more because those are all the logs.
Those are the logs from our Kubernetes cluster and from
our services and from the (UNKNOWN) through them.
I'll just filter...
Or maybe the errors in APM.
Only two documents.
Not really.
Let's see last 24 hours.
Yeah.
And you'll see from all the errors that are being captured
by APM we have an interesting spike in here, 2,000 errors.
Let me just go even further.
48 hours.
Yeah.
Let's click that spike.
And what Elastic is going to do is to explain.
There are a couple of errors in there.
They are really, really small,
but Elastic is going to explain that error spike.
By correlating a couple of field names with
the errors like service.language.name
nodejs, that's probably the front end.
The path name is the front end.
They are highly correlated.
So, the impact is high.
This impact is whether if the correlation to
the message rates differences high or not.
I mean, in other words, what is the majority
of errors in here related to.
I have already refreshed.
So, (UNKNOWN) is complaining like warning me.
Like we have new data, so you might want to rerun your analysis.
But anyway, so this is related to the front end.
From all the errors in there, what I would have to do in order to
maybe infer that this was an error that was exclusive to nodejs.
So, from all the errors in there, you see that it's yellow in there.
That's the part related exclusively to the front end.
Then we have our noise reduction right there.
So, the second thing we can do, we can see here is log pattern analysis,
let's click there.
Again, select our logs.
From the last 48 hours we have 14 million documents.
Let me do, I mean, actually do the same search again.
And then the category field will be message.
So, from the messages that are displayed in there, the categories...
So, the most...
What Elastic search is going to do here is that
it is going to group the log messages.
So, we are seeing that it has a fail actually in the cart store.
It's a different service.
Maybe...
Yeah.
Yeah, maybe that Redis was down for a while and
the cart service couldn't access the card store.
You can actually check that in the service map,
which I believe I have not shown yet.
We have the service map in here in which we can
see how the services actually communicate.
So, you go to the observability part, and then the APM service map.
You see in here we have the cart service, which is a dot net service,
the front end will access the cart service.
Cart Services will access this Redis instance in there
and see the latency and throughput of all those services.
And it seems that at 9 am, our anomaly was showing at 9 am,
there was a problem in there and it's actually
failed transaction rate and 1.3% in there.
Interesting.
Speaking of failures, we are going to in the next video,
do an issue detection.
Now, I'm not seeing in the APM part any anomalies painted in there
because I have not activated machine learning for APM in there.
That's what we are going to do in the next video.
Hope you are enjoying it so far.
And see you next time.