SPEAKER: Time to talk about root cause analysis.
Can we find the source, the origin of
an issue in our systems by using AIOPS,
by leveraging machine learning?
We certainly can.
And I have not chosen the open telemetry demo just
because it would save me development time.
We have 11 microservices there.
But we also have something else.
You might have noticed if you are following
along and you are deploying that in
your own infrastructure so you can try
it out and I really hope you do that.
You might have noticed that along with the UI,
when you apply the helm charts,
you get another thing that's called feature Flags UI.
And that allows you to actually trigger a problem
when fetching a specific product in that store.
But the documentation does not say what problem is that and actually,
the UI doesn't say what product is this.
We have to find it.
So, let's go to our terminal.
So, when you apply by using helm install or helm upgrade,
when you apply the values, you will get that message.
Yeah.
You will get Feature Flags UI.
Let's port forward and then access the Feature Flag UI.
Good.
And then we have productCatalogFailure, fail products
catalog service on a specific product.
It doesn't tell you what product it is.
Let's just enable this, let it fail for, I don't know,
maybe 10 minutes and I'll go back.
Why would we do that?
Let's go back to our APM.
Last 30 minutes and you will see our feature flag service in here.
It will start triggering.
And that failure most likely, feature flag service here.
So, that failure, failure in that service will propagate
hopefully up to the load generator actually
because the load generator is kind of interacting with our systems,
with our microservices.
And if feature flag fails,
then hopefully other services will fail as well.
I'll let this run.
Well, actually, failed transaction rate is
already so it's already failing actually.
This might be 0.2% in failed transaction rate right here.
Let's see.
So, root cause analysis is all about finding the source,
the root of the problem.
And in our APM, we do have distributed tracing,
and we can see that this is spanning across several services.
So, front end and cart service.
So, that's one interaction that is coming that is
passing through several of those microservices.
And then you will see, hopefully, yeah, feature flag service.
So, this front end is accusing a failure in there and product catalog
service that we know we have triggered the failure in there.
It's failing.
It's already telling me what product is failing.
But can I know for sure that the failing transactions
are all related to this product?
We can find that out with machine learning.
If we go over.
So, this is the latency distribution of everything all
the traces that were captured for the front end.
We can correlate the latency, see what fields are
highly correlated with the traces that are
or with the transactions that are taking longer
or with the transactions that are failing.
If I click failed transaction correlations, it will find correlations.
Among all my transactions, those are the ones that are failing.
And there is a high correlation with that specific product, OLJ.
Maybe might have the name in there.
But we have that product in here.
So, the failed transactions at the from end.
So, in the beginning of the transaction,
the failed transactions at the front end are all related
to this or for the most part related to this.
26% of the transactions right now,
they might be all failing at this precise minute.
But since we are looking at less 30 minutes,
so we have 26% of failure in there.
And that's because I know that already, that's precisely the product...
Open telemetry demo.
If you go to the documentation of the feature flag service,
if I'm not mistaken, let's see, what is it?
Oh, here, yeah, docs.
So, feature flag service.
It's clicking the wrong thing.
Feature flag service.
Still not this one.
Scenarios.
Oh, it was right here.
It was right here.
So, generate a product catalog error,
for GetProduct (UNKNOWN) for that specific product OLJ.
So, we can successfully, we could successfully
not doing anything special correlate
the failed transactions in the front end with that specific product.
And we have in the errors, we have it in there.
Oh, that's just the group and failed to get product,
that specific product right there.
So, yeah.
That error might be picked up by our anomaly detection as well.
If the failed transaction rate is raising,
failed transaction rate, it might get.
Let's just compare with the expected bounce.
So, my machine learning job, it will be picking up chunks of that data.
It might be a little bit lagging behind but it will pick it up.
It was a short video for root cause analysis.
We have shown issue detection and root cause analysis as
the main applications of machine learning for AIOPS at least.
And then the next video, we'll talk about how we
can actually act on top of those anomalies.
And I'll let this run to see if that failure,
how that failure will appear in an anomaly,
and then we can trigger an alert?
So, see you next time.