- Hello, everyone.
I'd like you to welcome you all
to this session on
Observability with Amazon OpenSearch Service.
My name is Muthu Pitchaimani,
I'm a specialist solution architect focusing exclusively
on Amazon OpenSearch Service
and I help customers build solution with the service.
This is agenda for today's session.
I'm gonna be discussing about what operational analytics is
and what observability is and where do they fit in.
And I'm gonna discuss some few use cases for observability
and what all the data or pillars of observability is.
And finally, I'm gonna describe some components
of Amazon OpenSearch Service that helps you to
accomplish observability.
Finally, there is a demo
that I'm gonna show you that shows you how
you can bring all this, put all these pieces together
and how you can use it to gain insight for your data.
What is operational analytics?
It is not a new field
it is a field that evolved from operation research
from industrial manufacturing three or four decades ago.
The aim of operation research
was to improve the business efficiency
and similarly the aim of operational analytics
which applies to-- which means
that applying operational research to modern day
computing systems is to improve business efficiency
under customer experience and value.
Traditionally, operational analytics has been done
using batch systems where you process a large volume
of data and then get some insight
from it that helps you to make some business decision.
The aim of operational analytics
is to remove this manual process
and to use missions in the process
so they can make the decisions faster and also in some way
shape or form in an automated fashion.
Operational analytics falls in three distinct categories,
the first of which is what is called
descriptive analytics.
When you look at a complex system, you should--
you need to be able to describe what the system is
in first place so that you
can understand and characterize behavior.
All the activities that you do is covered
under descriptive analytics.
Next is called the predictive analytics.
So once you know what the system is
and you understand its behavior
then you can predict how a system is going to behave.
All the activities that you do
comes into predictive analytics.
Finally, what is called a prescriptive analytics means
that once you know what a system is
and how it behaves, and you can predict what it is,
if some issue were to happen
then you can be prescriptive about a solution
either in an manual fashion or you can even
use the machine to take some automated decision.
That's what all the activity that you do there
is called prescriptive analytics.
So where does observability fit in?
It fits into the category of this descriptive analytics
that helps you to understand the behavior
of a complex system using some data that you gather
from the system itself.
So what is observability?
Observability, as I mentioned before
just like operational analytics, is a measure
of how well you can understand the behavior
of a system or the internal states of a system
from the knowledge of its external outputs
without knowing how the system itself behaves.
So observability is just to, is a measure
of how well you can infer the state
so that you can predict its behavior and finally
be prescriptive if any issues were to happen.
So what are some of the use cases for observability?
The first and the foremost use cases
in the field of microservices and container as
monolithic applications gets broken down into microservices,
you need to be able to know what is happening
within the system.
Or for example, you want to know how we request (indistinct)
to the entire system under (indistinct) service.
So that's one emerging use case where observability is used.
The next is what we call, like to call it
as a digital experience monitoring.
So this is where you have applications that interacts
with user and you would like to know what the user is doing,
what the user behavior is, and what part
of the systems are users more interested in
that helps you to make business decisions based
on those data, or it can improve
the overall efficiency of a system.
The next one is what is called this operation center
or AIOps or DevOps, where you can use observability
to reduce your meantime to knowledge.
That is if an issue were to happen
then you can quickly identify what the issue is
and once you know that, then you can formulate a resolution
and the time it takes to formulate a resolution,
which is termed as meantime to resolution.
The last application of data lakes,
last application of observability is in data lakes
is where when you process a large volume
of data and you have this large complex system
in tracking feature and you'd like to know
what the state of the system is in
and how you can characterize the behavior
so as to improve the data processing.
So what data can you collect from a system to improve the,
to understand the internal states of a system.
The first and the foremost is what is called as SLOCs.
These are time series events that happened,
or that happen that helps you to get an insight
into what is happening in the system.
These can be from your applications or infrastructure
or other parts of system.
Next is metrics, which gives you a numerical measurement
of your system state
or the resource utilization of a system itself.
And finally the last is what is called as a trace data.
It allows you to track a request
as it enters your complex system and as it goes
through it and finally returns after being serviced.
So that's it's, that's the last component of
observability itself.
So now let's look at architecture
or what are all the components that are involved, associated
and the services that you can use to achieve
or accomplish this observability within your system.
The first and the foremost is to the ability to
collect these three different types of data,
the logs, metrics and traces and open telemetries
and open source project that aims to do it
in a vendor-agnostic, or in a
programming environment-agnostic,
or your infrastructure-agnostic way.
It's an open-- it's aiming to be an open standard.
So OpenTelemetry is a set of tools,
STKs, and agents that helps you to collect these data.
So you will collect it and you will use other tool
what is called this DataPrepper.
It allows you to change this
or transform this data into a form that can be consumed
by a service like Amazon OpenSearch Service.
We all know there are agent-based tools
like fluentbit that allows you to collect metric
and log data from your applications and send it
to next layer or next part of the architecture itself
which is the storage layer. Where you can send--
so DataPrepper and fluentbit sends these logs
and metrics and traces
to storage services like Amazon OpenSearch Service
and there are other services that are available
that allows you to capture these metric data
and store at scale like Amazon Managed Service
for Prometheus, where you can directly write
those data as well.
So once you store the data and index it,
now they can be visualized, correlate
and which helps you to gain an insight.
You can use, again, Amazon OpenSearch Service
for visualization, or there are other services
like Amazon Managed Service for Grafana
that helps you to visualize the data.
You can use the metrics data coming
to Amazon Service for Prometheus, or you can get the traces
and log data coming from Amazon OpenSearch Service.
So we look at some of the components that built
inside Amazon OpenSearch Service itself.
The first and foremost is what is called as
observant UI that has many different components
including what is called event analytics.
Event analytics is a place where you can use
a language called as piped processing language.
The example for which is shown here that
allows you to pipe the data
from on source together and interact with it,
filter it, and create a visualization and so on.
Event Analytics is a panel that helps you to do do
the visualization easily using piped processing language.
We have another UA called Trace Analytics that
helps you to visualize this trace data
in a fashion that helps you to identify the throughput
or error rate or the latency within a service
very quickly.
Let's go a little bit deep
into what this Trace Analytics feature itself.
First and foremost, I'd like to talk
about what is called as distributed tracing.
As a request, a user request, enters a system, it travels
to the different parts of the system.
The distributed traces or tracing refers
to the method of observing the (indistinct)
as they pass through the system.
A trace is something that tells you a story
as our workflow as it propagates to the system.
Say, for example,, the user initiates a request
let's assume there is an e-commerce application
where a user is buying a data.
So typically the workflow will be like user looks at a--
so it's for the product, looks at a product,
clicks on a product, and then buys it,
makes a payment option
and then finally the order is complete.
So let's imagine a system in this case, if
the user is creating, say, for example,
the act of purchasing a span set of activities
that ends up calling different services.
Say, for example, you have to call your inventory
service to reduce your inventory, or you gotta call
your billing service to accept the payment and so on.
So the first, the ability to trace the
request as it goes through a different part of
system is what is called as distributed tracing.
So first I'd like to talk about span details.
This is a UI that gives you information about
SPAN and its associated details.
A span is a a single unit of work that is done
within the system collectively.
So this UI gives you the details about the
latency errors and the service timed summary as
the request travels through the system.
So these span details are connected,
are collected to the OpenTelemetry specification,
which was built from open tracing specification earlier.
We also support different other specifications
like on, other standards like Jaeger and Zipkin
and also provided specific specifications like
X-Ray and OpenTelemetry SDKs.
The next part of the system is what is called a service map.
This gives you an end-to-end view
of your application environment.
It collects, it creates a graph
through the connections through relationship
between the different parts of the system
and provides this error rate and throughput information
in such a way that you can, with a single glance,
you can look at where the issues are happening
in the system and so on.
So this helps you to isolate the hotspot
within the system based on these three different metrics.
The last is what is called as trace groups
this aggregates similar traces into a trace group
and this is based on your service name
or your APA name and so on.
So the unique thing about these trace groups is
that you can monitor the performance of these trace groups
and you can look at what the error rate was,
and the average latency and things like that.
It helps you to quickly spot anomalies that is happening
within your trace groups and allows you to
as I mentioned before, isolate the errors quickly.
Okay, and I'm gonna show a demo
of sample application that puts all this together.
Here is a demo that I'm gonna show you
about the different content that we talked about.
This is a demo of a sample application,
an e-commerce application.
This simulates the user behavior when you purchase an order
or places an order, pays for it, and so on.
This sample application is running in Amazon EKS
and all the microservices are running
in a different container
and you have these services called the payment service
or the authorization service and order inventory and so on.
So using OpenTelemetry
we collect all the traces and give it to DataPrepper.
And DataPrepper then prepares or transform the data
into JSON that can be entered
into Amazon OpenSearch Service.
And these containers also writes locks
and these locks are connected, collected by fluentbit
and they are return onto Amazon OpenSearch Service.
So this is how the sample application looks like.
In the sample application,
I just have a few buttons that simulates the user behavior.
So if you clicking
off each button simulates a user action here
and I'm gonna create an order
and then check out an order and then pay for it.
That completes one normal logical flow of operations.
Then I'm gonna log onto OpenSearch Service here dashboards
and it's already logged in with the username here.
Then I'm going to jump into the OpenSearch observability
UI which you can reach by going
to the left-hand side and clicking observability here.
And it lands in Event Analytics page by default.
And there are few other UA components called
Trace Analytics operation panels and so on.
Let's look at Trace Analytics.
This gives you a tracing dashboard that shows you
all the trace groups.
Trace groups are these collected traces that perform
on logical operations, are grouped
by the APA name or the service name.
So in this, in this hypothetical situation
the clients are complaining
that their audit checkout is taking a long time
and we are going to see which service is responsible for it.
And just by looking at the trace groups here
I can see from the dashboard
that the client checkout operation is
the one that is having an hundred percent error rate.
The pay order and the create order, they are working fine
and this UA also shows you the metrics
or the latency over a period of time
and the average latency that it is taken to.
And again, so the next view is what is called
a service map.
That gives you an overall end-to-end view of your system
and how your services interact.
In this case, the front-end client
or your front-end core interacts
with the authentication service
and also with order service and database and so on.
So, and also it shows you these error rate
and throughput metrics
and the darker it is, the higher its latency
and you can very quickly say that the hotspot is only
in this order on the front-end services.
Then if you select this error rate
you can see that the payment service is the one
that has the highest error rates of all.
Then you can quickly zero in on
which is the service that is having error.
Now so that is the service map view.
Next, I'm gonna look at the traces here
and the client check out traces associated with one trace.
I'm gonna click on those trace
and gives me the trace ID, and then
when you click on that one, it can give you all the trace
and span details here.
For example, this also show a view where gives you the time
that each service, time spent in each service.
In this case the client codes takes a long time
and the payment services we spend 23% of time there.
And finally, analytics services collects the locks
and initiates other metadata associated
with the services it spends
in the rest of the time here.
These are the interesting span details that shows your
timeline view of what is happening over here.
So the root span is this client checkout
and this initiates a HTP post
and that calls the payment service
and which is showing a error here.
So this is a nice timeline view of span associated.
You can also see it in terms of a span list over here.
So the next-- we already saw Event Analytics
and you can use your ppl,
the piped processing language to write your statements
and then show, interact with it in easy fashion here.
And you can also store and retrieve those visualizations
that you create using this panel,
the queries and the visualization panel.
So as I mentioned,
there are other operational panels that you
can create dashboards out of that you created
with the Trace Analytics and Event Analytics
and the notebooks helps you to organize a nice, tell a story
of doing a root cost analysis or something like this.
So that's the quick demo I had.
Thank you everyone.
I hope this has been useful
in getting introduced to what is Observability
and what are all different components
in Amazon OpenSource Service.
As a next step, you have this lab that you can go
and play around with to understand the, understand
and build the components for yourself.
Thank you so much.