- Hello, everyone.
I'd like to welcome you all today's quick introduction
on observability with Amazon OpenSearch Service.
My name is Muthu Pitchaimani.
I'm a specialist solution architect,
focusing exclusively on Amazon OpenSearch Service,
helping customers to build solutions with OpenSearch.
The chance are high that you have heard
about observability a lot in the last few years.
So let's look at what observability actually is.
"Observability," roughly defined, "is a measure
of how well you can infer the internal states
of a system by just looking at its external outputs."
Why do we need it?
Because, over the years, the complexity
of the system is growing, and it becomes quite impossible
to go look into each component of the system
and understand their behavior.
So it becomes very imperative to know
to characterize the behavior of a complex system
by just looking at it outputs.
So that's where observability comes in very handy.
So where does observability itself fit in?
To understand this, I'm gonna step back a little
and talk to you about what is called
as operational analytics.
Operational analytics is not something new.
It is a field that spanned out
of another field called operation research,
which has been going on for three or four decades
in industrial and manufacture.
So operational analytics is a field, the aim of which
to improve your business efficiency
under customer experience or value.
That's what it is.
Traditionally, when you run a business, operation research,
they did a large batch processing system
and then crunch the numbers
and then produce a report that helps you
to make a business decision through a manual process.
Operational analytics aims to replace
the manual process with a machine-enabled
or mission processing methods
so that you can arrive with this business insight quickly,
and there are different components of operational analytics.
The first and foremost is what is called
as descriptive analytics.
This is where you describe a complex system
to say what it is, then try to characterize the behavior.
The next one is called as predictive analytics.
This refers to all the business operations
and the things that you do that,
after characterizing the system,
you can predict how the system is going
to behave in the future, then lastly,
what is called as prescriptive analytics.
So once you know what a system is,
and so once you know how you can predict it,
prescriptive analysis tend to prescribe
a solution to the problem that's having.
This is where many exciting things,
like machine-learning-based,
and the artificial-intelligence-based system can be used,
and so that whatever that has been involving manual systems
can be kind of automated in a fashion.
So the mission becomes fairly intelligence.
So where does observability fit in?
Observability fits in this descriptive analytics,
where it tends to describe what a complex system is.
Okay, so what is the goal of observability?
Just like operational analytics,
the goal of your observability is to improve
the overall end user experience that you provide.
I mean, with your application and your services,
and if you ask an operations center person
what is the aim of observability,
they'll tell you that the aim of observability is
to reduce what is called this mean time to knowledge.
So mean time to knowledge.
What it means is that you need to be able
to understand that something is going
on within a system first,
then you need to understand what is going on the system
so that you can formulate a solution to it.
So this is measured in terms
of what is called this mean time to knowledge
and mean time to resolution.
So the one aim of observability is
to reduce these mean time to knowledge and resolution
so that you can improve the overall user experience
of a system itself.
So what are all the pillars of observability?
Observability is a means for a system to become observable.
You need different kinds
of data that need to collect from it.
The first and the foremost is the logs.
Logs are immutable events that occur over time
that helps you to get an insight
on what is going on within the systems.
So they are made from different parts
like your applications, your services
and infrastructure and so on.
Next is the metric data.
Metric data is a numerical measurement
of your state of your system that helps you
to understand the resource usage and things like that.
So over time, you can understand the trends,
and that helps you to predict what is going
on within the system later on.
The last and the final type of data is what I call traces.
Traces are data that provides you visibility
into the end to end flow of your user request,
from the time it enters the system
and until the time it is serviced.
So these three are the important data that allows you
to gain insight into your complex system.
So now, I'll give you a very quick introduction
into what are the companies that are there
in Amazon OpenSearch Service that helps you
to collect these observability data
and to get an insight from them.
The first and foremost is what we call
the Observability UI.
This is the UI that has different things
like Even Analytics, Trace Analytics.
Event Analytics is a UI that allows you
to interactively look at your logs,
metrics and also your trace information.
Pardon me, logs and your trace information using
what is called as Piped Processing Language.
The next is the Trace Analytics.
It's a UI that helps you to visually inspect
and interact with these traces that you collected
from the different parts of the system
and help to zero in on the issue very quickly.
We also already saw what Pipe Processing Language.
It's a language that you can use
if you're familiar with the Linux terminology
that you can use to interactively query
your data just by using pipes.
So you would take a data, pipe it into another system,
then it process it, and you can interactively build
the visualizations from it.
Next is what is called this Open Telemetry.
It's an open-source project that aims
to standardize the collection, the store
and visualization of all this data,
and the associated libraries, the SDKs and protocol itselfs,
and the Data Prepper is a component that consumes
the data from Open Telemetry and then prepares it
in a fashion that can be consumed
into Amazon OpenSearch Service.
Here is the architecture that shows you how
you can collect these data with different components,
store them and then visualize them.
For storing, you have a couple of options
with Amazon OpenSearch Service
and Amazon Managed Service for Prometheus
and for visualize, again, you have a couple of options
with the Amazon OpenSearch Service
and using Amazon Managed Grafana.
I hope this has been useful
in getting introduced to what is observability
and what are all the different components
in Amazon OpenSearch Service.
As a next step, you have this lab that you can go
and play around with to understand
and build the competence for users.
Thank you so much.