- Thank you for joining me today
to learn more about how Amazon DevOps Guru,
a machine learning powered service, from AWS,
helping developers and operators,
automatically detect anomalies
and improve application availability.
My name is Shivansh Singh,
and I'm a Principal Solutions Architect
here at AWS and a DevOps Guru Ambassador.
In this episode, I'm going to show you
how Amazon DevOps Guru for RDS,
a new ML powered capability, can automatically detect
and diagnose database performance and operational issues,
enabling you to resolve bottlenecks
in minutes rather than days.
But let's first understand what is Amazon DevOps Guru?
DevOps Guru is a fully managed service,
that uses machine learning informed by years of experience
from amazon.com, and AWS operational excellence.
These models can identify anomalous application behaviors
such as increased latency, error rates,
and resource constraints, and surface critical issues
that could cause potential outages or service disruptions.
It does not require you
to have any machine learning expertise to use it.
Once you sign up for the service,
it automatically starts monitoring the data and metrics
to understand your application's behavior.
DevOps Guru reduces noise
and help focus on the highest severity issues
with detailed insights to provide a quick MTTR.
DevOps Guru for RDS is a new capability in DevOps Guru,
that uses machine learning and anomaly detection,
to look for unusual and problematic performance events
in your Amazon Aurora MySQL-Compatible Edition
and Amazon Aurora PostgreSQL-Compatible Edition.
When it points one, it performs analysis on the anomaly
to determine what kind of conditions are contributing to it.
Some of the conditions that DevOps Guru for RDS
can highlight are, database weight states also known
as weight events that are creating bottlenecks for users.
For any anomalous weight state, DevOps Guru for RDS also
identifies SQLs that are likely contributing to the problem
as well as memory and the IO.
When DevOps Guru for RDS detects a performance bottleneck,
or operational issues, it displays its findings
in the DevOps Guru console and send notifications
via AWS services, such as Amazon event bridge
and Amazon simple notification service also called SNS.
Enabling you to automatically manage and take real time
action before they become customer impacting outages.
It has no manual step and you don't need ML expertise
or deep database expertise.
Now let's look at how Amazon DevOps Guru for RDS works.
It all starts with performance insights.
The database tuning metric and telemetry tools for RDS
and Aurora.
DevOps Guru for RDS relies on this data,
and only works if you have performance insights
enabled on your Aurora clusters.
DevOps Guru for RDS in just numeric metric only,
never any customer SQL commands or other sensitive data.
It particularly monitors the database load metrics
for anomalies comparing historical baselines
to current activity to pinpoint unusual behavior.
One of the optimizations DevOps Guru for RDS
has made for database workloads,
is to account for seasonality.
In many database workloads,
there are regular periodic spikes of activity from bad jobs,
ETL reporting, and other scheduled tasks.
DevOps Guru for RDS only highlights anomalies
that are not periodic, and that anchor outside
of the regular course of normal database activity.
Once an anomaly is detected, DevOps Guru for RDS
performs analysis on the metric that make up the anomaly
making note of the most prevalent weight states,
SQL commands and other metrics that coincide
with the anomaly.
A rule based algorithm processes the result,
and generates simple straight forward explanations
and recommendations.
DevOps Guru for RDS sends the basic anomaly information,
including severity, timeframe and resource IDs
to the main DevOps Guru service and console.
DevOps Guru has the ability to send SNS events
and integrates any database anomalies into the big picture
of pure full application stack.
Here in the DevOps Guru console, is where customers
usually first find out about a database ML.
Anytime DevOps Guru for RDS detects an anomaly
in an Aurora cluster, the DevOps Guru console
will highlight it, and direct you to the DevOps Guru
for RDS detail page for more information on the anomaly.
On the detail page, DevOps Guru for RDS
displays the analysis results,
providing the context necessary
to understand what is contributing to the database issue,
and also the results of the analysis that point you
to the best next steps for resolving the problem.
With that, let's go to the demo
and see the DevOps Guru for RDS in action.
I'm here on the AWS console
and I have few database instances in my AWS account.
You can go to AWS console
and search for RDS and click databases.
You can see all the database instances I have.
Now in RDS, we have something known as,
Amazon RDS performance insights.
Performance insights is a database performance dashboard
designed to help you quickly assess performance,
on the relational database workloads.
One of the core metrics of DevOps Guru for RDS measure
is the database load.
Simply stated, it is account of how many database sessions
are active at any given time.
A session is just a connection to the database.
But DevOps Guru for RDS, don't just count all sessions,
it only count active ones, that is connections
for which the database is in the middle
of fulfilling a request.
So that means, it's counting sessions that are not idle
but instead active in a call.
It turns out this counts specially, if you average it
over one minute, five minute, 15 minute or one hour,
is a very accurate measure of how stressed a database is.
Performance insights, collect discount once per second.
So I'll select one of my database instance
and I can see health of my dash database here.
This health dashboard is designed to detect
and visualize database health.
I can further drill down by expanding the visualization here
and see the detailed visualizations.
Just looking at this visualization,
you can see that there is something wrong.
There is an increase in the database load.
The number of CPUs are two, but the database load
has increased from 20 to 500.
What that translates into practice,
is that there are two workers on the database
that are doing the work, but there are almost 500 sessions
that want to do the work.
Which means vast majority of them are ready.
That affects user experience,
and it represents poor database health.
While performance insights help provide details
on the database health through a visual representation,
It does not alert or alarm the engineers about the issue.
That's where DevOps Guru comes into play.
DevOps Guru not only detects these kind of issues,
but also provides diagnosis and next steps
in terms of recommendations on how to solve those issues.
To see those issues, let's go to DevOps Guru console.
Here on the dashboard, DevOps Guru shows
the overall system health of the entire stack
of your application.
On the insights page, you can see various anomalies detected
by DevOps Guru.
Here you can see that DevOps Guru has detected anomalies
in my Aurora database.
Clicking on these anomalies,
you can see the detailed overview
of that anomaly on the inside details page.
It shows various metrics related to the database,
along with the visualizations.
To see the detailed analysis of the issue,
I'll click view analysis,
and it will take you to the detail analysis page,
where it shows you the database name, engine name,
anomaly type, anomaly start time and the duration
for which the anomaly lasted.
It also gives you a quick description
of what happened with your database
in simple to understand text.
To scroll down, you can see here,
it's the same graph which we saw
in the audience database performance insights page,
but it provides more details than that.
It highlights the detected anomaly in red
for which the customer was alerted via SNS.
It highlights the start and end time
as anomaly boundaries to easily visualize the duration
of the anomaly.
It shows the statistic analysis of the problem.
For example, in this case,
it shows that the size of the spike is significantly higher.
That is almost 500 times higher
than the typical database load.
This statistics is important because it's not just based
on the typical threshold value, instead, it's based
on the entire database load analysis.
and therefore DevOps Guru for RDS only alerts users
if there is a spike,
which is highly unusual compared to typical database load.
In addition, DevOps Guru for RDS
also provide detailed analysis and recommendations
to what you can do about the issue.
For example, it's showing here
that there's a high load of weight event
for the RDS database instance.
It shows the exact metric name,
Aurora lock thread slot putates related to the high load.
And if you click the metric name, it provides the meaning
of that metric in simple terms,
which does not require database expertise to understand.
Not only that, there is also a link
to the detailed documentation page related
to that specific metric, which tells you what are some
of the actions you can take,
to resolve the database problem.
And the reasoning behind why DevOps Guru
recommends these actions.
It's very important to note here,
that unlike commercial databases,
it's very difficult to find
detailed troubleshooting documentation
for open source database engines.
It either doesn't exist at all,
or it's very scarce in details.
This is where DevOps Guru for RDS helps reduce time
to resolution from days to minutes
and providing detail troubleshooting documentation.
Let's look into another use case.
This is an Amazon Aurora Postgrade database.
From this visualization,
you can see that there is an unusual spike in database load.
But this one is different than the previous one.
In this case, it's waiting for CPU weight on event.
And once again, we provide the details
about what a is CPU weight event means.
The troubleshooting documentation for the issue,
the SQL queries and the reasoning behind
why Amazon DevOps Guru think its unusual activity
is a problem with the database.
In addition, we are also doing something else here.
DevOps Guru is also looking at the system overall
and answering the question whether CPU resource
has also been overrun during the time of the problem.
And in this case it is.
When you click the related metric
you can see these visualizations where you see
that the CPU has really been exhausted.
It had effect on the database performance.
First, you see that there is a running task metric
that went really high during the time of the problem
and therefore correlated to the problem itself.
Another metric that people typically monitor
is CPU utilization for monitoring CPU exhaustion.
And you can see, it's also running very high
at around a hundred percent utilization
at all the time during the database problem duration.
The way Amazon DevOps Guru recognizes
that this is a CPU exhaustion issue,
is by correlating multiple metrics
and collecting signals that are pointing
all in the same direction.
In this case, there are two very strong signals
that are coming from the operating system metrics
that is running task and CPU utilization.
And there's also a very strong signal
coming from the database load as well.
Because CPU weight event has taken a very significant chunk
of the entire load.
In this case, it's almost a hundred percent.
We provide the reasoning as why Amazon DevOps Guru
think it's a problem.
And some of the recommendations
on troubleshooting the issue.
There are a few other use cases where Amazon DevOps Guru
for RDS can detect issues.
For example, due to database memory issue.
In this case, Amazon DevOps Guru detect a problem
with the database memory,
and it was derived by the strong signals
it received from multiple metrics
like out of memory kills, database memory usage,
and a number of database connections.
All signaling in the same direction
during the time of the database problem.
So that's DevOps Guru for RDS
and how you can use it to detect, diagnose
and remediate a wide variety of database related issue
in Amazon RDS.
I hope this episode gave you a good sense
about how DevOps Guru applies AI and ML
into your development and operations workflows
to improve your database performance.
Don't forget to watch the other DevOps Guru episodes
which also deep dives into other use cases.
Thanks for watching.
(gentle music)