Welcome to 'Back to Basics', I'm Adrian De Luca.
Today, we're going to look at
monitoring and observability of your serverless applications.
Think of monitoring as doing a basic health check with your doctor.
For your applications we look at things like
if your infrastructure is up or down,
or whether your software modules are working to expectations.
Observability is a little bit different.
It's more like doing a fitness test with your coach.
Are thing working well together to complete a user task?
Well, CloudWatch, is the perfect service for this.
It collects, monitors, analyzes, and acts on
all the things going on in your system.
So, let's get into it.
Here's a typical serverless application
that processes events which trigger other AWS services.
Treated as an automic operation,
it's important to trace what's happening in between,
and identify any problems, and why they occur.
The five things that you'll want to look out for are:
concurrency limits, errors, dead letter queue failures,
destination delivery failures, and the age of stream messages.
AWS Lambda makes it really easy to meet performance spikes
without any capacity planning.
But you still want to understand just how many invocations are going on.
Each AWS account has an account limit
set to default to 1000 per region.
So, we actually want to understand
how much of this we're actually consuming.
We measure this with the concurrent executions metric.
Think about this as the unit of scale.
Setting an alert at around 80%
is a good way to see when and how often you're getting close to maxing them out.
Depending on your application response time needs,
you may consider the throttles metric
if you're not using reserve concurrency limits.
You need to be careful with this,
as it can provide false positives as well.
Next, let's look at error alerts.
If your Lambda functions are throwing too many errors
this can cause a real degredation in user experience,
and make the users pretty cross too.
Understanding the rate at which these errors are happening is really important.
Here's the math that CloudWatch uses to work out error rates.
Align the alert threshold to your Service Level Agreement.
For example, if your SLA states that 99% of functions should succeed,
then set the error rate alert to 1%.
Several AWS services, like SNS and S3,
process their events asynchronously.
Lambda places the events in a dead letter queue,
and a separate process actually reads them off
before sending them to your function.
It tries running it three times
with one minute waits on the first two attempts,
then three on the third attempt,
before returning an error.
Functions can have permission errors,
misconfigured resources, or hit size limits.
So, watching out for these is really important.
Setting up a dead letter error metric
allows you to track how many times this happens.
Similar to dead letter queues,
functions can send an event to a destination.
These could be an SQSQ, an SNS topic,
a Lambda function, or an event bus.
These can all fail, so setting up an alert
against a destination delivery failure metric is a good idea.
Finally, you want to look at the age of the messages.
For event source mappings that read from services like
Kinesis, Managed Kafka, or DynamoDB,
you want to understand the age of the record
to help you understand whether there is some
performance or efficiency problems in your functions.
The iterator age metric helps us understand
the age of the messages that they receive.
When this metric starts to creep up,
it's an indicator that the function is not keeping pace
with the rate of new messages,
and is falling behind.
At worst, you can experience data loss,
since the data in the streams are only kept for 24 hours by default.
So, set up an alert for iterator age
to detect and rectify before a disaster.
I've personally used these patterns more than once.
Now you can use them too.
Check out the links below for more resources.
Thanks for joining us, see you next time.