[MUSIC PLAYING]
ALISA GOLDSTEIN: Welcome.
In this video, we'll walk through how to create alerts
from your logs in Google Cloud.
My name is Alisa.
And I'm the product manager for the alerting team.
Alerting is a horizontal observability experience,
and alerts can be created from any data streaming
onto your platform, from any service that
examines your metrics and logs.
There's a tremendous amount of data out there,
with billions of logs and metrics
streaming through systems worldwide.
The key questions that SRE and development
teams ask themselves are, what is important?
And how do I find the problem?
And how can I quickly fix the problem
before any serious consequences reach my end users?
We commonly refer to this as mean time
to detect and mean time to repair.
The lower the values are, the better your team is equipped.
Google Cloud's DevOps research and assessment in 2019--
we saw that well-equipped customers
were over 2,000 times faster in their time
to recover from incidents.
This happened while they also had over 200 times
more frequent code deployments than teams
who were not as well-organized.
With Google Cloud, when an alert is triggered,
your appropriate tools for logs, metrics, and dashboards
are all available for faster debuggability.
So to answer the question of what is important,
you'll want to first create an alert from your logs.
You want to create this if you want to be notified immediately
that a single event has occurred in your system.
This may apply for a variety of situations.
This includes security concerns, such as unauthorized access
or unexpected IP addresses.
This can include system runtime issues, like host errors
or production errors, or smaller events that
are impactful to your team.
So let's get started and walk through a demo.
We'll start off with the Logs Explorer.
Here, you can interact with your logs,
understand them by severity level,
and view this information over time.
You can run any kind of query that you like.
And this accepts standard regex expressions.
We have one saved here that we'll run.
And this search is for an error for a red team
at the 0 node in production.
If this event happens for our team,
we want to be able to create an alert.
So what we'll do is we'll come over here under Actions
in the top right corner and select, right here,
Create Log Alert.
This opens up a slide-out panel.
And you can start to fill in the form to create your alert.
First, we start off with an alert name
that is meaningful to your team if they see it.
And then we move into documentation.
The documentation is a playbook that
says, if this alert were to trigger, what
are immediate steps that your team should take in order
to debug the situation and to have a faster recovery time?
The documentation is a playbook that accepts
any kind of markdown language.
And we suggest that you also include
any kind of links to internal knowledge share documents
that you have on your side.
It also accepts any kind of variables
where you want to automatically pull in any data,
such as the project information, resource type, and others.
We'll go ahead and move on and choose which
logs to include in the alert.
Since we had this query, it automatically
pulled in the relevant log information.
And we can preview it to make sure that it's correct.
If it's not, you can always come back here and edit your query
and change the expression.
From there, you can set an optional snooze
time between notifications.
What this says is, if this event is happening
and you expect that it's happening at a larger scale,
you want to be able to quiet the notifications
while your team debugs.
So the default quiet period is one hour between notifications.
And you can set this as low as five minutes
and as high as seven days, or choose a custom period.
For now, we'll keep the default.
Then the question is, if this event were to occur
and the alert is triggered, who should get notified?
This is supported with all of the standard alerting
notification channels.
That includes email, SMS, mobile push, Pub/Sub, web,
and Slack integration.
Here, we'll go ahead and choose an email.
Once we click Save, then the alert policy is created
and we can go ahead and view it down here.
We can review the information to make sure it is correct.
And from here, we can see that the
logs that match, any incidents that
have happened historically, confirm the notification
channels, and documentation procedures.
When the incident occurs, according to our notification
channel, we'll get an email.
And the email will look something like this.
When your appropriate team receives the notification,
they can go ahead and click the View Incident
button, which takes them to the incident detail page.
From here, they'll be able to understand what's happening,
read the labels that are appropriate for this issue,
and begin to debug.
They can come in here to view the logs
and begin their investigation.
And so that, in a nutshell, is how
to create an alert for your logs and then
how to then begin exploring and debugging
when the incident is created.
If your team is looking to create
an alert after a certain amount of logs have been triggered,
then you're looking for a different feature.
And this is called a log-based metric.
It's available up here in the top right corner.
And this is Create a Metric.
This is available for teams that are
looking for alerts if there's a deviation from expected
behavior.
One example might be a retail company.
If, for example, they're expecting
x amount of checkout failures in y amount of time
and if that expectation is breached,
then they want to be notified.
That is different.
And here, what we're talking about
is log-based alerts, where we're focusing on anomalous events.
Next, the question becomes of how do you take the first step
and how do you cross the line to get started.
The first step to answering the question of what's
important in creating these alerts
is to read the Cloud alerting--
the Google Cloud alerting documentation
and to understand how to approach
this with your organization.
You can use the doc variables in your alert documentation
for faster debugging.
Today, we explored this in the UI.
And if you're looking to automate this at scale,
the same functionality and more is available via the API
to be managed by Terraform or other automation tools.
And as a reminder, if you'd like to be alerted
after a count of logs are present in your system,
then you're looking for log-based metrics.
Thank you for joining us, and we wish you
the best of luck on your observability journey.