YURI GRINSHTEYN: Imagine this scenario.
You're running an analytics workload
in Google Cloud that reads data from Google Cloud Storage
and processes it using a Dataflow job.
The amount of data that the job processes has been growing,
but your pipeline hasn't had any issues keeping up.
You're having lunch when you get an alert on your phone.
Your pipeline is returning processing errors.
You abandon your lunch, open your laptop,
and start investigating.
Sure enough, the Dataflow job is failing.
And it turns out that Cloud Storage requests
are resulting in 429 Too Many Requests errors.
This has never happened before.
What could be going on?
Well, it's possible you've exhausted
a quota in your project, and the job is being throttled.
My name is Yuri.
And I'm a site reliability engineer at Google.
Today I'm going to show you how to proactively monitor
quotas and get notified when you start to approach the limit so
that you can raise the quota and avoid problems like this one.
Welcome to "Engineering for Reliability with Google Cloud."
[MUSIC PLAYING]
OK, let's dig into this and figure out
if the errors you are seeing are actually
due to running out of quota.
We can start on the Quotas page.
Here we can see that we've run out of Egress bandwidth quota
in this region.
Let's go to the Products menu and select Metrics Explorer
under Monitoring.
Here we can plot quota usage against quota limit over time.
Both of these metrics are available for the Consumer
Quota resource.
The usage metric is called rate quota usage.
Rate quotas are typically used for limiting
the number of requests you can make to an API or service
and are applied over a specific time interval.
In this case, the quota is Egress bandwidth per second.
The other kind of quota is an allocation quota,
which restricts how many resources you
can have provisioned at a time, such as the number
of VMs in a project.
They reset when the resources are released, when the VMs are
deleted, for example.
The other metric we need here is the quota limit metric,
which tells us what the limit for this rate usage is.
There's another way to view this data.
We can use MQL, the Monitoring Query Language,
to actually calculate a ratio of quota usage to quota limit.
We'll post a link for this query in the episode notes.
Here we clearly see that our usage to limit ratio
exceeded 1 for a period of time.
This explains why our job was throttled.
So now we know that we did exhaust network Egress quota.
And that's why our job got throttled and started failing.
We submitted a quota increase request
which unblocked our job.
Our pipeline is back up and running.
We don't want this to keep happening, right?
Let's set up an alert that will notify us
if we get close to the new increased quota.
Let's start on the Alerting screen
and create a new alerting policy.
We'll use MQL to define this policy.
Paste the query here.
And run it to evaluate both the query
we've written to calculate the usage-limit ratio
and the threshold for that ratio that we want
to use to trigger the alert.
That threshold is defined here, in the last statement
in the query, using a condition statement.
Looks like our signal is correct.
We need to name our condition.
And specify how we want to be notified
the next time the alert fires.
We also want to specify whether the incident should
be automatically closed after some amount of time passes.
Let's name the alert.
And review the policy to validate
that we've defined our signal and threshold correctly.
And I think we did.
And that's how you can use MQL to calculate the usage
to limit ratio and set up alerting
to be notified when you start approaching a quota
limit in your project.
Great.
Now, if we start to get close to the quota,
we'll get an alert so we can avoid having
our job be throttled again.
That's engineering for reliability.
Thanks so much for watching today.
Don't forget to like and subscribe to never miss out
on more "Engineering for Reliability with Google Cloud."
See you soon.
[MUSIC PLAYING]