YURI GRINSHTEYN: Hello everyone, and thank you
for joining us today for this session.
This is OPS302.
We'll be covering monitoring as code.
I'm honored to have Pod Sandhu from the Schlumberger SRE team
here with us today.
Later in the session, he'll be discussing
how his team and others at Schlumberger
implemented the practice of monitoring as code.
He'll also give it to some lessons
that they've learned in this journey
that we think you can benefit from as you start yours.
Pod, thank you so much for being here today.
PARDEEP SANDHU: No thank you Yuri.
It's a pleasure to be here.
YURI GRINSHTEYN: My name is Yuri Grinshteyn,
and I'm a site reliability engineer
at Google Cloud on the customer reliability engineering team.
So here are the topics we'll be covering today.
We'll start by understanding the basics.
We'll define the concept of toil as it
relates to the practices of site reliability engineering.
From there, we'll cover the objectives
of as code practices.
Next, we'll review the specific Google Cloud monitoring
entities that can be automated and compared,
three approaches to automating.
I'll wrap up the first portion of the session
by showing you a couple of specific examples of automating
Cloud monitoring entities.
From there, I'll hand it over to Pod
to cover the lessons learned at Schlumberger, as they
have pursued this in earnest.
Let's get started.
So first, what do we mean by toil
and why is automation important?
Well if we go back to our fundamental principles
and practices, we'll recall that site reliability engineering,
or SRE, is a specific set of practices developed at Google
and since embraced everywhere that hold user happiness
and appropriate reliability at scale
as their ultimate objectives.
One of the major areas of focus for both the more generalized
DevOps movement and SRE, which is a specific implementation
of it, there's the focus on automating as much
of the repetitive manual tasks that specifically don't
add enduring value as possible.
Ultimately, our objective is to achieve the appropriate level
of reliability for our services and maintain that level, even
as user demand and load increases,
without having to proportionally scale the staff required
to support that service.
This is really the ultimate value of focusing automation.
It allows us to reach much greater levels of scale.
This is, of course, a very simplified version
of the answer.
You can find much more detail on why
we focus on reducing toil in the books
we've published on the topic.
I'll particularly encourage you to review
chapter 5 of the first SRE book called
"Eliminating Toil," which discusses
this in much greater detail.
So now that we've covered the value of automation,
let's talk about what we mean when we say that we want
to treat something as code.
Here are the things we'll be looking to accomplish.
Now a lot of this comes from the infrastructure
as code principles, but there's no reason
why we can't think of monitoring as code
using the same exact language.
So first, we want to be able to accurately determine
if there are any differences between what we want
to be the key to our monitoring configuration,
and what is actually currently the case.
And so to that end, we need a way
to precisely specify the desired state
and be able to retrieve the current state.
Next, we want our work to be reusable.
We want to be able to create modules and components
that other folks can use.
And again, we're in pursuit of scale here,
and that can't happen if every service team has to build this
from scratch, especially in a model where
SRE is a shared service across multiple service teams.
We want to use a declarative approach to specifying
our desired state.
This means that we want to define
the state we want, not the set of actions
to take to get there.
Now, one of the benefits of this approach
is that it's [INAUDIBLE],, which means
that we can run our automation no matter
what the current state is and still be sure we'll arrive
at the correct end state.
This is different than an imperative approach
where we specify the exact steps that should be taken.
That might work best for programming but not
for configuration.
And finally, in part because we are
SREs who value software engineering practices,
we want to treat these configurations in the same way
we treat other source code.
One important thing we want to achieve
is the other side of the reusability coin.
We want to both share our work and enable
others to collaborate and contribute to it.
And this is where code management
is incredibly important.
So I hope that you're now convinced of the value
of automation in general and applying as code principles
to configurations specifically at this point.
The next logical question is OK, how should we get started?
What can we actually automate?
So in Google Cloud Monitoring, there
are four primary entities whose creation, updates,
and deletions can be automated.
The first three are available today,
and that's where most of our focus will be.
They are uptime checks.
So this is a feature of Cloud monitoring
that allows you to configure synthetic checks to validate
things like URL availability, website performance,
and TCP connectivity on VMs.
Alerting policies.
So I'll get into their details in a bit,
but these are how you configure Cloud monitoring
to actually send alerts and notifications when
specific conditions are met.
And finally, dashboards are collections
of charts folks use to visualize the state of health
of their systems.
When looking to automate Google Cloud Monitoring
configurations, you have three primary options.
The first option is to use either the Google Cloud APIs
directly or make API calls in your code using
client libraries provided by Google or third parties.
This works very well if you have a robust software engineering
practice, and you need to include
monitoring automation in larger scripting or cognition
projects.
Your second option is to use Google Cloud Deployment
Manager.
This is a fully managed and hosted service
provided by Google.
It uses YAML as its declaration language,
and the current state of your configuration, once you apply,
is then stored in the Cloud.
You can interact with Deployment Manager using gcloud commands
or its own APIs.
And then your third option, and the one
I'll be using for our examples coming up next, is Terraform.
This is an open source toolkit provided by HashiCorp.
It uses its own language for declarations.
But it has an excellent multicloud support, which
is important for many users.
The open source version, which is
what I'll be showing you today, stores state locally
in dot tfstate files.
So these can be checked with the source control
just like the main configuration files.
So I imagine by now this is probably
enough theory for everyone.
Let's get into some specific examples.
Let's start with a simple example
of creating an uptime check.
As you may recall, uptime checks are
used to measure what service or TCP endpoint
availability [INAUDIBLE].
Here's what our Terraform file looks like.
There are five things to note here.
First, at the very top, we specify the provider
to let Terraform know that we're working with Google Cloud
and to provide the project we're operating in.
Next, we define the resource.
It's an uptime check.
In the http_check section, we specify the path and port
to connect on and whether or not to use SSL.
In the monitored resource section,
we specify that we're checking a URL rather than a TCP endpoint,
the host we want to connect to, and the project where
this uptime check will be created.
And finally, we specify a content match string
we want to validate in the response.
In the terminal, we'll run the Terraform plan command.
It will validate our configuration,
compare it against the current state, if there is any,
and determine what resources need to be updated.
Here, the plan is to add one new resource,
as you can see in the output at the bottom.
Once we've confirmed that the plan is correct,
we issue the Terraform apply command.
It will once again provide the plan
and ask for an explicit confirmation.
It will then provide the specific resources that
were created, changed, or deleted as a result.
And there we are.
We can now log into the Cloud console
and validate that our uptime check has been created.
If we want to clean up what we did,
we can issue the Terraform destroy command.
Similarly to apply, it will create and validate
the plan of action.
It will then ask us for a confirmation
before actually destroying the resource,
and then let us know when it's done.
And now we're back to a clean slate.
Let's take a look at another slightly more complex example.
Once we've created an uptime check,
we may want to get alerts if and when it fails.
Before we get into the code, let's review the basics.
We're going to create an alerting policy, which combines
both the condition to be evaluated,
the trigger notifications, and the mechanism
to be used for the notification itself
or the notification channel.
Our Terraform code needs to describe both of these.
So here's how we would configure the policy in the UI.
We would use uptime check URL as the resource and the check
passed as a metric.
We would then filter it through just the check we care about.
We would use the count false aggregator
to count how many of the locations from which
the check is running report failures.
Our condition will then trigger if any failures are
reported within five minutes.
Here's how we would write the Terraform
file to represent that.
Note that we specify the notification channel first
with type email and then email address where
the alerts would go to.
The output would then get stored in the variable.
We then define our actual alert policy
with a condition threshold.
There, we provide the metric filter, the comparison
operator, and the aggregations.
And then finally, we specify the notification channel
we've created up at the top using the variable all the way
at the bottom.
When we run Terraform plan, we see
that it's actually going to create
two resources, the notification channel and the alerting policy
that is going to use that channel.
When we run apply, we see that two resources
are created, as we expect.
As before, we can confirm that our alerting policy
was created in the console.
We can also confirm that the correct notification
channel was created and that our policy is [INAUDIBLE]..
If we want to undo what we've done,
we can again issue the Terraform destroy command.
Note that this time, two resources are destroyed.
Terraform is aware of the dependencies involved
and destroys the alerting policy first
and then removes the notification channel.
I hope this served as a good introduction
to the practice of monitoring as code
and how to use Terraform to automate and configure
in Google Cloud monitoring for your projects.
At this point, I'll turn it over to Pod
to talk about how the SRE team at Schlumberger
put these into a real world practice.
PARDEEP SANDHU: Thank you Yuri.
So I'm going to talk to you about Schlumberger's monitoring
as code journey over the past two years, starting with 2018.
At Schlumberger, there are many solution teams
developing apps and services for the Delfi ecosystem.
And each one of these teams was creating their own version
of the monitoring framework.
What I mean is they were creating dashboards or alerts
or even log-based metrics in their own silos,
using different coding languages,
or in some cases just configuring through the UI.
In terms of the big picture, it was obviously
great to see so many teams taking monitoring so seriously.
But the problem was, with this approach,
that very little code was being reused,
which lent it to being hard to build any best practices
or learnings.
And in most cases, word of mouth or presentations
were the only way of showing what each of the teams
were doing.
So what changed in 2019?
Well, a new central SRE team was created,
and it's the team that I work in today.
The purpose of this team is to provide governance
and alignment across the various solution teams in Schlumberger.
At the beginning of 2019, we started brainstorming
of what we wanted to focus on for that year,
and after reading various books on SRE from Google,
we decided it should be toil management.
After attending many presentations and demos
of each of the monitoring frameworks
from each of the teams, we wanted
to merge the implementation learnings into one toolkit.
So the idea of creating one toolkit or one framework,
we believe it would allow us to add new monitoring
capabilities that would benefit all the teams using it.
One example of this was when Google introduced the service
level monitoring APIs last year.
We could also ramp up solution teams much faster
with the use of this toolkit within their apps or services.
And by using the one toolkit, we could
spread the message of how we wanted
to monitor systems across Schlumberger more consistently.
So the fun part.
Creating the toolkit.
So what did we do?
By taking the best bits of each of the solutions monitoring
scripts, we created a toolkit to work with the Google monitoring
endpoints.
Key to the design of the toolkit was
to understand that most teams already
had monitoring setup, either through the code
or through the UI.
And we want to take the approach of backing up and restoring.
Backing up the aspects of monitoring
that they care about in the form of JSON files using [INAUDIBLE]
and providing the ability to restore from them.
Once we'd created the toolkit, we needed a way of teams
to adopt it because there's no point in having
a toolkit that no one uses.
So we created a second git repo, which they
could fork from to get started.
And both of these Git repos are an image in this slide.
Within the git repo, we provided the necessary pipelines
to take our approach of backing up and restoring.
We also provided the monitoring controls
that will need to be in place for all of our GCPs
for SOC 2 compliance.
And we also included other examples
of how to monitor different Google Cloud
technologies such as Pub/Sub, Google Storage Datastore,
Google Load Balancer.
And for the teams that were new to SRE practices,
we also included examples of how to set up SLIs and SLOs.
Within the git repo that the solution teams had forked,
they could also add their monitoring configuration
that was specific to their app.
We inner sourced both of these git
repos so that teams could contribute new features,
fix bugs, or even provide good examples of monitoring
certain aspects, which the other teams may benefit from.
Each of these Git repos are managed through pull requests,
which our team approve because we
have the broad knowledge of how all the other teams are using
the toolkit, and we can give advice
on the best ways of monitoring certain aspects of Google
Cloud.
Having everyone on the same toolkit
gives us the agility to be more dynamic
with either new technology adoption or compliance
requirements.
So what are the key takeaways from our learning
of our implementation?
I would ask you to go and look at the Google git
repo in github.
There you would find examples of how to back up
your alerts using Python.
And I think this would give you ideas
of how to add monitoring as code into your organizations.
So where are we today?
We believe that within your organizations,
you can start by implementing the following points that
are shown in this slide.
And within our organization, by consolidating
into a single toolkit or framework,
I'm going to explain to you how we
have benefited by doing this.
Number one.
New solution teams are no longer reinventing the wheel.
The learning curve is less steep,
and the time spent is significantly reduced
implementing monitoring.
Number two.
Monitoring is no longer an afterthought.
The automated deployment allows the solution teams
to implement monitoring early in their product lifecycle,
enabling them to have the insights into the reliability
and performance of the critical user journeys
and metrics that they've defined in their projects.
This will also improve the solutions overall reliability
at the time of launch.
Number three.
Continuity and continuous improvement
is achieved using code rather than
having to update multiple different documentation
sources.
And lastly, point number four.
The central support organization has a consistent interface
for monitoring, alerting, and reporting
across the various solution teams,
both for operational and business level needs.
This is an example of our service level objectives
compliance dashboard that we created in-house.
This gives us a single pane of glass for both business
and solution teams on how well our apps and services
are performing.
Each of the colored boxes represents
wherever a solution or shared service
is either in compliant, non-compliant,
or possibly even in feature freeze.
And with that, I'd like to thank you for listening to me today.
And I'll hand it over to Yuri.
YURI GRINSHTEYN: Pod, thank you very much for that.
It was extremely informative.
I really appreciate you being here today.
And thank you to everyone for watching.
We hope that [INAUDIBLE] found this useful,
and we certainly would encourage you
to get started with your own journey
toward monitoring as code.
Thanks again.
Take care.