SPEAKER: One of the first and most important steps
to optimizing your costs is being able to understand them.
Today, we'll talk about a few different ways
to understand your Google Kubernetes Engine costs,
and go over a few best practices.
Let's go beyond your GKE bill.
If you haven't watched the last video,
I definitely recommend taking the time to check it out,
since we talked a bit about teams and culture.
These concepts are super important for being
able to get a solid understanding of your costs.
I'll start by giving an overview of some ways
that you can understand more about your GKE infrastructure
and applications.
Then I'll go over how to turn these details
into actionable steps, and then finish up
with some cost-optimizing policies.
Your organization might have a platform and infrastructure
team that manages the clusters, which
can be completely separate from the dev team that actually
write the applications.
When every team understands the cost implications
for the decisions they make, like how many resources an app
actually needs, they can all work together
to get things optimized.
So where do you start?
Well, that's probably going to be the monitoring dashboard.
When using GKE, the cluster's Info page
shows a lot of details.
But we're looking for some details about our metrics,
so let's head over to the Monitoring page.
This is part of Cloud Operations, which
has a bunch of helpful functionality for monitoring,
logging, and alerting.
If it's your first time using Cloud Operations,
you'll have a workspace created for you or you can create one
manually.
Either way, as long as your cluster was set up
with Cloud Operations for GKE, a dashboard will automatically
be created, named "GKE."
This dashboard gives us the ability
to see our Kubernetes information
over a time series, grouped by cluster, node, pod,
and container.
We're on the Infrastructure tab, which is a perfect view
to get more details on the CPU and memory utilization,
and if there are any issues with the nodes.
The Services tab gives the same information,
but from a service-centric point of view--
useful when there's a service disruption.
There's also a workload tab that gives you an app-centric view.
This view is a helpful way to get an idea of what's going on,
but we can also get more detail by using the Metrics Explorer.
Here, we can create custom charts with options
to filter, group, and aggregate.
For instance, we can create a chart
that would let us see the CPU usage time by container.
You can even create a custom dashboard
that shows more custom charts.
If you use Cloud Logging and Cloud Monitoring to provide
observability into your apps and infrastructure,
you're only paying for what you use.
However, the more that they log and the longer you keep those
logs, the more that you'll pay for them.
Plus, the more external and custom metrics you have,
the more you'll pay for those, as well.
There's a link in the description that
gives a great overview and lots of tips on cost optimization,
specifically for logs and monitoring.
Also, we talked about multi-tenant clusters before.
And if you're using those, you probably
want to make sure that you're sending the logs back
to the tenants for their usage.
The cost savings tip here is to use exclusion rules
so that you can filter out the logs you don't care about
before you get charged for them.
There's another link in the description
that goes over setting that up.
OK, so we've seen the built-in dashboards,
but that might not even be enough.
If you want to get even more granular,
you can enable GKE Usage Monitoring
to set up automatic collection of metrics,
and then export then the BigQuery.
Once in BigQuery, you'll be able to see specific usage of CPUs,
GPUs, TPUs, memory, storage, and even network egress.
These details can be used to compare
actual usage versus allocated resources,
and then you can figure out where
you might be over-allocated.
Following the best practice of splitting teams or apps
by namespace also gives you the ability
to easily see the resource usage and cost for each one.
Labels can also be used for filtering.
The most important thing is to consider the culture
suggestions from the last episode,
and help make teams aware--
and even accountable-- for the costs that they incur.
Remember, high cost doesn't automatically
mean that something is wrong, but you
don't want to pay for things that you're not using.
Now that we've gone over the monitoring side,
let's take a look at some ways to take action
on those insights.
With the ability to see exactly how resources are being used,
you may also want to use Kubernetes Resource Quotas.
Resource Quotas let you cap the amount of resources
that a namespace can use, which is another great reason
to use multi-tenant clusters to split up by namespaces.
This can help you keeping any tenant
from consuming too many resources, which might then
trigger auto-scaling.
Limit ranges are another way to enforce resource limitations,
but they're actually at a pod and cluster level.
The real value comes from understanding detailed metrics
and using those to set appropriate quotas.
If you're using a version of Kubernetes
that supports a metrics server, there's
some good things to know.
The metric server helps collect and then expose metrics
to the Kubernetes metrics API.
When an application is using a lot of resources,
the metrics server can be used to figure out
when you should auto-scale.
That's great, but the metrics server
is still a deployment itself, which means that it actually
needs to be running somewhere.
As a cluster gets more nodes and deployments,
the metric server needs to grow along with it.
However, when it's time for the metric server to grow,
it has to be restarted.
If your cluster is changing quickly,
this could mean a lot of restarts, which could then
cause a small delay in auto-scalers,
and even small delays in auto-scaling
can add up to big impact.
So the best practice here is to make sure you
work with your metric server.
Newer versions of GKE support a resize delay
to help keep your metric server from restarting as often,
make sure that your version supports the scale-down delay
to avoid too many restarts, and ensure that you monitor
the metric server as a top priority
to avoid potential auto-scaling issues.
Another way to avoid potential issues
is to make sure that any configuration
changes are reviewed before they get deployed.
So if a team unexpectedly updates a config
with 1,000 replicas, you can deal with it
before it spikes your costs.
The Anthos Policy Controller is one way
to help automate this by checking, auditing,
and enforcing those policies that you've created.
These policies can help you control security, regulations,
and custom business logic that might
affect your costs or your application stability.
In addition to this, you can use a tool like KPT
to build this logic into your CI/CD pipeline,
so you're automatically validating any configurations
through the development cycle.
It's also a good idea to keep your eyes on the Recommendation
Hub.
All kinds of recommendations come up here,
from security to cost.
These are based on your usage, so there
are specific suggestions for how you
might be able to make changes and might
be able to save money.
For example, based on your usage patterns,
you could optimize your compute costs
by purchasing long-term commitments.
We'll talk about that more in the next video.
The key here is having visibility
into what resources your deployments, namespaces, apps,
whatever are actually using, and understanding how
you can optimize around that.
On the next video, we'll talk more
about optimizing your costs based on the machines that
actually run the applications.
Remember to check out the link in the description
for the full guide on GKE cost optimization.
[MUSIC PLAYING]