YURI GRINSHTEYN: Have you ever run
a workload that uses Virtual Machines in Google Cloud?
And while you're enjoying a cup of tea during a quick break,
your alerts are firing, the service is having issues,
and users are really unhappy?
You rush back to your computer and log into the Cloud Console.
The VMs are still up, and system resources are seemingly normal.
What can you do to troubleshoot from here?
Well, you might want to look at a few things--
whether your AP processes are running
on the VMs, how much resources they're consuming,
and whether there are any errors in the logs.
But as you may remember from our previous episode,
you need to have the Ops Agent installed,
running, and collecting and sending
data to get this kind of information from your VMs.
If you ever have issues getting installed,
running, or getting data, you've come to the right place.
My name is Yuri.
I'm an SRE at Google Cloud.
And today, I'm going to show you how to troubleshoot the Ops
Agent to make sure you always have the VM observability
data you need and get back to that tea before it gets cold.
[LIGHT MUSIC]
Let's start at the beginning with the installation.
There are two primary reasons why
you might have trouble getting the agent installed.
The first reason is that you're trying
to install it on an unsupported operating system.
You'll know that's the case if you see an error
message like this, where the installer isn't
able to download the right package for your OS.
The second reason is a configuration conflict,
specifically with the old monitoring or logging agents.
In this case, you'll see an error message like this one.
You can't have both the Ops Agent and the old agents
installed at the same time, so you'll
have to uninstall the old agents first.
Once the agent is installed, you can
confirm that it's running by using the system CDL status
command, like this, and looking for the confirmation messages
that the agent components are started and running.
If the agent is not running, you may need to start it.
If it has issues starting, you may have a conflict
or an invalid configuration.
Let's have a look at how to start the agent
and sort out startup issues.
If the agent isn't running for any reason,
you'll see a message like this, saying
that the service isn't running when you check its status.
Use the Start command to try to get it running again.
You might get a message like this,
saying that either the Cloud Ops Agent itself or one
of its modules failed to start.
The former usually means you have an invalid configuration.
Have a look at the documentation on how to configure the agent
to sort this out.
We'll, of course, link to it in the episode notes.
If a module can't start, it's likely
because they can't send logs or metrics to the API.
Often, this is caused by invalid credentials,
such as a disabled service account or an expired key.
Networking issues and proxy issues,
such as an invalid proxy configuration,
can also cause this.
Once you've verified that the agent is installed and running,
you should be getting its metrics and logs adjusted.
If that's not the case, you'll need to figure out why.
Generally, you'll be investigating
three types of issues.
The first has to do with the agent itself,
the second is with the sources of data
you're trying to ingest, and the third
is with sending collected data to the respective API.
Let's have a look at each.
If the issue you're seeing has to do with logs,
check to see if the logging module is running like this.
If the output is anything other than successful,
you may need to restart the agent
or validate its configuration.
If the logging module is running,
the next thing to check is whether it's
able to read the logs.
One way to check is to use the local metric endpoint,
like this.
If everything looks good so far, but you're still not seeing
the logs you expect, check to see
if the agent is getting any errors from the logging API.
You can do that in the logging module log, like this.
You might see permission errors if the API isn't enabled
in the project, the service account has been disabled,
doesn't have the right permissions,
the service account key expired, or if the agent is using
the VMs default service account, insufficient scopes on the VM
itself.
You might also be out of quota for the API.
You'll need to resolve these to resume log ingestion.
We'll use a similar approach to troubleshoot the matrix module.
Let's have a look.
You can check the status of the matrix module
by looking at module logs and syslog.
If there aren't any logs, the service
may not be running properly.
Otherwise, you can check the logs themselves
to see if there are any issues noted there.
This log is also where you would check
for any API errors, such as Permission Denied or quota
issues.
At this point, you should have everything
you need to sort out agent issues and never be in the dark
about your VMs running on Google Cloud.
Thanks so much for watching today.
Don't forget to like and subscribe to never miss out
on more "Engineering for Reliability" with Google Cloud.
See you soon.
[LIGHT MUSIC]