[MUSIC PLAYING]
SPEAKER: In this video, you will learn
how to use Cloud Logging to determine what caused a Google
Compute Engine instance to unexpectedly reboot or shut
down.
With a few exceptions, most activities
related to a virtual machine's lifecycle
are automatically added to Cloud Logging
in the form of an audit log.
Audit logs are easiest way to get insights
about a VM's activities.
When a VM reboots or shuts down unexpectedly,
try doing the following.
Select the VM that shut down or rebooted.
In this example, the VM named My Instance shut down and rebooted
unexpectedly several times in the last seven days.
On the log section of the VM details page,
click Cloud Logging.
Cloud Logging opens with a pre-built query that
filters for the instance ID.
Narrow down your data set using additional filters.
Select the log name dropdown menu
and look for cloud audit category, or search for audit.
Select the activity and system event logs.
Activity logs describe changes made to GCP resources,
such as compute instances.
System event logs describe events
generated by Google systems.
Narrow down the search to an approximate timeframe
when the VM reboot or termination occurred.
In this example, we know the shutdown occurred sometime
in the last seven days.
Each cloud audit log entry displays a compute instance's
label that describes the method associated
with a particular instance and which
principal, otherwise known as a user or service account,
was associated with that method.
The following log entries are activities.
Audited operations that cannot be executed instantly will run
asynchronously, and therefore may appear twice in the Cloud
Logging results.
The first entry represents the request itself,
and the second entry represents the outcome of the request,
such as a success or an error message.
For example, this log entry illustrates that, on April 4,
a stop instance method was issued to My Instance
by the principal user@example.com.
Then, on April 7, a reset instance method
was issued to the same instance by the service account
schedulerbot@example.com.
The following log entries are system events.
System events will always show system@google.com
listed as the principal because these events aren't
driven by direct user action.
This log entry illustrates that, on April 8,
My Instance was shut down by the operating system.
Note that operating system reboots are not tracked
by the cloud audit logs.
If the audit logs do not adequately
explain why an instance was rebooted,
then it was most likely caused by an operating system reboot.
In that case, consider investigating the operating
system logs for further clues.
Finally, these two log entries logged on April 9
indicate that the machine was shut down due to a host error
and was automatically restarted.
Occasionally, there might be problems
with the physical machine that hosts the VM.
This is referred to as a host error
and is indicated by the compute.instances.hostError
method, which means that the VM was terminated.
Some examples of events that might cause host errors
are physical hardware malfunctions
or a natural disaster that impacts
the location of the Google Cloud data center.
After a host error, the VM is set to restart automatically,
which is indicated by the compute.instance
s.automaticRestart method.
To protect your applications and services
from these potentially disruptive system events,
Google has shared several best practices that
explain how you can design robust systems in Google
Cloud, use patterns for scalable, resilient apps,
and encourages using managed instance groups
when appropriate.
In addition, Compute Engine also offers live migration
to keep your instances running during a maintenance event.
There are two methods which might appear after a Compute
Engine maintenance event.
The compute.instance s.migrateOnHostMaintenance
method indicates that the Compute Engine
used a live migration to keep your VM running
during a maintenance event.
Compute Engine live migrates your running instances
to another host in the same zone,
instead of requiring your VMs to be rebooted.
This allows Google to perform maintenance
that is integral to keeping infrastructure protected
and reliable without interrupting any of your VMs.
Live migrations are enabled by default,
except for VMs with GPUs.
But this can be disabled under the availability policy
of the particular instance.
The compute.instance s.terminateOnHostMaintenance
method indicates that a live migration was disabled or is
not supported on that particular VM,
so the VM was terminated during the maintenance event.
For more information on how to troubleshoot VM shutdowns
or reboots and a list of other common methods found
in Cloud Logging, check out the official troubleshooting
documentation with the link below in the description.
Thanks for watching.
[MUSIC PLAYING]