Hello, my name is Kathryn Hurley.
I'm a developer programs engineer at
Google for Compute Engine.
In this session, I'm going to show you some tips and tricks
for working with Compute Engine's metadata server.
But first, what is metadata?
Every Compute Engine instance has metadata
associated with it.
Metadata is essentially a key with an associated value.
Instances have a set of default metadata values,
custom metadata assigned by the user, and metadata
inherited from the containing project.
Using metadata, an instance can programmatically get
information about itself, such as its zone, machine type, or
host name, information about the owning project, such as
the project ID, or custom data
provided by you, the developer.
For example, if you're working with a Hadoop-like cluster,
you can use custom metadata to specify which node is the
master and which nodes are the workers.
Metadata is divided into default and
custom metadata values.
Default metadata is defined and set by
the metadata server.
You cannot change the default metadata values.
Examples include the project ID, the instance zone, and the
instance host name.
On the other hand, custom metadata allows you to define
arbitrary string key value pairs for
your instance or project.
We'll take a look at how to do this later in this video.
All metadata is stored on the Compute
Engine metadata server.
The metadata server is only available from
Compute Engine instances.
The route URL of the metadata server is http://metadata/
computemetadata/v1beta1/.
Metadata entries are available as subpaths
under this root URL.
Instances have access to these entries by making an HTTP
request to the metadata server.
Metadata entries are divided into directories
and keys, AKA endpoints.
Directories contain other metadata entries, while a key
maps to a specific value.
You can distinguish between the two by the trailing slash
in the name, which indicates a directory.
Metadata entries provide information about both the
project and the instance.
All project metadata entries are relative to the path
http://metadata/ computemetadata/
v1beta1/project/.
Examples include numeric-project-ID, which is a
key providing the numeric project ID of the project
owning the instance, and attributes/, which is a
directory containing a list of custom metadata values
assigned to the project.
To see a full list of project metadata entries, see the
documentation here.
The metadata server also provides
information about the instance.
All instance metadata entries are relative to the path
http://metadata/ computemetadata/
v1beta1/instance/.
Examples include machine-type, which is a key providing the
machine type of the instance, zone, which is a key providing
the zone of the instance, and attribute/, a directory
containing a list of custom metadata values assigned to
the instance.
To see a full list of instance metadata entries, see the
documentation here.
Now that you have an understanding of the metadata
server, let's take a look at how to use it.
Throughout the remainder of this video, I will show you
how to retrieve default instance metadata, including
the machine type in zone, return all metadata from a
directory, set custom metadata using gcutil when creating an
instance, use custom metadata in a startup script, and
update metadata on a running instance.
Let's take a look at how to get default metadata from the
metadata server.
I'm going to first start up an instance using gcutil.
gcutil is a command line tool that allows you to manage your
Compute Engine resources.
In order to use gcutil, I'll switch to my terminal window.
To start up an instance, I run gcutil addinstance
metadata-demo.
I select a zone, machine type, and image.
Keep note of the zone and machine type I'm selecting--
us-central1-b and n1-standard-1.
At this point, gcutil waits until the
instance is up and running.
When the instance is running, I can ssh into the instance
using gcutil ssh command.
Let's get the instance machine type from the
metadata server now.
I run curl using the roots metadata URL, plus the
instance directory, plus the machine type key to get the
value of the machine type.
You can see the machine type in the output here.
Now, let's get the instance zone.
You can see the zone in the output here.
Let's take a look at how to get all the contents within a
metadata directory.
We can retrieve all the metadata within a directory by
adding the recursive parameter to the end of
the directory's URL.
For example, let's take a look at the instance metadata by
making a curl request to the instance directory with the
recursive parameter set to true.
This recursive parameter can be added to any directory URL
to retrieve all the metadata content within that directory.
You can also set your own custom metadata.
Custom metadata allows you to define arbitrary key value
pairs for your instance or project.
Custom metadata can be achieved similarly to how the
default metadata is retrieved, by making an HTTP request to
the metadata server.
As an example, let's say you want to
start a Hadoop cluster.
To specify whether a node is a master or worker, you can
create a metadata key named role, where the values are
either set to master or worker.
To do so, I can use the following gcutil commands.
Notice an important flag here, the metadata flag.
This flag allows you to set a custom
metadata key value pair.
In this case, I'm setting the key role to the value master.
I could instead set this key to worker if I was starting a
worker node.
You can provide the metadata flag as many times as you like
to set multiple metadata key value pairs.
Once the instance has started, I can ssh into the machine.
And let's take a look at the custom metadata value of the
role key using curl.
You can see the value is master.
Now that you know how to set custom metadata, we'll take a
look at how you can use this metadata to introduce some
logic when the instance starts up to determine how the
instance behaves.
We can do this using something called a startup script.
What is a startup script?
A startup script is a batch script that runs when the
instance boots or restarts.
You specify the startup script for the instance when the
instance is created via the API or gcutil.
I have a startup script stored locally in a file called
startup.sh.
Here's what the script looks like.
The second line makes an HTTP request to the metadata server
to retrieve the value of the role key and assigns it to the
variable role.
The script then uses a conditional statement to test
the value of role.
If it's equal to master, then I am the master is printed.
And functionality specific to the master node is run.
Note, for simplicity, this is not shown.
If role is equal to worker, then just a worker is printed.
And the worker code is run.
If role is some other value, an error message is printed.
To start an instance with my startup script, I run the
gcutil addinstance command.
Notice the flag, metadata from file.
This flag allows me to specify the startup script.
I set the startup-script metadata key to startup.sh, my
local startup script.
I can also store my startup script on cloud storage.
In this case, I changed the flag to just metadata and used
the key startup script URL, and specified the URL of the
file on cloud storage as the key value.
You might be interested to know that the startup script
is itself metadata and can be retrieved via
the metadata server.
Once the instance has started, I ssh into the machine.
Let's look at the value of the role key using curl.
You can see that the value is master.
Now let's check the output from our startup script.
To do so, I look at the log file found at
bar/log/google.log.
At the end of the log, you can see that, I am the master was
printed successfully.
I'll skip going through the steps of starting the workers
in the interest of time, since they are essentially started
in the same manner as the master.
Now we'll switch gears and see how to change metadata values
on an instance that's already running.
As a token example, let's change the value of the role
key from master to worker.
I can use the gcutil set instance metadata command to
update my instance's custom metadata.
Here's the syntax for the command.
Note the flag fingerprints.
This is a hash of the metadata.
It must equal the current fingerprints on the instance
before the update can occur, as it
performs optimistic locking.
To get the current fingerprint, I run gcutil
getinstance.
The output displays the current metadata fingerprints.
I'll copy this value to use in the set
instance metadata command.
Now I can run gcutil set instance metadata to update my
role value to worker.
To confirm that this worked, I use gcutil getinstance to
retrieve the metadata.
You can see here that the role metadata key is
now equal to worker.
In this video, I showed you just some of the cool things
you can do with the metadata server, including how to
retrieve default instance metadata, including the
machine type and zone, return all metadata from a directory,
set custom metadata using gcutil, use custom metadata in
a startup script, and update metadata
on a running instance.
There are many more neat features of the metadata
server, including the wait for change metadata URL parameter
and custom metadata at the project level.
Check out the documentation for more information.