PAUL NEWSON: Hi.
My name is Paul Newson, and I'm a developer advocate
for the Google Cloud platform.
Almost all applications need to use
some kind of persistent durable storage to get their job done.
This might be to store information
about user accounts, images you want to capture or serve,
the time series of events that is happening that you
want to analyze, or pretty much any type of data
you can imagine.
Not surprisingly, the Google Cloud platform
can store all these things.
However, because different applications
have different storage requirements,
the Google Cloud platform has not one, but several ways
of storing persistent data.
Today I'm going to introduce each of the storage options
in the Google Cloud platform and talk
about their different characteristics
to give you some idea of which ones
you should use for your application.
Note that depending on your application,
you might use one or several of these storage services
to get the job done.
Let's start our tour in the Google Developers console.
I should mention that we are constantly
improving the Developers console to add new features
and improve usability.
So don't be surprised if the console you see
looks different from the one shown here.
You should be able to perform similar tasks to what
I demonstrate here today in future versions
of the Developer console.
Here we have a new project I recently created.
In the left navigation menu, we see the major categories
of services offered by the Google Cloud platform,
including Storage.
Under Storage, we see four of the storage services offered
in the Google Cloud platform-- Cloud
Bigtable, Cloud Datastore, Cloud SQL, and Cloud Storage.
There is a fifth storage option that is not listed here,
but is instead under the Big Data heading-- BigQuery.
BigQuery is both a storage service and a powerful analysis
service, which is why it is listed under Big Data.
We can divide these five storage services
into two categories-- structured and unstructured.
If the data you want to store can be organized into a table
structure with columns and rows, then it is structured data.
Examples might be user profile information, event logs,
sensor measurements, sales records, or stock trade data.
Structured data comes in many different shapes, sizes,
and usage patterns, and there is a great diversity
of ways to store it and interact with it.
So perhaps not surprisingly, all but one
of the storage services offered by the Google Cloud platform
are for structured data.
Cloud SQL, Cloud Datastore, Cloud Bigtable, and BigQuery
all store structured data.
The remaining storage service, Google Cloud Storage,
stores unstructured data, by which
I mean that you give Cloud Storage a sequence of bytes
to store, a place you want to store it,
and a name to identify that sequence of bytes,
and it stores them for you.
Of course, there may be internal structure
to that sequence of bytes.
For instance, it could be a zip archive
with a table of contents and individual archives inside
of it, or it might be a JPEG image file, which
has a well-defined format, or even a text file in CSV format
organized into rows and columns, which sounds
a lot like structured data.
The key is that Cloud Storage has no insight
into that internal structure.
It just faithfully stores and retrieves
the exact sequence of bytes you ask it to,
regardless of any internal structure
that may exist in those bytes.
At this point, you might be thinking this unstructured data
you're talking about sure sounds a whole lot like files.
And you'd be exactly correct, but Cloud Storage
calls them objects instead not because they're
not like files, but because files are generally
stored in a file system.
And the file systems we're used to interacting
with on modern operating systems generally
have a hierarchical structure and naming conventions
that Cloud Storage does not have.
Thus Cloud Storage calls the place
you store things a bucket and the things that you store,
objects.
Let's use the Developer console to get
started with Cloud Storage.
This project does not currently have any buckets.
So let's create one.
First you have to choose a name for the bucket.
It's important to note that bucket names are
global across all projects.
If you try to create a bucket with an obvious name,
such as Test, you will probably find that the bucket
name is not available.
As you can see, the Developer console helpfully
checks your bucket name while you're
typing it to let you know if it's a legal name
and if it's available.
Now we have a legal and available bucket name,
but we still have a couple of choices to make.
The first is the class of storage to use.
Standard storage is the default, and if you're
using Cloud Storage as part of an application,
is almost always the right choice.
It provides the best latency and highest availability
of the storage classes, which is usually what you want.
If, for some reason, you can't accept a lower availability
for the objects you are storing in this bucket,
you can choose the durable reduced availability
class of storage instead.
As the name implies, objects stored
with this class of storage are just as durable as standard,
but have a reduced availability expectation.
The service level agreement for the standard storage class
specifies 99.9% availability.
The durable reduced availability class
specifies 99% availability instead.
The trade-off for accepting the reduced availability
expectation is a lower price.
Pricing is subject to change.
So you should consult the current price list
for details on the cost of the different classes of storage.
The final class of storage is nearline.
Nearline storage is intended for archival scenarios
where you do not expect to access
the stored objects often.
Nearline storage also carries a 99% availability SLA.
Nearline storage also has slightly higher latency
than standard or durable reduced availability storage
with an expectation of a few seconds to the first byte,
compared with less than a second to the first byte
for the other classes of storage.
Nearline storage is significantly less expensive
than standard or durable reduced availability storage.
But there is a surcharge for accessing your objects.
A good rule of thumb for when to use near line storage
is that if you think you will access an object less than once
a month, then nearline storage will probably be the more
cost effective option.
If you expect to access an object more than once a month,
durable reduced availability storage
will be more cost effective.
The second choice is the location
where your data will be stored.
There are two types of locations available-- multi-regional and
regional.
Regional locations correspond to the region
supported in other cloud platform services
such as Compute Engine.
If you primarily intend to read and write to this bucket
from Compute Engine instances in a particular region,
then setting the location of the bucket to that same region
will provide lower latency and higher throughput
for that workload.
However, by setting the location to a particular region,
you restrict the storage to only that region,
which can be a disadvantage if your objects are
going to be accessed from many different places.
If your objects will be accessed from many different places,
then you can choose a multi-region location
which restricts the placement of your data
to a broader geographic area.
Currently, the choices are Asia, the European Union,
and the United States.
These multi-region locations can also
be useful when you want to ensure your data stays
within a certain jurisdiction.
With these choices made, you can now create your bucket.
Here we see our new bucket, which is, of course, empty.
We could use the developer console to upload objects.
But instead, I would like to quickly show you
how to use the command line utility gsutil that comes
with the Google Cloud SDK.
I have previously installed the Google Cloud SDK
and configured it to point to the project
and use my credentials.
We can use gsutil to see the bucket we just
created using the ls command.
gsutil uses a URI syntax for object names
where the gs prefix specifies objects
stored in Google Cloud Storage.
It can also access files in the local file system and objects
stored in Amazon S3 using the S3 prefix, which
is useful for transferring objects
between Cloud Storage and the local file system
or between Cloud Storage and Amazon S3.
Here I will quickly demonstrate how
to upload a large number of objects in a single command.
And we can switch back to the Developer console,
refresh the window, and see those objects in our bucket.
Now let's take a quick hands-on tour of the structured storage
options starting with Cloud SQL.
When we click on Cloud SQL, we are given the opportunity
to create a Cloud SQL instance.
We choose a name for the instance which
must be unique within this project,
choose a region in which to run the instance,
and a size for the instance.
There are lots of additional configuration options
that I won't go into detail on now.
But the one I wanted to point out is database version.
The important thing to notice is that you are selecting
an explicit MySQL version.
This demonstrates a key point about Cloud SQL.
It is a hosted MySQL service.
It is not similar to MySQL.
It is MySQL.
Whatever you do today with MySQL,
you can do with Cloud SQL.
But Google takes care of keeping the operating system up
to date, configuring application, performing
backups, and other ops tasks.
To underscore the point that this really
is a MySQL server we started, let's connect
to it using the MySQL client.
Let's create a new instance we can
use to run the MySQL client.
Because this instance is part of the same project,
no additional network configuration is required.
However, we do need to make sure we enable access
to Cloud SQL for this instance.
We can do this explicitly in Access and Security,
or we can allow this instance, to access all Cloud resources
that are part of this project.
Since we may use this instance for more than just
running the MySQL client, we will enable access
to all the APIs with this checkbox.
Once the instance is created, we can SSH into it
and install the MySQL client via [? app.get. ?] However, we
need to make a couple of configuration changes
to our Cloud SQL instance before we can connect to it.
First, we need an IPv4 address to connect to.
Then, we need to tell Cloud SQL that we're
OK with connections coming from the IP address of our Compute
Engine instance.
In this example, we only want one Compute Engine instance
to get access.
So we specify a network with a 32-bit prefix,
which effectively means a single host.
Finally, we need an account that allows
us to sign in from our host.
The default root accounts only work from local host.
So we create a new account that allows root to sign in
from our instance's IP address.
Now we are ready to connect to our Cloud SQL instance
using the MySQL client.
At this point, you have a fully functional MySQL
prompt with which you can do all the normal stuff
you would expect to be able to do from a root
MySQL prompt such as creating new databases, creating tables,
and so on.
But now you get the idea that Cloud SQL
is a fully functional MySQL database, which is fantastic
if what you want is a low maintenance MySQL instance.
It has everything you expect from SQL-- a rich query
language, primary and secondary indexes, asset transactions,
relational integrity, stored procedures, the works.
If you are familiar with MySQL or another relational database,
and you value the power of a full relational database,
and if your scalability needs are not too great,
Cloud SQL could be the perfect solution for your application.
However, the so-called NoSQL family of databases
was created to address some of the issues faced
by relational databases when maintaining
relational integrity and full ACID semantics while attempting
to attain massive scale in a cost effective way, which
brings us to Cloud Datastore.
Cloud Datastore was born as the structured storage
solution for App Engine, but is now
accessible from outside of App Engine as well.
Cloud Datastore scales gracefully
from very small database sizes to very large database sizes.
It grows with your application, both in terms of scale
and by being flexible when it comes to schema definition.
For those of you who are accustomed to third normal form
with full relational integrity, you
may find the NoSQL approach to schema definition
to be a little bit fast and loose.
And you'd be right.
It is fast, by which I mean highly scalable.
And it is loose, by which I mean it is highly flexible.
There is no equivalent to a create table
statement in Cloud Datastore.
You simply start storing things.
There is no equivalent to an alter table statement
to add new columns.
You just start storing things with additional columns.
We can demonstrate this in the Developers console.
There are a few important things to notice
about what we're doing here.
While this may look a little like defining a table,
it's not.
What we're doing is storing the very first entity of its kind
in our project's data store.
We can then store a second entity of the same kind,
but we can add new properties to it that
were not in the first entity.
You'll see when we create a second entity it knows which
of the first entity's properties were marked as being indexed
and asks us for a value for those.
But it does not demand that we provide
a value for every property that was in the first entity.
In fact, you can even create entities
that do not have values for properties that were previously
identified as being indexed.
At this point, it bears mentioning
that just because Cloud Datastore doesn't require
you to define your schema in advance
doesn't mean you don't need to think very carefully
about your data model.
You still need to think carefully
about what you're planning to store
and how you intend to retrieve it.
But the NoSQL approach means that when the time comes
to store additional information about a kind, which in a SQL
world would mean an alter table statement to add columns
to a table, you can simply start storing
new entities of that kind with the additional information
you now require.
You'll need a plan to either backfill old data
or deal with entities that are missing that data.
But that's true no matter what your database may look like.
Another thing to notice is that at no point
did we tell the Datastore how many resources should be
used for our structured data.
It is a true no-op solution with almost no operational overhead,
no need to specify instance sizes or cluster sizes
or anything of that nature.
Datastore is a great fit for an application database
that scales from 0 to terabytes of data as your application
grows in popularity.
However, if your scaling needs are truly massive,
there is another NoSQL solution you should
consider-- Cloud Bigtable.
Cloud Bigtable is a hosted version of the same Bigtable
technology that Google has been using internally
for over a decade.
In 2006, Google published a paper
describing Bigtable which resulted
in a flurry of activity in the open source community
to build similar systems.
One of those open source projects inspired by Bigtable
was HBase, which is part of the larger Hadoop ecosystem.
Cloud Bigtable is exposed through the HBase API, which
makes it instantly compatible with a great deal of existing
code written for the Hadoop ecosystem.
But it is a managed service, which reduces
your operational overhead.
If you know you're going to be story
over a terabyte of structured data,
if you have a very high volume of writes,
if you need read and write latency in the single digit
millisecond range with strong consistency,
or if you're already using HBase and you want
a straightforward migration path to a managed Cloud Service,
Cloud Bigtable is something you should look into.
Let's create our very own Cloud Bigtable cluster
to see what that looks like.
We have a couple of choices to make at this point.
First is to choose the zone in which our cluster will run.
Recall that for Cloud SQL, we selected a region.
And for Cloud Datastore, we selected nothing.
We simply started storing data and let Cloud Datastore
worry about where it should go.
Cloud Bigtable is a lower level database than both Cloud SQL
and Cloud Datastore, which is one
of the reasons it can scale so high
and provide such low latency.
But with such low latency guarantees,
your whole cluster runs in a single zone.
The second choice is how many nodes to provision.
As we increase the number of nodes,
we increase the available operations
per second and throughput, and also, of course, the cost.
You can go up to 30 nodes without even talking to us.
As the developer console shows, a 30 node cluster
is capable of performing 300,000 operations per second
with a throughput of 300 megabytes per second.
To put that in perspective, 300 megabytes per second
is about 25 terabytes per day, which
would add up to about nine petabytes in a year.
That's scale.
Similar to how we used the native MySQL
client to demonstrate that Cloud SQL really is MySQL,
I'm going to demonstrate how we can
use the HBase client to connect to our Cloud Bigtable cluster.
Here I am using the Quick Start script
that is described in the Cloud Bigtable Quick Start.
Setting up the HBase client can be tricky,
and the Quick Start packaging script
makes sure you have the right things on your machine
and understands how to discover and connect
to your Cloud Bigtable cluster.
We can see that we don't currently
have any tables in our cluster.
So we create one, then add a row to it,
then scan the table to verify the row is there.
Learning how to model data in Bigtable and use the HBase API
is beyond the scope of this video.
But this short example serves to show
that Bigtable does indeed play nicely
with the HBase toolchain.
Given the massive scalability provided by Cloud Bigtable,
why not make it your number one choice?
After all, everyone wants to be prepared
for the overwhelming success of their application.
So why not choose the most scalable database available?
Simply put, Cloud Bigtable does not effectively
scale down to small sizes.
The smallest Cloud Bigtable cluster you can create still
has three nodes and can handle 30,000 operations
per second, which is far more than a small application needs.
This isn't a problem from a performance standpoint,
but it is a problem from a cost effectiveness perspective.
These three nodes are dedicated for your sole use
and cost the same per hour no matter if you use them or not.
Contrast this with Cloud Data Storage model
where you pay by the operation.
So when your usage is small, so is your cost.
Each of the structured storage solutions
we have discussed so far-- Cloud SQL, Cloud Datastore, and Cloud
Bigtable-- are primarily operational databases.
They're intended to be used as part of an application.
The final structured storage service
we're going to talk about is BigQuery, which
is an analytical database.
The power of BigQuery is its ability
to run SQL queries over terabytes of data in seconds.
To demonstrate, I will use one of the several public data
sets available in BigQuery.
This dataset contains 100 billion rows of page view data
from the Wikimedia Foundation.
By running this row count query, we
can see that there are indeed more than 100 billion rows
in this table.
Now we're going to scan all 100 billion of those rows,
run a regular expression against each one,
then aggregate the page views for each matching row.
And there we have it.
We just scanned over three terabytes of data
in less than a minute.
I hope you've enjoyed this whirlwind tour of the storage
options in the Google Cloud platform.
We were barely able to scratch the surface of each
of these great products.
But hopefully you have a better idea
of which ones might be best for your application.
[MUSIC PLAYING]