Good afternoon, everyone.
My name is Dominic Preuss, and I'm
very excited to be here today at Google I/O talking
about the Google Cloud Platform.
I'm the product lead for Google's storage and database
products on the Google Cloud Platform.
As many of you know, data is everything,
and it's at the core of everything that Google does.
Whether it be the original BackRub algorithm or some
of the machine learning things you saw in the keynote
yesterday from Sundar, it really is the basis of everything
we do at Google.
So what we're trying to do is provide that platform
that is the best place for you to bring your data to be
able to do similar things to what
Google's been doing internally.
The way that we want to achieve this
is by building a fully integrated data platform
that is converged and makes it as easy as possible
for you get your data into the system, move it, process it,
and enrich it, and then derive value from it.
We don't want you to worry about where your data ends up
or where you decide to store your data.
We want to make sure that data is never in a silo
and that it's easy to do what you need to do
with it to complete your task.
So to achieve this vision, we've built
a suite of storage and database products
for customers to bring their data into the system.
This will give us a little bit of a level set
before I dig into some of the later parts of the talk.
The goal of this talk is threefold.
It's to give you an overview of our storage products.
It's to give you a couple of simple reference architectures
to give you an idea of how you might build a solution.
And then it's to give you a couple of examples
of customers that are currently doing this on Google Cloud
Platform.
So in this portfolio, we'll start all the way
at the object store.
So Google Cloud Storage is our object store.
This is where you put your objects, videos, or data
that you're trying to process via different products
in the platform.
It's used for media serving as well as a collection
point for data before you run your data processing pipelines.
What's interesting about Google Cloud Storage that's
fundamentally different from other offerings
is that it is inherently multi-region.
So when you put an object into the object storage
using our Standard class, your data
is actually replicated across multiple regions.
So it gives you a higher level of availability and ability
to serve that data back out to your customers.
Google Cloud Storage comes with three different storage classes
to match your user need.
The first one that I mentioned already is Standard.
The second one is Durable Reduced Availability.
And what that means is if you're willing to accept
slightly lower availability, we'll give you a better price.
And then our third one that we introduced
in the middle of last year is Nearline,
which basically allows you, for $0.01 per gigabyte per month,
to store your data in the same interface and the same product
that would store your more frequently accessed data.
Moving onto the relational side, Cloud SQL
is our managed RDBMS, Relational Database Management System
service.
So this is where you can go to get a fully managed MySQL
database with high availability and pay as you go semantics,
just like you would if you were rolling it on your own.
But instead of you worrying about the problems,
you can let Google site reliability engineers
worry about those for you.
These are really good if you're-- Cloud SQL is great
if you're building on a framework or have a piece
of software that requires a specific piece of software,
like MySQL, and really makes it easy for you to get up
and running with almost no work.
If you want to go down the NoSQL path,
we have two different offerings.
One is Cloud Datastore.
Cloud Datastore is our application database
that allows you to, from scratch,
just start storing data.
There's no requirements or defining a schema.
You don't have to provision size.
All you do is start storing your data in an unstructured way.
We then make it as easy as possible
for you to get access to that data
and allow you to scale up to any level.
We see this very much used in the mobile and web applications
as the default place that people go,
especially if they're building applications
on Google App Engine.
Our other NoSQL database is Cloud Bigtable.
So we published the Bigtable white paper many years ago
and spawned a lot of the work that's
being done in the NoSQL world.
We've now taken that internal Bigtable service
and externalized it as a service for customers
with a HBase-compatible client library.
So what this allows you to do is to take your HBase and some
of your Cassandra workloads, and bring them over and start
serving them off of Cloud Bigtable in a fully
managed way.
You no longer have to manage your clusters,
your HBase clusters.
You can just let Google manage those for you.
This is very frequently used in heavy read/write workloads.
So we see this in Internet of Things workloads,
collecting sensor data, time series databases,
and anything that has high throughput
and needs low latency reads.
Our last one is for caching.
So if you have an application in App Engine
that you need a caching layer, we
provide a fully managed Memcache service.
So again, this allows you to just use Memcache
without having to worry about how many nodes is it on,
how much storage do you have, how much memory do you have
on the node to be able to fit all of your keys, what happens
when your keys if you have to reboot your node.
All those things are handled for you in a fully managed service.
So now we'll start talking about reference architectures.
And I'm going to start with-- I'm
going to do all of these in two different ways.
We're going to start with the NoSQL world,
and then we'll do RDBMS, and then we'll do NoSQL.
I don't want to start a religious debate about which
is better, so I'm just going to cover both.
If you are working in the relational database
world, the quickest way to get up
and running from your concept of your idea
to a megabyte of data, is either from your web application
or from the web browser client or the mobile client,
to build your application, deploy it on Compute Engine,
and then store your structured data
in Cloud SQL and your unstructured data
in Cloud Storage.
So this way you can put all of your data
that's being generated off of your framework,
whether it be Rails or Django or any
of the other popular frameworks, and then you
can put your static data in Cloud Storage.
Basically, it's a couple of clicks to get these things up
and running, and then you can get--
and start serving your prototype out
to a small number of people.
Now, if you wanted to do the same thing with NoSQL,
the way that we do this is those same web
and mobile applications or clients
that are speaking to App Engine, which
then speaks to Cloud Datastore.
Now, the thing that we announced yesterday
that we're very excited about is adding Firebase support
for Google Cloud Storage.
So now those static files and you're
saving, either from your JavaScript web applications
or your mobile applications, can be
put into Google Cloud Storage via the Firebase set of client
libraries.
If you were doing this yourself, you'd write a few lines of code
in App Engine to be able to serve that information out,
and that data would be stored in Datastore for you.
Again, it's replicated, highly available,
and allows you to start taking your prototype out
to a larger number of people.
All right.
So you've got your prototype, and you've
shown it to a few people, and people like it.
So now, how do you push this into production
so you can start to support a real number of users?
The way that we do this is, we take those same web clients
and point them at Google Cloud Load Balancing.
Google offers a global load balancing solution.
Again, very different than other things that are on the market.
And then you would, behind that, put one or many Compute Engine
virtual machines to actually host your application.
So you would actually be hosting your code
and serving that code out of those VMs.
Again, you're serving your data into Cloud SQL.
You can have a high availability version of Cloud SQL
so that you have high availability
and never lose your data, and it's a single-click to do that.
And again, you're storing your data in Cloud Storage.
So you haven't actually changed anything.
You're using the same set of products,
but you're just provisioning them slightly differently
to make sure that you'll scale up to level of load
that you have.
Now, if we go back to the NoSQL path,
you can see that this is suspiciously
like our last slide.
And again, you have your web clients
in your mobile applications talking to App Engine
or Firebase, and then storing their data in Cloud Datastore
and Cloud Storage.
The beauty of going down this path
is that you can scale from basically zero and your concept
all the way up to terabytes of data, and all of that
happens completely without any intervention on your side.
You just keep throwing data at it
and we'll just keep consuming the data.
Whether it be in Datastore or Cloud Storage,
it's a per consumption basis.
There's no provisioning.
You just throw data at it and we'll go ahead and store it.
Great.
So now we've got this product on the market.
It's in the App Store.
It's doing amazingly well.
App's getting slow.
We need to really scale this out.
How do we support millions of users?
So the way that you would do this
is, if you're going down the relational database model,
is take your web and mobile clients-- speaking to the Cloud
Load Balancer, again, no changes are needed on the load
balancer-- and then you would just
be adding additional numbers of compute engines.
Now, you can use things like instance groups
and some of the other features as well.
But to keep it simple, we'll just
say that there's a number of VMs necessary to serve
your application, and then all of those VMs
and those applications are speaking back to Cloud SQL.
Now, the one thing we've done differently in Cloud SQL
is we've added read replicas.
So within Cloud SQL, you can, with one click,
add read replicas to basically scale out your read loads.
So basically all of your reads in your application
would go to the read replicas, and then you would just
send your rights to the master.
This allows you to continue to scale on top of Cloud SQL
without having to do anything other than add read replicas
and potentially change the size of the VM
that's hosting your database.
The other thing that's very-- is great about the way
that we've set up Compute Engine is that it's very easy for you
to add capacity.
So if you attach a certain number of persistent disks,
which is our block storage solution, to those VMs that you
need to add capacity, you can just
increase the size of the volumes without having
to restart the machines.
We call it hot grow, and it's a way
for people to add capacity to their applications
without having any downtime.
So now I want to start adding in some customers,
just to give you a little bit of context
of how people are using these.
So Ocado is a UK e-commerce provider.
They basically use Cloud SQL as a way
to not have to worry about managing their MySQL instances.
They allow Google to do that for them
and they focus on building their applications.
Applications that are very transactionally sensitive,
whether it be payments or if you have data that it has to always
be consistent, a lot of those tend
to go down the Cloud SQL path.
And so e-commerce and that vertical
tends to be very popular with Cloud SQL.
Now, if we go back to the NoSQL path,
again, this looks suspiciously like our prior slide.
Because to go from literally concept
up to terabytes of data, nothing has
to change in terms of your architecture.
As you add more and more clients,
both Firebase and App Engine will scale up
to serve those requests, and that data
goes into Cloud Datastore or into Cloud Storage
and there's no change necessary.
You can throw as much traffic at it as you want
and they'll just scale up to meet your needs.
This is the promise of the platform, where we can just
respond to the amount of demand that you're giving us,
and you don't have to worry about prepaying for capacity.
You don't have to worry about what the pricing does to you.
You know that if you use the product,
we'll give you your sustained use discounts on the VMs,
and you'll only pay for the data that you use in the database.
So a good example of going down this NoSQL path
is Khan Academy.
So Khan Academy uses Google App Engine with Datastore.
And we're basically able to scale from a small idea
up to a major website without having
to make any architectural changes to the way
that they build their applications.
The goal of all these things is really
to allow developers to focus on writing code and not
managing their services.
So we're trying to basically increase
the velocity of development to the maximum
by removing anything that isn't delivering value
to your end users.
So now we start to get into some interesting areas.
So now we're going past a terabyte of data.
And areas where we start to see large amounts of data
is really in the data aggregation space.
So there's a number of different ways
that you might be collecting data.
You might have a mobile client application
which is speaking back to Cloud Endpoints.
So Cloud Endpoints is our hosted API endpoints
that allows you to offload all the management of your API
to Google managed services.
And then the data that's coming in from those endpoints
can be written into the Cloud Bigtable which
we talked about earlier.
Another way is doing the same thing with the web.
You have a web application.
You're collecting a bunch of stats, time series data
that you need to push back to the Compute Engine, which
then gets collected in Cloud Datastore.
Again, you're just pushing data into Cloud Datastore.
You're not worrying about provisioning.
You're not worrying about, do you have
enough disk in the cluster.
All you're doing is pushing data into Cloud Datastore.
And the third one is the one that we're
seeing over and over again, and one of the things
that I'm most excited about.
We keep seeing, over and over again, people bringing us
really interesting data pipelines
and ways of aggravating and taking value of their data.
So a good example of this is sensor data.
So if you have millions of sensors
out there that are collecting data every minute,
10 minutes, hour, they have to send all that data back
to a single place.
What we do is we've seen people using Cloud Pub/Sub, which
is our messaging queue product.
It's a global product.
It's not zonal.
So wherever your sensors are in the world,
you can just push your data into Pub/Sub.
And then using Cloud Dataflow to then
process that data, either in a batch or stream mode,
to push that data into Cloud Bigtable.
And now we really are seeing people
start to operate the same way that Google does internally.
So internally, we use Pub/Sub for messaging.
We use Dataflow, usually in stream mode and not batch mode,
to then push all this data into Cloud Bigtable.
You're now starting to see people really implement
their data pipelines and their data collection the same way
that we do inside Google, and it's very exciting.
One of the things that we did recently
that I'm very excited about was a collaboration
that we did with FIS that basically they went and took
and created 25 billion synthetic market transactions
and were able to process them in an hour.
And so they were basically able to push
from putting data into Google Cloud Storage,
processing with Dataflow, putting it into Bigtable.
They were able to process those 25 billion transactions
in under an hour.
And at peak, they were pushing 34 million reads a second
and 22 million writes a second using 3,500 nodes of Bigtable.
So they were able to do that without having
to buy hardware, provision it.
We worked with them to make sure that we
had the capacity on hand for them to run the test,
and then they were able to execute that.
And it's something that we're seeing over and over again,
that these companies aren't able to run
at these levels of scale.
Because of just a procurement cycle
that would be necessary to acquire the hardware
to run a test of this size, we were
able to spin up in a couple of weeks.
So now we've gotten up to a petabyte of data.
But it wouldn't be Google if we didn't
talk about exabytes of data.
So how do you go from having a petabyte of data
into an exabyte of data?
And where we start to see this type of volume of data
is really in the archival workload.
So we've got a couple of ways for people
to ingest data into our platform.
So one way to do this is Google Cloud Transfer Service.
So we offer a product that will allow
you to grab your data, either from another Google Cloud
Storage bucket or from Amazon S3,
and transfer that data into Google Cloud Storage.
We will do this on a schedule.
If there are failures, we will retry and make sure
that all the data gets over.
And we'll optimize and paralyze the transfer to make sure
that we do it as quickly as possible.
Another area that we're seeing a lot of adoption,
especially from our enterprise customers,
is in the storage appliance space.
So we have gone out and worked with a number of the storage
appliance companies to partner with them
to create integrations for Google Cloud Storage.
So companies like EMC, NetApp, Veritas, Convault.
And the idea here is that these customers have several 1, 2, 3,
10, 100 petabytes of data that they've been sitting in a data
store that they're paying to buy disks to maintain this data.
And so now, instead of storing this on a capex basis
where they have to buy these disks,
they can transfer it all into Google Cloud Storage using
the same storage appliance that they use today,
and back all that data up into the cloud.
And they've gone from a capex model, where they're
paying for actual disks that they
have to be buying and storage arrays, to a pure opex
model where they're purely paying on a consumption
basis for storing that data.
We've helped companies save millions
of dollars doing this and it's something
that we continue to be very committed to.
Another way that you can ingest your data is offline ingest.
So we partner with Iron Mountain in North America and Prime
Focus Technologies in EMEA and in Asia-Pacific.
And you can actually send them hard drives,
and they will upload those files from the hard drives
into Google Cloud Storage for you.
So if you don't feel comfortable doing
all of that work on your own, we're
working with companies to make sure
that it's easy and secure for you to upload those files.
Now, any talk about storage wouldn't be complete
if we didn't talk about our partnership
and our marketplace.
So Cloud Launcher is where our partners
come to be able to launch solutions into the Google Cloud
Platform.
I've highlighted a number of them.
If what you really need is Cassandra,
we've got a preconfigured image through Bitnami,
one of our partners, to be able to launch a Cassandra into GCE.
Same as EnterpriseDB.
Worked with EnterpriseDB to get a solution up on GCE,
so that you can launch these solutions into the cloud,
so that if we don't provide a managed service that
meets your needs, you can manage your own solution on GCE
with a single-click.
MongoDB is another example.
A lot of interest in MongoDB.
Make it very easy for you to be able to spin up
those instances that are running MongoDB
to support your workload.
Same thing with Redis.
A very popular caching solution.
So we make it one-click possible to kick off
a VM that is a preconfigured version of Redis.
That's all the concepts that I wanted to cover today.
But the key takeaway that I want to make
sure everyone understands is that we
are striving to make Google Cloud
Platform the place to store your data in the public cloud.
We are faster, cheaper, and more reliable
than many of the solutions that are out there.
And then we're being fully integrated to make sure
that we can actually get your data,
move it, process it, and enrich it,
and allow you to drive value, whether that be
through BigQuery or running machine learning models
on top of it, like the ones that you saw at Sundar's keynote
yesterday.
Thank you, everyone.
[MUSIC PLAYING]