[MUSIC PLAYING]
STEPHANIE WONG: You can't really talk about storage at Google
without talking about how our production systems work.
How are all the hard disks and flash
drives on individual machines connected?
Let's boil it down.
Data is stored in hard disk drives or solid state drives.
SSDs and HDDs are deployed separately
from the applications.
For example, they can be stored in a dedicated storage
tray like this one.
In other words, we separate Storage from Compute.
Data needed by a machine is typically
not even in the same rack because machines
can access data in a different physical location
through our robust global fiber network.
Splitting Storage and Compute lets us scale them
independently as demand grows.
And machines can process requests much more efficiently.
We organize Storage and Compute racks
into rows of physical enclosures,
organizing groups of enclosures into clusters,
and putting multiple clusters in a single data center.
Each cluster depends on their own power, cooling, and network
infrastructure, a deliberate part
of how we design for data protection and reliability
at scale.
We've built our own warehouse-scale machine out
of hundreds of thousands of relatively inexpensive
machines.
At Google, it's rare to dedicate an entire storage
appliance in our data center for storing only one
product or service's data.
Instead, we spread a workload's data across multiple machines,
and workloads share network access to that storage.
Encryption is inherent in our storage systems,
and all data is encrypted prior to being written to disk.
But how does Google make storage accessible and scalable
across a global fleet of machines?
Remember, at planetary scale, it's
not unusual for individual machines, racks,
or even entire buildings to periodically fail,
so we need to build software to make the data stored on them
durable.
That way, when they fail, no data is lost.
Every layer of our storage stack,
down to the file system and the layer that writes to storage
devices, is a shared service.
Services like Search, Photos, and Gmail
workloads share machines at the data centers.
Resources within each machine, like Compute and Memory,
are allocated to each service.
At a machine level, disk, or D for short,
is a system that exposes the hard disk and SSD drives
attached to individual machines to other services
in the cluster.
It manages access to each machine's disk capacity
to maximize utilization.
At the cluster level, most data we store
has high durability and latency requirements,
so we built a file system called Colossus on top of D, which
is the foundation for many services like Cloud Storage
and Bigtable.
At the Colossus level, files are broken down
into a set of chunks that can be stored on different machines
in the cluster.
This data replication across machines
is the key to fast recovery and fault
tolerance against things like network failures.
For a given chunk, Colossus identifies a machine
to write the chunk to, and the client sends the chunk
to the D service that runs on the target machine
to perform the write.
Security is ensured through the encryption of each chunk,
and each chunk has a unique encryption key.
Because a single machine can be running multiple services,
and conversely, a service can be running on many machines,
services are constantly adapting to the amount
of resources they use.
Now you can see how distributed data better utilizes
our machine capacity and gives Google Cloud Services higher
reliability and performance.
For Google Cloud users, you have the option
to store data on local solid state drives,
zonal persistent disks that map to clusters across zones
in the same region, or regional persistent disks
across cloud regions or even storage buckets.
You can store data in a single region for high performance,
dual region for high performance and high availability,
and multiple regions for the highest availability.
Your data maps to different clusters
and protects it from zonal and regional failures.
D and Colossus manage access to the storage, store data safely
and securely, and efficiently use our hardware
so you get the best performance possible.
All of this is possible because of the speed of our network
fabric.
More on that next time on Discovering Data Centers.
[MUSIC PLAYING]