ALLON COHEN: Thank you for joining.
And welcome to our session on "File
Storage for High-Performance Workloads."
I'm Allon.
And I'm joined today by Gilles from our Google Silicon Team.
And we'll be presenting to you "File Usage
for High-Performance Workloads."
Before I begin, a little bit about myself,
my name is Dr. Allon Cohen.
I'm a 15-year veteran of the enterprise storage industry.
And I recently joined Google through the acquisition
of the Elastifile.
Elastifile developed the technology
that is at the base of many of the new capabilities
that we'll be presenting to you today.
So let's dive right in.
File storage essentially powers many high-performance workloads
in Google Cloud and throughout the industries,
whether it's rendering of the latest blockbuster
movies, discovery in life sciences,
including drug discovery for dealing
with the latest pandemics, or any such industry,
such as silicon and EDA.
But the truth of the matter is that file architectures are not
a one-size-fit-all technology.
Essentially, if you have a small workload,
you can use a local file system that provides you access
to one VM accessing that data.
But as you grow your systems and want to share more data,
shared file systems allow you to essentially connect
multiple VMs or containers to a single file system
through simplified interfaces and protocols such as NFS.
But as your needs grow and as you need more and more
compute power, it is not just enough to scale compute.
You also need to scale the back-end file system to support
those demanding workloads.
And that's where you need scalable file systems, which
also come in two flavors.
Parallel of file systems allow you
to connect using plugins on the client side
to multiple back-end storage nodes,
while scale out file system essentially retains
the simplicity of shared file systems,
allowing you to connect through protocols such as NFS,
while at the same time assuring that the storage
at the back end can also scale and match
the performance requirements of your front-end systems.
But moving HPC workloads to cloud
is not just about performance there
is a collection of features that you require to easily migrate
your workloads into the cloud.
Beyond the high capacity and high performance,
you want the elasticity to grow and shrink
on demand to really benefit from what the cloud provides.
You want to retain the simplicity of your local file
system and the way you use your scale
out file systems within your environments.
You want to be able to provision on demand.
You want to be able to integrate with cloud schedulers
so that you do not need to manually manage
the environment.
And you want that file system to maintain natively
all of the attributes of the file systems, such as locking,
that you used within your on-prem environment.
That is the reason that we are excited to introduce
to you Filestore High Scale.
Filestore High Scale is a new tier of Google's Filestore
product that is essentially a managed elastic scale out file
service that is designed to match the requirements of HPC
and HTC workloads.
With Filestore High Scale, we are extending the capacities
of the Filestore you're used to to capacities
of up to 320 terabytes.
We are supporting a background scale out architecture.
We are allowing you now to both grow
and shrink your environments for high-scale instances.
And we expand the performance of Filestore
to over 16 gigabytes per second, or more than tenfold
of our previous capabilities.
In terms of IOPS, you can now reach hundreds of thousands
of IOPS.
And even more so, and more importantly,
for HPC environments, you can now
connect thousands of front-end machines
to the back-end storage.
With these new capabilities, essentially, Filestore
enables you to migrate your workloads into the cloud.
You can now run all types of HPC workload
with a native NFS file system inside the cloud,
connect to it using either the VMs or containers.
And we'll even demonstrate to you
shortly that you can use preemptible VMs
and save considerable costs on your compute-side storage.
You can now support life sciences workloads,
manufacturing, finance, and more.
To demonstrate to you how such a file
system is used in production, I'm
going to turn it over to Gilles to present
to us how they use this technology for their EDA
workload.
GILLES-ERIC DESCAMPS: Thank you, Allon.
Hi, everybody.
I'm Gilles.
I've been lucky to have been able to implement
world-class networking, CPU, GPUs, and mobile SOCs
at different companies.
I've also been into the software industry, supporting enterprise
and personal batch queuing software.
And now, for two and a half years,
I've been at Google, implementing
world-class silicon.
Our needs are those of a typical HPC/EDA customer.
We rely on a lot of third-party commercial software
that we assembled into flows, each one
with hundreds of steps.
Our users use mix of interactive tool sessions and also batch
workloads across thousands of machines.
They share their data by using large data files that they want
to share with their colleagues.
And our application is not a web application.
We cannot afford to lose a single job.
Otherwise, it may hit the scale up to two weeks, for example.
Finally, we operate worldwide in multiple continents.
Our initial assessment of Filestore in 2018
before we took that journey was, it
look good that it was simple to instantiate.
It was compatible with Terraform,
so we could use infrastructure as code.
It was managed, so we didn't not have
to spend time watching how is the machine performing.
Somebody else was guaranteeing us that.
We were using 20-terabyte machines.
And Filestore offered up to 64 terabytes.
So that was great.
And it also grows online.
So we could start with small machine
and add more capacity as needed.
The problems we identified were, was the file system
a server file system, or just a file system?
And in the end, it's managed.
So we didn't care about those implementation details.
The big one which was initially throttling us was there
were no user or [INAUDIBLE] quotas.
So how do we make this work?
We decided to allocate one per user.
And finally, we identified that it was just
nice to have to have the feature of reducing online.
Our cloud architecture is based on using
one cloud project per chip.
Each user gets a virtual Desktop into the cloud,
which is basically a RHEL 7 machine inside your web
browser.
Those can be quite large, with 96 CPUs and 2 terabyte of RAM.
All the design files are stored on the NFS Filestore server.
We allocate at least one per user.
And we rely on that amount to enable collaborations,
so that each person can see other users' file.
And finally, we deployed the batch-queuing system
so that we can maximize license usage.
And the machines in the farm auto-amount the needed data
from Filestore.
So how did it go?
We started back in 2017 with 50-gig machine.
And Filestore turned out to be easy, reliable, and scaled.
We went in production with a beta service.
We started with a dozen Filestores.
And we ended up growing way past our plans and expectation.
So we have to say, thanks to the cloud
for delivering to us unlimited and instant capacity.
We ended up reaching close to 500 Filestore servers,
totaling 4.2-petabyte capacity, worldwide.
And so despite having 500 file servers, it was easy to manage.
Between Terraform and the management of Filestore,
it was great.
It was also very reliable.
We never lost a single byte, despite having 4.2 petabytes.
We had a couple of incidents.
And they were mostly related to performance
of temporarily unavailable data for a few hours.
The great thing that we discovered with Filestore
is, you all know what is the typical sysadmin duty.
A system administrator has to ensure that one user cannot
bring down other users by abusing the systems.
And so at other companies, with a traditional file server
appliance, I have personally experienced the overload
of a file server when an HPC application deploys
across thousands of cores.
So it's very difficult to identify
where the load is coming from.
The NFS protocol has no quality of service built in.
So Filestore gave us the I/O quality of service
we did not know we needed.
It solved by construction [INAUDIBLE]..
There was no way a single user could
impact the work of other users.
And finally, performance--
Filestore was able to provide enough IOPS and capacity
for most of our application.
Only the static timing signoff, which
runs across hundreds of corners, was too much for one Filestore.
So we mitigated this performance bottleneck
by manually sharding the runs across multiple Filestore
per user.
Now we are currently exploring Filestore High Scale.
And our users are reporting that they
are liking shared write, the fact
that you can size higher than 64 terabyte,
and that some workloads are running way faster than
on a single Filestore.
Now back to you, Allon.
ALLON COHEN: Thank you, Gilles.
So with that, we would like to actually demo the system to you
and its new capabilities.
We are going to simulate an HPC workload using a Filestore High
Scale instance.
We are going to automate that using Slurm.
And then we are going to spin up thousands of preemptible VMs,
connect them to the back-and Filestore system,
and demonstrate to you some of these supported performance.
The first step is, of course, launching a Filestore instance.
That is very simple, done through the cloud console.
And essentially, it runs in a very similar way to the way
you're used to launching Filestore instances so far.
But we add that capability of launching high scale instances,
allowing you to launch them with up to 320 terabytes of storage
in a single file system.
You can select the region and the zone
where you want to launch.
You can select the network that you
want to use and even control access using certain IPs.
You name your Filestore.
Let's do next2020.
And just press Create.
This is a fully managed system, so the scale out file system
is being launched for us at the back end.
And once it's ready, we simply have
an IP that we can start mounting clients to.
We'll do that now with Slurm for orchestration,
launching thousands of preemptible VMs.
Back in Cloud Console, we can see
that those VMs are launching.
And at the same time, Slurm is now initiating jobs
within each of those clients.
And those jobs connect to the Filestore High Scale
instance at the back end.
These are all preemptible VMs.
So Slurm can launch them while saving over
80% of the cost in terms of the compute for the systems
that you are using.
At the top portion of this screen,
you see the number of connections and the throughput
that is being generated by these workloads
as they run these jobs.
You can see that we are now already
at about 2,000 connections to our back-end Filestore High
Scale instance.
And you can also see that our throughput is reaching
25 gigabytes per second, allowing all of these clients
to run in parallel.
Now, because these clients are running on preemptible VMs,
it may occur--
and at this number, it definitely does occur--
that some of these VMs will actually be preempted.
But that is not an issue for this architecture,
because our orchestration system will essentially just
spin up these VMs again to complete the jobs that they
were running.
All of the data is persisted in Filestore High
Scale at the back end.
So the data is not lost, even though our VM
has been preempted.
So as you can see now, Slurm is relaunching VMs and making sure
that our whole job process has completed.
And that is the tale of completion
that you see at the back of this run.
So what we demonstrate here, essentially we
launched a 320-terabyte scale out file system
at the back end with a series of clicks from Cloud console.
We made sure that we can run preemptible VMs,
and launched, in this case, 2,000 preemptible VMs
against this single scale out file system.
There is no overhead, because with this orchestration,
you can essentially launch the VMs when you need them.
And when they are no longer needed,
you simply spin them down.
Everything is persistent on the file system.
And even the file system itself, you
can scale up and scale down on demand for your needs.
We connected thousands of VMs in parallel.
We showed over-- well over, actually, 16 gigabytes
per second of throughput.
And we completed 100% of our processing.
And all of the data was persisted in our back-end file
system.
If you would like to learn more about Filestore
and its new capabilities, you can check them out
in the links provided in this slide.
But the proof of the pudding is in the eating.
So we welcome you to go to Cloud Console,
launch a Filestore High Scale instance,
and test your own workloads to see the performance that you
achieve.
With that, I'd like to thank Gilles again for joining me.
Thank you all for being here.
And we look forward to receiving your questions
and getting your feedback on usage of the system.
Thank you.