(upbeat music)
- Hi, my name's Scott Jeschonek,
and I'm here to talk to you
a little bit about the storage options for HPC workloads
and specifically how you might use Azure to accomplish that.
To keep things simple,
I'm going to jump in and talk a little bit
about the various storage that you would consider
with a high-performance computing workload.
A lot of times you might put all of this
into a singular system,
but depending on the scale and the needs
of the particular aspects of your workload,
you might need a variety of tools.
And that's what this chart spells out for you.
For example, looking at the top left,
you might have a particular scheduler you use,
like LSF or PBS or Slurm.
And a scheduler typically deals with a lot of small files
and a lot of transactions,
and has to be very responsive to the HPC environment
as clients come online and do their jobs and mark them done.
You may want to consider using a database,
or you might roll your own scheduler system on a VM
with a local disk
to get the best performance.
That particular behavior might not fit as well
with some of these other ones.
So tools and tool chains,
this particular one might be,
let's say you have a very large
high-performance computing workload.
Let's say you're doing some sort of simulation workload,
and it's a lot of VMs or a lot of cores,
and you have to read libraries and source data in
as quickly as possible across a large number
of these virtual machines.
And you have to get that done very quickly
so that the job can start and the job starts
in a reasonably non-staggered way.
So you might need something that will give you scale
with very low latency for those reads.
And you may typically want that particular data
to be on a shared storage environment,
depending on the way that you're sourcing the tools.
For example,
perhaps the tools are sitting in a shared library
that gets updated from time to time.
So copying all of the tool files to the local disks
on all the VMs in the HPC cluster
can still be accomplished,
but you have to be cognizant of the change potential
that could occur,
and maybe you lose track of a change.
In terms of Scratch base,
ultimately,
that might be something in the middle of your workload
that the individual HPC cluster nodes
have to write out results that then are re-read back
into the job,
perhaps not even by the same node, but by another node.
And the ability to write that data out very quickly
and perhaps then have that data being concurrently accessed
by multiple nodes is very important.
It does also imply that there are many circumstances
where Scratch base might be local
to the individual HPC client
where they're reading what they're writing.
But there may be situations
where that becomes a shared store requirement.
And so you have to be aware that the data
might be accessed by other nodes.
Checkpointing might mean writing out the state of your job
at any given time
and subsequently, if there was an interruption,
being able to recover from that place.
And so that's very write-sensitive and you might consider
some sort of shared storage environment for that.
And then after you run your job,
you have to push the results out to something.
The results may be consumed immediately.
Say you're in a precision medicine environment,
and you've done some analysis,
and you're working with a patient,
this is the future.
You may have run some analysis,
and now you have need to consume that information
to help inform what decisions you're making
about a particular medication.
But maybe it's,
I need to put that data in a shared environment
where my researchers can then access it.
And so being able to put it somewhere that is accessible
is really important,
and it does vary depending on the various workloads.
I mentioned some life sciences scenarios there,
but it could be I'm a hedge fund,
and I'm doing Monte Carlo simulations,
and I just need that to go as fast as possible.
And then I really want to put it one place,
and I don't care about it after that.
Ultimately though, let's say that I am in life sciences
or autonomous vehicles.
I may have quite a lot of data,
and I need to put that data somewhere,
and I'm probably over time
going to want to not use a lot of that data.
Meaning that I have to keep it, but I may not access it.
And so that's where the notion of a tiered data lake
comes into play.
I would want to put that data in something that's vast
enough that I can store a lot of data on the order of
petabytes, but then ultimately,
I may percolate certain data up into a hot tier to read
down into an archive tier where I may never access it again.
But that archive tier is very inexpensive.
And so you have these various needs
in an overall complex HPC environment.
Frankly, the HPC aspect of this might be in the center,
and then there might be activity going on before
and after that particular job is done
that involved databases and other things.
So when you're making decisions
about what particular storage to use,
these are the kind of factors that go in
and the drivers that ultimately help you
make your storage choices.
Chief among them,
what are my performance and scale expectations?
Am I going to require a lot of concurrent access?
Let's say I'm running a small simulation
with a single VM.
It's not really a big consideration,
but if I'm doing a serious HPC workload with thousands of
cores and they're reading a lot of data,
and I need them to read that data quickly and run very fast,
then I'm going to care about the access
and throughput considerations.
Importantly, the larger the job gets,
the more important this concept of latency is.
So this is really down to,
let's say I have a thousand cores,
and they're all trying to access a particular piece of data,
and once they access that data,
then they can go on running the job.
If there is any interference or any slowness
or back-off in reading those particular files,
that could then serve to slow the job down.
It could cause some staggering in the job
that I'm not looking for
and then contribute to potential failure scenarios
in that situation.
File sizes and quantities.
My file sizes could range from a multi-terabyte
file that has a lot of genomic information in it,
all the way down to billions of 1K files,
depending on what it is that I'm doing.
And so it matters what I'm looking at
and how I'm going to read that data.
And ultimately, it matters a lot what I'm writing out to.
If I'm writing a very large file sequentially,
that's a vastly different use case
than if I'm writing out a lot of very small files.
And then all of that conspires
into what we call the "job runtime,"
which means that my storage choices
will help drive how fast my jobs will run.
And I have to assess whether or not the runtime matters
to my underlying business need.
Then ultimately, the rest of these factor in.
Access really boils down to what access methods
am I comfortable with
in terms of, for example,
disk means I'm copying data onto individual disks
in the HPC cluster.
There's a lot of use cases
where that is perfectly appropriate.
And Azure has managed disks
that you can attach to your VMs
that can help facilitate that.
But it might also be a shared NAS
where it's unstructured data,
so it wouldn't fit in a database properly,
and I need to read that data quickly
or write data out.
That might be the fit.
Blob, which is Azure's object storage environment,
that's the vast, cheap, and deep storage mechanism.
I encourage you to look at Azure Blob,
and it has this capability called Data Lake Service.
This is a namespace that you can put on top of your objects,
which means that you can get protocol access
to your objects
using things like HDFS or using NFS.
Then ultimately, things like protocols are going to matter
in the sense of,
let's say I'm running a workload in my datacenter,
and I want to put it up into Azure,
and let's say in my datacenter, I'm using NFS.
Can I use NFS in the cloud?
That's what this really comes down to.
I have to consider
if I'm running a Lustre environment in my datacenter today,
is there a capability in Azure
that would allow me to do that?
We do have tools such as
a very, very large Cray cluster store capability,
if you have a serious workload of that size.
But we also do have a lot of roll-your-own options
in terms of building BeeGFS and Lustre
and other capabilities.
So I encourage you to look at that, as well.
And it always boils down to costs,
which in that previous drawing,
I may have tiers of storage,
and I may also have different methods of access,
and the cost might vary with those different methods.
So to sum it all up,
you could take that same picture that I showed you before
with those different characteristics,
and you can apply different capabilities in Azure
to all of these things.
So in the case of scheduler,
I might use managed disks on a machine
that's running the scheduler,
which is perfectly fine.
Or I might use Azure's SQL DB service,
or I might roll my own MySQL,
it's really up to you.
And I might use something like shared storage.
In case of tools,
I might use something like local disks or managed disks,
or I might use Azure NetApp Files,
or we have a tool called HPC Cache
that you might want to look at if your data is still on-prem
and you want to serve that up into Azure compute.
In terms of Scratch base and checkpointing,
I might use Azure NetApp Files or Azure Files
or other capabilities like that.
And then I might want to consider
using a data lake,
and the data lake would be based on Blob.
I might use multiple tiers of storage
based on access and cost.
And from that data lake, I'd be able to serve up
and move data into these different environments
to leverage based on my needs.
So this is a way
that you could look at the storage architecture
that suggests different tools.
I encourage you to go to azure.com
and look at the storage options,
and you can see how these various capabilities
map to our services and products.
Thank you very much.
(upbeat music)