Hi, I'm Tanuja Korlepra, Principal Product Manager
for Azure HPC and AI storage
and I'm here to talk to you today
about our storage services for high performance computing
and artificial intelligence workloads.
Azure HPC and AI platform is purpose-built
for workloads across industry verticals,
like automotive, finance, energy and life sciences.
We understand that these are some of the most performance
and scale-demanding workloads that our users have.
And so, we are building the capabilities
for them to address those business needs in Azure.
We have built an entire stack of HPC and AI resources.
For Compute, we have H-Series VMs
that are optimized for intensive computation,
and N-Series for GPU accelerated workloads,
and even offerings like Cray supercomputers
that are running in Azure.
We've also added best-of-breed networking capabilities,
including express route, that can create private, reliable,
and fast connections between your on-prem infrastructure
and Azure data centers or within Azure data center
itself, using the fastest InfiniBand networking.
We have combined
these with highly performance storage offerings,
including scale-out, NFS cache, NetApp NAS offering,
object storage and parallel file systems like Lustre
on Azure and in Cray ClusterStor.
With these storage compute and networking services,
we have also built higher level services
to orchestrate your workloads
and reduce the technical complexity, making it
so that running your workloads in Azure
is as simple or even simpler than running those on-prem.
Then finally, to the intelligence we want to add on,
we have built transformative services
such as Azure Machine Learning, Azure Data Lake,
Azure ML Compute to allow you
to drive new innovations and new value for your businesses.
At Azure, our vision for storage
is to provide an intelligent data platform
that allows every organization to extract maximum value
from their data assets.
I'm going to start with a little bit on various storage
that you would consider for HPC workloads.
Now, different workloads have different requirements
for storage, scale and performance.
For example, you might have a scheduler like LSF
or PBS dealing with a large number of small files
and transactions, or you might have tools
for simulation workloads with lots
of VMs that read a shared set of files.
You might have HPC clusters writing out the results,
and then re-read those back
into the job needing concurrent access across nodes.
In some cases, you might have a lot of data,
most of which you might not use over time,
so you need an option to tear it down
to low-cost storage options for long-term storage.
So then you are making decisions
about which particular storage to use,
there are some primary factors to consider, including
the performance and the scale expectations,
and also the need for concurrent access.
For example, if I'm running a small simulation
on a single VM, it is not a big consideration.
But if I'm running a large workload
with 1000s of cores reading terabytes
to petabytes of data, I'm going to care about the access
and throughput considerations.
Aspects like protocol matter
when considering migrating your workloads
from on-premises environments to Azure
or for tools and applications
that are optimized for a particular protocol.
Like for example, parallel file system like Lustre.
And finally, the total cost of storage
and the complexity of using the storage also matters.
For some simplicity, if we broadly categorize
the performance requirements very broadly
for HPC and AI workloads across industries,
we observe a few prominent data access patterns
and requirements for storage.
Typically, high-throughput requirements
are for workloads like seismic processing for oil and gas
or for weather modeling workloads for gov
that perform sequential access
on a set of very large files for simulation jobs
that run for a very long time, sometimes even months.
These required throughput standard
in several 100 gigabytes per second.
The high-IOPS and low-latency requirements
are for workloads like electronic design automation
also known as EDA, for manufacturing
or for seismic interpretation for energy,
and these are the workloads that require concurrent access
to millions and billions of files,
or typically small files across hundreds
and thousands of nodes.
Now, these workloads require a very high IOPS
and sub-millisecond latencies concurrently,
which is a key element.
In this session, we are going to discuss
in detail storage services,
including Azure Blob Storage, Cray ClusterStor,
and Lustre Filesystem on Azure
for high throughput workloads
and Azure NetApp Files, Blob views,
and HPC Cache for high-IOPS workloads.
The first storage service I want to talk to you about
is Azure Blob Storage.
Azure Blob Storage is the foundational data store
with exabyte scale and high performance
that makes it possible for customers to run applications
in HPC, AI ML, analytics and much, much more.
It is a tiered storage
with premium tier offering low latency
to archive tier offering the lowest cost storage options.
Azure Blob Storage is the only object store
that offers true multi-protocol access.
This means Azure Blob Storage REST API,
Azure Data Lake Storage API, and NFS
can access the same data in a single-storage account.
This eliminates the need to make copies of data
for multiple tools, and thus eliminating data silos.
All of this comes with the foundational value
of using Azure Blob Storage,
which is industry-leading data management,
durability, availability, security,
and the ability to easily replicate data
within a region or across regions.
With the NFS v3.o support for Blob Storage, Azure
is the only public cloud object storage
offering native file system compatibility.
With this feature, you can simply mount
an Azure Blob container via NFS v3 protocol.
You can run your high-throughput, large-scale,
read heavy workloads in autonomous driving,
in genomics and workloads like rendering for media
and entertainment that are designed for NFS on Azure.
All of this data is stored as regular Azure Blob data
and is built at the exact same price as Blob Storage.
The flexible pay as you go Blob Storage pricing model allows
for many cost optimizations, and hence the overall low TCO.
Also, there is no minimum provisional capacity required
for any of your performance requirements.
You can also achieve the maximum throughput
for a storage account with either a single file
or a small number of files.
As I mentioned earlier,
with the multi-protocol access support,
all of the data is stored as Azure Blobs.
So, you can mount the data via NFS, or access it via REST
or Azure Data Lake Gen2 APIs.
You can also ingest a large amount of data
to Azure using tools like AzCopy,
and then run your existing on-prem NFS workloads in Azure
without any application refactoring.
Because NFS support in built on top
of Azure Blob Storage,
the rich set of Blob Storage capabilities
like data management, security and scale
are all available to data with NFS support.
For AI and visualization workloads
that require low-latency access
to locally cached data, BlobFUSE is an ideal solution.
It is a free, open source tool developed to provide
a virtual file system backed by the Azure Blob Storage.
BlobFUSE supports familiar POSIX commands
like open, close, mkdir, reads, and writes for Blob data.
It also allows multiple nodes to mount the same container
and locally access the data for low-latency access.
In addition, it also supports parallel uploads and downloads
from the Azure Storage account, offering data access
at a very high throughput.
The next service I'm going to talk
about is Azure NetApp Files, also known as ANF.
Azure NetApp Files is a Microsoft fully managed,
first-party NetApp ONTAP service.
We have worked very closely with NetApp
to offer this tightly integrated service and experience,
and it is sold and supported by Microsoft.
ANF makes it easy for customers
to run the most performance-intensive
and latency-sensitive workloads
in the cloud, including workloads like EDA
for manufacturing, risk workloads
for finance and seismic interpretation for oil and gas.
It also offers an easy lift and shift migration
of on-premises workloads that are using on-prem NAS
without any significant code changes.
With this offering, we have extracted away
all the complexities of deploying enterprise storage
like software and firmware upgrades,
and laying out disks and transformed it
into an easy pass service.
Customers get an Azure consistent experience,
including access through the Azure portal, CLI, PowerShell,
and REST API.
Support for ANF is provided through Microsoft
and through your existing Azure support agreements.
Similarly, ANF is consumed in build same
as any other Azure resource.
With ANF, we offer the widest choice of protocols.
We started with NFSv3 and have since added SMB3 and NFSv4.1.
NFSv4.1 brings file locking capabilities implemented
in the protocol,
which is highly valuable in large cluster deployments
that many of our customers have.
In addition, dual protocol access with NFS4.1 and SMB
and NFSv3 and SMB is supported to eliminate data silos.
To ensure ANF can service a wide range of use cases,
it offers a range of tiers, with each tier defined
by a ratio of throughput per capacity.
Ultra tier, that offers up to 128 megabytes per second
per tebibyte is an ideal tier for most IOPS
and latency-sensitive workloads.
Now, let's talk performance.
There are two ways to achieve
the throughput limits on ANF.
In the automatic QoS that we originally launched,
the gross performance is the product of the tier
of the pool to which the volume belongs
and the volume quota.
With this, you can get up to 4.5 gigabytes per second
and IOPs as high as 460,000 from a single volume
in the Ultra tier.
This year, we launched manual QoS,
where the throughput limit
can be defined individually per volume.
Because of this, you do not have to over-provision
the volume quota if you need higher throughputs.
The next service I'm going to talk about is Azure HPC Cache.
Azure HPC Cache is a high-throughput
and low-latency file system cache that is designed
for network file system loads.
It can allow access to data sitting
on the on-premises NAS to Azure Compute
to allow for the compute jobs to run on Azure VMs.
It allows you to even plug in to the Azure NAS,
such as Azure NetApp Files and Azure Blob
all in a single namespace,
which is quite a differentiated offering.
It easily scales from 100s of cores
to 10s of 1000s of cores
and it's easy to use with your familiar Azure PIs
and the protocol.
A large amount of data
for legacy HPC applications is on-premises.
Azure HPC Cache works great
for cloud bursting and hybrid,
which could be a combination of on-premises
and cloud storage HPC workloads.
Now, there are multiple skews and cache disk sizes
to choose from, to meet your performance requirements
in an optimal cost.
Standard cache, that's two, four and eight gigabytes
per second throughput offers read caching
and read/write caching.
A high throughput cache on the other hand,
with 4.5, nine, and 16 gigabytes per second throughput
and sub millisecond latency offers read-only caching
for performance demanding workloads.
To make it possible for our customers
to run the most performance-demanding
and long-running running HPC applications
in Azure, Microsoft
and Cray got into an exclusive partnership
to offer super computing capabilities in Azure.
Cray ClusterStor is a Lustre-based, single-tenant,
bare metal and fully managed HPC environment
in Microsoft Azure
It is available in three configurations, small, medium,
and large, depending on your workload requirements.
It is worth noting that Cray ClusterStor comes
with a five petabyte and three-year minimum commitment.
Given that customers pay for the commitment upfront, Cray
is ideal for workloads that are long running,
perhaps for several months at a time,
and the ones that demand very high performance
for workloads like seismic processing
or weather predictions and modeling.
With hierarchical storage management or HSM support,
the data from Cray ClusterStor can be teared down
to Azure Blob Storage for cost optimizations
and long-term storage.
Now, many of our customers
require high-performance Lustre Filesystem
to run their most demanding workloads.
Azure HPC Lustre Filesystem
is an Azure marketplace offering to provision
a Lustre Filesystem using the Azure portal very quickly
and very easily.
Using the flexible configuration options,
you can within minutes,
automatically deploy ephemeral VM disks
and persistent managed storage,
either standard or premium,
to meet the exact performance requirements
of your workloads.
It also creates a dashboard to show you the CPU, network
and disk stats for both metadata
and object storage server, to give you an overview
of your entire file system performance.
These metrics can highlight any bottlenecks
for the chosen configuration, either network, or disk,
or cached, uncached bandwidth to Azure Storage
for performance tuning that you can perform.
Like ClusterStor, the Lustre offering uses HSM
to import the data from Azure Blob for file system creation.
The standard Lustre HSM commands can be used
and the HSM restore command can be run
to hydrate the whole file system.
Also, the data transfer between the Lustre
and Blob Storage occurs in parallel
across VMs for faster access of the data.
And finally, we do believe that we need
to bring our ecosystem along for our customers
to be able to realize the performance and benefits of Azure.
We have a rich HPC storage partner ecosystem,
with ISVs like NetApp, Qumulo, DDN, IBM, Hammerspace,
and Excelero to make it possible to meet
all of our customer performance and scale requirements.
I do also want to share
some of our customer success stories.
BOSCH has been running
their autonomous driving re-simulation workloads fully
on Azure Blob Storage.
Their work on automated driving
and driver assistance requires processing
of a large set of data for their dev fleet.
Using NFSv3.o support,
they shortened the feedback loop
by running these large scale simulations faster,
more efficiently, and at a lower cost.
REPSOL wanted on-premises-like performance
for their reservoir simulations and analytics workloads
that required high throughputs.
With Azure NetApp Files,
they were thrilled to see the performance
that was over five times their on-prem performance.
And, Jellyfish Pictures ran
their rendering jobs using 90,000 processor cores running
in Azure, with the data still sitting
on their on-premises environments
by using the HPC Cache cloud bursting technologies.
In conclusion, at Microsoft,
we are making significant investments
in developing technology for existing
and rapidly evolving customer workloads in HPC and AI.
We continue to innovate Azure Blob Storage,
Cray ClusterStor, and Lustre Filesystem services
for high throughput demanding workloads
And, ANF, HPC Cache and BlobFUSE
for high-IOPS and latency-sensitive workloads
in addition to the entire HPC and AI platform innovation.
We want our customers to be able to run their tools
and existing stacks on the edge in the data centers
and in Azure seamlessly.
And, our goal is to make sure Azure performance
and scalability is available to all of our customers.
Well, I hope you found this session useful.
Here are some more resources for you to explore more.
Thank you for your time and have a wonderful day.