- Hi, my name is Klass Langhout,
and I work in the Azure Storage team.
I'm really excited today to talk to you about scaling
your globally distributed workloads with Object Storage.
In this session, we'll cover an overview of Object Storage,
talk about some key workloads and customer examples,
do a short demo on NFS V3,
switch gears and cover Azure Data Lake Storage,
which is built directly on top of our Object Store,
and then touch on the WANdisco LiveData platform for Azure.
So in Tad's session, you got to hear
about the entire storage portfolio.
In this session, I'm diving deep on Object Storage
which is Azure Blob storage and Data Lake storage, Gen2.
If we look at the larger environment
there are some specific projections to keep in mind.
The first is the astronomical explosion of data.
By 2025, projections are that we will hit
175 zettabytes of data,
roughly quadruple of what it is today.
And one zettabyte is equivalent to 1 million petabytes
to give you a slightly different way
to think about that number.
The second factor is that the majority of this data
is unstructured.
Providing the right platform to store, manage,
protect and extract additional value from this data is key.
Now looking at what the transaction rate is
that we processed in Azure Storage today.
During the 30 minutes that this session is projected to take
we will have processed 360 billion transactions.
So I first want to touch on what unstructured data is.
Basically not structured like a database with a schema.
Unstructured data includes types such as files
whether it produced by an end user
or a system such as a change order or an invoice.
Media including audio, images and video,
IOT and device data
whether from an autonomous car sensor array
or the thermostats we have in our homes.
And lastly, backup an archive data
from both on-premise produced as well as in the cloud.
So I first wanna dive into what Azure Blob Storage is.
It's highly scalable.
Being able to store tens of petabytes in a single account
and provide multiple redundancy choices to protect
from drive, zonal or even regional failure.
We also include security features
including AED authentication, policy config,
network security controls to ensure protection of your data.
We also have costs down to $1 per terabyte a month,
coupled with features
such as last access lifecycle management
where I'm just gonna be moved to lower cost tiers
that are not frequently used.
Besides management life cycle,
we also include features to make it easier
to understand what your data state looks like.
Even if it's billions of files
slash objects within a storage account.
And last we include premium tier
which is SSD based for your most demanding workloads
including HPC and analytics.
So now that I've about what Azure Blob Storage is,
let me talk about what it's good for.
The first is cloud native applications,
applications designed in the Azure cloud
from the ground up that make use
of the unstructured data storage capabilities of blob.
The second is Machine Learning and AI and HPC
even though there are some differences
they both have a need for high throughput, high capacity.
And some cases, low latency that can also be enabled
through tools such as HPC cash or blog views.
The fourth one is backup
and archival workloads where you have lots
of cool data that you wanna maintain
for a longer period of time.
And lastly, with analytics
which we enable through ADLS Gen2
which sits direct top of our object store.
Now I've mentioned the $1 per terabyte a month,
a couple times,
so I want to provide a picture
of all the different tiers we provide.
On the far right we have our two low cost tiers,
Cool, which is online
and Archive which is offline.
So the cost of Archive is so low
that you can replace your on-premise tape libraries
with Azure archive storage.
Which helps a lot
because you don't have to worry
about upgrading your LTO tape versions
or dealing with the hardware mechanics
of those tape libraries.
And you can lower the costs even further
with reserved capacity,
which can lower the cost up to 38% more
when you purchase in blocks of 100 terabytes
or one petabyte with one to three year duration.
And I previously mentioned lifecycle management.
So an example of that
would be to set a policy where after seven days
of not being read,
a blob or file can be moved
from Hot to Cool.
And then after 188 days of not being read
it could be moved from Cool to Archive.
And then after seven years deleted from archive
and that policy is completely
in your control as far as what those durations look like.
Now, the two tiers on the left include premium,
where premium is SSD based
which provides very low and consistent latency
and much better small object performance than Hot.
Now, for workloads which are very high
as far as transactions per second
Premium will also in some cases be cheaper than Hot
while providing that better latency
and that better throughput and performance.
So I did wanna touch on
three of the newer features we have coming out.
The first one is NFS V3 protocol support
on top of our Blob Storage.
NFS V3 enables both lift and shift
of on-premise applications that use NFS today,
as well as HPC workloads
which commonly also use that same protocol.
I also wanna add that NFS V3 is part
of our multi protocol abilities
that we're building directly
into our object store.
We don't have different containers or buckets
with different protocols
but we have the same data at rest
where you can access them concurrently
with these different protocols,
reducing the number of silos that you have to manage.
The second feature I wanna talk about is Index Tags.
Index Tags allows the user to add up to 10 indices per blob,
and then do a query against those indices
to return a subset of the blobs that are in their account.
So you can imagine that in the past
many customers would have a solution
with a separate catalog
or database to track their blobs
as well as be able to search against that catalog
to return a subset.
For some of those workloads
you no longer need to have that separate catalog
because you can create directly
into our object store.
The third feature I wanna mention is Inventory.
With Inventory,
you can set an optional filter
and then also specify which attributes you want to include
in your inventory.
And then the system will automatically generate
that inventory file once a day
in either a CSV or parquet file format.
This inventory file can then be used with Synapse
and other tools to produce statistics
such as the number of files in a container
or the capacity within a folder.
So it gives you a lot more functionality
and a lot better control to understand
what your data real estate actually looks like.
Now I wanna segue
into looking at some specific customer examples.
The first set of these are our backup partners.
These are six of our largest backup partners
who entrust their customers' data into Azure.
Azure Blob Storage provides an off premises
trusted cloud store for their customer's data.
Now I wanna take a deeper dive
into one customer, Net-Inspect.
So Net-Inspect has built one
of the most used enterprise quality management systems
focusing on aerospace companies
such as Boeing, Rolls-Royce, Honeywell and Raytheon.
Now, when you think about it
airplane manufacturing is incredibly complex,
including up to 26,000 suppliers in 60 different countries.
Now, when they moved to Azure Blob Storage
they imported hundreds of millions
of documents into Blob Storage
for the sole repository for all static content.
And they were able to do that based upon
the encryption at rest, private endpoints, network firewall
which made securing their workload easy.
I'll also add that they use Azure functions
as part of their workload
to perform data import, data export,
email notifications,
and other components of their end-to-end workflow.
With that, I want to do a quick demo of NFS V3.
So here I have a portal instance
that's looking at the Azure instances that I manage.
And I have a specific storage account
called nfsdemo that I've created.
Now, if I look at the nfsdemo account,
I can see that NFS was already enabled
because today you enable it
at storage account creation, not afterwards.
Now NFS V3 protocol has limited authentication
or security built in.
So we do require network-based security.
So in this case,
I have created a virtual network called demo.
And so my NFS endpoint will only be accessible
over this virtual network.
Now, in a storage account,
you can have one or many containers
and every container that's created
ends up being an NFS export
that can be mounted by an NFS client.
So you could have many NFS exports.
Now I've also created two VMs, one Linux
attached to my virtual network.
And then I've also created another VM
which happens to be Windows,
also connected to the same virtual network.
Now that I have these two VMs,
I can access my NFS export within those VMs.
So here's my first one, which is my Linux VM.
And so I've SSH in
and so now I do mount command
to mount nfsdemo dot blob dot Core dot windows dot net
to my Linux VM.
I can now go directly into that directory.
I see that there's nothing there.
So now I'm gonna go to my Windows VM
showing the concurrent protocol access,
and I'm going to use the Azure Storage Explorer tool,
which I've already authenticated to my storage account.
And I'm gonna pick a local file to upload.
And so now after I've uploaded this file,
I can go back to my Linux VM.
And just for grants also make a directory
and I have to do a pseudo.
So let me do that again.
Now I can do a directory listing
and I can show both files.
Now, the last thing I'll do is from this Linux VM.
I'm gonna also run a application called FIO
or File Input Output testing
which will actually do multiple read-write attempts
directly against the storage account
across four different processes.
So it's more of a stress test.
And when I do that
I also see that file show up on the Windows side as well.
So in this demonstration
I showed the concurrent protocol access
between NFS as well as the blog rest protocol
from two different VMs
running within that same VNET.
Now I'm going to move on
and then start talking about Azure Data Lake Storage.
Azure Data Lake Storage basically enables analytic workloads
which provides Better insights
across your unstructured data
especially when that data estate grows.
Now, the fact it's built on top
of our object store provides low cost
where you pay for what you use
and you can scale to petabytes.
And it's very open in that,
we enable the HDFS protocol
as well as the Blob Rest protocol.
And now in the future of the NFS protocol
where you don't have separate silos
to access that same data.
So I want to provide a typical workflow
for analytics and Azure.
This shows on the left
both structured and unstructured data
which is imported into Azure Data Lake Storage
which is the foundation for all analytic workloads in Azure.
Whether you then transform using Databricks
or do advanced analytic queries using Synapse
or visualize through Power BI.
It is the foundational component that drives all
of those experiences.
So I wanna talk a little bit
about what makes a Data Lake Storage platform great.
First, it's the massive scale,
the high performance,
and the durability that is required.
Secondly, it's the requirements to secure and protect
and make sure that the data is compliant
with all certifications necessary.
The third is the cost structure.
And Data Lake Storage
being built directly on top of our object store
provides that low cost performance.
And lastly, it's the fact that
it supports all the different analytic engines
whether it be Hadoop, Spark, or others
that customers want to use.
As well as the other Azure services that we support
through our multi-protocol capability.
So I want to focus on a few customer examples
for analytics workloads.
The first one is Rockwell Automation,
which is a world leader in industrial automation
and information employing over 22,000 people.
They're present in more than 80 countries
and focus on industries
such as oil and gas, mining,
food and beverage and automotive.
And they had a desire
to empower their business users
to have a data-driven mindset.
So they created a unified platform
for all their analytics needs, optimizing storage compute
and scale while reducing their TCL.
The process they use was to ingest all of their data
into ADLS Gen2, using Azure Data Factory.
They then did transformation
and curation using Azure Data Bricks
and then loaded the data into Azure Synapse Analytics
as well as using Azure Data Bricks, Azure AI,
and Azure Cognitive Search
for other analytics scenarios.
Now, the second customer I wanna talk about is Smiths.
Whether you've traveled by air,
had a doctor visit or used your phone,
or even heated or cooked in your house,
you've probably interacted
with a device or solution from Smiths.
They had multiple businesses
with separate data repositories.
Which required manual extraction
when they wanted to provide insights
across those businesses.
What they did is they consolidated
by ingesting into Azure Data Lake Storage, Gen2
using Azure Data Factory.
They then transformed the Azure Data Bricks
and then consumed via Power BI.
Now with Marks and Spencer,
their scenario was to pull parquet files
from their on premises HDFS clusters into ADLS Gen2
and then use Data Bricks to transform
and Synapse to run visualizations and create a dashboard.
Lastly, there's ABN AMRO,
which moved from a centralized model
to a data mesh architecture based upon ADLS Gen2.
Lastly, I want to talk about a partnership we did
with WANdisco to provide a turnkey solution
for Hadoop oriented Data Lake migrations.
The WANdisco live data platform for Azure.
With the live data platform for Azure,
you can deploy
and manage your Data Lake Migrations
using the same Azure management experience
that you enjoy today
through our portal or command line experience.
You can begin your Data Lake Migration in minutes
and not weeks or months
like you would with other solutions.
Most importantly,
the live data platform for Azure
allows you to begin migrating
your Hadoop oriented Data Lake
into ADLS Gentoo with zero business downtime.
The live data platform is integrated with Azure,
provides the same metered Pay As You Go billing model
as other Azure services.
Which means you will also get it
on the same as your monthly bill
and the same way that you can track
and monitor your usage through time.
Please check out the details in the Azure marketplace.
And additionally,
I wanna underline
that the first 25 terabytes migrated is free.
So there's three Takeaways I wanna underscore.
The first is that Azure Blob Storage
and Data Lake storage Gen2,
powers Azure unstructured and Data Lake Storage
Providing durability, security
and extensive data management and cost control features.
The workloads that it provides support for
includes everything from cloud native
to analytics, HPC, ML, AI
as well as backup and archive workloads.
And lastly, we have flexibility and consumption
where you can do PayGo
as well as reserve capacity
to pre-purchase to lower your costs even further.
And use the features that I mentioned
such as lifecycle management
which can lower your costs significantly.
Thank you for joining me today
to learn about Azure Blob Storage
and Data Lake Storage Gen2.
Here's some links to learn more.
Thanks again.
And I'm off to see if we cleared 360 billion transactions
during the session.
We are just getting started.