Hi, and welcome to the Azure IaaS Day On Demand sessions,
where you can learn about the latest greatest announcements
and updates on storage, compute, and networking.
As part of this session,
we will deep dive on application development
and Kubernetes based workloads.
My name is Vybava Ramadoss
and I'm a Product Manager in Azure Storage For Containers.
With me today is Jason Vallery,
who's also a Product Manager on Azure Storage.
Azure Storage provides cloud storage that is durable,
highly available and massively scalable.
We offer a wide range of storage services
to handle any of your data needs.
So you can choose the storage type
that best matches your workload.
We have three key types of storage.
One is Block Storage that is served by Azure Disks.
Next up is Object Storage that is provided by Azure Blobs
and our Data Lake capabilities.
And last but not least, we have File Storage
that is provided by Azure Files and Azure NetApp Files.
When we think about application development,
we see three common patterns.
First, we're going to talk about
containerized workloads that is leveraging the agility
and scale of containers.
Second, how you're going to use storage
to enhance your web and mobile applications.
So whether these are new or migrated
from on premises to the cloud,
and then driving innovation with serverless computing
and how storage can play a foundational role
in these applications.
First, let's start with containers.
Azure offers a plethora of options
for you to run containers.
These include Azure Kubernetes service,
Azure Red Hat OpenShift, Azure container instances,
as well as Cluster API for custom Kubernetes deployments.
Over the last few years,
we are seeing a significant increase
in the stateful workloads that are run on Kubernetes
at production scale on Azure.
These include databases like MongoDB and Postgres,
big data applications like Cassandra,
machine learning workloads, like Apache Spark or cube flow,
and other applications like Kafka, CMS technology,
CIC systems and so on.
These workloads use different types of storage
including Block, File and Object storage options
that are exposed by a CSI drivers
that are deployed and configured
on the Kubernetes environments.
When it comes to these stateful workloads,
there are four key workload considerations to think about.
One is resiliency of your application
as well as the infrastructure.
Azure Storage plays a key role
by providing the best data durability in the cloud.
We also offer capabilities like ZRS storage
that allows you to have Cross zonal resiliency.
Second, Performance Scale and of course, price performance
options that allow you to meet your workload needs.
So whether you need sub-millisecond latency
or extremely high throughput,
storage can always support you
with your different workload requirements.
Next up is operational simplicity,
so that you can manage your environments
on an ongoing basis, you know, after deployment.
These could include capabilities like automatic scaling,
data protection, as well as integrated monitoring.
And last but not least,
Built-in security so that you can use encryption-at-rest
and encryption-in-transit
or ensure that you always have secure access
to your endpoints.
We talked about a number of different storage services.
And all of these storage services
can be used with containers on Azure.
Let's look at these one by one.
Azure disks offers block storage
that it can be used to run databases
like MySQL, or applications like Cassandra or Kafka.
In addition to that Azure Files offers
fully managed file shares
that you can use both your Windows
and Linux based containers.
These can be used for shared configuration
or user workspaces
which typically required read write access
or multi-part access.
Azure Blobs provide a unique mechanism
to allow you to mount them as file shares
via either Blobfuse or NFS version three.
Now these can be used for analytics based workloads
or HPC workloads as well as machine learning applications.
And last but not least,
we enable Azure data files,
which is a NetApp offering in Azure,
but offered as a native service in Azure.
Now, you may be running NetApp environments on premises
with your container based applications.
You can now do this on Azure
and leverage as NetApp files via the NetApp Tried drivers.
In the last month or two months,
we've announced a number of key capabilities
on Container storage.
First up is the General Availability of the CSI drivers
for Azure Disk, Azure Files as well as Azure Blobs in AKS.
These include critical functionality like volume snapshots,
as well as support for the zone redundant storage.
In addition to this,
you can also leverage larger file shares with Azure Files,
as well as take advantage of the price drop
and reservations for Azure Files.
Not only that, we've announced the general availability
of NFS 3.0 on Azure Blobs,
as well as the preview of NFS 4.1 on Azure Files.
As an example of one of the announcements,
let's take a look at Zone Redundant Disks.
ZR disks can be used with multi-Zone AKS clusters,
and it synchronously replicates data
to all the different zones in a specific region.
This provides you cross zonal resiliency
for your application.
You can try this out today in four regions,
including West US 2, West Europe, North Europe,
and France Central.
Now let's take a look at this in action.
Here you have a Kubernetes cluster
that is deployed on West US 2.
And as currently running Kubernetes version one dot 21.
Let's quickly go over to the node pools and take a look.
Here we see that we have a node pool,
which has three nodes that are up and running.
Quickly looking at the node pool,
we also see that it's configured to deploy nodes
to zone one, zone two and zone three in West US 2.
Coming back, let's take a quick look at the workloads.
Choosing the Stateful Sets tab,
we see that a Stateful Set is currently running
in this environment,
which is a MySQL application.
This particular Stateful Set has a part that's running,
which is MySQL zero.
And this specific part is running on node two
within that agent pool that we just looked at.
This also is leveraging a persistent volume
with a PVC of data MySQL zero.
Let's take a quick look at this volume.
So if I put "kubectl get PVC,"
I will see that this volume is bound to the application
and this is the actual PB that is sitting behind the PVC.
Now, quickly looking at this PB,
we see that it is a disk as we expect,
as well as it is using the disk Q premium SSD ZRS.
Given that the pods are running on agent pool node two,
we see that this PVC is managed by node two as well.
Now what we're going to do is have this pod move
from one node to another,
which basically means it's actually moving
from one zone to another.
And to illustrate this quickly,
what we're going to do is issue a drain operation
on the node,
which will cordon off node two
on which this application is currently running
and move the pod over to a different node in the cluster.
We see here that the pods are getting evicted
since the node was cordoned off.
And it will now move the pod to a another node
in the cluster.
Oh, if we refresh the specific pod,
we see that the updates going through,
the pod is terminating
and it will refresh and come back live
on another node in the cluster.
Okay, we see that the pod was evicted
and now has moved to node three.
Now what we expect now is that the volume
that was available is now going on node two
is now going to be available on node three.
So, you see that the same volume
is now available instantaneously on node three.
Now just to summarize what we looked at here,
we have a multi-zone AKS cluster
with pods running in a single zone.
We then move that part from one zone to another
and the volume instantaneously shows up in the other zone.
We are able to achieve this by using Zone redundant disks
that synchronously replicate the storage
across all the zones in the region.
Now in addition to platform capabilities,
we have a number of integrated ISV solutions
from our partner ecosystem.
These solutions provide storage management and orchestration
for SQL applications,
as well as In-cluster,
Cross-cluster and Cross-region disaster recovery
with application awareness.
These solutions are tightly integrated with Azure storage
to ensure that you have a seamless end-to-end experience
when you leverage them on Azure.
Now with that, I will hand over to Jason
so we can learn more about modern apps
as well as our serverless technologies.
Thanks Vybava for that great overview
of Stateful containers on Azure Storage.
Now we're going to transition gears
and look at how we can leverage Azure Blob Storage
to build, deploy and manage
your modern web and mobile applications.
I want to review with you the landscape of services
that exist in Azure
that support just that.
App services is a fully managed environment
for the service side portion of your application.
You can deploy code that's written in common languages
like .NET, Java, Python, Go and many others,
and host them in a fully managed way.
And then there's Static Web Apps.
Static Web Apps provides you the capability to host HTML,
as well as your media assets like images and videos
and even client side applications built in JavaScript
with popular frameworks like Angular and React.
We have a Content Delivery Network
which can sit in front of these applications
that allow you to scale quickly to millions of users.
And we have API Management,
which provides you a gateway
that you can put in front of your APIs
to publish them out to your end users.
In addition to those capabilities,
we also have services that you can use to search
and enrich your applications based on the data within them.
Cognitive Search allows you to index
the contents of your files and search through them.
Cognitive Services allows you to process those files
and enrich them with an additional intelligence,
and maps and communication services
provide modules that you can include in your application
that allow your users to have more interaction.
Well, Azure Blob Storage is foundational
to all of those services.
It provides the scalability, the durability,
and the high availability of your data
to make all of those services work.
It provides you best in class security features
to ensure that you're compliant
with the most strict industry and governmental regulations.
And it does that in a cost effective way
with Pay As You Go billing that scales with you
as your application grows.
On top of all of those capabilities
as a Comprehensive data management framework,
which allows you to apply policy
and get the most from the platform.
This is foundational
to the most demanding workloads on Azure.
But in addition,
we have some unique capabilities that are worth looking at.
Our Premium Blob Storage tier
gives you the lowest
latency object storage platform in the cloud,
which is crucial for your interactive applications.
You don't want your users waiting for their data to load.
We also have Point-In-Time-Restore,
which allows you to roll back your entire data state
to a known good and time
with just a few clicks when the unexpected occurs.
And lastly, all of this is seamlessly integrated
with all of those great platform services
without requiring you to wire up custom
and complex integrations.
And how do you get your data to Azure Blob Storage?
We provide for you storage SDKs.
These are code solutions
that are built by Microsoft and Open Source
in the common languages, again,
like .NET, Java, Python, and others
that allow you to take advantage of all the best practices
of our storage platform.
But if that's too much,
you can interact with our platform
using just simple HTTP calls.
PUT and GET allow you to upload and download files.
And you can even do that from the command line with curl.
And when you're looking to migrate your data,
AZCopy provides you an end-to-end data moving application
similar to Robocopy.
We have PowerShell commandlets,
and Azure CLI binaries that allow you
to insert the data movement into your scripts
for deployment or other common tasks.
And if you want to use your interface on top of all of it,
Storage Explorer provides a great way
to go and manage your data and explore it.
In addition, even an Azure Portal makes it simple
for you to upload and download data directly
to your Azure Blob Storage account.
With that, I want to take a minute
to look at some of the common web and mobile patterns
we've seen with our customers.
There's no shortage of the imagination of our customers.
But I want to really focus in on three common scenarios
that we regularly see.
In addition to the obvious of just storing
and serving your data and fronting over the CDN.
Some emerging patterns include directly uploading
to Blob Storage from any internet connected device.
This could be a sensor or camera,
this could be a mobile application,
or this could even be a retail or point of sale device.
With our secure delegated
authentication and authorization model,
you can upload directly to Blob Storage
and know that it's been done by a trusted client.
And once that data is in Blob Storage,
you can use our multi-protocol access
to processing enriching.
In addition to cognitive search and cognitive services,
you can access your data
with any application that supports the NFS V3 Protocol,
or with our Hadoop driver for HDFS.
You can access any of the analytics tools
in the Hadoop ecosystem, including Spark and Hive.
I want to discuss one customer that did just that, Chipotle.
When you go to chipotle.com, to order a burrito,
you're experiencing a single page application
that was hosted inside of Blob Storage
and fronted with our CDN.
And when you go to order that burrito,
all of this service side logic
is in a .NET core application running an app services.
And then when you go into the store
to order a burrito at Chipotle,
those point of sale devices
are uploading your order to a Blob Storage account
so that Chipotle can have a single pane of glass
to the overall health of their business.
I want to spend a little bit more time
going deeply into cognitive search.
Azure Cognitive Search provides the capability
to index the contents of your files and search them.
It has indexers that are aware of common file types,
like images and videos,
as well as Office document formats
like Word, Excel, and PowerPoint.
It can extract from them text and other metadata,
like who created the file,
and even applying optical character recognition
on your scanned documents.
One customer that used these capabilities
to build a compliance and archive solution
is the Accounting Firm PwC.
PwC needed a solution to make sure
that they were compliant with industry regulations.
And they use cognitive search
to analyze all of their documents
and a rich capabilities and Blob Storage
to enforce the industry rules
that they need to be compliant with.
Now I want to shift gears a little bit to Serverless.
Serverless is a capability
that allows you to loosely couple your applications.
But what does that mean?
It means taking small components of the application
and running them in their own separate runtime
that can scale based on user demand.
Blob Storage is great at this.
You can have events that trigger these serverless functions
by when an object is created, deleted, or other events
that happen to your data natively.
Those get routed to an event router,
and then the runtime processes.
In the Azure serverless landscape,
we have a variety of services
that you can use to help you accomplish this.
Azure Functions is the runtime
where you can deploy your .NET Core, Node, Java
or other code
to a managed environment that will respond to those events.
Logic Apps provides you some pre-built modules
that allow you to compose them together
in a low-code way to solve business challenges.
And Event Grid manages the routing
to these different runtimes.
Azure Container Instances allows you to deploy containers
to Azure without managing
the underlying infrastructure, like AKS.
It is perfect for long running tasks.
Let me walk you through one common scenario
of processing and uploaded video.
When the video file has finished uploading
to Azure Blob Storage,
it will create a Blob created event
that gets routed to Event Grid.
Event Grid is responsible for filtering that file,
filtering that event and notifying the downstream function.
Azure functions has code in it
that then notifies Azure Video indexer,
which is a capability of our cognitive services
to do frame by frame analysis
and metadata extraction from videos.
Simultaneously, another message was routed to my queue,
that queue was being monitored
by a number of Azure Container instances
which had transcoding software running
inside of the container.
The transcoding was preparing that video file
for web and mobile delivering.
A customer who had exactly this scenario was Cloud9.
Cloud9 is an e-sports company who hosts
massive video game tournaments.
And when their players come to play a match,
those matches are recorded,
and then uploaded to Blob Storage.
After the matches are over,
they transcoded the videos using a similar pipeline,
and made them ready for web and mobile delivery,
as well as indexed them to get
as much intelligence as they can
about what happened during that match.
And with that, I want to invite you
to review some of our container storage documentation
as well as our overall Azure Storage docs.
And then we would like your feedback
specifically on Container storage.
If you go to aka.ms/ContainerStorage survey,
we'd love to hear from you.
And if you're interested in data protection for Kubernetes
we'd like you to participate in that survey
AKS backup for you.
If you have any questions or you'd like to reach out to us,
AskContainerStorage@microsoft.com
and AzureStoragefeedback@microsoft.com.
I want to thank you for attending today's session.
And I recommend you look at all of the IaaS
Storage Day sessions on Storage
and hear from some of my coworkers
on the great innovation happening on our platform.
Thank you.