Hello everyone. My name is Raji Easwaran.
I am a Product Manager on the Azure Storage team,
and I'm here today to talk to you about
massively scalable storage platform
for your analytics workloads.
So let's get started.
There's various types of storage
that are available to customers today
depending on the type of workload
that you're planning to run.
You've got disk storage or block storage,
if you're using virtual machines.
You've got general purpose object storage and file storage.
What we're going to talk about today
is primarily around object storage.
On Azure, we call that the Azure Blob Storage platform,
data lake storage is essentially a capability
on the Azure Blob Storage platform,
and that's what we're going to touch on today.
Let's understand why analytics is so important.
It all starts with the data.
You've probably heard the cliche saying
that data is the new oil, but it is a fact.
Data is so important for all businesses today.
Your businesses typically run
on the data that you have within the organization
and it's what makes the businesses even possible
in this new digital economy.
Regardless of which industry you're in,
whether you're a retailer,
whether you're in the healthcare industry,
education, it doesn't really matter.
Data is going to be very crucial
for every possible industry type
because all of these industries run on data,
including even consumers like ourselves.
We rely on data.
For example, our Fitbits,
the watches that we wear.
We rely on all of that data
for us to make decisions in terms of what we eat,
how we exercise, how we live.
So data is going to be very critical for us now
and well into the future.
And businesses that don't understand
the value of the data that they have
are not going to be able to survive.
It is a competitive advantage that organizations do have
and they have to leverage that.
So now that we understand
the value and the importance of data,
we also need to understand what you can do with it.
Once you're able to analyze the data that you have
within your enterprise,
you can start now looking at what's the current state
of my business?
How am I doing?
How am I performing?
Your knowledge workers,
or even your C-level executives,
can now start taking decisions,
data driven decisions based on that data.
So you're now more well-informed
when you're making these very business critical decisions.
And last but not the least,
you get to take humans out of the equation.
Decisions that are being made today
that take a lot of time because humans are involved,
can now be replaced by machines.
Machine learning, artificial intelligence,
all of these are going to help you now reduce that time
to the insights that you need to get from the data
that you have.
So let's talk about why data lakes are important
for these analytics and for the decisions
that you need to go and make.
Number one, in order for you to get better insights
from that data,
you need to be able to look
at a diverse amount of data or diverse sets of data.
Structured, semi-structured, unstructured data,
there are so many different types of data out there.
You need to be able to collect all of that data
in large volumes in a single place
such that you can now start making sense
of all of the data together.
Second, since you've got large volumes of data,
you also want to be able to store that at fairly low cost.
And so cost-effectiveness is also going to be a very
integral part of that decision making process.
And last but not the least,
you want to be able to use that data
and you don't want to be pigeonholed
into using a specific set of solutions.
So you need that openness,
you need that broad ecosystem
in order to act upon that data.
Let's now go talk about
what services or what products and capabilities
we have on Azure in this space.
Like we said, there's going to be large volumes of data
in different formats.
You need capabilities to ingest that data
onto your data lake.
Services like Azure Data Factory
is a good example of what you could use
to ingest these massive volumes of data
into your data lake.
You've got analytics engines like Synapse Analytics,
Databricks, or HDInsight
which is open source Hadoop infrastructure on Azure
to prepare that data curated,
aggregated, and also put them in data warehouses.
For example, Synapse Analytics also includes
data warehousing capabilities.
So all of these analytics engines
allow you to process that data
and model or serve that to an application,
whether it's to a data scientist
who's using some kind of an data science workbench,
or someone who just wants to visualize them
using BI reports,
in which case you can use things like Power BI.
And at the very heart of all of this
is Azure Data Lake Storage,
because that is where you're collecting all of this data
super low cost and the platform that can handle
variety of data formats
and also massively scalable platform.
Why is Azure Data Lake Storage
so important for you to build your data lake?
Number one, the scalability and the performance.
And we'll go into a little bit of
the capabilities that we have on the platform
around scalability and performance.
Like I said, cost effectiveness,
object storage prices.
So, super cheap.
Third, you don't want to just
put your data on a platform without having the ability
to govern it,
or provide access control around it.
So we've got capabilities to allow you to
govern access to that data.
And last but not least, Data Lake Storage like I said,
it is the foundation
for a lot of the analytics platforms out there
and so you need something that
is scalable from that sense as well.
Let's take a look at Data Lake Storage
solution for analytics
and why it is differentiated
compared to some of the other solutions out there.
Number one, it is built on top of Blob storage
which is a proven, massively scalable platform,
storage platform that Azure has had
for for more than a decade now.
What we've done on that platform
is just added a few capabilities that are unique
to bring out
the performance for analytics workloads.
So, number one,
we've added a hierarchical namespace,
which is essentially a way for you to organize that data
within the object storage platform in folders and files,
which are typically how analytics engines interact
with that data.
On top of that,
we also have a multi-protocol access platform.
Which means depending on the kind of workload that you have,
if you didn't have this capability,
you're likely going to be using three different services
or four different services
because each service needs to access that same data
with a different protocol.
With this design, you store the data once
and depending on the protocol that we use,
we allow you to access your data using that protocol,
all with the same data, but not having to create any copies.
So that's a huge benefit.
We also have other capabilities that we've added
which is the HDFS compatible driver,
which you will use on your Hadoop clusters,
and a combination of all of this
is what we call the Data Lake Storage capability on Azure.
So now let's dig a little bit into the scalability
and the performance aspect of it.
Scalability, for example,
because you're talking about petabytes of data
and petabytes and terabytes are the new norm.
People don't deal with gigabytes of data anymore.
You need a platform that is massively scalable.
Data Lake Storage is a massively scalable platform.
Number two, you don't want to be thinking about
pre-provisioning that capacity
and having to deal with
adding capacity when your workload scale.
All of the scale is on demand.
We will scale as your workload grows.
And we also want this to be massively scalable
from a regional availability standpoint.
Azure has a presence in 60 to 70 regions across the globe
and Azure Data Lake Storage
is available in every single one of these regions.
So if you've got data sovereignty requirements
where you cannot take the data out,
we are there.
We are scalable from that point of view as well.
Now, let's talk about the performance
aspect of the platform.
In any other cloud storage or object storage platform,
when you want to organize your data,
you likely organizing it into folders and files.
Now let's take an operation that you're likely to do
as part of an analytics workload.
You've got a folder of data
and you want to delete it.
In the normal object storage platform,
an operation like a "delete,"
means you've got to enumerate all of the files in the folder
that you're trying to delete,
and you've got to go delete them one by one.
And so if you've got
N number of files in there,
in order for you to complete that operation,
you're talking N times the order of magnitude
for you to actually go and complete that operation.
Now let's look at how we do this on Azure Data Lake Storage.
The benefits of having a hierarchical namespace
implementation on the server side
allows you to treat that folder as a first-class citizen.
And so if you want to go and make that same delete,
you're not enumerating all of the files in the system.
You're doing one metadata operation against that one folder.
You delete it super quick
and instead of completing that operation in N times
what it takes for you to delete a file,
you're completing that operation in milliseconds
because it's not even an actual delete operation,
it's only a metadata operation.
So this is an example of how the benefits of the platform
that we have on Azure
can kind of translate into performance characteristics
for you on your computer infrastructure as well.
So now let's talk about security.
Like I said,
you need the ability to go and provide access control
for that data.
On Azure, you have variety of levers
to protect that data that you have on the platform.
Number one, AAD based authentication mechanisms
such that you can make sure you're authenticating
all of the users that are accessing your platform.
We also have a whole bunch of authorization mechanisms.
What's unique to Data Lake Storage
is what we call the POSIX based access control mechanism
where you can not only authorize using
shared key signatures
or role-based authentication mechanisms,
but also at the folder level and the file level,
you can now set up these access control lists
such that you can now start governing access
at super fine-grain levels.
Then let's talk about the actual data itself.
We allow abilities for you to go and crypt that data,
whether it's an address or whether it's in transit.
We also allow you to specify your own keys
or allow Microsoft to manage the keys.
And last but not the least,
there are a lot of audit and compliance requirements
that your organizations might have.
And we have industry wide certifications on the platform
that gives you that peace of mind
that your data is safe with Microsoft.
Let's now touch on the cost effectiveness.
As I mentioned,
Data Lake Storage is built on an object storage platform.
So it is super cheap to begin with.
If you're coming from on premises,
especially if you've got a Hadoop
environment already on-premises,
you're having to deal with the challenge of
not having the ability to scale your compute
and your storage separately
because it is a monolithic platform.
So if you need the capacity for compute,
you wind up
stranding some of the storage capacity.
In the cloud,
and of course for Data Lake Storage as well,
you have that flexibility of separating the compute
and the storage,
and so you can manage your storage costs independently
of your compute costs,
and that's an advantage in itself
when you move to the cloud.
The other element of cost effectiveness is the performance.
As you've seen, the performance on Data Lake Storage
is way better off,
much faster than the traditional object storage systems.
And so an operation that would have taken
N times more on another platform
will take milliseconds on our platform.
What does that mean?
It means if I had to run the job for five minutes
or 10 minutes,
it actually doesn't have to take that long
when I'm doing some of these metadata operations that fast.
And so it also translates
to compute cost reduction for you
because of the high performance on the storage platform.
And then about data management,
when you store data within the storage platform,
we also give you the choice of multiple tiers.
You've got a premium tier where you pay more,
but you get way faster performance
because the latencies are much lower.
You also have hot and cold tiers,
and you've got even archive tiers
depending on whether you're trying to archive your data
or back it up.
In addition to the availability of these tiers,
we also give you tools, policy-based mechanisms.
We call that lifecycle management on the platform.
We allow you to manage the life cycle
of the data that you have
and tier them from a high cost year,
all the way down even potentially to an archive tier.
So that also gives you capabilities to manage the cost.
Then let's look at the openness of the platform itself,
and we've already discussed this.
On Azure, we have a whole slew of analytics engines
or ingestion mechanisms or visualization mechanisms.
We're very tightly integrated with all of these services.
Data Lake Storage is the foundation
for all of these analytics solutions on the cloud.
But in addition to that, we don't stop with Azure.
We also work very closely with third-party companies
across the board,
whether it's integration and preparation
where you're bringing the data in and preparing
that for consumption,
or an analytics engine or a BI solution
that's acting upon it,
or a governance solution
where you're actually trying to specify access control
on that data.
And last but not the least,
if you are an on-premise customer
who's trying to migrate the data
from an on-premise Hadoop cluster to Azure,
we work with partners on that as well.
So we have a very wide ecosystem
and we like to provide that choice to customers
in terms of which solution you actually want to use
on your data lake implementation.
Let's spend a few minutes on data lake migrations.
On Azure, you have a variety of options,
whether it's Azure Data Factory,
it can be used as an ingestion or a preparation tool.
It can also be used as a migration tool.
If you're trying to migrate data from your own premises,
you can use Azure Data Factory.
You can use data box.
We can ship you an appliance,
you copy the data,
and offline migration of that data over to Azure.
Or like I said, we partner with WANdisco,
it's a third party company,
to help you with the migrations.
Let's just look at the WANdisco solution itself.
So it's a partnership we've had with WANdisco
over the past few years.
The benefits of WANdisco solution,
it's very tightly integrated with Azure in the portal.
It's a very daunting task for a lot of customers
because when they've got massive amounts of data
that they've collected for over years,
migrating all of that data
with all of the integration checks
on to a cloud is a pretty daunting task,
and a lot of customers will actually spend weeks, months,
just planning and preparing.
And we're trying to take that complexity
and that hardship out of the equation for such customers
who are looking to move to the cloud.
Because it's so tightly integrated with the Azure portal,
discovering it is super easy.
You go to the marketplace,
you look for the Live Data Migrator Solution,
we call it LDMH, for Azure.
You find it,
it's just like another resource,
you go through the prerequisites.
The prerequisites essentially are
hey, I need to go and set up an express routes.
I might have to go and configure my network.
I need to know exactly which storage accounts
I need to put that data.
So you take care of all of those prerequisites.
Then on-premise, you need to make sure you've got the agent,
the software on the on-premise side
to be able to migrate that data over to Azure.
So you take care of all of that.
And then when you come to Azure,
the experience is actually super easy and quick.
You go to the portal.
You say, I want to create a migrator.
You create it.
Once you've created the migrator,
you're now saying, okay, I need a target.
I need to know where that data is going to land on Azure.
So you go to the targets.
You say, I want to go create the target.
You've already created the storage account.
So you configure the target
and say here's what I want my data to go,
here's the storage account that I just created,
please create my target for me.
The target gets created and you've finished
step one of the process.
Now, what you want to do is
you want to kick off the migration.
You've already set up your on-premises edge node
because you've got the software installed there.
You've defined your target.
You now need to map them.
So you go create the migration piece
and you say, okay, here's my source path
which is your on-premise path where the data
is going to come from.
You want to pair it up with the target
that you just created.
You also specify options
in terms of how do I want to migrate the data?
You can say, start the migration immediately.
You click a few buttons
and boom, you've actually created that.
And as soon as you've created that resource,
the data just starts magically flowing in.
It's as simple as that.
After a few clicks on the Azure portal,
after you've taken care of your prerequisites,
you've been able to start migrating terabytes
and petabytes of data.
We've got feedback from several customers
that they've been able to get up and running in minutes.
In addition to that,
the portal also gives you the ability to monitor
how the migration is doing.
You can look at all of the metrics right
from within the Azure portal,
such that you're not playing any guessing games.
Next, let's look at a customer
who's actually benefited from all of the capabilities
that I've just spoken to you about.
Smiths is a United Kingdom based
industry technology provider.
If you've walked into a hospital
and looked at the devices at the hospitals,
or you're walking into an airport and you've seen a scanner,
potentially even some of the appliances
you have in your house,
you might not even realize it
but you might have Smith's components
in any of these devices that
you're casually observing
as we go about our normal day to day business.
So Smith's has around 22,000 employees.
They have customers from 200 countries worldwide.
They actually operate from
50 regions.
So they are globally distributed company.
They had some issues where
they had massive volumes of data
across all of these 50 divisions,
and in order for them to actually get the insights
from this data,
they had around 800 applications
that they had developed over time.
What they wound up doing
was they created a data lake implementation.
They brought all of that data across all of these divisions
into Azure Data Lake Storage,
they used Azure Databricks to prepare and curate that data
for all of these divisions within the company.
They leveraged Power BI
to replace what was originally a very manual
and error prone reporting set up that they had,
and automated everything.
And since it was low cost data lake implementation,
they got the benefits of cost.
They got the benefits of save time,
and they also got the benefits of accurate data
that they're getting from the platform.
So that's a great example of how a customer
has leveraged our solution.
So key takeaways for you today.
Number one, the platform that we have
is a low cost,
purpose-built solution primarily for analytics,
it's super performant,
massively scalable,
and has a very vast,
extensive integration with all of the ecosystem
that's out there
to help you get the value of the data
that you have within your organization.
So thank you for spending the time today.
Please visit us
at the links that you see here,
and have a good rest of the day.