>> Hello, this is Lara Rubbelke.
Today on Azure Friday, we're going to be learning all
about Azure Data Lake Storage Gen2.
With me today, I have James Baker who is the Program
Manager-in-Charge of Azure Data Lake Storage Gen2.
James, can you talk to us about a little bit about
what you want to share with us?
>> Yeah. Thanks Lara. I'm super
excited about the new storage service,
Azure Data Lake Storage Gen2.
What we're offering with this is
a new capability dedicated to analytics,
built on top of Azure Blob Storage.
>> Okay.
>> So, what that comes up is that it becomes
a Multi-Modal Cloud Storage Service,
the first in the Cloud and we're super excited about this
because it really delivers the best of both capabilities,
driving the storage service and becomes
the future of what Cloud Storage becomes.
>> What does multi-modal mean?
>> Yeah, okay. So, previously in the Cloud,
what we commonly had was the things called Object Stores.
>> Right.
>> So, Azure Blob Storage is an example of Object Store,
X-ray is an example of Object Store,
JCS is an example of Object Store.
What they are is have a very flat hierarchy,
just simply store the objects very low overhead,
makes them very scalable and very cost efficient.
>> Right.
>> So we can see that here.
Just a simple example.
What we've got is the convention of
putting slashes in the name,
but as far as the storage is concerned,
it's just part of the name and
doesn't consider the structure.
The other sort of paradigm that is very existed
if you've ever used your local computer,
you are used to a Hierarchical File System
where the File System organizes this data into
a structure of directories and sub-directories
leveraging just simply organizing
the data in a sort of an efficient manner.
More importantly, analytics frameworks
have taken advantage of
this arrangement and they
have built special features around it.
So what we're going to see today is an example of
how the analytics engines
take advantage of those features.
But, in the Cloud we've either had to do
an Object Store or a
File System and we've never
really been able to bring these things together.
So customers have had to make a choice of
trading one set of
qualities off for another set of qualities.
>> Or store their data twice.
>> Or store their data twice, correlating [inaudible] of data.
>> Right.
>> Which is exactly what goes on.
So what we're doing with Azure Data Lake Storage Gen2
is combining those qualities,
providing it in a single
unified capability two endpoints.
So we've got an Object Storage endpoint,
we've got a File System endpoint
all converging around the same data.
So that's what makes this so dramatic and
so important for analytics workloads.
So no longer do you have to sort of
compromise in your choice around
cloud-based storage whether you want
the qualities of Objects Store or File Systems.
>> So, you can use that same data in
that stored ones and use it no
matter what kind of application need you have.
Whether it's analytics or your typically I just
need to be able to store that data and access it.
>> Exactly right. Yeah.
>> Great.
>> So I want to show now is
a quick demo which shows how important
having File System capability and
Object Store scale is to analytics workloads.
>> Okay.
>> So if we go into the demo here,
here where I am in the best show for my HD and Cycliste.
I can do this on Azure Databricks as well,
just happened to have pick HD inside.
One of the things we can do is,
we've always been able to look at
examples of how we've been able to talk to,
there is the command, how we've been able
to talk to remote storage.
This isn't our new capability.
We've always had the words be driver the ideal driver.
But a nice thing to point out here
is a new capability that we have introduced,
the IBFS Driver, the Azure Blob File System Driver.
So this is now built across the top of
Azure Data Lake Storage Gen2
which gives us access to this great capability.
So here we've got an example of a directory.
Let's have a quick look at him.
If I count the objects in here, there you go.
There's almost 5,000 objects in there.
Now, this might sound a little bit trite,
but what is very common in analytics workloads is that
the job process is in parallel and
typically writes the output into a temporary location.
As the job completes,
it renames that temporary location
to the real output name location.
In file system based activities,
this is a metadata operation.
But in Objects Stores which we are
approximating a file system what this means,
because Objects Stores don't have the notion of
directories and don't even actually
have the notion of renames,
what this means is that
the file system driver has to go through,
get a listing of the all 5,000
files and then do 5,000 copies
and 5,000 deletes of the original and
that's how it approximates a rename of a directory.
While it does this quite successfully, it takes time.
So if I was to for instance,
execute that command on [inaudible] with 5,000 files,
this would take in the order of minutes.
So as we run this now because it's
a first class file system,
what do we got? Two seconds.
On that DNM a lot of that is the JBM starting up.
So that's an example of how we get
significant performance improvement which
directly benefit analytics workloads.
>> Okay.
>> The other piece we want to talk
about with Azure Data Lake
Storage Gen2 is how this is now
the convergence of our two
existing remodel storage services.
We've always had the Blob Storage Service which
it has great qualities for cost effectiveness,
regional availability, redundancy options.
You can either have local, zonally,
geographically redundant options which gives
great abisity stack around your data.
We've also got Life Cycle management
where you can tear the data out.
We know that when data is new it's very,
very valuable but as it gets older
more than 30 days customers
have often got a choice of saying, "Oh well,
the value that I get out of it versus the cost of storing
it becomes negative" and so they
delete the data but they don't
really want to delete the data,
what, they just want to store the data cheaper.
So we've got tiering capabilities
with the hot cool and archive tiers in
Azure Storage where you can store that up to
a tenth the price
that you would have of some of the hotted tiers.
So we've got a lot of cycling around that which is super
important for Analytics Data when you're
building these enterprise data lakes.
Other things we're got a larger partner ecosystem
around Globe Storage and
we want to be out to take advantage of that.
We don't want to
recreate the ecosystem surround our new service,
we want to be out to leverage the existing ecosystem
that we've got.
Then we've got the Data Lake store service and
this has always been built for analytics.
It's primarily optimized for Hadoop in terms
of the frameworks that work on Hadoop
such as Spox such as [inaudible].
This is what the data like store was optimized for.
It had support level security which is super
important to folks building enterprise wide Data Lakes.
You need to be able to secure these things.
You can't just have full access.
I don't want to give you the keys to my entire Data Lake
and give you access
especially if you're in the accounting department.
>> Right.
>> So in the marketing department we want to
keep out data separate from your accountants.
That's super important that we can't build one without
a good structured Atheling system
and so that's what
the existing idealists service brings us.
The other things is that it had very large scale.
It's no secret that
in the Blob Storage service we used to have
very sort of rigorous skiles at the account level,
skile limits and that caused
problems for a lot of big data analytics.
Idealists went and solved that and we're bringing
that quality through to the new idealized Gen2 service.
So really what we're
saying is we're bringing the qualities that people
love out of both of the separate services and
converging them into the same
service and that's super important.
Because again, it gets you out of the terreniable.
You're getting the best of both worlds and bringing
it and not only storing
the data once and not having to move it.
You know all of these moving data
around, super problematic,
super expensive, multiple copies of data terrible.
So, bringing it all into the one single place and doing
analytics and ingest and
all of the work that you want to do on it.
Machine learning is super important.
So, I just want to jump in now and do another demo.
This is a fairly basic demo because it uses Hives,
maybe not the coolest thing in those shop anymore,
but it does reinforce a lot of the capabilities
that all of these analytics engines take.
So, one of the first things you do,
here we've got a simple little bit of table definition
which goes and puts
structure around an existing set of data.
So, we don't copy the data with this.
We want to be out to create metadata around this.
>> So, what you have here,
you have a Hive statement.
>>Yes.
>> It's creating a table and it's
applying a structure over existing data.
>> Yes.
>> So, this is your traditional Hive.
>> Yes.
>> You're doing this against
this new converged storage system.
>> Exactly right.
>> So.
>> So, if I jump over into Storage Explorer,
we can see the data here.
That one is very common especially around things
like logs data is that it's already structure.
We get logs pilling in by the year,
by the month, by the day, by the hour.
So, this is very
typical of the way the data is already structured.
So, hive the architects of
hive recognize this quality not
just because it's a naturally occurring phenomena
around the way we organize data today,
but it also provides
huge performance opportunities for the hive query engine.
So, when you see here, we've got all that metadata.
One of the key pieces of metadata we
put out there is the notion of partitioning.
So, this partitioning structure mirrors
the natural structure that we've got now
embedded within first-class directories and are
approximations they are directories that we're saying.
So, the partitioning mechanism we're
putting out here mirrors that.
Then the other thing we say is we give it
a specified location and so that location
there again is just using
the IBFA straw that we saw in the first demo.
>> Yes, I see that.
>> It's just specifying a location.
So, again it's not saying, "Oh,
I need to move the data into my database."
I'm just pointing the metadata out of
that existing location and away we go.
>> Then it infers the
hierarchy structure that you have in
your partition by that is in
that hierarchy area in that directory as well.
>> Yes, exactly right.
So actually, and that's
a great point because it comes in two parts.
The first one we're running the DDL there because
that's just metadata that comes back very quickly.
So now, if I go and run this command we can see that.
Let's try and minimize that away a little bit.
We can say there is a blobs logs type there.
If I scroll down through the description of it,
you can see all of these fields that we
specified and that's great,
now we've got the partitioning information
there and the key point
we've got is the location which
again preserves that IBFS location,
the data isn't moved.
So, the second part of what we want to
do is go and discover those partitions.
Hive's got the slightly quirky command,
the MSCK repair table command which goes and crawls
over the store from that location we
specified and discovers all
of those partitions automatically.
So if I kick this off again this
is able to take advantage of the directory structure
which is part of the hierarchical file system which is
part of IDLS Gen2 is able to go and
efficiently crawl this data or discover
the partitions and build them into
the metadata that's keeping around this table.
So, when this comes
back which hopefully won't be too much longer,
this will be great going catus
outlier here it is, it's finished.
Now, give us the results.
Oh, it's hidden the results.
Oh, there we go, right, perfect.
We can see now we've got thousands of partitions which
reflect the thousands of directories that
we had in that location.
But what that means now is that I
can go and issue a command,
a SELECT statement in hive that
uses predicate which aligns
to the partition structures that we've got.
What that allows hive to do is say, "Oh,
I can prune out all of these partitions,
99.9 percent of the data,
I don't even need to process,
I don't even need to look at.
So it prunes out these partitions,
runs the query and this query isn't cached,
it's not using any fancy tricks
of the engine is simply now saying,
"Oh, I only need to go and
process this very small piece of data,
and so now it can serve the results back to me very,
very quickly even after applying all of
the metadata pieces that we want as the serializer.
So that's a very simple example of
how hive makes the directories,
makes the hierarchical file system
beneficial to analytics workloads in a very real way.
So, last thing I want us to talk
about is what this means to
the Azure platform because we've got a lot
of 1st party analytics services,
we've got HDInsights that we saw here,
we've got Azure Databricks,
we've got SQL DW,
we've got many analytic services which offer great value
to customers that can
run and extract value out of their data.
Azure Data Lake storage Gen2 is being integrated
into these services and we're
really pleased about how that's going.
The other people we want to say is that
this is now available to third parties
as well and we're working with a lot of
great third party partners
to make sure that they're integrating here.
So that it's not a choice,
we're not forcing you down one analytics engine solution.
If you've got a commitment to
a specific ATL provider
or a specific data governance provider,
we want you to come onto the platform, keeping,
preserving those relationships because
I know they are important to customers.
So, we've got the third parties that we've been
speaking to the ISV ecosystem are really
resonating with this well because previously
they had to integrate twice with Wasby and ADL,
now they're going to have to do a single integration,
so the partners are super pleased about it.
They're also super pleased about the qualities that are
bringing in so they're very eager to
come and build on top of this.
The second point I want to make here is that,
and you've already touched on it
is no longer olens of data.
There is just one data like.
As long as we give it structure,
we make it manageable so it
doesn't turn into a data swamp.
You no longer need optimized stores
or optimized copies of the data
simply because my analytics engines works there.
We want to make sure that IDLS Gen2 performs it's
best for all analytics capabilities
regardless of what their data axis pattern needs.
So we're making changes and
we're not just treating this as a stream of
bits where optimizing it based
on the access patterns of all these analytics engines.
>> Then also makes it available,
much more easily available across all regions.
>> Of course, yes and that's
a key point because it's built on Azure Blob Storage.
>> Which is everywhere.
>> Yes. At the end of the IDLS Gen2
is a set of capabilities built on top of Blob Storage.
It's not even offered as a separate service,
you go and create a single storage account,
you just simply enable rename I suppose.
So yes, Blob's Storage is in all 50 regions,
IDLS Gen2 is also going to show up in 50 regions.
>> Great.
>> Cool. So, last place.
>> Ready to go.
>> Yes. So we're in public preview now,
we're super excited about that.
The response has been great.
So, here's a sign up link
that gets us access to the public preview,
we just want to know a little bit about
the workload information that customers are going
to be using so we make sure
we're understanding the workloads that are applying.
It's not about filtering
people out or anything like that.
We've got a great product information page,
and so there is now a unified story
about how to build enterprise Data Lakes on Azure.
So, please come and consult that.
We've also got documentation and we're going to build out
these documentations throughout the public preview
so that it really tells
a great story and customers can really and easily
understand how it's benefits apply to them.
>> Fantastic. Thank you so much James.
I am also really excited about what
we've got with the Azure Data Lake Storage Gen2.
I'm looking forward to working with
the customers and their solutions with this service.
So, great work, really great work.
>> Thanks a lot.
>> All right. We've been learning all about
Azure Data Lake Storage Gen2 here on Azure Friday.