>> Hi everyone. It's Jeff King.
I'm a Senior Program Manager in Azure Storage at Microsoft.
I have here my friend, Paul.
Paul, do you want to introduce yourself?
>> Yeah. Hi Jeff. It's Paul Scott-Murphy from WANdisco,
VP of Product Management.
I'm very pleased to be here with you today and tell you-all about
the exciting things we're doing
with the LiveData Platform for Azure.
>> Before we dive into that, let's just
go ahead and talk about some
of the challenges with data lake migration.
Data lakes, Hadoop environments are central
to every enterprise, they're critical.
You want to maintain the business continuity,
you can't have any downtime,
you have very strict SLAs,
and you got to keep that in the back of your mind
as you are thinking about
lifting or modernizing your analytics environment
as well as moving the data.
Then there's the data alone. There's a lot of data.
You can't move petabytes easily.
There's just a lot of complexity;
there's just a lot of stuff that you have to consider.
That's why we wanted to address it.
We wanted to address it with a turnkey migration solution.
There's nothing else like it on the market.
What do we have available today,
and what can folks use today for migration?
>> When you look at the types of
technologies that organizations try
to use to bring large data to the Cloud,
they are burdened by the fact that they have to build that
on tools that weren't targeting that outcome.
They're not migration tools, they're tools perhaps to
copy static data and nothing more.
They don't deal with those core challenges around maintaining
business continuity while a migration is underway.
They can be expensive and
cumbersome to implement in organizations.
Those technical resources don't want to be
spending their time building migration solutions,
they should be spending their time getting
value out of the data in these environments.
Really, the existing tools aren't adequate,
and we'll talk about what we've done with
the LiveData Platform to really move past that.
>> Let's talk about that. Let's dive
into that right now. Paul, do you want to take it?
>> I'm happy to talk about it.
The LiveData Platform for Azure,
and of course that's something we've been working
on jointly for some time.
We've recently gone to public preview with this,
and introduced two offers;
the LiveData Migrator, which is focused clearly
and specifically on the challenges around data migration.
A data-first migration approach for
big data environments going from on-premises to the Cloud.
Then LiveData Plane tackles the next step around
maintaining the ability to use
those datasets in each of those environments.
Bringing those two solutions together
under a single platform that's
implemented and available as
a Native Azure resource is really what we've done.
>> This is going to be something where as
you are building, and migrating,
and modernizing all that,
you're going to be able to watch, and monitor,
and control your migration through the same Azure portal using
the same Azure CLI and
tooling that you use today to manage the rest of your Azure estate.
It's pretty slick stuff.
You'll be really excited.
Let's talk real quick about what the deployment model looks like.
What you have here is in the LiveData Migrator world,
you've got binary package that you pull down from the portal.
You run it on your on-prem environment,
and it's just sitting there and slurping up all of the data,
capturing all the changes that you
make in your on-prem environment and
pushes every single one of
those changes into your ADLS Gen2 account.
Then from there, you can set HDI if you want to stay in Hadoop,
and you can also run Databricks.
Every Azure data service that's
available today supports ADLS Gen2.
Once you get into data in Gen2, you're golden;
you can build out whatever you need from that point on.
The LiveData Plane is similar to LiveData Migrator,
except for the fact that since
you need that active-active consistency,
you need to have an instance of LiveData Plane running
in every environment that you
need to have active-active consistency.
You've got LiveData Plane, in this case,
running side-by-side HDI as well as Gen2,
and you also have LiveData Plane running on on-prem.
They handle and coordinate the consistency.
We'll go into details about how all this works in another episode,
and we'll do a deep dive and all that.
For now, we'll just keep it high level.
I don't know about you, Paul, but I love every time you do a demo,
it just tickles me every time to see this work.
It's so slick, it's so easy.
Can you not just
do it for me but can you do it for the audience as well?
Can you show them how this works?
>> Yeah, absolutely. We've put a lot of effort, of course,
into making sure that the solution,
particularly around Cloud migration is as turnkey as possible.
The process starts from
the Azure portal where a user
interacts with it as they normally would.
They will search for the LiveData Migrator solution and deploy
an instance of this resource type
directly in their Azure environment.
The process they're following there is
exactly like deploying any other Azure resource.
You choose the Resource group to which it's deployed,
the Region, you'll give this instance of this component,
we call a Migrator a name,
and you make a choice at that point about what type of on-premises
deployment you'll need to have that run
alongside your Hadoop environment.
You follow through the same process of assigning
metadata like tags to the Migrator instance.
The validation will take
place to ensure that the information you're
providing will result in a successful deployment of the resources,
and you'll click "Create" in order to create an instance of
the Migrator that is going to run in your Azure subscription.
The deployment of that is entirely automatic and leaves
you with a deployed Migrator instance to give you
the opportunity to download the component that you'll then deploy
alongside your on-premises environment
to continue with the actual migration of data.
>> Paul, that's really awesome, man.
Maybe I don't want to migrate my entire file system.
Maybe there's only a certain set of directories,
or subfolders, or certain areas,
or certain datasets that I want to migrate.
Can I do all that as well in the portal?
>> Yeah. The process takes
advantage of this tight integration with Azure so that the user
can control and select the data they want to
migrate entirely from the Azure portal.
The way they go through that,
I can show you right now.
You identify the data that you want to migrate
and you create that in the form of what we call a migration.
To get to that point though,
you need to first define where your data will reside in Azure.
You create what we call a target.
You give it a name, you select the Gen2 storage account,
and the storage container within
that to which your data need to be migrated.
Again, this is just like you're
creating any other type of Azure Resource.
You'd assign the same metadata,
perhaps an owner, review,
and create that target definition.
Having completed that, you've just
defined where your data will be migrated to.
With the target in place,
once that deployment is complete,
the next step is in creating what we call a migration.
That's the point where you get the ability to define the subset of
data from your Hadoop environment that you want to
migrate into Azure Storage.
To do that, you go back to the Migrator instance in
your Azure Portal and you create
an instance of what we call a migration.
From the on-premises environment,
in selecting the data that you want to migrate,
you may have applications running.
Here's an example of an application like
a TeraGen job where we're generating
data into this directory we're calling
LiveData demo and the TeraGen one input there.
To select back for migration,
we create an instance of migration by
adding that through the Azure portal directly.
We give the migration a name,
we specify that we're delivering that to
the target we defined before,
and then choosing the path from our source environment,
in this instance, that LiveData demo directory.
We give that migration some settings,
tell it that we're going to automatically start it,
and then create that instance of the migration.
Once that's created, that's everything the user needs to
do to have those data that are continuing to
be generated or modified in
the Hadoop environment on-premises
migrated into their Azure storage.
We can see here, drilling in,
if I just refresh that view into LiveData demo directory,
the output from our TeraGen job is already running.
We can see that TeraGen data
being created in my Hadoop environment.
But with the migration in effect,
we can view that now going to the migration instance,
seeing that that migration status is live,
that means that we can go back to the container in
our Gen2 storage and immediately see that the data being modified,
and created in our Hadoop environment are
already available in our Gen2 storage container.
Now here we see the LiveData container drilling
into that LiveData demo directory,
and we can see exactly the same Hadoop data
available with that simplicity
and ease in our Gen2 storage account.
>> Wow, that was really great, Paul.
Really appreciate you showing the demo.
There's plenty of information available.
I just want to let you know that there
is an offer available right now.
It's an exclusive offer with Azure,
the first 25 terabytes that are migrated using
LiveData Migrator or LiveData Plane for Azure is free.
Thank you for your time.
Thank you Paul for joining me in this episode,
and we'll see you guys on the next one.
Thank you very much.
[MUSIC]