- Hello everybody, this is Paul-Scott Murphy from WANdisco.
I'm gonna be presenting to you to today with Mada
from Infosys on Managing the Business Risks
of Large-scale Hadoop Migrations.
So the agenda for the conversation will really be
talking through some of the drivers
that organizations are faced with in cloud migration.
We'll be talking specifically about the challenges
that emerge from that with large-scale data migrations,
what we refer to as the data migration gap.
The risks and technical challenges that emerge at scale
differ from those for small scale environments,
so we really wanna give a clear understanding
of what those are.
We'll make reference to industry studies,
and the technical approaches that WANdisco takes
to resolving those challenges
with we call our LiveData platform.
Mada will talk in specific detail about the application
of this technology, along with the processes
and blueprints that Infosys provides
to lend their expertise to large-scale data migration,
and make reference as well to a case study
for a U.S. retailer that's taken this approach
for a hugely successful migration of multiple petabytes
of data in the Azure cloud environment.
So we'll talk through the technical details behind that
and give you a really clear understanding
of how this applies in environments a platform
such as the Databricks platform are in constant use
for analytics infrastructure.
So with that, I'll hand over to Mada
to explain Infosys' role in our partnership.
- Let me start with reinforcing
the importance of migration and modernization
in any enterprise's journey.
You know, what our vision for any enterprise
which is data oriented is for them to become
what we call a live enterprise
where, you know, everything is alive.
It feels like an entity
which is like any other living thing.
The foundation for this is absolutely
the modernization journey, including migration to the cloud
that takes place and it's (mumbles) it's an imperative
for any enterprise.
And the challenges though are the fact that
data applications are no more tier two.
Applications in most enterprises,
they are tier one, they have huge volumes,
even compared to a few years ago.
That makes this a little challenging for us
to do migration the old-fashioned way.
What happens when you have a large volume of data,
the processes and tools that we use
for this, for solving this,
when the scale was smaller in the gigabytes
and low terabytes is
these tools and process fail.
We have done a lot of migrations over the years.
As the scale grew, the processes and technologies
that we use for this do have to keep pace with it,
and that is due to the fact that
outages are unacceptable.
You know, if we have to be down for days at a time
that wouldn't be acceptable to most of our customers.
Applications need to keep on moving
while the migration is taking place.
And the scale of the migration would further
compound that challenge, and this is where we have
brought together the technology
from WANdisco's LiveData platform
and the processes and techniques that we use at Infosys,
to do this in a seamless, painless way for our customers,
and launching them toward success
in their journey to become live enterprise.
That's it, hand it over to Paul
to talk more about what technology we use
under the hood and talk to us more about that.
- Thanks very much, Mada.
WANdisco plays a role in large-scale data migrations
that's very specific to the challenges that emerge
when data exists at scale.
He's already made reference to the fact that
the continued operation of applications
is critical for many organizations
looking to migrate to cloud infrastructure,
perhaps to modernize their application
and analytics platform, maybe from an on-premises
Hadoop environment to using Spark and Databricks
in the cloud.
But there are many other reasons for migration as well.
So really what we refer to as the data migration gap
is how you go about answering that question
of the challenges that emerge with data at scale.
How do you migrate potentially petabytes
or exabytes of what's increasingly becoming
business critical information,
without causing disruption to the applications
and platforms that take advantage of those data sets?
Ensuring data consistency between environments, as well,
so that you can guarantee your data are available
in the form you need at the outcome of your migration,
can be important for that.
So WANdisco refers to this, as I said before,
as the data migration gap.
I wanna make reference to an industry study
that we've conducted in order to determine, you know,
what's important for organizations looking to migrate
to more modern analytic platforms in the cloud,
and really the focus there was around the business risks
associated with large scale data migration.
In querying organizations through this study,
it was very clear in the responses that were obtained
from that that there were a few key challenges
that most businesses saw as critical
to their migration to the cloud.
In the first instance, the majority of organizations
were unwilling to accept
business disruption throughout migration.
In fact, the vast majority indicated that only
a matter of minutes or hours of down time at most,
if not, no down time was all that was acceptable
for the purpose of large-scale data migration,
and if you think about what that requires
when you're conducting a migration,
it really means that applications need to continue
operating throughout the migration itself.
It takes time to move data at scale,
and you can't afford the option of ceasing operations
while that migration is underway.
The second key result that came out of that study
was the questions around data consistency.
Of course, organizations using large-scale data platforms,
Hadoop environments, Apache Spark environments,
Cloud Object Storage, and the like, are increasingly
doing so for what's becoming business critical data.
They're running their organizations on it.
They're integrating with their partners,
and employing the ability for access
to the analytic platforms and the data itself
to be available with confidence that the data
represent what the business needs to use and consume.
So the outcome of requiring data consistency
throughout migration and being able to guarantee
that data are not lost or modified through migration
is very important as well.
The third key element that we saw in response to the survey
was the concerns around the level of effort involved
in actually performing a migration.
Application migration, data migration itself
can be time-consuming because of the need
for the effort involved in dealing with the details
of accommodating the migration needs throughout.
Any chance of that being automated or optimized
either through technology or process then becomes
hugely important for how organizations
address what we refer to as the data migration gap.
So I'll talk a little bit about WANdisco's approach to this
which we term LiveData.
WANdisco's technology platform is geared
specifically towards answering those questions
around how you can migrate to modern
analytics infrastructure, particularly in cloud environments
without incurring the overheads associated
with the expended IT effort, or risking data consistency,
or needing to incur down time in applications
while a large-scale migration is conducted.
So LiveData is what we refer to as the ability
to guarantee that your critical data remain
continuously available throughout migration,
regardless of environments, regardless of platform,
and regardless of the scale of data,
even while those data continue to change.
So to talk a little bit about how WANdisco does this,
how we employ a LiveData approach to achieving
this outcome, conceptually it's quite straightforward.
We obviously productize this through technology offerings
we refer to as LiveData Migrator,
which is available as a standalone product
and obviously is cloud services as well,
but the approach it takes is
conceptually quite straightforward.
Firstly, we intercept activity between clients
and their use of underlying storage.
The reason we intercept and interject in that activity
is so that we can use consensus across a distributed set
of deployments using data at a global scale
to obtain an agreement on any changes
that those applications want to make to the data itself.
We replicate those changes between environments
in order to take the outcome of that consensus led approach
and maintain data consistency by replicating data between
environments prior to it being changed at the source.
So we agree on the replication.
Applications make changes to their source environment
and WANdisco's technology uses this approach
of employing distributed consensus
to guarantee a consistent outcome
across each replica of data regardless of the scale.
Now, for pre-existing data we also introduce
a scan of content at the source.
We only need to do that once.
We don't need to repeatedly scan content.
We don't need to schedule migration,
or schedule small batches of the movement of data
from one place to another.
We conduct a single scan of source,
and we replicate ongoing changes
with global consistency by employing this
consensus based approach.
So the benefits that result from that are
firstly it allows us to automate data migration.
If you're moving from Hadoop,
or some other platform on-premises,
into a platform like Spark running in the cloud,
we can automate the data migration by simply
selecting data sets to be replicated,
initiating migration, and leaving applications
to operate as they would have before,
eliminating the down time and business disruption
that would result from other approaches.
We can guarantee that data won't be lost or modified
during transition because we're employing
a global, globally capable consensus based approach
to replicating changes to data.
This obviously improves the overall efficiency
of the data replication, and in the end enables you
to achieve migration more rapidly
with minimal or zero down time
and disruption to business operations.
So to give you an example of that,
I'll hand it back to Mada to talk through a case study
referencing a U.S. based retail environment
that has taken advantage of this technology
along with the process led approach that Infosys adopts
to migrate multiple petabyte cloud based storage environment
between Azure Databricks as the primary workloads, thanks.
- Thanks, Paul.
So the customer that we are referring to is
a U.S. based retailer who has
an Azure and Databricks based environment.
The customer in question is a large retailer
based in the U.S.
The environment that they have is
Azure and Databricks heavy.
Having built this over a number of years,
having migrated from (mumbles) into Databricks.
It still used a large number of legacy components,
namely ADLS Gen2
and Parquet for, you know, tables and organization.
The target was to go to the latest generation, ADLS Gen2,
for the target platform, which enables a better
security model and also unlocks a number of capabilities
with respect to analytics and AI.
It would also convert into a Delta Lake format
for storage and organization,
and the challenges are pretty straightforward,
multiple petabytes of data that needs to be migrated
with a large number of application workloads
with interdependencies between them
which cannot really be done in one big bang approach.
The approach we took was to deploy WANdisco's
LiveData platform and combined with our processes
and methodologies around inventorying analysis
at the beginning and validation
from an independent perspective post-the migration,
at the end before cutting over.
This provides a lifecycle view of what approach we took.
The analysis and design of the entire inventory
takes place at the very beginning of the engagement.
We splice and slice and dice this into multiple waves.
In this case, we had about five different waves
that tackled different workloads.
For example, the data engineering workloads,
some of the data engineering workloads also had
near real time data ingestion happening
which couldn't really be stopped,
as well as some of the workspaces that belonged
to the teams and consumers who had
their own lifecycle with respect to how they develop,
deploy, code, and deal with the data.
The approach that we took split this
into five different waves,
and each wave looked something like
what's in the middle of your screen.
The historical data was migrated using the one scan approach
that Paul talked about at the very beginning
of that wave.
Then turn on LiveData replication
that keeps the data in sync.
When we are ready cut over once the code is remediated
and tested, that happens in a parallel track.
We validated this using what we call as a parallel run
which is right at the bottom of the bottom track
over there that keeps it running in parallel
with the ingestion, ingested data kind of float
across both platforms running in parallel
for a period of time.
Typically, it's about a few days, maybe a week,
and then that gets validated by the business
during that period and gets cut over.
That process allowed us to kind of seamlessly migrate this
minimizing the down time
over a shorter period of time than expected.
This kind of gives you an overview
of the migration architecture that was deployed.
As you can see, that was ADLS Gen1
and the Databricks environment associated with that,
and the Gen2 and the Databricks environment
associated with that on the right side
which is our target environment.
The LiveData platform provides all the necessary toolkits
to both migrate this data, as well as keep this data
globally consistent, once the data is initially migrated.
And the outcomes were a testament to the technology
as well as the processes that we deployed here.
The over 3 1/2 petabytes of data was migrated over four,
and over five phases and it took about
10 weeks to complete this migration.
We targeted a release every couple of weeks, you know,
to happen over the weekend to minimize
the impact to the business,
and the security requirements which were stringent
were met by the process.
The scale it took to migrate this was delivered
by the underlying LiveData platform
to perform this migration,
and the risk was minimized
to a near zero, and that was basically the,
makes the migration process absolutely easy
and painless for the customer.
And we of course migrated the Databricks workloads
to Delta Lake as well.
- Thanks very much, Mada.
Hopefully that was a good presentation for all
on the business risks and technical approaches
that can be taken to large-scale data migration.
We've made reference here to some industry analysis
of the business risks as seen by organizations
wanting to adopt cloud and migrate large-scale
analytic platforms to cloud infrastructure,
reviewed the technical approach that can be taken
with WANdisco's live data platform,
as well as the process that our partner Infosys
puts around that, and made reference to our successful
case study of, again, large-scale data migration
to Azure Data Lake Storage Gen2
for a predominantly Azure Databricks environment
for multiple petabytes of data.
So thanks very much for listening.
If you're interested in any further information
about WANdisco's LiveData platform,
you can contact partnermarketing@wandisco.com,
or obviously go to our website, or the Infosys website
to hear more about what we do jointly.
Thanks very much.