[MUSIC PLAYING]
BRUNO AZIZA: Hi, everyone.
I'm Bruno Aziza, and welcome to another episode
of Data Journeys.
This is where we come to learn from data leaders--
their growth, their lessons.
And today, I'm excited for you to discover Ramnik.
She's at LiveRamp.
Ramnik, thank you so much for spending the time with us
today.
RAMNIK KAUR: Thank you, Bruno and team, for this opportunity
to talk about LiveRamp and our journey related to GCP.
BRUNO AZIZA: All right, Ramnik.
Well, let's go straight into it.
What does LiveRamp do?
RAMNIK KAUR: So LiveRamp is world's leading data
connectivity platform.
We make it safe and easy for our customers
to use customer data effectively and efficiently
wherever it may be.
We help our clients to connect to data
within their enterprise, collaborate
with partners, as well as activate data
across broader ecosystem.
We maintain world's largest and most accurate identity graph.
We are also industry's neutral data connectivity platform
at scale, working with more than 50 of leading SSPs and DSPs.
We have more than 600-plus technical partnerships.
We have integration with the digital leaders,
as well as social platforms, with more than 75 million
connected TV households.
We also work with 160-plus data providers
across leading categories such as CPG, transaction, auto,
retail, B2B and more.
Our world's leading governance design and privacy-first design
ensures that the data is used safely
and effectively, and its use is controlled downstream.
We have world's one of the largest infrastructure, which
includes Hadoop deployments with more than 90,000 CPUs,
300 terabyte of memory, and 30 petabytes of storage.
We run more than 100,000 YARN applications
to deliver billions of records per day to our partners.
BRUNO AZIZA: So one of the largest Hadoop deployments
in the world.
90,000 CPUs, 300 terabytes of memory,
30 petabytes of storage, 100,000 YARN applications.
So you know what it means to deploy data at scale.
Let's talk about the use cases, and particularly the migration.
RAMNIK KAUR: So a little bit about our journey to Dataproc.
So after our GCP migration, we first
started continue to use our existing Hadoop
vendor, Cloudera Director.
But we had multiple challenges, including reliability issues,
ease of use, as well as auto-scaling issues,
as well as support from the vendor.
So it helps us thinking that we need
to start looking for a better solution for a Hadoop cluster
management.
We did a pretty detailed analysis,
and following were the factors we
used for this evaluation between different vendors.
First thing first, starting with the support,
we really needed expert Hadoop support from our new vendor,
as well as close partnership with the team when
we are working through the new solution.
Dataproc surely came as one of the solutions.
Especially even during analysis time,
we got very good support and collaboration
from the TAM team, as well as customer success management
team, working closely and helping us
with the journey for migration.
The second main area was multiple features offered
from Dataproc as solution.
First, built-in auto-scaler.
As we mentioned, we had our in-house auto-scaler
and we had multiple issues, as well as overhead management.
Dataproc providing a auto-scaler helped
us move away from that overhead, as well as
remove the challenges and make it
much easier for teams to auto-scale clusters
based on the workload.
And second was ease of use.
Dataproc APIs make it really easy for our teams
for creating, deletion and updating of cluster.
Dataproc also offers data from Terraform support,
which makes it easier for teams to have Hadoop cluster sitting
right next to with the rest of our infrastructure.
And finally, Dataproc are supporting
all and updated big data technologies,
especially support for Spark and MapReduce,
which was so critical for LiveRamp.
And further down the list, very high performance
in terms of faster cluster creation,
deletion, and scaling.
From our analysis, we found that the cluster creation
and deletion, even adding 100 nodes, removing 200 nodes,
all these timings were less than five minutes, which
was so critical for teams, in terms
of high performance for cluster management.
And finally, Dataproc offered high reliability,
which was one of our previous challenges.
It offers clearly externally SLA-managed solution,
which enabled us to stay on top of our external SLAs
for our customers.
And cost always-- although the primary focus for migration
was actually to migrate cost as it,
but we were aware of premium configuration.
Post-migration, as we started looking into doing more cost
analysis, migrating to PDM really
helped us with the cost optimization
for Hadoop clusters, and we are in some of our clusters,
we are even now at 40% PDM percentage,
which really helped us reduce our cost
for Hadoop infrastructure at LiveRamp.
BRUNO AZIZA: Wow, that's really quite impressive.
When we're talking about performance here,
you're talking about cluster creation, deletion,
less than five minutes, adding 100 nodes
in less than five minutes, removing 100 to 200 nodes
in less than five minutes.
So it's giving you a great infrastructure
to scale to your ambition.
Can we talk a little bit about the use cases?
How is the data used?
What are people at LiveRamp doing with the data
that you have created such a great environment to stand on?
RAMNIK KAUR: So most of our data is in terms of batch processing
data, basically.
So we have multiple products here-- like onboarding, FBR,
data networks and data partnership,
which requires multiple Hadoop pipelines, workflows here
at LiveRamp administrate, and delivering more than billions
of records per day to our customers,
delivered to multiple different partnerships.
BRUNO AZIZA: And it's data scientists, business analysts?
Who are the folks interacting with this?
RAMNIK KAUR: So mostly it's big data engineers at LiveRamp.
Plus, we have a whole separate unit of our data scientists
and data analyst as part of another product suite, which
is LiveRamp Safe Haven
BRUNO AZIZA: That's great.
So you've got internal use and you're building data products
on top of that data.
Now, you've told us a lot about the benefits
of the platform for you.
I'm curious about the best practices-- what you've learned
throughout this journey.
So we like to start with the positives.
So what would you say the three things
that everybody needs to consider when they
start a journey like yours?
RAMNIK KAUR: So one of the best practices
we followed as big data infrastructure team was,
even before we start the migration,
we worked on developing some tools to gather metrics
so that we can compare our performance and reliability
before and after migrations.
So some of the key metrics which we look into
was job runtime, job and task waiting time,
filled jobs, filled tasks, comparison
of preemptive lifespan and impact
of that on our reliability for our jobs.
So these metrics and dashboards really helped
the teams involved into migration
to compare the performance and reliability
before and after migration, and help us make good calls even
during the migration.
The second main area was focus that we kept migration
as the primary focus to improve reliability and migrate cost
as is-- which helped us actually complete the migration on time.
Although, after post-migration, we did
spend a good amount of time in terms of cost optimization.
And finally, working closely and making
the best of our excellent customer success management,
as well as TAM support.
It really helped us through our journey for Dataproc migration,
as well as, after migration, helped
us to stay connected with the product management, as well
as the engineering team, and stay
connected on the roadmap for new features.
BRUNO AZIZA: That's great.
So creating metrics before and after the migration.
So you know if you're improving and how you're improving.
I'm curious, did you do this by yourself?
I know you mentioned Google's team help,
but did you also involve a partner in this deployment?
RAMNIK KAUR: So mostly it was like us.
We have a centralized big data infrastructure
team working with the engineering team on our side,
as well as the customer success management and TAM team.
BRUNO AZIZA: So we've talked about the best practices.
Let's talk about the flip side of that-- maybe the things
that people need to avoid.
It might make sense at the beginning,
but maybe they shouldn't happen or maybe they
should happen later.
What are some of the don'ts that you would suggest people
to consider?
RAMNIK KAUR: One of the items that we learned
was that we don't rely directly on alpha and beta products
without having explicit partnership with the TAM team,
as well as the engineering teams.
Specifically, any pre-GA projects
don't come with the correct SLAs.
So it is always a good idea to have partnership established,
like early conversations with the engineering
team and the product management team,
so that we can do good risk analysis before we start trying
out those features and make it part of our rollout
internally in our teams.
So in one good example for us was working and looking
into the feature enhanced flexibility mode,
which we started early conversations
while the feature was still pre-GA,
but working with the TAMs, and we
were able to successfully roll this out at LiveRamp.
BRUNO AZIZA: What else should we consider avoiding,
or maybe paying attention to?
RAMNIK KAUR: For sure.
Don't forget about the network quotas, as well as
another limitation and other limiting Cloud factors--
especially with the Dataproc.
It offers us to have much higher node count
than our previous Hadoop vendor, which
also meant that we had often to update our IP space ranges,
as well as update our quotas--
especially when we were working through multiple VM
configurations and disk configurations.
And finally, don't forget about the interaction
between preemptibles and CUDs.
We know that CUDs make compute cheaper,
and preemptibles make it a lot cheaper.
So specifically when you are looking post-migration,
when you start looking into further migration
into preemptibles, it was a good idea
to work closely with the Cloud FinOps team
so that we can see what is the impact of CUDs utilization
as we migrate to preemptibles.
BRUNO AZIZA: Well, Ramnik, this was amazing.
We learned a lot from you.
Setting up metrics, thinking about performance,
and not just partnering on the technology side
but also partnering on the financial operations side.
Ramnik, thank you so much for spending the time with us
today.
RAMNIK KAUR: Thank you for having me.
BRUNO AZIZA: And I hope people are going to reach out to you.
In fact, if you want to find out more about customers just
like this, click on the link down below
to sign up for the series.
Until next time, I'm Bruno Aziza.
[MUSIC PLAYING]