Hi, welcome to another episode of 'This is My Architecture'.
I'm David John, and I have with me, Lakhan, from ZS Associates.
-Hi Lakhan, welcome to the show. -Thank you so much for having me.
So let's get started. Tell us something about ZS Associates.
ZS is a global professional services firm,
where we provide technology, analytics, and strategy consulting
to various clients across the industry.
We have a keen focus on life sciences and their various functions.
Great.
So we're here to talk
about your self-service enterprise data engineering platform.
What does this platform do?
Yeah, so for any life sciences client, as it starts to grow, different functions
become more and more data-driven,
they often start to face challenges while scaling their analytics journey.
For example, they start to face challenges
in building those data integration and ingestion pipelines.
Second challenge they face is the turnaround time
they require to spin up the infrastructure which is consistent, compliant, secure,
because it often requires deep tech skill, or multiple approvals across different agencies.
Third challenge faced by data analysts and data scientists
is time they take to discover data sets in a data lake
and get access to the datasets.
What we have here is a self-service enterprise data engineering platform,
which tries to solve those typical challenges
with a key goal of shorter time to market for these functions,
security at scale, all with optimal cost.
I completely agree with the challenges you've highlighted,
and that's a good set of goals you set right at the start.
So let's dive a bit deeper.
So, who are these users, and how do they interact with the platform?
Yeah, so our first users are data engineer
who wants to set up an infrastructure for their application, let's say.
So, they log into our self-service portal, where they select the type of infrastructure.
It could be a batch ETL,
it could be real-time change data capture type of a pipeline,
or it could be a custom multi-tier web application.
Once they make the choice, they also have to provide
cost center project code information, which we eventually use to charge back.
Once this selection is made, our portal makes a REST request to EKS,
where a service running on EKS creates a request record in PostgreSQL
with all the metadata information, and then creates an event on SQS,
which in turn triggers a workflow, automation workflow,
for infrastructure provisioning.
After the workflow is completed, again,
the service sends a notification using SES with all the information
about the infrastructure provision back to the request tray.
So that's an impressive automation, I should say. So I want to dive a little bit deeper.
I want to know what happens when the user actually submits the job.
Yeah, as I said, there is an event sent to SQS, which provisions the infrastructure.
What happens here is, there is another job in EKS
which picks up this message and takes various actions
based on the requested infrastructure type.
Let's say it is a batch ETL,
the first action it will take is creating an EMR cluster.
Then it will create various databases in Glue catalog.
Then it creates S3 buckets, and launches an EC2 with containerized airflow
or NiFi where data engineers will be running those data engineering pipelines.
And then lastly, creates IAM roles which integrates all of these resources.
At the end, it will, again send a message using SES
with all the information about the project resources provisioned,
and how to access them back to the user.
That's good, the orchestration pipeline uses a lot of AWS services.
It's fantastic.
But with this much automation, I'm curious to understand
how do you manage the data governance, security, and discovery part of it?
Sure.
Well, first thing I will tell is,
all of our data sits on various S3 buckets across AWS accounts.
What we have done is controled the access to this S3 bucket through Glue catalog,
so everything access through Glue catalog.
We have crawlers running in EKS, which crawl the changes in metadata
and makes it available in our self-service portal as a data discovery dashboard.
That's when users like data analyst and scientists come to this dashboard,
select different data sets they need access to and hit request.
After this, most of the automation is same as the infrastructure provisioning,
with a key change in the requested data set having to be approved by the data owner.
So, before we provision anything, we require the use of SES
to notify data owners to accept those requests.
Once they accept it, it triggers an event in SQS,
again, followed by a job running in EKS,
which then updates the IAM role, or it updates the Ranger policy,
if it requires fine-grained access control.
And again, at the end, sends the same information
about the data access provisions back to the requester.
Right, so you even have manual approvals in your orchestration pipeline?
And the way I see, it's the Glue catalog and IAM, which are the crucial services,
which help you to do data discovery, security, and governance. Fantastic.
I'm interested to know the scale of this platform, you want to talk about it?
Sure. At this point in time, our EKS hosts a hundred-plus microservices.
Of course, many of these microservices are supporting these workflows and the UI portal,
but a large part of this is reusable data engineering services.
For example, there are services for compute lifecycle management,
there are services for different data processing,
there are services for data quality rules, running quality rules on datasets.
All these reusable services, on a rough scale,
we receive millions of API requests every month,
and hundreds of data access requests per month from data analysts and data scientists.
So, I assume your platform services are running on EKS at scale,
and your application setup, all the applications are running in EMR?
Fantastic.
So, what are some key challenges this platform has addressed for you?
Yeah, like I said before,
these are some of our key goals from the platform,
which is shorter time to market for functional teams.
The way we achieved is by automating data access governance workflows,
which before this used to take six weeks,
now it takes few hours with a human in the loop who approves the request.
The infrastructure provisioning which used to take months
with all the approvals needed from security and compliance agencies,
now it takes a few minutes.
The other data engineering reusable services
helped reduce the build of data engineering pipelines by 30 to 40% for our engineers.
Secondly, the security at scale, we have built in all the security requirements
needed to provision this infrastructure and governance flow.
For example, anonymization, masking, encryption, and fine-grained access control.
We control the costs by using autoscaling capabilities of EKS and EMR.
Additionally, everything getting provisioned through this platform is tagged,
which gives the functional team ability to keep check on expenses every month.
It's fantastic, these goals seemed very challenging at the start of the conversation,
but I'm really surprised how your data engineering platform
was able to meet those goals using so many AWS services.
Thanks a lot, Lakhan, for being on the show.
Thank you for watching 'This is My Architecture'.