Hi, and welcome to 'This is My Architecture'.
Today, I'm here with Florian from TIS.
Hi Marina.
So Florian, before we start with the whole use case,
I would love to know a little bit more about TIS.
Yeah, sure.
This is a cloud solution to manage corporate payments.
This means we integrate our enterprise customers with their banks,
help them to improve operational efficiencies
through streamlined payment processes,
lower risks, managed liquidity,
and help them to gain strategic advantages through better insights.
So Florian, the architecture that you brought today
is about a data lake.
Can you just briefly describe to us,
which business challenges did you solve through this architecture?
Yeah, actually we solved two main business challenges with the data lake.
First, it's about managing our data growth efficiently and cost-effective,
and second, it's about making data available
to our internal platform services,
to customers and partners in a secure and compliant way.
And actually I can imagine that since TIS is a financial provider service,
privacy and security must be key for you.
So how is that being reflected here?
Actually, we follow their basic security pillars
such as lease privileges, data locality, data isolation and automation.
And what we did is basically to have a multi account set up,
where we have different accounts for storing the data in the data lake,
and separated them from processing of the data.
Okay.
And for the data, we again have multiple accounts
separated by data category
for public data, internal data, and customer data.
And since you have different data categories
and also different accounts,
how is it actually that you are securing those?
Is there any access control or permission set of policies
that you have on top of it?
Yes, we use Lake Formation,
which is based on the Glue metadata catalog
and use the fine grain permissions from Lake Formation
together with Lake Formation tags
to restrict access,
and for example, also to ensure that our data scientists and analysts
don't get access to personal identifiable information.
Okay, I see.
And this architecture seems to be a little bit complex,
so you must for sure want to avoid some manual errors.
So that's why I was wondering if there is some automation on top of it.
Yeah, sure.
We automate our overall infrastructure provisioning
through Terraform code.
And in the case of the data lake,
we not only provision our infrastructure with Terraform,
we also use Terraform to provide data set descriptors,
which allow us to describe data products
in a consistent way in the data lake.
Okay, I see.
And what does this exactly enable to you?
This on one hand, enables us
that we can manage security,
and compliance of the data products in the data lake consistently.
And on the other hand,
enables us to build a data marketplace
based on the metadata
and the information provided by Lake Formation in Glue.
And I can imagine that this marketplace
also enables you to just create new business cases as well, right?
Yeah, actually it allows us that our data scientists and data analysts
can access the data lake
through the data marketplace,
browse through available data sets, explore the data
according to the permissions which are granted to them.
Perfect, so now we have a lot of context,
but I'm still lacking the question of how do you ingest the data?
Actually we ingest the data from two primary sources.
On the one hand side,
we ingest data from the internet,
where we use AWS Lambda
triggered by EventBridge,
and store the data in the S3 buckets of the data lake.
And on the other hand, we use the Database Migration Service
to access the bin lock files of the MySQL Aurora server
and also write the files on a continuous base to the S3 bucket.
And Glue is accessing the data to create the metadata catalog
based on the data in S3.
And also with the Glue workflows and Glue ETL processes,
we then transform and process the data as needed.
Okay, thanks.
And I can imagine that this is being handled at scale, right?
So can you just maybe enlighten us with a couple of stats
on how much data you're ingesting nowadays?
Yeah, actually we store roughly 15 terabytes of data in the data lake,
and the data is growing about 500 gigabytes every month.
Okay.
So now we have the ingestion part covered,
but what about the consumption part?
Yeah, basically the data lake
is consumed on the one hand side from the analysts
through the data marketplace,
but we also integrate our platform services
through an API Gateway
to actually access services and capabilities of the data lake.
I see.
So this architecture seems to be really complete,
but is there anything in your wishlist,
that you would like to implement or improve later on in the future as well?
Yeah, we currently already work on implementing
the first machine learning models
where we experiment with SageMaker,
we investigate using Macie,
to automatically identify PII data in our data sets,
and we also make first experiments with the Kinesis family,
for example, to ingest our logs from the infrastructure
and from the applications to make in-depth analysis on our logs.
So Florian, thanks a lot for showing us how is it possible
to enable security and data products
which make this architecture unique.
It was a pleasure, thank you.
And thanks a lot for watching, 'This is My Architecture'.