Welcome to 'This is My Architecture'.
I am Hawn
and today we're joined by Senthil
from Discover Financial Services.
Thanks for having me.
So before we get into your architecture,
can you tell me a little bit about Discover Financials
and what you're trying to solve for.
Discover Financial Services is a digital banking
and payment services company.
Discover is one of the largest cloud providers in the United States.
Okay.
And so what are you trying to achieve with this architecture?
So we have a cloud data platforms within Discover.
So we enable data scientists and business users
to build a model and deploy model in the cloud.
So can you walk me through the flow
of how you're able to provide that analytics for your data sciences?
Yeah, we have multi-account and multi-VPC strategy in the production.
So we have a PCI account and PROD account.
Our S3 landing buckets sits in the PCI account.
So there are various data sources on-premises.
From the on-premises data sources,
we have data ingestion pipeline running in our on-prem
which uses the AWS private link,
and then it ingests the data to the landing bucket.
And we also have external data coming to our on-premises.
So we use AWS transfer for SFTP to ingest the data into S3.
Once the data comes to the landing bucket,
which is in part of the PCI, we have the triggers enabled,
and which goes to an SNS topic.
From SNS it is consumed by security services
which are running in EC2 to perform tokenization.
And we also do the file format conversion
and basically curate the data.
And the curator data gets ingested to another S3 bucket
which is in the production account.
And from here, it gets synced to EDW.
And we also have analytics and machine learning services
running in EC2 and other container platform within the PROD account.
And they consume the data from S3.
So the general purpose analytics and machine learning consumption,
we only have it in the PROD account,
wherein in the PCI account,
it is only restricted for security and data processing services.
So it sounds like you not only provide the analytics,
but also ensure that the data is PCI compliant
and tokenize for the data science to build the models.
Yeah, of course.
So the raw data which gets ingested to the PCI account,
it contains a PCI data.
And that's the reason we want to segregate the PCI and PROD account,
and then we only do the tokenization and the initial data processing
within the PCI account,
and then ingest the data to the PROD account.
And with that, you're able to limit the PCI scope.
Yeah, so the idea to have two different accounts
is to have the PCI account very minimal,
so that the surface area is less,
as well as you will be able to provide appropriate evidence for the audits,
and then like we can stop anyone from directly consuming
the data from the PCI account.
And then the scope of the PCI compliance itself will remain within this account,
not for the entire production account.
- So let's talk about scale.
How much data are you processing?
So our data warehouse like whatever we have in S3,
it's a petabyte scale data warehouse.
So that initial data which lands in the landing bucket,
we have the S3 lifecycle policies enabled,
which gets removed after certain number of days,
however, we have an archive bucket
that is where we maintain the data for the purpose of audit.
Overall, in terms of the scale, we have multiple petabytes of data.
That is outstanding.
Well, thank you so much for going over your architecture.
Thank you.
That's really amazing how you're able to provide
that PCI compliant data set
to enable your data scientist to perform the data model processing at scale.
- Thank you. - Thank you for that.
And thank you for watching 'This is My Architecture'.
[Thank you for watching]