RON PONTAFARO: Hello, and welcome to a Google Cloud
databases fireside chat.
Today, we're going to be hosting Yuvaraj Sankaran, who's
the chief architect for the data fabric platform at Equifax.
Andi Gutmans, our VP and GM for databases,
is going to be talking to him about the data fabric
platform, why Equifax created it,
and the database as it leverages.
I'm Ron.
I'm from Andi's product team, and we can just start.
Yuvaraj, can you tell us a little bit
about Equifax, please?
YUVARAJ SANKARAN: Sure.
As a global data analytics and technology company,
we play an essential role in the global economy
by helping financial institutions, companies,
employers, government agencies make critical decisions
with greater confidence.
Our unique blend of differentiated data, analytics,
and cloud technology drives insights and decisions,
which help people during critical financial lifecycle
events, like buying a home, buying a car,
starting a new job, et cetera.
Equifax's products and services Play an essential role
in making these events possible.
Equifax is headquartered in Atlanta
and supported by more than 11,000 employees globally.
We operate in 24 countries in North America, Central
and South America, Europe, and the Asia-Pacific region.
ANDI GUTMANS: Can you tell us what the Equifax data
fabric is, and what is the story behind it?
YUVARAJ SANKARAN: Sure.
After the 2017 breach, the company
had a new leadership, a new vision, a new direction,
and wanted a cloud-native global data management platform
to support all of Equifax's businesses.
We wanted to reinvent ourselves and transform our business
through technology.
We are a 120-year-old company.
We have built many data silos over the years,
and we wanted to build a platform
to bring all the data together and break down these silos.
Our data fabric is a cloud-native data management
platform built using Google Cloud products.
The platform models Equifax's business
as part of its core domain model,
provides information security capabilities, metadata
management capabilities, along with the ability
to manage the data lifecycle, which includes ingestion,
keying and linking, journaling, and purposing of the data.
ANDI GUTMANS: Data fabric sounds really awesome.
How do you explain the value of the data fabric
to your business stakeholders?
YUVARAJ SANKARAN: The data fabric
enables onboarding new data sets much more rapidly compared
to how we did it in the past.
It also helps governing and auditing these data assets
in a much more streamlined and straightforward manner.
This is a critical capability, since Equifax
is one of the most highly audited
and regulated companies in the world.
A few examples.
ID Foundry is a new product platform
that we are building using the data fabric.
The monetizable data, we call them, purposed views,
were assembled in a few weeks compared
to the many months it would have taken in the legacy platforms.
Traditionally, every exchange was
built as an independent data silo
and it was very hard bringing these different data
assets together.
The reason for this expedited ability
is the data fabric allows simply configuring
these purposed views based on our enterprise. journals.
Once the configuration of the views are complete,
the data is ready to be consumed.
Another example could be starting a new exchange
with the fabric could be a matter of setting up
the exchange's profile in the data fabric.
The profile includes schemas, ingestion purposing rules,
and other metadata.
Once the profile is created and validated,
the exchange is ready for consumption
by the products downstream.
In summary, the data fabrics enables
bringing new data assets, new exchanges, new products
to market much more rapidly than before.
ANDI GUTMANS: Can you explain that data lifecycle journey
and maybe provide an example?
YUVARAJ SANKARAN: One of the core concepts,
or one of the core aspects of the Equifax business
is that we build data exchanges.
Examples, we build a consumer credit exchange,
we build employment exchanges, we build wealth exchanges,
we build telecommunication exchanges globally.
These are essentially data pipelines
that connect all the different suppliers,
assemble these disparate data assets into an exchange,
and then build products out of them.
For example, the employment exchange,
we receive paychecks from our suppliers, paychecks
and benefits information from our suppliers.
We collect them and build them into an exchange
and produce an employment verification report.
This employment verification report
is commonly used in an employee hiring process.
In the data fabric, the data we get from our suppliers,
we call them contributors-- a contribution could have
several observations-- is taken through
the following lifecycle stages.
So during ingestion, we will standardize, normalize,
cleanse, and curate the data.
These are observations, paychecks in this example.
Once the paycheck is normalized, cleansed, and curated,
we take it to the keying and linking stage of the lifecycle.
Here, we identify who these paychecks belong to, right?
This is we call this the identity resolution process.
Once we identify the individual, we assign a data fabric key
to this paycheck.
Once the entity resolution is complete,
we take it to the journaling stage,
where we journal these paychecks as a time series
in Google Cloud Bigtable.
Once the journaling is complete, we
take it to the purposing stage where
we apply the business rules, get the data ready for monetization
and consumption by the products downstream.
The purpose views are also housed
in Google Cloud Bigtable.
ANDI GUTMANS: I really love the data lifecycle
and how you built it.
Can you tell us a bit more about the technologies under the hood
that you're using?
YUVARAJ SANKARAN: Absolutely.
The data fabric is completely built on Google Cloud products.
We use Cloud Composer to orchestrate the pipelines.
We use Upserve to bring the data in and use Dataflow
for running these pipelines and move them
through various lifecycle stages.
We have metadata services that store data on Cloud SQL,
on Postgres.
Our microservices are hosted on GKE,
and the journals themselves are hosted on Cloud Bigtable.
We use BigQuery Looker, Vertex AI, and related technologies
for the data analytics side.
But the primary production operations, the database
is Bigtable.
This is where our Enterprise Journals are,
which is our source of truth.
We also use Bigtable to monetize our purposed views.
We use Bigtable for our identity resolution services, as well.
ANDI GUTMANS: And what made you choose Google Cloud,
and I would say more specifically
using Bigtable as a system of record for the critical data?
YUVARAJ SANKARAN: Equifax being a data analytics company,
we needed a partner with good data and analytics
capabilities, and we felt that Google
had the best of breed technologies in the space,
so we thought it's a good match.
Bigtable is a highly reliable, efficient, cost effective
offering from Google Cloud.
We needed a highly scalable, eventually consistent,
low latency database.
So for example, our credit exchange,
we receive about 100 million observations a day,
which need to be made available for monetization
in a very short amount of time.
Overall in the fabric currently, we
have about 225 billion records and growing.
We expect to receive about 500 million observations a day
in 2022.
Our products have a pretty low latency requirement, right?
A credit report is supposed to be delivered in 100
milliseconds to the customer.
This includes finding, retrieving, assembling
and running business rules for every request that we receive.
So the database typically gets about 10% of the time,
so 10 milliseconds or less.
So we needed a database technology
that provides significant low latency capabilities,
and Bigtable fits the bill for us here.
ANDI GUTMANS: That's really impressive latency and scale.
What do you think are some of the other things
that our customers should know about
as far as Bigtable's concerned?
YUVARAJ SANKARAN: Bigtable is a hosted, serverless, cloud scale
database.
It takes the responsibility away from us
for keeping critical infrastructure up and running
all the time, so that our teams can focus on the business
problems we have.
We knew going in that this is what we expected from Bigtable,
but it was a pleasant surprise, the convenience and the ease
of use we got from using a platform of this scale.
The main things that the customer should know
are the planning of the workloads.
Pay attention to the quarters.
Pay attention to the key designs,
so that the data is distributed and you don't get hot spots.
All of this, of course, is documented in Google's best
practices and absolutely important
that customers follow those best practices.
The other thing I want to highlight here
is the collaboration that we have with the Google product
engineers.
We work with the team to understand and set up
multi-region capabilities, and we
were able to collaborate with them
to reduce the replication latency that we
found when we set this up.
And we found that the Google engineers were
very collaborative and helpful and we were
pleasantly surprised at that.
ANDI GUTMANS: That's really great to hear.
If you could ask us for one new feature on Bigtable,
what would that be and how are you dealing
without that feature today?
YUVARAJ SANKARAN: So we were asked this before.
Bigtable secondary indices is the ask.
The current state is Bigtable organizes the data based
on row keys, and the access pattern
is based on this row key.
And since we host all of our journals in Bigtable,
we needed other ways to get to this data.
And so we built these secondary indices ourselves
using Bigtable.
And the ask is if Bigtable can make it part of their roadmap
so that we don't have to maintain that ourselves.
ANDI GUTMANS: I cannot promise, but I can definitely promise
that it's noted.
Thanks a lot for joining us today.
It was really exciting to hear about Equifax and the data
fabric you've built. I'm going to hand it over to Ron
right now.
RON PONTAFARO: And as Andi said, thank
you so much for being with us today, Yuvaraj, and telling us
about the transformation journey at Equifax.
Andi, thank you, as usual, for hosting.
And on behalf of everyone here in Google Cloud Databases,
thank you for joining us.
See you next time.
ANDI GUTMANS: Thank you.