Welcome to 'This is My Architecture'.
I'm Jennifer at AWS.
And I'm joined today
by Baldev from Toyota.
Hi, Baldev, thanks for joining us today.
- Hey, thank you so much for having me here. - Sure.
Many of us know about Toyota, can you tell us a little bit about, more about Toyota?
Yep. It's a great company to work at.
And we have a very awesome culture.
People probably around the world know about Toyota,
that it produces high quality cars.
Awesome.
So can you tell us a little bit about this architecture
and how this solves your business challenges?
Sure.
The problem that we solved is the on-prem systems,
we had data warehouses and data marts
which have been purpose built.
And they have domain data in there,
but it has been a challenge to get all of the data into the cloud.
So with this architecture,
we are trying to converge the data warehouses and data marts into the Lake House.
That's great.
So how are you transforming and ingesting this data from on-prem to the cloud?
Sure. The journey starts with ingestion.
So around EC2...
We have Kafka...
We have Talend...
And then we have Databricks.
So the data is securely gotten through the Direct Connect.
And using these technologies, which are running on top of EC2,
they pump the data into these layers.
That's neat.
Can you tell us more about the data types and how that is moved into S3?
Sure. So I'll start from top down.
One is Glue, where...
What is this data about, right?
So the catalog aspect, the meta data aspect is all stored in Glue.
S3 actually has the data, and then when it comes to RDS,
it's about the various processing that's happening.
How are the metrics looking like, so this helps with storing all of that.
And in here, we have the source data...
We have the enterprise data, and then the final product.
So you can correlate the source data as the main origin, the enterprise data,
where you can think of it like the data warehouse, and then the product,
you can think of it like the data mart.
So I see Redshift and Athena services, how are you leveraging these two services?
Yes, we have various actors.
One is the analysts, like our data scientists,
and then we got business users who do a lot of self-service.
So these two engines solve different purposes.
One is the data mart, which we're talking about here.
So this is pumped into Redshift.
And this gives us a read optimized layer.
The other aspect is on the self-service where data which is not in Redshift,
for that we use...
Spectrum as well as Athena, where these engines are on the fly.
Spectrum is a fleet of servers which extends on the fly.
And then these two engines go directly against S3 to fetch the data
in an effective manner.
And since we have the data in Parquet and DeltaFS, they're pretty fast.
And the analysts and business users and data scientists are able to process
this data much faster.
And the data, since it's enterprise, we can do cross domain joints,
and the extension is way beyond just a data warehouse and data mart.
Okay.
So your end users are able to consume and, you know,
process the reports and get reports out of this data.
- Correct? - That's correct.
So this has helped us with achieving the data warehouse
and data mart convergence into the Lake House.
And we started with vehicles, but now we have parts,
customer finance and so on, which are going to come along.
This has helped us with time to market...
No data duplication.
And we've already seen the benefits because recently,
an application which typically takes months to a quarter,
to a year, took us only a month.
That's a fantastic story.
Thank you for walking us through your Lake House architecture.
And thank you for watching 'This is My Architecture'.