Hello. This is Yongjin Cho from Amazon Web Services.
In this 'This is My Architecture', we've invited Yousu Mok, who built the BI platform
on Lotte ON, the new e-commerce platform of the Lotte Group.
Hello.
Thank you for coming today.
First, please introduce yourself.
I'm Yousu Mok from the Big Data Business Team of Lotte Information Communication.
Thank you for inviting me.
The architecture that I will introduce today is Lotte ON's BI platform.
Our platform provides sales, link traffic, and multidimensional analysis results of online stores
or open market sellers of Lotte affiliates in the semi-real time.
BI platforms have various characteristics depending on the purpose of use.
What was the most important thing when you were designing this platform?
When I planned and designed the platform, we aimed to integrate the platform with business data within the affiliates.
The users in our platform are salesmen, marketers of affiliated distributors,
and executives.
By providing the integrated management of affiliated distributors and indicators for each affiliate,
the management visibility of the e-commerce business is increased,
and various linkage systems are added centering on Lotte ON
to provide the data analysis in web and mobile formats.
So, the platform itself was designed to be very flexible,
and highly scalable.
It also has high performance because you considered the various affiliates and users.
Then, would you please explain what kind of data analysis you use through this platform and how you get insights.
First, let me explain by looking at the diagram.
We have various data sources such as traffic DB, DynamoDB, RDS, Redshift, and Elasticsearch.
The structured and unstructured data is stored in the Data Lake
by using the third-party ETL tools and Glue.
This stored data is purified in Glue by using Spark, Python, Redshift Spectrum and Procedure,
and then the purified data is stored in Redshift.
This stored data is provided to customers by using BI tools.
In order to provide as much real-time data as possible, different pipelines such as 10 minutes, 1 hour, and daily
are configured and utilized according to the task.
Yes, I think they are providing data analysis results in different time units.
In that case,
if you have to process data every 10 minutes, it will be difficult when the amount of data is huge.
How did you configure it?
First, we used Redshift's Spectrum feature
to process data every 10 minutes.
The Redshift Spectrum's SQL is used by directly accessing data.
In the case of the contrast indicators such as yesterday, last month, and last year,
the daily batch is configured and used in advance.
Yes, when I look at the architecture, the next part seems more complicated.
There seem different data sources.
Those different data sources would have different formats.
How did you integrate them?
Yes, first, we are taking Parquet format as the data format.
And, in the case of overlapping data such as traffic DB and DynamoDB,
we do that in Parsing.
First, in the case of traffic DB, it is saved in the Data Lake
as Avro format.
The saved data is converted to Parquet format and used in Parsing by utilizing
the Redshift Spectrum function.
In the case of DynamoDB, the columns of nested data are declared as streams,
the declared data is stored in Redshift, and the JSON function in Redshift is used
in Parsing.
Thank you for the explanation.
You've explained how to integrate data, and the current architecture.
What's your plan to evolve this architecture in the future?
Yes, we plan to provide forecasting insights by using the AWS Forecast service
and expand real-time analytics through streaming data collection.
I understand that you are going to use AI or machine learning to advance the analysis results
and the construction of a streaming data pipeline that take less time.
Thanks for your explanation in detail.
Today, we have introduced the architecture of Lotte ON's BI platform.
Thank you, Yousu Mok, for providing such a detailed introduction.
This has been 'This is My Architecture'.
Thank you.