Hello. I'm Bryan Kang from 'This is My Architecture'.
I'm with Takyong Kim, Senior Engineer at Korean Air.
Yes, I'm Takyong Kim from Korean Air.
I work on the data team in the IT strategy department and work on a project to build an enterprise cloud analytics environment.
Hello. First, please introduce the project about constructing the data analysis environment.
Yes, it is a project to build an environment in which Korean Air can analyze a lot of data generated
while running its business and data of each part such as cargo ERP in the cloud environment.
We are migrating the existing on-premises data warehouse to Amazon Redshift.
We are promoting the modernization of our analysis environment using several services from AWS.
Yes, there seems to be a challenge to solve something while constructing this architecture.
What challenges were you trying to solve?
Yes, as you know, Korean Air is not a startup, but a company with a traditional airline business with a 50-year history.
Many existing operating systems were intricately intertwined.
So, in the analysis environment, such data does not affect the operating environment and the data is collected and analyzed.
And then the analysis speed is accelerated from batch data analysis to real-time data analysis,
and the cloud environment is good for agile development.
I was worried about how to apply existing governance and security policies into the cloud.
Okay, now let's take a look at the architecture.
There is a DMS at the front? Since DMS is usually our Database Migration Service,
customers who transfer data to the cloud use it to transfer data.
This time it is in the middle of the pipeline architecture.
Yes, we are using an ongoing replication (CDC) feature for our DMS.
In the source DB, the Oracle DB creates reservations, tickets, and boarding data
for all Korean Air customers.
This data is replicated to the target RDS in real time by using DMS.
So, in general, data is replicated within seconds.
In the case of large events such as flight departures, millions of data is generated from the source,
and data synchronization occurs within up to 3 minutes.
Yes, the database in front of this is the Oracle DB, and the target database seems to use RDS.
Is it the same type of database?
Not really. As mentioned earlier, in order to minimize the impact within the operating environment,
we left the existing DB intact, and used PostgreSQL which is an open source DB as the target RDS.
So, the DB type changed, but I was able to easily convert it by using AWS,
SCT, Schema Conversion Tool.
If you look at the layer you are saving, it seems that there is a part that stores data in S3 as well.
Yes, in addition to the RDS that we are targeting, the history of how the data has changed is
stored in S3.
S3 is much cheaper than a DB and provides unlimited capacity,
so we use it as a core service to expand our analysis environment to Data Lake.
In addition, the data entered in S3 could be easily searched through our Athena service.
Would you explain the subsequent processing of data?
Yes, the data stored in RDS or S3 is processed by ETL through AWS Glue service
and loaded into Redshift. Therefore, Fact and Dimension data loaded in Redshift
is used by our BI tools to help executives and business users understand the objective status of Korean Air's business
and discover insights.
Yes, it's called Code Series. In other words, you are using Code Pipeline, Code Commit, and Code Build, etc.
How do you apply this Code Series?
Yes, since we have thousands of jobs in the Glue service, it was difficult to manage them all in Glue.
So, we do the configuration management for Glue Code through Code Commit and Code Series.
We are using DEV and PRD multi-accounts. Code Series are for the automated distribution
of multi-accounts.
Yes, you are using a lot of managed services to discover another architectural feature.
What are the benefits of using these managed services?
For example, because we (AWS) directly manage the services, it is expected that the manpower for operations
may be reduced a little, right?
When we operate the system, basically, there are things that are necessary for IT operation, such as server management
with security patches. We feel less burden around that.
It has become an environment where you can focus more on the data required by the business field
and data processing.
Since the managed service is served, the configuration is not stuck in the time when the version was initially configured,
since new services are constantly coming out. For example, functions such as Glue 2.0
or Glue Studio came out with the new features that we needed. So I received a lot of help.
It also has made me keep studying AWS services.
Yes, please tell us about the future direction of this architecture.
As we move the data from the existing operating environment to the analysis environment by using DMS,
it becomes possible to perform data analysis in real-time.
In addition, customer data such as SKYPASS and other data are more integrated into Data Lake.
I have a plan like that.
We will try to create better customer service by integrating customer data
and doing more in-depth analysis such as machine learning.
Thank you.
This has been 'This is My Architecture'. Thank you.