Hi, welcome to This is My Architecture.
I am Dickson, Solutions Architect of AWS.
Today, we are glad to have Dick from Hong Kong 01.
He is going to talk about their data analytics architecture.
Dick, can you tell us about Hong Kong 01?
On top of online media, Hong Kong 01 also develops all types of business,
like charity crowdfunding, online food ordering services.
Since you guys have so many online platforms, you must need plenty of data analytics.
So what are the considerations in terms of architecture?
Actually, Hong Kong 01 values data analysis.
We have two main concerns for data platforms.
First, it should be able to process a large amounts of data.
Second, since we want the data team to focus on data analysis,
we adopted a serverless architecture
in order to reduce team maintenance time.
Would you talk about how a daily use case functions on the architecture?
Sure, I will take the collection of clicstreams as an example.
When our users browse the website,
the browsing history files a request to ALB,
which executes the request via Lambda.
Next, Lambda forwards the request to Kinesis Firehose,
and finally stores it in our S3 Data Lake.
As we can see, Kinesis is commonly combined with S3,
while ALB and Lambda is a relatively new support function.
So why do you guys go for ALB, rather than the traditional docking of API Gateways with Lambda?
Doubtlessly, API domain is a more versatile platform,
but our use cases are relatively limited.
Therefore, given the costs, we decided to adopt ALB.
Smart move! It is cost-saving.
Also, how do you employ the Glue part?
As a matter of fact, Glue is mainly used for data catalog,
and for storing some S3 table schema.
Raw data will be stored in S3,
and Glue will know what tables are available.
I figured you still have to do data transform and data cleansing,
so how is that done?
We are not using the traditional Hadoop or Spark platform.
Rather, we make use of Athena's CTAS function,
plus Airflow's scheduler function
to do data cleansing and data transform.
In the end, the transformed data is stored in S3.
Pretty neat!
You guys use Athena to perform transforming,
and then update it into S3 as an ETL.
I see that you also have Athena and Redshift.
When is each service chosen?
Actually, all the transformed data will be stored in S3.
To support BI tools and Dashboard,
we will transfer data to Redshift,
so our reports can be generated quickly.
At the same time, when needed, our business analysts
will be able to access those raw data via Redshift Spectrum
to come up with a more comprehensive data report.
So data analysts might use Athena,
while business users mainly access Redshift via BI tools.
In terms of machine learning, one will need to returns to S3 Data Lake to fetch the data.
I also noticed something unique.
What's going on with the combination of BI Tools with Lambda?
In fact, there are many restrictions with BI tools.
We may not be able to freely set a time for refreshing.
As a result, we schedule a Lambda function
and use it as a headless browser.
We can imitate the refresh of a manual data report,
while avoiding real manual work.
Anyways, Lambda is used to imitate manual F5 refresh.
I see a very complete data ingest in place for clickstreams,
so what are the future plans?
In fact, we value the needs of real-time users.
Therefore, in terms of Kinesis Firehose,
we plan to not only sending the data to S3
but also forward a copy to ElasticSearch.
This way, our business users can search for real-time data
in ElasticSearch through Kibana,
while aiding them to make some decisions.
As a result, Kinesis can generate some dashboards with slightly lower latency.
As for the dashboards with higher latency, we can do some more complex data analysis.
What an excellent data analytics architecture!
It provides many serverless features.
Thank you, Dick.
Thank you all for tuning in to this episode of This is My Architecture.
See you next time.