Welcome to 'Back to Basics'.
To gain business insight, you and your data engineering team
build and manage data pipelines
to extract, transform, and load data.
ETL provides the foundation to your data analytics and machine learning applications.
You can apply a series of business rules in ETL
to clean and organize data to address business intelligence use cases
such as monthly reporting.
Today, we will discuss modernizing ETL process
with an event-driven serverless data pipeline you can build on AWS.
Let's dive in.
When you start building an ETL data pipeline from the ground up,
you begin with provisioning infrastructure resources such as servers,
networking to get your ETL tools implemented, configured and tested.
This provisioning process could take weeks and months,
depending on the complexity of your environment and ETL tools.
Once deployed, you will spend time managing and maintaining ETL tools
by ensuring servers are patched, applications are updated.
I have experienced patching and fixing to stabilize ETL tools
so that overnight ETL batch jobs could continue.
It's not fun.
Once your ETL tools are deployed,
you will need to configure it to automate ETL processes.
For a batch job, you configure ETL processes to run when data becomes available.
A common method is running jobs on schedule.
Because data loads vary,
even a small batch job may run for different duration of times.
This may result in a waste of paying idling resources between two job runs
or a job failure due to an extra large load of data.
Many of you have experience with this,
and you may think, what if an ETL tool can easily scale
and you only need to pay for the resources that've been utilized?
What if your ETL tools are always available
for business logic configurations and you have no underlying infrastructure to manage?
Here comes AWS Glue.
Given there is a wide range of data integration use cases
you can design ETL pipeline solutions for,
we will discuss an implementation example
that you can modernize in scale ETL pipelines
using AWS Glue
by building an event-driven serverless ETL data pipeline on AWS.
A starting point with an ETL process is ingesting data.
You need a scalable data store for the ingested data
when building a scalable ETL data pipeline.
You can pick Amazon S3 as your serverless data pipeline
and use it as the primary data store of all your data.
By using Amazon DataSync SFTP application, or other techniques and applications
the document data arrives in a bucket on Amazon S3 service.
To make your users easily and securely find and access data,
the ingested data needs to be cataloged based on its schema.
You can scale and automate this process
and apply security access rules by using AWS Glue Data Catalog.
To turn this ETL data pipeline to be event driven,
so you avoid paying idling resources between the scheduled jobs,
you would want to trigger the data processing upon the data arrival in S3 bucket.
You can do this using an AWS Lambda function invoked by Amazon S3 trigger
to start an AWS Glue Crawler that catalogs the data.
You can program the AWS Glue Crawler
to scan data in all kinds of repositories,
including Amazon S3 and database services, classify it,
extract schema information, and store the metadata
automatically in AWS Glue Data Catalog.
To scale this event driven ETL data pipeline,
you would want to run jobs in parallel when required.
You can use Amazon SQS to manage the large volume of Amazon S3
triggered invocations in the event of varied data loads arriving simultaneously.
When AWS Glue Crawler is finished storing metadata in AWS Glue Data Catalog,
you invoke a second Lambda function using an Amazon EventBridge event rule
so AWS Glue Catalog can be used to guide ETL operations.
The AWS Lambda function starts an AWS Glue ETL job
to process and output data into another Amazon's S3 bucket.
For example, you may want to use an AWS Glue ETL job
to convert the data to a desirable data format,
such as Apache Parquet format.
You may also modify the ETL job to achieve other objectives
like more granular, petitioning compression, or enriching of the data.
To make this ETL pipeline more scalable,
you would want to automate the monitoring and notification process
as soon as the ETL job finishes, another EventBridge rule
sends you an email notification
using an Amazon Simple Notification Service SNS topic.
This notification indicates that your data was successfully processed.
There you have an event-driven, scalable,
highly automated ETL data pipeline
that you have no servers or underlying infrastructure to manage.
In this episode, we explored an implementation design
of building an event-driven serverless data pipeline on AWS.
I have used this design more than once.
Now you can use it too.
Check out the links below for more details.
See you next time.