Amazon Managed Workflows for Apache Airflow
is a managed environment in the AWS cloud
for the open source Apache Airflow project.
Apache Airflow is a popular workflow management platform
The project was open sourced from the
very first commit, and the Airflow community
has hundreds of active members who support each other in solving their problems.
Airflow is commonly used for tasks like
running ETL jobs
managing Machine Learning pipelines, and automating DevOps tasks.
Customers love Airflow's supportive community
and robust ecosystem. But many have asked us:
"Can AWS help us to deploy and rapidly scale
Apache Airflow environments without having to set up and maintain
the infrastructure ourselves?"
Today, using Amazon Managed Workflows for Apache Airflow,
you can author, schedule, and monitor
workflows using Airflow inside AWS
without facing the common challenges involved in running your own
Airflow environments.
Managed Workflows take care of your Airflow environment setup, scaling,
and security, as well as handing
upgrades and monitoring.
This means you can spend a lot less time
managing Airflow environments, and a lot more time using them
to execute your data processing workflows in the cloud.
Getting started with Managed Workflows is easy.
You can create an Airflow environment and run your first
workflow in minutes. And, Managed Workflows
environments come pre-configured for high availability
and automatic scaling. Here's how to get started.
First, in the AWS Management
console
navigate to Managed Apache Airflow
click on create environment, and start by entering a name for your environment.
Here, you can also see the version of Airflow
that will be installed. Managed Workflows run
the actual open source Apache Airflow software in the
environments it manages for you.
Minor version patches for environment will be performed
automatically during a maintenance window you can define. And,
when major version upgrades are available,
you can apply them on-demand, with the ability to roll back to the
previous version if required.
Airflow workflows are expressed
as a collection of all the tasks you want to run, organized in a way
that reflect their relationships and dependencies.
In technical terms, this collection of tasks
is called a Directed Acyclic Graph, or DAG.
To define workflows in Airflow,
you write DAGs using the Python programming language.
Managed Workflows uses Amazon S3,
our Simple Storage Service, to access the Airflow DAG files.
So, the next step is to specify an existing
S3 bucket and path where your Airflow
DAG files will be located. You can also optionally
install Airflow plugins and Python modules
required by your DAGs by providing a plugins zip file
and a requirements text file. We'll click next to continue.
Managed Workflows create
the resources for each Airflow environment
inside an Amazon Virtual Private Cloud or VPC.
So, you'll need to select the VPC you want to use.
Each environment automically spans
multiple Availability Zones, or AZs, so
you need to select private subnets in two different AZs
from your VPC as well. Next, you need to
choose how you want to access the Airflow UI.
You can choose the Private Network option, which will create a fully-private
endpoint accessible only from inside your VPC.
Or, you can choose to create a publicly accessible endpoint.
Both options are authenticated and authorized
via AWS's Identity and Access Management System
or IAM. For this demo,
we'll choose the public network option.
Managed workflows provide automatic
worker scaling for you. When setting up a new environment, you
select an initial environment class, and a maximum
number of workers. The environment class controls the amount
of vCPUs and RAM available for the worker nodes
and can be monitored and changed at any time.
Managed Workflows will scale the worker nodes up to your maximum
to accomodate your workloads, and when there are
no further queued or running tasks, it will remove those additional workers.
The service will also automatically
monitor the health of the worker and scheduler nodes,
restarting them on fresh containers when required.
Managed Workflows make it easy to run Airflow
with in-built security. All of your data is encrypted
by default. You can choose to use an encryption key
managed by AWS or select a key that you've configured
in the AWS Key Management Service.
To provide environment observability, you can configure
the level of logging to persist for your tasks, as well as for the
web server, scheduler, worker nodes, and DAG processing logs.
These logs, as well as your environment
metrics, are stored in our observability service,
Amazon CloudWatch. You can also provide tags
for your environment if you wish.
To execute tasks, the Airflow workers assume an execution role.
This gives you a simple and secure way to access
other AWS services from your tasks.
Here, you can either select an existing role, or have the console help you
to create a new role for your workers. After selecting a role,
click create environment.
Once the environment becomes available
click the Open Airlflow UI link
to access the environment. As you can see,
this is the same AIrflow U that you would see if you downloaded
and installed Airflow on your own.
To finish this demo, let's upload a DAG from our workstation and run it in the cloud.
We'll view our Airflow environment details and click the link
in the DAG code in Amazon S3 section.
Then we'll click into the DAG
folder and upload a new DAG file.
After a short wait, the new DAG shows up
inside the Airflow UI, and clicking in to it,
we can load the Graph View to examine what it does.
This example DAG performs an ETL style workflow,
copying CSV files from a public dataset into an Amazon S3 bucket,
perfoming a join and aggregation across those CSV files,
and storing the result in S3 as well.
This DAG uses a built-in Airflow sensor to check
S3 and a built-in operator to execute
serverless SQL queries against CSV files
using Amazon Athena. But, because Managed Workflows
are running the same open source Apache Airflow
that you can download yourself, your workflows
can use any of the built-in sensors and operators that are part of the Airflow project,
as well as the features from any plugins
or Python libraries that you choose to install into your environment.
Finally, as we mentioned before,
Managed Workflows integrates with Amazon CloudWatch for logging and monitoring.
No matter how many Airflow environments you create,
you can view their logs and monitor all their essential metrics
like Task Delays, DAG Load Failures, and more,
from one central location.
To learn more, please visit our website
and thanks for joining me for this brief overview.