- From health care to education to advertising,
data is helping businesses innovate to new heights.
A key challenge is not just how to find data,
but how to make it useful.
Data scientists and data engineers
often need to build data pipelines to ingest, process,
and analyze their data.
These ETL and ELT type workflows usually
involve multiple services that need
to be coordinated and executed at the right time
and under the right conditions.
In our previous video, we learned
about how orchestration is the automation, management,
and coordination of workflows.
In this video, we'll focus on how
you can orchestrate your data workflows in Google Cloud using
Cloud Composer.
This is where Cloud Composer, Google Cloud's fully managed
orchestration service, comes into play.
You can use Composer to create, schedule, and monitor
end-to-end workflows using Python.
Composer is built on the open source Apache Airflow
Project, which has a huge community of contributors
and users.
And since it's a fully managed service,
you don't have to worry about managing Airflow deployments.
Composer takes care of infrastructure management
for you, so you can focus on actually building your data
pipelines.
Composer supports many different use cases,
but most of its users are data engineers
who are building workflows to orchestrate their data
pipelines.
There are several key advantages that Composer has for the data
workflows use case.
First, the Airflow Project has hundreds of operators
and sensors you can use.
These make it easy to communicate
across multiple Cloud environments
with very little code.
In addition to Google Cloud operators,
there are multiple operators for other common interfaces
like MySQL and Slack.
Second, Composer has sophisticated task management.
You can parallelize and branch tasks and define
how you want to handle failure.
For example, you could send out an email to the team
if something goes wrong along the way.
And third, keeping track of your workflows
is straightforward with Composer's built-in monitoring.
All of this is important as your workflows scale
and become more complex.
And finally, Composer comes with enterprise-ready security
features, like customer-managed encryption keys and VPC service
control support, enabling organizations
to use it right away without worrying
about security compliance.
It's important to note that Composer is not
designed to transform, process, or pass along
large amounts of data.
Instead, Composer is great for managing and coordinating
tasks delegated to other products that handle your data.
Some products you might want to coordinate using Composer
include BigQuery, Dataflow, Dataproc, and Data Fusion.
These are all great at operating directly on big data.
Let's take a look at an example use
case of how Composer can use to orchestrate a big data
pipeline.
Composer tasks are defined in the form of a Python Directed
Acyclic Graph, or DAG.
First, we monitor a Cloud Storage bucket
and trigger the pipeline as soon as the file appears.
Then we kick off a Cloud Dataflow pipeline.
Next, we execute BigQuery SQL job
to validate the process data.
And finally, we delete the object
from the Cloud Storage bucket since we no longer need it.
Along the way, if anything goes wrong,
a Slack notification could be sent to team members.
And there you have it.
Data orchestration is becoming increasingly important
for data engineers and developers
to manage their tasks and services.
By having Composer orchestrate workflows on their behalf,
data engineers can focus their efforts
on the more creative aspects of data engineering.
Make sure to check out the next video in our series, where
we'll learn more about how to build workflows
in Cloud Composer.