SPEAKER: How are you coordinating and managing
your workflows?
If you're using cron jobs with custom scripts,
you know that it can get super tricky to stitch together
all of your services and applications
together to build a workflow, especially as your workflow
scales in size and complexity.
In this video, we'll explore some of the orchestration
services that Google Cloud has to support your workflows.
So what is orchestration?
Orchestration refers to the automated configuration,
coordination, and management of computer systems and services.
You can think of orchestration as the process
of stitching all of your services and applications
together to build an end-to-end workflow.
Perhaps you want to automate the execution of a single service
at a specific time of the day.
Or maybe you have a more sophisticated approach
of not just automating one service
but automating, coordinating, and monitoring
multiple services over longer periods of time.
There are several orchestration tools
that Google Cloud has to help support your workflows.
These include Cloud Scheduler for scheduling
single repetitive tasks, Workflows
for complex multi-service orchestration,
and Cloud Composer for data-driven workloads.
First, we have Cloud Scheduler.
Cloud Scheduler is a simple scheduling tool
that allows you to schedule the execution
of a single HTTP-based service on a recurring
schedule you define.
Many customers use Scheduler alongside other Google Cloud
tools like Pub/Sub and Cloud Functions
to execute their code in a serverless manner.
Cloud Scheduler is great if you need
to call a single service at regular intervals.
For example, you could use Cloud Scheduler
to schedule the execution of a Cloud Function
that sends out daily emails with top news
stories about technology.
But what if you have multiple services
that you need to chain together, feeding the output of one
service to the next?
Or what if you need to apply complex logic to determine how
and when services are invoked?
This brings us to Workflows.
Workflows is a service for orchestrating
multiple HTTP-based services into a durable and stateful
workflow.
Like Cloud Scheduler, Workflows allows
you to automate the execution of HTTP-based services running
on Cloud Functions and Cloud Run as well as external services
and APIs.
But unlike Scheduler, Workflows gives you
the power to connect multiple microservices as part
of your larger workflow.
When you create your workflow, you
can use either YAML or JSON.
You specify the order of the services
as well as how the result of one step
can be used as either an input to the next step
or as a condition to determine which step to execute next.
Workflows is perfect for chaining together
microservices, automating infrastructure tasks
like starting or stopping a VM, and integrating reliably
with external systems.
To execute a workflow, you can manually trigger it
via an API call or in the Cloud Console,
or you can set up a recurring schedule with Cloud Scheduler.
If your focus is on engineering data pipelines or big data
processing, you should consider Cloud Composer.
Composer is a service designed to orchestrate
data-driven workflows and is built on the popular open
source Apache Airflow project.
Since it's a fully managed service,
you don't have to worry about managing Airflow deployments.
Composer takes care of infrastructure management
for you so you can focus on actually building your data
pipelines.
It also supports your pipelines wherever they are,
whether that be on premise or across multiple Cloud
platforms.
Like Workflows, you create a task
for each step in your workflow, configure the order
of the tasks, and specify which tasks to execute next
based on some conditions.
Tasks are expressed as Python Directed Acyclic Graph, also
known as a DAG, that can be scheduled
to run at a time of your choice or in response
to a particular event.
Operators allow you to communicate
with services across multiple cloud environments and on-prem.
In addition to many operators in the Apache Airflow ecosystem,
there are over 150 operators for Google Cloud alone.
For example, your Composer workflow
might start by triggering a data ingestion job in BigQuery,
then start a batch processing dataflow pipeline,
and finally send a Slack notification to teammates
that the data processing workload has finished.
Both Composer and Workflows are great for orchestrating
multiple services and handling long-running workflows.
However, there are several key differences to highlight.
Composer is designed for orchestration
of the transformation of data as part of ETL or data engineering
workflows.
Workflows, on the other hand, is focus on
the orchestration of HTTP-based services built
with Cloud Functions, Cloud Run, and external APIs.
While Composer is designed for orchestrating
batch workloads that can handle a few seconds of delay
between tasks, Workflows is designed
for more latency-sensitive use cases.
And while Composer tasks can be modeled
as a directed acyclic graph, Workflows
can work for use cases where you need loops and jumps
to already executed steps.
So there you have it-- an introduction to orchestration
in Google Cloud and how you can use Scheduler, Composer,
and Workflows to manage and configure your services.
Check out the next video on data orchestration
with Cloud Composer.