PRIYANKA VERGADIA: Welcome to "Google Cloud Drawing Board,"
where we doodle our way through the cloud.
Today's topic, What is Dataflow?
This video is divided into chapters.
So watch the full video or skip ahead
to any section of your choice.
Data is generated in real time from web sites, mobile apps,
IoT devices, and other workloads.
Capturing, processing, and analyzing this data
is a priority for all businesses.
But data from these systems is not often
in the format that's conducive for analysis
or for effective use by downstream systems.
That's where Dataflow comes in.
Dataflow is a serverless, fast, and cost-effective service
that supports both streaming and batch processing.
It provides portability with processing jobs
written using open-source Apache Beam libraries.
And it removes operational overhead
from your data engineering teams by automating
the infrastructure provisioning and cluster management.
How does Dataflow work?
In general, a data processing pipeline involves three steps.
You read the data from a source, transform it,
and write the data back into a sink.
The data is read from the source into something
called a PCollection.
The P stands for Parallel, because a PCollection
is designed to be distributed across multiple machines.
Then it performs one or more operations on the PCollection,
which are called transforms.
Each time it runs a transform, a new PCollection is created.
That's because PCollections are immutable.
After all of the transforms are executed,
the pipeline writes the final PCollection
to an external sink.
Once you've created your pipeline
using Apache Beam SDK in the language of your choice, Java
or Python, you can use Dataflow to deploy and execute
that pipeline, which is called a Dataflow job.
Dataflow then assigns the worker virtual machines
to execute the data processing.
You can customize the shapes and size of your machines.
And if your traffic pattern is spiky,
Dataflow autoscaling automatically
increases or decreases the number of worker instances
required to run your job.
Dataflow's streaming engine separates
compute from storage and moves parts of pipeline execution out
of the worker VMs and into the Dataflow service backend.
This improves autoscaling and data latency.
Now, how do you use Dataflow?
You can create Dataflow jobs using
Cloud Console UI, gcloud command online interface, or the API.
You have options for creating the job.
You can use Dataflow templates, write a SQL statement,
or use AI Platform Notebooks.
Dataflow templates offer a collection
of prebuilt templates with an option
to create your own custom ones.
You can then easily share them with others
in your organization.
Dataflow SQL lets you use your SQL skills
to develop streaming pipelines right from the BigQuery web UI.
You can join streaming data from Pub/Sub
with files in cloud storage or tables in BigQuery,
write results into BigQuery, and then build real-time dashboards
for visualization.
You can use AI Platform Notebooks from the Dataflow
interface itself to build and deploy
data pipelines using the latest data science and machine
learning frameworks.
Dataflow inline monitoring lets you directly access job metrics
to help with troubleshooting pipelines at both
the step and the worker level.
How to secure Dataflow pipelines?
When using Dataflow, all the data
is encrypted at rest and in transit.
To secure the data processing environment,
you can turn off public IPs to restrict access
to internal systems, leverage VPC service
controls that help you mitigate the risk of data exfiltration.
Additionally, you can create a pipeline
that's protected with customer managed encryption keys.
Now, what does it cost to use Dataflow?
Dataflow service usage is billed in per-second increments
on a per-job basis, depending on streaming or batch data.
For batch data processing, you can
utilize the flexible resource scheduling
feature that reduces the cost by using advanced scheduling
techniques.
Each Dataflow job uses at least one Dataflow worker,
and the price depends on the worker configurations.
Where to use Dataflow?
Dataflow is a great choice for any batch or streaming data
that needs processing and enrichment
for the downstream systems such as analysis, machine
learning, or data warehouse.
Some examples are stream analytics enabling
real-time business insights; real-time AI
enabling predictive analytics, fraud detection,
personalization; processing streams of log data,
unlocking health insights for your systems;
and of course, any other data aggregation
and analysis scenario.
Want to learn more about Dataflow?
Check out cloud.google.com/dataflow.