Hello, Cloud gurus and welcome to the GCP service spotlight. In this series,
we put the spotlight on specific GCP services,
walking you through detailed technical demos to explore their features,
benefits, and costs. I'm Karlos Knox.
And today we're going to talk about Google Cloud's Dataflow.
Well, during this episode, to help me explain what Dataflow is and how it works.
I have a couple of props. In my hand, I hold a raspberry PI or small computer,
if you will, attached to a heart rate sensor.
And this device is going to automatically collect data every one minute,
taking my heart rate for one hour,
I've configured this device to send data to Google Cloud. Now,
collecting this data in Google is similar to what all businesses do today.
Data's being collected from all areas of business websites, mobile apps,
IOT devices, and other workloads. Capturing processing,
and analyzing this data is a priority for all businesses. However,
data collected from these systems is not always in a format or location
required for analysis.
And that's where Google Cloud's Dataflow can be of service.
Dataflow intakes data from a source,
performs some transformations and deposits the data in a destination location.
Dataflow is a managed service for executing a wide variety of data pipelines to
collect, transform, and write data to storage or a data warehouse.
And we create Dataflow jobs using three methods,
the Google Cloud console UI SQL query language or
AI platform notebook.
Dataflow supports both batch and streaming pipeline jobs when written using the
Apache beam SDK.
Now Dataflow provisions the environments necessary to execute the creative
pipeline, which is known as a Dataflow job.
What do you need to run Dataflow jobs?
Dataflow is serverless and Dataflow automates the provisioning of resources and
managing a cluster of machine Dataflow can automatically scale the workers if
demand exceeds capacity and scale them down and things calm down.
To recap,
Dataflow executes both batch and stream processing pipelines.
You're going to create your pipeline using the Apache beam SDK and Dataflow
scales out resources based on set project quotas.
If Dataflow sounds pretty good by now,
you're probably wondering how much does it cost and the answer there is,
it depends.
There are separate charges for batch and streaming jobs with streaming jobs
being slightly more expensive.
Both batch and streaming processing jobs will utilize compute resources,
but these resources will appear on your bill as Dataflow,
not the individual components. So back to the price,
each job uses at least one Dataflow worker and price depends on
worker configuration, which is the compute instance size you select.
Cost per Dataflow is billed per second per job resource use.
Let's switch over to the billing page so you can see it firsthand.
You will be billed in one second increments for VCPU memory,
persistent disk storage utilized for processing.
There are also charges for data throughput and Dataflow shuffle,
which moves the data to the processing components and to the final destination.
Usage will be displayed on your bill in hourly increments. Let's recap,
the costs were charged based on the compute instance type of the worker nodes,
Dataflow shuffle, add some additional cost for data movement.
Now these prices are subject to change so check the pricing webpage for details.
One more thing for batch processing jobs - you can use flex RS.
If you can run your jobs on a flexible schedule,
then you can select to mix in preemptive VMs in your worker nodes for a flat
discounted costs. Let's quickly recap a few of the benefits Dataflow offers.
Automated resource management. It's serverless,
Means you get to focus on the business needs,
not managing the tools performing the data processing.
Dataflow even watches for hotspot or bottlenecks and rebalances processing
accordingly. And if Dataflow can't rebalance due to usage,
the service will skill the worker nodes to a pre-allocated maximum during peaks
and scale them back down to a minimum during slow ingestion periods.
That brings me to my favorite benefit of using Dataflow.
There are multitude of templates available that perform ingestion transformation
and loading the data into a destination. Let's recap the benefits,
automated resource management, dynamic workload rebalancing,
and those Dataflow templates. That brings us to the demo of Dataflow.
We're going to use a Dataflow template to move my collected heart rate device
data from a pub sub topic in Google Cloud to a big query table.
Let's head to the demo and check it out.
I'm sure you remember that raspberry PI from the introduction that was going to
collect heart rate data. This demo begins at the Google IOT core service.
We start here because this is where the data from my makeshift heart rate sensor
arrives in the Google Cloud platform.
Now I registered this device in a heart rate data registry and based
on the heart rate data registry settings,
which says these events will be collected in a pub sub topic at the following
location, which is your project,
topics and heart rate store. Now to make this easier,
I just sent a couple hundred sample heart rate event messages.
Now I have a tab already open to the pub sub subscription dashboard,
and I'm going to switch there. Now,
these are the pieces of our data pipeline that we're going to use with Dataflow.
The topic is the location where the heart rate data is stored.
The subscription to the topic is the source for Dataflow to retrieve
the heart rate messages,
a subscription enables data to be pulled from a topic.
But what we're here for is to validate that the messages are arriving in the
Google Cloud platform. Success.
We've just validated that the data from the heart rate sensor is currently
stored in the heart rate data pub sub topic.
Now we could scroll through the data using this preview window and gather
information about the message such as time region, project,
and even what's in the body of the message,
which contains the heart rate sensor ID,
a time collected and the unique ID of each message with the source of our data
in Google Cloud.
We still have one more step before we actually get to Dataflow and that's big
query. And if you've never used big query, well,
it's like a large scale Excel. There'll be a table with columns and rows.
In this demo,
our data set is called heart rate data and the table is called the heart rate
data table.
This table has four fields that match the message data collected from the
sensor.
And this table will be the destination location for our Dataflow job.
The final tab I have open is the dashboard for the Google Cloud storage bucket.
And in that bucket is a temporary folder and we use this temporary storage
as a temp location for Dataflow to perform the transformation of our data from
Jason to table format.
Now you have the layout of the components that we need to use Dataflow.
So let's use Dataflow to move the data from our pub sub topic to the
big query table.
So the data's better set up for analytics and visualization.
Let's get to it to get started with Dataflow,
let's select the navigation menu.
Now I pin the Dataflow selection to my shortcut list,
but you will find Cloud Dataflow in the big data section of the menu.
Now we'll right, click and open that link now.
And from the Dataflow page, we can use jobs and notebooks.
We can also create a job using the SQL query language. However,
we're going to create our job from a template and to create our Dataflow
template. We need to supply three pieces of information,
the job name we'll call this the heart rate data mover
for the regional endpoint. All of our data is going to us central one,
which is the closest to me.
However you could select the region that works for you.
And the third piece of information we need is a Dataflow template. Now,
now there's templates to just get started.
Then there's templates to process data continuously for streaming data.
There's a section to process data in bulk or batch. Finally,
there are some utilities to do compression and conversions on the data,
as well as the ability to load our own custom templates.
The template we're going to use is at the very top under continuously stream,
and it's the pub sub subscription to big query. Now,
once we select the template, the Dataflow process goes into overdrive.
There's lots of information that's shown in the screen on the right-hand side is
a how to use this Dataflow template overview.
It outlines each of the steps that taking place based on your selections on the
left-hand side of the screen.
Now I'm going to close that shortcut window and let's take a quick look at the
diagram.
The first step reads from a pub sub subscription.
Next it's going to convert our messages from Jason to a table in row format,
following it up,
but writing the successful records to the destination location on the left-hand
side is the continuation of our Dataflow job creation wizard,
which now requires three additional pieces of information.
It basically walks us through creating the job.
The first parameter that we need to input is our pub subscription,
where we get the data from.
If I head over to the pub sub subscription page, I'm going to close this window.
I'm going to copy this description name,
which contains the end point needed to access this subscription.
Also the name we input for our Dataflow job. Next,
we need a big query table to output the data ingested in this data
processing pipeline. Now in big query, I'm going to take a shortcut.
I'm going to use the query table option,
and then I'm going to copy the entire path of this query. It's project name,
dataset, name, and table name. Copy that.
I'm going to head back over to Dataflow and paste in the path to the big query
table Dataflow expects the table path to be in the form of project name,
colon table, name dot table.
So swapped a period after the project for colon
and the final piece of information we need to supply for our Dataflow job is a
temporary location to write the data in order to provide transformations,
we're going to use a temp folder in the Google Cloud storage bucket called HR
temp data for this purpose.
And with those three pieces of data,
the Dataflow template job is ready to submit.
I'm going to click run job now and sit back and relax while Dataflow does the
hard work now is this is the first Dataflow job running.
There are no Dataflow resources.
Currently active Dataflow is going to trigger an auto-scale action,
which will provision compute resources necessary to carry out this job.
So if we watch the right-hand side of the screen under the resource metrics,
we can begin to see resources come online. Now,
this process is going to take a few minutes for all the pieces to connect and
the worker resources to be available. In the meantime,
let's head over to big query and I'm going to keep track of how long it takes
for data to arrive. Welcome back.
It took approximately two minutes and five seconds.
From the time we created our job for the data to be moved from pub sub to big
query. And when I say moved, that's accurate.
If we go back to the pub sub subscription and click on view messages by
accessing those messages,
Dataflow has deleted them from pubsub and now the only storage location is the
big query service heading back over to big query.
We can now scroll through the information in a table format,
or we could use some simple query language like order by time collected to
sort by the time it was actually received.
Once we had the data in a format that we like, we can explore the data,
what data studio I'll change a couple of things around.
We'll take the heart rate data and sort it by the time collected.
And if we go back and select a line chart,
we can see the graph of our heart rate data.
And you'll notice there's a little spike at the end.
And that's the power of using a tool like Dataflow,
to move data to an analytics program where we can quickly visualize
the differences and what's going on in our environment,
the power behind visually seeing the data makes it really easy to find any
anomalies.
Now Dataflow will continue to process any data sent in a streaming fashion,
which means as it arrives, it will be moved into big query. In addition,
we could use data studio to create charts and dashboards that showed us the
various trending patterns over time.
I hope this helps you understand more about how you can use Dataflow in your
daily job. You can use Dataflow to move data from a source,
transform it,
and then write it to a destination using built in templates using SQL language
or AI notebooks.
So what's the verdict? With no hardware or software to manage Dataflow,
templates, supports both batch and stream processing,
it auto scales up and down.
Now some of the cons to create custom jobs,
you will need some knowledge of the Apache beam STK,
or how to use the SQL query language to build your data pipeline. Now,
since you invest evaluate or revisit Dataflow,
my recommendation is to invest Dataflow extracts data from an input source,
transforms the data and loads it to an output destination such as the database,
data mart or data warehouse for reporting analysis and
synchronization using Dataflow templates.
We were able to move data from pubsub to big query using a few clicks by
providing the correct end point Dataflow is serverless,
which means there's no maintenance of the pipeline execution environment.
Simply define the jobs and let Dataflow handle the risks.
So that's it keep being awesome Cloud gurus.
And I hope to see you next time on the GCP service spotlight.