DEBI CABRERA: Data by any other name would still be as sweet,
but how dost thou process and transform data?
AKA, data is very powerful, but how do you
process and transform it?
Join me as I show you with Apache Beam.
[UPBEAT MUSIC]
In this video, we'll walk through a demonstration
of Apache Beam.
If you haven't seen the last video,
I recommend checking it out since we talk about some
of the key concepts that make up a pipeline in Beam.
There's a link in the description.
In this example, we'll walk through using
Beam to count the words from Shakespeare's "King Lear."
And we'll show what it looks like to use Google Cloud
Dataflow as a runner.
This is a great starting example of a batch data job,
since we're working with a file which is bounded,
as opposed to streaming data processing, which is typically
unbounded because there's no fixed start or end.
Let's take a look at what our batch data pipeline does.
Here's what the pipeline looks like.
The first thing we'll do is read from a text file that contains
the entire copy of "King Lear."
After that, we'll use transforms to split
each line into individual words, count
the instances of each word, and output the results.
Each PTransform takes a PCollection as input
and sends one out as output so we can chain together
the transforms.
So first things first.
We can check out the input file.
This is just a short snippet, but the entire book
is included in the text file we'll be using.
Of course, there's lots of words in this book.
So let's get started counting them.
In this example, we're using Python.
So the first step is to install the Apache Beam files we need.
After that, we can run the word count example
and specify the input and output files.
Since we're not specifying a runner,
the default is to use the direct runner, which runs the pipeline
right on your machine.
If you're just getting started or looking to debug,
the direct runner is likely how you'll
want to run your pipelines.
After all, Beam is designed so you can run the same pipeline
code on different runners.
Your local environment might be fine for "King Lear,"
but what if your batch data job was
counting all of the words for an entire library's
worth of books?
In that case, we can just provide a few more options
to the code and use Google Cloud Dataflow as the runner instead.
The input file comes from a publicly-accessible Cloud
Storage file, while the output will be a Cloud Storage
bucket that I created earlier.
We're also specifying to use the Dataflow
runner and some other details, like which Google Cloud project
this runs on.
Once we run this command, the job
will be created and Dataflow and start processing.
Let's take a closer look at the code.
The core part of the code happens here,
after the pipeline has been set up with these options
that we provided.
This first part of the pipeline reads the text
from the input file and creates a PCollection of each line
from the file.
From here, we use read transform from the line's PCollection.
The first transform to split the line into separate words,
one transform to turn each word into a single kind of one,
and another transform to group and sum
all of the words and counts.
The pipe operator for Beam and Python
is short for applying a transformation to a collection.
So with this short block of code,
we're able to chain the PTransforms together
by using the output PCollection of the previous transform
as the input.
ParDo is a transformation that takes each element
from the input PCollection and performs a function on it.
In this case, we have a custom regex
that simply looks for groups of non-white space characters
and splits each line of text into words.
The final transform formats the text using a custom function
and finally outputs it to the bucket.
This is a fairly straightforward pipeline,
so these few transforms are all we need.
While this is running, let's take a look at Dataflow
in the Cloud Console.
You can see the status of the job
in each step of the pipeline.
The name for each transform step comes from the code.
You can also click on each step to see more details.
Over time, the interface will update,
and pretty soon, the pipeline will be complete.
Beam is designed to run pipelines
with parallel processing.
So if you're working on a large batch or a streaming data job,
the pipeline still looks the same.
Thankfully, Google Cloud and Dataflow
will handle all the infrastructure details for us.
So with just that short bit of code,
we've been able to get a count of all the words.
Now, if we take a look at the output file,
we can see the count of all the words in "King Lear."
However, you might notice that there are two separate counts
for the word "king."
That's because we didn't take capitalization
to account when grouping.
The easiest way to fix that would
be to add another PTransform.
After we split the sentence up into words,
we could simply lowercase all the words
and then pass that PCollection over to the group and sum
transformation.
Apache Beam is a powerful tool to help you process data.
And the word count example we just went over
is a simple way to get started.
There's some links in the documentation
with more information, including an interactive tutorial you
can take right in the Google Cloud Console.
Thanks for watching.
[UPBEAT MUSIC]