SPEAKER: Processing data can be complicated,
especially when the volume of data
is variable, and even more so when the data is streaming.
There's lots of it out there, and you usually
need to work with it before you can start to get some value out
of it.
But how do you work with data efficiently,
and how do you scale it?
In these videos, we're going to talk
about Apache Beam, a flexible and open source
tool for processing data.
Specifically, it's built to unify batch and streaming
parallel data processing.
Let's start by talking about why Beam exists and then
go through how it works.
[MUSIC PLAYING]
You may be familiar with the original MapReduce paper
from back in 2004.
Over the years, the usage of MapReduce
grew exponentially in Google.
But people started running into limitations
with complex multi-stage pipelines.
The answer to this was FlumeJava,
which was released as a paper around 2011.
There was also a need for accurate, low
latency scaled pipelines, which the MillWheel focused on.
Shortly after, these two technologies
came together to become the foundations for Apache Beam.
Beam is designed to allow folks the freedom
to compose highly expressive batch or streaming
pipelines with the language of their choice
and to run those pipelines on a number of different runners.
Supported languages include Java, Python, Go, and SQL.
And you can also choose the runner,
which is where the pipeline will actually be executed.
For these videos, we'll be using Google Cloud Data
Flow as our example runner.
The flexibility of Beam goes beyond programming languages.
It also covers lots of use cases.
You can set up pipelines to deal with everything
from simple transportation, which mainly involves ingestion
and simple transformations, all the way to creating
a continuous intelligence solution.
As you can probably see, portability and scale
are key parts of Apache Beam.
So let's look deeper into how Beam
works by going over pipelines and a few key concepts.
Pipelines are basically the description
of how you'll be working with data.
Pipelines are essentially made up of one or more data sources,
a collection of transformations, and then one or more places
for the data to end up.
This pipeline also represents a directed, acyclic graph.
Let's talk about some of the core primitives
that make up a pipeline.
A PCollection is an immutable, unordered collection of values.
And it can contain a bounded or unbounded number of elements.
Beam uses PCollection objects as inputs and outputs,
operating on them with transforms.
A PTransform is an operation in a pipeline.
These transforms are basically functions
applied to each element of one or more input PCollections.
Depending on the runner, multiple workers
that cause a cluster may execute the code in parallel.
Ultimately, the transform produces one or more output
PCollections after the function is applied.
Since PTransforms input and output PCollections,
you can chain them together to make a pipeline.
And then there's the pipeline itself.
The pipeline object helps describe
the relationship between PCollections and PTransforms.
Since it can support one or more inputs and outputs,
you can even chain together pipelines
to support whatever kind of business logic you might need.
As I said before, data can be tricky to deal with.
These concepts might seem pretty straightforward
when looking at batch data, but streaming event data
can introduce more complexity.
Thankfully, Beam is designed to handle both batch and streaming
data processing with ease thanks to the ability to group data
using windows.
A window subdivides a PCollection
by using the timestamps of each individual element.
There is multiple types of windows
that can be used to group elements,
such as fixed windows or session-based windows based
on clusters of events.
It makes sense that an event probably
has a timestamp of when it happened,
but that doesn't necessarily mean that's when your data
processing receives the event.
Think about a mobile device that temporarily loses connection.
You'll know the timestamp of when the event happens,
like the different actions a user might take
while playing a game offline.
But when the device gets a connection
and sends all of those events over at once,
you may see a significant difference in the timestamps.
That's why we separate event time from processing time.
If you were to use the processing time as a window,
you might end up with events in the wrong area,
and your groups could be mixing up events.
Grouping by event time gives you the right groups.
But that introduces another question.
How do when you've received all of the data for a window?
Handling late data is done with triggers and watermarks,
but we'll talk about how that works in an upcoming video.
So today, we covered a brief introduction to Apache Beam
and started to go over some of the key concepts.
There's plenty more to cover.
And in the next video, we'll go through an example
of putting Beam to the test.
If you'd like to learn more, check out the links
in the description below.
And as always, thank you for watching.
[MUSIC PLAYING]