[MUSIC PLAYING]
DEBI CABRERA: Data comes in many shapes and sizes.
In the last videos, we introduced Apache Beam
and walked through a batch data processing job.
But how do you handle processing streaming data?
Let's get into it.
First things first, we need to define batch processing
and stream processing.
Batch processing uses data that has a fixed size,
because it represents a data set that's already completed.
We went through one example by looking
at all of the words in Shakespeare's "King
Lear" in the last video.
But it could also be a much larger data
set, like all the words in every book in a library.
Once you get your hands on all of the data,
you can start processing it.
Beam refers to this type of data as bounded,
because you already have all the data you'll want to work with.
Stream processing uses unbounded data,
where you don't have a complete set of data
because it's still being updated.
If you think about a social media feed,
for example, there's constantly new data being added.
And you may need to process new events as they come in.
Since the data set won't ever be complete,
you need to process events as they stream in
and compute results on aggregated data.
In order to process unbounded data,
Beam uses something called windows.
Let's break down what windows are.
In Beam, a window generally represents a range of time.
By default, Beam will put every piece of data
into the global window, which represents
the range of time between negative infinity and infinity.
When you perform any operations or transforms on your data,
they will apply to the entire window.
This is what you'll usually want when working
with a batch processing job.
But what happens when you're working with an unbounded data
set?
Instead of using the entire global window,
you can divide elements between smaller intervals of time.
These windows represent time ranges,
so you can process the data in each one.
For example, you could have 60-minute windows
and count the number of elements in each window.
The count of each window would tell you
how many pieces of data were processed for each hour.
Because each window is eventually complete,
you can process them independently and produce
a final answer for that window.
The global window will contain all the data
as it's being processed.
But that's a pretty large range of time.
When you define how you want your data to be windowed,
you can configure things like how large
the windows are and whether or not they'll overlap.
Before we dive deeper into exactly how windows
work though, the right configuration
depends on how you want to process your data based
on its timestamp.
But what do we mean by a timestamp?
Well, if we think about an event that
happens on a mobile device, it's easy enough
to grab the time that the event happened,
according to the device.
We call this event time.
However, that event will also need to be sent to a system,
like Beam, for processing.
There will always be a difference between the time
that the event actually happened and when
the event is processed, or the processing time.
Especially when you're receiving events
from multiple devices with different connectivity
and latency, these events will almost always
arrive out of order.
If you use processing time to aggregate your events,
you'll only get an approximate result at best.
Instead, you'll want to use the event
time to handle your data correctly and get
accurate results.
Let's take a look at an example so we
can see why processing time won't give you accurate results
and go over how windows work.
Here's a simple example data set of events
that you might see for a mobile application.
We can see that each of these events
is for the same user ID and each item
represents different user actions, like tapping
or swiping.
For this data, the event time property
will represent the actual time that the event happened and was
collected on the mobile device.
To keep things simple for this video,
we'll look at the event time as a number of seconds
rather than showing a full Unix timestamp.
If we wanted to aggregate and count the number of events
for this user, it would be a pretty straightforward count.
We could group the events by the user ID
and then count them, which would give us
a total count of five events.
As we mentioned before, even though we're
looking at these events ordered by timestamp,
there's no guarantee that they'll actually arrive
for processing in this order.
Data in a distributed environment
may be inherently unordered.
And the Apache Beam model doesn't make any assumptions
about data ordering.
That's why we mentioned in earlier videos
that peak elections are immutable and unordered.
OK.
Back to our example data.
We know that there are some events
and we want to process them.
If we had all of the events ahead of time,
we could consider it bounded data
and use a batch processing job.
In reality, we'll often want to process unbounded data
with the possibility of more events coming in at any time
and handle events arriving out of order.
That means we'll need windows.
If we don't configure anything for windows,
Beam will place all of our elements in the global window.
If we wanted to get a count for the events,
we'd have to wait for the data to be complete.
And since our example represents unbounded data,
new events could arrive at any time
and the window will never be complete.
There is a way to get speculative results when
working with unbounded data, but we'll talk more
about that in another video.
In order to get the complete results we're looking for,
we'll need to subdivide the data into groups
that we know will eventually complete using windows.
We'll start by using fixed windows.
These are the windows of the same size that repeat, like,
every hour or every 15 seconds.
Using our same data from before, we
can use fixed windows that have a size of five seconds.
Using these windows, we can do a simple aggregation
by counting the data inside of each window
after the window is complete.
So the first window has a count of three events
and the second window has a count of two events.
The right size for your window depends
on your use case and your data.
But you can specify the length to Beam
and it takes care of the rest.
Beam can estimate and track when a window is complete,
which we'll talk about more later in this video.
However, we did talk about this being unbounded data.
That means more events could come in at any time.
And they're not guaranteed to arrive in the same order
that they happened.
And just like that, here is a new event.
While this event looks the same as our other ones,
the event time is actually at 4 seconds.
So the event itself would've happened before the events that
happened at 6 and 8 seconds.
Since data like this can continue to arrive at any time,
Beam needs to determine when to stop adding new events
and process the data inside of a window to produce output.
Triggers are the mechanisms that Beam actually
uses to emit the aggregated results for each window.
The default afterwatermark trigger can be overridden.
And there's multiple types of triggers
that you can customize, depending on how
you want to process your data.
We'll talk more about those in the next video too.
When collecting and grouping data into windows,
Beam uses a watermark to keep track of how up to date
the events in that window are.
As data flows through a pipeline,
Beam keeps track of the oldest event timestamp that it
expects to see at each step.
This is how Beam can handle and estimate
when a window is actually complete
and all the data is expected to have arrived,
which also means Beam can process the complete window.
Once the watermark for a window has passed,
the default afterwatermark trigger
will emit the results for that window.
Any data that arrives for that window after this point
is considered late data, which we'll also
discuss in the next video.
Let's see what this looks like with our example data.
Since we're trying to illustrate the differences between event
time and processing time, we'll take a look
at when these events actually arrived in Beam.
This is a pretty optimistic example,
because most of our event time and processing times
are very close together.
However, the last event has a bit more delay.
Since we're using 5-second windows,
the data will be split into two windows.
The tap event at 8 seconds was actually
processed at 9 seconds.
But the overall data would still be processed in the same way,
since that event is still part of the same window.
On the other hand, the final swipe event
has an event time of 4 seconds but was processed at 9 seconds.
If we were using the processing time in our windows,
we'd have three events in each window.
But that doesn't reflect when the events actually happened.
We would normally want that new swipe event
in the first window, since that's actually accurate as
to when the event happened.
So we can see that using event time
is the right way to get accurate results.
But Beam still needs a way to know
when it can expect all the data in each window to have arrived.
Since out of order data could arrive at any time,
Beam needs to use a watermark to estimate when the window is
actually complete.
Let's look at just the events that would be
contained by our first window.
Because the process time for the final event
is after the closing time for the window,
whether or not it makes it into that window
depends on the watermark.
The delay in reordering between event arrival
can vary over time.
So the watermark will make progress at different rates.
If the watermark for this window hasn't
passed until after the new event is processed,
then that event will make it into the window.
Once the watermark passes, the window
will be processed with all four events in it.
But if Beam determines that the watermark has passed
before the new event is processed,
Beam would close the window without including the event
and discard it as late data.
This is the default behavior for handling late data.
And just like most of what we talked about in this video,
you can customize how late data is handled.
So to recap what we've talked about in this video,
there's bounded and unbounded data sets,
which are aligned with batch and stream processing.
Beam uses windows to handle data,
dividing up the data into segments
so it's easier to work with.
Fixed windows have distinct starting and ending
times for the data that goes into them.
Using event time gives you accurate results
rather than using processing time.
And Beam uses afterwatermark as the default
trigger to know when there is no more data
to be expected in a window.
We'll talk about handling late data, triggers,
and different kinds of windows in the next video.
So stay tuned and thanks for watching.