[MUSIC PLAYING]
DEBI CABRERA: Sorry I'm late.
Thanks for letting me in.
Good thing we're allowing late data into our windows,
speaking of which, let's pick up right where we left off,
dealing with windows and late data in Apache Beam.
In the last video, we looked at some sample data
and how Beam fits that data into windows.
With our example data, we set up five-second fixed windows
and talked about how some events might not
make it into the window if the watermark has already passed.
Data is considered late if it comes in after Beam has already
processed the window.
So what exactly does that mean for your data?
The default behavior is to drop late data, which means it's not
used in any data processing.
Sometimes this might be what you want, depending
on your business logic.
However, if you want your pipeline
to process data that arrives after the watermark passes
the end of the window, you can apply an allowed lateness
when you set up your windows.
This option gives your windows the chance
to react to late data.
If allowed lateness is set, the default trigger
will process the new data and send out the results
immediately.
So if we set the allowed lateness to 10 seconds,
the final event from our example will still
be processed because it's less than 10 seconds late.
The result of this late processing
will be sent to the next step in the pipeline.
Beam uses watermarks to provide an estimate
for knowing when no more data is expected for a window.
And allowed lateness gives you the ability
to allow late data into the windows.
When we look at our example data and the windows
after giving 10 seconds for allowed lateness, well,
that's not quite what we were expecting, is it?
It looks like there's two windows with the same start
and end.
That's because they're actually the same window
but two different window panes, which
is what we call it when a single window aggregation is output
as two separate elements for downstream processing.
Depending on how you want to handle data,
elements in a window might need to be processed several times.
The actual results from processing a window
are called window panes.
And they fall into three categories--
early, on time, and late.
Up until now, we've been dealing with on-time panes, which
are the unique output that contains all the received
data when Beam considers the window complete
based on the watermark.
But this new window pane is actually late
because it has some additional data that's
been allowed into the window, thanks to allowed lateness.
When a window has multiple outputs,
Beam can handle passing on the results in two ways.
These two window panes, one on time and one late,
have the same window properties, like start and end time,
but not the same data.
This happens when the data is discarded
and Beam passes along the new process
data as a separate window pane.
If we don't want the data to be discarded,
we can set the windows to accumulate data
rather than discard it.
Allowing accumulating data gives us
this, where the late window pane now
includes all of the data from the first pane as well.
Instead of just passing along the new data,
beam will pass along the entire aggregation
as a replacement for the previous window pane.
Ultimately, it's your business logic
that will dictate which of these is right for your pipeline.
If you want to include late data, deal with late data
differently, or discard it entirely,
that's all possible through how you set up the windows in Beam.
With our example, the first window having two panes
means each of those panes would be sent down
the pipeline for whatever other processing you have set up.
One benefit to these multiple panes
is that you can actually process results
before the late data comes in and then process them again
with the late data.
We've mentioned a few times what Beam does by default,
such as the after watermark trigger.
But one of the properties of the Beam model
which makes it so powerful is how you can
configure all of the behaviors.
Beam uses these triggers to determine
when to send off the data from a window through a window pane.
And there's lots of options for how to configure them.
Again, the watermark is Beam's indicator
that there is no more expected data for that window.
The trigger for a window is the actual mechanism
that sends off a window pane.
After watermark is the default trigger that's actually
responsible for this, which will only
happen once per window after the watermark has been cleared.
That trigger will happen no sooner
than the watermark, which will then send off the window
pane for further processing.
Depending on if you're allowing late data,
events after the watermark might go into the same window
and eventually become a late window pane.
But after watermark isn't the only trigger you can use.
Instead of our small example data,
what if we were collecting millions
of events over a large window, like 24 hours?
Having the final data be processed
after waiting the full 24 hours is important for completeness.
But that might increase how long it takes to make decisions
based on that data.
Thankfully, you can use data-driven triggers
and processing time triggers to get early results.
Data-driven triggers can take action
based on things like the count of events in the window.
So even if the window is 24 hours,
you could trigger an early window pane after the first 20
events are processed.
In the same way, you can use a processing time
trigger to get an early window pane based on the processing
time.
So if you want to see some early results after only 12 hours
of processing, you can use a processing time
trigger to get an early pane.
But that's not all.
Triggers can be composed together
for more flexibility on when you want a window pane.
By using AfterAny, you can group triggers together
so only one of the subtriggers needs
to be met in order to tell Beam that the window pane is ready.
Another common case is to use repeatedly
to have the trigger continue to process the window
as new data arrives.
If we take a look at this code, this trigger
will happen every time 20 events are
processed or after two hours.
It's important to note that these triggers can
be customized, but they only set the minimum requirements
for sending out a window pane.
Especially since Beam is built to do distributed
data processing, there might be a bit of extra time
or a few more events than the minimum.
Keep that in mind as you experiment with triggers
and customize to fit your needs.
So far, everything we've talked about
is based on fixed windows that have a start time and end time
and don't overlap.
Even if you don't declare a window,
Beam still uses the global window,
which is great for batch jobs but not so useful
when dealing with unbounded data.
If you want to use windows that overlap,
you can use sliding windows.
A good way to describe sliding windows
is to think about doing a moving average.
If you wanted to get an average of the number of events
in a 10-minute period but wanted that average
to update every minute, sliding windows would be a good fit.
Since these windows overlap, any events
in that overlapping time period will
be processed in both windows.
It's a good idea to understand how intensive the event
processing is and potentially perform
any processing before windowing to avoid heavy resource usage.
There's one more kind of window, session windows.
These windows are a bit different.
Instead of start and stop times, you specify a gap time.
This gap time is how long a window
should wait without seeing any activity before closing.
Using the same example of mobile device data from before,
we might use a session window with a gap time of five minutes
to automatically create windows based
on when the user is active.
If the user doesn't have any events for five minutes,
the window will close and continue processing.
So, to recap, windows subdivide data.
And Beam uses triggers to send off
window panes, which could be early, on time, or late.
Allowed lateness can be used to allow late data into a window
past the watermark.
Windows can be fixed, sliding, or session.
And all of these give you the ability
to really customize your processing pipeline.
Of course, this is a lot of information,
and we are just scratching the surface.
There's a link in the description
below to a detailed notebook where
you can try out the code for yourself and get hands-on.
I definitely recommend you check it out
if you want to learn more.
Thanks for watching.