stream processing is a term that groups
together the collection integration
and analysis of unbounded data.
First, we're going to talk about the characteristics
of the data that stream processing systems
are built to handle.
So there's two main types of data.
We've got bounded data. Unbounded
data,
bounded data is finite. It
has a discreet beginning and end really.
This is any data that you store in the kind
of a transactional database. Uh
it's associated with batch processing.
An example would be a point of
sale data in a shop whenever
you make a purchase in a shop, that
data will be stored in the database.
And then maybe at the end of the day, end of the
week, even quarterly, a batch
process will run against that database
and generate reports to stake for stakeholders
to review on bounded data is
infinite. It has no discrete beginning
or end.
And this type of data is association
to extreme processing.
An example would be an air quality
sensor, let's say a sensors recording
a data point every second.
So over a year, you know, this sensor
is recording data 24 hours a
day, 365 days
a year. There's no beginning and end.
And so this is the kind of differentiation
between these two.
So let's have a look at the characteristics
of what one of these unbounded data streams
might look like. So your stream
processing software needs to be able
to handle all of these points.
First we've got small data
records so often Jason
XML or even see
SV fragments.
So you've got the small
in size but there can be huge
volumes of them
and when summed up over a large period of time,
the actual data storage volumes can be significant.
Data volumes can also be extremely
high. Um you can get
low volume data streams, but
if the low volume um consistently
then they often handled by complex
event processing tools rather than stream
processing, stream processing, it
are focused on streams
that at some point have a high volume
data distribution can be
inconsistent as well. So you
know, you might be record recording, let's
say the location of vehicles
well during the night, you might have very
few data points coming in, but then
during rush hour you might get a surge of
data.
So your infrastructure in, your software has
to be able to handle these bursts
and these busy periods, but on
the flip side it also needs to be able
to handle the quiet periods, you
don't want to be able paying for software
and hardware. During these quiet periods,
data can also arrive out of sequence
network issues,
um you know, hardware downtime,
it can cause the data to arrive late
in your infrastructure, your
stream processing tooling needs to be able
to handle this data arriving
out of order and deal with it.
So here's some common event streams,
enterprises really being engulfed
by a growing volume of streaming data now
and you can see there's a huge variety of sources,
I think one key thing to note
as well as location is playing
an increasingly larger role, um
GPS chips are now extremely
prevalent. They're much much cheaper.
So again the extreme processing system,
it should be able to handle a
data with a location or element
and it should be able to analyze and process
that data in an intelligent way.
So it's a historical transaction data
is still important. But really this
all of the discussion around big data,
it really consists mainly of streams
of observable events from from
one of these events sources.
This is a no review of the difference between
batch processing and stream processing.
So I pull a few things out here. The
first thing to know is that with stream processing
the jobs must be running continuously.
That process the data
and those jobs that are running continuously. They
must be able to produce constant
results. obviously if it's streams
running 365 days a year,
you can't have a period of time where
your stream processing framework is not listening
because you may lose data. Whereas
we batch processing your running infrequent
jobs that produce results, wants
the job has finished running
the other thing in the last row there
with batch processing your processing
the data after storing the data.
So this enables you to do very complex
analysis because the data stored in a database,
you can pull out specific
parts of that data to do analysis
on them in memory and then save the data
back with stream processing
your processing the data before
it's been stored to a data store.
So you're processing in memory
and then maybe you're storing the data.
So again you're dealing with
larger volumes here but you're
also um it limits
the type of analysis you can do uh
must be simpler because this continually
running job makes it harder to
do very very complex analysis. Often
with analysis you you window
the data so that's how you split the data
for analysis, you window
it based on time and then you can group
the data and do some simple analysis
on it. So these are the type of
analysis we see people doing on stream.
I'm working with streaming data.
We've got three main groups here. The first
group is filtering and aggregating. So this
is all about reducing data volume
as you might filter data out based
on an attribute or you might group
data together. For example you
may have an air quality sensor logging
data every second.
What you could do is build a workflow
that aggregates that data over a minute
takes an average. So then rather
than saving every point
To the database, you're just saving one point
every 60 seconds
enriching the data is another common
workflow.
Obviously often data coming
from a stream is very raw and
basic it might have an I. D. It might have a time
stamp it might have one value
attribute. so you might want to join
that data to an external database
or a P. I. Enrich that
data in memory before storing it to disk.
You've also then got event detection. And
event detection is interesting because
often with event detection you might not
even store the data at
all in a database. You might
just in memory, look for patterns against
that data and once you can identify
a pattern you trigger a downstream
process or notification to
a stakeholder.
So in conclusion stream processing
it's really been a game changer. It allows
organizations to deliver insights
across massive data sets on
a continuous basis.