Hi everyone today, we're going to look at the frameworks
available to you for processing your real time
data. Real time data is data
that is generated and delivered for
immediate processing from sources such
as sensors, mobile devices and
applications. Before we look in
more depth, the processing frameworks
available to you.
What we're going to do is take a more in depth
look at real time data. I want you
to be able to look at your real time data
and understand it and characteristic
because that will help you choose the right framework
for processing it.
So there's two main types of real time
data. There's event data and stream data.
Event data. This is self
contained messages. Typically low volume.
These are generated when a well defined
condition arises. So this could be a
one off event like,
you know, a river flood triggering
event into a system, a
vehicle incident where
a stream data, on the other hand, this is when
your data source is it's
a continuous flow of data is unbounded,
which means it has no discrete beginning or end.
So this could be a sense of recording temperature,
a sense of recording position, something
like that.
Now. Typically these data records are small
in size but the complexity
arises in designing your architecture
to handle the data volumes as these volumes
can be extremely high. So here's
a comparison between event data and
stream data. Let's have a look at an example.
So event data, let's have a look at the
river flood example. So
if you have a river you may have a sense
of high up on the riverbank
and that sensor detects maybe
it's kind of a buoyancy, has buoyancy
and it detects when the river reaches
that level.
When the river reaches that level, a
message is sent into the infrastructure
to warn that the waters reached a dangerous
level. Now that trigger that
event, it only happens once compare
this to stream data with stream data,
you may have a sense actually
located in the river
that's giving real time information on
flow. Um you know the depth
of the river, things like that. And these
centers are sending data back continuously
uh into the infrastructure for processing.
So you can see very, very different
volumes of data. One of them is
is very very infrequent and
low volume
and the other stream data is
continuous. You need to look at your
real time data and understand which of these
two categories it falls into.
So next let's take a more
in depth look at the real time processing
frameworks available to you to
handle these two types of data. So
batch processing or batch processing
can handle event data or
streaming data. Um basically
for either of these two you would store the
data to a database and then you'd
run some matching tools against that data.
Now this can be complex for stream
data. Just because of the huge data
volumes involved, you might have
to use, you know like a big cloud database
to store of that data but
it is possible um you're processing
the data after its start and it can be
very high late and see you know minutes,
two hours event processing.
Now this type of processing framework
is obviously related to event data
streams. So a job runs immediately
in response to event, it's his medium
latency. So you know the
job isn't running all the time listening
so there is some latency with the
processing engine coming up online to
pull that job.
Um and with this you
can do very complex analysis so
you basically have all of the
analysis tools available to you that you have
in batch processing
um but yeah
you're reacting to the event and then
you may be storing the data um
but this this this type
of framework is very powerful but
it's really suited to kind of um
low volume messages. So
um you know back to the example
before if a message comes in
because there's a flood event in the
river um the infrastructure
would consume that message processes
would start and they would run and then things
downstream would happen. Stream processing.
Um This is continuously running
jobs that produce constant results
and so here rather than a job
starting and stopping, you have
engines running continuously
listening for the data so there's
no latent seen starting a process
which means you can get very very low latency
down to milliseconds.
Uh Typically with stream
processing you're connecting to message
queues. Event streams, uh
awesome transactions in another application
you could connect directly to hardware
as well, potentially like to the device,
compare this to event processing.
This is more you see
also message queues for event processing
the lower volumes but event
processing also supports static static
files. Things like emails arrive
in files being uploaded.
Web hooks been updated, things like that.
Stream processing is better suited to
connecting to endpoints that continually
producing events. So
things like Kafka or even lower level
things like TCP I. P. With stream
processing, you can only do simple
analysis.
The key thing here is in order to be able to handle
these huge data volumes you're
doing in memory processing before the
data is stored in your infrastructure
because of this. In memory processing.
Um you have a kind of a
less tools available to you for
doing analysis but you can still do aggregation
filtering, enrichment, geo
enrichment with fMI proximity
analysis and event detection as well.
So these are kind of the three real time processing
frameworks you need to figure out
based upon your requirements and your data characteristics,
which one is the best fit for you
with that for me we obviously support batch
processing.
Event processing is supported through
ftp server automation
and stream processing is through
FB service streams.
So batch processing in
conclusion is near real time, often
minutes but more likely hours. The
data is stored and then processed
event processing. Each
discrete event is handled separately,
with connections between events being
stored in persistent storage. So
this is good if you have real time low
velocity
and then stream processing, there is if
you have much higher velocity
and data is processed in this context,
in memory before storing.