Section 3.0, Data
Flow. The point of this Section is
to teach how data flows through the system. We want to
make sure everybody understands how that data flow works
because it does have implications as to how you configure your
PI Tags. Also, we'd like you to be able
to tune the amount of data collected by a tag using
some specifications. They are called the compression
specifications or exception and compression taken together.
And these together determine how much data we end up storing
in the system. So that's the objective.
Let's first go over the New Event Processing. New Event Processing
works something like this, and let me highlight this.
If you could imagine the very first or the -- say
you just turned on the system, and, and the components
have just been, just been fired up,
and the interface starts up. It starts sending data
to the PI Server. Before it
sends data to the PI Server, that Interface is going to do something
called the Exception Test. And, the result of that is the
Exception Report. The Exception Test simply checks to
see if the values are changing enough from the
previous value to be worth sending on to the PI System.
Now that all happens here on the left side
here which indicates it's off
on an Interface somewhere. Everything on the right side
of this illustration is going to happen on the PI Server.
So the Interface does the Exception
Test. It passes the data in this direction
to the PI System. Now the syst...the
data that makes it through that test goes into something we call
the Snapshot. The Snapshot Subsystem manages
the Snapshot memory structure. It's basically a
very, a, a, -- the most recent value
for each tag in the system resides
in the Snapshot. Now that data does not necessarily go
straight to our Archives yet. That's not
necessarily going to get down here.
The reason it does not get down there is because there's another step.
See, we would like to do more data compression on
that, on that data that's been sent in.
We want to use a data compression algorithm
doing something we call the Compression Test to
reduce the amount of data further without really
losing much accuracy. And I will explain how we do that
in just a few minutes. So we do the Compression Test.
The results of the Compression Test are sent to an Event
Queue. That Event Queue is a queue that sits on
the PI Server. It simply fills and empties,
generally fills and empties because as soon as we put data
into the Event Queue we take it out of the Queue,
throw it into the Archive Cache where it is then stored
on disk. So, why do we do this? Well, we do
this because one day you may find that this
Archive is not available. Either you have had a disk failure
or somebody forgot to restart the
Archives after a manual backup. But, for whatever reason,
that Archive is not available. That's why this
Event Queue exists. The Event Queue is going to fill up
and after it fills up entirely,
then will overflow and start overflowing to some files.
We have files that we maintain,
pieventqueue.dat, that will
fill with the data that would ordinarily go
to the, the Archive itself.
Now, once this situation resolves itself
over here, once this is no longer offline,
once this is back online, at that point
the data comes out of the Event Queue without
any interaction from the System Manger. So, it's basically
automatically. We take the data out of the Event Queue,
put it into the Archive Cache, and then from the Cache
we send it directly into the PI Serve. Now you
may notice this Cache is representing two directions here.
Not only is it placed where we store things on its way to
the Server, but it's also where we store
things that are being pulled off of the Archive,
so that users can look at the data.
So, it's a smart kind of a Cache. We replace
data in the Cache on a least recently requested
basis. In other words, if somebody has not requested data in a long
time, we simply remove it from the
Archive Cache in favor of this data here coming in this
direction that's been more recently request. Now the point
of all that is that -- well, first of all, we do have a
place to store this temporarily until we can write it to disk,
but secondly, and more importantly, as people
work their, or with with their clients, they are going to be
bringing data into the Cache that's of interest to them.
And if it's of interest to them, using good caching,
you know, kind of a, a, algorithms,
we can be pretty certain it's going to be interesting to other people
as well. So, since we are trying to keep
well-populated this Cache
with interesting information that's being requested on a regular basis,
then, whenever you go to retrieve data from the Archive,
chances are that data may already be sitting in a
Cache that's sitting in memory. So, and, of course, a memory
call is a lot faster than getting this data off of
the physical hard drives. So, it's a smart Cache
for retrieving the data or for holding the data
that's both on its way to the system and also data that's
come out of the system and
being, you know, queued up for people to look at,
to, you know, to answer the queries of the PI
clients like ProcessBook and RtWebParts. So,
taken together, this is the overall Event Processing
that gets data ultimately onto these PI Archives.
Section 3.1
Exception Filtering. Now, the point of discussing
this data flow and the Exception Test is that
what we, or how we do this is going to impact how
you create tags. And, again, maybe this is belaboring the
obvious, but if we go out to our Point Database -- this is one of
our Points in the Points Database. If you take a look at the
Classic Section -- I am sorry, that the wrong place --
if you look, take a look at the Archive Section, you will find
there is, there are these two fields, Exception
Deviation and Compression Deviation. And, the, currently
they are shown in Engineering Units, but they also could be shown in --
as part of the percentage of the Span.
In any event, these are settings that each tag in the
PI System has, and it's these that
determine the deadbands that we use when we are
deciding whether to bring the data into the system and
then whether to store the data on the, into the Archives.
So, this has some real-world application. We
want to make sure you are storing enough data without storing too much.
And, if you are thinking, hey, this is silly.
Disk space is cheap. Why not just store it all?
It's true, disk space is cheap. The reason we do this is not so much
for disk space so much as for the speed of
retrieval. With good data compression, your,
your, your trends will retrieve tens,
if not hundreds of times faster than if you simply
dump every single second of information into
your Archives. We do have customers who choose to do that.
That's fine. All of this can be turned on and or off as you choose.
So, let's take a look at this data flow with regard to
the Exception Test. As I mentioned earlier, on the
Interface side, this is now -- we are talking
about an Interface Node, either an API
Node, a PINet Node, UNIX-based, Windows-based,
but basically on one of these Data Collection Nodes.
We are going to - the Interface is going to
scan data from the Data
Source and then the Interface, itself, is going to
do the Exception Report. Now the Exception Report,
or Exception Test, simply
checks to see if the value is worth sending on to the PI
Server. If it is, we send it to the Server.
Here's the dividing line between the Server and the Interface.
We send data to the Snapshot
where, at this point, it goes through a compression algorithm
on the Snapshot si... -- or on the Server side, and
from there it's stored in the PI Archives. So,
we check again to see how we are going to best
store it, because sometimes we can, depending on how
the data looks, we can store just the peaks and valleys and
without losing any data resolution
to speak of, we still have, we
saved a great deal of space on the computer.
And also retrieved these things a lot faster as a result.
Now, when you look at this, you are saying, okay, well, what,
what is this I have heard lately about the Compression
Test being done on the Interface side? That is true. There's
an option that you can with, with the newer version of
buffering, you can set this up so that the Compression
Test is actually done over here on the,
on the side of the Interface
by a buffering subsystem. And,
with, with that in place, what it simply means is that we
flag those values that will have passed
the Compression Test if it had been done over here. That would
have passed the Compression Test if it had been done here.
Now we still send all the data
to the PI Server. It's simply that some of it's been marked
as data that will pass the Compression Test. And,
again, we do this so that -- are actually doing this
in pursuit of high availability and true data fanning
where the exception,
results of the Exception Test are sent to
multiple PI Servers. But, suffice
it to say, really from the effective point, it does not
quite matter where this is done, but there are two tests:
a test to determine whether it's worth sending on to PI;
and then another test to determine how we are going to store
it in the Archive. And, in just a few minutes, we will see them,
the mechanism that we use for doing that.