Exception reporting and compression
testing offer you the opportunity for maximum efficiency
with your PI points. The idea is to only store meaningful data in the
PI archives while discarding all the noise.
Although PI is capable of storing enormous amounts of
data, it is important to store only this meaningful data in order to make your
PI server run the most efficiently and as effectively as it can.
So have you ever
refreshed a trend only to see some data points disappear?
What you are seeing is the effect of exception and compression, but you
don't need to fear because you'll see that the
information being transmitted, the fidelity of the trend
is still preserved with the points that have been kept and stores in the PI archives.
In this video we'll go into what actually is going on here.
So first off, why only store meaningful data?
So the first one is obvious, disk space. You'll take up less
disk space if you only store meaningful data.
Next is network traffic. With properly
set exception and compression settings, there will be less data sent over the
network. Now this happens when the original data is sent
from the PI interface to the PI server. And whenever a client
like PI ProcessBook or PI DataLink
calls that data, we can significantly cut down on the network
traffic load with properly tuned compression and
exception settings. Lastly, performance.
Storing excessive data in the archives is extra work for the
PI Server. And whenever clients call for big periods
of archive data, much of the archive data
will likely to read from the hard disk.
This can be an expensive operation, especially for big chunks of data.
And lots of meaningless data equals slower
client read times. So if you only store
meaningful data in your archives, then the PI server can retrieve
bigger time ranges of data faster, giving you
quicker access and a happy PI server.
So there are two parts of this.
There is exception and compression. We are going to go over the first
test, exception first. Exception reporting takes place on the PI
interface node before the value is ever sent to the PI server.
And the point of exception is to reduce
the communication burden between the PI server
and the interface noise by filtering out what is obvious
noise. So here I have a thermal couple
monitoring temperature. And this thermal
couple has a certain instrument precision
where anything that falls within that instrument precision
is really just noise. Now I have an initial value right here
and then I have other values coming in periodically every
few seconds. Now we can see that most of these values
are within my instrument precision. B hasn't changed very much
from A. And it is not meaningful.
The thing is, is it within the instrument
precision and it doesn't mean anything significant and we shouldn't
base decisions on this value. So what is great about the PI
[inaudible] testing is it allows us to
define a dead-band to reflect that instrument
precision. And what is more, you can individually tune all of your
PI tags to have their own exception settings.
So the PI tag attributes that define
exception are ExMax which defines the
length of time we'll go before we for sure
report a value. And ExDev
which defines the width of this dead-band.
Now let's have a quick look at how this
works. We have a starting value right here. And we
call this the Snapshot value -- this is the
current value as it is sent to the PI
server. It defines the current state of our
process. Now as we
see as we go on, raw values are falling within our dead-band.
And we see one has fallen out, and this is an exception and we are
going to keep it. However, if we just stored the
Snapshot value and the new exception value,
we'd miss what was actually on with the data. So we end up
sending the previous value as well -- the exception and the one right before it.
Now this is easiest to see if you watch it in
action. So here we have some raw values that were scanned
in at data source. Now we are going to see what happens as
exception is applied to them.
So here are the raw values that have passed the
exception algorithm and have arrived at the Snapshot table of the PI server.
You can see in this scenario, not that
too many passed exceptions. But in your scenarios, quite a few
might. It all depends on the process
and the exception settings. The next test is compression. This is a more
thorough test that defines which data is stored in the
PI archives. Unlike
exception, compression can have a slope. This is
why we sometimes call it the swinging door algorithm
because it looks like a door viewed from above
with a width and a length. And it can swing to
match the slope of the data. Now unless you are using the
PI buffer system to buffer data, compression testing is
taking place on the PI Server by the PI
Snapshot subsystem before data is actually sent
and written to the archive. If you are using the PI buffer
subsystem, then the compression is performed on the interface node before it is
sent to the PI server. This is important in case of a
collective so that both collective members have the same data.
So the idea here again is to
store only meaningful data without losing any meaningful fidelity
of the information. Since every sensor and PI tag is
different, it's important that you are comfortable with the
mechanics of compressions so you can confidently
set your settings -- the compression deviation and compression
maximum -- to match what is appropriate for you.
So the compression algorithm is based upon three
values. There is the most recently archived value, the
current snapshot value, and the incoming value.
Let's have a look.
So in the beginning there is only the most recently
archived value and the current or snapshot value right
here. Remember the snapshot value is the most
up to date value that has passed exception. It is the current
instantaneous state of the process -- thus we call it the snapshot.
However as time goes on it is entirely possible
that this snapshot won't be needed in the PI archive.
So the first step is to calculate the slopes. We have an
archive value, we have a snapshot value and we have the compression deviation settings.
And we'll calculate the maximum slope and the
minimum slope. A reference slope is
calculated using the incoming event
and compared to the maximum slope and the minimum slope. Then we have a decision to make.
Will the existing snapshot
value be written to the PI archives? If the
incoming event falls within this angle like it does here,
the incoming event becomes the new snapshot value and we can
discard the old snapshot
value. Then we recalculate the
minimum and maximum slopes. Notice
now that our angle has narrowed a bit.
Now again, an incoming event comes in
and we compare it to our maximum and our minimum
slopes. However now there is one additional rule
applied. The slope is calculated as the new slope and the
minimum slope, needs to always be narrowing.
So we'll see this one, despite the compression deviation reaching
down to this SMin,
that exceeds what our old SMin is.
And one of the rules about compression is that these angles but
always be narrowing. So when our next snapshot value comes
in, we see that it is outside of our acceptable
angle here. This means that we are going to
keep this value and this one will become the new snapshot
value. So we
defined a new archive value that will be written to the PI
archives and we start the whole process over again. We can also
see that between any two archive values, the compression
deviation defines the parallelogram
that holds any data that happened between those two archive
values. So again, this is
easiest to see in action. So watch this,
we are taking the raw values from our exception test before and
applying compression to them.
So now we can see the
raw values scanned at the source.
What passed exception, what passed
compression? The trend lines that PI ProcessBook or PI
CoreSight will draw
and we can see that all the values from the raw
source still fall within the compression deviation that we setup.
So exception
settings should generally be slightly less than the
instrument precision. Or you can set it to be about half
of the compression deviation. For more details you can consult this
KB article on our website -- 3226OSI8
Compression settings --
we recommend this based on a ratio of the
snapshot events to the archive events. Ideally this is going to be a
ratio of 10:1 or 100:1 and at
minimum it should be a ratio of 10:1 or 3:1.
Anything less than that is going to be pretty hard on
the PI server. To check this, you can open up
system management tools and go to snapshot and archive statistics and look at
snapshot events and archive events
to get your ratio.
When is it appropriate to turn
compression off? OSISoft generally recommends
not ever turning off compression but it is appropriate
for a few scenarios. For laboratory values or
manually entered values or totalized points where every value is
significant, it may be appropriate to turn compression off.
Also there may be government requirements regarding data collection
that require disabling compression. If you set compression
to off, then all exceptions are archived.
Now a suggested better option to turning
compression to completely off is to
set compression to on but to set the compression deviation
to zero. This means that successive
identical values will not be archived.
For example a value of OFF that is sent to the
archive and that is followed by another OFF is not added.
This is much more efficient.
So that is exception and compression.
You start with raw values then you apply exception, then you
apply compression. The trend that comes up in ProcessBook
or PI CoreSight fully represents the fidelity of the data within
the instrument precision. For more information
check out our tech support website at techsupport.osisoft.com