It's time to talk about PI Buffering, and
this is a technology that's been a success from day one. The fact that we
have a mechanism to buffer data
if there's any kind of a disconnect between the interface and the PI
Server ends up being very successful. By buffering
what we mean is if the interface is collecting data
and sending data, you know, as it normally does, in this direction
over to the PI Server, if something happens to this PIServer --
let's say it crashes or the network crashes or
something a little less severe -- I do not know, say this, this Server
here's been brought down so that you could update the
Operating System or upgrade PI. Well, if buffering's
not enabled then, you are simply going to lose the data. You know, the data's
moving in this direction, has no place to go, it just, you know,
falls off the face of the earth into the proverbial bit
bucket. If buffering is turned on, what you
have instead is you have got this little application,
the Buffering Subsystem or the buffserve, you know.
I will explain the differences in just a second. But, you have got a little
application here. That application acts as a kind of a
go-between between this Interface and
the PI Server. So, when the Interface sends data
it no longer sends data to the PI Server.
The Buffering Subsystem kind of intercepts it.
In fact, what, what we normally see is the Buffering Subsystem
has to be started before the Interface so that it can do
this kind of impersonation. See, the Interface does not really even
know that this is happening. All the Interface does is it
starts to try to send data, but if the Buffering Subsystem
is there first, then it's going to interpret that --
or intercept that data and just kind of pass
that on then to this PI Server
over here. Now, the whole thing seems kind of pointless until
you get to the point at which, again, you crash this or
this network goes down, or you bring this down for updates.
If you do any of those things, then at that point,
the data over here which, you know, normally would
be going into the bit bucket, that will just sit there and buffer
on this server. And, I will explain what the buffer
looks like, exactly, you know, how we, how we do it. We go into
Memory, then into File, etc., etc. Anyway, to make
a long story short, this, basically, is going to save
a lot of data in the event
that, that there is some kind of a problem over here. Now, when we
first introduced this years and years ago, it was immediately a success.
You know, not only because it saves data, but it also
gives the system manager more options when, when
it's not an emergency. Again, let's just say the system manager
needs to bring this server down to do an upgrade.
Before buffering was around, you actually ended up with
a lot of PI System managers they'd have to come in nights or weekends
to do this kind of work because, you know, nobody wanted the
data to be unavailable during the workday when engineers were trying to
do analysis and things like that. So, it ends up
being very, very successful way of, you know, ensuring
that you have got some kind of redundancy built into the,
into the system just by its very design. So the buffering
does that. Now, technically, what it is, is it is a
First In, First Out queue, a FIFO queue.
It's not a stack. We will go into some of the details of that in just a
bit. The buffering that
you implement is going to be something that will work with
all of the interfaces that are on this machine. So, let's say you have
got, you know, one, two three different
interfaces that are sitting over here. Each interface
is going to be sending data to the PI Server.
But, as I said, this, this buffering application here
kind of intercepts all that. So it's, it's truly a go-between
between the PI Server and any number of interfaces that
are running on that, on this machine over here.
Now, this is implemented as a service. This is where I switch
over and take a look at my Interface Node here.
And I take a look now
at this Services Applet. You will see
we have this PI Buffer Subsystem. We also
have something called PI Buffer Server. There's actually two different mechanisms
for doing a buffering, and I will explain both of them --
why, why you choose one vs the other, in just a bit. And, we
use the ICU to configure this buffering.
So, if I go into Tools, Buffering, this is where we get to
choose which of the two buffer types we use,
and some of the settings for that. Now,
when you set up buffering, you are going to want to make sure
that the interfaces that are running on this node
have used, or have been specified to have
a dependency on that Buffering Subsystem.
So, if we take a look over here at my OPC Interface,
or excuse me -- this is my OPC Interface right here.
If we take a look at the Service Section here,
you are going to want to add this PI Buffering Subsystem
as a dependency here.
Actually, this is the Buffer Server.
That's the service that uses the API.
The PI Buffering Subsystem is this right
here. This is the, the second of the two ways of doing it.
So, one or -- depending on which one you are using, you want to make sure
that this interface is dependent on that. And,
in that, in that way you can make sure that this is
going to be started up in the right order. The buffering first and then the interface.
So, here's a chart that shows the two.
The first is the API Buffer Server. That's the older of the
technologies. Second is this PI Buffer Subsystem.
And, from the point of view of, you know,
advantages, one vs the other, and why you choose them,
as you can see the PI Buffer Subsystem, it does
require a more recent version of the PI Server. So, if you are running an older
version of PI, that's really it's your only choice to use this
older version. Now, the PI Buffer
Subsystem does the compression test actually on
the Interface Node which has some advantages I will
describe a little bit later on. The
API Buffer Server has a 2 GB limit
as to the size of the buffer, and this Buffer Subsystem does not
have that limit. And, as you can see here, the,
the PI Buffer Subsystem has a, a much bigger
throughput channel there. Let's take a look at
where you make this decision. It's under Tools,
and under Tools, this Buffering Option.
So, as you can see, you have the, the choice here of this PI
Buffer Subsystem or this API Buffer Server
right here. Let's start with the more recent one, the PI
Buffer Subsystem. This is the one we recommend. It's
better for a number of reasons. The, the throughput as you saw was much much
faster, but the in... interesting thing about
this is, where you have multiple PI Servers, it ends
up doing some things that are, that are much better for that particular
situation. So, let's pretend that we have got a situation here
where, you know, we have got buffering in place, but
in addition to this first Server, we have got a second PI Server
over here. This particular PI Server
is going to be the recipient of a copy of
the data. We would call this N-Way Buffering. So the
buffering's going to go in this direction, but then it's also going to go in this
direction, as well. Well, when you are doing N-Way Buffering
like that, what you'd like to do is make sure that
there's exactly the same data
on both PI Servers. And, if this is a PI Server
in a collective -- a collective is, is implemented as part of
our PI High Availability --
if you are doing this as a collective you, you know, that's
the whole goal of the, of High Availability. You want completely
the same information on both Servers. If you are doing
the data compression over here -- data
compression over here means what you end up sending
is going to be exactly the same data to both
servers. Now, if you use, instead of the
PI Buffer Subsystem, you use the API Buffer Server,
that actually does not do the data compression over here.
What that does is it sends the raw
data, uncompressed data, it sends that to
both Servers that are -- or, or, the, you know, the end
number of Servers that are, you are buffering to.
And, as you may be familiar with --
you know, when, when data compression occurs, data compression really
has to do with the order in
which you get certain data. So, it's, it's actually possible,
you know, if you are, if you are bringing in data
raw, the same raw data sent to two different PI Servers
that have, you know, slightly different Current Values
sitting in them, and slightly different most recent
Archive Values, you could actual, actually end up with slightly different
date, data. It's not going to vary by much but it's going to vary
enough that we would strong suggest, instead, you use
the PI Buffer Subsystem. Now, another reason
we recommend this is because, when you are using a
collective, if you were to add a third PI Server to that
collective -- you know, for my last example -- well,
this PI Buffer Subsystem will automatically be informed
of that. That's something you do not have to configure manually. If
you use this API Buffer Server to do
N-Way Buffering, then that's something you'd have to configure manually
yourself. In, in our next video, we are going to go into the details
of how you actually configure this. So, stay tuned for that.
What I would like to now is just go into some of the details of how
the Buffering Subsystem works. This is the
Buffering Subsystem, the more modern of the two. We will get into
how this actually stores the data. So, we have got a Data
Source over here. It could be a Distributor Control System or a
SCADA System or a paper machine or whatever
source of your real-time data. That data's coming into
the Interface Node. Our Interface is running on here.
And if you, you know we send the
Value to the PI Server, and the PI Buffering
Subsystem is running, then the Buffering Subsystem
is going to, as I said, intercept that data.
It's going to do the Compression Test
on the Interface Node, and then
send that data along to the PI Server. Now it will
send it, temporarily, into a little queue in Memory,
and, if everything's good over on this end, it will just continue to
send that. So, using this illustration here --
See, this is what normal data flow
would look like. We have gotten these Values that come into the
system. We do a Compression Test.
Once we do the, or when we do the Compression Test,
at that point, you know, we try to flag
which Values are going to not
make it into the Archive, that are going to be --
basically, fail the Compression Test. So, those two there with the *s
those are going to be sent over to the Pi Server as
Snapshot Values, but they are not going to make it into the Archive.
These over here, these Values will make it into the Archive
'cause they have passed the Compression Test. But anyway, we have not
gotten to that point yet. We are still sitting on the Interface Node.
So, the Interface Node normally,
through buffering, is simply going
to put that here. And then once it's,
you know, once it's in that queue, just as quickly as it gets in there,
we are going to send that data over to the PI Server.
Once it gets to the PI Server, it goes into the Snapshot.
From there into something called the Event Queue, and from the Event Queue, we
send it to the Archive Cache. From the Cache, we write it to disk
quickly, and then we also keep a copy in the cache
just in case somebody needs to look at it. Anyway,
that's normally what will happens with data collection
on and with buffering turned on and everything working
fine. Now, if the Pi Server's not available
for some reason -- let's say the network goes down or
the Server's being brought down for upgrade --
when that happens then, you know, these Values over here,
that we are not going to be able to send those. So, what we are going to do is we are just
going to start queuing data up
into this first queue
here. Now that,
that queue is going to be read
in the order in which we brought the data in when
we re-establish the connection. So, as you just saw with that
animation, you know, once the connection was re-established,
we do take that data out, in the order it came in, so it's
a First In, First Out, you know, queue.
We take that out. We bring that into the Archive and,
you know, everything's fine. We, we have not lost any data.
Now, if we, if we have
the same situation occur again --
but in this case it does not resolve itself so quickly,
if this buffer fills, this first buffer,
then we are going to overflow to a second buffer.
There you go. As you can see, this Buffer is actually called
pibufq_ and then
the name of your server, and we append some
numbers at the end of it just indicating which file it is.
Now, if the first one fills, then what we are going to do
is we are going to start a second one.
As you can see that first one filled. We will go ahead and start
the second one. At this point in time,
communication's been re-established. So, when the communication's been
re-established, this is a FIFO queue, so
we will go ahead and start retrieving data from, you know, the first
data that was brought in. So, at Value #7,
8, 9, 10, etc. -- they are going to be brought in.
Now, remember, as we are bring this in, we are still adding
more Values to the end of this Queue. Yeah, we are doing
it -- a combination of what we keep in memory
and what we put on these, these files here.
Now, eventually, eventually we will catch all
of that up.
Yeah, there we go. We are seeing this queue and we have,
we have eliminated the first queue. We are now
retrieving data from this queue that does still have data. And, eventually,
that's going to catch up. And then, finally, we will,
you know, of course, in time, we will take the data out of this memory queue
over here. Now, I did not really explain this well enough
originally. This part of it is going to be the
queue that sits in Memory. We basically, we will keep this queue in
Memory and swap the data out into a
file as that queue in Memory -- that queue that's in
Memory fills up. And then we do the behavior you just
saw when it comes to filling up
and emptying those file queues. Now,
even though we do not have any kind of Events happening
right now that are problematic here, you will
notice that if you go into the Program Files, PIPC
dat Directory, you can actually see this
pibufq. That's the file that will fill up
originally if the memory queue fills. That's the one that we are
going to go ahead, you know, and fill first. If that one
fills before we have had a chance to empty it, we will go ahead and
create another one, append some numbers at the end of it, etc.
And it's that, you know, the, the name pibufq.
This is the name of the Server right here.
Osisoft-trng -- that's my PI Server Name -- .dat.
Now with the other
older API Buffer Server, we
have a, a similar file. It's just got a different name.
So it's called APIBUF_ and then the name of the
PI Server, etc.