Section 13.0--
Troubleshooting the PI System.
Well, when troubleshooting the PI System, or really troubleshooting
anything for that matter, the first thing to do is to try to isolate
the problem. Is it a client problem? Is there a problem
with the network? Have, do other things work on the network?
Are there updates that are not working on the interface?
Or is it, or is it instead on the PI Server?
Now a good way to distinguish between the two, between an interface
and a PI Server problem, is to look at the value of the
tag, 'sinusoid.' If you have not,
or if you chose to install those default tags,
like 'sinusoid,' then if you have
historical values and current values for
'sinusoid' the problem is probably gonna be with
your interface. See, 'sinusoid," if you're
having problems, and 'sinusoid' is not available,
then that would be an indication of a general, kind of an
archive failure or a network failure. But, if it's just
your, certain interface tags, and this
is working fine, then you've isolated this to
be associated with that interface. Now,
also, take at look at the Server and the Client
Log Files. I've mentioned this before, but
probably the single best place to go for
our, or for
troubleshooting the Client Applications,
is to go into Program Files,
and within there, PIPC, and then
in the dat Director. So, within the dat you'll find
pipc.log. So, just about any
kind of a client problem is gonna be reflected there.
And then, likewise, if you go into the PI
System,
System Management Tools,
look in Operations and look at the Message Log. So,
we store just about everything that goes on in the PI System,
we store just about everything there. You know, even
things that, well, you not might, might not
imagine. So, there's a, a huge amount of data that you'd find
within that Message Log.
And then the other thing to check is, you know, check your
Performance Monitor tags, your I-O Rate tags.
What do they look like? Are you actually,
you know, is your I-O actually consistent with what you'd
normally expect? Take a look at those things.
Here's some questions we suggest
you ask yourself when you are in a troubleshooting session.
First, check all the things, the components, on the server.
So, you know, take a look at some of these things. We're not gonna go through
all them, now. But, you know, are the services running?
Have you looked at the Windows Event Log,
etc.? If it's an interface problem,
take a look at the section here. And, if it's a client
problem, take a look here. A lot of these things
are very common sense troubleshooting. Like for example, can you
check the suspected bad component against,
you know, good, other good components?
Now, another resource you should check out is the --
this is in the PI Server System Management Guide --
it's a troubleshooting checklist. Now,
this checklist started out as a,
kind of a checklist for our tech support folks,
just to kind of -- those who were new -- just to try to get a
handle on some of the problems. And I've noticed, if I,
if I'm kind of stumped by a problem
and I just go through this checklist one at a time, about
half the time it seems like the problem jumps right out at me. So there's
a lot of good advice packed in this checklist. For example,
testing out API Snap from a Remote Node,
and then also from the Home Node. See, these are
two different ways of checking to see if the
communications are working. So, for example, if you're using
piconfig and that works, but apisnap does not work,
it's an indication that you've got some kind of an API
or network problem. So, give that
tasklist or that checklist a, a try.
Another thing to do is to simply
look at the Task Manager.
So, for example, let me go over to the PI Server,
and within the PI Server,
you can choose,
choose the Task Manager and just look at the CPU
usage. Look at the amount of CPU
that you're getting for each of your
processes, and check this on a daily
basis. And
a lot of system management is simply checking the same
utilities over and over again to see which ones
seem to be acting up.
And, of course, if you have created any
Performance Tags, this would be a good time to go in
and look at those Performance Points, the I-O Rate Tags,
etc.
An excellent
utility that's very simple to use is pisnap. You
can run this on the PI Home Node
Now, if I run, you know, on this, this PI Server, if I
run pisnap, that's simply doing what we call
an apisnap to localhost.
And, from there, we can type in the name of a PI tag
and we can take a look at what its status
is as far as the snapshot value goes and the
archive value. So sometimes this can be very useful because
if the snapshot's there and the archive isn't there, you know you simply
have some kind of an archive problem. It can help you isolate
where the problems may be.
And another utility you can use
is within the -- now, this would be within
the System Management Tools under
Operation. Now, you'll find that there is
a set of snapshot and archive statistics
that you can look at and update.
So, for example,
what I did just now, I took a look at both
the -- all the snapshot and the archive
metrics here, and they are updating once
every five seconds, so this five second update
is showing me the values in the last five seconds
and, and these, this
column over here shows me the values since the last time I
stopped and restarted PI. So, this can be interesting
if you see a lot of things like out-of-order events
in either the snapshot or the,
or the archive. That's an indication that somehow you're getting data
that's coming in out of the normal time series, and that's
something that's, that's really worth, worth looking at.
And from here there's certain specific things to look for.
For example, the archive backup flag. If that's set equal to
one, that means that a backup is probably in progress
or perhaps failed and left the archive
offline. Also, if you
notice the events sent to the queue
and there -- that event
sent to queue just does not, does not continue to update,
then that may mean that you've got some kind of an archive
problem, and in that case, the events in the queue are
going to keep growing and growing. So, watch
out for that. The, this would be an indication
there's some kind of a, a,
event or a buffer that's filling
on the Home Node because the archive is not able
to meet the performance
of, or the demand. So, you would see that
by looking at this Events in Queue and the number of
Overflow Queues.
Now the first line of defense in doing any troubleshooting
is the Message Logs, the System Manager Message Logs that
we store on the PI Server. So here's what it looks
like. We've used it a great deal in this class so far.
So, I would simply go into Operation and Message
Logs. And this is where I, where I would go through and
search through the Message Logs. Again, you can look through
the local pipc.logs as well. Now,
the second place to go, in addition to the Message Logs
where you'd find those server-based
messages, is to look in pipc.log.
Now, probably the, one of the easiest ways of bringing
that up is using ICU,
Interface Configuration Utility. I still find this
is the easiest, most intuitive way of opening this file.
So, you go into Tools, and within Tools, go into
Log Files, and you're gonna be looking for this file
and all the older versions of that file.
Yeah, see Tools, Log Files. I can see the
latest pipc.log.
So this holds the messages for all the API-based applications.
And, of course, that's going to include interfaces on a Remote Node.
On the Server Node, it's generally gonna,
it's generally going to include all the standard PI Interfaces
like Perfmon and Random, etc.
So you can use Notepad,
as we saw, to edit this. You can also choose
Continuous, which brings up that watch-log, and just watch it
continually.
If you do find a specific error message
in the log files, then, by all means, go into the support
site, and within the support site you can
choose to search for those Message Logs. So, for
example, negative 10701 is gonna
have a specific meaning, and the more specific an error message
the better your chances of finding somebody
that's seen that problem as well. So, this describes some of
those instances.
Also, if you are making use
of the SDK Based
Clients, and
this is on a, a Remote Node, not
on the PI Server -- but if you're using SDK Based Clients
like ProcessBook and DataLink, then you
will find there is a, the --
there is a log that is
generated in a Log Directory that is
searchable only by that PI System Management Tools.
The Message Log plug-in. You can also use
pigetmsg from
the DOS prompt in the pipc1911admin
directory. But the, again, the better tool would be
to use the Message Log plug-in and you can open
that file manually. So, you'll be
able to open up log files and also
other types of the login, the dat.,
dat files that we use for messa...some types of
message logging.