In this video we will go over the general approach of troubleshooting an issue with
asset analytics. The approach presented in this video is applicable to a wide variety of issues
though we will be focusing on a specific issue which is that of analysis outputting
calc fail. For all of the technical details please see the comments below this video.
Now before jumping into the troubleshooting, let's look at our setup. We have a compressor asset with
three attributes: we have two attributes these input and output pressure that will be the
inputs to our analysis notice that they're both PI point data reference we therefore have historical
and real-time data. This third attribute here the pressure difference is the output of our analysis
and it also has a PI point data reference as we can see the current value is calc failed.
Now let's look at our analysis. Our analysis is a simple pressure difference analysis.
Therefore we're simply subtracting the output and the input pressure.
The first step in troubleshooting is to make sure that PI System Explorer is able
to calculate this analysis correctly using real-time data. So I'm going to
click on this evaluate button and make sure that we get a numerical output and not calc
fail. As we can see PI System Explorer is able to correctly calculate this analysis.
If we would have had a value of calc failed we would have had a meaningful error at the bottom
of this window. Now the next step is to make sure that PI System Explorer is able to calculate this
analysis using historical data. So I'm gonna right click on this analysis and select preview
results and I'm gonna preview the results for the last hour. Now the idea here is that compare this
Delta P column with our inputs. The Delta P column is the output of our analysis and we
want to see if there are any calc failed values in the preview results. If there are any calc
failed values we can then go and compare with our inputs, the output pressure input pressure
and see if there's maybe unexpected values in our inputs. Looking at the preview results it doesn't
seem like there are any values of calc failed I therefore need to pursue my troubleshooting.
All right the third step now is to go and try and backfill and recalculate this analysis. To
do this I'm going to right-click on the analysis and I'm going to go and select backfill and
recalculate. Now it's very important to note that backfill recalculate is different than
preview results. Preview results were asking PI System Explorer to perform the calculation
therefore our user is being authenticated for the input data and it's PI System Explorer
performing the calculation. When we select backfill recalculate we are asking the PI
analysis service that's running as a dedicated service account to perform the calculation;
therefore it's the dedicated service account that's used for authentication on the
input data and it's the PI analysis service performing the calculation.
So now let's try and backfill and recalculate this analysis and see if we are able to remove
those calc failed values. So I'm going to recalculate the analysis,
I'm gonna select permanently delete existing data and recalculate to see if we can get rid
of those calc failed values and the recalculation is done; let me go back to the attributes tab hit
refresh. We can see the value of calc failed was replaced by a numerical value maybe the problem
is a result now let's wait a few seconds and hit refresh. Oh, the calc failed value is back. Okay
so it doesn't seem like we've resolved the issue but we've at least narrowed it down;
we determined that this issue is specific to real time calculation and does not happen during
a recalculation. Okay at this point in time I need more information to troubleshoot this issue.
So the next step is to go increase the logging level of the PI analysis service to get more
information. To do this I'm gonna open file system Explorer and I'm going to navigate to my analysis
folder that's in my PI PC folder under the 64-bit Program Files. I can do this by simply typing in %
PI home 64 % and then go to the analytics folder. Now the PI analysis service uses three processes:
the PI analysis manager, the PI analysis processor and the PI recalculation processor.
Each of these processes have a corresponding configuration file that we can see right here.
Now the PI analysis manager is used for managing client connections;
the PI recalculation processor is used for backfills and recalculations and the PI analysis
processor is used for real-time operations. Since we're having an issue with real-time
calculations we'll go and edit the PI analysis processor configuration file. And I'm gonna do
this using notepad plus plus and I strongly recommend using notepad plus plus as there's
some neat features that typical notepad or other word editors do not have. And now we're gonna go
and modify some information in this logger rules sections. Now before doing any of the
modifications it's very important to understand that we don't want to make any mistakes in the
config file. If we make a mistake it could render the PI analysis processor inoperable.
It's also important to understand that when we increase the logging level the logs for the PI
analysis processor will grow quite quickly and therefore consume a lot of disk space.
By default the logs are located on the C Drive so before doing this we really want to make sure that
there is sufficient space on our C Drive and we also really want to make sure to come back to this
configuration file and bring the logging level back down once our troubleshooting is done. We
don't want to leave the logging level on for an extended period of time. To change the logging
level we need to go and change the min level from info to trace. Now there are quite a few different
loggers here so it's important to pinpoint which loggers we need to increase the level for.
In this specific case we're having an issue with either the evaluation of our analysis. So I'm
going to change the evaluation to trace or the data retrieval for the input data our analysis
so I'm going to change the data retrieval to trace. If you're not sure in your case you could
try increasing more loggers than not enough to trace just make sure you have a sufficient disk
space and make sure to bring these back down to info once that's done. I'm going to hit
CTRL-S to save this configuration file. I'm going to navigate back to PI System Explorer
because I need to make sure that the issue is gonna happen again so I'm gonna hit refresh.
Okay I got another evaluation now let's go look at the logs. Let's go back to file explorer and the
log file is located in the sub folder of program data. Program data is typically on the C Drive and
we can navigate to it by doing % program data % and then I'm going to navigate to OSIsoft,
PI analysis notification and logs. The complete file path to the log files is
right here. Now since we were dealing with a real-time execution problem I'm gonna go look
at the PI analysis processor log. I'm going to right click and open this with notepad plus plus.
Oh wow there's a lot of information here. I'm gonna need a strategy to go through this log
file because reading this line by line would be way too time-consuming. Okay I'm gonna go back
to PI System Explorer to my analysis tab and I'm going to right click on the analysis and select
copy. What this does is that it copies the full path of your analysis to your clipboard. So I'm
gonna go ahead and do that and then I'm going to go back to notepad plus plus and hit CTRL-F to
bring up the fiind dialog box and then CTRL_V to paste in my full analysis path. As we can see here
there's a complete path to our analysis and I'm going to use a neat feature in notepad plus plus
and select find all in current document to find all of the log messages related to this analysis.
So let me click on find all in current document and we found two lines where this analysis is
mentioned. Let me go to the first line and see what that message says. Okay so we can
see here this is a message linked to evaluation and I have quite a bit of information here;
I have the trigger time of the analysis, I have some information on our inputs and on our outputs.
Alright let's go look at our inputs, the first input output pressure I have a value of 135,
the second input input pressure oh I have a bad value here and my analysis I'm doing
output pressure - input pressure so I'm doing 135 - bad okay this makes sense
that I'm getting a value of calc failed since I'm subtracting an integer to a digital state value.
Okay at this point in time I've identified the root cause as to why my analysis is
outputting calc failed is because one of my inputs has a bad value.
And as we can see we had a bad value that was replaced shortly after by a numerical value. So
it seems like the interface or the application that's writing to this PI point for some reason
is writing a bad value and then replacing that bad value by the correct numerical value. Now the
proper solution to resolve this issue completely would be to investigate the PI interface or the
application that's writing to this PI point and determine why a bad value is being written
and if it's possible to fix the application or the interface to not write a bad value.
If we find out that it's not possible to replace this bad value or prevent this bad
value from being written, an alternative solution would be to enable this analysis
for auto recalculation by going on the Advanced tab here and selecting automatic
recalculation down here. Now note this is a new feature of AF 2017 R2 and therefore
you must have at least this version to be able to utilize this feature.
Now to recap we were having an issue with an analysis outputting calc failed;
we confirm that PI System Explorer was able to properly evaluate the analysis in real
time. We confirmed that PI System Explorer was able to properly calculate the analysis using
historical data by doing a preview results; we made sure that the PI analysis service was
able to calculate this analysis using historical data by performing a recalculation. Since none
of these three steps provided a hint as to what the root cause may be we went into the log file
and increased the logging level from info to trace and we consulted the logs and found out that one
of the inputs actually had a bad value and this is why our analysis was outputting calc failed.
Now don't forget to go turn the logging level back down to info;
I hope this video was informative, thank you.