In this video I'll be reprocessing archives to
clear corruption, combine the data from two different
archive files and to split a large archive file into smaller ones.
Archive reprocessing is a task performed
on offline archives that will take the event data from one
or two archive files and rebuild a new archive.
This task will recover any recoverable
events from corrupted archives and different options can be
passed in order to manipulate the output in specific
ways. The first reprocess I'll show you
is how to recover events from an archive that we've determined to have
corruption. This process will also be necessary
if you are back-filling to archives created before PI Data Archive
2012 to points that were created after the
archive file was. Because you can only
reprocess offline archives, the first step is to make sure
that your archive is offline. If the archive you are trying
to reprocess is primary the only way to take it offline
is to stop the PI Archive Subsystem service.
If your PI Archive isn't primary, you can choose
to unregister it or to stop the PI Archive Subsystem.
If you are trying to decide between unregistering the archive
or stopping the PI Archive Subsystem, keep in mind
the data sent to an archive gap will be lost.
And while stopping the PI Archive Subsystem will prevent
you from making an archive gap, no users will be able to
read PI data while it's down. This is my
corrupt archive and since I won't be back-filling any data
to it, we'll reprocess it by unregistering it.
I'll then open a command prompt as administrator
and then navigate
to the %PISERVER% environment
variable slash bin using the d flag.
The program we want to use is
piarcss
and I'll first pass a question mark to show you that this will display the
help dialogue with the proper use of the command and then all of
the options we'll be using below.
My first reprocess will be
piarcss to call the program
and then if. Here we have to provide
the file path to the input file. To make sure I
don't make any mistakes, the safest way to do this is to
navigate to the archive file in Windows Explorer
which is on my D drive in PIARC. And it
was timestamped on the 8th at
9:17.01.
Before we executing any reprocess, it's good practice to create a copy of your
archives before you reprocess. So I'll copy this
archive onto my backup drive F in case something
goes wrong.
Back in my D, PIARC folder, I'll
copy the folder path and then paste it into the
command line.
And then copy and paste
the file name of the archive.
The next argument we are going to
pass is of followed by the filename of the
output. Since I want to create a new archive file
as the result of this reprocess, I'll manually
provide a file path that doesn't exist yet. I'll
use a similar file path to the name of my input file
except I'll add an _reprocess at the end.
As of
PI Data 2012 or 3.4.390,
the archive subsystem will do an integrity
check on archive files before reprocessing. A corrupt
archive is likely to fail an integrity
check. So if you know your input file is corrupt, you need to
add no input check to continue the reprocess.
Next I'll add a -f0
to tell piarcss that I want to make a fixed archive
and use the same file size as the input
file. This archive is a historical
archive. So piarcss will use the same historical
start time and end time of the input file. But if it was
a primary archive, I'd want to include -oet primary
to signal that I want the output archive to be a primary archive.
Finally I'll send
the resulting messaging to a text file on my D drive
called reprocess_ and then
the file name of my input
file.
After I hit enter, the archive subsystem will work behind the
scenes to reprocess the archive. And in order to
check its progress, I'll navigate to the D
drive and open the text file I created.
We can see that piarcss is reprocessing
my corrupt file. And since the output file
didn't already exist, it's going to be
created. And if we scroll down to the bottom of the
text file, we can tell that it's not quite
done. So let's close notepad and be patient while it
finishes.
Once their process is complete,
printed at the bottom of the text file will be a complete message.
Following by some statistics about the output file.
This reprocess was a success, so I'll
go back into PI System Management Tools to
register my archive back into my archive gap.
The next reprocess I'll do
is reprocess an archive to split an archive
file into smaller archives by manipulating the
start time and end time of my output, as well
as filtering input events by time. This procedure can
also be used to change the end time of a particular
archive if you are trying to stretch it over an archive gap or
to turn an archive to fit into an archive gap that
already exists.
This first archive on my PI Data Archive was made before I configured
my automatic archive creation tuning parameters and it's a
little too large for my system. What I'm going to do
is break it into two smaller archives that have similar
sizes as the rest of my archive file. This
file starts at 7:42 and
runs until 9:17 the next morning. So I'll
separate it by breaking it in half around
2:30AM. I'll start by unregistering it
and then make a copy of it
on my backup drive again.
And back in the command prompt, I
can run two different piarcss commands to create two
separate output files. The first will be
piarcss then if
and the file path to the large archive file I just unregistered.
So let's go copy and paste the file
path
and then the file name
again.
And I want to make an output file of the same name
because this archive will start at the same time
except for I'll add an _split at the end.
Now this time I'll specify an output start time
of the 7th of February
in 2016 at 1942.
Which is the start of the large archive file
in day, month, year, hour, minute, second
format. And I'll specify an output end
time of the 8th of February at
2:32.
I also want this archive to be fixed at
256 MB like my other archive files,
but I don't need to add a
no input check because this time I want to be warned
if this archive isn't healthy. I do however want to
add a filter_x which is filter
exclusive option to filter out the input events to match
my output time.
Filter exclusive will cause the reprocess
to ignore any events that occur outside this time-range
but also ignore any events that fall on the event end time
of this time-range. I'm doing this because the
next archive will start on this archive's end time and I
don't want any events to appear in my archive files twice.
If your PI Data Archive is older than 2010,
you won't have access to this command. And the
best way to handle events that occur on an archive boundary
in that case is to end your earlier archive one millisecond
before the next one starts.
I'll send the output file to a different text file and then
run the command.
And in order to check its progress,
I'll navigate to the D drive and
open the text file I created.
You get this warning message if
you use an -oet with a filter or filter x command.
And it's just to let you know about the problem that I
described about double archive events on archive boundaries.
Since we are aware of this, this is nothing to worry about.
So we'll come back in 30 seconds to look at our output
file.
Here we can see that the reprocess is
started and we are going to create a new output
file. Scrolling to the bottom of the message log, we can see that it's not
quite done. So let's come back once it's completed.
After waiting a while,
scrolling to the bottom of my text file, I see a
success message with my output archive statistics.
Now back in the command prompt, I want
to create the second half of my archive split. So I'll input
the same archive file
but this time I'll
make an archive that starts on the half way mark of my
large input archive.
And it ends when my large archive originally ended.
And filter anything that's outside the
last half of my large archive. I'm using
filter instead of filter exclusive this time because I
want to include any event that might happen to be
on the exact time of this archive.
I'll make it a fixed archive again at
256MB. And then we'll continue our procedure
of sending the output to a text
file, running the command,
and then opening notepad to verify
that it started successfully.
Here it looks like it understood the input file and it's going to create a new
output file. So let's close notepad and wait for it
to complete.
Once it completes, I'll check for a success message
one more time. And after I confirm that it is successful,
I know that I can go back into System Management Tools
and register my two split archives to fill the space
that my large archive used to occupy.
And here we can see the result
of turning one large archive into two smaller ones.
The large task I'll be using
reprocessing for is to merge data from our recovered
interface buffer into my current archives.
Back on my D drive, I'll show you that I have an
archive file that I have from reprocessing a corrupt buffer file a
few days ago. And it needs to get merged into my
already existing PI data. But I'm not sure exactly what
time period it
covers. So back in the command prompt, I can go back a folder
into the PI ADM directory. Now here I can
run pidiag ahd to
dump the header of any archive
file. And if I target my
D: buff_reprocess.arc file,
I can see
that this file covers between
9:12:12 and 15:04:43
on the 9th of February
in 2016. This means that back in PI System Management
Tools, the reprocessed archived file starts
about half way through this archive and
goes a few hours into this archive. So I'll
take note of their start and end
time, and then unregister them so I can
merge my reprocessed buffer data. Before I go
back to my command prompt, I'll visit my
piarc folder again to make a copy of these files onto my backup
drive.
Going to my command prompt and not forgetting to return to my
PI
bin directory, I'll reprocess the reprocessed buffer file
by typing if and then typing out the
folder that it's in and then hitting the
tab key to automatically cycle through all of the files in this folder
until my reprocessed archive file path shows up.
My output file will be the first archive file on the
list. So let's go copy and paste that file path
that filepath
and that filename
into the command prompt.
I'll specify that I want the output
file to start at the same time and end at the same time
as my first archive file.
As well as filtering out any input events that
don't also fall within that time.
I'll choose to use filter exclusive here to make sure
that I don't do duplicate any events that might be in the archive boundary.
Again I'll send reprocess messages to a
text
file
and then hitting
enter I'll go see that the reprocess started without errors.
Once we see that the reprocess
is in progress, we can wait patiently until it
finishes.
After that reprocess completes
and the text file reports a success, I'll
reprocess the same file into my other archive
file specifying its different start time and end
time as well as filtering out inputs that don't fall within that
time period.
I'm using filter instead of filter exclusive
this time because I want to include any event
that might happen to be on the exact end time of this archive.
I'll repeat the same process of sending the same output
to a text file
and checking the text file to make sure
that the reprocess starts
successfully.
After being patient again, opening the text file and scrolling to the
bottom we can confirm that this reprocess
was also a success.
Now back in PI System Management
Tools, I can register those files again with the new
events that were recovered from the buffer queue.
Reprocessing is an advanced skill for PI
admins. And it's important to understand the implications and
results and to do it properly. Luckily we have lots of
resources and examples on the topic.
KB00159 is an overview of the
piarcss utility.
2367OSI8 will step you through recovering from corruption.
KB00590 shows you how to split
archives. 3108OSI8
will show you how to merge archives.
And KB00713 has a series of other examples
that you can use as references.