In this video we'll learn how to backfill data using
PI Config. For a write-up, please check the link in the description.
Backfilling data allows us to
write data into the archives after the fact. PI Config
is just one method to do this. You can also use
other methods such as PI UFL, PI to PI, or
PI RDPMS. Keep in mind that PI
Config writes data to the PI Server very
quickly so if you have a lot of data to write we highly recommend writing it to
an offline PI Server. If you're still not sure
what method to use please contact tech support.
The first think we want to do is
analyze what sort of data we need to backfill.
We'll need to know how many points should be created, what the
configuration of these tags should be, and also approximately
how much data is going to be written in total.
In our case,
I've already created the points and configured them to collect
data. You can also create fresh points which have no data in
them. The advantage there would be that you can actually use
the PI Servers Compression Algorithm in order to minimize
disk space usage while still keeping the fidelity
you choose in the compression settings.
However, please keep in mind that if you do plan on
using new tags you need to make sure that no data is written to
them otherwise you will not be able to utilize the compression settings.
In order to do this you could
either set the point source to some nonsense value
or, set scan to zero. This will prevent
new data from being written to those points.
The next thing we'll want to do is take a look at the
archives currently on the PI Server.
In order to do this you can use PI SMT.
Here we pull up a list
of the archives and we can see the start time and the end time for each one.
If you're using
a version of PI Server before 2012,
you're also going to need to reprocess all of the archives that were created
before the new points were generated. In this case we only have
one, but if you have multiple archives you'll need to reprocess
all of them. Now whenever reprocessing
archives we highly recommend making a backup.
So let's go ahead and do that.
Here we navigate to the current backup
directory and get it started.
Now that we have
a successful backup we can go ahead and start reprocessing these
archives. The first thing we want to do
is un-register the archive. You can do this using PI
SMT.
Remember,
we're going to want to reprocess all of the archives with a start time
before the points were created.
When we reprocess the archives records will be created for
these new points. Once we can verify that the
archive has been successfully un-registered, we can
use the offline archive utility in order to start
reprocessing. This utility is
located in the BIN Directory.
The syntax we'll use is "PI
arcss-inputfile",
the path to the archive to be
reprocessed.
-output file and the path to the
file we'd like to create the new archive.
We'll also
want to specify the duplicate flag in case there are duplicate
events.
Now that we have a reprocessed archive we can go ahead and register
it. We can use
PI SMT in order to do this.
Notice that this archive only goes back
to December of 2011.
If we take a look at the data that we wish to
backfill,
we see that it actually extends back to
August of 2011. This means that we need
to create another archive in order to hold that
older data.
Again, we can use SMT to do this.
Here we create an archive with a fixed start and end
date, setting the start time to the oldest
data point we wish to backfill, and the
end time to the start time of the oldest archive such that
there are no gaps between the archives.
And we'd like to make this a dynamic
archive.
Earlier in the video we mentioned that you could use empty points in order
to take advantage of the PI Systems Compression Algorithm.
If you did decide to create these empty points and you would like to take
advantage of the compression and you're using a pre
2012 version of the PI Server, there is one more
step that you will need to follow. Those points currently
have "point created" as their status.
We will need to remove this as the snapshot value if we intend
to utilize the compression. In
order to do this we'll use a script. You can find the script
in the write-up but I have it here copied into a text file.
This script will create a text file
"snap to delete" and it will put this in ADM,
the folder where PI Config is situated.
In order to run the script
we'll navigate to the
ADM directory and into PI
Config we'll feed this script.
Now we can verify
that the "snap to delete" file has been
created and here are the new points which have "point created" as a snapshot value.
If I would go on to delete
"point created" as a snapshot value in
all of them I could simply remove those points from
the list. Once I'm
satisfied, I can go to PI Config
and run the following commands, again
these can be found in the write-up.
And if all goes well,
we will have deleted "point created" from the
snapshot of these points.
Once we have done this we'll be ready to
start backfilling data.
You'll notice that the file
we're using is a CSV.
Excel is a good tool to manipulate
CSV's.
Here you can edit the columns and then export into a CSV
file.
You'll also notice that the first three lines of this file contain
PI Config commands.
The third line indicates what the order
of the values is. In this case, a tag, a
time, and a value.
Now we're ready to
actually backfill the data.
Note that we've
configured the PI archives and we've confirmed that we have all our
data in the proper format. We can go ahead and use
PI Config again to actually backfill the data.
In this case we're going to want to pipe our
modified CSV
into PI Config and then let her run.
Now that we've
finished writing the data into the archives we can go into
ProcessBook and confirm that it's actually there. So if we zoom
out, it looks like we've
got it all back.
Remember, whenever using PI Config
you have full access to the server and you do
want to test a small amount of data first just to make sure you're getting
the expected results.