- [Donal] Hi, my name is Donal O'Leary.
I work for the National Ecological Observatory Network
as a Data Science Educator.
Thank you for joining me today
for the EGU 2020 Data Help Desk.
Today we're gonna go over
NEON's instrumented systems data,
how they're stored and organized,
'cause it's not always intuitive at first.
So I hope by the end of this short video,
you'll have the tools and knowledge you need
to move forward with using our IS data
in your own research.
So what we're looking at here
is the Explore Data Products page.
Megan Jones gave a great
discussion of how to use this in the previous video.
So today, we're gonna be looking at
rather than phonology data,
we'll take a look at
photosynthetically active radiation data
or what we call PAR.
So if we type PAR into the search bar here,
you'll see we start to get some filtered results.
And this first one here
is the one we're interested in.
Photosynthetically active radiation.
Something I wanna point out
is in this description,
it says that observations are made
by sensors located at multiple heights
of the tower infrastructure.
We'll get to that in just a second.
But I wanted to highlight that before we get started.
So if you click on the Download Data button over here,
it'll take you to the Data Download page.
And just,
really quickly I'm gonna go through this,
we'll select three months of data
from the Wind River Experimental Forest,
let's do September through November of 2019.
Some good example data I'm used to working with here.
So we select the site
and the date range,
we will want to include the relevant documentation.
And let's use the expanded package.
We'll agree to the citation policies.
If you click download data,
that will initiate your download.
I already have this ready here.
So I'm just gonna open this up
and show you what it looks like
inside of this zip file that you're gonna download.
So if you open that up,
make this a little easier to see here.
You'll see that
within the zip file,
we have our five PDFs for a relevant documentation
that will have all the information that you need,
every little technical detail about how we
collect these data
and how we validate them before publishing.
And then you'll see three more zip files here.
These are the site months
of data that we downloaded.
So we can see 2019 of nine,
10 and 11 for September through November.
Let's open one of these up
and take a closer look inside.
All right, we have quite a lot of files here.
So I'm gonna zoom in make it easier to see.
And let's go through them here quickly.
If we go down to the very bottom,
there's four kind of metadata files down here.
This is an XML file of an EML,
if you're familiar with that,
metadata style,
you'll know how to use this.
If not, you can read more about EML files
through your favorite search engine.
Next, we have the README file.
If you open this up,
it'll describe all sorts of things
including the naming convention
for these very long complex file names
and the data files we have up here above.
The next two are really relevant to the,
this particular data package.
So let's open up the variables file,
and take a look in here.
So this is a CSV that contains information
about the different variables that are reported
in this data set.
So it'll tell you which table it comes from
these part data are separated into one
and 30 minute averaging intervals.
Then we have the field name here.
So this is the
field of the particular data set
within the the actual data CSVs.
Some of them,
you can kinda guess what they mean,
just by the name,
the start and end date are pretty obvious.
But we have this nice human readable, description here
that'll give you a more detailed information about
what that variable is actually reporting.
This one is really useful,
the data type will tell you
specifically what type of data
these are reported as.
The units of course are essential.
Something I didn't know
before really digging into this data product is that
we're actually counting the micro moles
per square meter, per second,
as incoming energy,
not just wattage or something like that,
but a more raw, basic scientific metric here.
And then you can see which download package it's a part of,
you can see our basic download package is
really just the relevant variables
to the data products.
But then in our expanded data set,
we report all sorts of quality control metrics.
So anything that ends with a QM
is a quality metric or quality flag
as we like to call them.
This gives you an indication
of just how rigorously we validate our data
before we publish them and send them to you.
Last one is the publication format.
This will tell you if it's like a date time stamp,
exactly what format we're using,
are the values rounded,
should they be integers?
This is really valuable.
So you can get a sense of the level of precision
that we're reporting in these data.
Next one I'm gonna look at here
is the sensor positions file.
So before we go into this,
let's talk about the PAR,
sensors in particular.
I chose this data product because it is
one of our many instrumented data products
that are collected at many locations.
Some of our instruments,
we only have one of each site.
But for things like soil arrays,
and these PAR sensors,
and temperature sensors,
along the tower,
we have multiple locations.
So you can see this is a
schematic of our many different
flux towers that we have across the observatory,
all the way over here on the right
is the Wind River Experimental Forest.
This is in southern Washington
in a temperate rain forest.
So these trees are huge
and this flux tower has to be above the trees.
So it's a very tall tower.
And it has a number of these booms
that come off of the side of it.
Each one of these booms has a PAR sensor on it.
So you can see that
we have PAR readings for many different vertical positions.
Those vertical positions are reported then
in this sensor positions file.
So if we open that up,
you can take a look here.
And see, the first variable being reported
is the horizontal.vertical positions
that kind of concatenated together.
They're all in the same horizontal position,
they're right there in the tower,
which is horizontal position zero.
And then we have many different vertical positions,
including two vertical position eight,
which are the upwards and downwards pointing PAR sensors
at the top of the tower there.
If you scroll over to the X, Y and Z offset,
this will give you a spatial reference
for exactly where the sensors are
relative to our datum spike in the ground.
And the Z offset you can see is the
height of the sensor in meters above the ground.
So this flux tower goes all the way up to 74 meters tall.
And that's where these data are reported from.
Last thing I'm gonna do is point out the actual data files
'cause that's where you're interested,
in working with the data, right?
So again,
check out the README file for the exact
naming convention for these data files here.
But now you might have a better understanding
of why we would have 16 different data files
reported for this one data product at this site.
We have the one and 30 minute averaging intervals,
and across these eight different vertical positions.
So if you open up one of these,
this is maybe more what you expect to see
when you're looking at our data here.
We have a time of the reading,
the beginning and ending of that averaging interval,
and then the different variables
as reported in the variables file.
So that's a quick overview
of how these data are laid out.
Of course, if you have any questions,
please contact us at NEON.
Thanks for joining.