Hello and welcome to how do you get 30 000 photo attachments out of arcgis enterprise or arcgis
online my name is carl von stetten i'm a gis analyst with central san where i've worked since
2000 and have been using fme since about 2002. central san is a wastewater collection and
treatment agency serving nearly 500 000 people in central contra costa county which is located in
the san francisco bay area in northern california our headquarters are located in martinez
this is an agenda of the discussion this morning i'll talk about
why we had to extract arcgis attachments some of the technical issues that were involved
and some tips for developing arcgis attachment workspaces
so we were assigned a project to inspect all of our sanitary sewer access structures and we
have a variety of types as you can see from the table there and in total we've got almost 39 000
structures and we needed to improve the location precision for these structures in our gis system
now many of these structures are concealed under tree canopies so we are not able to use aerial
imagery to fine tune their location so the project involved having someone go out and visit each
structure take a close-up photo of the structure cover and then two or three additional photos to
help identify the location and the in the context we also needed to capture whether the structure
was located in bike lanes pedestrian walking paths or in public or private streets we also
needed to identify any potential safety hazards and flag those for review and corrective action
so our group built a field data collection application using a combination of arcgis pro
portal for arcgis enterprise and the field maps application for ios
here you can see one of our field operations specialists
is collecting the inspection data and taking photographs
we then built a nice dashboard in arcgis enterprise to track
progress and examine the collected data notice the map and pop-ups in the lower right corner
and particularly the photo attachments we needed to pull those photos out of arcgis enterprise
so that we could load them into our document management system
we use laserfiche we also needed an export of some of the inspection and photo attachment attributes
so that the photos could be bulk loaded into laserfiche with their required metadata
so i'm going to make kind of a bold statement here and that's that
working with attachments in arcgis can be challenging but fme can make it easier
so i'm going to answer some of some questions now such as how are attachments stored how does
fme read the attachments and how do you save attachments to files
now when working with attachments we encountered a number of challenges we're using the arcgis portal
feature service reader but the same issues apply to the arcgis online feature service reader as
well and that's that the reader optionally exposes the attri the attachments as a list attribute
if there's at least one attachment on that feature and each attachment in
the list is made up of a number of attributes including one that stores the actual document
now the document itself is stored as a base 64 encoded binary large object or blob
photos if there's exif data attached it's not stored in those attributes
now one important note as of fme desktop 2021.1 the version we're currently using
the feature reader transformers do not support accessing feature attachments from portal or
arcgis online you have to use an actual reader so let's get into the solution now
so as you can see this is not a large workspace there's
a reader eight transformers and a writer and this application i'm using the esri arcgis
portal feature service reader but all of the settings and behavior that you're going to see
are identical for the esri arcgis online feature service reader
for the purposes of this solution i don't need the geometry so the second transformer in on that
workspace is a geometry remover i'm getting rid of the geometry because
it will improve the performance of the remaining part of the process
now i'll walk through the rest of the workspace in more detail
we start with the reader now you must set the include attachments setting on the reader
to yes in order to get the attachments and their data
if this is set to no attachments are completely ignored by the reader
i then use the tester transformer to make sure there's at least one attachment on each feature
there's no point passing any features on that don't have attachments in fact it's likely an
error will happen somewhere downstream if they aren't filtered out the arcgis online attachment
attribute is the list attribute that holds all of the attachments for each feature
if there were no attachments on a particular feature that list attribute would not exist
thus the testers checking for attribute is not missing
and the ones that features that have that attribute are allowed to pass
the attachment data is stored in a list attribute here i'm using a list exploder
transformer to explode the list so that for each attachment a new feature will emerge
with the original features attributes and the attachments attributes combined so basically
if the original feature had three attachments three features will emerge from the list exploder
next i use the attribute manager transformer to remove
most of the unneeded attributes from each feature
i also built the file name that i'll be using momentarily to save each attachment to a file
here any spaces are removed from the attachment's name attribute and then it's appended to our
common name identifier of the sewer structure which came from the original feature itself
also note the data attribute that attribute came from the attachment
and contains the attached document or photo we'll be using that in the next few transformers
remember i mentioned earlier that the attachment is a base 64 encoded binary large object we need
to decode it first using a binary decoder transformer so that we can write a usable
file to disk we also need to make sure that the character encoding remains set to binary
with the attribute file right transformer you can write the contents of a single attribute
directly to a file on disk in this case once again i'm using that data attribute
that now contains the binary object free from the base 64 encoding for the file name
i'm using a user parameter that stores a folder path and then appending a backslash
and the file name file name i created earlier with the attribute manager
also again note that the target file encoding is set to binary so that the file is written properly
now earlier i mentioned that the arcgis attachment didn't store any exif data
for the photos arcgis doesn't really know or need to know much about the attached documents
in order to get the exif data for the photos each jpeg file needs to be read back
in fme so that the exif attributes can be exposed the feature reader reads back
each of the files that were created by the attribute file writer in the previous step
i chose to use the single output port to keep things simple and to merge any exposed jpeg
attributes with the attributes of the incoming feature in this case i'm exposing one particular
exif attribute jpeg exif date time there are dozens of exif attributes that can be used
lastly i'm using another attribute manager to set up the data that i need to write out
to excel which will be used when everything is imported into laserfiche i parse the date from
the exif data because it's read in as a string and not a native fme date i'm also building the
virtual folder path that will be used inside of laserfiche when the documents are bulk loaded
so in developing this workspace we found a couple of things that i thought i'd pass on as helpful
tips because the reader is loading not only the features but also the attachment data along with
them the data will load much more slowly than if you were just reading the features alone
even in a high speed environment so the first tip is to limit the number of records to read
until you're ready to actually do a full production run and the way you do that is to
set either max features to read or max features to read per feature type in the navigator panel
and you'll find that under reader parameters and under features to read within that
then remember to go back and clear that setting when you're ready to actually do
the full production run so you can read all of your features the second tip involves feature
caching now feature caching is a great feature when you're doing development
but it will add significant amount of time and resource consumption to the process when
you're doing the full production run or reading a large number of features and their attachments
it may also crash your computer because it's basically writing out those features to a a
feature store file on your hard disk and if you're reading a lot of features and a lot of attachments
that may end up filling your hard drive up completely and then causing a system crash
so make sure that you turn off feature caching when you do your production run
so the results of the workspace after we ran it in production
we got two output products on the left you can see there's a windows folder that has about 33 000
jpeg photos in it and on the right we have the excel spreadsheet that has about 33 000 rows of
all of the associated metadata for those files and together those can then be used to bulk load all
of this data into laserfiche the fme process took about 34 hours to complete for this batch of data
and once all those photos were loaded into laserfiche now they're accessible from links
in our vertiges geocortex web applications as you can see on the left after selecting one of our
structures and getting the information on the the pane on the left-hand side of the web map clicking
the link for asset management documents shows all of the available documents and then clicking the
link within that gives you the photos so any one of our staff now has access to this information
so earlier i made a statement that working with attachments in arcgis can be challenging
but fme provides the tools that can make it easier
so now that all those photos are loaded into laserfiche as i said they're accessible to
all of our staff through multiple web applications and because we built this
workspace in fme we can reuse it again as we have two more phases of work to complete
and when all is said and done we'll probably have more than
80 to 90 000 photographs that will have to be loaded into our document management system
so to sum up if you need to read and extract arcgis attachments for any reason
fme can help you to do it efficiently now other attachment-related workflows can be
developed using some or most of the workspace that presented such as pulling down attachments
from portal or arcgis online and then putting them into a local or enterprise geodatabase
and before we go i wanted to share a few resources that may be helpful if you're
developing a similar workspace the first two links there are to fme documentation for the
the two feature service readers the format attributes for portal and for arcgis online
uh then there's a link to an fme community article that deals with the whole base64
encoding issue that i encountered and lastly the fme documentation for the jpeg reader
it documents all of the format attributes that you can work with including all of the xf1s
well thank you again for your time this morning if you have any questions or
want to reach out for more information there's my contact information
and i hope you enjoy the rest of the fme world tour thank you again