hi I'm Kevin Weller a certified FME and FME Server Professional,
Systems Integrator, GIS bootcamp instructor, and software engineer by trade
I tend to push the envelope when it comes to treating FME workflows as custom software
requiring modular architecture automated testing and other software engineering methods
I own modern geosystems a safe software service partner specializing in comprehensive geospatial
solutions architecture implementation and application development we're often serving
clients with complex requirements involving advanced FME Server and Cloud deployments
one such client needed an up-to-date replica of their Salesforce data including location
data and an ESRI compatible PostGIS enterprise geodatabase so they could more easily integrate
it with other enterprise data including geospatial studies involving a high availability ArcGIS
enterprise deployment we installed for them between Salesforce and their geodatabase they
needed changes in one to be reflected in the other in near real time so everyone would be
on the same page including sales staff managers and geospatial market analysts
so I decided to implement a bi-directional data synchronization scheme using the relatively new
streams feature in FME Server streams are a powerful means of quickly processing small
packets of data that can come at any time often many times a second whenever a stream
is active the workspace associated with it runs continuously processing each information packet
as a feature this eliminates the significant overhead inherent in starting and stopping the
workspace for each event and the stream monitors the workspace job to make sure it continues to run
even if it has to restart the job due to workspace error or system reboot as long
as the workspace implementation is efficient the system can handle very high volumes of activity
such as when many online users are busy making changes in Salesforce and the geodatabase
in our solution there are two streams Salesforce to geodatabase and geodatabase to Salesforce
the streams are configured via the FME Server web portal you can see both streams here along
with the operational status of each we did however run into a few technical challenges
during implementation one such challenge was that neither the Salesforce streaming API nor
the PostgreSQL notification protocol is directly supported by FME at least not yet our solution
was to implement a couple of system services in python to continually forward these messages from
their native sources to SQS queues this is part of the implementation for one of those proxies
here you can see how we configured the Salesforce to geodatabase stream to run the corresponding
streaming workspace the workspace parameters specify the required web and database connections
as well as the queue to monitor for Salesforce data change events
each of our streaming workspaces includes an SQS connector transformer to receive cue messages
note that the mode is set to stream which means the transformer will run continuously waiting for
events and passing each event on as a feature to be processed by the rest of the workspace
we had to tackle a few more challenges along the way for instance our implementation had
to avoid synchronizing Salesforce controlled metadata and read-only computed fields back to
Salesforce so we generated a self-maintaining data dictionary to keep track of which fields
were which type we also had to unpack and repack structured Salesforce fields composed of several
values using combinations of python collars and other transformers inside reusable custom
transformers another challenge arose from the fun fact that there are many Salesforce APIs
and we had to use several of those in the overall solution depending on what we were trying
to accomplish at any given time however these various APIs are terribly inconsistent in terms
of parameter structures and data representation so we encapsulated and hid these differences within
custom transformers as well and standardized data representation in the local geodatabase a lot of
Salesforce installations are configured to run background processes that further alter data after
it's been created or modified by a Salesforce user or API call for example one such process
might geocode an address after a user has created or changed it resulting in multiple messages from
the Salesforce streaming API all caused by the same initial transaction these exchanges
between the two systems can react to each other such that they go back and forth forever trying
to assert dominance over the data this is known as a race condition in computer science jargon
we implemented an algorithm in the sync workflow to detect and correct for these conditions finally
even though FME Server streams are self-repairing we had to deal with situations where the sync
streams go down for any length of time due to some uncontrollable situations leaving the data
unsynchronized so we also implemented a nightly batch migration using FME and the Salesforce bulk
APIs to capture any changes that might have been left behind by such events during the previous day
this was also the best way to get all the relevant Salesforce data into the geodatabase in the first
place streams are a powerful new feature of FME Server that enable fast processing of high volumes
of real-time data data synchronization between systems is but one use case for streams but it's
worked beautifully for us we now have a Salesforce real-time data sync solution that by our contract
with our client we can reuse for other clients or even within our own business processes
I hope sharing our experiences with streams will inspire you to imagine how you might
use this real-time FME Server capability thank you for your time and attention