welcome to the data synchronization
pattern session in this session we'd
like to talk about the different types
of data synchronization patterns that
exist so to be able to do that we want
to look at what a synchronization
pattern would consist of and the
different types of characteristics it
might have and some of those include
what type of data are we trying to
synchronize is there any metadata about
what we're synchronizing and sometimes
that includes status or different types
of conflicting data and then what is the
actual synchronization logic that the
pattern is using to synchronize the data
and keep the metadata in sync as well so
to be able to understand different types
of patterns we'll look at those
characteristics and then we'll look at
for or which can be considered classic
patterns so there are two read-only
patterns and then there are some
readwrite patterns and these aren't
necessarily patterns that you have to
use but these are classes of patterns
and these are a single example of the
set of types of patterns that can give
you ideas about how you're going to
synchronize your data but one thing
that's very important to note before we
even get started and that is you'll need
to look at one of these patterns and and
look at how you're trying to synchronize
your data and what your business logic
and business problems are because
depending on the type of data you have
you may need to look at some of these
patterns and customize one and create
your own and that's probably the best
answer as far as data synchronization is
concerned but to understand how you
might want to approach things it's much
better to have a few examples to look at
so let's get started the different
characteristics that we might care about
in a synchronization pattern would be
the data itself and inside of out
systems where we have a offline data
synchronization framework we need to
know what type of database entities are
going to be involved in the
synchronization what type of local
storage entities
and are these similar or duplicate
entities that are just on both sides or
do they actually have different types of
structures that need to be mapped
besides those types of entities we may
need a metadata entity to keep track of
other information about the
synchronisation and then we may actually
change some of the local storage
entities to have metadata attributes to
keep track about how often that entity
has changed or what type of change has
occurred and then last but not least
when we look at the data sometimes we
may need to keep track of this metadata
in this other information depending on
whether we're online or offline so for
online we may actually have the ability
to do the update directly to the server
but we're only going to use these
patterns when we're offline and then
we'll synchronize after that so
sometimes the actual work that we're
doing will depend upon whether or not
our methods and our capabilities in the
device are sensitive to whether we're
online and what type of network we have
so those are different types of things
that we need to consider and factor in
to be able to understand our patterns
well and then the second big thing is
what's the logic and the logic is really
what are the steps that we need to
synchronize the data to the server and
this is going to depend on the specific
pattern and that's why we have four
different classes of patterns that we
want to talk about so to support all of
the different patterns we're going to be
using the different actions that are
built into the the framework so when we
go to the logic tab we know that there's
the offline data sync folder full of
client actions and there is the offline
data sync action the configuration and
the trigger and then down below for
server actions we have an offline data
server data sync action so in our case
we're going to be constantly looking at
what would we implement in the offline
data sync and what would we implement in
the server data sync
and in reality when we say that the
client is going to call the offline data
sync action that action is also going to
be as part of what it does calling the
server data sync so we're not really
don't worry about the configuration as
to whether or not we're going to sync on
the login and some of those other things
and we're not going to worry too much
about triggering it you can trigger the
offline data sync anytime you want that
really ends up calling what's in this so
we're going to focus and looking at
these patterns on the offline data sync
client action and the server data sync
server action so let's go ahead and look
at the four different patterns so the
simplest of all of the synchronization
patterns is a read-only data pattern and
that basically means that the server has
the data and it's allowed to update the
data in the database the device will
either not change any of the data that's
in local storage or if it is changing
the data it's never sending the data
changes to the server so it has a local
copy in general it shouldn't be changed
in local copy but even if it is it has
its own local copy and anytime we do a
synchronization we're just going to get
the things that are in the database
server and all of those entities are
going to come to us so if we look at the
synchronization steps we basically have
to have the client call the server to
get the data and that would be this
server data sync action the server is
going to return all of the data so here
in server data sync we can see that it's
going to do some checks to make sure
that we have access to the data it's
going to log the fact that this
synchronization is occurring and then it
goes and gets the data so in all of
these examples we're going to pretend
that we have a contacts list and that
we're going to be getting the
information for that contacts list so
the server is going to grab all the
contacts it's going to store them in the
output of this action and then in the
output this action it will be able to
pass them along to the client the other
actions inside of this client action so
from there the server returns all the
data the client then will delete
everything in local storage so it'll get
rid of all the local contacts at the
moment and then all it will do is
the things that have come from the
server and create all new local contacts
so when we think about this from a high
level this basically is a big push it
sends all the data every time we go
ahead and update all the data every time
and we're not changing any of the data
so there isn't a lot of synchronization
here it's more of a here we're going to
push all of the data to you and you're
going to be able to get new sets of data
whenever you choose so this is very
simple there isn't really any
negotiation or changes between the
device and the server and not only is it
simple but it's not terribly efficient
because if we have huge sets of data
we're getting all of the data every time
and we may already have 95% of the data
and only 5% has changed but instead of
sending us the 5% that's changed it
sends us everything but this is very
simple it only takes a few actions to
implement it's very quick and if the
data is small it will happen fast in
between the server and the device so it
can be a simple decent pattern when the
amounts of data are small and and this
is the simple type of logic that we need
the second class of types of
synchronization would also be read-only
where the device isn't going to change
the data but now it's optimized don't
give us all of the data only give us the
data that's changed so again only the
server will be able to update the
database the device is not going to
change any of the data or if the device
does change the data we're not trying to
send those changes back to the server
because this is all read-only so in this
case the synchronization steps the
client is going to call the server but
we have some metadata we actually kept
track of the last time we called the
server to do a synchronization and in
the last successful synchronization the
server told us what time it was when we
did that synchronization so we'll start
by getting that last sync time and then
we'll call the server and ask it to
synchronize the server will again do it
checks and things that it needs to do
and it will check for all of the
contacts that have changed
since that sink date it will also look
for any contacts that have been deleted
since that sink date and then it will
set those in the lists and return them
to us as output so once the server
returns all of the changed or new
entities and all of the deleted entities
we on the device and on the client side
will create or update all of the local
contacts from the things that have
changed and then we'll loop over all the
ones that have been deleted and will
delete them the server will have also
returned to us the new synchronisation
time when it just sent us all this data
and we'll go ahead and update that in
our last sync metadata entity so that we
keep track of that synchronisation time
and then we can hold on to all the data
and track all the changes again until we
start another synchronization and start
the whole process over again so this is
only going to send the change to data to
us which means there's a lot less data
it's more efficient so if you have a
huge amount of data that needs to be
synced this is only going to sync a
small portion of that data because it's
only the changes so this type of pattern
can be a really good pattern if you
expect the devices to need to hold large
amounts of content so the next class is
readwrite so we've only been looking at
read-only data and that can be a little
unrealistic you know there are devices
that are going to consume data but
usually if the device has the data we
may want to allow the user to go ahead
and update it as well so in those
readwrite scenarios one of the patterns
that we can use is called last right
wins and that just basically means that
whatever the last update occurs it's
going to occur it's going to push it on
to the database and that's going to be
the correct data so we're not really
going to worry about the time when the
updates occurred so readwrite last right
wins says that in the synchronization
steps the client on the device side will
retrieve all of the entities that have
changed so locally on our device we're
going to see if we've added any contacts
updated any contacts or delete of any
contacts
and once we get all those lists we're
going to call the server with a list of
changes so from our offline data sync
we'll call the server data sync and then
the server will take all of our creates
and go ahead and create them so anything
that was added gets created anything
that's updated gets updated and anything
that was deleted gets deleted and now
that it's updated everything on the
server side it can retrieve all of the
contacts again and set that list as part
of the output and send it back to us so
at that point the server now has the
server data sync has passed to us all of
the updates and everything that is
currently on the server so again we do a
simple delete of everything that we have
in local storage and then we create
everything that the server just gave us
back in the new one so again all the
data is going to be returned with each
synchronization so again if we have a
large data set this may not be the
appropriate pattern to work with we may
want to incorporate into this adding the
sync time and only looking for things
that have changed at a given time but in
general because we're allowing changes
to be made on our side we do have to
push all this over to the server and
then the server sort of has to tell us
about other things that have changed and
if you don't do the full set it can get
a little tricky as far as tracking all
of that data and that gets us to the
last class which is let's do readwrite
and keep track of when those changes
occurred because what happens if on our
device we make a change to a particular
contact but because this is all
distributed data and many devices are
talking to the server someone else also
updates that contact
well if multiple people can update the
same piece of data then I may ask the
server to go ahead and update something
I've changed but it may already have
been changed so the question is can we
have a pattern that can detect the
conflict and then help us resolve the
conflict so this is the most complicated
of the examples that we have and there's
a lot of steps to this so the first step
would always be to make sure that before
the client tries to do
synchronization it checks if there are
any unresolved conflicts so if a
conflict had been reported to us before
this then we should have had taken some
sort of action and sometimes that action
is to automatically resolve the conflict
and sometimes it's to allow the user
through the user interface to choose to
update the conflict and resolve the
issue if the user doesn't do that they
may accidentally kick off another
synchronization without having resolved
the conflict so we may want to warn them
that hey why don't you fix this conflict
before we try to do a synchronization
again so most of the time it would be
helpful to abort the synchronization at
that time now once we know that there
aren't any unresolved conflicts then on
the device side we would retrieve all of
the changes and then we would call the
server with the list of changes and the
last time we did a synchronization the
server would then check the data for any
types of conflicts and it would create a
list of the conflicts and now that those
conflicts are separate it can update the
rest of the data and then it can return
all that data to us on the client-side
and the device as the server returns us
the list of data and the list of
conflicts will check and look at the
conflicts and see if they can be
resolved will then delete the local
storage set of information create new
ones and then try to resolve any of the
pending conflicts so this is interesting
and it's a good list as far as what's
happening but we don't have the graphics
and the reason we don't have the
graphics on the side here is because the
graphics are rather large so when we
think about this pattern there's a lot
of things happening in the action flow
just to do the offline data sync so to
kind of go through it again real quickly
we check if there are any conflicts and
if there aren't any conflicts then we
would go ahead and get all of the
contacts that may have changed and send
them to the server the server looks at
our changes and attempts to determine if
there are any other conflicts with
things on the server
for each added contact it checks and
keeps up-to-date and as it looks for all
of the contacts when we get things
returned from the server it will go
ahead and keep track of the conflicts
and whether they've been resolved if
they haven't been resolved then it will
kind of wait but once they are resolved
then we'll have a list of contacts that
can be updated so once we get things
back from the server will delete all the
contacts will attempt to do the updates
and then we'll also update the last time
that we've done synchronization so here
we've seen everything that happens on
the client side and we haven't
necessarily looked at what the server is
doing as far as its logic and this is
the server logic so the server logic
will begin a synchronization pattern by
logging what it's doing it will then
look for any of the new contacts that
need to be created it will also look for
any of the ones that need to be updated
and if it finds one that's been updated
that has also been changed then it knows
it has to add that to the conflict list
if not it can just update the contact
and we're good for each delete it can
look at the delete and then it can also
say ok was that changed before the
delete occurred and if that's true then
this becomes a delete conflict otherwise
we just delete things and we go now that
we have a set of conflicts we can build
them into the conflict list we can go
out and try to get all of the contacts
that aren't part of the conflict list
and send all of those lists back so you
know at that point here we would be
getting back the list of all the
contacts and all of the conflicts and if
there are conflicts it goes through this
path to try to resolve the conflicts if
there are no conflicts then it just goes
down the normal path of deleting the
contacts and adding them back in and
updating the last time that we've
synched now in this case when we look at
why are we updating the last time that
we've synced we're not really using it
on the client side that's really just
being used on the server side to make
sure that it can check
what's been changed since that
particular time so this is obviously the
most complex one and is something that
can be really interesting
so in general we've gone through these
for synchronization patterns and they're
good representatives of different
classes or types of synchronization if
you want to look at these and learn more
and if you want to see how they work you
can actually download these so if you go
out to the OutSystems 10 documentation
underneath developing an app there's a
section for using data and in the using
data there's an offline section and
there's a special offline data
synchronization patterns section where
it has detailed explanations of all of
these and it gives you the capability of
downloading these patterns that have
been implemented for the contacts an
example so sometimes it's nice to hear
someone talk about these but it's a
whole lot better to get your hands on
them and maybe even you know run them
with some data and test them out and see
how they work so in this section we try
to talk about synchronization patterns
and the different types of
characteristics they might have and then
show the examples of the four typical
types of patterns that can occur but
when you go to implement your own you're
going to probably want to customize
things to be specific to the types of
logic and the types of data that you
have but this should be a good way to
get started and hopefully now you
understand some of the capabilities and
the types of patterns that you could use