[Music]
cool thanks guys you can hear me okay
all right yeah so good morning I'm
really excited to be speaking at Jay s
conf I've never really spoken at a
conference before or at least one with
thanks so yeah and it's really exciting
because in 2015
Patricia Garcia gave a talk here in
Berlin at Jay s conf
on an offline first data and I wasn't
there for it but I saw it a year or two
later on YouTube and kind of figured out
where her LinkedIn profile was there was
no job so I found her github profile and
I found a link to a repo at Justin's
organization which is field intelligence
where I work now so I emailed Justin in
two weeks later I was on a plane to
Nigeria and now Patricia and I are
colleagues and I'm speaking here about
offline first stuff so the work we do is
in trying to deliver health care
commodities where it's a bit tougher to
deliver them expanding access with
partnering with federal governments and
with down to small pharmacy traders and
doing that with software supply chain
software so we were headquartered in
Abuja where we work it's in the middle
of the that just below the red part in
Nigeria and this map is population and
kind of one of the reasons why Nigeria
is because it's huge and it's growing
like crazy it has a massive market and
it's like about one-sixth of the
continent like 200 million people we're
a small team of software developers
for Asians people we have an office
there a small one here in Berlin and
then a couple operations offices in
Lagos and in Nairobi so three years ago
four years ago when the company started
you kind of asked the question okay what
tech stack are we gonna use for pharmacy
supply chain management you can buy one
you can use one that's already built
there's a lot of them you can customize
on top of a development platform like an
Oracle or in s AP a lot of warehouses
and places and G RIA do this
or you do what a lot of large companies
do a lot of large tech companies in
Fortune 500 companies where they just
build their own and if you're gonna do
that and it's 2015 2016 you're probably
gonna choose like a very boring but
sensible stack maybe something like
c-sharp net or Django with Python or
Ruby on Rails something with a Postgres
or some sort of sequel relational
backend which works really nice with
enterprise resource planning you get a
lot of stuff out of the box a lot of
tools that can really help you but for
us our requirements were a lot different
than that and the contrast that I kind
of like to invent people will talk about
these cities in different ways but I
like to think of Abuja and Lagos as the
two places we need to operate and Lagos
for people who have been there is a mega
city it's like a country it's it's 20
million people it has tons of growth
tons of money tons of opportunities and
then it also has networks that sometimes
work and sometimes don't work you're a
small pharmacy you connect to the
internet on a cell tower there's tons of
cell towers you see them everywhere but
then there's a traffic jam right outside
of your business and suddenly 10,000
people are trying to use the same cell
tower and you don't have network but you
need to keep working and then Abuja is a
planned administrator
city of just like a million people and I
always have internet there people don't
carry cash they pay with bank cards our
partners at the federal government have
this giant command center room that has
network it has HD TVs it has a rotating
web camera that follows you as you talk
and they need to see what's going on in
the whole country
so we want to make something that's
better than just a traditional boring
web application we need something that
can work in both of these places and so
that means doing something that's
offline capable and offline is not
binary it's not a it's an offline
capable app or it's or it's an online
app you have well it looks crazy on the
screen you have like a lot of different
options the first one is what we call
small offline these are names we made up
for these categories and it's kind of
like a marketing gimmick that a lot of
web-based apps use where they say we're
offline capable and people who aren't
software developers don't know what that
means but like we know what this means
it's like kind of like you're working on
an issue ticket and github and you lose
the network connectivity you can like
keep typing you know the data will stay
there
maybe it's thrown into local storage but
you don't want to like refresh you don't
want to click around so we couldn't use
this model it's not going to work for us
medium offline which is our word for
like just using a native app like a
iPhone app an Android app or writing for
a plot like Windows or Mac OS is a lot
more attractive and I've done work on
apps like that before for pharmacy
supply chain
our company has as well and it's not
something that we're necessarily going
to rule out in the future but it is
still very manual there's question marks
for how offline it is because it totally
depends on you as the developer what
kind of database are you working with
what rules are you setting up and how
are you syncing with remote so like the
example is like you pay to have offline
features in consumer apps like like
Spotify and like duolingo that give you
offline features that duolingo seems to
work really well but like Spotify
seems like every time I get on a plane
it forgets what I said to download
offline big offline is really important
to think about because this is how
hospital IT systems work most of them
historically this is changing but what
you've had over the past couple decades
is excuse me like a very large on-site
deployment of servers even a server
warehouse
you have IT staff you have a local area
network and it's because clinicians need
their software to work offline it's not
like offline is new to tech companies in
this industry but as I said like that's
not going to work for us we can't put a
server in like right now we're working
with about 30,000 pharmacies and there's
hundreds of thousands to plan for so we
went with the web with offline first in
the web that gets us kind of the things
we want low IT support you can do the
whole thing offline but it's still the
internet it's still sinking and there's
a lot of talks on which distributed
database to use like the basic setup for
an offline app web app is you put a
service worker in you tell it to cache
the static application and then you work
with local storage in the browser which
has become a lot more attractive over
the last several years you get to use a
lot of the users available disk space a
percentage of it but how do you work
with the in browser database and what do
you replicate with and couchdb and pouch
TV which some people at this conference
helps build are kind of like the tools
that most people end up using we're
really happy with the area that they
cover for us if you know of some cool
tool that somebody has or like this
question that I often get is like oh why
don't you just write your own super cool
like replicating thing that that uses
blockchain and Kafka and is
Torsten and the thing is that like with
these two our problem is never in the
actual replication protocol whenever we
don't have engineer's sitting around
wasting time trying to figure out what
steps went wrong if there's a little bit
of network the document sink so these
two have been really useful for that
like somebody gave a talk a while ago
where they said friends don't let
friends build their own replication
protocols Gregor said this and yeah all
of our problems are elsewhere we have
our stack we're sticking with it and
like what could go wrong with this
somewhat non-traditional application and
for those of you who've maybe done work
in like global public health you know
that it's not actually this very well
clearly administered highly educated
system of people with clear incentives
like it's constantly in a state of
emergency the whatever the current
situation is or or however the current
administration's are working there's
just this constant need to build things
quickly and make things happen and very
sort of unclear incentives it's not like
a market economy that's just working off
of a bottom line taller so we started
hitting like a lot of problems right
away even though all of us had had
experience in building these kinds of
applications we were building them at a
very much bigger scale than what we've
done before and the first problem you
hit right away is what do you sync you
have to segment the data somehow you
can't I mean we did we started by just
giving all of the data to every client
we say like you get the whole database
but then over time you need to start
segmenting on something because if it's
all gonna get stored in the users
browser it doesn't matter how much space
they have the data is growing linearly
over time we're getting hundreds of
thousands of documents of reports and
shipments so you can't just tell the
user take
everything and the way you segment data
using these databases using CouchDB is
you set up a partition per a user you
say this user is going to get these
documents and the way you get it from
the main database is you just sync them
so your rules for setting this up are
just entirely custom code your it's
entirely up to you so you have to take
into consideration all right what are
our rules we need to you know somebody
in Delta state should not be seeing the
same data as somebody in Lego state
ok cool geography next time time-based
data becomes a really big issue because
you need to sync some of it but not all
of it but then this comes into like the
domain model of what is okay to cut off
because like to calculate how much stuff
you have at a store or how much money
you have in your bank account it's
usually like what was the opening
transaction and then let's add all of
the debits and credits over time until I
get the current balance but if I only
sync your last ten transactions that's
not gonna give you the right data to get
the right and number so you look at what
to sync as a developer and you have all
of these different choices the other
thing that I'm not going to spend a lot
of time on is that with supply chain
management you have really crazy access
rules you have tons of dynamic lists of
who's allowed to see which certain
sponsored commodities at which certain
locations two years ago at this month so
there's not like a ton of yeah really
clear ways to model your your data next
is network storage and this is kind of
exciting normally in an application you
have a database that you're working with
and so your code to access the the data
is just in one place and you try and
keep it extracted maybe someday you're
going to reap lat form and you're going
to choose a different database
but typically you're working from one
type of data store if we had gone the
traditional like web framework
application and with online offline you
have data that can be coming from the
index TV browser database through pouch
you have data that could just be like in
an in-memory
excuse me cache you have requests from
the browser to get remote data that you
don't have locally and then you have
back-end service functions that are also
talking to the same database that need
to do similar business logic so
developers don't really have a framework
for where we're busy we need to throw
our code somewhere if this was rails I'd
have an endpoint I'd have a new table
and set up a new entity and I put the
code under it and even if it's messy
everybody in the organization is used to
that pattern but for us it was just like
just put it wherever you want like let's
put it in the lambda let's put it in
this back-end script let's put it you
know in the front end and then over time
you don't have like an isolated place
because you're working with different
network storage adapters the biggest one
of the bigger problems that isn't just
us on the front end doing web-based
applications is is modeling JSON
modeling sort of without a relational
database and there isn't as far as we
know so far like there isn't one clear
way to do this that always wins there's
a lot of different strategies to deal
with the problems so normalization like
you want this table and this table and
this table to be neatly separated and
you have foreign keys between between
them like this is a database that's
awesome that's what you want and so you
you're a developer and you're like I'm
going to do this in JSON so I'm going to
create a document and a document and a
document and a document and then to get
this document and to join on them you're
making HTTP requests over a network
because you don't know what the relation
is yet so so then you say oh okay that's
too much I'm gonna put everything in the
JSON document
and then a user says hey we need to
change the name of this item you say
okay let me just do a bulk update on
half a gig of data and hope that there
aren't any conflicts so and it's not
always clear what the right strategy is
for this sometimes it helps to
denormalize sometimes it helps to
normalize and document conflicts
Patricia Garcia's talk had some really
cool in-depth details about strategies
for using multiple documents to have two
users working on the same entity at the
same time without causing conflicts yeah
and then the other one is like around
what you sync to who so like kind of
access stuff you might have a list of
pharmacy commodities that you need to
send to a supplier but it turns out
there's two suppliers one for cold chain
commodities one for non cold chain
commodities so you need to split the
document so it's it's like a business
rule that impacts how you're modeling
the JSON and when a user says hey I need
to make a new type of database table you
know couch people what do I do you say
like it really depends on what you're
doing so it makes having a single
framework for everything difficult so
deciding how to segment data and what to
sync to users we created this thing we
call an ID dispenser and it's just a
remote endpoint that when a user starts
may initialize the application we have
the browser make a call to a lambda and
the lambda takes the user who it is
what their location is what programs
they have access to and it goes and it
talks to CouchDB in it says hey what
what documents does this user need couch
says ok based on your business logic its
X Y & Z lambda says thanks sends it to
the browser and then the user has a
subset of the data that they get to use
and as an aside like this is so far been
working for us but if you're going to
use like infinitely scalable serverless
functions on behalf of clients remember
that other resources of yours are not
infinitely scalable
you might DDoS your own database so yeah
this is really exciting like what we
started to do in our API is with this
problem of sometimes your network
storage adapter is a local database
sometimes it's a remote database
sometimes you're a node is having api's
that have all of that to find in them
per entity it's one API that we use in
all of our different places in lambdas
in the back end and on the front end in
scripts and your API can know what kind
of network storage adapter you're using
so you have situations where you ask hey
give me all of the reports and the API
goes okay cool I'm looking in the
database I have found three months worth
of reports and that's what I know about
that's offline and the user then says
but I want to page back further than
that this isn't fair that I should only
see what's offline and the API at that
point says I'm gonna switch network
storage adapters and I'm gonna go fetch
the remote Docs and give that to you and
then a lot of this ties back into like
how do you design this kind of
application for the user you have to
tell the user what's going on if they
look for a report on their facility some
facility of its remote maybe they don't
submit reports that often and they want
to know has this report then submitted
and they're working offline the ID
dispenser told them you should know
about this range of documents you should
have these offline so if you go and
check and the documents not there
display to the user this thing doesn't
exist I don't care about what's on the
remote I know that this document was
just never submitted in the first place
or that it's there then if you're going
too far back to a segment of the data
that we can't sync something that's much
older we tell the user and we display in
the UI like this has got to be an online
resource this is something that you're
not going to have offline
so if you have network you make the
remote call comes back says yep we don't
know about that report or yep here's the
report display it but if you're offline
you have to display to the user hey like
we don't know like we were offline we
don't know if this report exists or if
it doesn't exist so yeah the last thing
is some pictures where this is like a
remote clinic that has a pharmacy and
you can see there's like a little bit of
a VSAT which is for satellite internet
and this clinic had pretty good internet
at the time it was like 2014 and they
had like 512 kilobits a second down and
that was fine like they could have an
application on site on a laptop and
dispense point of care to patients so as
patients would come up still for the
supply chain not for the clinicians
necessarily but you would track the
movement of pharmaceuticals
electronically and so when you're
designing for this place you think well
hey we can just use a network we'll just
really make sure that that VSAT always
works well we'll talk to the IT team
will be really serious about it but
that's not the case this VSAT would very
frequently get slightly misaligned and
the clinic would lose internet and
what's really kind of the point of using
this somewhat difficult and
non-traditional application is that for
that pharmacy like it didn't matter the
application worked fine it continued to
work they could create patients they
could create dispenses they could look
up their stock they could transfer stock
internally and then oh I should show the
pharmacy the pharmacy is like that one
in the back they're all red and looked
the same but the the cool part was that
when a shipment would come once a month
from the central warehouse what would
you do you would have to manually enter
that
like you're using an offline system and
then also worse is that you have
hundreds and hundreds of transactions
daily about really important commodities
that the central team needs to plan and
think about that's why they asked us
programmers to build something so if
you're not gonna send that data back
home then it's like why are you you know
why are you tracking it in the first
place unless it's something later some
analysis or something which is what a
lot of systems end up being used for so
the team would do is they would walk to
this River and I like this photo because
it looks really nice but this River was
like really directly behind the pharmacy
it's like just less than a kilometer and
there's a cell tower over there now that
when you're at the river and you have a
cellphone and you're tethering to your
laptop you have fine connectivity and
they didn't really do this every day
they didn't need to but they could just
go over there whenever a shipment would
come they would learn from the remote
about the shipment
it would update their inventory levels
and then they would sink to the central
server and the central team these are
all the commodities we dispensed over
the last couple of weeks and the central
team could see oh hey cool they need
more of this they're good on that and we
can know what to do next
even though they're VSAT is misaligned
and the IT team is busy so yeah that's
kind of like the it's really exciting
work like we have to know a lot about
JavaScript and we have to do new things
and try and talk our own team and our
own self into trying to use patterns and
put code in the right place so it can
feel really frustrating and at times we
many times we will kind of look at each
other and be like what are we even doing
like what why don't we just make a rails
app or a django app like this is so hard
but like ultimately it means an
application that is more like robust
like it can get us more data and it can
work in the places we want to work in so
even though like a year or two ago it
was looking pretty scary
there's been a lot of learnings from us
about stuff we've done wrong that have
made it possible to kind of safely start
delivering this type of application at a
pretty large scale so I'm out of time
but thanks very much guys for your time
and it's happy to talk to you today
[Applause]
[Music]