(upbeat music)
- Hello and welcome.
My name is Seth, and I’m an engineer at Google.
You can find me on the internet at @Seth Vargo.
In this session, you'll learn about the exposure notification system
and how we built scalable backend systems
that serve millions of users with minimal operations overhead.
Let's jump in.
If you're not familiar, EN is a joint effort by Google and Apple
to help governments and global communities
fight the pandemic through digital contact tracing
using iOS and Android mobile devices.
When mobile devices are in close proximity,
they exchange random rotating identifiers via Bluetooth.
When a user tests positive for COVID,
they can opt to upload those identifiers to a server.
Other mobile devices download those identifiers,
often called keys, from the server, and perform local matching.
Such a match generates an exposure notification.
This design means that the server never knows which keys
or how many matches have occurred.
As you can see, the client, iOS and Android devices,
have a lot of responsibility here.
But this talk is going to focus specifically on the server bits.
Now you might be thinking to yourself, “Well, this sounds easy.
The client is doing all of the work. I go get tested and I upload my keys.”
But, this has a problem.
There's no requirement that I actually go and get tested.
There's nothing that stops someone from uploading random keys to the system.
So, there’s actually two main server components in the system.
And each of those server components have
their own subsystem of services and background jobs.
When a patient tests positive for COVID, a clinician or a caseworker authenticates
to a verification server and generates a time-bounded code.
The clinician shares this code with the patient
and the patient enters this code into their exposure notifications app.
Let's take a concrete example.
Suppose you go to the doctor and you get tested for COVID.
If that result comes back positive, your doctor's office may call you.
They would give you a code.
You would put that code into your local health authority
exposure notifications application.
When you do that and upload that key to the verification server,
that verification server validates the code
and you get back a longer lived token.
Now, the patient would never actually see this token
because most of this interaction is handled by the app or the operating system.
The token is then used for another request to the verification server
along with HMAC of the on-device exposure keys.
This is used to generate a certificate.
The mobile device or operating system uploads the temporary exposure keys
and the certificate to a key server.
The key server first validates that that certificate is valid
before accepting those keys into the system and later distributing them to the CDN.
This validation leverages asymmetric cryptography.
So there's also a background synchronization process
between the key and verification servers.
But again, this happens in the background and is not in the main request path.
At their cores, the key server is responsible
for receiving keys and exporting them to a CDN.
And the verification server is responsible for issuing and verifying codes.
But, like many software systems the servers also have a broad surface
of auxiliary functionality, some of which has evolved over time.
Let's take a minute and zoom in on just the key server for a moment.
In addition to facilitating the uploading and downloading of temporary exposure keys,
the system is responsible for talking to multiple upstream services and APIs.
We made careful and intentional choices to pick Google Cloud services
that minimize our operational overhead.
For example, Cloud Scheduler, Cloud KMS, Cloud SQL,
Secret Manager, and cloud storage
And most of them have an SLA of 99.9% or higher.
Our network is fully managed by Google Cloud and our compute layer is Cloud Run.
Cloud Run is a serverless platform.
We’re only billed for the exact resources that are consumed
and each service can be scaled independently from its peers.
We can set a minimum number of instances per service
to mitigate cold boots.
And we can set a maximum number of instances per service
to help control costs.
As a concrete example, the exposure notification service,
the service where COVID-positive humans
upload their keys, can scale independently of the other services.
This architecture has enabled us to have a relatively small operations team
while still maintaining reliability
to service hundreds of millions of users every day.
We also depend on other fully managed Google Cloud services
including the Cloud Operations Suite,
which handles all of our logging, monitoring, alerting
and tracings so that if something does go wrong,
we get notified.
The verification server’s architecture is very similar
using Cloud Run as the compute layer.
There are however, a few additional things worth mentioning.
First, the system makes significantly more use of Google Cloud KMS,
sometimes in the path of requests.
Second, the system makes use of client side and memory caching
as well as a centralized caching mechanism
using Cloud memory store running in Redis mode.
Finally, since clinicians need to authenticate
to the verification server,
this system uses Google Cloud identity platform
to manage username and password logins and to enforce multi-factor authentication.
We have the opportunity to have a completely greenfield project.
And in doing so, we were able to do a couple of fun things
that are worth pointing out.
The first is that we were able to implement a full browser testing framework
for our user interface components
with no outside dependencies, besides Google Chrome.
We call these tests clicky-clicky tests
because we leverage Chrome DP to spin up an in-memory headless, Google Chrome instance
and actually click buttons, execute JavaScript, and verify styles.
This is only important for verification server.
That’s the only thing that actually has browser interface.
But nonetheless, it's great to have no outside dependencies
besides Google Chrome and our programming language.
Second, this system is actually fully open source.
The source for both the key and verification servers
is available on GitHub.
And finally, this system is heavily optimized for release time and release cadence.
This is largely because, at the early stages of the project,
guidance around COVID and risk scoring and modeling
was changing on an hourly basis.
We needed to be able to push changes very quickly
so we could adapt to this ever-changing environment.
And as with any project, we weren't exactly perfect.
Even if it was greenfield and we had no dependencies
we obviously were under quite the timeline crunch
as COVID spread rapidly.
Furthermore, our architecture isn't simple.
In fact, our architecture is quite complex.
Even though we chose serverless technologies
and we optimized the system to be as operationally simple as possible
we still needed to think through the design of the system.
We were very fortunate on this project
that myself and others had experience building
and running serverless systems at scale.
But it’s very easy to mis-design a system.
One thing we weren't quite prepared for was the client support.
A mobile application or operating system is essentially on-premise software.
And we have no control over when, if ever, the end user updates.
This leads to an ever-growing compatibility matrix.
However, unlike other kinds of applications,
dropping support for an older app version
or operating system could put people's lives at risk
for the exposure notifications project.
And finally, all of our background processing is stateless.
Since this background processing happens at our compute layer on Cloud Run,
we only pay for the few seconds that a job executes.
If it doesn’t finish in time, the next invocation picks up where it left off.
In practice, this means we can bin-pack many background jobs into the same space
which dramatically reduces costs compared to, say, a long-running Kafka cluster.
Thank you all so much for joining.
If you’re wondering how we measure the effectiveness
of the system while still preserving individual privacy,
you’ll wanna check out my colleague, Robert Kubitza’s, talk
on exposure notifications, privacy preserving analytics
or MPAA for short.
You can check it out and the Google Cloud Dome.
Thank you so much.
(upbeat music)