Welcome to 'Back to Basics.' I'm Simon Carter,
and I'm here today to talk about reference data
and how to manage it centrally without introducing performance headaches.
So firstly, let's talk about what we mean by reference data.
Reference data is non-user data that comes from outside the application,
and is generally considered part of the configuration of the application.
Examples include countries, state names and abbreviations,
IDD dialing codes, public holidays, supported currencies, service codes, and types.
It can also include the content of UI drop down lists,
such as the various delivery options for postage.
It's important for reference data requests to have very low latency, but also low cost.
Your application looks at reference data for most interactions,
such as when displaying web pages, and validating transactions.
The example we're going to look at today is validating IDD codes,
so when the user enters their country, you can default the right IDD code for their phone number.
As reference data tends to be slowly changing,
it can be tempting to hard code it wherever it's needed.
This approach is super fast, but it costs more for
multiple copies of the data have to be stored in each container or virtual machine.
In large applications, I've actually seen reference data grow to over 100 megabytes,
so this is not a good approach.
It's also hard to maintain, when reference data changes, it requires a code deployment
and all code deployments need to change at exactly the same time to avoid inconsistent behavior.
I really don't recommend this pattern.
The simple approach is to store reference data in one place, the back end database.
As reference data volume grows, it needs ever-increasing cash
to provide a low latency response time.
It's tempting to be able to rely on the back end database,
scaling memory and CPU whenever you need to.
But you really end up spending way more than you have to with this approach.
You could add a database read replica and segregate reads from writes.
But this doubles the cost of your data tier without really addressing the underlying problem.
So while reference data does change slowly, such as when countries split or join,
it also needs to change in response to operational events.
Imagine your delivery partner is suddenly unable to deliver,
and you need to change the UI drop down options for delivery for parcels.
So we have these core reasons of how to manage our reference data.
We want to maintain it centrally, we want to segregate reference data from transactional data.
And we don't wanna hard code it.
Another key reason for that is change control processes in enterprise,
are very time-consuming for code changes, but very lightweight for reference data changes.
The best way in AWS to manage a reference data store
is with Amazon ElasticCache for Redis.
ElasticCache is a fully managed, scalable, in memory data store with extreme performance.
It provides most read or write operations in less than a millisecond,
and support hundreds of thousands of requests per second from a single instance,
and hundreds of millions of operations per second within a cluster.
So while ElasticCache is often used as a look aside cache in front of a database,
in this example, we're going to use ElasticCache as the database.
ElasticCache for Redis allows us to use arrays, lists and sets,
which are perfect for lookups, UI elements, codes and other simple data.
Now let's look at an end-to-end architecture that leverages ElasticCache for Redis.
In this approach, when a microservice needs reference data,
it typically fetches it directly from ElasticCache.
Deployed in a highly available configuration, it automatically recovers from the loss of a node.
Request for non-reference data continues to be served by the back end database.
If you're after performance, you'll still want to segregate reads from writes with a replica.
Although this architecture is super simple,
it shows how you can achieve impressive performance just by segregating different data types.
Remember, ElasticCache is suited to all kinds of fast moving data
such as user sessions, leaderboards, and geolocations
and data that needs to expire, such as credit card data.
Okay, here's my hot tips.
For critical workloads, you want to deploy ElasticCache in cluster mode,
so that you can recover in the event of a node failure.
Also, if you're unsure how much storage space you need,
make sure you provision 50% headroom,
so you can always resize on the fly without interruptions.
For monitoring, you want to make sure you get alerts
if CPU goes above 90%, or if freeable memory drops below 100 megabytes.
Look out for any usage of swap because that will dramatically lower your performance.
Look out for cache misses.
That means your application is looking for reference data that doesn't exist.
And make sure that cache evictions are always zero. You don't want things being removed from your cache.
Depending on your process for loading reference data,
you'll either want hourly backups, or deployment process that populates the data.
Amazon ElasticCache for Redis allows you to segregate reference data from transactional data,
and achieve both performance and scale with cloud architectures.
Check out the links below for more details, and see you next time.