(funky music)
- Welcome to the Azure Cosmos DB Essential
series season two.
My name is Mustafa.
I'm a product marketing manager for Cosmos DB.
And in this episode we're going to cover a little bit
about data modeling.
And when we look at data modeling specifically
we're gonna look at schema design.
So before we dive into schema design,
let's define schema just to give everyone a refresher.
Schema is the structure of how you're going to
arrange different properties or columns within an item.
And that item could be a document, a vertex,
or even a row for all intents and purposes
if you're in a relational setting.
These are all different ways of
combining properties or columns.
And what you're going to see when you look at Cosmos DB
or other NoSQL databases is that we typically represent
relationships slightly different
than in a relational context.
In a relational database we have normalized tables
and we draw relationships across them.
In NoSQL we do not.
So let's dive into an example right away.
For this example we're going to imagine that we are building
a database for an insurance claim system.
When you normalize the data model,
what you typically end up doing is
you break up things by entity.
So you might have an entity called claims.
You might also have a set of entities
called the claim line items.
And what happens is as a patient files a claim
that claim might have a bunch of different little
line items under it.
So using the example, let's just say I tore my ACL.
And as part of tearing the ACL,
I had to go get both an MRI to get the ACL examined
as well as get ACL reconstruction surgery.
Now when we go and model that in a relational database,
you're typically normalizing where you have a claim
that might have metadata.
So when we file my claim into the insurance,
we'll have diagnosis, insurance, insurer, payer, provider,
all of that metadata may be stored in a claim object,
or even separate in their own independent tables.
To simplify, the MRI, the reconstruction surgery, et cetera,
would be rows in a claim line items table.
And then these of course get joined together
from a foreign key, likely the claim ID.
So what's the problem with normalization in this context?
There's a subtle thing that's happening here,
which is by normalizing we are incidentally
optimizing for storage,
and there's a hidden penalty for compute.
There's no problem until we scale.
So from an application point of view,
a lot of times if we were to build an application with,
let's say a rest API,
as you post these pieces of information into the API,
what you're generally doing is on every write
you're shredding the data.
So we have to shred that claim and then write multiple
entries under the line items table.
And then every time I read that data,
we have to reassemble everything.
So we're gonna go, we're gonna look at the disk,
grab the claim.
We're gonna go look for the claim line items.
And then we're gonna do a cross product
in the form of a join
to join these two pieces of data back together.
The challenge with joins is they don't work very well
in a transactional scenario that scales out.
And so whenever you do a joint across multiple nodes,
doing the cross-product across a node boundary
in a sharded architecture tends to have
very heavy performance penalties,
both in terms of throughput and latency,
because the join requires a high degree
of node to node coordination.
And this is exacerbated by the more nodes we have.
So instead of joining across the two short database shards
or partitions imagine that if I had a hundred
database partitions doing joins,
it's gonna be pretty prohibitive to do that many joins
across those partitions.
So in summary at the micro level,
normalization is just fine.
But once our data starts to get larger and larger,
we start to run into heavy compute challenges
working across partitions.
So what happens if we denormalize?
A common technique to denormalize is to effectively
take objects and embed them into another object.
And so this is where you take advantage of the type system
where the type system supports nested objects.
A less optimal way to denormalize is to nest
all of the line items under a single claim document.
Now in this case,
what I have here is some sample Jason we're going to use
the document or your data model, just as an example.
And what you have here is we have a bunch of claim
information for my ACL tear,
which we can go and query for by let's say the care team.
We can look at all of the claims that were filed
for between
a certain date or for a certain doctor.
And alternatively, if we want to we could also filter on
any other field in the metadata.
But zoom out and we'll see the challenge here.
When you denormalize many items under a parent document
or a parent object,
what would happen if the number of line items
is exceedingly large?
Now imagine that instead of having just two line items,
we have let's say many line items,
maybe a hundred thousand.
In such a case you might end up running against boundaries
for what is the largest record size
that can be in your database.
So there's actually another approach for denormalizing.
Typically it's pretty intuitive for people to do
what we just did, embed children into the parent entities,
but what usually yields a better design
is to actually do the reverse.
Instead of embedding children into the parent record,
embed and copy the parent record into the child records.
So with this third option in this case what we're gonna do
is we're gonna take a document line item.
And here I have a document for
the ACL reconstruction surgery,
as well as the document for the MRI.
And because we frequently query off of certain fields
in these documents what we're gonna do here
is we're actually gonna copy the parent object
and duplicate it across these other documents.
Now effectively, there's gonna be some trade-off.
Engineering is all about navigating the right trade-offs.
By duplicating information I'm now taking a hit on storage.
I'm having to store the claim metadata more than once.
In fact, I'm gonna have to store it based on
a multiplier of how many line items I have.
But the advantage to this is for my example workload,
if I'm frequently querying for a set of line items
one by one over time,
I'm adding a line item and adding onto this
as the patient makes progress through their treatment plan.
Now what you'll find is that writes can be made very cheap
because rather than having to append an object into this
parent array and having to rewrite this entire document,
we can go and do tiny much smaller updates.
So in summary option one is normalization
and the trade-off is between compute and storage.
Option two is denormalization by nesting a child
in a parent object,
which becomes a trade-off between storage
and write throughput.
Option three becomes a sacrifice on storage
at the benefit of faster reads and writes.
And to take it a step further with option three,
we can selectively query a set of line items
based off of whatever filter conditions we decide.
This is how to effectively optimize
for compute over storage,
because compute is honestly the bulk of the costs
and storage is and has continued to be more
and more commoditized over time.
Now, a lot of relational users might be hearing this
and thinking, well, couldn't we just normalize the data
in Cosmos DB and then do a follow-up query to get around
this whole issue of denormalization?
The answer to that question is absolutely yes.
As we do that, one of the high level themes you're going to
find here is that when you normalize data,
what you're effectively doing is optimizing for writes.
The big drawback for duplicating parents is if I let's say
as an example frequently changed the insurer
or whoever the provider is on my patient claims,
well I get that's unlikely.
Let's stick with me here for the hypothetical.
If I'm designing a data model that does frequently update
metadata in the parent,
what ends up happening is you have to update that parent
object multiple times for each duplicated parent.
And so one of the things you need to look at here is
normalizing has some benefits
if you're optimizing for updates,
because the lack of duplicates means it's easier to do so
by avoiding the fan out update.
Now because we are not doing cross products
across the database by denormalizing,
you can grab all the information within the scope
of a single query,
making updates very cheap as a counterpoint.
So what is the impact of normalization, denormalization,
on how developers want to think about their data model
in Cosmos DB?
Number one is of course to keep in mind that the goal
when using NoSQL is to optimize for compute over storage.
Compute is going to be the dominant cost
in any modern cloud system.
So it's honestly a much smarter thing
to optimize around compute than storage.
Optimizing for compute means you don't need to understand
the data access patterns.
Rather than starting with a set of entities
and then thinking about the relationships,
what you're really going to do is you're going to think
about how data typically gets read and written.
When working with flexible schema,
as you remember if you tuned into the last episode
on partition strategy,
the first thing we're always going to recommend you do
is figure out your ratio of reads to writes.
If you have something that performs let's say 10 times
as many reads over writes,
you likely want a bias towards optimizing for the reads.
And likewise, if you have the reverse,
then it makes sense to optimize for the writes.
Optimize for the aggregate workload.
But of course if you're going to have to make trade offs,
don't sacrifice the performance or cost
of let's say 90% by optimizing for the 10%.
If the opposite is true and you have more writes than reads,
the main thing you want to pay attention to of course
is update patterns.
A tiny insert is going to be less expensive
than a much more expensive update.
And that's something you should always keep in mind.
Okay, so we've gone through schema design
around normalizing and denormalizing.
We've talked about rigid and flexible schema.
Before we dive into actually getting data into Cosmos DB,
let's go through a couple of tips for getting the hang of
this concept of database design that comes with NoSQL.
But the first tip we give is that you want to avoid doing
follow-up reads or having a break up a query
into multiple database reads.
So if there are filter conditions
you continuously query off of,
those a lot of the times can be candidates
for duplication across your documents.
This way rather than having to look up an insurer,
and then find many line items for a given insurer,
duplicate that insurer in each document
so you don't have to look everywhere.
Data that's written together often belongs together.
Data that's not rendering together
should be separated out into separate records.
That way when growing in a single giant record,
the basic building block or unit grows.
Another tip we typically give is to segment data
by frequency of reads as well as mutability.
And so what we mean here is
how often the document mutates or changes.
And what you'll find is there are certain properties
that change with high frequency
and certain properties that do not change
with high frequency.
An example for this would be,
let's say a different use case
which is building our product backlog.
When I look at a product catalog,
the quantity you have an inventory is going to change
much more than the item's title, picture, or description.
And so by separating these things out what you can do is
actually optimize for both reads and writes
by having a data model that aggregates
a lot of the frequently read properties
that are read together.
Now the third tip is it's much cheaper
and easier to think about the first two tips upfront
than optimize this after you've built your application.
Once you settled on a data model,
your objects will often look the same
throughout the application, as well as the business logic.
So it's easier to plan out what these things
will look like in advance.
Alright, so that concludes the fourth episode
about schema design, normalization and denormalization,
in Cosmos DB.
I hope you enjoyed going through some of the day one things
to think about and tune into the next episode
where we'll talk about getting data into your database
with Azure Data Factory.
(funky music)