TODD KERPELMAN: So Cloud Firestore
doesn't exist in a nice, pristine world where
your database requests march in and out in an orderly fashion,
like British people properly queuing at their favorite chip
shop.
Instead, it's chaos.
You've got requests coming in from all sides,
all trying to access or change the same document all at once,
sort of like Americans on Black Friday when
there are TVs on sale.
And you have to make sense of that chaos.
Luckily for you, you have a very powerful tool at your disposal
to bring order to this chaos.
And that's transactions and, to a lesser extent, its cousin,
batched writes.
So if you want to avoid weird race conditions in your code,
these are the tools you're going to want to master.
Let's find out how on today's episode.
Hey, wait a minute.
That was mine.
I was going to own--
[MUSIC PLAYING]
So let's talk about transactions.
Generally speaking, you're going to use transactions
in cases where you want to perform a bunch of operations
on the database, but you kind of need
them to be performed all at once, meaning
that I don't want the database to change out from under me
while I'm in the middle of making my changes.
And if I'm making a bunch of changes,
I don't want users being able to read
my set of documents in that partially changed state.
And I certainly don't want a set of operations failing halfway
through, leaving me in a weird and inconsistent state forever.
It's got to be all or nothing.
For example, let's say I'm changing
a value that exists in a couple of places in the database.
This is something you're going to be doing pretty
often in our de-normalized database world
where you might have redundant data in a couple of places.
Like, let's say I've got our literary app,
where I've got a collection of books
and a collection of authors.
And it just so happens that I have the author's name recorded
in a few places, both here in the author document,
but also here in each of the books.
Now, let's say we hear from Charles Dickens' literary agent
that he wants to re-brand himself a little bit-- you
know, to stay relevant.
So I could try to perform this changes,
just like three or four different
write requests in a row.
But if my app crashes or my flaky network connection
disconnects midway through making these requests,
I'll be in a state where some of my database has the old value
and some of my database has the new value.
And that could mess up my app in all sorts of insidious ways.
And heck, even if my client doesn't die,
if another client happens to read-in these documents
while I'm in the middle of making these writes,
we could end up in the same situation.
And what would happen if another client or cloud
function were also trying to change my author's
name at the same time?
If all these writes were happening one after the other,
I could end up in a situation where half the documents
contain my changes and the other half contain theirs.
Or let's take another example.
Say I've got a role-playing game.
And I'm trying to buy an item from another player.
Well, we'll need to remove gold from my inventory
and give it to the other player.
But then we'll also need to remove
her plus-3 pen of accounting or whatever
and add it to my inventory.
But what would happen if this series of events
got interrupted, for some reason?
Depending on when or how it fails,
I could end up with no gold and no pen,
or all the gold and a pen, or even weirder cases
where both of us have a copy of the pen.
And I'm sorry, but I can't respect
any MMO that doesn't obey the laws of conservation of mass.
Hey, buddy, where'd you get all those pens?
Are those really yours?
Whoa, whoa, whoa, don't write on me.
That's a plus-3 pen.
My stain remover's only plus-2.
So all these cases can be handled by transactions
to ensure there's no funny business going on here.
Now, there are two types of transactions you
can perform in the database.
The first one is a much simpler version,
known as a batched write.
Basically, a batched write lets you perform a bunch of writes.
And these can be document creations, deletions,
alterations, what-have-you, all at once in a batch.
You tell the SDK what changes you want to make.
It tells the database.
And the database will apply your changes to all the documents
that you've specified.
If any part of your batched write fails along the way,
none of it will go through.
The entire batch will be rolled back to its original state.
This is known as being atomic in the database world, which
is way less exciting than being atomic in the superhero world,
but still useful.
The point here is that you don't have
to worry about your write failing halfway through.
And you don't have to worry about somebody else's writes
getting mixed up with yours.
Sure, if there's another client trying
to make these changes at the same time
with their own batched write, they
could totally blow away your changes
when your write is done.
But this weird hybrid state where you end up
with a mix of my changes and their changes all at once--
that won't happen.
The really nice part of all this, by the way,
is that batched writes are generally easy for the database
to perform.
In fact, in many ways, a batched write
is actually more efficient than trying
to make a bunch of individual write calls.
And the implementation is pretty straightforward, too.
It's not much more different than performing
any other database write.
So when would you want to use these?
Well, basically anytime you want to change a bunch of related
documents and you don't really care
about what the old value was.
One case might be when you have your de-normalized data living
in multiple places among your database
and you want to change them all at once.
So looking back at our author and book example,
this could work.
If we know we need to change our author's name in just
these places and our new value isn't really
reliant on the old value, well, then we could send up
these four writes in one nice, little batch.
But now, what about our role-playing game
where our two players want to trade items?
It seems like what we could do here
is create a batched write where I
get a plus-3 pen of accounting and 100 less
gold and my trading partner gets 100 gold and no pen.
But here's the problem--
this batched write would only work
if I know the copy of my data is up to date.
And it might not be.
Maybe I don't have real-time listeners set up.
So I'm actually working off of data
that's, like, 10 minutes old.
Even if I do have real-time listeners set up,
there are race conditions that could
happen where the data that's on my client
isn't quite up to date with what's on the server.
For example, let's say I decide to send down this
write to perform this trade.
But in the meantime, some other event in the game
has awarded me 1,000 gold coins.
Awesome, right?
Well, it would be until my trade comes along and sets
my gold value equal to 200 coins.
And just like that, I'm out 1,000 gold pieces.
Or maybe my partner has already traded away her plus-3 pen
to somebody else.
My trade function doesn't know that yet.
So I still get the plus-3 pen.
And hey, guess what?
My players have just discovered an exploit
where they can keep creating an infinite number of plus-3 pens
in the world.
And don't think that there aren't clever players out there
who are looking for logic issues just like this
and chomping at the bit to exploit them.
So in this situation, we're going
to want to turn to a transaction.
A transaction is like a batched write,
but we tell the database that we want to read in some documents
first to make sure we're working with the most up-to-date data.
And we're going to assume that these documents aren't
going to change until the transaction is complete.
Now, the code for creating a transaction,
it looks a little strange at first.
And I encourage you to check out the documentation
for a sample or two.
But once you've written a couple of these things,
you'll get more comfortable with it, and it'll be second nature.
So the way transactions work is something like this.
First, in your transaction, your client
is going to read-in a document or two from the database.
Next, your client is probably going
to perform a small amount of logic,
based on the data you get back from that read.
Third, you will then tell the database
exactly what changes you want to make, based on this data.
Then the database will check and see,
have any of these relevant documents,
either the ones I've read or are about to write to,
changed since I started my transaction?
Now, if nothing's changed, it'll go ahead
and commit all these changes at once, which is
an almost instantaneous step.
But if something has changed, we're
going to go back to step 1.
The client is going to re-fetch those documents.
And it's going to try the entire transaction all over again.
And we repeat this process either
until the transaction finally succeeds
or we've retried it too many times
and the transaction fails.
Now, this process might seem strange to you
if you're coming from other databases
where a transaction typically means
that a portion of the database is locked,
and nobody can make any changes until that transaction is
complete.
That strategy certainly can work, too.
And in fact, our server libraries do work that way.
And I'll talk about that more in a little bit.
But when you're dealing with mobile clients that
might suddenly lose battery or drop network connections
or go into airplane mode, locking your database
is often a bad idea.
You don't want to be in a situation
where your database is locked and waiting
for some final bit of code from the client
that it's never going to get.
Instead, we don't lock anything and just retry
the entire transaction if the database sees
that something has changed.
Now, this is a strategy known as optimistic concurrency control.
It's based on the idea that most of the time, our transaction
will probably work just fine.
So let's optimize for that case.
And in the rare case where it doesn't work out,
we can go through the hassle of redoing the entire thing.
And if you think about it, this kind of makes sense.
The whole NoSQL model is based on the theory that reads happen
way more often than writes.
And in fact, given some of the limits
on how often you can write to the same document,
that is probably going to be true.
So maybe my Black Friday analogy wasn't so good.
Maybe it's more like a Black Friday sale
where copies of cassette tape players are on sale.
I'm sure you might get a few small bursts of interest
from hipsters buying them ironically,
but it's not quite the madhouse you were expecting.
Yeah.
Yeah, you should probably diversify, dude.
All right.
So let's go back to our trade example
and think about how we'd want to accomplish
this with a transaction.
I think, first off, our transaction
would request the documents representing the inventory
from both of our players.
Next, it would perform a little logic
to make sure this trade is still possible.
Do I have enough gold to make the trade?
Does my partner still have the pen?
Then we'd perform a couple of writes on our transaction,
based on the idea that the data we have here on our two players
is completely up to date.
That gets sent down to the database, which would then
do a sanity check to make sure that nothing has
changed since our first read.
And then it would go ahead and commit the changes all at once.
So those weird edge cases are avoided.
Like I said, if some Cloud Function comes in and gives me
1,000 more gold pieces before this transaction is complete,
well, hey, guess what?
Our database is going to notice that.
And it's going to retry the entire process
and make sure that extra gold doesn't get lost.
Now, there are a few additional rules and guidelines
about transactions that you should know about.
First, you have to perform all of your reads
before you can do any writes.
And on the mobile and web clients,
these have to be individual document reads, not queries.
Second, because your client might retry your transaction
multiple times, make sure there aren't any side effects
in your transaction code.
Don't increment any variables in the middle of your transaction
or show pop-up dialogs or anything like that.
Do all of that work in your completion handler.
Third, because your transaction will basically
have to rerun itself anytime any one of these documents changes,
don't go overboard with the number
of documents you're trying to fit into a single transaction.
Basically, you want to involve as many documents
into a transaction as makes sense logically.
But don't try to add in other unrelated documents
into the same transaction, thinking that you're
being super efficient.
You're not.
Fourth, transactions will fail if you're offline,
which kind of makes sense, if you think about it.
By writing a transaction, you're saying, hey,
this write is totally dependent on having
up-to-date information from the database.
And that's probably not going to be
the case when you're offline.
Finally, you can only write to 500 documents
at a time with a transaction.
And this applies both to transactions
and to batched writes.
So what are good examples of transactions?
Well, it's funny, a few weeks ago, I
would have said that anytime you want to increment or decrement
a value, you'd need to do that in a transaction,
because by definition, incrementing a value basically
requires that you have the most up-to-date data.
But the Cloud Firestore team just
recently released the ability to do exactly that--
numerically increment or decrement
a value in the database without having
to resort to a transaction.
That'll make your life a lot easier.
And I recommend checking out the documentation for more info
on these fun new functions.
So looking at our MMORPG, if I only wanted to reward a player
some gold coins for completing a quest,
well, that's something I could now possibly
do, using this fancy, new numeric add function.
On the other hand, if you need to perform any other logic
around these changes, that still might need
to be done in a transaction.
For example, if my user wants to spend some of that gold,
well, I'd need to put that into a transaction,
partly because spending gold while simultaneously adding
the item to my inventory is probably something
you'd need a transaction for, but also
because I need some logic to make
sure my user isn't spending more money than they have.
And that's not something I can do
within the numeric add function.
It will happily let me have negative gold.
And frankly, any other situation where
you want to transfer data between documents,
you'll probably want to put that into your transaction.
The example everyone always uses for a transaction
is like when you've got a banking database and you've
got a customer looking to transfer money
from one account to another.
You can kind of see why this would be
important to handle atomically.
OK.
So up until now, I have generally
been focused on transactions from the perspective of client
libraries-- mobile devices, web pages, and the like.
But you can also write transactions from the server.
And yes, this also includes Cloud Functions.
Now, in general, these server side transactions
work very similar to how the client libraries work,
but with two big differences.
First, because your typical server
doesn't go through a tunnel or go into airplane mode
or run out of batteries, we do the more traditional document
locking technique instead of this optimistic concurrency
model.
Second, we are able to run full-blown queries
during the read part of a transaction
and not just read individual documents.
And that's often useful for cases
where you need to handle de-normalized data.
So let's see if we can apply what we've learned
to my restaurant review app.
Are there any cases where I'd want
to use either a transaction or a batch write?
Well, one good example might be updating our average score
for a restaurant.
Let's say that I'm keeping track of my average score
for each restaurant by storing a running
average and a total number of reviews for that restaurant.
When somebody adds a new review, I
have a Cloud Function that would grab
this average, the total number of reviews,
and then calculate the new values based
on our incoming review.
Well, with both these operations,
it's important that, A, I have the latest and greatest data,
and, B, I don't have multiple processes updating these values
at the same time.
So this seems like a pretty clear opportunity
to use a transaction.
A second, maybe slightly contrived, example is,
let's say San Francisco, in a bold new marketing move,
officially decides to change its name to San Funcisco.
Come on, guys.
This is my greatest branding idea
since Funyuns and Funfetti.
What do you mean, "I'm a fun trick pony?"
How would I change that city name
for every restaurant listed in the database?
Well, this one's interesting.
If I were using the server SDKs, I could create a transaction
where I would perform a query for all restaurants where
city equals San Francisco.
Then I could iterate through the entire results of the query
and build out a bunch of write statements
to change the city name in each of these documents.
And then I could get everything done in one nice transaction.
But there are two big concerns here.
First, remember that transactions are limited
to 500 document writes total.
And San Francisco probably has a lot more than 500 restaurants.
So I would need to run this transaction multiple times.
And I could totally do that.
I'd just keep track of how many documents
I get back in my initial query, and I'd just kind of
repeat the transaction until this number is zero.
Now, yes, I do lose some of the benefits
of a single transaction by breaking this up
into several operations.
Specifically, there will be a small amount of time
where a client could see half of my restaurants
in San Francisco and the other half in San Funcisco.
But you know what?
I think that's OK in my specific case.
And I'll be honest, anytime you're
migrating a huge number of documents like this,
you're probably going to need to do it in smaller chunks.
You can't migrate your entire database all at once.
The second issue is that this transaction
is going to lock down a large number of documents at once.
Remember, server side transactions
go with the more traditional locking strategy.
And locking so many documents down at once might slow down
writes to my database if I'm not careful.
Now, I can help address this by adding a limit to my query.
If I were to limit my query results to, say, 50 items,
then I'd be locking down fewer documents at once.
But I would be running this transaction many more times.
And actually, therefore, it might take a few more
milliseconds to completely migrate over
all of my documents.
It's a bit of a trade-off there.
You can decide what makes most sense for you.
So there you go.
Hopefully, this gives you a better understanding
of transactions and batch writes and when
you'd want to use them, which is probably more often
than you originally thought.
So give your app another look and see
if you find any cases where you might
want to convert a normal update into a transaction.
Once you get the hang of writing a couple of these,
it certainly gets easier.
So thanks for watching.
I will see you soon on another episode of "Get to Know Cloud
Firestore."
But if you'll excuse me, that guy still has my TV,
and I'm going to get it back.
Hey.
Hey, you there.
I need that TV.
For watching TV on.
Well, well, yeah, I have one in my living room,
but I don't have one in my bathroom.
[MUSIC PLAYING]