You replace memcachedb with Cassandra for the precompute stuff?
That solved the right contention issue.
At that point,
Every listing has a lock --you have to lock around.
So you have this listing, which is a set of IDs
of all of the links in a subreddit, right?
And you have to modify that.
You have to lock around that item,
and that's at the subreddit level.
If a lot of people are hanging out in one subreddit,
that thing will have a lot of locking going on.
We talked a little bit just in the office hours I think for Unit 4,
somebody had asked about
in datastore, if two users tried to register the same username at the same time,
and datastore doesn't enforce any uniqueness constraints
on a field in the database. How do you manage that?
And my answer was
you can either use transactions in the datastore, which I don't fully understand,
because I'm new to a datastore, or you can use memcache,
which is one of the 101 uses of memcache,
it has this global lock.
So you guys are moving away from that? >>Yes.
When you use memcache as a locking service,
the problem is
if you lose a single memcache node,
then you lose the site,
because you can't throw away that node
without potentially,
say half the apps can't see a memcache node,
and the others can.
So the ones that don't see it decide that they're not
gonna talk to that guy anymore, and they try
to lock on a different set of servers.
When we store data on multiple nodes of Cassandra.
It's different from how we store data
across different nodes of memcache,
which is--correct me if I'm wrong--when you distribute across memcache,
you basically hash your key to a particular memcache box,
and there's no notion of replication.
Yeah, it's similar in Cassandra.
This node has a key space,
and it just happens to go +1 and -1.
And you can do the same. You can do replication like that in memcache, as well.
Now, were you guys using--
when I left, we were using a naive memcache library
that basically took a key and said,
which box is this hashed to, and would store it on that box.
If you lost that box, it would effect the hashing of every other key.
Losing a memcache box was really painful.
So we'd replicate memcaches for mostly space.
We had more space--things wouldn't expire out of the cache fast enough
or would expire out of the cache too fast.
But that had the big balance setup that
we lost a single memcache box
all of the keys would get rearranged.
Right. Module hashing.
We are using consistent hashing now.
The way that works is
it basically builds up this ring similar to this,
and instead of mapping
keys 1 through 1,000 to this box,
it actually sort of assigns them to a place on a circle
and finds the nearest server on that circle.
When this node fails, it will just go to the next one.
We have 10 nodes, we lose 1.
Only 1/10th of your keys get redistributed.
If you're using consistent hashing,
a key might hash to this space on a circle.
And if you lose this guy, the key still goes in that space,
but all of the keys over here stay in the same spot.
When you're using modular hashing--
remember we talked about modular in unit 6--
that's a really naive way to distribute keys.
But all of a sudden you go from modular 10, because you have 10 servers, to modular 9.
Instead of losing 1/10th of your keys,
you lose 9/10th of your keys; that's a big problem.
We actually had this happen a couple of times while I was there.
You lose one memcache box,
or more likely, you misconfigure it,
and the app service doesn't see it anymore--
All of the sudden, your cache is not warm anymore,
and you've got users who are just like
hitting the databases, pummeling the hard cache or the memchache.
We call it the hard cache.
And all of a sudden, your load profile completely changes, and everything explodes.
That was a really nice improvement.