Anything else?
Lock contention is a big thing for us right now.
As I was saying with this Cassandra stuff--
Whenever you vote on a link in a popular subreddit,
it has to lock on that subreddit's listing.
An example of that biting us recently--
we have these queue processors,
and all they do is they
get that queue item that says that
somebody's cast a vote.
So you basically have a set of machines that are reading from this. >>Right.
So all these apps are writing what happened to the queue? >>Right.
Then you've got a bunch of machines
that are reading from the queue.
And what they do is, they sit there.
They pull that out, and then they say, okay, that means I have to update these listings,
and I have to record the vote in postscribes and cassandra.
And they go through all of this stuff,
and that involves a lot of the locking in here right now.
And so we had a lot of these queue processors up for votes,
and we get a lot of votes simultaneously,
so we need a lot of them,
but we had too many, it turned out,
and they were all fighting each other for those locks.
Just having the number of those queue processors
actually sped up queue processing in general.
You need enough queue processors to actually handle the depth of the queue,
but if you have too many,
they spend too much time fighting each other.
And one of the ways we're working on that
is we're getting rid of the locking in the cassandra stuff.
And we're trying to get rid of locks as much as possible in general.
Locking--
It's a common theme.
In python itself
when we first switched to Amazon we had this weird issue where
Python's multi-threading--running two threads at once,
running two pieces of your program at the same time,
is not state of the art--
would be a nice way of putting it.
Python was spending so much time locking its data structures
so two threads could access the same data structure at the same time
that it was actually slowing down the
computer's ability read traffic over the network,
which was causing it to spend more time
switching between threads.
It created this weird network/CPU thrashing issue.
Where the way we solved that at the time was we just made python single threaded,
and we'd only handle one request at the same time.
I think you guys are probably still doing that.
Yeah, we very rarely use threaded processes.
The ad servers use threads, but other than that
we just have lots of separate processes.
You guys use lots of processes on 1 machine,
and the OS then can do the task switching for you.
The OS is Linux; these days, is pretty good at it.
And they spend a lot of their time waiting
for something in the back end.
Well, this is really great. Thank you so much for coming.
One thing I'd like to point out is that
everything we've talked about here, the main things--
memcache, Zookeeper, Cassandra, Hadoop, AMQP, NGINX, HAProxy--
This is all free software.
It's pretty wild how far you can get without paying for anything.
The things that you have to pay for are
the computers to run this stuff, but all of the software
and all of the code behind all of the software is online and free.
Also, the vast majority of reddit's code is also open source and online,
so if you wanna look at this stuff--
What's the URL for that? >>Github.com/reddit.
I'm gonna make a hole right in the middle of our picture here.
If you go into this code,
we switched to git at some point,
and my name isn't on a lot of this code anymore,
but if you go in there, you will see
a lot of the code we've written in this class for
hashing and passwords and all of that stuff--
It exists in reddit somewhere.
It's really common stuff.
It's cool, and you can see all of this stuff's also on the reddit mailing list,
where people discuss these architecture changes and that sort of thing.
Thanks again. Good job.
Watching you guys grow Reddit has been really cool.
There were some dark days, and you guys have really done an amazing job growing that site.
It's really impressive. >>Well, it wouldn't be where it is without you.
All right guys, thanks very much for watching, and we'll see you in the next one.