You just built a web app.
And suddenly, it becomes an overnight success.
Now, you find yourself struggling to scale it.
We at YouTube face the same problems.
I'm Sugu, and I'm going to talk about how we solve them,
specifically about storage, which is the hardest part.
On top of this, we also open sourced
the project named Vitess.
So you can also use it for yourself.
Vitess is written in Go.
And since Go is an upcoming language,
I'll also talk about how that worked out for us.
When YouTube was first built, we started out very simple.
We had just one database.
Then our QPS started going up.
So we added replicas and sent some of our retraffic there.
That kept us going for a bit.
Then, we ran out of write capacity.
What you do now?
That's when we figured out how to shadow databases.
Once we did this, we thought we had arrived.
That's when someone came and asked, what about DR?
I was like DR, what is that?
OK.
We'll work on disaster recovery.
Once we took care of that and a few other requirements,
we ended up with something like this.
And at this point, we had created more problems
than we had solved.
With these many servers, the problems just multiplied.
The simple operations become complex and error prone,
and things start to fail all the time.
And at the same time, the number of developers went up.
And every push, we had some toxic query
that would keep taking the database down or the other.
So this is how Vitess was born.
We had three goals.
One was to automate maintenance tasks.
Two is to write a query mediator that
will protect the databases from bad elements.
And three, make it such that this big farm
appears as one single database.
So how did we architect Vitess to achieve these goals?
The first thing we did was to write a proxy server that
will mediate queries to MySQL.
We called it vttablet.
It actually does some pretty cool stuff.
I'll give you a glimpse of what it can do in a moment.
Then we built this topology cloud.
It's basically a lock server to keep track of all our servers.
It's the one link.
For open source, we used ZooKeeper,
and internally, we used Chubby.
And using this, we took control of common workflows,
like taking servers in and out of rotation,
master failover, et cetera.
And finally, we built vtgate, which
is the main server that the apps connect to.
This server makes all the databases
look like a collective whole.
Internally, what it does is to use the pathology
to figure out where to send the query, which will eventually
reach the vttablet where the actual work is done.
So what does this proxy called vttablet do that is so awesome?
So the first thing it does is connection pulling.
The cost of a MySQL connection is around two megabytes.
And imagine how much this would be with tens of thousands
of connections.
And if your app becomes popular, this
is the kind of demand you'll see.
The second thing that vttablet does
is to have its own cache, also known as the rowcache.
See, MySQL's buffer cache works well for full table scans,
because it reads data in blocks.
But this is not very efficient for random access
reads that fetch one row at a time, which
are typical of mobile and web apps.
So what vttablet does is to store such rows
in its own cache.
And any subsequent reads are first
checked with vttablet's rowcache before deciding
to send the query to MySQL.
And depending on your usage present,
the rowcache could give you an order
of magnitude increase in your serving capacity.
Vttablet has more optimizations, like for example,
if a query is executing, and other identical requests
come in, it just holds off on those requests
until the existing query has returned.
Then, it returns that result to all the queries
that are waiting on this.
And you'd be surprised at how often this happens.
There are many other safeguards that vttablet
lets you customize like blacklisting queries,
or limiting the number of rows a query can fetch.
You can see the full list on our documentation.
And finally, there's the stats module
that exports all kinds of info about what's
going on inside vttablet.
Our [? SRAs ?] just love this.
Speaking of which, how is Vitess performing at YouTube?
Well, it's been serving YouTube's MySQL traffic
since 2011.
And the first problem we solved was the growing number
of connections.
Our servers now handle tens of thousands of connections,
and we serve hundreds of thousands of QPS,
and all of this at low client side
latency of three to five milliseconds.
So in short, Vitess is high performance, scalable, storage
system.
Let's see how it's positioned compared to other products.
On one side, we have the relational databases
that give you rich functionality and data guarantees,
but they don't scale out very well.
And on the other end of the spectrum,
we have the NoSQL solutions that give you good scalability,
but they're not strong on data guarantees, et cetera.
So what Vitess does is to push MySQL
a little closer to the NoSQL datastores
by supporting sharding and replication.
And along the way we make some trade offs
on transactions and data consistencies.
And this allows us to scale just like NoSQL datastores
while still maintaining some of the cool functionality
that relational databases give us.
So this kind summarizes what Vitess is about.
I'm going to switch gears now and talk
a bit about our experience with Go.
We evaluated Go in late 2010 when
Vitess was just being conceived.
And the main reasons why we decided to use Go,
well, that one, we liked how the Go team approached problems.
While everyone else was fascinated with complexity,
we found that they were actually trying to simplify things.
And this was refreshing.
And two, we had confidence in their ability to execute.
And other than this, we kind of just like those features.
In reality, the true awesomeness of the language
is something we realized much later along the way.
So how does Go stack up against C++.
If you're looking at raw performance,
C++ will almost always win.
But then there are other considerations.
Like in my experience, once a server is deemed viable,
people want it to just do more and more.
And with language like C++, the code tends to lose
maintainability as it grows.
And even if you're a good programmer,
it becomes increasingly harder to keep it maintainable.
And it starts to take longer and longer to produce new features.
We haven't seen this happen with Go yet, and at the same time,
Go's performance is pretty close to C++.
And this makes Go a better choice than C++ for most cases.
How about Python?
Python's expressibility is amazing.
And with Python, you can do with statements, custom list
comprehensions, function decorators, and much more.
But these are great only if you want
to write like 100 or 500 line program that
performs a quick task for you.
What if you want to write hundreds of thousands
of lines of code using many engineers?
Then, you start running into a whole class of problems
you haven't thought of before.
There is this lack of formalism.
When you write a function in Python,
there's no easy way to tell what those parameters should be.
There's a testability problem.
It's very hard to prove to yourself
that a program will not blow up once it goes live.
And at the end of the day, Python
is an interpreted language.
It eats a lot of CPU.
So Go fills a unique gap for those
seeking to write high performance servers.
It's fast enough that performance is not a concern,
and at the same time it's expressible enough
that you can grow your server functionality without losing
maintainability and readability.
So this is a very brief presentation of Vitess and Go.
If you want more information, you
can follow the links provided here.
Vitess is still under active development,
but most of its core features are done.
The project is open sourced.
And you should be able to download it and give it a try.
We are behind on documentation, but you can always
contact us if you need help.
And we are very eager.
Thank you for listening and happy hacking.