We have a really fun question today from Blind Five Year Old
who asks, "Can you provide some insight into how Google uses
Percolator, Dremel, and Pregel?"
All right, so what are Percolator, Dremel, and Pregel?
These are completely different tools,
and I'm doing the mental translation from internal code
names to external code names.
So I'll try to keep them straight.
Percolator is basically a layer on top of BigTable.
Indexing in the old days used to be
done in what we'd call batch mode.
And so it's like you wait for the next train,
and the trains run every hour or every day,
or something like that.
So you have to wait.
And then all the people get on the train,
and they all head out.
And they're good to go.
Caffeine and Percolator-- you can
see the connection between a percolator and caffeine--
takes that notion of batch indexing
and moves it to incremental.
So it's more like as passengers arrive, they can get in a taxi,
and they can leave immediately.
So instead of waiting for 1,000 people
to get ready to leave once a day on the train,
people can leave whenever they're
ready by just getting in the taxi.
So that's the notion of moving from batch indexing
to an incremental index.
So Caffeine was the overall system.
That's the external name for it.
And Percolator is a part of that,
that helps make sure that that whole system runs and runs
well.
Dremel-- I'll use a metaphor here a little bit.
You know how you might use a MySQL database,
and you say Select, Start, from whatever.
And it works for pretty good sized databases.
Dremel is a lot like MySQL, except for huge databases.
So there's some architectural differences.
But on a practical level, what Dremel lets people at Google
do is take databases on the size of the web
and do very fast queries over them.
And so all sorts of people use them
to do all sorts of analysis.
Even the spam fighters use them.
It's a fantastic tool.
And there are papers about many of these projects,
and then there are often open-source implementations
of some of these as well.
But Dremel lets you do really great, fast data analysis
extraction over very large databases,
and lets you play with them in basically an interactive mode.
Pregel is basically a system to deal with graph problems.
So there was an interesting blog post in 2009
on the research blog post that basically
said once you have all this graph stuff done as a library
or as a set of code and machines to implement them,
they said that they could do page
rank in about 15 lines of code.
So those are 15 very expensive lines of code,
but you have to get the system right and architected
well so that you can handle those kinds of graph problems.
So you can imagine we might have somebody new come in and want
to come up with a better version of page rank,
or come up with a completely different way of trying
to compute reputation over links,
or any other type of graph problem.
Maybe you've got some connections
between people or something like that.
So you can imagine wanting to run that computation very fast,
and Pregel is a system that let's you do that.
One of the fun things about being at Google
is there is a lot of great infrastructure
and a lot of great machinery to help
you do really fast processing, experimentation,
and then once you find something that's
a good signal for search quality or some way
to improve how we crawl the web or something like that,
we're able to deploy that on top of our architecture
much more quickly.
I guess the downside is sometimes
it's a little harder to take something and open source it,
because it's got vast chunks of Google proprietary stuff.
But you really do get to stand on the shoulders of giants
and save yourself a lot of time and work
by not having to implement everything yourself.
There's a lot of great machinery here at Google
that let's do that instead.
So thanks for the question.
It's an interesting one, and hope that helps.