Alexis Moussine-Pouchkine: Hi, everybody.
We're here at Devoxx 2014.
And I'm here with Martin.
MARTIN GORNER: Hello.
INTERVIEWER: Hello, Martin.
And Martin has an interesting session
here at the conference, which is called something
like "Nobody Does MapReduce Anymore at Google."
So Google kind of invented MapReduce,
and now you tell us, just a few years later,
when everybody is using it in various implementations
that this was a bad idea, in fact, right?
MARTIN GORNER: Well, the MapReduce paper
was published in 2004.
That's 10 years ago.
You can imagine that the train has been moving since then.
But actually, yes, the title of the talk
is a little bit provocative.
There is still a lot of MapReduce going on.
The reality is that it would be, I think,
a bad idea to launch MapReduce as a service
today because what we are using in-house
is much more powerful than that.
And that's actually what we will be releasing shortly.
It's called Google Cloud Dataflow.
INTERVIEWER: So Dataflow.
So this is based on another paper, which, I believe,
is called FlumeJava.
MARTIN GORNER: Yes, exactly.
INTERVIEWER: And it's a higher level of abstraction.
So maybe MapReduce is just one way
to implement whatever you express in Dataflow.
So walk us through what Dataflow is.
Maybe for the developer, it's about dealing with data, right?
MARTIN GORNER: Exactly.
What you want to do is crunch huge amounts of data.
And when you do this for a living,
you realize that even for small problems,
one MapReduce stage is usually not enough.
You want to have at least two of them or maybe five of them in--
INTERVIEWER: The pipeline of MapReduces.
MARTIN GORNER: In a different graph.
Not just a pipeline.
It could be a graph.
INTERVIEWER: A graph, yes.
MARTIN GORNER: But two MapReduces,
merging the results, going into a third one,
and then doing something else with the data again.
And this, if you try to implement it using MapReduce,
you end up with a lot of plumbing
because you need to handle all those intermediate results.
You need to actually deploy your MapReduce clusters,
and all that.
So there is a better way.
And the way we are suggesting to developers to do it
today is to use a very small number
of high-level primitives-- in Java, for example--
and define the computation using those primitives.
Those primitives work on parallel collections
and are executed lazily, which means that the code you write
is simply used by the system to infer an execution
graph in memory.
Once we have that graph, we can optimize it,
and we can deploy the optimal cluster topology
for actually running your computation.
Deploy it, run it, tear it down.
Of course, you still get all the benefits
of MapReduce, which means it's a parallel execution framework.
The life cycle of the nodes is handled for you.
Catastrophic events, like a node dying unexpectedly or not
responding.
That's handled for you.
You really only focus on the math, the computation
you want to do on your data.
INTERVIEWER: So walk us through, briefly,
what the API looks like as a developer.
How do I develop and use this?
What is my input, output, operations?
What does the API look like?
MARTIN GORNER: So the basic piece of data you act on
is a P-collection.
That's a parallel collection.
It's an immutable collection of data.
And then you use primitives to work on it.
The most basic primitive is parallelDO,
a parallel operation, a function you provide.
That's the equivalent of Map or Reduce
in the MapReduce paradigm.
And then the second important primitive is a group by key.
So normally, when you do a parallelDO,
you do computations on your data,
and you also assign keys to your data.
So the output is key value pairs.
And then, again in the MapReduce paradigm,
you have a group by key, which is called a shuffle
stage, which will put all the data with the same key
together and apply another, usually an aggregation
on those data.
So parallel collections, parallelDO group by key.
And then a third primitive, which
is an associative parallelDO that has a lot of good stuff
that can happen when your parallelDO is associated.
So with those three primitives, it's
enough to describe most of the computations you want to do.
And then FlumeJava also adds primitives-- well,
not primitives, but functions, helper functions,
which are already implemented on top of those,
like counting or adjoin, or a top operation.
INTERVIEWER: OK.
So you mentioned FlumeJava.
That's the paper that was published around 2010, I think.
MARTIN GORNER: Yes.
INTERVIEWER: It's based on experience
Google has running this type of technology in production.
But what we're offering here is a product called Dataflow.
And that's a service we're offering
because you want to focus, as a developer, on that level of API
you just described and leave to Google everything else, which
is setting up the environment, the network, the nodes,
and spinning out the resources in the most
efficient and fastest way to provide a result back.
MARTIN GORNER: Exactly.
And the beauty of it is that if you have a complex computation,
which actually results in a fairly complex network
topology for your processing cluster,
if there is something you need to change,
it's a couple of lines in your code.
And you hit Enter, and that cluster
gets redeployed with a different topology.
That's really how we want to manage a compute cluster.
You don't want to be there changing the nodes
and rewiring them because you need to compute something
slightly differently.
INTERVIEWER: Yeah.
So the other thing is the data you work on,
is this like read-only data?
You have a batch of data?
Or can you have some sort of streaming?
Or is there anything--
MARTIN GORNER: Well, that's the second, I would say,
magic strong point of Dataflow is that once
you have your computation described
in using the high-level primitives,
to turn this into a streaming computation,
what you do is that you change your inputs and outputs.
Instead of reading from a file, you read from a stream.
INTERVIEWER: OK.
MARTIN GORNER: You add one line of code,
which is please now work with this time window.
And that's it.
This cluster becomes a real-time data processing cluster
that does everything the batch processing cluster was doing.
There's still node life cycle, nodes dying, and all that.
All this is handled for you.
But now it can also automatically
scale with the volume of data.
And if you think about it inside,
it uses parallel sets of nodes for executing your parallelDOs.
It can figure out how many nodes to put in parallel
to actually sustain the flow.
And those parallel processing groups of nodes,
they're all actually put together
in a complex execution graph.
But you don't have to worry about that.
There's an optimizer that worries about that.
And there is an executor that worries
about putting the whole thing on autopilot.
INTERVIEWER: I like all those things
I don't have to worry about.
So when can I, when can folks start using this?
This is Dataflow, that's a service, a product,
Google Cloud is offering soon?
MARTIN GORNER: So we've been using it
for many years at Google, which means that it's not
a completely non-baked technology.
And in the session today, you will see a live demo.
INTERVIEWER: OK.
MARTIN GORNER: So that's the status.
INTERVIEWER: So, yeah.
That's a good plug for your session,
for people to watch this on [INAUDIBLE]
as soon as it's published.
MARTIN GORNER: Indeed.
INTERVIEWER: So, Martin.
I think that's all the time we have for it.
So thanks for doing this, and good luck with the session.
And folks, keep an eye on Cloud Dataflow.
MARTIN GORNER: Thank you.
INTERVIEWER: Thanks, everyone.