Now we're going to talk about a pretty cool new
tool. Something that goes between the map and the reduce
phase, called the combiner. Before we do that, let's review
how you may have calculated the mean in the last
problem. And I don't know exactly what you did,
but one thing you may have done is, something like
this. First, you probably had your mapper go through the
data. I should say your mappers, go through the data,
and output, key value pairs that looked
something like, day of week, amount spent. Then,
for each day of the week, your reducer probably kept a total of the sum and
a count. So actually maybe your value here,
was the amount of money spent. And then
also a one, to help increment the count.
And then you probably would divide the sum
by the count, and it looks pretty good. But
remember, net produce is happening on this whole network
of data nodes. Each of these little boxes, we'll
pretend is a data node. And your data's spread
out across all of these different machines. And now
let's think about where each of these three steps
took place. So, we're not going to focus on what
happened but where it happened. Well, for step one,
the mappers were all over the place. There were
mappers, anywhere there is relevant data. But, for any
given day, all of the reduction had to happen
on a single machine, where that reducer was living.
And actually, that's enough, we've already exposed a problem.
Because if there's a lot of data, even if
each individual output was simply a day and a
number, depending on the number of records we have,
transferring all of this, all this data to
this machine, with reducer lift, that, that's a lot
of bandwidth. That's going to be a lot of traffic
on our network that we don't necessarily want, and
it doesn't actually seem like we need to have.
So what if we could do some of this
reduction locally? What if on each machine, before we
send our key-value pairs off, off to be reduced,
what if we could do some pre-reduction? And it
turns out we can. And that pre-reduction happens with these
things called combiners. And before you practice using combiners
let me try to sell you a bit more on
their value. So as you know, when you use
a map reduce job you get some output like this.
This is where it's displaying how much mapping has happened
and how much reducing. But you get this tracking URL.
And if you open that URL in a browser you get a job page, and that job page
looks something like this. And actually if you scroll
down below here you'll see some more interesting statistics.
And in fact we ran two jobs. They were
identical jobs, except one implemented combiners, so they were
reducing locally as much as possible, before shuttling that
data off to that machine that held the end reducer.
Let's compare the statistics from those two jobs. The
one with combiners, the one without. So let's compare what
happened on these job pages. The one without the combiner
versus the one with combiner. So, a lot of stuff
is the same. The mappers to the same amount,
the input matches the output for without and with. That's
as expected. In fact, a lot of things look pretty
similar. But where things get really interesting is down here
at reduce input records. So, the reducers, had to handle the whole
4 million records that the macro had put when there is no combiner. But when
there was a combiner, these 4 million
mapped output records are combined into 412
reduced input records, or reducer input records.
That's pretty amazing. That's a huge change.
In fact, the reducers had to shuffle far fewer bytes. And for time
spent, with the combiner this job took 31 seconds to run versus without where
it took 43. That's a significant difference. So you can really use combiners to
improve the efficiency of your map reduced
jobs. And we're going to try that next.