So, Sarah's just shown you the code. And one of
the nice things about using hadoop streaming is that it's
really easy to test your code outside of hadoop. So,
let's take a look at how to do that. Our mapper
takes input from standard input. So, in order to test
it, we can just run it from the command line
and type data in to test it. Here, I'm just
typing, as standard input, a couple of lines of sample data.
Six fields, separated by tabs. And then when
I hit Ctrl+D, to simulate the end of
input, you can see that the mapper outputs
the results, just as I'd expect. Even better, we
can just build a small sample data file, and pipe that into the mapper. So let's
do that. I'm going to just take the
first 50 lines of purchases.txt and save those into
a test file, which I'll call testfile. Now,
I can just pipe that to the mapper by
saying cat testfile, pipe that to mapper.py. And
that gives mapper that data a standard input, and
as you can see, the mapper produces its
output again to the command line. If we have
problems, we could just go back and edit the
mapper until it works. It's really nice and fast
to be able to do this without having to run
a complete hadoop job every time during the development phase.
We can do a similar thing with the reducer. It's
expecting a set of lines, each of which looks like
the store name then a tab then the value. So,
again, we can create a sample file which looks like
that and pass it in to the reducer. But even
nicer, we can test the entire pipeline. Remember that the mapper's
output is sorted by the hadoop framework, and then passed
to the reducer. So, we can simulate the entire thing
on the Unix command line, like this. I cat testfile.
I pipe it to the mapper. I then pass that
to the Unix sort command, and pass that output to
the reducer. When I run that, that simulates the entire
map followed by shuffle and sort, followed by reduced phase,
and as you can see, I've got the output from
the reducer. So, now that we've tested this on the
command line we can test it on the cluster. The best
practice when you're developing map reduce jobs is first to
test locally with a small data set before you run your
code on the entire, huge, set of data. So, now
that we've tested on the command line, we can test our
code on the cluster. Best practice, when you're developing that
reduced jobs, is to first test with a small data set.
Before you run your code on your entire
huge set of data. But, we're already, pretty,
confident here. So, let's just run the thing
on the wholepurchases.txt file. We'll use the HS
alias to save some typing. I specified my
mapper, my reducer, my input directory, and a
new output directory which we'll call output two.
And off hadoop goes. It starts running the job.
It turns out that we can see what's going on on the cluster by taking a look at
the hadoop job tracker web user interface. So, you point your
web browser at the job tracker, which on our machine is just local host on
port 50030. And here you can see that this just one running
job. If we click on the job name, then that takes us to a page,
which shows us the progress of the job, and a
lot of other interesting information as well. We can drill
down and look at the individual map, or reduce tasks
just by clicking on the words map or reduce. And
here we can see that two tasks have completed, two
tasks are still running. If I click on one of
the completed tasks, I can even go as far to
look at the logs from that task. So, here's our job.
We're now 25% of the way through our reducers.
As you can see, we get graphs at the
bottom of this page to show us exactly what's
happening with the job. So, we nearly finish with
the job, when the job's finished, hadoop fs minus
ls of output 2 shows me the just I'd
expect is a part dash 00000 file in that and just as we did before we can hadoop
fs minus cat that file to see our actual results.