Google Compute Engine is a great place for running Hadoop
to solve your big data challenges.
Compute Engine runs on Google's advance
infrastructure, developed through 14 years of experience
building high performance web services, supporting billions
of queries per day.
So you'll get consistent I/O performance from Compute
Engine that scales out predictably, which is
extremely helpful when you're developing solutions with big
data tools like Hadoop.
I'm going to walk you through a demo of setting up Hadoop on
Compute Engine using a sample application from our Google
Cloud Platform repository on GitHub.
Hadoop is an open source implementation of the
MapReduce paper published by Google.
Google has been using MapReduce for many years to
distribute data processing workloads over large clusters
of machines.
Today, we're going to show how to process 255 gigabytes of
text to compute the number of times each word appears in our
data set, and then output the results in a
sorted list using Hadoop.
First, we'll download the Hadoop getting started sample
from GitHub.
This sample sets up a Hadoop cluster on Compute Engine that
executes a MapReduce task.
To download the sample, simply run that git command from
within the directory where you'd like to
download the sample to.
Once we've got the sample downloaded, we'll cd into the
solutions Google Compute Engine
cluster for Hadoop directory.
Next, we'll need to make sure that we've got the necessary
prerequisites to work with Compute Engine and Hadoop.
The first tool we need to set up for use is gcutil, which is
a command line tool for interacting
with Compute Engine.
Follow that URL to download and install gcutil, then open
up the readme.mb file that's included with the Hadoop
sample and find the gcutil section.
Follow those instructions to set up the default project and
SSH keys to be used by gcutil.
The other tool that we'll be using is gsutil, which allows
us to run commands on Google Cloud Storage.
Follow the URL you see on the slide to
download and install gsutil.
Once we've got gcutil and gsutil installed, we're ready
to set up our Hadoop cluster.
First, we'll need to download and
install the Hadoop package.
Let's do that now.
You can see that when I run an ls command, that I'm currently
in the sample directory, which is where we'll run the
following curl command to download the Hadoop package.
Then, we'll extract the downloaded Hadoop package
using the tar command.
Then we'll apply a patch that's included with the
Hadoop sample.
And finally, we'll update the package again
using the tar command.
Next, we'll download the Open JDK and dependent packages.
To begin this step, let's create a folder called
deb_packages under the root directory of the sample and
then cd into that directory.
The sites where you can download these dependent
packages from are listed in the sample's readme file,
under the section titled Download Open JDK and
Dependent Packages.
Listing the files in deb_packages, you should see a
similar list of six files once you've completed this step.
This sample uses a Google Cloud Storage bucket from
which Compute Engine instances will download the Hadoop
package and generated files for the Hadoop setup.
Here's a short gsutil command you can run to create a bucket
for this purpose.
Be sure that the bucket is created in the same project in
which we'll be running the Hadoop cluster.
Notice that I'm specifying the -l flag with a value of
US-CENTRAL2.
Doing so ensures that our cloud storage bucket will be
created in the same region as our Compute Engine instances
that will be running Hadoop.
This way, our data transfers across both services will be
extremely fast, and we take advantage of the Google Cloud
platform's capability of free data transfers between cloud
services in the same region.
Next, we'll set up our client ID and client secret, which
will be used to access the Compute Engine API.
Go to cloud.google.com/console and click on the APIs and Auth
section to create a registered app for your project.
Then open the file gce_cluster.py in an editor
and replace the values for CLIENT_ID and CLIENT_SECRET
with the values that you just created in the Cloud Console.
We have a few more dependencies that we'll need
to have in place for our Hadoop cluster, which you can
find listed in the sample's readme file under the section
titled Download and Set Up Python Libraries.
Just follow the step by step instructions there to get the
Python API client library ready to use.
Now that we've got everything in place, we're ready to
create a Hadoop cluster using the sample application.
The main script of the sample is a Python file,
compute_cluster_for_hadoop.py.
To get familiar with the commands available, you can
run this script with the --help flag.
This script has four subcommands--
setup, start, mapreduce, and shutdown--
which we'll test out right now.
Starting with the setup command, we'll run compute
cluster for Hadoop setup, followed by our project ID and
the name of our Google Cloud Storage bucket.
The setup command creates an SSH key used for communication
between Hadoop instances.
It also uploads the Hadoop package and its dependencies
to Google Cloud storage so that Compute Engine instances
can download them to set up Hadoop.
Running setup also creates a firewall in the Compute Engine
network to allow users access to the Hadoop web consoles.
OK, now that the setup process has completed, we're ready to
start up our Hadoop cluster.
We can do that by running the compute cluster for Hadoop
script with the start subcommand, followed by our
project ID and the name of our Google Cloud Storage bucket.
After that, we can specify the number of instances we'd like
to start up in our Hadoop cluster.
By default, it starts six instances, one
master and five workers.
Let's bump that up to 97 instances and
start up the cluster.
And the startup process is complete.
That took roughly 8 and 1/2 minutes, pretty fast for
starting up a 97-node cluster with Hadoop ready for use.
The startup process returns a response which includes links
to the HDFS Console and the MapReduce Console.
Here's the HDFS NameNode Console, which shows that the
Hadoop file system is running as expected, with all 97 nodes
that we created.
And here's the MapReduce Console, where we can monitor
our MapReduce jobs as they're running.
With our Hadoop cluster up and running, we can now start our
MapReduce job.
Using the compute cluster for Hadoop script, we'll run the
mapreduce subcommand and pass it the following values.
First, we'll specify the project ID, followed by the
Cloud Storage bucket which will be used for temporary
storage while our MapReduce is running.
Next, we'll specify an input, which is a Cloud Storage
bucket where our input file or files reside.
This bucket currently contains the results of a Hadoop
MapReduce that I ran previously to pull out a
subset of English titles for every object in the
255-gigabyte Freebase source file that Google has made
publicly available.
So this input bucket now holds a single large text file with
nearly 200,000 titles, formatted with one
title on each line.
The output value is a Cloud Storage bucket where the
results of our MapReduce will be placed.
For the mapper, we'll specify the shortest-to-longest-mapper
Python program, which is located in the sample
directory of our sample application.
For the reducer, we'll specify the
shortest-to-longest-reducer Python program.
The next value we'll pass to the mapreduce subcommand is
the number of mappers, which we're setting here as 95.
Finally, since we want a single file containing our
MapReduce results, we'll specify a
reducer count of one.
Now let's run the command to start our MapReduce.
And now our MapReduce job has completed, and it only took
two minutes to run.
We can open up the Cloud Storage bucket that we
specified as the output location in our mapreduce
command to see the results of our MapReduce job.
And there's our output file.
Let's open the file to see the results.
And there we are.
Here's our word count results, with the words listed from
shortest to longest and in alphabetical order.
Excellent, success.
Now that our work is done, the final step is to
shut down the cluster.
Jumping back to the terminal window, we can easily do that
by running the compute cluster for Hadoop script with the
shutdown subcommand and specifying our project ID.
There, the cluster is now shutting down.
So we've now seen the process of setting up a Hadoop cluster
on Compute Engine using the Getting Started sample.
We also saw how to use the sample to run a MapReduce job
that processes a large data set and generates a result.
Using the Hadoop sample application as a starting
point, we've got all the pieces in place to begin using
Hadoop on Google's powerful infrastructure.
Thank you for watching.