Hi! Let's get started with the Introduction to MapReduce Framework lesson.
In this lesson, you will learn about the MapReduce Framework, identify its benefits and how it
works, and finally you run and monitor a word count MapReduce job.
The MapReduce framework integrates with HDFS and provides the same benefits for parallel
data processing.
It parallelizes and distributes computations to where the data is stored. Replication increases
availability and increases opportunities for data locality. It is more efficient to move
the computation closer to where the data is located rather than moving the data to where
the application is running.
The framework schedules and monitors tasks.
It hides complex distributed computing tasks from the developer.
Enables developers to focus on writing the Map and Reduce functions.
Let's go over the process of how MapReduce works. A MapReduce job is a unit of work requested
by a client. The job consists of input data which is usually stored in HDFS and a MapReduce
program which is created by the developer.
In this example, we have two HDFS input files. Hadoop divides the input files into fixed-size
pieces or "chunks" named input splits. The Input split is usually the same size as an
HDFS block which is typically 128 MB and it contains many records in the form of key-value
pairs.
Hadoop creates one map task (Mapper) for each input split. It runs the user-defined map
function on each key-value record in the input split and generates new key-value pairs. Hadoop
tries to run the MapReduce tasks on the same worker nodes where the data is located.
The Reducer task starts with the Shuffle and Sort step. It downloads the grouped key-value
pairs to the local machine, where the Reducer is running. The individual key-value pairs
are sorted by key into a larger data list. The data list groups the equivalent keys together
so that their values can be iterated easily in the Reducer task.
The Reducer takes the grouped key-value paired data as input and runs a Reducer function
on each one of them. Here, the data can be aggregated and filtered to produce new key-value
pairs.
The output of the Reduce phase is the union of all the records.
Let's go over a simple MapReduce Word Count example.
The MapReduce process is started.
The input text is divided into key-value pairs input splits with several records in each
input split. For example, the first input split contains the words Hello, World, Bye,
and World.
Hadoop creates one map task (Mapper) for each input split. The Mapper gets the key-value
pairs as input. The key in this case is the offset into the file, which is not useful
in this example, and the value will be a word such as Hello, World, Bye, and World. The
mapper task is started on the node that contains the data that is needed in the processing.
The goal of the mapper function is to parse the sentence and emit each word as a key,
with a numeric value "1". Each mapper emits a word with the number 1 everytime a
word is found. For the first mapper, the output is the key-value pairs , <World,
1>, , and . Each mapper finishes the initial processing and sends
the output to the Shuffle and Sort phase.
In this phase, the individual key-value pairs are sorted by key into a larger data list.
The data list groups the equivalent keys together so that their values can be iterated easily
in the Reducer task. For example, all key-value pairs from the mappers that contained
are now grouped as .
Next, the Shuffle and Sort copies the local sorted data as input to the reducers. The
reducer receives each key (that is, the word) and a value "1" and simply adds up or
aggregates the "1"s to compute the word count. In other words, the reducer combines the mappers
output into a total. For example, the reducer combines the key-value pair
from the previous step into . Finally, the output of the Reduce phase is
the union of all records that are produced by the reducers and the result is written
to the output file.
When you are running a MapReduce job, Hadoop does its best to run the map task on a node
that contains the HDFS block that is required by the Map task as shown in the first example
where the data block and the mapper are located on the same worker node. This is called data
locality optimization.
Sometimes, all three nodes hosting the HDFS block replicas for a map task's input split
are running other map tasks; therefore, the job scheduler locates a free map slot on a
node in the same rack as one of the HDFS blocks as shown in the second example.
If both situations are not possible, Hadoop uses an off-rack node in the cluster, which
results in an inter-rack network transfer as shown in the third example.
In YARN, mapper and reducer slots are replaced with containers. This is covered in the YARN
lesson.
In the Oracle Big Data Lite VM, click the Terminal icon on the toolbar to display a
terminal window.
For the remainder of this lesson, we will go over the classic MapReduce word count example
which counts the number of word occurrences in two text files in HDFS. We will use the
Oracle Big Data Lite VM for this demo. On this screen, we navigated to the /home/oracle/exercises/wordCount
directory, and then listed the contents of this directory.
If you are using the BDLite VM for this demo, you'll need to change the privileges on the
.sh files to read, write, and execute as shown on the screen before you can run any of the
highlighted scripts.
The WordCount.java program finds out the number of occurrences of each word in file01 and
file02. The code for the mapper function is on lines 14-26. The code for the reducer function
is on lines 28-37.
You can use the cheat.sh script to run the entire Word Count demo. On the screen, a summary
of the tasks performed by this script is shown. To run the script, enter ./cheat.sh at the
command line.
After you successfully run the cheat.sh script, you can perform the required clean-up by running
the cleanup.sh script. This is required if you plan on using the cheat.sh script again
to re-run the word count demo.
On the next few screens, we will run individual scripts that you can find in the /exercise/wordCount
directory which will perform the same tasks that are found in the cheat.sh script so that
we can provide some explanation as to what the commands do.
You can run the compile.sh script to compile the WordCount.java program.
The javac command compiles the WordCount.java program and it creates a compiled Java class
file. The -cp or classpath option specifies where to find user class files. The -d or
directory option, specifies where to place the generated class files, wordcount_classes
directory in this example.
The createJar.sh script creates the jar file. The jar command packages the application into
a single archive file and places it in the current directory.
The c option creates a JAR file. The v option lists the name of each file as it's added
to the JAR file. The f option specifies that the output should go to the new jar file rather
than to stdout. The -C wordcount_classes/. specifies that
everything in the wordcount_classes directory should be archived.
We will count the number of occurrences of each word in the file01 and file02 text files
using the WordCount.java program. The two files contain customers' feedback on a fictitious
car insurance company. The customers' comments will be used as the basis for the sentiment
analysis.
The copyFiles.sh script copies file01 and file02 from the local file system directory
to the HDFS directory; First, we will need to create the wordcount and input HDFS directories
which will contain the two input files. After we create the directories, we run the copyFiles.sh
script.
The runWordCount script contains the Hadoop jar command that submits the MapReduce program.
The command contains the name of the JAR file that contains the Map and Reduce user-defined
code and the path for the input and output HDFS directories.
Next, we execute the runWordCount.sh script to perform the word count.
Informational text from the Hadoop infrastructure is displayed on the screen to help you track
the status of the job. When the job is completed, the command prompt is re-displayed. The highlighted
information shows the MapReduce job ID that you'll need if you want to kill the job while
it is running. You can track the MapReduce job by using the URL provided.
Click the Browser icon on the Big Data Lite VM toolbar. Click the Hadoop drop-down list,
and then select YARN Applications.
The YARN Web UI All Applications page is displayed. This page shows the jobs that are managed
by YARN and their status. In this example, there are a couple of MapReduce jobs which
correspond to the word count programs that we ran twice. The status of both jobs is SUCCEEDED.
YARN is used to manage resources on a Hadoop cluster.
To access the JobHistory Web UI on the Big Data Lite VM, click the Hadoop drop-down list,
and then select JobHistory.
The JobHistory Server archives jobs' metrics and can be accessed through the JobHistory
Web UI or Hue. You can click a job's id url in the Job ID column to display the job's
details. Let's click the first job id in the list.
Note that this MapReduce job has two mappers and one reducer. You can click the Map or
Reduce links for the job to display additional information. Let's click the Map link.
The Map Tasks for the selected MapReduce job page are displayed. Let's click Go back to
go back to the previous page.
Click the Reduce link to display information about the Reduce task.
The Reduce task page shows a single Reducer for the selected MapReduce job.
Finally, let's take a look at the contents of the wordcount/output HDFS directory. After
running the MapReduce job, this directory contains two files: _SUCCESS which is a simple
flag that MapReduce generates to indicate that the job ran successfully. The part-r-00000
file contains the output of the MapReduce word count program. The output file name usually
contains either an m or an r as part of the name depending on whether the job was a mapper
or reducer. It also contains a gvfive digit sequential numeric number starting with zero.
Let's take a look at content of the part-r-00000 file using the hadoop fs -cat command. The
output shows some of the customers' comments that can be used for sentiments analysis such
as Best, Disappointed, Expensive, Terrible, Worst and so on.
In this lesson, you learned the benefits of MapReduce, and how to run and monitor a MapReduce
job.
Thanks for watching!
Learn more at: docs.oracle.com/en/bigdata