In this lesson we're going to look at one specific example of a
programming environment for dealing with big data
applications running on large computation clusters. And
this programming environment is called map-reduce
programming environment. And in this programming environment,
the input to the application is considered as a set of records Identified by
a key value pair. So the big data app developer, supplies the
run time environment with two functions called map and reduce. And these are the
only two functions that the app developer has to supply to the programming
environment and both map and reduce - Take user defined key value pairs,
as inputs, and produce user defined key value
pairs as outputs. So both the input and
the output to each of the map and
reduce functions, written by, the domain expert, are key
value pairs. Once again, these are key value
pairs defined by the app developer. Specific to
the particular application that he or she is
coding. Let's consider a fun example. In this example,
I'm going to say that we're looking to
find specific names of individuals in a corpus
of documents. Maybe the names are Kishore, Arun, Drew, and so on. These are the
specific names that I want to find in a whole corpus of documents. So the
input to the application is a whole corpus
of documents, and we are looking for specific
names and the number of times those names
occur in those documents. So that's what the
appllication is going to do. So the input key
space for the application is the file name.
And the value part of the key value bares the content of the file. So if you
have n files in the corpus of documents that we want to analyze in this fashion,
then you have N key value pairs corresponding to each of the different files and
the respective contents. So this is going to
be the input to the whole application and
were going to structure this application using this
programing paradigm of map reduce. And we will
see how the app developer will use
the map-reduce framework to accomplish the goal that
we set out. Which is to find the
number of occurrences of unique names, in this corpus
of documents. In this example, the user defined
map function is looking for each of the unique
names that we're interested in the corpus of
documents. So the map function will take as an
input a specific file and the contents of
the file as the key value pair, and emit
a value that corresponds to the number of times each one of these names occur in
the input file. Now, a simple version of
the map function may emit a one every time
it encounters the name Kishore in the input
file, or the name Arun in the input, or
the name Drew in the input file. The output of the map function is a key value
pair, where the key is the unique name
that you're looking for. One of the unique names
that you're looking for, and the value is the
number that says how many times it found that
particular name. As I said earlier, a simple version
of the map function could emit the value of
1 every time it finds one of the specified
names that it is looking for in the file.
Or a slightly more elaborate mapper may actually
combine the number of occurrences of a particular name
in the input file and then [UNKNOWN] that as a sum of all the times a particular
name occurred in this key value pair. In
either case, the output of the map function is
itself a key value pair. Note that it is
different from the input key value pair. The output
key value pair that the mapper is emitting is a unique name and
a value that is associated with that unique name and from this example it should
also be evident that we can spawn as many instances
of the map function as the number of unique files that we
have in the input document corpus. Further, since each of the
map function is working on a different key-value pair as an input, they're all
independent of one another and there is no need for coordination among them. And
that's the feature of an embarrassingly parallel
application. The output of the mappers are
the input to the reducers. In other words, the output key value pair from the
mapper is the same as the input key value
pair for each of the reducers that you see
here. Again this reduce function is something that is
written by the user. And you notice that what
the mapper is doing is when it notates a
particular name in this example that it is looking
for like Kishore then this mapper is going to
send the value associated with Kishore to this reducer.
Similarly this mapper is going to send the value
associated corresponding to Kishore to the same reducer. And
all the mappers in the same fashion are
going to send the respective values that they phone for
this unique name Kishore, to this reducer. Similarly,
the values phone for the name Arun, is going to
be sent to this video set by all the mappers, and so on. So the number of
reducers that we will have is the same as the number of unique names that
we are looking for in the corpus of documents. This is where work needs to
be done by the programming environment to plumb the output of the mappers To the
inputs of the reducers. Potentially, the mappers could
be executing on different nodes of the cluster,
reducers can be executing on different nodes of
the cluster, and the work that is involved in
this plumbing is really making sure that the outputs
of the mappers are fed to the corresponding correct
reducers in the entire distributed system. For example,
the output that is coming out of the mapper
corresponding to Kishore has to be sent to this
reducer from all the mappers. The output corresponding to
Arun coming from all these mappers has to
be sent to this reducer, and so on. That's
what we mean by the plumbing that needs
to be done by the programming environment to facilitate
the communication That needs to happen between instances
of the mapper and the instances of the reducers.
The work performed by each one of these reducers
is going to be aggregation of the values that it
got for each of the unique names that we're looking for in the input document
corpus. So this reducer's job is to
aggregate all the instances of Kishore that it
got from all the mappers. So it
is receiving the key-value pair corresponding to the
name Kishore and the number of instances
that they found in the input key-value pair
that they processed, and the reducer is going to
aggregate that And the output of the reducers, all
the reducers, is once again going to be a
key value pair. It is exactly the same as
the input key value pair that the reducers got.
Namely, the key is the unique name that they've
been assigned to aggregate, and the value is the
total number of the occurrence of this unique name.
In the corpus of documents we started with.
So this is the roadmap for creating a
map-reduce application for a chosen application that works
on big data. So in this case, the
chosen application was looking for unique names in
a corpus of documents, and all that the
domain expert had to worry about, is deciding
what is the format of the key value pair,
which is the input to the whole application, in
particular, the map phase of the application. And also, what
is the format of the intermediate key value pair that
is going to be generated by the mapper function. And what
is the work that needs to be done in
each of the mapper and the reducer to carry out
the work that needs to be done for this particular
application. Beyond that, the app developer does not have to
worry about any of the other details, such
as how many instances of mappers do we need
to create. How do we create the number of
instances of reducers corresponding to what this application is
wanting to do, and also worrying about the plumbing
from the output of the mappers to the input
of the reducers, all of those, are things that
the app developer does not have to worry about.