The question is why map reduce is
a programming framework for big data applications.
It turns out that several processing steps
in giant-scale services are expressible as map
reduce. For example, let's say that you're
looking for seat availability for selected dates
to selected destinations on different airlines. That
can be expressed as a map reduce computation.
Let's say you want to access frequency of
the URLs on the website that you've designed.
That can be quoted up as a map
reduce application. Let's say you want to create. Word
indexes to facilitate document searches on the web.
That can be coded up as a map
reduce application. Or let's say that you want
to do ranking of pages. When a user is
doing a search, how to present present the
search results May depend on the popularity of pages
and that is what is often referred to as
page ranking. So if page ranking has to be
done for all the documents that is another
application that can be [UNKNOWN] up as a map-reduce
application. The list goes on. All such examples that
are mentioned share some common property. They're embarrassingly parallel,
and they're common in giant-scale services. And all
of them tend to work on big data
sets. Therefore there is plenty of opportunity for
taking advantage of the computation resources that are
available in a data center. And all such
applications need domain expertise in terms of what
to do with the data. Which is expressible
as a map function and a reduce function, and
this is the only thing that is required
of the domain expert to provide to the programming
system, because that is domain expertise that lives
with the app developer. So here is another example
of how an application may be structured as
a map_reduce application. And in this case, I'm showing
you a page ranking application that is, I'm
interested in knowing what is the popularity of different
URLs that occur in a document corpus. So the keyspace that is input to this
application is a set of URLs. And the key value pair that is coming into each
of the mapper is a source URL and the contents of that web page that corresponds
to this particular URL. So what this mapper
is doing is, in the given page defined
by this URL the contents of which is the input to this mapper it is looking for
different targets. Maybe it is looking for a particular URL target one, another
URL target two, and so on, target N, and that's what each of these mappers are
doing. So the keyspace that is output from the mapper is unique target names.
So the keyspace that is output from the
mapper is unique, target URLs and, the value
is the URL in which it was actually phoned. So the corpus of input URLs it is
taking, and saying well, in this particular URL
I phoned this target. If it did. It
is emitting that this target was found in
a particular URL. And this reducer is going to get
all the hits for a particular target that
was found in the input corpus of URLs. So
all the mappers they're going to send their results to
this reducer if they found in the input -
Surl the target, target 1, if they did,
they are going to send their results to this
reducer. Similarly if they found target n, each of
these mappers are going to send this reducer that
in the input URL they found this target
n and the job of the reducer [INAUDIBLE] Is
once again aggregation, and you have as many reducers
as the number of unique targets you're trying to
identify in the input dataset. Very similar to the
previous application that we went over. So the output
of the reducer is going to be the specific
target that this guy has been asked to accumulate.
And, a source list, meaning all the source pages in which this particular target
was found. So each of these reduces is going to find the number of times a
particular URL is found in the input
corpus of webpages that came into the system
as a whole. For instance, if I wanted to find out how many times my webpage
appears in the universal web pages all over
the world, we can take the entire corpus of
web page available in the universals input, and the
map that's we're going to look for occurence of my
web page in each one of those input web
pages. And if they find that, they're going to
send it to this reducer and if target one
corresponds to my web page then this reducer is
going to say, okay, show his webpage, was found
in this list of source webpages all over the
Universe. That in a sense gives a rank for.
That particular web page that we're looking at. So
we're able to rank the target web pages 1
through n based on the number of source web
pages that contain that particular target. And that's what
page ranking is all about. So I'm giving you
yet another example of how This map
reduce functionality can be applied for an application
such as page ranking. All the heavy lifting that needs to be done, in terms of
instantiating the number of mappers, instantiating the
number of reducers, The data movement between the
mappers and the reducers, all of those chores,
are taken care of by the programming environment.
All that the domain expert had to do was
to write the map and reduce function that is unique
to particular specifics of his application. The rest of
the heavy lifting is all done by the programming framework.