We saw the system's issues at a macro level from the point of view of
administering the engine that powers internet scale computing. How do
we program services such as websearch and airline reservation on top of the
data center cluster resources? How do
they exploit the hardware parallelism that's available
in these clusters? We will learn about
the map produced programming paradigm that answers
these questions. The term Big Data, has
become a buzz word. Computations in giant
scale services are usually simple, but they work over large data sets and hence,
the name Big Data. And naturally, because
they are working with a large data set,
these computations take a long time to compute.
For example let's say that I want to
search for John F Kennedy's photographs in all
the documents that are available on the web.
It's going to take a long time. And that
is an example of a Big Data computation.
Similar examples include online reservations, shopping, et cetera.
And applications that work on Big Data would
like to exploit the pluralism that's available
in the cluster at the data centers. I
mentioned in the last lesson that data
centers these days comprise of computation elements on
the order of thousands and even 10,000
nodes that are ready to do computations. And
we would like to exploit the computational resources
available in such Big Data centers to do
computation on Big Data. For example, if it is John F Kennedy's photographs, I'm
looking in all the different documents. We
can parallelize that's search for Kennedy's photo
in all the documents in parallel on all the available nodes in the data
center. And since computations are also called
embarrassingly parallel computations. What does this mean?
Well, there is not much synchronization or
coordination that is needed among the parallel threads
that comprise the applications and that run
on different nodes of the computational cluster. Looking
for John F Kennedy's photographs in all the documents on the web, is a good
example of such an embarrassingly parallel application. What
are all the issues in programming in the
large, on large data sets, so the data sets are also large and the computational
resources on which we want to run this big data application is also vast.
So some of the interesting issues in
programming in the large include, how to parallelize
an application across let's say, 10,000 machines.
How to handle the data distribution and plumbing
between the producers of data and consumers of
data. Remember these are embarrassingly parallel applications, but different
phases of the application will require data from one
phase of the application to be passed on to
the next phase of the application. That's what
we mean by plumbing between the producers of data,
intermediate data to be more specific, and consumers of
that intermediate data. And of course, one of the
biggest challenges in Big Data applications on
large computational clusters is failure handling. Recall what
we said about the nature of these
data centers. They employ thousands and thousands of
processes. When you have so many parts, in any setting, failure is not a
question of if it will happen. It is a question of, when it is going to happen?
So that's a fact of life, and therefore, programming models for Big Data, have
to worry about the fact that failures are to be expected in this environment.