So we mentioned, a number of complete technologies that are used in cloud
computing and also we mentioned a number of types of functionality that
cloud computing technology needs to provide.
Again, let me offer a birds eye view of two popular stack for
big data that are used in cloud computing platforms.
The two concrete stacks I want to show you is the Hadoop Big Data Stack,
it's an open source stack and then also the Berkeley Data Analytics Stack, BDAS.
This is another open source stack.
These are by the way, not the only options.
There are number of both propriety stacks and
also a number of other open source stacks.
For instance, one example is the so-called HPCC stack by LexisNexis.
This actually, even predates Hadoop.
However, it wasn't necessarily popular or
widely used outside of the LexisNexis environment.
When it comes to the overall popularity and adoption rate,
both in commercial and academic settings or research settings,
these two are probably the dominant stacks out there today.
The Hadoop architecture looks like this.
At the bottom layer, there is the data storage layer,
the Hadoop file system and this takes care of taking large amounts of data and
somehow, splitting them and replicating and making sure the data doesn't get
lost and making sure that there are no hot spots in the system, etc.
Then there is a processing framework that can allow computation to
operate on such data and deals with the scheduling of what are the specific
notes that particular data manipulation operations need to be scheduled.
Ideally, the scheduling should be done in such a way,
so that you don't have to constantly move data from one note to another,
depending on what is the task that needs to process it.
We're really not going to talk in more detail about the Map Reduce framework.
The intent of these is just as an illustration to provide you with
an overview of the breadth of technologies in this space.
There's a component Hbase and this provides a table like view
of the stored data that is more similar to with what we're familiar
with in the context of databases, but it is not the same as the relation
of database representation of the data as were used in the SQL environment.
In order to support data processing and
data query operations like what SQL databases support,
there's a language front end hive that will take an SQL query and
then translate it into a number of these Map Reduce operations.
That will then operate on top of the data that's stored in this Hbase
tables on top of the distributed file system.
Order number of higher levels services that would also provide the same kind of
things.
So end users can use the R environment in order to describe certain
rules about how data should be processed and analyzed or
we can use machine learning algorithms, one of a collection of many that
are supported in the Mahout component that's part of the Hadoop stack.
And a number of other technologies that provide more specialized
type soft functionality or different kinds of interfaces to the end users.
At the lowest level, however,
all of these end up being a number of these more primitive operations that can
execute on top of the distributed file system where the data is stored.
And there are a number of supporting services that are needed in order to
deal with locker or coordination or to provide the streaming of data into
this distributed environment and this is what the Berkeley BDAS looks like.
There's similarly a component that's the data storage layer and
then there is a in memory component that is the in memory file system,
so that you can avoid the caches.
You can run the regular Hadoop Map Reduce engines on top of this layer or
you can run another type of programming model,
another processing framework called Spark.
There are a number of front-ends for the Spark in terms of the kinds of
execution models that are supported, so whether it's in SQLite queries or
stream processing or graph processing or machine learning types of
operations that should be executed on top of this processing framework.
Another component that's important to mention is this Mesos component
that's the lowest level scheduling framework, the lowest level
resource manager that allows the resources to be partitioned and
then used by different types of frameworks.
You may have a partition that's a Hadoop partition.
A partition that's a Spark partition.
A partition that's an MPI partition.
And using this stack,
the two will be able to basically coordinate and negotiate among them and
then you'll have elasticity across these very different types of framework.