One benefit of cloud computing is that it empowers anyone
to porpetually have infinite resources.
As long as you can pay, you can have the resources that are required to work
on maybe hard problems that involve lots of data and
that require lots of processing.
Here are some illustrations that I've found pretty informative.
To illustrate how much of data are we dealing with in
terms of our everyday life in our world.
So if we just think about one year, 2011, there
were 1.8 zettabytes of data that were created in that one year.
This amount of data is equivalent to everyone
in the US tweeting 4,320 times a day.
Pretty much not doing much else.
Or another way to think about it's this amount of data
is 200 billion high definition movies that are each about two hours long.
If you lived for
47 million years, you'd be just about done watching these 200 billion movies.
Or to store this much data, it would require so
many of the Apple's 32 gigabit iPads, that if you stack them
up together, it would be 25 times higher than Mt Fiji.
Pretty powerful these illustrations.
And this is all the data from 2011 only.
Clearly not every application needs access to all of the data
that's generated that's available in the world.
But a lot of the applications for scientific discovery,
for improving social services, for
a number of other types of applications, they do require a lot data.
Looking at cloud computing as a big data engine,
there are a number of layers that are necessary in order for
the cloud to provide that kind of infrastructure and
adopt those kinds of services that are needed by these big data applications.
Cloud platforms that offer as a platform as a service stack for
big data processing.
At the minimum have to have some layer that enables the storage and
access of the data from cross many, many nodes.
And also the ability to describe applications in a certain
parallel fashion, and ideally to do that easily, so
that the data can be processed across potentially many nodes.
We talked earlier in this course that access to memory and
access to local memory is faster than going to disk.
And particularly if you have to go to disk that
need these nodes that's attached to another node.
So most of the cloud stacks also incorporate some data caching
layer where the data can be brought in memory, or
maybe even a cross multiple memories, in the distributed system, big data system.
I said a cloud platform for
big data must incorporate some data processing framework.
But commonly, when we talk about small or smaller data,
the way we are analyzing it is using things like SQL queries or
some other type of querying mechanism.
And so often cloud stack, big data stacks, would incorporate some language
front end that would make this data processing a little bit more easier.
One common thing that people do with big data, that they analyze it.
They search for certain patterns, so
they perform some data mining operations that often involved common functions,
such as some machine learning algorithms.
So just as it's common for
cloud big data platforms to support popular language front-ends,
it's also common for them to support popular analytics libraries.
And also, a lot of the types of data that we're interested in
isn't just created once, and then that's it, that's a complete data set.
Instead, there's continuously generated knowledge.
And then potentially, the answers from the analysis on that
knowledge are derived continuously and updated over time.
So therefore, a common service that needs to be incorporated
in big data clouds would be something that provides as streaming abilities,
both as inputs and as outputs.