>> We're in the exciting and ever-changing age of big data.
The volume, velocity, and variety of
data has been increasing multi-fold in recent years.
Take for instance a factory.
There could be thousands of
IoT sensors or Internet of Things sensors in a factory,
each producing petabytes of data.
While it's great to have that much data so you can understand how
our machinery is functioning and places for improvement,
how can we actually process it all when we have that much?
More than just that, how can we process it
all quickly and efficiently?
Well, welcome to the world of Apache Spark.
Hi, I'm Brigit Murtaugh,
and I'm a Program Manager on the.NET team here at Microsoft.
In this series of.NET for Apache Spark 101 videos,
we're going to be diving into the exciting world of
big data analytics within the.NET ecosystem.
So what is Apache Spark?
Apache Spark is a general-purpose distributed processing engine
for analytics over large datasets,
typically petabytes or terabytes of data.
Put maybe a little bit more simply,
Apache Spark is a great tool we can use to
analyze a lot of data in a quick and easy-to-understand way.
So you don't have to be a data science expert to understand it.
There's quite a few useful things
that we can do with Apache Spark.
For instance, there's Spark SQL
which allows us to analyze structured data,
such as data stored in a CSV or in a database of some sort.
There's Spark Streaming, which allows us to analyze and
gain insights from data live as it's being produced.
So in our factory example,
we could gain meaningful insights from our data
live as it's coming from our IoT sensors.
So if there's some sort of malfunction in the equipment,
we could detect it and go address it.
There's also Machine Learning capabilities of Apache Spark,
so that we can combine the powers of big data and
machine learning to get faster training or prediction.
There's also Visualization opportunities,
so that can gain meaningful insights into our data,
through graphs, through plotting or other visual means.
So in order to understand how Apache Spark works,
there's only three main components that we need to discuss.
The first one is the Driver.
The Driver consists of the user's program.
So for instance that could be
a C# console application that you've written,
and it also consists of this Spark Session.
So the job of a Spark Session is to take the user's program,
so to take that C# console app,
and to divide it into smaller pieces which can be known as tasks.
Those tasks will be divided among our second component,
which are the Executors or the worker nodes.
So each worker node is going to get its own task.
With that, it'll be able to
execute a small part of our user's program.
The third component of our architecture is the Cluster Manager,
which helps manage allocation of resources and how
our program overall gets divided and then executed in the end.
So now that we know how Apache Spark works, it sounds great.
I want to be able to use it to analyze all of my data,
all of my terabytes and petabytes of data.
But up until now,
the popular APIs for Spark were written in languages like R,
and Python and Scala.
But what if I want to use
ApacheSpark with my preexisting.NET code or knowledge?
Well, we now have a great solution and it's
known as.NET for Apache Spark.
So I go to the.NET website which is just D-O-T.NET.
I can scroll down and find
the Machine Learning section because we've combined
the machine learning and big data sections together.
So I'll go past ML.NET for now,
and I'll see.NET for Apache Spark.
So.NET for Apache Spark is a free open source,
and cross-platform big data analytics framework.
Like some of the scenarios we had mentioned earlier,
like Spark SQL and Streaming and Machine learning,
we can use.NET for Apache Spark for all of those applications.
The.NET bindings for Spark are written on the Spark interop layer,
designed to provide high performance for multiple languages.
It's also compliant with.NET Standard,
meaning anywhere that you already have C# or F#,
you'll be able to use.NET for Apache Spark,
meaning you can finally use
big data analytics in any of your.NET code bases.
As we had mentioned,.NET for Apache Spark is open-source.
So you can go ahead and check out the GitHub,
to learn more and to
contribute to the open-source community there.
Thanks for joining me in this video where we
learned all about Apache Spark and
the exciting new tool known as.NET for ApacheSpark.
Stay tuned to the other one-on-one.NET for Apache Spark videos,
to learn about all of the exciting
applications we can start building and
running using big data analytics, and the.NET ecosystem.