>> Hi, I'm Brigit Murtaugh,
a Program Manager on the.Net team here at Microsoft.
Welcome back to.NET for Apache Spark 101 where we
explore the exciting world of big data
analytic s within the.NET ecosystem.
In this video, we're going to be looking
at Batch Processing with.NET for Apache Spark.
So what is Batch Processing or what is Batch Data?
Batch Processing means that we're working with
data that's already been stored.
So for instance, we could be performing
log processing which means analyzing
user logs for maybe a website or
a server of some sort and we could gain insights
such as which pages are most popular to
our users or what type of commands do they typically run.
Another application for Batch Processing is data warehousing,
such as analyzing a mass amount of data
that we have stored somewhere in Azure Storage.
In this example that we'll go over in this video,
we're going to be looking at some GitHub's projects data.
So we have a CSV file that has a list of
various GitHub projects including their name,
the URL who created it,
and the number of times they've been forked from.
In our case we want to do some data prep and then we
want to know on average how many forks does each language have.
You can see that that column H there represents
the number of times each project has been forked.
So here I'm opening up a C# console application
that have already created and it's called Batch Spark app.
I've already installed the Microsoft Spark NuGet package,
so I'll be able to have access to all that great,
done it for Apache Spark functionality.
At the top, I need to make sure that I'm importing
over the Microsoft Spark SQL and SQL
functions into this program since I'm going to be
doing batch data processing and that relies on Spark SQL.
Once we actually have the Spark SQL and
Spark SQL functions imported over,
we can go ahead and start off by creating a Spark session.
So we're going to create a Spark session at the beginning of
any.NET for Apache Spark application that we write.
Because the Spark session is going to be in charge of creating
tasks and dividing our information
over those worker nodes or those executors.
So with my Spark session,
I'm just calling it a "Spark."
I go ahead and build it,
and I can give my app the name,
"My first batch app."
Pretty suitable for creating my
first.NET for Apache Spark batch application.
And then, I just call GetorCreate and
my Spark session has been successfully created.
So now that I have access to
all of the.NET for Apache Spark functionality,
I can go ahead and start creating a DataFrame.
So DataFrame is what we want to
use to read in any sort of structured data.
So in our case, we're going to be using a DataFrame to
store the projects CSV file that I had mentioned earlier.
We can go ahead and define a schema for
our data which is going to be
what type of values are in each column.
So for instance, in our schema here,
we can see that I have values for the ID of my project,
the name of the project,
when it was created,
how many times it's been forked from and when it was last updated.
I also have a type for each of those column names.
Now that I've defined a schema,
I can go ahead and determine what type of file I want to read in.
Since my data stored in a CSV or comma separated values file,
I'm going to choose this.CSV method into call in
the name of my file, which is projects_smaller.csv.
Great. So now I've read in my input file,
and let's just say I want to see how it looks.
I can go ahead and call my DataFrame object,
and then the show method which is going
to print my DataFrame to the screen.
I'll start off with a fresh terminal window here.
Then I'm going to navigate into my batch Spark apps directory.
Now that I've done that, I can build my app,
the build succeeded, and then I want to run
the Spark submit command which is
actually going to run my program.
My program successfully ran and we can see my output here.
So while the output does have all the information and columns,
we can see it's a little bit messy.
Some of the columns are
overlapping and there's a lot of null data.
So we want to do something called Dataprep or data
preparation which is cleaning up our files,
so that we can better analyze it and make sure we don't
accidentally try to analyze some sort of null columns or values.
So I'm going to start off by dropping
any rows to have any values or null values,
and I can do that by calling DataFrame NA functions.
With DataFrame NA functions,
I can determine how I want to deal with missing values in my data.
So for instance, if later I know I'm going to want to
perform some sort of Math or other calculations,
I might want to just get rid of any rows that have
any values to avoid any sort of errors in my code.
So in my case, I'm going to say that I'm dropping any which means
I'm dropping any rows that
have some sort of missing values and then.
Now that I've cleaned up and removed any values,
I also want to remove some unnecessary columns for my data.
Since we saw our DataFrame was kind of
long and overlapping with itself.
So with that, I'm still going to use my cleaned projects
DataFrame and I'm going to call the drop method,
and then pass them to the drop method the names of
the columns that I want to get rid of or that I want to drop.
This will ensure that my DataFrame is more
concise and only contains the information that I need.
I don't think I'll need that ID, owner ID,
or URL of my project for my overall goal for my analytics.
And as one final step,
it's always good practice to call
Spark.Stop to ensure that our program
can stop running successfully
once you've completed all of our execution.
So I'm going to open up my command prompt again and build
my project since I've written
some new code and made some changes to it,
and then I can call Spark submit again.
Now, we can see what the output from here.
We have a much cleaner DataFrame.
There aren't any null values like there were
above and there aren't too many extra columns.
The DataFrame isn't overlapping with itself.
So now that we've done some data path and cleaned up our data,
we can actually start doing the calculation we wanted which was
to see the average number of times each language has been forked.
So in order to do that,
since we want to find something for each language,
it would make sense to group our data by each language.
So I'm going to create a new DataFrame here
when I'm performing a new calculation,
and I'm going to call the GroupBy method.
And with GroupBy, I can determine which column
of my DataFrame I want to group by information by,
so in my case it's language.
Then the.Agg stands for the aggregate method,
and that means that I'm applying
a method to each group that I've grouped by.
So in my case, I wanted to find
the average number of times each language has been forked,
going to pass the average method in the aggregate method.
Okay. So now that I've successfully
performed the calculation that I wanted to,
I can go ahead and perform one more step
that I can see my data a little bit more cleanly.
I'm going to call the DataFrame that I
did the grouping in the aggregation
with and specifically call order by, and then descending.
So that I can get the most forked from languages at
the top since I'm most interested in seeing those first.
All right. Then, I also want to call
shows that I can actually view my output,
it's always a good method to call.
So I'm going to open up my command prompt
again and rebuild my app.
I can see that it succeeded and
Spark submit to run my application once more.
There we go. We can see that we have the average number of
times each language has been forked and it's in descending orders.
The top forked languages are at the top.
Thanks for joining me in this video where we created and run
our very first batch processing app
using.NET for your Apache Spark.
Be sure to check out the other
one-on-one videos in this series to learn
even more exciting functionality that we can create
using big data analytics and the.NET ecosystem.