>> Hi, I'm Brigit Murtaugh,
a Program Manager on the.NET team here at Microsoft.
Welcome back to.NET for Apache Spark 101,
where we're going to talk about the exciting world of
big data analytics within the.NET ecosystem.
In this video, we're going to write and run our very
first.NET for Apache Spark application on our own computer.
It's going to take less than 10 minutes. So let's get started.
So I'm going to start off by going to the.NET website,
which I can do by just typing in dot.net.
Then, I can scroll down here
and find the Machine Learning section.
If I continue scrolling further past ML.NET,
I'll find.NET for Apache Spark.
I'm just going to go ahead and click on that,
and I can see now that I'm on
the landing page for.NET for Apache Spark,
and I'm going click on "Get Started".
So with this, getting started tutorial,
we're going to learn how to write and run our very
first.NET for Apache Spark application in 10 minutes or less.
So I'm going to say, let's get started.
We can see that we have to start off by installing.NET.
Fortunately, I've already installed that here,
so I can check by opening up
a command prompt and just typing in.NET.
I can see that it's a recognized command,
so I've successfully installed.NET.
If I hit "Continue",
I can see that I need to install a few more prerequisites.
I need to install Java and then also 7-zip.
7-zip is a tool we can use to
unzip different compressed files that we'll be working with.
But again, I've already installed these,
but it won't take too long for when you install it at home.
The next step is actually installing Apache Spark,
and in this case, it'll be a nested tar files.
That's when we're going to start using 7-zip.
We just have to set a couple of
environment variables and elements of our path to
ensure that spark will be able to
find all the dependencies and downloads it needs.
Since I've already gone ahead and done these steps,
I can go ahead and check that everything installed correctly.
So I'm going to copy this command here.
Go back to my command prompt,
right-click to paste it,
and then let's run it and we
should be able to see the version of Spark that we're using,
assuming everything installed correctly.
Great. So we can see that our command prompt has filled in here,
and we can see we're using Spark version 2.4.1.
Awesome. We're going to continue on in our tutorial here,
and the next step is to install.NET for Apache Spark.
So we'd already installed Apache Spark in general,
but now we specifically want to install.NET for Apache Spark.
Again, you'll go ahead and say some variables for that as well.
I've already gone ahead and done that,
so I'm going to continue on in our tutorial.
So now, we're at the point where we're actually going to create
and build our first.NET for Apache Spark app.
So in these commands here,
what we're doing is creating a new app called mySparkApp,
and then we're changing to the directory
where mySparkApp is located.
So I'll go ahead and copy this command here,
open up a command prompt and right-click to paste them in.
So now our program has been created.
With the CD mySparkApp,
we've successfully navigated it into
the directory our new app was created in.
I'm going to continue on.
Now I can see the next step is to install a NuGet package.
So with this command, we're specifically adding
the Microsoft Spark NuGet package to our application,
which is necessary that it can have
access to all the Spark functionality.
We're specifically going to be installing version 0.4.0.
It's going to copy that command,
right-click to paste it in,
and now we've successfully added
the Microsoft Spark NuGet package to our app.
So continuing on, so the whole point of Apache Spark and.NET for
Apache Spark is to work with
data and to read it in and process it in some way.
So with this application,
even though we're not going to be reading
a terabyte or a petabyte of data,
we'll be reading in a small input file,
and the goal is to count how many times each word appears.
So here on this step,
we can see that we've been given what should be in our input file,
which is going to be called input.txt.
So I'm going to go ahead and copy this, and then,
I need to navigate to where it my app is on my local machine,
scroll down, find mySparkApp.
So within my directory here,
I can say "New",
and I'll find the "Text Document",
I'm just going to call it input.
I'll open that up, and then I can
just paste in that input from the browser.
I'll save that and exit out.
If we continue on,
we can see that the next step is to actually code our application.
So as I had mentioned, the point of
our application is to perform a simple word count.
So with that input.txt file,
we're going to count how many times each word appears.
So I'm going to go ahead and copy the code in here.
I'll go back to my directory,
and I can just open up my program.cs in any Notepad,
or Microsoft Visual Studio, or anything like that.
I'll use Notepad here and just remove all of the preexisting code.
I'm going to paste in
that code from the browser, and I'll save that there.
Awesome. So now we have our input file set and we
already have the code we need for our application.
I'm going to continue on.
So now it's time to actually build and run our app.
So I'm going to copy this command here
into the command prompt to go ahead and build our app.
We can see here, build succeeded,
so our app was successfully built.
Now, before we do Spark submit,
which is the command we need to
actually run and submit our Spark program,
we want to make sure we modify one more environment variable.
So if I go to my environment variables
and then I just click on "Environment Variables" here.
We want to make sure that we have
a.NET assembly search paths variable,
which is important so that Apache Spark and knows where defined
the DLLs or just any of
the dependent files that are Apple need to run.
So you can see here that the value I've given
that variable is the location of mySparkApp,
and then specifically the bin folder, the debug folder,
and in the version of.NET Core that I used to build my app.
So since that's all set,
Spark should be able to find all the files that
needs to successfully run my application.
So I'll say "OK" here.
It's always good practice when you modify an environment
variable just to go ahead and start with a new command prompt.
So I'm going to start with a new command prompt
and navigate back to my Spark app.
I'll just filled it one more time to
make sure that everything builds correctly.
Build succeeded, great.
So now we can run the Spark submit command.
So with Spark submit,
you can see that
there's several components to it that we're going to
need anytime that we want to run
or submit in Apache Spark program.
So we have the.Net runner which is
necessary to run a.NET for your Apache Spark program.
We have the path to the jar file for our program,
which is also necessary for any app that we're running,
and we also have the location to the mySparkApp.dll.
So there's several parts to this Spark submit command.
We always need to have the.NET runner,
the jar file for our program,
and the DLL for our program.
So we can see we have all of those aspects here.
So I'm going to hit "Enter" so that
our program can actually run and be submitted.
We can see we have our output here and it's just what we expected.
We have a table of word counts
of each word that was in our input file,
and the number of times they appear.
So that's awesome, we got what we expected.
Thanks for joining me in this video.
We created and ran our very
first.NET for Apache Spark application,
and it didn't even take 10 minutes.
Be sure to check out my other.NET for
Apache Spark one-on-one videos to continue
expanding your.NET big data analytics
knowledge and create some other really cool apps.