>> Hi, I'm Brigit Murtaugh,
a Program Manager on the.NET team here at Microsoft.
Welcome back to.NET for Apache Spark 101,
where we go over the exciting world of
big data analytics within the.NET ecosystem.
In this video, we're going to deploy a.NET
for Apache Spark app to Azure Databricks.
First, let's start off by discussing what it means to
run or deploy an app locally versus in the Cloud.
If you solve one of the earlier videos in this series,
you saw that in order to get started
with.NET for Apache Spark on our own machine,
we need to do certain downloads and
dependencies like we needed to install Java,
and set up some variables and set some paths.
But if we want to run our.NET for Apache Spark app in the Cloud,
all we need to worry about is that all of
our dependencies and app files are uploaded somewhere to Azure.
In this video, we're specifically going to be deploying
our.NET for Apache Spark app to Azure Databricks.
Now, Azure Databricks is a fast, easy,
collaborative Apache Spark-based analytics
platform optimized for Azure.
It was designed in collaboration
with the founders of Apache Spark.
So you can really ensure the Azure Databricks takes the best of
Databricks and the best of
Microsoft to give us an optimized solution.
So to start our deployment journey,
I'm going to visit the Azure Portal which I can get to you
by just saying portal.azure.com.
The Azure portal is where all my Azure resources are located,
and where I can start working with Azure Databricks.
So I'm going to go ahead and if I
wanted to create a new Azure Databricks workspace,
what I would do is I create a resource,
go over to Analytics,
and then I can find Azure Databricks right here.
There's just a few steps that we would need to fill in to
create a new Azure Databricks service such as a name,
choosing a subscription, and a location,
and what type of pricing tier we want to use.
But I already have an Azure Databricks service created,
so I'm going to go to my Resource Groups,
open up my Resource Group,
when I scroll down I can find that I
have an Azure Databricks service right here.
So what I'm going to do is just click "Launch Workspace" so that I
can see what all is included in my Azure Databricks workspace.
So it signing me into Azure into Databricks right here.
Now, we can see all the functionality
to Azure Databricks has to offer such
as creating new notebooks and new tables and clusters and jobs.
So in order to start working with Azure Databricks,
I need to have a few different credential set up.
So I'm going to open up a new command prompt,
and in order to install and work with those credentials,
I need to actually have Python installed.
So I'm going to double-check to make sure I have Python installed.
I can see that I've Python 3.7.4.
I exit out of that.
Now next, I need to install that
the Databricks CLI or command line interface,
and I can do that using pip3 which stands for
Python three or a way to install Python three libraries.
Pip3 install Databricks CLI.
Now, I've already installed the Databricks CLI.
So in my case, we can see that I
get requirement already satisfied.
Just to double-check that everything installed correctly,
I'll run the Databricks command.
After successfully installing the Azure Databricks DLI,
I want to go ahead and actually authenticate.
So this is how I'm going to log into
my workspace and deal with jobs and clusters.
So what I can do to make sure that I have all of
my authentication credential setup,
is I can run Databricks configure token.
So now, we can see that it wants to start off by entering
a Databricks host which should begin with HTTPS.
So in our case,
if we look at the URL,
we can see that we have https eastus2.
So I'm going to say HTTPS and
eastus2 is location I had chosen
during creation of my Azure Databricks service.
Azure Databricks.NET.
So we have the host information set up.
Now we need to enter a token which is
a special identifying string that'll
allow us to login and connect to our Databricks workspace.
So what I can do to get that token,
is I'll go back to my workspace,
and then I'm going to click on "Accounts" and "User Settings".
I'll say generate new token and I can call this one is
for the exciting 101 video.
We can just leave it as a lifetime of 90 days.
I'll say generate here.
Then, I'm going to copy this token.
I'll go back to my command prompt and paste it in here,
where it asked for a token.
So now we successfully have
our authentication credential setup and so
it'll be able to access
information in our Azure Databricks workspace.
So now that we successfully
authenticated into our Azure Databricks workspace,
it's time to download some of the files that we'll
need for executing and running our Spark app.
So I'm going to go ahead and go to github.com/dotnet/spark,
which is the repository for.NET for Apache Spark.
The couple of files that I need can be
found within the deployment folder here.
So first, I'm going to need
db-init.sh which will help me
with my Databricks submission. So I'll click on that.
I can say Raw and then I'll right-click in here,
and say "Save As".
I'll just be able to save that script somewhere in my computer,
and we'll wonder remember where we
saved on our computer for later.
I can see that I already have it saved some,
so I'm not going to save it again in this case.
After db-init, we also need to install worker.sh,
which will also be another helper script for installing and
running dependencies and path files with our cluster.
So I can say Raw, right-click,
"Save as", and that would also be able to save that script here.
So now when I go back,
now that I have my two helper scripts that I need,
there's one other file that I need from this GitHub repository.
So I scroll down on this deployment page.
I can see a link to the Microsoft Spark worker.
Now, the Microsoft Spark worker is going
to determine what version of.NET for
Apache Spark and what version of.NET
Core is going to be used when we're running our application.
So in this case, it'd be using
NET Core app 2.1 which also works at NET Core app.
2.2, and it would be using
Apache Spark or.NET Apache Spark Version 0.4.0.
Now, even if you're on Mac OS or Windows machine like I am,
you want to make sure that you install the Linux version
because the clusters that we're going to be
using are going to run on Linux.
So you would just go ahead and click this,
and it would let you download the NET Core app.
2.1 and.NET for Apache Spark 0.4.0,
Linux version of the Microsoft Spark Worker release.
So now that I've downloaded
all the necessary files and scripts from the GitHub repository,
I can go ahead and publish my Spark app.
So I'll go ahead and open a new command prompt,
and I'm going to navigate to my Spark app.
Now the command I need to run is rather long,
so I have it saved over here.
So I paste this in here.
We can see that I'm running a Publish command
and I'm going to be using NET Core app 2.2,
and I'm going to be using Ubuntu which I
want to use some version of Linux since,
again, my clusters are going to be using Linux.
So I'll submit that.
So now that my app has been successfully published,
I can go ahead and in the File explorer,
I'll locate where my app is.
Then, I just need to go to the Bin,
Release, Net Core app 2.2.
I can see here, I have this great folder where
I can see that's the type that I published to.
Then I have a published folder here.
If we open up that published folder,
we can see that it's a lot of the dependencies and
files that my application needs to run.
So then when we want to run our application in
the Cloud or on Azure Databricks specifically,
our cluster is going to need all of these files,
all of this information.
So what I'm going to do is right-click on this "Publish" folder,
and I'll say "Send to",
compressed the zip folder.
This will compress and zip up all the files
that our application will need to run in the Cloud.
We can see here we have published.zip
created in our apps directory.
So now that we've zipped up our published app files and we
downloaded all the dependencies we
needed from the GitHub repository,
we can go ahead and upload all those files,
our app needs to run to Azure Databricks.
How we're going to do that,
is copy them to the Databricks File System or a DBFS.
So in order to do that,
I have some commands to setup already.
So I'm going to start off by going back a level in my directory.
Then first, what I want to do is copy over all of
those files that I had gotten from
the GitHub repository onto the Databricks File System.
So I'll right-click to paste them.
We can see that I've already uploaded these files,
so it's telling me that resource already exists.
But if you haven't already uploaded these files,
you wouldn't see that message.
But we can see that the structure of these commands
in order to upload files to Databricks.
As you see Databricks,
FS stands for File System, CP for copy,
and then the name of the file,
and specifically the location where we're uploading it
on the Databricks File System.
So right now, I uploaded the two helper scripts that I
needed which were db-init and install worker.
I also uploaded the Microsoft Spark Worker release.
Next, what I need to do is uploaded the input file to my app.
So I'm still using input.txt.
So I'm going to go ahead and copy
over input.txt to the Databricks File System.
After doing that, there's
a couple more files I need to upload but we're almost done.
I'm going to go ahead and upload the jar file from my app.
You may have noticed if you've seen the other
one-to-one.NET for Apache Spark videos,
that whenever we call Spark-submit,
we always have to have the jar file to our app.
So we want to make sure to upload
the jar file to the Databricks File System.
Finally, I'm also uploading publish.zip which is
that ZIP file that we just created
once we published our.NET Core app.
Now that we've uploaded all the files to
Azure Databricks that we needed to run our program,
we can go ahead and actually start creating the Spark job.
So I'm going to go back to
my Azure Databricks workspace and I'm going to create a new job,
I say create job here.
Let me just exit out of that download.
I can give my job any sort of name,
so I'll say "My Great Video Job".
So first, I'm going to configure Spark submit.
Unfortunately, I also have that command already prepared.
When I paste that in here,
we can see that it contains all of
the same components that we usually
have when running a Spark-submit command
locally on our own machine.
So you can see we have the.NET Runner,
with the path to the jar file,
but now in this case we also have
the published app files if you publish.zip.
We also have the name of our program which is my Spark app.
Go ahead and confirm that.
So now, I'm going to go ahead and modify our cluster,
which contains all of those worker nodes
that are going to be executing
individual tasks or smaller chunks of our user program.
So I'm going to go to "Advanced Options" and then Init Scripts.
Within this, I need to go ahead and put in the path to DB Init,
which is the scripts necessary to just get things started to
run an Azure Databricks.NET for Apache Spark app.
So earlier when I'd uploaded
the DB Init to the Databricks File System,
I had used the path spark-dotnet/db-init.sh.
I'll say "Add" here and I've successfully added
my DB or Databricks initialization script to my cluster,
and I can hit "Confirm".
So now I've configured settings for my cluster in spark-submit,
I can go ahead and run my Spark job.
So I'll say "Run Now",
and we can see now the first run of migrate video job has begun.
Now, it'll take about five minutes for this job to complete.
So now we can see that our job has
completed running and we can see that its status is succeeded.
So let's go ahead and click on "Logs".
We can see the standard output from our job.
We can see it's what we'd expected.
When we'd first created my Spark app,
it was a simple word count application.
We can see that we got that word count table
in our output log here.
Thanks for joining me in this video where we deployed
a.NET for Apache Spark application to Azure Databricks.
Be sure to check out the other.NET for
Apache Spark one-on-one videos to learn
even more about the world of
big data analytics within the.NET ecosystem.