Hi everyone, my name is Misha Desai and I'm a program manager on the Azure Synapse Analytics team.
In this session we'll show you how you can create and configure your very own Apache Spark pool for your big data and analytics
workload. So now I'm going to switch over to the
Azure portal and navigate to my Synapse workspace.
Now once I'm in my synapse workspace, I'm going to switch over to the Apache Spark
pools tab and create a new pool.
So now as I'm creating my pool, there are a few different options that we can choose from.
So let's walk through each of them.
Now the first thing I'm going to do is name my spark pool.
Now in the next setting, what you'll see is that we've offered a default
which provides the best price per performance option.
Now what this offer also includes is a high memory to CPU ratio,
so it's great for some of your in memory analytics.
Now in the next setting we're just going to switch over to some of the smaller node
sizes just to reduce some of the cost for the spark pool that we're using.
In the next setting warrants a little bit more of a discussion.
Once enabled, the auto scale feature monitors
metrics like the sum of unused cores, memory and so on,
and then automatically scales the number of nodes in the pool up or down
based on the job that you're running during creation of the pool, you can specify the minimum and maximum number of
nodes that can be set when auto scale is enabled.
Autoscale then monitors and resource requirements in the load and scale the number of nodes up or
down based on the resource requirements.
In addition, there's no additional cost for this feature.
Now, one thing I wanted to call out here is that there's no cost for simply just creating a spark pool.
What you're built for is the time that the Spark pool is running or executing a job,
and so going through this process and simply creating the Spark pool will not lead to a direct cost.
So now I'm going to switch over to some of the additional settings,
so now the next setting we'll talk about is the auto pause setting.
The auto pause setting spins down the pool after the specified amount of idle time.
A pool is considered idle if no jobs or sessions have been submitted in the allocated amount of time.
So now let's take a look at some other component versions that come pre-installed with your your Apache Spark pool.
o by default each pool will come with a specific set of spark. Python, Scala, java, .Net and Delta Lake versions.
And what some of these language services will also
include a set of libraries are packages that are installed on each of the pools by default
Now for example, for languages like Python,
you'll automatically get the anacondas distribution.
So all of the libraries that are available through that distribution,
you'll be able to access from your spark pool.
Now, in some cases you might have libraries are packages that you need that
are not included by the default distribution,
and so for Python specifically, you can upload a requirements dot TXT file
and there's additional information in the documentation about the specific formatting of the file.
But here you'll be able to specify things like additional libraries or versions of those libraries that you might want.
So now, once you instantiate your pool, you automatically see that all of libraries that you've specified have automatically been installed.
So now let's switch over to the final setting.
Pull tides, allow you to monitor the cost of cloud resources used by various groups in your organization.
You can specify tags as key value pairs when you create a pool,
and so here I'll add A tag, say called
owner, and I'll just make sure I mark myself as the owner of this.
So the last thing I wanted to cover is the RBAC that will be required as you use.
various spark features.
If you plan on using features related to library management,
or if you plan on reading and writing to SQL pool databases via the Spark SQL connector
or even creating spark databases and tables, it's
really important that you contact the owner of the storage account and verify that
the following role assignments have been made
So you want to make sure that one the
workspace MSI has been assigned the storage Blob contributor role
and then you also want to make sure that you and any other users that might be using these features are also
assigned the same role. So this will help make sure that your
workflow using your spark pool is seamless and so you don't hit any are RBAC
requirements or gaps as you go about that.
So with that, we're going to go ahead and review and create our Spark
pool, and once the deployment is finished, I will be able to start using my Spark applications.
Thanks and thanks for listening in.