Hi Everyone, I'm Kristin, a Cloud Technical Resident at
Google Cloud and I will be covering how to orchestrate batch
Spark notebook executions on Dataproc Serverless
This will include typical customer use case
and solution along with the step-by-step demo on
Google Cloud Platform to run your Spark workloads without having
to provision manage or scale the cluster.
It supports Batches and Sessions.
We'll be focusing on running notebooks as batch jobs on
Dataproc Serverless in this video.
When running enterprise-grade data science and machine learning workloads. These
are some known challenges that Dataproc on
Google Cloud Platform can help address
Scalability and Performance. The scale of data and
models are large and tend to grow over time. And as we're
aware of it, more complex and application gets the
more difficult is to address capacity limits,
bandwidth issues, traffic spikes, and
many more. Manageability - for typical
enterprise ML deployment,
planning and configuring appropriate cluster size,
machine type, performance tuning and maintaining costs
are complex. So there is a need for more flexible
and manageable solution that lets you
just focus on the core business logic of ML that matters
the most to your business.
Typically customers want to lift and shift notebooks
from their legacy environment to Google Cloud Platform.
In this scenario, customers want to operationalize
and orchestrate that existing notebooks with minimum
efforts.
The migration is a two-step process. First, we
will migrate and stage Spark notebooks that were
in legacy Data Lake to Google Cloud Storage.
Then we will orchestrate notebook executions on Dataproc Serverless
as a batch job.
As this video is about orchestrating notebooks on
Dataproc Serverless, I haven't covered notebook development in the
video, but we recommend you use Vertex AI Workbench for
interactive notebook development on Jupyter UI provided.
Now, let's look how you can create a batch job to
deploy notebooks on Dataproc Serverless
So here we have vehicle_analytics_executor python file
shared in the description sections, so clone
that and submit it as a PySpark job to Dataproc Serverless
This executor python file executes the
spark notebooks and write the output to the Google
Cloud Storage.
From Dataproc on Google Cloud Platform. We're going to navigate to serverless
Serverless > Batches. Then create a job for our
notebook execution.
We leverage this vehicle_analytic_executor python script
we have shared in the description section which calls the
papermill utility that will take in the arguments and
do the execution for it.
So we can pass input and output notebook path as Argument
and submit as PySpark job.
If your notebook requires adding python packages,
you can pass the requirement.txt file
path here and you could also add additional python
files, JARs or any other reference files
that are in the Cloud Storage when submitting this job to
Dataproc Serverless
Once this job is submitted and completed, it will store the
execution results in the target Cloud Storage path.
It will take a few seconds to execute our sample
analytics notebook. Once that's done, you could
view the result from output console or navigate to the
output notebook path that you've assigned at a job creation and
download the file to your machine.
This is a sample output notebook of vehicle analytics you
will see in your bucket that executes few fundamental
data analytics on a Spark session. You can
check out the GitHub link for code and more details.
You can orchestrate this job using Dataproc Workflow Templates
or Cloud composer. Now, let's take a look how
we can orchestrate this using Cloud Composer.
We will use this specific DataprocCreateBatchOperator()
batch operator from Airflow library to create a
batch job on Dataproc Serverless.
For batch configuration, you will need to include 3 things
that we just covered so Job Type, Python file
and Arguments.
Sample DAG file can also be found in the GitHub repository added
to the description.
Before executing DAG, we have a few things to confirm.
First, you need to create a Composer environment if you
don't have it already.
Second, you need to place all the source files in
your staging bucket.
Again, all the files including vehicle analytics
executor python file, sample dataset, dependencies,
reference files and Jupyter notebook can
be found in the repository in the video description.
Please make any additional changes as you like.
For third step, from your Composer environment, you
will click here to open Airflow UI
and go to Variables to set up variables that you're
calling from Airflow code.
If you've added additional variables to DAG file,
please make sure you declare those variables here.
Finally, you're going to navigate to the environment configuration tab and
go to the DAGs folder to separately store your
DAG. So that Composer can sense and run it
when it's triggered.
Once you've placed your DAG files in the folder, you
can view your DAG name and you can trigger it.
So I click on my vehicle analytics tag, and
I've just triggered by pressing the play button
on the top left.
You can navigate to this Graph tab and monitor the executions.
The tasks will be executed in the order as specified and
it will show you different colors indicating different results.
And from this Grid tab, you can see by the
task how much time each task is taking or
total has been taken and real-
time log as well.
Once all the tasks have been successfully executed,
you can view the output result and log from
here as well.
So in this video, we saw how Dataproc Serverless
can be used to orchestrate and run batch Spark
notebook application at scale without having to
provision and manage complex clusters.
Running end to end data science workloads and Dataproc Serverless can help
reduce the data value gap. Please check out
the GitHub repository link and blog in the description for
getting hands on and for more details :)