Are you running a lot of Apache Spark
of Hadoop workloads on Google Cloud Dataproc?
There are several ways to monitor Hadoop and Spark applications.
In this video, we will see how to configure a Persistent History Server to monitor the Spark applications.
I am Anu, I am a strategic cloud engineer at Google.
Let's get started!
In this video, you will learn what is 1. Persistent History Server
2. How to setup a Persistent history Server and
3. Some of the best practices of setting up a Persistent History Server (PHS)
What is Google cloud Dataproc?
Dataproc is a fully managed cluster platform for running BigData frameworks
like Apache Hadoop, Apache Spark and various other BigData open
source tools and frameworks to process and analyze vast amounts of data.
Let's take Spark workloads,
how can you run your existing Spark workloads on Dataproc?
With Dataproc,
you can run your Spark workloads
on Dataproc serverless or long running Dataproc on Google Compute Engine or
Google Kubernetes Engine clusters or Ephemeral Dataproc clusters.
What is Dataproc Serverless?
With Dataproc Serverless, you can run Spark batch workloads without
having to provision or manage the clusters and servers. With an Ephemeral cluster,
Life of the cluster is tied to these Spark jobs.
What do I mean by that?
You create a cluster,
You configure it to run specific Spark jobs and the delete the cluster.
You can use Dataproc workflow templates or Cloud composer to orchestrate this.
These clusters can be benchmarked and sized based upon the Spark jobs.
These job scoped ephemeral clusters are effective for batch processing.
The challenge with Ephemeral Clusters and Dataproc Serverless for Spark is
that you will lose the application logs when the cluster machines are deleted.
Persistent History Server enables access to the completed Hadoop and Spark
application logs for the jobs executed on Ephemeral clusters or Serverless Spark. Persistent History Server is
stateless and it constructs the Spark UI of the application by reading
the application event logs from the Google Cloud Storage bucket.
Now we will see how to configure a persistent history server.
First step is you need to create
a Persistent History Server Google Cloud Storage Bucket (GCS) in advance.
Configure the logging directory property
when creating Ephemeral Clusters or Serverless Spark batch jobs.
This is the directory in the Google Cloud Storage bucket to store the event logs.
So far, we have a Google Cloud Storage bucket created and configured for the clusters to write the log to the bucket.
Now create a standalone cluster
and configure it to read the event logs from the Google Cloud Storage Bucket.
This standalone cluster constructs the Spark UI of the applications by reading
the event logs from the Google Cloud Storage bucket.
Now we will see a demo of setting up a Persistent History Server.
Let's navigate to the Google Cloud shell in the console.
This cloud shell provides command line access.
I have created
a Google Cloud Storage Bucket to store event logs from ephemeral clusters.
That step is not included in the demo.
This gcloud command creates a persistent history server.
You will find the details of the code in the description of the video.
Now the cluster creation is in progress.
This command basically configures
the cluster to access these Spark and MapReduce job history files and Yarn
aggregation logs from Google Cloud Storage bucket.
Now the cluster is created.
The public documentation has all the details on how to configure this.
The link is in Description section.
I highly recommend you refer to that when setting up.
If we check the configuration and click Show properties,
it shows Map Reduce job history directory and Spark event log directory,
done directory, intermediate directory and event log directory.
These directories are used by Persistent
History Server to construct the Map Reduce and spark UI from the logs.
Now you can access the Map Reduce
History Server UI and Spark History
Server UI by navigating to the web interfaces section.
Now we will see how to configure Ephemeral
cluster to write the logs to Google Cloud Storage bucket.
This gcloud command creates an Ephemeral cluster
and configures it to write the logs
to the same gcs bucket that the Persistent History Server is accessing.
This cluster creation is in progress...
Now the cluster is created, we can
submit a sample Spark job to the Ephemeral cluster.
I'm going to test with a Spark Scala job.
In the main class, I am passing
the main method of the scala application.
This is a Spark Pi program.
It calculates the rough value for Pi
using an approximation method. I can pass the corresponding job files.
I'm setting the argument to be 1000, which indicates the number of job classes.
Now I click submit.
This job is submitted to the Ephemeral cluster.
And this job writes the event logs to the Persistent History Server GCS bucket.
To check the running application, you can access the Yarn UI by navigating
to the web interface section of the Ephemeral Dataproc cluster.
The status of the job is completed.
So now we can navigate to the Spark History Server of the Ephemeral cluster
and this shows the Spark job application details.
If you click on the app ID,
it shows that there are 1000 tasks, which is the argument we passed.
Next, we can navigate to the Spark UI or the History Server instance.
And this presents the application details
and corresponding event logs of the jobs run on Ephemeral cluster.
If we click on the app ID,
it shows the stage level, task level details.
This is how you set up a Persistent
History Server to monitor and analyze all the completed applications!
Now we will see some of the best practices of setting up Persistent History Server.
Best practice 1: To start with a single persistent history server for a GCP
project is optimal. If there are multiple applications running,
if you want to isolate and monitor
specific business applications, you can create two or more Persistent
History Server gcs buckets, then create two or more Persistent History
Servers pointing to the corresponding buckets in the project.
Best practice 2: In the event of a zonal or regional failure,
you can quickly create a new Persistent History Server in another region,
pointing to the same Persistent History Server GCS bucket.
Best practice Three we recommend you benchmark.
With your existing Spark applications in the testing environment, before
promoting the Persistent History Server to production.
In production, monitor the instance
for memory and CPU utilization. In case of performance
degradation in the Spark UI, you can
recreate the Persistent History Server with increased
machine size with higher memory.
Best Practice 4: By default, the Spark
daemon memory is ~3.8 GB.
If many users are trying to access
the Spark UI and dig into the job application details, or if there are long
running Spark jobs with many tasks like 50,000 or ~100,000,
the 3.8 GB heap size could be too small.
This might impact the spark UI performance. In such scenarios,
increase the heap size to 8 GB or 16 GB in Persistent History Server.
Best Practice 5: Configure a 30 day lifecycle policy in the Persistent History
Server GCS bucket to periodically clean up the event logs.
This will improve the performance of the
persistent history server Spark UI considerably.
In this video we saw what Persistent History Server is, how to configure a
Persistent History Server, and some of the best practices.
Persistent History Server is important for troubleshooting issues
and for engaging Google Cloud technical support to troubleshoot and explore.
There are various other scenarios to consider
when setting up a Persistent History Server.
I am sure this video sets the foundation. To stay on top of all the latest changes,
periodically review Dataproc release notes.
This helps you to adopt new features and new functionalities.
If you are interested in exploring more,
please check out the Description section in this video!
Thank you for your time! Thank you for watching!:)