Hi! Let's get started with the Introduction to YARN lesson.
In this lesson, you will learn the following: Identify YARN, its benefits, daemons, and
architecture. Configure YARN to use the pluggable Fair Scheduler. Identify Cloudera's Static
Service and Dynamic Resource Pools. Use the yarn application command to list, track, and
kill applications.
YARN Is a programming model or framework for Distributed Computing. Unlike the MapReduce
framework, YARN separates the resource management and processing components. It is a resource-management
framework for Hadoop that is independent of the execution engines.
Integrates with HDFS to provide the exact same benefits for distributed parallel data
processing on the cluster.
Uses a Master-Worker node architecture.
Schedules and monitors tasks and re-executes failed tasks.
Sends computations where the data is stored on local disks (data locality), when possible.
Hides complex "housekeeping" and distributed computing complexity tasks from the developer.
At a high level, Hadoop architecture has the following layers: Distributed storage using
HDFS or NoSQL Databases. Resource management using YARN and cgroups (control groups). Distributed
processing using MapReduce, Hive, Spark, and others processing engines. Metadata store
using Apache Hive.
Let's go over some of the benefits of using YARN. Scalability:YARN ResourceManager concentrates
exclusively on scheduling, resulting in better cluster management. Compatibility with MapReduce:
Existing MapReduce applications can run on top of YARN without disruption to their existing
processes. Improved cluster utilization: There are no named map and reduce slots, which helps
use cluster resources more efficiently. Support for workloads other than MapReduce.
An HA Hadoop cluster is made-up of master and worker nodes used for both distributed
storage and distributed processing of the massive amount of data stored in the cluster.
The NameNode daemon runs on the Master nodes on the HDFS side of the cluster and it holds
the metadata about the cluster and manages the data storage.
For distributed processing and resource management, we have the YARN's Active and Standby ResourceManager
daemons which run on the Master nodes on the processing side of the cluster.
MapReduce 1 framework used the concepts of Mappers and Reducers slots. A mapper slot
could only run a mapper task and a reducer slot could only run a reducer task.
The YARN framework uses the concept of containers which represent the memory and CPU resources
given to an application on a node in a cluster. A vcore, or virtual core, is a usage share
of a host CPU. YARN currently supports memory and CPU. A container is supervised by the
NodeManager and scheduled by the ResourceManager. For now, you can think of containers as a
request to hold resources on the YARN cluster for an application.
The table on the screen lists and defines the YARN daemons for your reference. We will
discuss the daemons using diagrams starting on the next screen.
Let's go over the architecture and components of YARN. On the Master nodes, we have the
Active and Standby ResourceManagers services.
The ResourceManager is the master daemon that communicates with the client, tracks resources
on the cluster, and orchestrates work by assigning tasks to NodeManagers on the worker nodes.
The ResourceManager has two main components: The Scheduler and the ApplicationsManager.
The Scheduler is responsible for allocating resources or containers to running applications.
The ApplicationsManager is responsible for accepting job-submissions from the client,
negotiating the first container for executing the application-specific ApplicationMaster.
The ApplicationMaster is the first process run after the application starts.
The ResourceManager keeps all jobs' information in memory. For completed jobs, it moves that
information from memory to the JobHistory server to free up memory.
The NodeManager daemon runs on a worker node. It is responsible for launching containers
for applications and monitoring their resource usage such as CPU, memory, disk, and network,
and reporting this information back to the ResourceManager.
Again, the ApplicationsManager negotiates the first container for executing the application
specific ApplicationMaster.
The per-application ApplicationMaster has the responsibility of negotiating appropriate
resource containers for the running application from the Scheduler, tracking the containers'
status, and monitoring the progress of running tasks. The ResourceManager and the NodeManager
form the data-computation framework.
Each NodeManager on a worker node tracks its own local resources and communicates its resource
configuration to the ResourceManager, which keeps a running total of the cluster's available
and used resources. This way, the ResourceManager knows how to allocate resources as they are
requested.
DataNodes, NodeManagers, and other services such as HBase and Oracle Big Data SQL require
a basic amount of resources set aside to operate properly. Once resources are allocated to
the various components, the rest can be allocated to YARN. A task runs as a process inside a
container. This is where your code runs which requires memory, CPU, and HDD resources.
Let's go over a simple example of submitting a Yarn application to the ResourceManager.
The application requires HDFS data blocks A and B, 16 GB of memory, and two vcores.
The ResourceManager requests a single container for the application from an available NodeManager.
NodeManagers send heartbeats to the ResourceManager to indicate that they are available to run
jobs.
The NodeManager starts the ApplicationMaster for the submitted application in that container.
The ApplicationMaster then requests the subsequent containers from the ResourceManager to run
the tasks for the submitted application. It also communicates with the NameNode to find
out where the data blocks are located.
The ResourceManager tries to grant requested resources where the data is located. In this
example, blocks A and B are located on worker nodes 2 and 3; therefore, the ResourceManager
assigns any free containers on worker nodes 2 and 3.
When all the tasks are completed, the ApplicationMaster sends the result set to the client application,
informs the ResourceManager that the application has successfully completed, de-registers itself
from the ResourceManager, and shuts itself down. The output is written to HDFS and the
completed job information is written to the JobHistory server. The resources used by this
application are released.
The scheduler is responsible for deciding where and when to run tasks. YARN supports
the following pluggable schedulers:
First In, First Out (FIFO). Allocates resources based on arrival time.
Capacity Scheduler which is the default in Hadoop. It allocates resources to pools, with
FIFO scheduling within each pool.
Fair Scheduler. Allows YARN applications to share resources in large clusters fairly.
It is the default in CDH5 and later releases and is also used in Oracle BDA.
The Fair Scheduler bases scheduling fairness decisions only on memory but you can configure
it to be based on both memory and CPU what we call Dominant Resource Fairness, or DRF.
It organizes applications into queues (pools), and shares resources fairly between
these queues. Every application belongs to a queue. The default queue is named default.
It grants YARN containers to the queue with the least amount of allocated resources.
The Fair Scheduler configuration is maintained in two files: the yarn-site.xml configuration
file and the fair-scheduler.xml allocation file. You can edit the two files directly,
but it's not recommended.
The configuration file contains properties and is used to configure the ResourceManager
and NodeManager.
The easier way to configure YARN to use the Fair Scheduler is to use Cloudera Manager
which uses an intuitive UI to change the underlying .xml files.
You can assign the Fair Scheduler class in yarn-site.xml configuration file directly
to enable the Fair scheduling policy in your YARN cluster as shown in the partial yarn-site.xml
file on the screen. The property name and value are highlighted.
You can find detailed information on the available properties that you can set in yarn-site.xml
using the URL shown on the screen. The partial table shows two such properties.
You also need to configure the Fair Scheduler allocation file named fair-scheduler.xml to
reference the scheduler used and to specify the queues and weights which are used in the
fair share calculation, and so on. Next, you will need to copy the yarn-site.xml and fair-scheduler.xml
files to all nodes in the cluster, and then restart the ResourceManager and NodeManager
daemons.
The easier and preferred way to configure YARN to use the Fair Share scheduler is by
using Cloudera Manager.
Enter /scheduler class in the Search text box in Cloudera Manager, and then select yarn:
Scheduler Class from the list.
In the Configuration tab, click the default value link at the bottom of the page.
By default, the Fair scheduler is selected. To choose a different scheduler such as Capacity
or FIFO, click the appropriate option, click Save Changes, and then follow the prompts.
For detailed information on configuring the Fair scheduler, use the URL shown on the screen.
Cloudera Manager provides the following features to assist you with allocating cluster resources
to services:
Static Allocation Pools which is a percentage of the cluster resources, and Dynamic Resource
pools.
When you have multiple processing engines such as YARN, Oracle Big Data SQL, Impala,
and others running on the same cluster, you can use the Static Service Pools feature to
divide the cluster resources between the different engines. For detailed information on Static
and Dynamic allocation pools, use the URLs shown on the screen.
To work with Static Service Pools, navigate to Clusters > Cluster name (if you are managing
several clusters), and then select Static Service Pools. The Static Service Pools page
is displayed. To configure the Static Service Pools, click the Configuration tab. The Static
Service Pools wizard is displayed. Follow the prompts.
To allocate resources within a single engine such as Yarn, Cloudera Manager offers Dynamic
Resource Pools. This enables you to divide the resources between different user groups
inside a single engine such as YARN. To work with Dynamic Resources Pools, navigate to
Clusters > Cluster name (if you are managing several clusters), and then select Dynamic
Resources Pool Configuration.
In this example, the gray boxes represent the allocated static pools for different services
in the cluster and the blue boxes represent the Dynamic Resource Pools for different groups.
HDFS and YARN were both assigned 40% of the cluster resources and Oracle Big Data SQL
was assigned 20%.
You can dynamically distribute resources that are allocated to a static pool. In our example,
we defined three dynamic resource pools for YARN: training, Dev, and PM with weights of
1, 3, and 2 respectively. A weight given to the resource pool determines how to allocate
resources relative to other resource pools. For additional information, use the URL shown
on the screen.
Continuing with our example, let's say an application starts and is assigned to the
PM pool. Let's assume that other applications are using the training and Dev pools. In that
case, the PM resource pool receives 10% which is calculated using the formula shown on the
screen. Take a moment to review the formula.
To access the YARN Web UI All Applications page on the Big Data Lite VM, click the Browser
icon on the Big Data Lite VM toolbar. Click the Hadoop drop-down list, and then select
YARN Applications.
The YARN Web UI All Applications page is displayed. This page shows the jobs that are managed
by YARN and their status. In this example, there are a couple of MapReduce jobs which
correspond to the word count programs that we ran twice. You can click the application
id link for a job in the ID column to display the job details.
To access the JobHistory Web UI on the Big Data Lite VM, click the Hadoop drop-down list,
and then select JobHistory.
The JobHistory Web UI page is displayed. The JobHistory Server archives jobs metrics.
You can click a job's id link in the Job ID column to display the job's details.
To display the ResourceManager Web UI in Cloudera Manager, click the yarn service link in the
list of available services on the CM Home page. The yarn page is displayed. Click the
Web UI tab. The Active and Standby ResourceManagers are displayed in addition to the JobHistory
server.
Click the Active ResourceManager Web UI. The ResourceManager Web UI All Applications page
is displayed.
Scroll down the page to view other jobs. For each job, this page shows the application
id, the username, job name, application type, queue where the job ran and more.
To view information about the scheduler in your cluster, click Scheduler. The Applications
Queues section shows details about your scheduler (Fair in this example) and the information
about the applications running. In this example, there are no applications that are currently
running.
You can use the yarn application command to list, track, and kill running applications.
The partial table on the screen shows few of the options that you can use with this
command. For example, you can use the yarn application command followed by the -list
option to display the list of the currently running applications. Among other things,
the -list option displays the application_id that you can use after the -status option
to track the status of an application.
You can use the URL on the top of the screen to view the complete list of the yarn application
user commands and options.
In this lesson, you learned about YARN, identified its benefits, daemons, and architecture. You
also learned how to configure YARN to use the pluggable Fair Scheduler. You learned
about the Static Service Pools and Dynamic Resource Pools features in Cloudera. Finally,
you learned about the YARN application command to list, track, and kill applications.
Thanks for watching!
Learn more at: docs.oracle.com/en/bigdata