Hi! Let's get started with the Introduction to Hadoop Distributed File System (HDFS) lesson.
In this lesson, you will learn about the Hadoop Distributed File System or (HDFS) and its
basic concepts and components.
A Hadoop cluster is made up of master and worker nodes which are physical servers that
are used for both distributed storage and distributed processing of massive amounts
of data.
HDFS is fault tolerant. Therefore, detection of faults, and quick and automatic recovery
from failures is a core architectural goal of HDFS.
HDFS stores large files. It breaks those files into blocks which are then replicated three
times and distributed across different nodes in the cluster. Redundancy yields fault tolerance.
If a server fails, the other two servers automatically take over its workload.
HDFS supports MapReduce and Spark processing frameworks.
HDFS is scalable. If you need more storage or processing resources, you simply scale
out instead of scaling up and add more nodes to the cluster.
Finally, Hadoop doesn't require expensive, highly reliable hardware. It's designed
to run on clusters of commodity hardware.
Let's look at a simple example of an HA Hadoop cluster. It is made-up of master and worker
nodes used for both distributed storage and distributed processing of the massive amount
of data stored in the cluster. The NameNode daemon also known as service, runs on the
Master nodes on the HDFS side of the cluster. The NameNode holds the metadata about the
cluster and manages the data storage.
HDFS also contains worker nodes that run the DataNode daemon or service. The Data is stored
and processed on the worker nodes. There is usually one DataNode daemon per worker node
in the cluster. A DataNode daemon manages the storage attached to the node that it runs
on and communicates with the NameNode.
On the Processing side of the cluster, we have YARN and the master nodes that run the
Active and Standby ResourceManagers. We will cover YARN, MapReduce, and Spark in later
lessons.
HDFS High Availability or HA deployment addresses the Single Point of Failure (SPOF) of early
Hadoop by running two redundant master NameNodes in the same cluster to provide a Fast Failover
if needed.
The Active NameNode is responsible for all client operations in the cluster.
The Standby NameNode acts as a hot backup to the Active NameNode, maintaining enough
system information to provide a fast failover if needed.
Let's go over the HDFS concepts of files, blocks, and replication factor using a simple
example. A client is writing a movieplex1.log file to HDFS. The file size is 350 MB. When
HDFS ingests the file, it breaks it up into (3) blocks or chunks: A, B, and C. Assuming
that the default block size is 128 MB, blocks A and B are both 128 MB each whereas block
C holds the remaining 94 MB. The (3) blocks representing the log file are replicated to
different nodes in the cluster using the default replication factor of 3. Replication increases
availability and increases opportunities for data locality. It is more efficient to move
the computation closer to where the data is located rather than moving the data to where
the application is running.
The NameNodes store file system metadata such as file name, size, replication factor, and
access rights to the files. File blocks information and locations in the cluster. Total number
of files in the cluster, total number of DataNodes in the cluster, and much more.
DataNodes use local disks in the commodity servers on which they run for persistence.
All the data blocks are stored locally, which improves performance.
DataNodes serve read and write requests from clients. They perform block creation, deletion,
and replication based on instructions from the NameNode.
DataNodes report their status to the NameNode by using heartbeat messages and Blockreports
to detect and ensure connectivity between the NameNode and the DataNodes. Receipt of
a heartbeat implies that the DataNode is functioning properly. A blockreport contains a list of
all blocks that are available on a DataNode. When a heartbeat is no longer present, the
NameNode assumes that the DataNode is no longer functional and it unmaps the DataNode from
the cluster and keeps on operating normally. When the heartbeat for the DataNode returns
(or a new heartbeat appears), it is added to the cluster transparently without the user's
intervention.
Let's go over the example of writing a file to HDFS. This example uses the movieplex1.log
file that we used earlier.
The client communicates with the NameNode and requests to write the movieplex1.log file
in the file system's namespace, with no blocks associated with it. The NameNode checks
to make sure the file does not already exist and that the client has the right permissions
to create the file. If these checks succeed, the NameNode makes a record of the new file;
otherwise, file write fails and an error message is displayed. The NameNode checks the available
nodes in the cluster and determines where the blocks and the replicas will be written
HDFS supports creating data pipelines. A data pipeline is a connection between multiple
DataNodes that exists to support the movement of data across the servers in the cluster.
A client application writes a block to the first DataNode in the pipeline. That DataNode
takes over and forwards the data to the next node in the pipeline. This process is continued
until all replicas are written to disk. In this example, a pipeline is created between
the 3 DataNodes and block A is written to DataNode 1. When block A is written successfully
to DataNode1, it is replicated and written to DataNode 2, and then DataNode 3. Therefore,
the data is pipelined from one DataNode to the next.
The same pipeline process is repeated for Block B.
The same pipeline process is repeated for Block C.
When each block is replicated successfully to the assigned DataNodes, Acknowledge messages
from the pipelines are sent back to the client to communicate that the blocks are copied
successfully.
This diagram illustrates rack-awareness in HDFS. The placement of replicas is critical
to HDFS reliability, availability, and performance. For the common case, when the replication
factor is three, HDFS's placement policy is to put one replica on one node in the local
rack, another on a node in a different (remote) rack, and the last on a different node in
the same remote rack.
To access the NameNode Web UI on the Oracle Big Data Lite VM, click the Start Here icon on your Big Data Lite VM desktop.
From the Web browser toolbar, select Hadoop > NameNode bigdatalite.localdomain:8020.
The NameNode Web UI is displayed. The NameNode exposes its Web UI on port 50070. You can
use this page to display useful information such as the number of files, directories,
and blocks in the cluster.
In this lesson, you learned about the Hadoop Distributed File System (HDFS). You also identified
HDFS design principles, characteristics, and components.
Thanks for watching!
Learn more at: docs.oracle.com/en/bigdata