Hi! Let's get started with Creating an Oracle Big Data Service Cluster video.
In this video, you'll learn how to plan and create a simple Cloudera Distribution
Including Apache Hadoop (CDH) cluster using the Oracle Cloud Infrastructure Console (OCI) and Big Data Service (BDS).
You can set up administrator groups and administrator users in several different ways, depending
on your requirements.
In this video, we describe a simple scenario that has three types of administrators.
A Cloud administrator has full access privileges to all resources in the tenancy, including
those used by any other services in the tenancy.
A Big Data Service administrator has privileges to manage one or more clusters and the resources
associated with them such as groups, users, storage, and so on.
A Hadoop cluster administrator has permissions directly on the cluster, to perform management
operations programmatically.
Before you can create a cluster, a Cloud administrator must set up your Big Data Service using the
OCI Console.
In this video, we'll create a simple CDH cluster using the existing resources shown on the
screen such as a tenancy, a compartment, an Administrator group, a BDS administrator,
policies, and a VCN; however, in later videos and in the BDS documentation, you can learn
how to create new resources instead of using existing ones, if needed.
The first policy statement grants the cluster administration rights to members of the ua-admins group.
The second policy statement grants network administration rights to the members of the ua-admins group.
The third statement allows the service to use the network.
The table on the screen shows some of the useful documentation resources that you can
use to learn how to create a tenancy, a compartment, and other resources.
It is a good practice to plan your CDH cluster
before you create it.
This includes choosing the cluster's instance type, shape, and layout.
For the cluster instance type, you can choose bare metal compute or virtual machine (VM)
compute.
A bare metal instance uses a dedicated physical server for the Hadoop node for the best performance
and strongest isolation.
VM instances are less expensive than bare metal instances and are useful for building
less-demanding applications that don't require the performance and resources of an entire
physical machine for each node.
Next, you select shapes for the Hadoop Master/Utility and Worker nodes.
The shape determines the number of available CPUs, memory, and type of storage among other resources.
For layout, a cluster can either be Highly Available (HA) or non-HA.
Use HA clusters for production environments for resiliency and to minimize down time.
Use a non-HA cluster for developing applications and testing functionality at a minimal cost.
A Master node runs the critical services for the cluster to operate such as the NameNode,
ResourceManager, and so on.
A Utility node generally contains utilities used for accessing the cluster such as Cloudera
Manager, Hue, and so on.
Worker nodes store and process data.
Our simple non-HA cluster will have one Master node, one Utility node, and three Worker nodes.
We will use the VM.Standard2.4 shape for all nodes in the cluster.
This shape provides 4 CPUs and 60 GB of memory.
Standard shapes are designed for general purpose workloads and suitable for a wide range of
applications and use cases.
They provide a balance of cores, memory, and network resources.
Nodes based on standard VM shapes use network-attached block storage.
The shapes of the Big Data Service Master nodes and the Worker nodes don't have to match.
But the shapes of all the master nodes must match each other and the shapes of all the
worker nodes must match each other as well.
We will use 250 GB for the Block Storage size for both the Master and Utility nodes, and
750 GB for the worker nodes.
For more information about the available Compute Shapes, use the url at the bottom of the screen.
It's time to create our BDS cluster.
Log in to the OCI Console.
On the Sign In page, select your tenancy, enter your user name and password, and then
click Sign In.
The Oracle Cloud Console Home page is displayed.
Click the Navigation menu in the upper left-hand corner of the Console Home page.
Under Data and AI, select Big Data.
On the Clusters page, click Create Cluster.
At the top of the Create Cluster panel, provide the cluster details as follows:
In the Cluster Name field, enter the cluster name.
In the Cluster Admin Password field, enter the cluster admin password.
You'll need this password to sign into Cloudera Manager and to perform certain actions on
the cluster through the Cloud Console.
Confirm your password.
For a simple non-HA cluster, un-check the Secure and Highly Available (HA) check box
which is selected by default.
The Cluster Version field displays the latest version of Cloudera 6 that is available to
Oracle which is deployed by BDS.
In the Hadoop Nodes section, you first provide the details about the Master and Utility nodes:
Select Virtual Machine as the instance type.
In the CHOOSE MASTER/UTILITY NODE SHAPE drop-down list, select VM.Standard2.4.
In the BLOCK STORAGE SIZE PER MASTER/UTILITY NODE (IN GB) field, enter 250 GB for storage size.
Since we are creating a non-HA cluster, the Number of Master and Utility Nodes read-only
field shows only two nodes: A Master node and a Utility node.
In the Worker Nodes section, provide the following details about the Worker nodes:
Select Virtual Machine as the instance type.
In the CHOOSE WORKER NODE SHAPE drop-down list, select VM.Standard2.4.
In the BLOCK STORAGE SIZE PER WORKER NODE (IN GB) field, enter 750 GB.
In the NUMBER OF WORKER NODES field, enter 3 which is the minimum allowed for a cluster.
In the Network Settings section, specify the following information:
In the Cluster Private Network section, you specify a CIDR block for the cluster private
network that will be created for the cluster.
This private network is created in the Oracle tenancy and not in your customer tenancy.
It is used exclusively for private communication among the nodes of the cluster.
No other traffic travels over this network, it isn't accessible by outside hosts, and
you can't modify it once it's created.
All ports are open on this private network.
In the CIDR Block field, enter a CIDR block to assign the range of contiguous IP addresses
available for this private network.
In our example, we used CIDR block 10.1.0.0/16 to avoid overlaping with the 10.0.0.0/16 CIDR
block range that we already used for our VCN.
In the Customer Network section, select the compartment name that contains the VCN, and
then select an existing VCN to use for the cluster from the CHOOSE VCN drop-down list.
The VCN must contain a regional subnet.
Select your regional subnet in your compartment from the CHOOSE REGIONAL SUBNET drop-down list.
If you plan to expose any IP addresses outside the subnet such as to access a node via the
internet, you must select a public subnet for the cluster.
In the Networking Options section, select one of two options:
Select the Deploy Oracle-managed Service gateway and NAT gateway (QUICK START) option to simplify
your network configuration by allowing Oracle to provide and manage these communication
gateways for private use by the cluster.
These gateways are created in the Oracle tenancy and can't be modified after the cluster is
created.
Select the Use the gateways in your selected Customer VCN (Customizable) option to allow
the cluster to use the gateways in your customer VCN.
You must create and configure these resources yourself.
In the Additional Options section, you can associate an SSH key with the cluster.
This enables you to access the cluster from the command line.
You can specify an SSH key using either the CHOOSE SSH PUBLIC KEY FILE option, or the
PASTE SSH PUBLIC KEY option.
We already selected an existing public key named mykey.pub.
Next, click Create Cluster.
The Clusters page is re-displayed.
The status of the new cluster is initially Creating.
Once the cluster is created, the status changes to Active.
To view the cluster's details, click the cluster's name link in the Name column.
The Cluster Details page displays your cluster's information.
In the Network Information section, the Subnet fields refer to the VCN that the cluster uses.
The NAT Gateway Configured and CIDR IP Address Block fields refer to the cluster's private
network.
Let's scroll down the page.
The list of cluster nodes section displays the five nodes in the newly created cluster.
You can click any node link to display its details.
The name of each node is the concatenation of the first seven letters of the cluster
name, followed by two letters representing the node type such as mn for a Master node,
un for a Utility node, and wn for a Worker node.
The numeric value represents the node type order in the list such as Worker nodes 0,
1, and 2.
Click the Actions button to maintain the cluster.
You can use the View Cluster Details context menu to add nodes, block storage, Cloud SQL,
and also to rename or terminate the cluster.
The screen shows some of the useful BDS, Hadoop, and OCI resources.
Thanks for watching!
Learn more at: https://docs.oracle.com/en/cloud/paas/big-data-service/