(bright airy rock music)
(computer keyboard clicking)
- Hi, I'm Aditya, a Cloud Support Engineer
here at the AWS headquarters in Seattle.
Sometimes customers ask me,
"What is Amazon EMR
"and how can I use it for processing data?"
I tell them Amazon EMR is a managed cluster platform
that simplifies running big data frameworks,
such as Apache Hadoop and Apache Spark,
on AWS to process and analyze vast amounts of data.
to process and analyze vast amounts of data.
such as Apache Hive and Apache Pig,
you can process data for analytics purposes
and business intelligence workloads.
Additionally, you can use Amazon EMR
to transform and move large amounts of data
into and out of other AWS data stores and databases,
such as Amazon Simple Storage Services, Amazon S3,
and Amazon DynamoDB.
Today, we'll discuss the architecture of Amazon EMR
and the step-by-step process to launch an EMR cluster
using the AWS Management Console
and then run a sample Hive application.
So let's start with talking about
the Amazon EMR architecture.
There are several types of storage.
Hadoop Distributed File System
is a distributed, scalable file system for Hadoop.
EMR File System uses HDFS or Amazon S3
as the file system in your cluster.
Local File System refers to a locally connected disk.
Cluster Resource Management manages cluster resources
and scheduling the jobs for processing data.
Uses YARN as the default.
Data Processing Frameworks.
Different frameworks are available
for different kinds of processing needs.
For example, Hadoop MapReduce, Tez, and Spark.
Applications and Programs supports many applications,
such as Hive, Pig, and the Spark Streaming library.
Now let me walk you through the process
of creating a sample EMR cluster
by using the AWS Management Console.
Then we'll run a Hive script as a step
to process sample data that is stored in Amazon S3.
But before you do that,
there are some prerequisites for your sample cluster:
sign up for AWS,
create an S3 bucket to store output data,
and create an EC2 key pair.
Once you have completed the prerequisites,
sign in to the console.
As you see, I'm already signed in to AWS Management Console.
Now let's select the region.
For demonstration purposes,
I'll just leave Ireland as the default.
Now let's navigate to EMR service.
Click on Create cluster.
We'll give it a meaningful name.
And we leave the Logging as checkmarked
and the Launch mode as Cluster.
And for the release version,
we're gonna use the default one, which is the latest one.
And then for Applications, let's select the Core Hadoop,
which includes Hive as well.
Now for Hardware configuration, we can leave it as default.
And for EC2 key pair,
we can select the EC2 key pair which we created earlier.
And Permissions, leave it as Default.
And you will see there will be default EMR role
and the EC2 instance profile already listed there.
Now let's click on Create cluster.
It will take some time for this cluster to spin up.
While the EMR cluster is spinning up,
let's prepare your sample data and script.
The sample data is a series
of Amazon CloudFront web distribution log files.
The data is stored in Amazon S3 at the URL that is shown,
and always be sure to change the region
to the same as where your EMR cluster is spinning up.
The sample script
calculates the number of requests per operating system
over a specified timeframe.
The script uses HiveQL,
which is a SQL-like scripting language
for data warehousing and analysis.
Again, the script is stored in Amazon S3
at the URL that is shown,
and, as always, be sure to change the region
to the same one as where your EMR cluster is spinning up.
Process your sample data by running a Hive script as step
via the AWS Management Console.
In Amazon EMR, a step is a unit of work
that contains one or more Hadoop jobs.
You can submit steps when you create the cluster
or when the cluster is running
if it is a long-running cluster.
The Hive script and sample data used by the script
have been uploaded to Amazon S3 for you.
To submit your Hive script as a step, select Steps.
Click on Add step.
Now here we'll select the Step type as Hive program.
Now go ahead and add Script S3 location, Input S3 location,
and Output S3 location.
In this case, we're gonna use the bucket
which we created as a prerequisite
for our Output S3 location.
So here we can see, for the Output S3 location,
I've used the bucket
which was created as a part of the prerequisite,
and I have given the folder name as output.
Now let's click on Add.
So now the step has been added,
and if you refresh this,
you can see now it's in the Running status.
Now we'll wait until the steps complete.
As you can see, this Hive program is now completed.
Now let's try to submit our Hive script
directly on the master node wire SSH.
But before we do that,
we need to open the ports for Port 22.
To do the same,
let's click on the Security groups for Master node.
This will take you to the EC2 Management Console.
Now let's select the ElasticMapReduce-master security group.
Click on Inbound.
Next, click on Edit.
Now click on Add Rule.
Then we'll select the SSH.
And the Source, I'm just selecting as My IP.
Click on Save.
So now that we have added Port 22 in the security group,
let's go back to EMR AWS console.
Here you will see the Master public DNS.
Now click on SSH.
Now this will give you the complete command
to SSH into the master node of this Amazon EMR cluster.
Now let's copy this.
Once you have this copied,
now let's move over to the terminal.
Now let's go ahead and paste the command here,
and then press Enter.
We select yes.
And now you will see
you're logged in to the master node of this EMR cluster.
So now let's go back to the EMR Console.
Let's navigate to Steps and then select this Hive Program.
And here you will see the complete command
to run this script.
Now let's copy this complete command.
And now we'll go back to the terminal
and paste this command.
And before we press Enter,
let's go ahead and change the output folder name.
This is just to make sure
that we can differentiate between the output
of two different jobs.
Now let's press Enter.
Now you will see the Hive job is starting,
and it will take some time to complete this job.
As you can see, the job is completed now.
To view the results,
let's navigate to the Amazon S3 console.
We'll search for Amazon S3.
And now we'll select the bucket.
Now select the output folder.
And this is the output file.
Let's download this.
And now let's open this file.
And here you can see the results of this Hive job.
Now reset your environment.
After you complete this tutorial,
remove your Amazon S3 bucket
and terminate your Amazon EMR cluster
to avoid incurring additional charges.
Congratulations, you have created an EMR cluster
and ran a sample Hive job.
For more information on analyzing big data with Amazon EMR,
please access this link.
Thank you for watching,
and happy cloud computing from all of us here at AWS.