[Music]
I am a Hadoop systems engineer
here at the a tablet headquarters in
Seattle today I'm gonna show you how to
run and troubleshoot a SV disappear job
on an Amazon EMR cluster let's get
started a soup recipe is an application
which enables copying large amounts of
data between Amazon s3 and HDFS s3
recipes install on earmark lustre by
default you can run extra days to be
using EMR step or execute the command in
a terminal on the EMR master node as you
can see that I've already logged into
the AWS management console let us go
ahead and open the EMR web console now
let us click on the cluster which is
currently running
and since you wanted to add a step on
the cluster let's go to the steps tab
so here you can add steps which will
basically one on your EMR cluster let us
go ahead and click on the add step
button
so here we can fill out the details
about what step you want to run since we
are going to run the asteroid SCP job
we'll leave the step type as custom jar
we'll give it a name and the jar
location will be the command run our jar
since we're going to run the asteroid
disappea command on the master node and
the arguments will be the astra disobey
command which will contain the source
and the destination path or the files
that we want to copy since we have all
the details we need for our step let's
go ahead and add the step
this will take some time to change the
status from pending to complete it as
you can see that the step is completed
let me go over to the terminal and show
you that the files were copied to HDFS
in our EMR cluster now that I'm in the
terminal and this is the master node of
the EMR cluster let us run the Hadoop FS
command to check whether the files were
copied to the cluster or not
so as you can see that the files are
copied from su bucket to HDFS we can be
sure that the step was finished
successfully in addition to copying the
files you can also aggregate multiple
files into a large file and also change
the file compression type please see our
documentation for more details and to
know more about the features about SV
DCB before I show you how to
troubleshoot a job let me give you some
background on asteroid s CP it is an
extension of the open-source tool
disappea so it creates a MapReduce job
underneath it yesterday CP is optimized
to work with a SS particularly Amazon s3
which is an object storage solution
widely used as a data leak therefore
troubleshooting s will disappear job
will involve troubleshooting a MapReduce
job also if we wanted you in a nested
SCP job then the tuning practices of a
normal MapReduce job will apply here now
let's take a look at the log files of s
through the CB so let me switch back to
the console now that I'm back in the
console we can see that we have a field
as we disap a job so we're going to find
out the reason white field so if we look
at the log file section you will see
four log files this will show you
important information about how the job
ran and the root cause of white field so
if we look at the first log which is the
controller log
so this will show you the environment in
which the job ran like the command the
EMR step ran in the background and user
which ran the step and other important
information but since we want to know
why the job failed let us look at the
standard error log and see what our
reason it is showing us so now we can
see that the reducer task failed to copy
the files so that's a good start to know
why a virtual fail but this still
doesn't give us a root cause as in why
the reducer does failed to copy the
files so in order to dive deep let us
look at the MapReduce job that a three
DCP step ran so we will click on the VG
ops link so here we can see that it ran
a MapReduce job with the job ID which
ends in zero zero zero seven we will now
go to the View tasks so this will show
all the mappers and reducers that the
job launched so here we can see that all
the reducers have failed so in order to
find out white field let us click on the
attempts of one of the reducers
we have the system log for that
particular field reducer so let us click
on one of them so if you scroll down
this log file you can see that we are
seeing an error message which says that
no such file or directory so that gives
us some idea why the job failed because
it wasn't able to find the source file
in the directory in this case this error
usually happens because the EMR cluster
is not able to access the source files
so the next steps will be to review the
bucket policy and ensure that the
permissions are in place such that the
EMR cursor can read these files once you
have fixed the permissions
please rerun the step to know more about
common errors of s4 disappea please see
our documentation thanks for watching
and happy cloud computing from all of us
here at AWS
[Music]
you
[Music]