Nick Rimmer: Hello, and welcome to this video
where I'm going to show you how to create an
Amazon Web Services S3 batch job. So you can use
Amazon S3 batch operations to perform large scale
batch operations on Amazon S3 objects. You can
execute a single operation on lists of Amazon S3
objects that you specify. A job as a basic unit of
work. A job contains all the information necessary
to execute the specific operations on a specific
list of objects. Let's get started.
So you're going into an S3 bucket with some files
in that bucket. I've got that, and it's called
techsnipsUK and I've got five test files in there.
And I've also got a bucket which I'm going to use
as a destination bucket. So in this video, I'm
basically just going to copy those files to
another bucket. So what's the first thing we need
to do? Well, we need to get an inventory up and
running. We need to go into our source bucket,
which is techsnipsUK, go into management, and then
click on inventory. You'll see we don't have an
inventory to find at the moment so we click on add
new. I would give the inventory a name.
This inventory will be used by the batch
operations for moving our files. We'll just select
the techsnipsUK inventory as our destination. And
we are going to take the default output format of
the CSV. S3 batch operations can use this
particular format without having to modify it.
Object versions, we're going to stay with the
current version and we're not going to use any of
the older versions within that bucket. We can also
specify optional fields, but we are going to go
with the defaults for the purpose of this
demonstration. Encryption, I'm selecting none at
the moment but it is available.
So click on Save. Now, two things happen. First it
tells you the inventory successfully saved. It can
take up to 48 hours to deliver the first report.
Then you'll see the second thing that's happened
is the bucket policy successfully created. This
policy is applied to the destination bucket that
will hold the inventory. So this just allows us to
have the inventory file created and then written
to the destination bucket. So if I go into our
inventory bucket, you'll see some inventory files
that I created earlier. Here's the file we're
looking for, it's manifest.json. So to actually
start a batch operation, we need to click on
create job. It asks for the manifest format. So
it's S3 inventory report, and then the path to
that manifest.
So let's have a look in our texts that momentary
bucket. And we'll select the latest one we have.
You can put an object version ID, this is for
buckets with versioning enabled. For the purpose
of this demonstration, we're not going to put an
ID in. Now we come to the operations. So we can
choose put, which is just basically a copy, we can
invoke a lambda function, we can replace the tags
on the bucket, we can replace the access control
list, or we can actually restore files. For our
purposes, we're going to choose put. It's asking
us the destination bucket, we'll select browse and
we will select mytechsnip bucket.
Okay, so I proceed the warning and it's basically
that versioning is not enabled on the bucket. I'm
not going to enable that at the moment, we don't
need to for the purposes of this demonstration.
We'll just click that we acknowledge the existing
objects with the same name will be overwritten,
which essentially means that we run this two or
three times those files are going to be
overwritten, okay.
Storage class, we'll choose standard for
frequently accessed data. And encryption. Again,
I'm just going to choose none at the moment, you
can use S3 managed encryption keys or KMS as well.
Access control list, we're not going to change
this, we will stay with the same permissions as
our source object. And tags, we're going to copy
the existing tags. We'll also choose the option to
copy existing meta data.
We need to give this a description. Again, we'll
just leave it at the default, the priority. We can
modify this after numerous batch jobs running and
you can assign them a priority. Then we have a
completion report which will request and the scope
of that will be on all tasks. So the destination
let's just put that number in the inventory
bucket. Now permissions, we're going to use an IAM
role, identity access management role. I have to
find one already called S3 batch role. Now
defining IAM roles is out with the scope of this
particular video but if we go click on View,
you'll see that this is basically just a simple
role I've called the S3 batch role. I've given it
full access to S3.
So we just click on Next. So now we come to the
review page. So it tells us a region and manifest.
We're doing this EU London, the path to the
manifest, now remember the manifest is your
inventory of files. The operation we want to carry
out it's gonna be a PUT operation. So we're going
to copy files to mytechsnip bucket. You'll then
see the additional options we requested our report
after it. And the role we're going to use is S3
batch role. So we just click on create job. There
we have the job there. We can go in, we can view
the details of the job. This gives us this type of
status. So you can see a zero percent complete,
total succeeded zero. What we want to do now is
confirm and run. See our job details. Now come up
and we'll click Run job. So we go back to batch
operations, we just refresh the should run just
shortly. So there you'll see it's now complete,
the total of five objects moved. This is 100%
complete.
The completion report is in our inventory bucket.
So let's go back to our buckets. And let's have a
look in the destination bucket. And there you see
our five files have been moved. So if we go into
our inventory bucket, you'll now see the job
report. And we've downloaded that report. Now
you'll see that it's told us the file name, the
bucket, and its status which is successful. Thank
you for watching this video.