Welcome to this demo lesson
where you're going to get the chance
to experiment with S3 event notifications and Lambda.
Now, you're going to use both of those technologies
to implement a simple
event-driven serverless image processing workflow.
Now, how this is going to work
is we're going to configure S3 event notifications
on an S3 bucket
and we're going to call this bucket the source bucket.
Now, when any images get uploaded to this S3 bucket,
an event is going to be generated.
That event will be passed through to a Lambda function,
which will be invoked.
That Lambda function will gather all of the data contained
within that event object.
It will load the original image
and it will process that image,
in this case, generating five different pixelated versions
of that image and putting them into a processed bucket.
Now, we're going to implement this step by step
so you'll get the chance to understand each
of the individual components.
Now, attached to this lesson is a link,
which contains full text-based instructions
for this demo.
Also included will be any assets
or anything else that you'll need
as you move through the demo.
Now, to get started, we're going to need
to be at our AWS console.
So let's move across to that now.
Okay, so now we're at the AWS console.
Just make sure that you're logged in to an AWS account
with an IAM identity with admin permissions.
And as always, make sure
that you have the Northern Virginia region selected.
Now, the first thing that we're going to do
is create the S3 buckets,
which will store the source images
and the processed images.
So we're going to create two S3 buckets.
So to get started, let's move to the S3 console.
Now, this will either be in recently visited
or you can type S3 in the search box at the top
and then click to move to the S3 console.
Now, we're going to be creating a source
and a processed bucket,
and both of these are going to share the same base name.
So you need to decide on a base name.
In my case, I'm going to use the name pixelator.
So what we need to do is click on Create bucket.
And under bucket name, type your base name.
And again, you need to choose something unique.
So I'm going to use pixelator as my base name,
and then at the end of this,
we'll put hyphen and then source.
So this is the source bucket of the application.
Now, you can leave everything else as default.
Make sure that you've got us-east-1 selected under Region
and then scroll to the bottom and create the bucket.
Now, if you do get an error
that the bucket name is already in use,
then you need to choose a different base name
for the bucket.
So just to reiterate,
this is the base name that I chose.
You should pick something different.
Next, we'll create a second bucket,
and this is going to be the processed bucket
for this application.
So we'll use the same base name,
so I'll click on Create bucket.
Again, I'll type pixelator
and then hyphen, and then processed.
And you should use your own base bucket name
and then just append -processed onto the end.
Again, choose us-east-1 for the region,
and you can leave everything else as default.
Scroll down to the bottom and click on Create bucket.
Okay, so now we've created both the source
and processed buckets for the application.
The next step is to move across
to the IAM console to create the Lambda execution role.
So that's what we'll do.
You can go back to the main AWS console,
and if you've used IAM before,
it will be in the recently visited services,
or you can type IAM in the search box at the top
and then click to move to the IAM console.
Once you're at the console,
you'll need to click on Roles
and we'll be creating a service role
that a Lambda function can assume.
So go ahead and create the role.
Now, I'll have to pick a trusted entity type
and it's going to be an AWS server.
You should see Lambda listed under common use cases,
or alternatively, you can click on this dropdown,
scroll all the way down to the bottom
and it's listed there
but generally, Lambda is a common use case.
So select Lambda.
And this is going to mean that the role
that we create can be assumed by a Lambda function,
and that's what we want.
So go ahead and click on Next.
Now, we're not going to assign any permissions
at this point because we're going to come back
in the next step and create an inline policy.
For now, just scroll down and click on Next.
Under role name, call it PixelatorRole.
Verify that under the trusted entities
you can see lambda.amazonaws.com.
If you can, everything's good.
You can scroll down to the bottom
and click on Create role.
And this has created PixelatorRole,
which can be assumed by a Lambda function.
Now, in the instructions for this demo lesson,
I've also linked a policy document.
So I want you to go ahead and open that policy document.
So this is a JSON document,
which we're going to use to give permissions
to anything which assumes this IAM role.
So it's got some placeholders.
It has REPLACEME four times at the top
and YOURACCOUNTID listed twice.
What I want you to do
is to copy this entire document into your clipboard
and then move back to the IAM console.
Once you're there, click on PixelatorRole,
and then under Permissions
and then permissions policies,
we're going to add permissions
and it's going to be an inline policy.
So we need to go ahead and create an inline policy.
Now, we're going to edit this directly using JSON.
So open the JSON editor
and delete whatever policy document is listed there already
as a template and paste in your clipboard.
Now, don't be concerned
if it lists errors at the bottom.
We just need to update these placeholders.
Now, the first part of this that we're going to update
is the resources under the S3 allow action,
and this is going to allow our Lambda function
to access both the source and the processed S3 buckets.
Now, there are two lines for each bucket:
one which refers to the bucket itself
and another which refers to any objects in that bucket.
So we need to update both of those.
Now, if you do need a reminder of your bucket names,
click in the search box at the top,
type S3 and open that in a new tab.
What we need to do is to get the base name
for our buckets.
In my case, it's pixelator,
but again, this will be unique to you.
So just make sure that you select the base name
of your bucket, and copy that into your clipboard.
And then go back to the IAM console,
select one of the REPLACEMEs in these ARNs
for the S3 component, and then just paste in
what you just copied to your clipboard.
So we're replacing the REPLACEME placeholder
with the base name of your bucket.
And then we need to do that three more times.
So in my case, we've got the pixelator source bucket,
objects inside the pixelator source bucket,
the pixelator processed bucket
and then objects inside that bucket.
Now, the next thing that we need to update
is the account ID placeholder
of these second two statements.
So these allow the Lambda function
to create a log group,
and create log streams and put log events,
and these are required in order
that the Lambda function can log to CloudWatch Logs.
Now, to get your specific account ID,
click on the account dropdown at the top right,
and it's this account ID that you need.
Now, this has got hyphens in.
It's actually a 12-digit number,
and we don't need these hyphens.
Now, if you click on this icon next
to the account ID,
that copies this number into your clipboard
and automatically removes these hyphens.
So go ahead and click that button,
and then highlight the YOURACCOUNTID placeholder
and paste in your real account number.
Now, again this needs to be your account number,
rather than mine.
So do the first one on the second statement here
and then replace the second placeholder,
so your account ID with your real account number.
And once you've done that,
both of these errors should disappear
and then we're good to click on Review policy.
Now, in my case, I got an error.
It appears that there's a leading space in this policy.
So I'm gonna scroll all the way up to the top.
I can see it here.
So I'm going to select the first curly brace.
Put my cursor right before that
and just press Delete to delete that first leading space.
And once I've done that,
I'll go ahead again and click on Review policy
to move forward in the process.
Now, under name, we're going
to call this inline policy pixelator_access_inline.
So go ahead and enter that
and then click on Create policy.
And this means that when this IAM role is assumed
by the Lambda function every time it's invoked,
it gets access to all the permissions that it needs
to perform the task.
So it can access the source bucket
to load in the original image.
It can access the CloudWatch Logs
to log out any status events
and then it can access the processed bucket
to store the output.
So that's everything that we need to do
from an IAM role perspective.
We've created the role and assigned it permissions
so that when the Lambda function is invoked
and assumes this role,
it has access to all the permissions that it needs
to perform the tasks.
Now, the next step is that we need
to create the Lambda function.
So this is the code for the Lambda function.
This is the code which performs the pixelation process
on an image that we upload to an S3 bucket.
Now, just to briefly step you through how this works,
first, we import a number
of relatively standard Python modules.
So these are available
within the Lambda runtime environment.
But we're also importing one called Pillow,
known as PIL, and this is a library,
which allows Python to perform operations on images.
Now, this isn't inside
the standard Lambda runtime environment,
and so we have a number of options.
We can create our own Lambda layer, which contains PIL.
Alternatively, we can use a Lambda layer created
by somebody else which contains these libraries
or we can create a Lambda deployment ZIP,
which we create containing these libraries
and we use this as we upload this code to Lambda.
And that's what we're going to do.
We're going to create our own deployment ZIP.
So we need to provide these libraries
for this function to run.
Now, before we do that, just at a really high level,
this is what this code does.
So first, it reads an environment variable
from the Lambda runtime environment
called processed_bucket.
We have to provide it with the bucket
to store the processed images.
Then it uses the boto3 libraries to connect to S3.
It receives an event,
because remember, this is an event-driven workflow,
so basically, when an object is uploaded
to the source bucket, an event is generated
and this event is passed into the Lambda function
when the Lambda function is invoked.
So Lambda is able to extract
from that event the source bucket
and the object name.
So both of these are available to the Lambda function.
Then what it does is it creates a temporary name
for this object, so a random string.
It sets the path to download the source image to
inside the temp folder of the Lambda runtime environment.
It then uses the boto3 library
to download that original object.
And then it runs the pixelate function five times,
generating five different versions of this image
with various different sized pixels.
Essentially, all this function does
is to decrease the size of the image
and then increase the size again,
which causes it to become pixelated.
We run this five times
and then again, we use the boto3 library
to upload each of those five new pixelated objects
to the processed bucket.
So we need this code for our Lambda function to work
but equally, we also need these additional libraries,
which don't come as part
of the standard Lambda runtime environment for Python.
Now, at this stage, you can either choose
to create your own deployment ZIP
or you can use one that I've already created.
If you're going to the one that I've already created,
then you can just watch the remainder
of this set of instructions.
If you do want to create your own deployment ZIP,
which will be good experience
for both the exam and the real world,
then you should definitely follow along
and do this in your own environment.
Now, I'm running on macOS,
and so the instructions will work fine
on that operating system.
They should work fine on Linux
but if you are using Microsoft Windows,
you'll need to vary the instructions
because Windows uses different commands and utilities.
Okay, so I'm going to go across to my command prompt,
and the first thing that I'm going to do
is create a folder called my_lambda_deployment.
So I'm going to use mkdir and then my_lambda_deployment.
Then I'm going to move into that folder,
and I'm going to create a sub folder called lambda,
and then move into that lambda folder.
Now currently, that folder's empty
because we've just created it.
Now, into this folder, the first thing
that we need to store is the function code itself.
So this is lambda_function.py.
So go ahead and save this entire code
within a file called lambda_function.py in that folder,
and I've included a link to this code attached
to the instructions for this demo lesson.
So now if I do a listing,
I can see this file inside this folder, lambda_function.py.
Now next, we need to get those additional libraries
for the imaging functions of Python.
So how I'm going to do that,
I've attached a link to this lesson
of this website, so pypi.org,
and this is the Python Package Index website,
and this is where you can get downloads
of any Python modules that you might need to use.
So I'm going to get the module that we need
to perform the imaging operations
and it's for Python version 3.9.
So I'm gonna go ahead and search for Pillow,
which is the name of this library.
And we can see right at the top,
Python Imaging Library Fork.
So go ahead and open that.
Now, your version might be slightly different
and that's okay.
The crucial part is for whatever version of Pillow
that you download, we need to get the version
that matches the version of Python
that we're using for the Lambda function.
Now, when I'm creating this demo, that's version 3.9.
So I'm going to click on Download Files,
and then I'm gonna scroll down
and we've got it for different Python versions.
So version 3.8, version 3.7, and so on.
So I'm going to scroll down
and I'm going to look for one for version 3.9.
Now, we can see for version 3.9
that we've got downloads for lots
of different operating systems.
Now, the one that I'm looking for
is for Python 3.9 for Linux
and using the x84_64 architecture.
So in my case, it's this one.
Just make sure that it's the one that has cp39, not pp39.
So it needs to be cp39.
Once you've located that, go ahead and click on it,
and you'll be able to save that file,
and you need to save it into the same folder
that we've just been working in.
So the Lambda folder inside my_lambda_deployment.
And now if I do a listing,
I'll be able to see that I've got the lambda_function
and then I've got this downloaded file,
which contains the Pillow libraries.
Now, this is actually a ZIP-compatible file.
So I need to unzip this file
into the current folder,
and I'll do that using this command.
So unzip and then the name of the file.
And that will inflate that file
into the current folder.
And now if I just clear the screen
and just do an ls, you'll be able to see we've got PIL,
we've got this original download that we did,
we've got the dist-info file,
and then Pillow.libs,
and that's everything we need.
So now I'm going to remove the file
that I just downloaded, clear the screen.
Just do another listing,
and what you should end up with
is the lambda_function.py,
Pillow.libs, the Pillow, then the version dist-info
and then the PIL folder,
and that's everything that we need
to create our .zip file to deploy into Lambda.
Now, what we need to do now is create a ZIP file
with the current working directory
at the top level of that ZIP file,
and we'll do that using this command under macOS.
It's going to create the ZIP file one level up
in the tree, so in the parent directory,
and it's going to call it my-deployment-package.zip,
and it's going to zip the current working directory,
which will include all of these files and folders.
So I'll go ahead and run that command,
and that will create the deployment ZIP.
And then if I do cd .. to move up one level and do an ls,
we can see that we've created
the my-deployment-package.zip file,
and this is what we're going to be using
to create our Lambda function.
Okay, so this is the end of part one of this lesson.
It was getting a little bit on the long side
and so I wanted to add a break.
It's an opportunity just to take a rest
or grab a coffee.
Part two will be continuing immediately
from the end of part one.
So go ahead, complete the video
and when you're ready, join me in part two.