Welcome back.
This is part two of this lesson.
We're gonna continue immediately from the end of part one,
so let's get started.
Now, don't worry if any of these commands are unfamiliar,
just do your best to understand what each one of them does.
Again, I've included attached to this lesson,
a full set of instructions
where you can copy and paste these commands
and worst case, you can use my pre-created deployment zip,
which I'll demonstrate using during the next stage
of this demo lesson.
So let's go ahead and move on to the next stage
where we have to create the Lambda function.
So go ahead and move back to the AWS console
and I'm just gonna go ahead and close down
some of these excess tabs and then click to move back
to the main AWS console.
Okay, so we need to move to the Lambda console,
but just before we do, just make sure yet again,
that you do have the correct region selected.
You need to be in us-east-1, which is Northern Virginia.
This is especially important in this step,
because the Lambda function that we're going to create
needs to be in the same region as the S3 buckets.
Now, to move to the Lambda console,
you might find it's in recently visited services,
regardless, you can also click in the search box at the top,
type Lambda and then click to move to the Lambda console.
We're going to be creating a function,
so go ahead and do that.
There are various ways to create a Lambda function,
but we're going to be authoring from scratch,
so select that option.
For function name, choose pixelator.
This only needs to be unique
in the specific region of your account,
so you can also use this name.
For runtime, click and change it to Python 3.9.
For architecture, make sure you select x86_64.
That's the architecture that we're going to use.
Expand change default execution role.
We're going to use the role that we created earlier.
So select use an existing role, click in the dropdown
and then select the pixelator role
that you created earlier in this demo.
Once you've done that, go ahead
and create the Lambda function.
Now, when you're creating Lambda functions
from the console UI,
you can often directly enter the code into this box,
but because we're using a function
which requires additional libraries,
we need to upload the deployment zip.
Now, you can either upload the deployment zip
if you created one in the previous part of this demo
or alternatively, if you want to use my pre-created one,
I've attached a link inside the instructions for this demo
of how you can download that zip
which will be the same as the one you've created,
so you can use either of them.
In either case, to use a deployment zip,
you can click on upload from
and choose either an S3 location
or to directly upload a zip file
and that's the one that we're going to do,
because we've got the zip file to use.
So click on zip file.
Then click on upload.
Next, I'm going to go into the my Lambda deployment folder
that I did earlier in this demo.
So here's the zip file I created earlier in this demo,
I'm gonna select that.
Gonna click on open, you should do the same
and then save that zip file
and that will upload it to Lambda
and use it anytime this function is invoked.
Now, that we've created that Lambda function,
we just have a few bits of configuration
that we need to change
to make sure this function invokes without any issues.
So to do that, click on configuration
and the first thing that we need to do,
is to set an environment variable
so that the function knows where to place
the processed images, so the output images of this function.
So to set that, click on environment variables
and then click on edit and add an environment variable.
Now, the key of the environment variable
is processed_bucket.
And what you need to put in the value field,
is the name of your bucket, the processed bucket.
Don't make a mistake at this point
and put the source bucket or any other bucket,
it needs to be the processed bucket.
So as a reminder, to get that,
if we click in the search box and type S3
and then open that in a new tab,
you'll see we have two buckets, source and processed.
You need to copy down the name of your processed bucket.
Please make sure it's the processed bucket.
This is really important.
So copy that into your clipboard
and paste that in to this value box.
Now, why it's important to get the correct bucket here,
is because this is the bucket where the function
is going to output the five processed versions of the image
that we upload to the source bucket.
The worst possible thing that we could do here,
is put the source bucket here
instead of the processed bucket.
That would mean every time the Lambda function was invoked
once for an image upload,
it would output five images and those five images
would each invoke another Lambda function,
so that would be exponential growth.
It would run five times and then 25 and so on.
So that will have huge cost implications.
So just make sure that you put
the processed bucket in this field and then click on Save.
Then we need to go to general configuration
and click edit and we need to change the time-out
of this Lambda function, which by default is three seconds.
If we upload large images for processing,
it's possible that it will take longer to process
than three seconds
and we don't want the function timing out.
So change this to one minute and zero seconds
as opposed to three seconds.
So one and zero and then click on save.
Now that's the function configured.
The next step is to configure the trigger.
So what causes this function to be invoked?
And we want it to be invoked
based on S3 event notifications.
So anytime an object is uploaded to the source bucket,
we want it to invoke the Lambda function.
So click on add trigger.
In the trigger configuration, click and locate S3.
Now, you're asked for various options,
such as the event type,
so what type of events should invoke this Lambda function?
And leave that as the default,
which should be all object create events.
If it's not that then set it to all object create events.
We don't need to set anything for prefix or suffix.
We do though need to set the bucket,
which will invoke this Lambda function.
So click in this dropdown and select the source bucket.
Again, this is really important
that you pick the correct bucket.
Do not pick the processed bucket
and do not pick any other bucket.
It needs to be the source bucket
you created at the start of this demo lesson.
Once you've set that, you'll actually need to confirm that,
because again, you need to get into the habit
of setting different buckets for the event generation
and the output of the function.
So we've set one bucket for the source
and we've configured it to output to the processed bucket.
So you can confirm that we're using two different buckets.
If we use the same bucket, as I mentioned before,
it can cause exponential function invocation
and that can come with some significant costs.
So we don't need to worry about that,
we've set this correctly.
We've used source here and we've outputted
to the processed bucket
by setting that environment variable.
So this is set to source.
The environment variable is set to processed.
Assuming that's good, then click on add.
Now, that means that we now have
an event notification configured on our S3 bucket.
If we go to the tab that we should still have open
to the S3 console and go inside the source bucket
and then click on properties and scroll down,
you should see that it's now configured
for event notifications
and it's set for all object create events
and the destination type is a Lambda function
and it's the destination function of pixelator.
So make sure that it's configured
in the same way in your environment.
Now, that's everything that we need to do.
All that remains is to test
how this serverless event-driven flow functions.
So to do that, I'm gonna get you to open a number of tabs.
So initially, let's close down this S3 tab
and go back to the main AWS console.
So I'm going to want you to open two tabs to S3.
By now, it should be in your recently visited services
and you can go ahead and open that in a new tab twice
so that you've got two tabs open.
Now, for each of those tabs,
first, I want you to open the source bucket.
In the other S3 tab,
I want you to open the processed bucket.
So that means we can easily see what's in the source bucket
and what's in the processed bucket.
Now, go back to the main AWS console
and you might have CloudWatch
under recently visited services.
Regardless of if you do or not,
go ahead and click in the search box at the top
and type CloudWatch and then open that in a new tab.
Go to that tab.
Now, depending when you're watching this demo,
you might see the Legacy console
or the new version of the console.
The choice is yours which to select.
AWS are constantly making changes to the interface,
so you can either use the old or the new.
In either case, move across to log groups
and just leave this tab on log groups.
Then move back to the main AWS console.
You should see Lambda in recently visited services,
just right, click and open that in a final new tab.
Go to that tab and open the pixelator function.
Now, what this function will do whenever it executes,
is it will log to CloudWatch logs,
and right now we don't have a log group for this function.
That will be created the first time the function is invoked
and logs any information to CloudWatch Logs.
So we can't see anything yet and that's fine.
If you invoke this function a number of times,
you might see that this already exists,
but currently for me, it doesn't.
So what I want you to do,
is I want you to go to the source S3 bucket tab,
click on upload and then I want you to upload an image
into this S3 bucket and I've included some examples attached
to the lesson instructions for this demo.
In my case, I'm going to use the nohair.jpg,
which is a picture of a hairless cat.
There's also Cupcake or you can use your own image.
In my case, I'm going to use nohair.jpg,
so I'm gonna open that
and then I'm going to upload that into this S3 bucket.
Now, again, before we click,
we're using the source bucket for this application
and what I want to do is immediately
after we click on this upload button,
I want to move across to CloudWatch logs.
So click on upload.
It's gonna take a few seconds to upload that.
Once it has, you can click on close
and then move to the CloudWatch tab
and once you're there, click on this refresh icon.
What you'll see after a few seconds,
is it's created a log group called /aws/lambda/
and then the name of the Lambda function,
which is pixelator.
So go ahead and click on that log group
and the first time that this Lambda function invokes,
you'll probably have a single log stream.
Now, you're gonna have a log stream generally
every time this function invokes.
If you invoke a function multiple times in a short duration,
it's possible they'll be in the same log stream,
but generally, you're gonna have a log stream
every single time that we invoke this Lambda function
by uploading an image.
Now generally, whenever we are diagnosing Lambda functions,
we want to be looking at the latest log stream
inside this log group.
In my case, this is the only log stream
inside the log group and so we're going to go ahead
and open up that log stream.
Now, in my particular case,
we can see the request is successful.
We've got the start event.
We've got the end event
and we can see that the Lambda function
took two and a half seconds in total
to both load the original image and generate the output.
Now we see at the top here,
I've got the Lambda function to print a copy
of the event data structure which it receives.
And so if we just expand this for a moment,
this is the JSON that the Lambda function will receive
when it's invoked.
And you can see that it contains
various bits of information.
Now, the important things that I want to draw attention to,
is that the Lambda function receives
within this event structure,
a copy of the bucket that this event was generated from.
So in this case, it's pixelator-source.
It receives the ARN of that bucket
and it also receives the object key.
So in this case, object key is nohair.jpg
and this is the source image that this Lambda function
is going to process and based on this source image,
it's going to generate five pixelated versions of this image
and it's going to output those into the processed bucket.
So it looks as though this serverless application
is working as we expect.
So now let's have a look,
let's move to the processed bucket.
Again, we'll need to hit refresh
and we should see five different versions of this image.
So we've got the name nohair.jpg,
we've got eight by eight, 16 by 16,
32 by 32,
48 by 48
and 64 by 64.
Now at this point, what we're gonna go ahead and do
is select all of these images and then click on open.
What you'll probably see, is your browser
will offer to save each of these images,
so we're gonna go ahead and do that.
So save each of these images to your local machine
and if you go ahead and open those images,
you'll see different levels of pixelation.
So as a reminder, this is the original image.
This is a cat with no hair or an alien, one of the two.
But what we'll see if we open each of the output images,
we'll see pixelated versions of this image.
So this is the eight by eight version.
So the hairless cat or alien is barely distinguishable.
Next, we'll move to 16 by 16,
which you can see has smaller pixels
and slightly more detail.
This is the 32 by 32,
so we can see more detail, smaller pixels.
This is 48 by 48 and this is 64 by 64.
So each of these images has more and more detail
versus the original source image.
And all of these images have been generated
by the single Lambda invocation.
It's an event-driven image processing workflow.
The original image is loaded using the event data
that's passed into Lambda when it's invoked,
the image is loaded from S3 processed
and outputted onto a destination or processed bucket.
And this all occurs in a way
which means that there's no cost
unless this workflow is being used.
And even then, AWS includes generous free tier allocations
both for S3 and for Lambda functions.
So it's rare that this type of workflow
will incur any charges.
Now, you can go ahead and try that with different images.
You can experiment with it
and just explore the overall architecture.
Just as a reminder, this is what it looks like.
So we've got all of these different components
and you should use this as an opportunity
to understand each one of them.
Now, at this point, all that remains
is for us to clean up the account
and put it back into the same state
as it was when we started this demo.
So to do that, go ahead and move back to the Lambda console,
open up the pixelator function
and then delete that function.
So that's the first step.
Go ahead and delete that Lambda function.
Then move to the IAM console
and we need to delete the Lambda roll that we created,
so click on roles, locate pixelator role,
click on it and then delete that role.
You'll need to confirm that process,
go ahead and do that, which will delete the role.
And then the last two steps are to empty
and delete both of the S3 buckets.
So move back to the S3 console.
Let's start with the source bucket,
so select it and then click on empty.
You'll need to confirm that and then click on empty.
Close down this dialogue.
Make sure it's still selected, click on delete
and again, you'll need to confirm that
and then delete that bucket
and do the same process with the processed bucket.
So select it, empty it,
confirm that, that'll empty the bucket,
close this dialogue down.
Make sure it's still selected and then click on delete.
Confirm that again and delete the bucket.
And that's the account back into the same state as it was
at the start of this demo.
So that's it for this demo lesson, I hope you've enjoyed it.
You've implemented a pretty simple
event-driven serverless image processing pipeline,
which uses S3 event notifications together with Lambda.
Now, this is a pretty simple example,
but it can be used to create much more complex workflows,
potentially just using Lambda or even using step functions.
In this demo though, we've kept it simple
and just implemented something nice and easy,
which did use a custom deployment zip containing libraries
which are not contained
within the Lambda runtime environment.
So this has given you a little bit of experience
at creating a custom deployment zip for use with Lambda.
At this point though, that is everything I wanted to cover.
Go ahead and complete this video.
And when you're ready, I'll look forward to you joining me
in the next exciting theory or demo lesson.