hello everyone on this video we will look at aws lambda functions and lambda functions are
really just actions that you can have triggered by some other actions so the goal of this example is
to drop a file into one s3 bucket which will then trigger a lambda function that will actually push
that uploaded data to our stream that we created in the previous video that means it will go from
the stream to fire host to the new bucket so a lot happening at once but let's let's go through
it step by step now some other prerequisites here and here are the different roles that we have now
two other new services we'll be looking at are sns which is a messaging system and cloudwatch
which will allow us to really monitor and review what's what the functions are actually doing
and it'll help us debug and move our way through this so step one let's create from scratch
we'll give it a name s3 to kinesis
we will use python
and we will allow lambda to create a new role
with all the permissions so that we don't have to worry about that specifically for this example
here's the base view that we're looking at
now if we hop back in here if we look at our roles we can refresh and we'll see
this new role is here we want to make sure this role has full permissions to both s3 to
kinesis and to sns we don't want any issues here in terms of it being denied access to any of
these things that we're going to be working with here in this example so let's give it all those
let's add a trigger in this case s3 it will be our upload bucket so anytime
something gets put in to this bucket it will trigger this function that we create
in the destination we just wanted to have a topic so we'll create a topic here
again call it demo sns create
let's try again it should recognize it now and there it is so let's put that
here as our destination
so as soon as something gets put in s3 it will trigger this function that we work that we are
going to create and send a notification to the sns service all right so let's now scroll down and
take a look at the actual function to test this out let's just put in a message here print success
and that way we can test it all right so let's add a file
just going to add this team csv file that we worked on in the past on previous videos
so now that it just got dropped we expect something to happen we expect this to fire
but in our case nothing's going to happen other than printing that success message
so let's go and explore this check this out we can see there is a log event and there it
is it printed success that's good that means this function was triggered and it executed
but now we want it to actually do something we don't want it to just you know that doesn't
really add any value of printing let's add some sort of logic here and this is where you
can actually get into some very simple etl type of work different transforms whatever
so for the sake of time i've copied and pasted a script that i've pre-written
and i will walk through this a bit with you and also provide this in a github
and essentially what this is doing is it's parsing through the events json object reading the content
capturing each of the columns and then allowing us to create a custom json record
so if you look at the bottom here what we're saying here is as soon as something gets uploaded
to our source bucket it will trigger this lambda function now this lambda function this code that
it's looking at is creating a custom json object and ultimately pushing it to our kinesis stream
now we also have our kinesis stream hooked up to fire hose and fire hose will land it
in another s3 bucket and the idea here is it's going from a raw data to being slightly
transformed and a more a cleaner more consistent json object and then landing in that new s3 bucket
so it's a lot happening but you know once it's set up it's pretty cool how it all works together
now to help show you what this is what this actually looks like i'll print out
these different objects that from the events so we'll say event object the bucket and the csv key
and we'll also print out just the full event so the entire raw data event
and this is something we'll be able to monitor in cloudwatch so that we can see what it looks like
so i'm going to comment everything else out so we'll just have those printed objects and we're
not actually going to process anything just going to print out what the event looks like
so now i just uploaded a new one we should again trigger another lambda function
but this time it's printing out more information so let's go to our logs here and check it out
and here it is so now you can see what this new event record looks like and start to see
what we're trying to parse out you can see the records which is what we're looking at there
traverses down into the bucket name there's s well s3 and then the bucket and then name
so that's kind of how we're we're drilling down into this
object etc this is how we know what actual elements we're actually concerned with here
now we have an error that says source is not defined
and that is from a typo
but you can just understand that that would display the name of the bucket
if i didn't mistype it so now let's save and run the whole thing and see what happens
this data will get dropped into the upload bucket which will
trigger this lambda function this lambda function is going to push the information
to the stream and therefore through firehose and land into a new bucket
okay so we uploaded it this should trigger this this function which again will push it to
our stream which will do the rest we can look at our logs
i think that was the wrong one here we go
so we have a success here we should see something now there we go
so that means that upload was successful successfully triggered
the lambda function that pushed it to the stream now if we go to the other bucket
this should be the cleaned version and there it is it dropped it in here
and this actual record should look a little better than the raw data which if you remember was just
you had a lot of metadata with it this should just be what we want there it is id team
city so it makes it then much easier for a future etl process to filter that out
now this is a very simple example but you can imagine as you get a ton of data happening in
real time this allows you to capture it have a trigger and automatically move it to a new bucket
that's much cleaner and easier to work with so i know that was a lot in one video i hope you found
it helpful please leave comments if there are any questions and happy to help thanks for watching