- [Instructor] Hello, everyone.
Today I'm gonna show you how you can automate
the PGP file decryption process
for any files uploaded to Amazon S3 or Amazon EFS
via the AWS Transfer Family service
using the managed workflow feature.
For the purpose of this video,
I will assume that everyone is already familiar
with what the Transfer Family service is
as well as what Transfer Family managed workflows are.
If you're unfamiliar with either of these topics,
please refer to the video that is linked
in the description below,
as it provides a great summary
regarding what the Transfer Family service is
as well as what Transfer Family managed workflows are.
Quick overview of what I'll be covering today in this video,
I'll start by going over a short refresher
of what PGP is and its typical use cases.
I'll walk through each step of the managed workflow
and explain how the PGP decryption Lambda function works.
I'll go over the custom Transfer Family identity provider
that I'll be implementing in this video
and explain why you may want to use this
over traditional Transfer Family managed users.
I'll then explain how all these different pieces
fit together using an architecture diagram.
And I'll finish out the video by showing you how
to configure the solution within your own AWS environment
as well as provide a quick demonstration
of what the managed workflow looks like in action.
For those of you who may be unfamiliar with PGP,
it stands for Pretty Good Privacy.
And it's an encryption system used
for sending encrypted emails and encrypting files.
PGP is mainly used today to help organizations
meet various security compliance requirements
and to ensure that only the intended parties
have access to specific files.
The main challenge, however, is that PGP encryption
and decryption add a layer of operational complexity.
Today I'll show you
how you can remove this undifferentiated heavy lifting
by automating the PGP decryption process
using a Transfer Family managed workflow.
So the managed workflow contains four steps in total,
one of which will manage the PGP decryption process.
The first step is to copy any file that is uploaded to S3
via the Transfer Family to an S3 prefix named archive.
The next step then tags the file that was copied
to the archive prefix as archived.
Then the workflow orchestrator
invokes the custom Lambda function,
which performs the actual PGP decryption process.
And the Lambda function then moves the decrypted file
to an S3 prefix named decrypted files.
Lastly, the originally uploaded file gets deleted from S3
since it was archived in step one
and decrypted and moved to the decrypted files prefix
in step three.
If any errors occur during a workflow's execution,
the exception-handling steps that you specify are executed.
Several things can cause these exception-handling steps
to be invoked, such as if the file decryption process fails
for any reason, if the time threshold for a step is reached,
or if a step fails for other reasons,
such as inadequate S3 permissions on a copy step.
Regardless of the reason which causes the workflow to fail,
the exception-handling steps for this workflow
will copy the file that failed
to an S3 perfects named failed decryption,
tag the file with the status of failed decryption,
and then delete the originally uploaded file.
The way the custom PGP decryption Lambda function works
is by first, it downloads the newly uploaded file from S3.
It then checks if the file is encrypted or not.
If the file is encrypted,
the Lambda function pulls the private key required
for decryption from secrets manager
and performs the decryption.
Once the file has been decrypted,
it is moved to the S3 prefix named decrypted files
and a success message is sent back to the managed workflow
using the send-workflow step state API.
Once the workflow receives the success message,
it continues with the next step of the workflow.
If the file decryption process fails for any reason,
the Lambda function reports this failure back
to the managed workflow,
and the exception-handling steps are invoked.
In this demo,
I will use a custom secrets manager-based identity provider
for Transfer Family user authentication.
However, please note that this custom identity provider
is optional, as the PGP decryption Lambda function
will work the same regardless
of how you choose to authenticate users.
The reason I'm using this custom identity provider
is because it provides several extra features
that Transfer Family managed users do not currently support,
such as username and password authentication
and source IP address checks.
Now that you're familiar with each step of the process,
let's put it all together with this architecture diagram.
First, the user sends an authentication request
to the Transfer Family server endpoint,
which is then forwarded
to the custom Lambda function identity provider.
The Lambda function will then check
if the authentication credentials provided
match what is stored within secrets manager,
for example, a username and password.
If these credentials do match,
then the user is authenticated
in the Transfer Family server.
Step four is where the main part
of the project actually begins when a user uploads a file
to S3 via the Transfer Family.
Once the file is uploaded,
the Transfer Family managed workflow gets invoked,
and it goes through the four steps
that I discussed earlier in this video.
For the purpose of this video,
I'll be going through this configuration process
rather quickly.
For detailed step-by-step instructions
on how to recreate this project,
please refer to the read-me file
on the AWS Samples GitHub Repository for this project,
which is linked in the video description below.
So the first step of the project
is to clone the GitHub repository
into our CloudShell environment.
After that, we'll change into the new directory
and give the setup environment Bash script
executable permissions.
Then we'll run the setup environment Bash script.
So right now, this Bash script is downloading the package
and binary file which are required
to make the custom PGP decryption Lambda function work.
After it downloads all the required files,
it will zip them up and push them to AWS as a Lambda layer.
If you're not familiar with what Lambda layers are,
they're used to provide a convenient way
to package libraries and other dependencies
that you can use with your Lambda functions.
Now that the script is finished building my Lambda layer,
I'm going to deploy the first CloudFormation stack
by copying this command from the read-me
and pasting it into CloudShell.
First, give the stack a name.
I'm going to name mine PGP decryption stack.
Then select a region.
I'm just going to press enter for this one
to leave it as default.
Now, enter a name for the S3 bucket
that the stack is going to create.
I'm going to name mine PGP decrypted files,
then respond to the rest of the prompts.
For a full list of all my responses
to the stack-creation parameters,
refer to the read-me file on the GitHub repository.
Right now, this stack is creating the required
IAM rules S3 bucket Lambda function,
PGP private key secret, and managed workflow.
Now I'm gonna deploy the second CloudFormation stack
which will create a Transfer Family server
with the custom identity provider I mentioned earlier
in this video.
First, I'll create a new directory named temp
and change into it.
Now I'm going to grab the link
to download the CloudFormation stack
and paste it into CloudShell.
Now I'll unzip the folder
containing all the files I need to deploy.
And then I'll run the command sam deploy -guided.
Now I'll give the stack a name.
I'm going to name mine Transfer Family server,
but you can name yours whatever you'd like.
After that, accept all the options
as default simply by pressing enter.
Same as with the last stack deployment,
refer to the GitHub read-me file
for a full list of my responses
to these stack-creation parameters.
Now that we've ran the setup environment Bash script
and deployed both CloudFormation stacks,
I'm going to create a new Transfer Family user
for the custom identity provider.
So I'll navigate to the secrets manager console
and select store a new secret.
Then I'll select other type of secret,
and I'll create four different key value pairs
for password, role, home directory details,
and home directory type.
Please note that the secret key names are case-sensitive,
so be sure to name the keys exactly the same as I do.
To get the value for the role field,
I'll navigate to the CloudFormation console
and select stacks.
Then I'll select the name of the first stack we deployed.
Then I'll select outputs.
Now I'll copy the IAM roles arn,
navigate back into secrets manager,
and paste that into the value field for my role.
Now for the home directory details field,
I'll navigate to the read-me file,
copy the secret value from the table,
and paste it into secrets manager.
Now I'll switch back to CloudFormation once more
to grab the name of the S3 bucket that I created.
I'll copy the name, and I'll replace the part of the secret
that says insert S3 bucket name
with the name of my S3 bucket.
Then I'll change to where it says insert username
and insert whatever I want the username to be.
For the purpose of this video,
I'm going to name my user test user.
Lastly, for the home directory details field,
I'm going to paste in the word LOGICAL
in all caps for the secret value.
The reason I'm doing this is to create a logical directory
for my Transfer Family user.
This logical directory can be used
to set the user's route directory
to a desired location within my storage hierarchy.
By implementing a logical directory structure,
I can avoid disclosing absolute paths
and S3 bucket names to any end users.
If you'd like to learn more about logical directories
with Transfer Family users,
please refer to the link in the description below.
Now I'll click next, and I'll give my secret a name.
For this custom identity provider, the secret must be stored
in the format of server ID/username.
To get the Transfer Family server ID,
I'll go to the CloudFormation console,
select the name of the Transfer Family server stack,
and then select outputs.
I'll copy the server ID
and paste it into the secrets name field.
Then I'll add a backslash followed
by whatever I assigned as the username.
So in my case, I will enter test user.
All right, now I'll press next,
leave all the options as default,
press next again, and then press store.
Now that I've got my user configured,
I need to paste in my PGP private key
into the secret that was created
by the CloudFormation stack I deployed earlier
in this video.
So I'm gonna open up my PGP private key, copy it,
go back into secrets manager,
replace the text that is serving as a placeholder,
and paste in my private key and select save.
The last step I need to complete is
to attach my newly created Transfer Family managed workflow
to my Transfer Family server.
To do this in the Transfer Family console,
I'll select the server, then select edit,
then scroll down to where it says managed workflows.
And I'll select the newly created workflow.
For the managed workflow execution role,
I will select the role
that's named PGP decryption managed workflow role.
I'll then scroll down to the bottom and select save.
Now that I've finished the configuration process,
let's see how the managed workflow looks in action.
To start off, I'm gonna open up a PGP encrypted file
stored locally on my machine.
As we can see, the file is indeed encrypted,
and no useful information can be gained from this file.
Now I'm gonna connect my Transfer Family server.
So first, I'm gonna copy the endpoint ID
from the Transfer Family console.
Then I'm gonna open up my file transfer client.
I'll be using WinSCP,
but you can use any file transfer client
that you would like.
The process will work the exact same.
So now that we've got WinSCP open,
I'm gonna paste in the endpoint ID.
The method for authentication will vary
based on how you deployed your Transfer Family server.
Since I deployed the Transfer Family server
with a custom identity provider,
I'll be using a username and password to authenticate.
Now that I've successfully connected
to my Transfer Family server,
I'll upload the encrypted document that I opened earlier
as well as some other test documents.
Now that these files have been uploaded,
let's go into the S3 bucket
to see how the managed workflow has handled these files.
Right off the bat, I can see the three different prefixes.
Let's look at the archive prefix first.
Once I click into this prefix,
I see another preface that is named after the username.
This is done in order to sort files
that are uploaded by different users.
Once I click into this prefix,
I can see all the files that I uploaded
have indeed been archived.
Now I'm going to navigate into the decrypted files' prefix.
And again, we see the username prefix.
Once we click into that, we can see the file
that was initially encrypted has been decrypted.
Now I'm going to go ahead
and download this file and open it.
And as you can see,
the file has indeed been successfully decrypted.
Lastly, I'll look at the failed decryption prefix,
where I can see two files that I uploaded
that are purposely intended to fail the decryption process.
The first file was encrypted with a different key pair
than what I have stored in secrets manager.
The other file was actually never encrypted.
It was just a .TXT file that I renamed to .GPG.
Since both these files are unable to be decrypted,
we see them here in the failed decryption prefix.
Now I'm going to quickly walk through
what the CloudWatch logs look like for this process.
To access these logs within CloudWatch, select log groups.
Then select the specific Transfer Family server
and then the specific workflow.
So we can see that the workflow execution started.
Then it copied the file to archive.
After that, it tagged the file for archive.
Then it invoked the custom PGP decryption Lambda function.
And lastly, it deleted the original file.
Now I'm gonna quickly shift over to a different log
to show what the exception-handling steps look like.
As soon as the workflow orchestrator receives notification
that a step in the workflow has failed,
it executes the exception-handling steps,
as we can see here.
That completes today's video.
To recap, today we went
over a specific Transfer Family managed workflow
to automate the PGP decryption process.
We walked through the process of how to configure this
within your own AWS environment
and finished up by providing a demo
of what the workflow looks like in action.
Thank you all for watching.
I hope you have a great day.