- [Narrator] Hello everyone.
In this Tech Talk I'm going to introduce you
to a new feature in AWS Transfer Family, SFTP connectors.
Before I discuss the new feature,
let's review the primary use cases customers enable
with Transfer Family.
The core use case is simple file sharing.
Customers use Transfer Family's fully
managed endpoints to provide access to data,
stored in AWS storage over common protocols like SFTP.
This allows customers to easily share files
with external third party users
or applications without refactoring.
When customers formalize this third party data sharing
into a centralized business function
and designate a central team responsible
for administering those exchanges, they often refer
to this practice as managed file transfer, or MFT.
Customers use AWS Transfer Family to build
and deliver cloud native MFT
for their businesses and partners.
Finally, with the addition of AS2 in 2022
customers are now able to exchange EDI documents used
for transactional business to business
or B2B communications with their business partners.
AWS Transfer Families fully managed
AS2 Service provides a modern interface for EDI
enabling customers to simplify business
critical functions such as supply chain operations.
It was AS2 where Transfer Family first
introduced the concept of a connector
which enables customers to use a fully managed
service to enable outbound workflows with their partners.
Now we have extended our outbound capabilities to
include the SFTP protocol.
SFTP connectors is a fully managed service
which simplifies copying data
between remote SFTP servers and Amazon S3.
Some customers refer
to these workflows as client operations.
These types of fully managed SFTP client
operations are most frequently utilized
as part of a managed file transfer architecture.
When an MFT architecture includes both server
and client operations, we call
that a bi-directional MFT workflow
and with the introduction of SFTP connectors,
AWS Transfer Family now supports bi-directional MFT
by providing both a fully managed listening endpoint
for SFTP, FTPS, or FTP,
as well as a fully managed connector for outbound SFTP.
MFT customers are now
able to receive files from third party users
or automated pipelines over these common protocols.
Process these files to refine them
or create new information and distribute this data
to external third parties over SFTP.
Additionally, SFTP connectors support retrieve operations,
which means a connector can connect
to an external SFTP server and pull a file to be stored
in your Amazon S3 buckets.
Driving automated pipelines
like enriching your data lake with third party data.
SFTP connectors will scale on your behalf,
meaning you do not need to do the heavy lifting
of determining decision points like what type
of compute platform to use
or how to build in AZ level redundancy.
SFTP connectors is easy to set up and integrates
with other AWS services offering an API,
which can be integrated with your code pipelines
and your preferred scheduling tools.
As a fully managed service, SFTP connectors
maintain certain compliances based
on industry standards like HIPAA, preventing you
from having to not only build your own solution
but ensure its compliance on your own as well.
And SFTP connectors are cost efficient.
You will only pay for a connector when you use it
and based on the data you move with the service.
The real value of SFTP connectors has realized its scale
when you reduce the burden
of managing and maintaining infrastructure
in complex code and automate these pipelines,
saving time for your operations and development personnel.
While using this fully managed service
to enable key pipelines to deliver data
into your S3 buckets, you break
down the need for data silos.
From S3, your data can be incorporated into AI, analytics,
data lake and many other workloads.
This fully managed service will log all
of its activity into CloudWatch, ensuring
that you have the complete logs you need
for your compliance and governance.
All of this value connectors provides while only requiring
that you pay for the service when you use it
and based on how much data you move with it.
There are no minimum commitments or ongoing fees.
Let's quickly review the architecture
of transfer SFTP connectors
and then we will discuss how to get started.
An SFTP connector is configured
for access to a remote SFTP server.
The connector configuration holds all the information
necessary to make a connection to the remote server
to either retrieve files from the source or send files
to the destination.
The files sent or retrieved are persisted
in Amazon S3 buckets where they can be used
by your critical pipeline, such as analytics,
machine learning, persisted as a long-term archive,
or distributed out to external subscribers as content.
There are many more use cases.
So let's get started with SFTP connectors.
We support deploying SFTP connectors
with cloud formation or RCLI and API,
but a connector is also easily configured in our console.
Within the AWS Transfer Family console
and the connections menu, you can select Create Connector.
We now support creating a connector over either AS2 or SFTP.
When you select SFTP and select next,
you will be able to store a URL for your connector
which may be a host name or an IP address.
You are also welcome to use a colon at the end
and specify a custom port for this external server.
Additionally, you'll want to select a role which has access
to the location in S3 where you want to send
or retrieve your data to or from,
as well as a secret and secrets manager
which will hold the authentication details,
more on that in a second.
Additionally, you can select the logging role
which allows all of the activities
of your new connector to be logged
in CloudWatch for later auditing and reporting.
Additionally, on the connector configuration console,
you are able to store the host key
for your external server to ensure your connector
is reaching the right source
and apply any tags you might want to
for your connector as well.
Finally, you're able to select
or create a secret from AWS Secrets Manager.
This secret will be the source
for any login credentials needed to authenticate
to the external server.
If you click on the dropdown, you'll be allowed to choose
from an existing list of secrets in your account
or if you haven't previously stored this secret
you can link directly to store a new secret
in another window.
Your secret will be stored as another kind of secret
and should contain a username as well as either a password,
a key, or both.
SFTP connectors will support authentication flows
for keys, passwords, or keys combined with passwords,
based on your external servers login requirements.
Once you have configured the details
of your secret and selected next
you'll be able to give your secret a name.
We recommend a naming prefix such
as AWS transfer connector to be able to easily
identify the secret in your connector configuration console.
Once you apply a name and select next,
your secret will be available in the secrets console.
If you're adhering
to the concept of least privileged access,
you'll want to select your secret
and update the role you created
for your SFTP connector to include only
the specific ARN of the secret that contains
that connector's authentication details.
With your secret created,
you'll have all the information you need to
create your SFTP connector.
So SFTP connectors will enable outbound
push architectures.
One customer example includes a payment application
which uses a transfer family SFTP server
to upload its transaction details at the end of each day.
In this use case, the incoming file needs to be compressed
and encrypted, and pushed to an external SFTP server
in a partner's network.
Utilizing AWS Transfer Family workflows,
automated file processing operations are enabled
on the transfer server to
both compress and encrypt the file.
This file is in stage to a specific prefix
and an S3 bucket where an S3 event is
configured to call an AWS Lambda function on put.
This Lambda contains a low-code call
to the API for connectors
which makes the external connection to the partner
SFTP server
and pushes the newly staged file automatically.
SFTP connectors also enable inbound pull architectures.
In this customer example,
a pharma company is retrieving clinical research data
from an external partners SFTP server
and storing it in S3.
In this architecture, the customer uses an event
bridge schedule to periodically call a lambda function
and this lambda function retrieves a list
of files from a DynamoDB table
and calls an SFTP connector to retrieve the files
and store them in S3.
In this customer's use case, it was desired
that a notification be sent when a new file was staged.
So an additional S3 event was configured to
call a lambda, which sent a notification
to an SNS topic when a new file was written
to the configured prefix.
These are just some of the workloads you can enable
with AWS Transfer Family's SFTP connectors.
To learn more or get started with AWS Transfer Family
please visit our site where you can find more content
about our service, our use cases, and SFTP connectors.
Thank you for viewing our Tech Talk
to learn more about AWS Transfer Family and SFTP connectors.