AWS DataSync makes it simple and fast to move large amounts of data between on-premises
storage and AWS.
DataSync makes it easy for you to move data by automating both the management of data
transfer processes and the infrastructure required for high-performance, secure data
transfer.
I'll demonstrate by using DataSync to move data between my on-premises NFS storage system
and AWS.
First, I'll configure a DataSync agent to read data from my on-premises NFS storage.
To do this, I'll browse to the DataSync web console and select "Get Started" to begin
the Agent creation process.
I've previously installed the DataSYnc agent near my NFS server following the instructions
provided here.
Next, I'll choose a service endpoint for the agent to connect to.
I have a dedicated VPN connection to AWS so I'll choose "VPC Endpoints using AWS PrivateLink.
This will ensure the agent connects to the endpoint without having to traverse the open
Internet.
If I didn't have a dedicated connection to AWS, like a VPN connection or AWS direct connect,
I could choose to use a public service endpoint instead.
I'll use a service endpoint that is already associated with a VPC and subnets in the destination
region, and also choose a security group for the endpoint.
This security group is configured to allow only my DataSync agent access the service
endpoint and noone else.
Finally, I'll provide the Agent's IP address and request an activation key from the agent.
Once I have the activation key, I'll provide a name for my agent and select 'Create Agent'
to finish.
Now that the agent is created, I can setup a task to transfer files from my on-premises
NFS system to AWS, by selecting "Create Task"
First, I'll choose to "Create a new location" and specify an NFS location type for the source.
I'll select the agent I just created, and then provide the IP address of my on-premises
NFS server, as well as the path to the files I want to move.
Next, I'll create the destination location and choose from Amazon S3, Amazon EFS, or
Amazon FSx for Windows File Server.
I'll select S3, choose an existing S3 bucket and provide a prefix that determines where
the files will be located in the bucket.
Finally, I'll I attach a CloudWatch LogGroup to the task and choose detailed monitoring
to see individual log records for all the files processed by the task
I'll start the data transfer task and then wait until it is finished.
The duration of the task depends on the number and size of files transfered, as well as the
available bandwidth.
In my case it takes about 23 minutes to securely move over a million and a half files totaling
over 132 Gigabytes from my on-premises NFS server to Amazon S3.
I also want to transfer a subset of the files from my NFS storage system to an existing
Amazon Elastic File System.
To do so, I'll create a new task, this time reusing the existing source location that
I created for the previous NFS to S3 task.
I can use this same location for as many tasks as I want.
Next, I'll indicate that the destination location is an existing EFS file system and I'll provide
the name of an existing subdirectory on the files system as the destination target.
I'll give the task a name and use the defaults for the remaining task settings.
For this task, I don't want all of the files from my NFS server to be transfered, so I'll
use a simple include pattern as a filter.
Now only files that match this pattern will be moved.
I'll start the data transfer task and then wait until it is finished.
The duration of this task is significantly shorter than the previous task because I only
moved a small subset of the files from my on-premises NFS system.
Finally, I want to transfer some of the NFS files to an existing Amazon FSx for Windows
File System on a recurring schedule.
I'll start by creating a new task, and like before, reuse the existing NFS source location.
Next, I'll select an existing FSx file system as the destination location and I'll provide
the mount path on the file system to use as the destination target.
I'll also provide the user credentials necessary for DataSync access the FSx server.
To finish creation of the task, I'll specify a recurring schedule for the task to run,
once an hour.
I'll use the defaults for the remaining task settings and indicate that only basic logging
be used for this task.
When I start the task, I'll use an include pattern to only move a subset of files from
my NFS storage.
This task will now run once an hour and only move those files that have changed since the
previous execution of the task.
In this demo I've shown how AWS DataSync makes it simple and fast to move large amounts of
data online between on-premises storage and AWS.
For more information, visit the AWS DataSync product page