Amazon FSx for Lustre is a service that provides cost-effective, high-performance, scalable file
systems for compute workloads. Integrated with Amazon S3, FSx for Lustre makes it easy to
process S3 datasets with a high-performance file system. In this demo, we’ll see how to create an
FSx for Lustre file system and link it to our S3 buckets, allowing us to keep our S3 objects and
file system contents synchronized automatically. Let’s begin by creating the file system from the
Amazon FSx service page. We’ll select “Amazon FSx for Lustre” and click Next. Let’s call our file
system “fsx-demo” and configure its storage and throughput settings. We’ll give it 2.4 TiB of storage
capacity on a Persistent SDD file system, with 1,000 MB/s/TiB of throughput, which equates to
2400 MB/s. For the VPC Security Group, we’ll choose the “fsx-demo” security group, which was
created ahead of this demo. This security group is configured to allow TCP access on the required
Lustre traffic ports from an existing EC2 instance that we’ll be using to mount the file system. Under
data repository import/export, we'll check this box to indicate that this file system will be used to link
to an S3 data repository. This disables automatic backups, which are not allowed on file systems
linked to S3 buckets, as your S3 buckets are still the primary storage location for the full data set.
In the Logging card, we can choose to log failures and warnings in Amazon CloudWatch Logs so
that we’ll have visibility into any issues that arise when importing from or exporting to the linked S3 bucket.
After clicking Next, we confirm our settings and click Create file system.
And in just a few minutes, the file system is ready. Now, let’s open up our new file system in the console and link up
our S3 buckets from the Data Repository tab. Links to S3 buckets, or prefixes within those
buckets, are called data repository associations (or DRAs), and a file system can have multiple
DRAs. Let’s create two of them. The file system path is the directory on the file system, under the
mount target directory, that will be mapped 1 to 1 with the S3 bucket or the specified prefix on the S3 bucket.
We’ll enter “/dra1” as the directory name and link this directory to the “sample” prefix
in an existing S3 bucket called “fsx-demo-bucket”. We also have the option to load metadata for S3
objects in the data repository after the link is created, which will populate the file system with
file metadata and directory listings for the objects in the S3 bucket. We’ll skip the option to load
metadata for S3 objects in favor of using a data repository task later. This can be advantageous if
you plan to use only a subset of the files under the prefix initially. The default automatic import and
export policies will keep the file system and the S3 bucket synchronized with each other by
automatically importing updates from S3 and automatically exporting updates on the file system
to the linked S3 bucket. We'll leave this configuration as it is. Now that the first DRA is
created, we’ll quickly create the second one. The process is the same as before, except this time
we’ll use “dra2” as the directory name and link it to an existing S3 bucket called
“fsx-demo-results-bucket”. We’ll use this 2nd DRA as a target for some file system operations that
we’ll run later on. Let’s mount our new file system onto an EC2 instance and see how everything looks.
If we open the file system in the console and click Attach to see the instructions on how to do so.
First, we switch over to our existing EC2 instance running Amazon Linux 2 and install the open-source Lustre client.
Then, we create a directory to use as a mount target. And finally, we copy and paste the mount command
from the earlier screen. Now that the file system is mounted, we can see both of our buckets under
the “/fsx” directory; however, they both seem to be empty. This is because we haven’t actually
loaded any of the bucket metadata yet. Let’s switch back to the console really quick to take care of that.
We’ll select both DRAs and click Import Task. For the “fsx-demo-bucket”, we’ll have it import metadata
from the prefix “sample/input”, and for the “fsx-demo-results-bucket”, we’ll have it import
metadata from the “output” prefix. Since we enabled Logging when creating the file system
earlier, we can leave the Completion report option disabled, as any errors in the import process will
be recorded in CloudWatch Logs. The required metadata has now been imported into the file system.
Let’s go back to our EC2 instance. Great! Now we can see all of the files from the
“sample/input” prefix in the “fsx-demo-bucket”. Let’s make some changes on the S3 side.
We’ll upload a new file to the S3 bucket, upload a new version of README.txt,
and delete the DELETE_ME.txt file.
Looking at the file system again, we see the DELETE_ME.txt file is gone, NEWFILE.txt has been added, and README.txt
now says “Hello, world!” instead of the previous sample text. This synchronization works both ways: Let’s move the
README.txt file to the “/fsx/dra2/output” directory, which is linked to “fsx-demo-results-bucket”, and is
currently empty. The README.txt has disappeared from the first bucket and is now in the results bucket.
And finally, if we modify the README.txt file on the file system, we can see our changes reflected when we
download it from the S3 bucket in the AWS console. To learn more about Amazon FSx for Lustre's
integration with Amazon S3, visit aws.amazon.com/fsx/lustre.