Amazon Elastic File System (EFS) provides a simple, scalable, and fully-managed shared file
system for use with AWS Cloud services and on-premises resources. Whether you’re
developing a new containerized application or modernizing an existing monolith, there are many
types of applications where a shared persistence layer is an essential ingredient in achieving a
resilient and durable architecture. These include stateful, long-running applications like content
management systems, whose files change frequently as editors publish new content or
administrators install plugins, and applications that need to share data with one another, possibly
spanning multiple availability zones, such as machine learning frameworks or collaborative data
science workspaces. Amazon EFS is the natural choice for these types of use cases. As a
fully-managed service, it’s simple to set up, and it will scale to petabytes of capacity and several
gigabytes per second of aggregate throughput automatically as you add and remove data. And if
your workload requires higher throughput than that, you can achieve even more by using
Provisioned Throughput! When you use Amazon EFS in conjunction with your containerized
applications on AWS, you can feel confident that your shared storage solution is as elastic as the
container infrastructure that depends on it. Let’s take a look at an example scenario of one of
these use cases: We have an Amazon Elastic Container Service, or “ECS”, cluster running a mix
of tasks using both ECS on EC2 and AWS Fargate. The containers are all running a web
application called Cloud Commander. We’ll use it as a hypothetical production application for today.
An application such as this might need varying degrees of read and write access to shared
storage, and the familiar interface of a file manager will allow us to visualize these ideas
more easily. The ECS on EC2 version of this application is running behind an Amazon
Application Load Balancer, and the Fargate version is running behind another. If we open up
the first load balancer URL, we’ll see an empty directory where we can drag-and-drop a file to
upload it, and then double-click on it to view it. Notice though, that if another user, represented by
this other browser window, loads the same application, the file we just uploaded isn’t there
anymore. The other version of the application running on Fargate has the same behavior, and
neither application can see what has been uploaded to the other. What’s happening here
is that the files we’re uploading aren’t being saved to a common location - they’re being saved locally
to whichever container happens to process the upload request, so there’s no guarantee that
different users would see the same files. That’s definitely not what we want here - so let’s fix it!
In this demo, we’re going to use Amazon EFS to create a shared filesystem, and then attach it to
our existing Amazon ECS and AWS Fargate services. This will enable our example file
manager application to interact with the same files, no matter which container handles our
requests. Once we’ve got EFS hooked up to both versions of our application, we’ll then explore how
AWS IAM Authorization and EFS Access Points can simplify configuring more granular access to
our data for different scenarios. If you’d like to try this out on your own later, you can use the
AWS Cloud Development Kit, or “CDK”, to deploy the initial state of this demo environment to your
own AWS account. A link to the sample code repository for that is available in the video
description below. But for now, let’s head over to the AWS console, open the Amazon EFS
dashboard, and create our filesystem! Creating a basic EFS filesystem is easy. First, we
choose the VPC the filesystem will be created in and its mount targets. In order for all of our
containers to be able to access it, we must create a mount target in each Availability Zone. As a
general best practice, we’ll choose the private subnet for each Availability Zone for our mount
targets. We also need to select a security group. The default security is pre-selected for us, but
we’re going to choose this “efs-demo-fs” security group that was created by the AWS CDK sample
code so that we can limit access to the filesystem to just our containers. Next, we’re shown
several different filesystem settings that we can configure. We’ll leave most of them alone for this
demo, but let’s give the filesystem a name and enable encryption of data at rest using the default
AWS Key Management Service key for Amazon EFS before moving on. We’ll come back to the
additional client access settings later on in this demo. For now, let’s continue, and then click
“Create File System”. Our EFS filesystem was successfully created! Now let’s go make it
available in the Cloud Commander application running on our cluster! To do that, we open
the Task Definitions link in the Amazon ECS menu, select one of the task definitions, and click
Create new revision. First, we’ll scroll all the way down to the bottom and click on the name of the
existing volume to modify it. Instead of Bind Mount, we’ll choose EFS as the volume type and
then select the File system ID for the EFS filesystem that we just created. We’ll get into
Access points later, so let’s leave that alone for now. The root directory value tells us which
directory in the EFS filesystem will be mounted as the root directory inside the ECS host. Leaving it
as just “/” will mount the entire EFS volume’s contents. We’ll enable Encryption in transit, and
finally, we’ll leave EFS IAM authorization disabled for the time being and come back to it later.
Since we already had the Bind Mount volume set up earlier that we changed to an EFS volume, we
don’t need to make any changes to the container definition. For reference, if we were adding a
completely new volume, we’d now need to go up to the list of container definitions, click “cloudcmd”,
and add a new mount point under the Storage and Logging section. That’s not the case here, though,
so we’ll cancel out of that, scroll down to the bottom, and click “Create” to create the new task
definition revision. We can start deploying this by choosing the Update service option within the
Actions dropdown menu. Please note that at the time of this recording, it’s necessary to select
1.4.0 as the platform version. We can otherwise leave the rest of the inputs alone as we click
through each of the screens, and then keep an eye on the deployment status until ECS has
finished starting our updated tasks and stopping the old ones. This process is identical for both
ECS on EC2 services and as well as for Fargate services, so we won’t walk through it step-by-step
in this video. Same as with the previous walkthrough, we just need to wait until ECS has
finished deploying our updated tasks and cleaning out the old ones, and then we’re ready to try
everything out. Now that the cluster is showing only the updated tasks running, let’s
open up both of our browser windows from earlier and see what happens. The first thing we notice is
that the test files we uploaded at the start of the demo are gone. This is expected, since the file
was previously being saved on a data volume that was no longer available once the container was
shut down, which is one of the things our new EFS filesystem is going to help us avoid. Let’s try
uploading a few new files. OK, we can upload files and view them when we double-click. After
refreshing the listing a few times, and pretending to be a different user using a different browser, we
can see our files are still there. Now let’s see check and see what happens if we refresh the
window using the other application running on AWS Fargate. Great! Our files are still there
too. By simply creating an EFS filesystem and mounting it to use as a volume in our containers,
we have persistent, shared storage accessible by both of our example Cloud Commander
applications running in multiple containers, and we know that our data will still be there even if we
terminate these containers and launch new ones. But this is a pretty basic use case. In a
real application, we’re going to have several different services that don’t all need the same
level of access to the same filesystem. The traditional approach of relying on POSIX
filesystem permissions can make this very difficult to configure and may even require creating
multiple EFS filesystems to ensure that everything is locked down correctly. Instead, we’re going to
take a moment to dream up a very complicated scenario, and then see how easy it can be to set
up if we incorporate IAM Authorization and EFS Access Points into our solution! So, what’s our
scenario going to be? Well, a service is going to have one of three types of access to a filesystem:
read-only, read/write, or none at all. We have two services in our demo, so let’s have our
complicated scenario involve all of these combinations! We’re going to set things up as
follows: (1) each service will have its own private storage, shared with containers of its same type,
that only it can read and write; (2) each service will have a location it can write to, that the other
service can read from, but not write to; and (3) both services will have access to a common
location that they can both read to and write from like they have now. OK, let’s go back to EFS
and configure this! We’ll create the Access Points first. An Access Point can be thought of as
an application-specific entry point into a certain location in an EFS filesystem, this lets us manage
things like data access and user identities more easily. We’re going to create some Access Points
for each of the three scenarios mentioned earlier. First, let’s configure the private Access
Points for each of our services that only they are able to read from and write to. Let’s start with
Fargate. We’ll call the Access Point “FargatePrivate” and have its root directory begin
at “/private/fargate”. We also have the ability to specify a POSIX User and Owner. The values
entered under POSIX User will tell the mounted filesystem who we are, and the values under
Owner will define the ownership and permissions for the directory we specified when it gets created
the first time we connect. To keep things easy to remember, let’s use “7000” as the Owner ID and
Group ID for anything related to our Fargate tasks, since the word “Fargate” has seven letters. As for
permissions, we only want our Fargate tasks to be able to read from and write to here, so let’s set the
permissions to 750. The configuration for “EcsPrivate” is the same, except we’ll input “ECS”
instead of “Fargate”, and we’ll use “3000” for the IDs, since “ECS” has 3 letters. Next, we’ll add
Access Points for each service that they can write to and the other can read from.
And finally, we’ll add a common location where both services can read and write freely, and then click “Save
access points” to create them. It’ll take a few moments after saving for them to become
available, at which point we can define how access is going to work. Let’s scroll back up and
set our filesystem policy. There’s no policy in place right now, so our filesystem is using the
default policy of allowing any clients with network access to connect, which is why we were able to
hook up the filesystem to our containers so easily before. To get our policy started, let’s check off
these three settings to configure new defaults: “Disable root access”, “Enforce read-only access”,
and “Enforce in-transit encryption”. After clicking “Set policy”, we can see the JSON tab has been
populated with the familiar IAM-style policy configuration. Now we need to configure the
policy for our own scenarios. We’ll do one together step-by-step, and then fast-forward
through the rest. If you’d like to review any individual policy statement more closely, a copy of
the entire policy is available at the same repository linked in the video description. Let’s
configure “FargatePrivate”. We want to allow our Fargate service, or more specifically, the Task
IAM Role that it’s using, to be able to mount and write to the FargatePrivate access point that we
defined below. We can get that Task IAM Role’s ARN from the Fargate task definition.
It needs to be able to mount and write to the filesystem, and we’ll specify that this policy statement applies to
that “FargatePrivate” access point.
Since it needs this same level of access for the “FargateShared” scenario, we’ll go ahead and
add that access point here as well. The final policy looks like this, and we can use the “Sid” values to
help us tell at a glance what each section is for. Let’s save this policy and go back to our task definitions to
make the access points available. We’ll quickly run through how to create the new task definition
revision for the Fargate application. First, let’s add the volume for the “common” access point, making
sure to also check the “Enable IAM Authorization” box so that the filesystem policy we created earlier gets
evaluated. Next, we’ll do the “private” access point for Fargate,
and finally, we’ll do both of the “shared” scenarios - Fargate and ECS on EC2.
We don’t need our “files” example volume anymore, so we’ll remove that before going to the container definition
to add all of these new mount points. Note that since our sample application container uses the “/files” directory
as its root, we need to be sure to mount our access points within that “/files” path in order for them to be accessible.
There’s the mount point configuration for “common”...
both “shared” volumes...
and the “private” volume just for the instance of our application that’s running on Fargate.
The process to do this for the ECS on EC2 application’s task definition is identical, aside from the names, so
we won’t go through that step-by-step in this video. After creating both of the new task definitions and
updating the services, though, we can open up our two browsers again and see that they each have three
directories - “common”, “private”, and “shared”.
They can both read and write from each other’s files under the “common” directory...
read, but not write, each other’s files under the matching “shared” directories...
and they each have a private space for storing files under the “private” directory that the other can’t
interact with at all.
Support for Amazon EFS on both AWS Fargate and Amazon ECS for EC2 is available in all AWS regions.
To learn more about using Amazon EFS with your containerized applications, including documentation,
blog posts, and the sample code from this demo, please visit the links in the video description below.
Thanks for watching!