- Hello, my name is Devabrat Kumar,
and I'm a Product Manager on Amazon S3.
In this presentation, I'm excited to talk to you
about Mountpoint for Amazon S3.
Mountpoint for Amazon S3 is a new open-source file client
that makes it easy for Linux-based applications
to connect directly to Amazon S3 buckets
and access objects using file APIs.
Using Mountpoint for Amazon S3,
you can mount your Amazon S3 bucket
on your compute instance and it presents S3 objects
as files in the local file system.
It translates local file system API calls
to rest API calls on Amazon S3 objects.
With Mountpoint for Amazon S3,
you can achieve high single-instance throughput
to finish jobs faster.
You can reliably scale up and down over thousands
of instances and read large objects processing petabytes
of data in S3 data lakes.
With Mountpoint being generally available,
it is ready for use with your production workloads.
It comes with AWS support.
This means customers with AWS business
and Enterprise support plans get 24/7 access
to cloud-support engineers
to support their use of Mountpoint.
Mountpoint for Amazon S3 is ideal for large-scale
out workloads that process petabyes
of data using random and sequential read operations
and sequential write operations for creating new files.
Common use cases include machine learning training,
autonomous vehicle simulation,
genomics analysis, and image rendering.
These applications perform read-heavy access on hundreds
of terabytes to petabytes of data in S3 data lakes,
and require S3's high elastic throughput
as they scale up and down over thousands of instances.
While they read large files over one onboard instance,
they write to a file from a single instance at a time.
This means they do not need shared file system features
like file locking.
However, if you are looking to run data lake applications
that need to collaborate or coordinate
on shared data across instances or users,
we recommend using Amazon FSx for Lustre with Amazon S3.
But if you're running a data lake application
that needs to read and write large objects
without shared file system features
like locking or POSIX permissions,
we recommend using Mountpoint for Amazon S3.
Next, let's see Mountpoint in action,
and see how it works for your Amazon S3 buckets.
As you can see, I have Mountpoint
already installed on my EC2 instance.
The first thing that I do is to create a directory
where I'm going to mount my S3 bucket calling it mntdir.
Next, I'm going to use Mountpoint
to mount my S3 bucket at this mount location.
With Mountpoint running in the background,
my directory now reflects the content of my S3 bucket.
Here we can see that Mountpoint creates
a directory structure using the keys of my S3 objects.
Now, I can use common Linux command line tools
with my S3 objects.
For example, I can create an archive file
using the tag command,
so I can say tar, let's call it test.gz.
And I'm going to use files from my S3 bucket
which is mounted at this new directory location
that I have created just now.
So I have a bunch of PDFs here,
and I'm going to create an archive file called test.gz.
Behind the scenes, Mountpoint is streaming
my objects directly from S3,
so that I don't need to do any manual work
like copying my files down to my local storage first.
I can just run my command as if the files
were already on my local disc.
Let's quickly look at the archive file that I just created.
So there it is.
Next I can write this file
to the bucket as if it is a local file system.
So I can do cp test.gz and then the mount location
where I want to write it to which is just my S3 bucket.
I can quickly verify using AWS CLI.
So I'm just going to do a list command on my S3 bucket
and see that this file was actually written there,
so there it is.
With mount one, customers can run business-specific tools,
like ffmpeg while accessing S3 as a local file system.
For example, I can run ffmpeg and then let's say,
I want to downscale a file
that I have stored in my S3 bucket.
I'm choosing this machine-learning session from one
of our previous reinvents, and I'm going to scale it down.
Let's say to 480 is to 320,
and I'm going to save the result
on my local instance in this case.
So let's just call it scale result.mp4,
and let's just run this command.
So again, behind the scenes,
Mountpoint is streaming my object directly from S3,
so that I don't need to do any manual work
like copying it down.
And I can see the result here,
the file that was created just now.
Thus, we see you can use Mountpoint
to access your S3 bucket as a local file system
and use standard Linux commands
as well as business specific tools on it.
Finally, I'm unmounting my bucket from this location,
and you can use it for any other purpose.
With this, we conclude our demo today.
We are really excited as you get started
with Mountpoint, and we are looking forward to hearing
from you on how you use Mountpoint
for your production use cases.
Thank you.