Welcome to another episode,
I'm Adrian De Luca.
Today we're going to talk about
bulk data transfer into the cloud.
This could be for log or transactional data,
for analysis, post-processing
or content distribution
and users could be literally coming from
anywhere around the world.
Well, it turns out that S3
or the Simple Storage Service
is perfect for this.
Here's a common interaction:
you've got users coming in over the internet,
uploading their files via HTTP PUT
with those files being stored as
objects in a S3 bucket.
Now, there's a couple of considerations.
First: How do you manage identity and access?
Second: How do you secure the data once it's there?
Third: How do you implement detective controls?
You know, the stuff that the security folks
tell us to do for auditing and logging.
And finally: How do you onboard new users?
Identifying your users and what they had access to
is really important from a security perspective.
A common anti-pattern that I see
is creating the credentials using the secret
access keys and distributing them directly to the users.
It's not a good idea, because if they leave
their organization, they can take those credentials
with them and still have access to the data.
This is where the Identity Access
Management Service comes in.
It allows you to define specific actions that
can be performed on those resources.
So, if we look at the pattern here,
you're going to create IAM users,
group them together and assign a role
with a policy that restricts them only
to the functions that they can do,
in this case, an HTTP PUT.
This gives you the flexibility to add, remove users
really easily.
Another thing that you want to add to
the IAM policy is to lock down the location
from where they're coming from.
So, using this source IP Condition key
can restrict the IP address of where they can
upload their files from.
And finally, we also want to monitor
the policy for changes.
Here we can create an AWS Config Custom Rule
to be able to pick up those changes.
If we see that the keys are no longer being used
we can trigger an alert and then using
a Lambda function, remove those users' keys.
Next, we're going to look at how you secure the data.
Amazon S3 provides server-side encryption
by default with Amazon managed keys,
but you can bring your own master keys as well.
Going back to the pattern, these are all
stored in the AWS Key Management Service,
so it's something that you don't have to worry about.
Now, we talked a little bit about IAM before.
Here's where we can lock down the objects
and resources that they have access to
using access controls.
To be able to identify different organizations
you may want to think about using
prefixes and delimiters.
Also, if you find that you need to maintain
multiple copies of the data,
you can switch on versioning.
And if you want to provide users a custom URL
you can use the S3 access point service.
One thing that I definitely recommend
is that you take all your objects for good practice.
Implementing detective controls allows us
to identify potential security threats
as well as incidents that have happened.
So the first thing you want to do is enable monitoring
and collect some usage metrics from all the
different services that you're using.
So for that, switch on CloudWatch.
This enables us to keep a close eye
on all the performance thresholds
inside of your architecture.
To track access and changes to objects
CloudTrail helps us keep track of
all the different calls that have happened
for audit purposes.
Elasticsearch is really great to comb through
all the S3 logs, help correlate events
and find anomalies that have happened.
And finally, if your data is subject to
governance or compliance, or you just want to
stay away from storing personally identifiable information,
Amazon Macie can really come in handy.
So far, we've looked at how to identify
different users, what they have access to
and how to secure your data,
as well as the detective controls that help
manage the environment.
But what about onboarding your users?
Here you may want to think about creating
a little web app, one that allows your users
to log in, create their profiles
and assign different access and permissions.
If they're integrating it into an existing
application through an API,
you may also want to think about using
an organizational LDAP, something like
Active Directory.
Or you could use Amazon Cognito
to manage the user pool.
Check out the resources below for more details.
And thanks for tuning in.