- Hi, my name is Vaibhav
and I'm Senior DynamoDB Specialist Solutions architect
at Amazon Web Services.
Many a times my customer have asked
how they can migrate their Amazon DynamoDB tables
between AWS accounts.
While there is no direct one click way of achieving it.
But in this video, we'll discuss at least four options
that can help you migrate your DynamoDB tables
between AWS account.
Let's try to understand why you might need
to migrate your Amazon DynamoDB tables between AWS accounts.
I have worked with many customers
who started their AWS journey with a single AWS account,
and now they're adopting multi account strategy.
This is one of the reason they want to migrate their
existing DynamoDB tables into other accounts
to make sure they have their non-production
and production workloads deployed in different accounts.
I've also worked with customers
who want a backup table available in a different AWS account
as part of their cross account backup strategy.
The other scenario I can think of is that you need
to share your production data in non-production account
for testing or debugging purposes.
The last reason could be there are business compliance
that might need you to migrate your DynamoDB tables
to some other AWS account.
These are the four options we will discuss in this video.
To migrate a DynamoDB table between AWS accounts,
let's explore the first option that uses AWS backup service
to perform cross account migration of DynamoDB table.
In this option, we use AWS backup service
to create the backup of the DynamoDB table in source account
and then copy the backup from source AWS account
to destination account.
Once the backup is available in destination account,
we restore the table in destination account.
This method requires both source
and destination accounts
to be in the same AWS organizations.
This is how the high level architecture looks like.
First, we'll create a customer managed KMS key in
destination account and share it with source account.
Then we'll create a backup vault in destination account
using the KMS key we have just created.
We'll also allow access
to this backup vault from organization.
Similarly, we'll create a customer managed KMS key in source
account as well and share it with destination account.
Then we will create a backup vault in source account.
Using the KMS key we have just created in source account,
we'll allow access
to this backup vault from AWS organizations.
Now we'll create a backup plan in AWS backup
in source account that will create a backup
of Amazon DI mood table in source account,
store it in source account backup vault,
and copies it to the destination account backup vault.
Also, once the backup is available in
destination account, we restore the table in
destination account.
Please note that this method requires both source
and destination accounts in same AWS organizations.
It also requires the customer managed KMS keys to be shared
between source and destination accounts.
Now let's follow this process step by step.
Go to management account in AWS organizations
and make sure you have enabled these items in
AWS backup settings.
Then in the destination account,
create a customer managed KMS key
and share it with source account.
Then create a backup vault in destination account using the
KMS key we have just created
and allow access to this backup vault from organization.
Similarly, create a customer managed KMS key in source
account as well, and share it with destination account.
Then create a backup vault in source account using the KMS
key we have just created in source account
and allow access to this backup vault from
AWS organizations.
Now create a backup plan in a AWS backup in source account
that will create a backup
of Amazon DynamoDB table in source account.
Store it in source account backup vault,
and copies it to destination account backup vault.
Once the backup is available in destination account,
reso the table in destination account.
This way your table will be migrated
to other AWS account using AWS backup service.
Now let's explore the second option
to migrate your DynamoDB table between AWS accounts.
We can use DynamoDB export to S3
and import from S3 feature for this.
This is the high level architecture for this approach.
First, we use DynamoDB export to S3 feature
to export data from dyod table in source account
to Amazon S3 bucket in destination account.
Then we use DynamoDB native import from S3 feature
to import data in new DynamoDB table in destination account.
As a prerequisite, make sure
that the S3 bucket in destination account should allow the
source account for S3 put object operations.
Let's see these steps in detail for destination account.
First, create the S3 bucket in destination account
and allow this bucket to perform.
S3, put object operations from source account.
This can be achieved using this sample bucket policy
now head to source account
and enable point in time restore
for your source time motivate table.
Please note enabling point in time restore is mandatory
for export two S3 feature.
Once point in time restore is enabled.
Export the data to the destination account S3 bucket.
Depending on the amount of data in your DynamoDB table,
it can take some time for the data to get exported to S3.
Once the export is finished, head to destination account
and use DynamoDB Import from S3 feature
to import data from destination S3 bucket
to destination AWS account.
Please note this feature can work
outside AWS organizations also,
but import from S3 can be used only for new DynamoDB tables
and can't be used in existing DynamoDB tables in
destination account.
Let's see the third option that uses DynamoDB export
to S3 feature with AWS view to migrate your DynamoDB table
between AWS account.
In this option, the method
of exporting the data from DYOD DB is same as second option,
but we use AWS Glue to import the data in DYOD db.
This method allows you to transform your data in AWS glue
before importing to dyod DB
and also allows to import data in existing DynamoDB tables.
This is the high level architecture for this approach.
First, we use DynamoDB native export to S3 feature
to export data from DynamoDB table in source account
to Amazon S3 bucket in destination account.
Then we use AWS glue crawler
to catalog the exported data.
Once the data is cataloged, we use AWS Glue jobs
to import the data from S3 and import it in existing
or new DynamoDB tables in destination account,
like second option as a prerequisite.
Make sure that the S3 bucket in destination account allows
the source account for S3.
Put object operations.
Once the data is exported to S3 bucket
and destination account, use AWS glue crawler
to catalog the exported data.
Once the data is cataloged, we use AWS Glue jobs
to import the data into new
or existing DynamoDB tables in destination account.
You can also use Amazon EMR
to migrate your DynamoDB tables between AWS accounts.
This option comes in handy if you are already
using EMR with Hive.
This method can also work outside AWS organizations
and data can be imported in existing DynamoDB tables.
This is how the high level architecture
looks for this approach.
First, make sure that the S3 bucket in the destination
account allows the source account
for S3 put object operations.
Then we use Amazon EMR running Hive in source account
to export the DynamoDB data
to S3 bucket in destination account.
Then we use Amazon EMR running Hive in destination account
to read exported data from S3 bucket
and write it in Dion Moody table in destination account.
It's a good practice to launch EMR clusters in private
subnets in A VPC as an additional security measure,
and to make sure that no traffic goes via internet.
Consider Amazon S3 VPC endpoint
and a route to DynamoDB in the private subnets.
The role given to EMR cluster in source account should allow
S3 put object operations
to S3 bucket in destination account.
Once you have the EMR cluster
with high ready in your source account, connect
to the master node of your EMR cluster using SSH
and use hive command to export the DynamoDB table
to S3 bucket in destination account.
This is the sample hive command
that exports the DynamoDB table
to S3 bucket in destination account.
Once the data is exported
to S3 bucket in destination account, connect via SSH
to the master node of the MR cluster running hive in
destination account and use hive command
to input data in existing
or new DynamoDB table in destination account.
This is a sample hive command to write data from S3
to DynamoDB.
All the methods discussed in this video are also documented
in detail at AWS Knowledge Center.
You can scan this QR code
and access the repost link to read it.
This video is part of DynamoDB Nuggets playlist on YouTube
where we try to explain you important DynamoDB concepts
in form of videos that are less than 15 minutes in duration.
You can scan this QR code
to access the DynamoDB Nuggets playlist on YouTube.
I hope you found this video useful
to learn different options
to migrate your DynamoDB tables within AWS accounts.
Do let us know your feedback in the comments section.
Also, let us know the topics on which you would like us
to make the next ODB Nuggets video.
Thank you.