Welcome to 'Back to Basics'.
Today, we're going to talk about database replication.
In the simplest use case, this can be between two AWS accounts
in the same organization.
A common requirement is to copy data between two different environments,
for example, between production and UAT to support testing,
and to share data between two separate teams
that manage their own databases.
Another use case is to support hybrid architectures
where an application and the corresponding database
is moved to AWS,
and an on-premise database needs to be kept in sync
to support on-premise applications that haven't migrated yet.
Today, more frequently, data is also stored in
third-party applications or SaaS solutions that are host on AWS.
In this situation, we still need to have the capability to
combine all our data for various analytics use cases.
So, as you can imagine now, it becomes more challenging
to implement a secure and scalable pattern
for all of these use cases.
In this episode, we're going to walk you through the steps
for setting up the secure and scalable pattern
for database replication between two AWS accounts.
In our example, we have a consumer web application
running in a private VPC across two AZs.
This application wants to run advanced analytics
on all of the portfolio holdings.
In order to do this, this application needs to
have access to the portfolio data that is available in a SaaS application
that is used by their business to manage all portfolio holdings.
The SaaS provider runs and maintains an isolated collation
and SQL server database for each customer.
Things you want to consider for this architecture are the following:
first, from a security perspective,
no communication should occur over the internet.
And that we can lock down ingress-egress traffic
to dedicated private IPs form into the third party.
Second, from a flexibility perspective, that we might want to use
a different database engine like Amazon Aurora,
on the consumer side, to reduce costs down the road.
Third, from a scalability perspective, that we might require
dozens or hundreds of similar network connections
with other AWS accounts in the future.
These requirements lead us to use PrivateLink for network connectivity
as VPC PI won’t be scalable.
And to use Database Migration Service (DMS) for database replication
as it provides cost engine replications.
In our example, we want to replicate the database
in the SaaS application VPC, to a SQL server instance
in the customer VPC.
VPC endpoints make it easy to create a private connection between two VPCs
of different AWS accounts in the same region.
Traffic between both VPCs don't have- don’t leave the Amazon network
and don't require public, IP addresses.
For cost regional use cases, we can extend this pattern via VPC PI
to other VPCs in other regions.
To expose this database to other consumers,
we need to front end the database with a network load balancer.
In the network load balancer, we register as a private IP
and the part of the database instances as a target group
and adjust the security groups of the database
to only allow traffic from a network load balancer.
Finally, we can set up the VPC endpoint services
by creating a service configuration and associate it with
the network load balancer.
For security reasons, we require manual approval
for all connections.
The third party can now request an interface endpoint
for the previously-configured AWS PrivateLink service in the VPC.
The interface endpoint can span one or two availability zones,
it’s redundant and scales horizontally.
After the service provider approves request,
the endpoint is available as an ENI in the third party VPC.
As previously mentioned, we use AWS DMS,
database migration service, for the database application.
And as a first step, we set up a DMS instance
in the third-party VPC.
The DMS instance requires connectivity to the source and the target database.
We configure the security group of the source database
to allow access from the DMS instance.
Access to the interface endpoints can be controlled by endpoint policies.
So, by default, endpoint policies allow full access
to the service.
So, in this case, we need to tighten up the security
and define an endpoint policy that allows only access
from the DMS instance.
So now, we have everything in place from a security and networking perspective
to continue with the setup of the database replication.
As a best practice, you wouldn't put in line any firewall
or network inspection device between the PrivateLink endpoint
and the database.
In our example, we want to configure
the SQL server instance in the third-party VPC
as the source database in DMS.
For the target database, we configure in DMS the private IP
of the VPC interface endpoint, and the SQL part of the database
in the customer VPC.
Alternatively, you can also use private DNS names
for VPC interface endpoints.
We have chosen to set up DMS in the SaaS provider or third-party VPC
as this gives control of future DMS changes
for sample frequency of replication tasks to the third party.
Depending on the needs of your setup, this pattern can be also set up
the other way around.
DMS can continue to replicate data with high availability
between two data stores
Tasks defined in DMS can perform a full load,
catch changes or ongoing application between two data stores.
I've used this pattern more than once, especially for
security-concerned organizations.
Now, you can use it too.
Check out the links below for more details.
See you next time.