Seamless Shift: Migrating from Amazon RDS for PostgreSQL to Azure Database for PostgreSQL
Published Jan 15 2024 05:59 AM 2,560 Views
Microsoft

Migrating from AWS to Azure, particularly when transitioning databases like PostgreSQL, demands a meticulous strategy to ensure a seamless and efficient process. Crafting a well-thought migration strategy is paramount, as it not only mitigates potential risks but also optimizes the utilization of Azure's features. This strategic shift encompasses careful consideration of cost, downtime, networking, security, data integrity, performance optimization, and resource utilization.

 

The equivalent of Amazon RDS for PostgreSQL in Microsoft Azure is Azure Database for PostgreSQL - Flexible Server. Azure Database for PostgreSQL is a fully managed service that offers built-in high availability, data protection, automated maintenance, predictable performance, elastic scaling, enterprise grade security, monitoring and automation, and industry-leading support.

 

In this blog, we'll delve into the critical aspects of why a thoughtful migration plan is indispensable, providing a comprehensive understanding of the technical nuances involved in the Amazon RDS for PostgreSQL to Azure Database for PostgreSQL - Flexible Server migration journey.

 

Migration Considerations

The key considerations to take note of before starting the migration activities are as follows:

PostgreSQL AWS to Azure-key.jpg

Cost and Sizing

Cost

Sizing

  • Choosing the right tier: Consider the following factors when choosing an Azure Database for PostgreSQL tier: Instance Type, vCores, Memory, Storage, Backup Storage. Read more about various tier and offerings here: Compute and Storage options in Azure Database for PostgreSQL – Flexible Server. Please note that Azure Database for PostgreSQL - Single Server is scheduled for retirement by March 28, 2025. We strongly recommend you upgrade to Azure Database for PostgreSQL - Flexible Server. 
  • Best practice(s): Choose a higher compute and memory SKU to complete the migration quickly. You may scale back easily to the desired configuration post migration. Additionally, leverage IOPS scaling to increase the number of IOPS for the duration of migration without increasing the storage size.

Migration Downtime

  • Regardless of the migration approach employed for workloads, there could be a potential risk of downtime. This is often a result of the necessity to quiesce or temporarily pause an application or dataset to guarantee the continuity of its copy, validation, resetting the server parameters and other critical processes.
  • Best practice(s): It is important to connect with business and application teams to inform, plan, and analyze the impact of downtime during the cutover.

Version compatibility and dependency

  • Ensure that your database is compatible with Azure Database for PostgreSQL, or you have the necessary tools and processes to convert your database to be able to run it on Azure Database for PostgreSQL. Please note that we don't support PostgreSQL version 10 and older for Azure Database for PostgreSQL - Flexible Server. For more details refer to Supported versions - Azure Database for PostgreSQL.
  • Best practice(s): Application compatibility issues can arise when migrating to a different version of PostgreSQL. These issues are primarily due to changes in the database system's behavior, features, or syntax between versions, deprecated features, changes in default setting, extension compatibility, character set, etc. Comprehensive testing is crucial to identify and address compatibility issues before migrating to a higher PostgreSQL version.

Resiliency

  • Azure Database for PostgreSQL provides features that protect data and mitigates downtime for your mission-critical databases during planned and unplanned downtime events. Built on top of the Azure infrastructure that offers robust resiliency and availability, the flexible server has business continuity features that provide another fault-protection, address recovery time requirements, and reduce data loss exposure.
  • As you architect your applications, you should consider the downtime tolerance - the recovery time objective (RTO), and data loss exposure - the recovery point objective (RPO). For example, your business-critical database requires stricter uptime than a test database. Overview of business continuity with Azure Database for PostgreSQL - Flexible Server.
  • Best practice(s): You should consider the trade-offs between availability, performance, and cost when choosing HA architecture for your database. Zone-redundant HA provides the highest level of availability and isolation across multiple availability zones, but it may incur higher network latency. Same-zone HA provides high availability within a single availability zone with lower network latency, but it does not protect against zone failures. You should also implement retry logic in your application to handle the transient errors or timeouts during failover process and monitor the failover status using Azure Monitor.

Networking and Security

gyanisinha_0-1702366466146.png

  • Alternatively, review if there is AWS direct connect and Azure express route already setup between AWS and Azure. However, consider the cost factor.

Performance

  • Evaluate the performance of your database and applications on Amazon RDS for PostgreSQL and Azure Database for PostgreSQL, and make sure you have the plan to optimize performance as needed. Query Performance Insight provides intelligent query analysis for Azure Postgres Flexible Server databases. It helps identify the top resource consuming and long-running queries in your workload.
  • Query Store feature in Azure Database for PostgreSQL provides a way to track query performance over time. For more details refer to: Query Store - Azure Database for PostgreSQL - Flexible Server.
  • Azure Database for PostgreSQL - Flexible Server has an intelligent tuning feature that's designed to enhance performance automatically and help prevent problems. For more details refer to: Intelligent tuning - Azure Database for PostgreSQL - Flexible Server.

Other considerations

  • Support: Make sure you have a plan in place to ensure that you receive the necessary support and guidance as you migrate. Contact Microsoft support team.
  • Operational Excellence: Consider CI/CD, logging, diagnostics, and monitoring requirements.
  • Upskilling needs: There are ample resources and programmes from Microsoft to get your team up to speed with Azure Database for PostgreSQL. Reach out to your account contact for training needs. You may also refer to self-study training modules: Browse all learning paths and modules - Training | Microsoft Learn.

 

Migration Options and Approaches

Although there are different options for migrating from Amazon RDS for PostgreSQL to Azure Database for PostgreSQL, broadly migration approached can be categorized as either offline or online, as shown below:

 

gyanisinha_2-1705325895496.png

 

Offline migration

With an offline migration, the source server is taken offline, and a dump and restore of the databases are performed on the target server. Offline migrations are always the preferred choice if your application can afford some downtime, as they are simple and easy to execute.

 

Offline Approaches

Details

Dump and Restore

You can use pg_dump to extract a PostgreSQL database into a dump file. Then use pg_restore or psql (depending on the dump format) to restore the PostgreSQL database from an archive file created by pg_dump. For more details refer to: Dump and restore - Azure Database for PostgreSQL.

pgcopydb

pgcopydb tool automates running pg_dump | pg_restore between two running PostgreSQL servers. The pgcopydb tool provides performance gains by allowing below two aspects:

·         Bypass intermediate files for the TABLE DATA

·         For each table, build all indexes concurrently

Please refer to the GitHub link for more information:

GitHub - dimitri/pgcopydb: Copy a Postgres database to a target Postgres server.

Online migration

With online migration, the application can continue to access the database during the migration process, except for a short cutover window when the final switch is made from the source to the target server. Online migration is the best choice if your application can only afford minimal downtime. Migrations of most OLTP systems, such as payment processing and e-commerce, fall into this category. However, online migration is more complex and requires more planning and testing than offline migration. You need to ensure that the source and target servers are compatible, replication lag is minimal, and the cutover window is short and well-timed.

 

Online Approaches

Details

Azure DMS

Fully managed Azure service that helps you migrate your databases to Azure with minimal downtime. There are no costs incurred while running the migration using DMS. For more details refer to Tutorial: Migrate RDS PostgreSQL online to Azure Database for PostgreSQL.

Open Source

(Logical Replication, Debezium, etc.)

Logical replication is a method of replicating data objects and their changes, based upon their replication identity. Please refer to: Using logical replication to replicate managed Amazon RDS for PostgreSQL and Amazon Aurora to self-m...

 

Debezium is an open source distributed platform for change data capture. For more details refer to: Debezium

Commercial

(Striim, HVR, etc.)

The Striim platform is an enterprise-grade data integration and streaming solution for moving data in real-time to the cloud. Microsoft and Striim providing exclusive Striim licenses at zero-cost for online database migrations including CDC to Azure Cosmos DB, Azure SQL and ORCAS. These Striim licenses provide:

·        Online migrations for homogeneous and heterogeneous migrations

·        Change data capture (CDC) for continuous, real-time data movement

·        In-flight data processing

·        Near-zero downtime migrations

 

HVR is a powerful software product that enables real-time homogeneous and heterogeneous data replication. HVR uses various CDC (Change Data Capture) methods to replicate changes between databases, directories (file locations), as well as between databases and directories. For more details read: Introduction - HVR 5 Documentation (fivetran.com).

 

Depending on the downtime window and business impact, you may decide to use offline (pg_dump and pg_restore/pgcopydb) or online mode of migration using DMS. The final choice of migration approach depends on factors such as, permissible downtime, cost, ease of use and criticality of applications.

 

Migration Phases

 

gyanisinha_0-1702364644945.png

 

Below is the guidance for each of the phases of migration, as shown in the image above:

 

Discovery and Assessment

  • Conduct customer discovery to understand the business objectives and motivations.
  • Assess the existing Amazon RDS for PostgreSQL architecture and evaluate the target Azure Database for PostgreSQL Architecture.
  • Confirm the downtime expectations with the stakeholders.
  • Estimate cost savings: Part of the business's plan to migrate to Azure could be to reduce costs. Use the Azure pricing calculator to estimate monthly running costs. You can save on your Azure Database for PostgreSQL costs when you pre-purchase compute resources for one or three years. Refer Reserved compute pricing - Azure Database for PostgreSQL for more details.
  • Gather AWS inventory information to understand the AWS SKU’s and pricing tiers used.
  • Finalize the list of the databases that needs to be migrated. Clarify and confirm from the customers if there are any transformations and modernization requirements.
  • Finalize target SKU based on the requirements of Azure Database for PostgreSQL. Validate service, AZ, and DR availability in the region.
  • If there are any special requirements that cannot be met due to restrictions on Azure Database for PostgreSQL, also consider PostgreSQL on Azure Virtual Machine as a migration destination.
  • Identify upskilling needs.

Plan and Design

  • Ensure to evaluate the key considerations and the assessment in the discovery step impacting the migration plan.
  • Define and establish target state architecture. Finalize the migration method/approach.
  • Validate and finalize the network connectivity strategy between single to flexible server.
  • Review migration readiness.
  • Plan and prepare detailed migration plan, involving key stakeholders, based on the responses gathered in Amazon RDS for PostgreSQL to Azure Database for PostgreSQL technical evaluation.
  • Ensure to review the key considerations impacting the migration plan.

Migrate

  • Perform smoke test/pilot migrations before actual migrating for production:
    • If you are migrating to a higher PostgreSQL version, test for application compatibility.
    • All stakeholders who need to be involved should be available and ready. The production migration requires close monitoring.
    • For an online migration, the replication must be completed before you perform the cutover, to prevent data loss.
  • The key steps for migration are:
    • Prepare the source and target servers. This includes creating the flexible server with the desired configuration, setting the server parameters, configuring the networking and firewall rules, and creating the databases and users on the target server.
    • Start the migration and monitor the progress and status.
    • Perform the cutover when the replication lag is minimal, and the application traffic is low.
    • Refer to the migration documentation for the selected method and tool.
  • Post migration activities:
    • Perform validation and data integrity checks.
    • Redirect all dependent applications to access the new primary database and make the source server read-only. Then, open the applications for production usage.
    • After the application starts running on the target flexible server, monitor the database performance closely to see if further performance tuning is required.

Monitor and Govern

Optimize and Modernize

  • Identify opportunities to further optimize cost, performance, security, scalability, and operational excellence. Conduct well-architected (WAF) review to ensure that the workloads are optimized as per the best practices and standards.

Known Issues and Limitations

It is also important to consider the known issues and limitations of the migration tools you choose to use. Refer to the links below for more information:

Conclusion

By meticulously considering various factors, employing the right tools, adhering to best practices, and addressing limitations, you can seamlessly transition your database from Amazon RDS for PostgreSQL to Azure Database for PostgreSQL. Success lies in thorough preparation, comprehensive testing, ongoing performance monitoring, and optimization.

 

If you have any feedback or questions about the information provided above, please leave a comment below or email us at AskAzureDBforPostgreSQL@service.microsoft.com. Thank you!

 

1 Comment
Version history
Last update:
‎Jan 15 2024 05:59 AM
Updated by: