AI-assisted Oracle-to-PostgreSQL schema conversion in Visual Studio Code
By AI Omar Rajawat, Pranay Lohia, Gautam Juneja, Vikas Nimmagadda, Anil Dogra, Aditya Duvuri We’re seeing significant interest in migrating database workloads from Oracle to Azure Database for PostgreSQL. Historically, schema conversion has been one of the most technically challenging and expensive steps in that journey, demanding specialist knowledge of both engines and stretching migration timelines before a single row of data moves. Recent advances in AI-assisted schema conversion are changing that, turning what used to be a long, manual effort into a faster, more accessible, and lower-cost proposition. This post looks at what that shift means in practice for teams moving to Azure Database for PostgreSQL flexible server. Oracle schema conversion is where the complexity of translating schema and code objects to PostgreSQL becomes visible. Packages, procedures, triggers, custom types, and dependencies built up over years must be mapped to PostgreSQL-compatible definitions while preserving the relationships that make the schema work. Generally available since May 2026, the feature is built into the PostgreSQL extension for Visual Studio Code, published by Microsoft. It helps teams convert Oracle schema and code objects — tables, views, constraints, packages, procedures, functions, and triggers — into PostgreSQL-compatible definitions for Azure Database for PostgreSQL flexible server, with no separate conversion utility to install and no disconnected workflow to manage. It brings schema discovery, conversion, compile validation, and review into one project-based experience. Teams can connect to Oracle, select schemas, and configure a Microsoft Foundry connection in the same project. The extension then translates Oracle-specific constructs, compiles and syntax-checks converted DDL into scratch schemas on Azure Database for PostgreSQL flexible server and surfaces unresolved items as review tasks that teams can work through with GitHub Copilot agent mode. Why schema conversion deserves a better workflow Traditional conversion tools can produce a useful first pass, but the long tail of the process is rarely solved by generating replacement DDL alone. Teams still need clear answers to practical questions: What converted successfully? What needs attention? Which Oracle constructs require a PostgreSQL design decision? Which items should be reviewed first? We designed the schema conversion experience around those questions. The goal is not to hide complexity behind a single score. It is to help teams make steady progress while keeping the work visible and reviewable. What the schema conversion experience provides The experience guides teams through a schema conversion project rather than a collection of separate scripts. It discovers the selected Oracle schemas and converts both the relational model and the code that runs on it: tables, indexes, sequences, primary key, unique, check and foreign key constraints, views and materialized views, synonyms, and Oracle object types — along with the PL/SQL that is usually the hardest part of the migration. Packages and package bodies, package-level state, standalone procedures and functions, and triggers are translated into PostgreSQL functions, procedures, and trigger functions. Oracle-specific constructs are mapped to PostgreSQL equivalents rather than dropped or stubbed out. REF CURSOR and SYS_REFCURSOR become PostgreSQL refcursor; CLOB and BLOB columns become text and bytea ; and NUMBER and VARCHAR2 are mapped by precision and length to their closest PostgreSQL types. Oracle date functions such as ADD_MONTHS, LAST_DAY, MONTHS_BETWEEN, and TRUNC are resolved through the orafce extension, which the project detects and flags for you before deployment. Every converted definition is compiled against scratch schemas on Azure Database for PostgreSQL flexible server, so the deployment script you end up with is an organized, dependency-ordered set of PostgreSQL SQL artifacts that has already been proven to build. Objects that still require human judgment are surfaced as review tasks. Teams can inspect the source and converted definitions side by side, work through the remaining items, and use GitHub Copilot agent mode for guided assistance — keeping automation and human review in the same workflow. How it works: the system architecture Under the hood, the conversion engine follows one principle — the language model is a single, bounded stage; it never has the first or the last word. Deterministic steps decide what the model sees and what it is allowed to produce. Deterministic in. Rule-based extraction reads the Oracle DDL and metadata, then a dependency-graph decomposition splits the estate into bounded, dependency-ordered chunks — so every object is converted in the context that keeps it correct. Bounded conversion. A tiered model strategy through the Microsoft Foundry connection translates each chunk with structured, contract-wrapped input and output. Even very large PL/SQL packages are split and converted member by member, so nothing is trusted as a monolith. Deterministic out. Converted objects pass through review, then compile-and-verify against scratch schemas on Azure Database for PostgreSQL flexible server, and finally dependency-ordered deploy assembly. Unresolved items become review tasks, and every object carries a per-object audit trail. A continuous-improvement loop closes the system: the engineering team maintains a versioned regression suite of supported conversion patterns, and an executable benchmark tracks regressions as the pipeline evolves. Proven in production The approach has been exercised on real enterprise estate. Across representative production runs totaling more than 60,000 schema objects; conversion reached roughly 98% overall — with several schemas converting at a full 100%. The hardest tail, PL/SQL package members, now compiles at 96% across more than 20,000 members thanks to targeted coverage and a resilient compile stage. Conversion outcome and review status are separate measures. Objects that convert and compile cleanly are safe to deploy as they are; the rest are deliberately routed into a prioritized review queue rather than silently accepted. In a representative single-schema run, no object ended in a hard conversion failure, and a cleanly generated object can still involve a PostgreSQL design decision. That is the workflow operating as intended: automation absorbs the volume, and review tasks to keep the remaining judgment calls visible, ordered, and auditable. Measured, not asserted: the SchemaBench eval Quality is verified by running it. SchemaBench, the evaluation framework, deploys each converted schema to a live PostgreSQL database and probes real behavior — whether constraints still fire and whether objects still resolve — rather than comparing DDL text. It scores seven weighted dimensions: semantic fidelity, structure, constraints, completeness, performance, target idioms, and maintainability, behind hard gates. On the e-commerce benchmark, the strongest model scored 96.1 overall with 100% semantic fidelity and a ~98% behavioral-probe pass rate. Every failure a migration hits becomes a permanent regression test the next run has to pass. Learn more: Oracle to Azure Database for PostgreSQL schema conversion overviewFaster, Safer Version Upgrades for Databases with Large Objects
By Varun Dhawan, Ilan Benschikovski, and Alexander Kukushkin - Azure PostgreSQL, Microsoft Faster, Safer Upgrades for Databases with Large Objects TL;DR: We improved major version upgrades for PostgreSQL databases with very high large-object counts. For upgrades targeting PostgreSQL 15 and later, large-object metadata is now handled more efficiently, reducing memory/temp-space pressure and helping previously risky upgrades complete more reliably. Why this matters Some PostgreSQL workloads store documents, images, PDFs, scanned files, or attachments as large objects (LOBs). In normal operations this is fine. But during a major version upgrade, very high LOB counts could make the schema dump step slow, memory-heavy, or fail. This improvement is about making that upgrade path safer and more predictable for Azure Database for PostgreSQL flexible server customers. What changed? During a major version upgrade, PostgreSQL uses pg_upgrade , which internally runs pg_dump to move schema and metadata into the new version. For databases with millions of large objects, the older upgrade path handled large-object metadata one object at a time. That created high memory and temporary-space pressure during the schema dump phase. This fix changes the upgrade path. Instead of processing large-object metadata one object at a time, PostgreSQL now transfers that metadata in bulk. The actual large-object data is not changed; only the upgrade metadata handling is improved. Why this is different This improvement builds on upstream PostgreSQL work that makes large-object metadata handling more efficient during upgrades. We brought that benefit into Azure Database for PostgreSQL flexible server for supported PostgreSQL 15+ upgrade targets, so customers with large-object-heavy workloads can benefit without waiting for a future PostgreSQL major version. Before vs after Area Before After Metadata handling One operation per large object Bulk metadata transfer Memory/temp pressure Grew heavily with LOB count Much flatter and more predictable High LOB counts Risk of OOM or temp-space failure Completes more reliably for PostgreSQL 15+ targets Customer workaround vacuumlo + scale-up often needed Less reliance on LOB-specific workarounds for PostgreSQL 15+ targets In plain English: the upgrade no longer has to carry paperwork for every large object one by one. It moves the metadata in bulk, which makes the upgrade faster, safer, and less likely to fail at very high LOB counts. The numbers We tested upgrades from PostgreSQL 13 with large-object counts ranging from 10M to 500M. The older path is represented by PostgreSQL 13 → 14. The improved path is represented by PostgreSQL 13 → 15. Note: These figures come from a multi-database test where large objects were spread across 100 databases. Because pg_dump runs per database, single-database workloads with the same total large-object count may see different runtimes. Cap Outcome summary Large objects Older path Improved path What changed 10M 48 min 16 min 3x faster 20M 1h 02m 18 min 3.4x faster 30M 2h 41m 21 min 7.6x faster 50M Failed at higher scale 26-30 min Now completes 100M Failed 54 min Now completes 500M Failed 4h 01m Now completes Key takeaway: this is not just faster. At higher LOB counts, the improvement changes the outcome from upgrade fails to upgrade completes. Who benefits from this? You should care if your database stores large binary content using PostgreSQL large objects. Workload pattern Why it matters Document management PDFs, contracts, scans, and archived files Attachment-heavy apps Files stored inside PostgreSQL instead of external storage Legacy apps using lo APIs LOBs may have accumulated for years Image/archive systems Millions of binary objects can build up quietly Previous upgrade failures Failures during schema dump may map to this scenario Copy/paste: check your large-object count Run these checks in each database you plan to upgrade. 1. Count large objects in the current database -- Count PostgreSQL large objects in the current database SELECT current_database() AS database_name, count(*) AS large_object_count FROM pg_largeobject_metadata; 2. Check large-object storage footprint -- Estimate large-object data and metadata size SELECT pg_size_pretty(pg_total_relation_size('pg_largeobject'::regclass)) AS large_object_data_size, pg_size_pretty(pg_total_relation_size('pg_largeobject_metadata'::regclass)) AS large_object_metadata_size; 3. Understand ownership and ACL shape -- Inspect large-object metadata shape SELECT count(*) AS total_large_objects, count(lomacl) AS large_objects_with_custom_acl, count(DISTINCT lomowner) AS distinct_large_object_owners FROM pg_largeobject_metadata; 4. Find top large-object owners -- Top large-object owners SELECT lomowner::regrole AS owner, count(*) AS large_object_count FROM pg_largeobject_metadata GROUP BY lomowner ORDER BY large_object_count DESC LIMIT 10; What should I do before my next major version upgrade? If your situation is... Recommended action Target is PostgreSQL 15 or later Target PostgreSQL 15 or later to benefit from improved large-object metadata handling. Target is PostgreSQL 14 or earlier Prefer PostgreSQL 15+ where possible; very high LOB counts may still hit older-path limitations. Very large or unusual database Restore a copy and rehearse the upgrade before production. Suspected orphan LOBs Consider vacuumlo only after testing. It can delete valid LOBs if your app uses custom references. Any major version upgrade Keep healthy free space and leverage pre-upgrade validation checks to validate extension/schema compatibility first. Bottom line If large objects were making your PostgreSQL upgrade risky, this improvement makes the upgrade path safer and more predictable. For large-object-heavy databases, upgrades targeting PostgreSQL 15 and later now show faster runtime, lower memory/temp-space pressure, and successful validation up to 500M large objects. Learn more Major version upgrades in Azure Database for PostgreSQL flexible server How to perform a major version upgrade PostgreSQL vacuumlo documentationTop 10 Performance Optimization Techniques for Azure Database for PostgreSQL Flexible Server
Introduction Performance optimization is one of the most common challenges faced by organizations running business-critical workloads on Azure Database for PostgreSQL flexible server. As your workloads grow it’s common to encounter high CPU utilization, storage bottlenecks, autovacuum issues, excessive temporary file generation, and connection saturation. The good news is that Azure PostgreSQL flexible server provides several built-in capabilities to help optimize performance, improve scalability, and reduce operational overhead. This article explores ten practical techniques that can significantly improve database performance and reliability. 1. Choose the Right Compute SKU Performance starts with selecting the appropriate compute tier. Azure PostgreSQL flexible server offers: Pricing tier Target workloads Burstable Designed for workloads that don't require full CPU performance continuously. Best suited for proof-of-concept environments, and development builds. Not recommended for production workloads. General Purpose Provides a balance between CPU and memory with scalable I/O throughput, making it suitable for most production workloads. Examples include servers for hosting web applications, mobile apps, and enterprise applications. Memory Optimized Suitable for high-performance database workloads that require in-memory performance for larger buffer cache sets, and higher concurrency. Examples include servers for processing real-time data and high-performance transactional or analytical apps. Learn more about Compute Tiers here. 2.Enable and Use Query Store Query Store is one of the most powerful performance tools available. Query Store automatically captures the following and keeps them available for review: Query execution statistics Runtime metrics Wait event information Historical execution trends It organizes the data into time windows, so you can spot database usage patterns. Data for all users, databases, and queries is stored in a database named azure_sys in the Azure Database for PostgreSQL instance. It’s generally recommended to monitor query store from Azure tools, KQL, etc. Learn more about Query store here. You can also view some useful scenario for query store and some Best Practices for Query store 3.Leverage Built-In PgBouncer Connection Pooling PostgreSQL uses a process-per-connection model, which means every connection consumes memory and CPU resources. Azure PostgreSQL flexible server provides built-in PgBouncer support for eligible SKUs . PgBouncer allows multiple application sessions to reuse open backend connections and significantly reduces overhead. Benefits include: Lower memory consumption Faster connection handling Improved application scalability Reduced CPU overhead Learn more about PgBouncer here 4.Use Azure Troubleshooting Guides One underutilized feature is the built-in troubleshooting experience available directly in the Azure portal. Guides are available for: CPU troubleshooting Memory troubleshooting IOPS analysis Temporary files Autovacuum monitoring Autovacuum blockers These tools provide actionable recommendations and visualizations without requiring external monitoring solutions. Learn more about Troubleshooting Guides here. 5. Monitor and Tune Autovacuum Autovacuum is critical for maintaining PostgreSQL performance. Without proper vacuuming: Dead tuples accumulate Table bloat increases Statistics are not refreshed regularly Query performance degrades Transaction ID wraparound risks emerge Use Azure's built-in Autovacuum Monitoring TroubleshootingGuides to identify: Vacuum lag Blocked autovacuums Table bloat Inefficient cleanup operations Azure now also offers adaptive tuning capabilities to optimize maintenance behavior. Learn more about Autovacuum tuning here 6.Optimize Storage and IOPS Planning Many performance incidents originate from insufficient storage planning rather than inefficient SQL. In Azure PostgreSQL flexible server: Storage and baseline IOPS are closely related. Learn more here. Larger storage allocations provide higher baseline IOPS. Auto-grow prevents storage-related outages Note: Storage can only be scaled up and will always be double in size. SSDv2 auto-grow will allow customized growth settings in future release. For write-heavy workloads, monitoring storage utilization and IOPS is essential. Best practice: Enable Storage Auto-Grow Monitor Read/Write IOPS regularly Scale storage proactively 7.Investigate Temporary File Generation Large sorts and hash operations that exceed available memory spill to disk and generate temporary files. Symptoms include: Sudden Latency Spikes Increased IOPS Slower query execution Azure TroubleshootingGuides provide dedicated temporary-file analysis capabilities that help identify offending queries. Frequent temp file generation often indicates: Missing indexes Undersized work_mem Large sorting operations 8.Use Intelligent Tuning Azure PostgreSQL flexible server includes Intelligent Tuning capabilities. The service continuously observes workload behavior and automatically optimizes parameters related to write operations. Examples of tuning include: checkpoint_completion_target max_wal_size min_wal_size bgwriter settings This reduces administrative effort while helping maintain consistent performance. Learn more about Intelligent Tuning here. 9.Optimize Checkpoints and Write Workloads Checkpoint spikes frequently appear in escalations involving high IOPS and latency. Aggressive checkpoint activity can: Generate excessive disk writes Increase latency Consume IOPS capacity Monitoring checkpoint behavior and ensuring WAL parameters are properly configured can significantly improve write-intensive workloads. Azure intelligent tuning can assist in this area as well. 10.Metric Monitoring Optimization should always be data-driven. You should track the following: CPU utilization Memory pressure Active Connections Oldest Query IOPS consumption Combining Azure Metrics, Query Store, and PostgreSQL statistic views allows teams to distinguish between normal workload spikes and true performance degradation. PostgreSQL statistics views provide valuable workload insights. For example, pg_stat_activity can be used to identify long-running or blocking queries, pg_stat_user_tables helps track dead tuples, vacuum activity, and statistics refreshes, while pg_stat_statements (if enabled) help identify the most resource-intensive queries by execution time and frequency. Learn more about Metric here Conclusion Performance optimization in Azure Database for PostgreSQL flexible server is not just about changing a few parameters and hoping for better results. It requires a structured approach that combines workload understanding, proactive monitoring, proper sizing, query optimization, and platform-native capabilities. By leveraging Query Store, PgBouncer, Intelligent Tuning, Autovacuum Monitoring, Azure Metrics, and Troubleshooting Guides, you can significantly improve database efficiency while reducing operational effort. References Compute Options - Azure Database for PostgreSQL | Microsoft Learn Query Store in Azure Database for PostgreSQL Flexible Server - Azure Database for PostgreSQL | Microsoft Learn PgBouncer in Azure Database for PostgreSQL Flexible Server - Azure Database for PostgreSQL | Microsoft Learn Autovacuum Tuning - Azure Database for PostgreSQL | Microsoft Learn Intelligent Tuning in Azure Database for PostgreSQL Flexible Server - Azure Database for PostgreSQL | Microsoft LearnLog Insights in Minutes: A Simpler pgBadger Workflow
Sometimes the fastest way to understand a PostgreSQL workload is not another dashboard. It is a good log report. pgBadger is a PostgreSQL log analysis tool that turns raw PostgreSQL logs into an interactive HTML report. It helps summarize query activity, connection patterns, errors, temporary files, lock waits, autovacuum activity, and more. Earlier guidance for generating pgBadger reports from Azure Database for PostgreSQL Flexible Server focused on exporting logs through Diagnostic Settings, storing them in a storage account, and then using tools such as BlobFuse and jq to extract PostgreSQL log lines from JSON files. That workflow is still useful when customers centralize logs across multiple servers. However, if you are already using the Server logs feature in Azure Database for PostgreSQL Flexible Server, there is a much simpler path. In this post: You’ll learn how to generate a pgBadger HTML report from Azure Database for PostgreSQL Flexible Server by downloading native PostgreSQL .log files directly from the Azure portal. No storage account, BlobFuse mount, or JSON extraction required. Fast path Configure log_line_prefix . Enable Server logs for download. Download the PostgreSQL .log files. Run pgBadger with the matching prefix. Open pgbadger-report.html . Why use this workflow? With Server logs, you can download native PostgreSQL .log files directly from the Azure portal and run pgBadger locally. Older path Simpler path in this blog Diagnostic Settings → Storage account → BlobFuse → JSON extraction → pgBadger Server logs → Download .log files → pgBadger Area Older Diagnostic Settings workflow Server logs workflow Export path Diagnostic Settings to storage account Download .log files directly from the portal Format JSON payloads need extraction Native PostgreSQL .log files Extra tooling BlobFuse and jq JSON parsing None Best suited for Centralized or multi-server logging Quick per-server analysis Outcome Flexible, but more setup Faster path to pgBadger Recommended: Use the Server logs workflow when you want a fast, low-friction way to generate a pgBadger report from one Azure Database for PostgreSQL Flexible Server. When should you use this workflow? Use this workflow when... Use Diagnostic Settings when... You need a quick report for one Flexible Server. You centralize logs from many servers. You want to run pgBadger locally. You need long-term retention or workspace-level querying. You want to avoid JSON extraction. You already have automated log export pipelines. Before you start A machine where you can install or run pgBadger. A working Perl runtime. Git Bash on Windows, so the multi-line shell commands work as shown. Portal access to your Azure Database for PostgreSQL Flexible Server. Permission to update server parameters and enable Server logs. Important: pgBadger can only analyze what PostgreSQL logs capture. To populate query timing and slow-query sections in the report, enable log_min_duration_statement before collecting logs. Logs collected before that change will not include duration data. Workflow overview Task Type Rough effort Install or prepare pgBadger One-time setup per analysis machine 5–10 minutes Configure log_line_prefix One-time setup per server 2–3 minutes Enable Server logs One-time setup per server 2–3 minutes Download logs and run pgBadger Repeatable 2–5 minutes Install or prepare pgBadger on the machine where you will analyze logs. Configure log_line_prefix so pgBadger can parse each log line. Enable Server logs, so PostgreSQL logs are available for download. Download the logs and run pgBadger locally. 💡Pro tip: Start with a narrow log window first. Use one or two hourly log files, confirm the report looks right, and then expand the analysis window if needed. Step 1: Install pgBadger Before generating a report, you need pgBadger available on the machine where you plan to analyze the downloaded PostgreSQL log files. Run this on a Linux VM, WSL, or another Linux-based environment where you can install packages. Note: Azure Cloud Shell may work for quick testing, but package installation and build-tool availability can vary by session. For repeatable analysis, use a Linux VM, WSL, or another environment you control. Copy and run sudo apt-get update && sudo apt-get install -y git perl make gcc && \ git clone https://github.com/darold/pgbadger.git && \ cd pgbadger && \ perl Makefile.PL && \ make && \ sudo make install && \ pgbadger -V What good looks like: The install command completes successfully and pgbadger -V returns the installed pgBadger version. Step 2: Configure log_line_prefix This is a one-time server configuration step. The log_line_prefix parameter controls the beginning of each PostgreSQL log line. pgBadger uses this prefix to extract useful fields such as timestamp, user, database, and process ID. In the Azure portal, open your Flexible Server and go to Server parameters. Search for: Parameter log_line_prefix Set this value %m user=%u db=%d pid=%p: Then select Save. In Server parameters, confirm that the custom value is saved for log_line_prefix . Figure 1: Set log_line_prefix so pgBadger can correctly parse timestamp, user, database, and process ID from each log line. Prefix tokens Token Meaning %m Timestamp with milliseconds %u Username %d Database name %p Process ID After this change, log lines should look like this: Example log line 2026-06-22 19:00:00.070 UTC user=pgadmin db=highcpu pid=3805603: LOG: statement: SELECT 1 FROM pg_extension WHERE extname='pg_stat_statements' The matching pgBadger prefix for this log format is: Matching pgBadger prefix %m user=%u db=%d pid=%p: You will use this same value later in the pgBadger command. What good looks like: The server parameter is saved, and new PostgreSQL log lines begin with timestamp, user, database, and process ID fields that match the pgBadger prefix. Step 3: Enable Server logs for download This is also a one-time setup step. In the Azure portal, open your Flexible Server and go to Server logs. Enable: Portal setting Capture logs for download Set the retention period based on how long you want logs to remain available for download. For example, a 7-day retention period keeps logs available for download for 7 days. In Server logs, enable Capture logs for download and choose the retention window. Figure 2: Enable Capture logs for download and set a retention period long enough to cover the analysis window you want to inspect. What good looks like: After Server logs are enabled, hourly PostgreSQL log files appear in the Server logs blade and can be downloaded from the Azure portal. Once enabled, hourly log files appear in the Server logs blade. The files are named by date and hour, for example: Example log files postgresql_2026_06_22_19_00_00.log postgresql_2026_06_22_20_00_00.log Step 4: Download and organize the logs locally From the Server logs page, select the .log files for the time window you want to analyze and download them. For example, to analyze activity between 19:00 and 21:00 UTC, download: Example files to download postgresql_2026_06_22_19_00_00.log postgresql_2026_06_22_20_00_00.log On your local machine, create a folder for that analysis window. A simple convention is to use the Mon-DD format. Folder name Jun-22 Place the downloaded .log files inside that folder. Your local folder structure should look like this: Folder structure pgbadger-13.1/ pgbadger Jun-22/ postgresql_2026_06_22_19_00_00.log postgresql_2026_06_22_20_00_00.log Step 5: Generate the pgBadger report Open Git Bash from the folder where pgBadger is located. For example, if pgBadger is inside the pgbadger-13.1 folder, open Git Bash from that folder. # Action Command 1 Set the folder FOLDER=Jun-22 2 Confirm files ls -lh ./$FOLDER 3 Run pgBadger Use the full command below. Copy and run FOLDER=Jun-22 ls -lh ./$FOLDER perl -X ./pgbadger -f stderr \ --prefix '%m user=%u db=%d pid=%p:' \ ./$FOLDER/*.log \ -o ./$FOLDER/pgbadger-report.html Command breakdown Part of command Purpose perl -X ./pgbadger Runs pgBadger and suppresses non-critical Perl warnings. -f stderr Parses PostgreSQL stderr log files. --prefix '%m user=%u db=%d pid=%p:' Matches the log_line_prefix set on the server. ./$FOLDER/*.log Analyzes every .log file in the selected folder. -o ./$FOLDER/pgbadger-report.html Writes the HTML report into the same folder. When the command completes successfully, you should see output like this: Expected output Parsed 12134249 bytes of 12134249 (100.00%), queries: 26684, events: 83 LOG: Ok, generating html report... What good looks like: pgBadger finishes parsing the logs and creates pgbadger-report.html in the selected folder. Step 6: Open the report Open the generated report: Copy and run start ./$FOLDER/pgbadger-report.html The report opens in your default browser. The final report is created here: Generated report path Jun-22/pgbadger-report.html What the report can show The pgBadger report gives you a quick view into the workload shape for the selected log window. For example, in a sample run across two hourly log files, pgBadger summarized: Total number of queries. Number of unique normalized queries. Query traffic over time. Events such as errors and fatal messages. Session and connection patterns. Once the report opens, start with Global Stats to confirm the time range, total queries, normalized queries, and query peak. Figure 3: Start with Global Stats to validate the selected time range, total query count, normalized query count, and query peak. Query volume and normalized queries Many raw queries can often reduce to a smaller number of normalized query patterns. This helps identify whether the workload is spread across many different query shapes or dominated by a smaller set of repeated statements. Example: In this sample run, 26,684 queries reduced to 59 normalized query shapes. That suggests the workload is mostly a small set of repeated statements, which can help focus tuning effort. Traffic patterns The SQL Traffic section helps identify spikes, quiet periods, and workload changes over time. Figure 4: Use SQL Traffic to identify query spikes, quiet periods, and workload changes during the selected log window. Figure 5: Review the query breakdown to compare read vs. write volume and query-type distribution for the selected Server logs window. For example, if the report shows a steady baseline followed by a sharp spike, that spike can be correlated with application activity, batch jobs, synthetic tests, or operational events during the same time window. Query duration If query duration shows 0 ms or the slow query sections are empty, it usually means duration logging was not enabled when the logs were collected. In that case, pgBadger can still show query counts and events, but it cannot calculate the slowest queries, total execution time, average duration, or maximum duration. To unlock those timing sections, enable log_min_duration_statement , collect fresh logs, and rerun pgBadger. What pgBadger cannot infer from missing logs pgBadger reports are only as complete as the log data you provide. If PostgreSQL did not log duration, lock waits, temporary files, or autovacuum activity during the selected time window, pgBadger cannot reconstruct those details later. To analyze... Enable before collecting logs Slow queries log_min_duration_statement Lock waits log_lock_waits Temporary files log_temp_files Autovacuum activity log_autovacuum_min_duration Repeatable copy/paste block Reusable command block Change only FOLDER for each new analysis window. Copy and run FOLDER=Jun-22 ls -lh ./$FOLDER perl -X ./pgbadger -f stderr \ --prefix '%m user=%u db=%d pid=%p:' \ ./$FOLDER/*.log \ -o ./$FOLDER/pgbadger-report.html start ./$FOLDER/pgbadger-report.html For another date, change only this line: Update this value FOLDER=Jun-22 Examples: Example folder values FOLDER=Jun-23 FOLDER=Jul-01 FOLDER=Aug-15 Optional: Improve report quality pgBadger can only analyze the information captured in PostgreSQL logs. The default logs may be enough for query frequency, connection activity, and errors. For deeper performance troubleshooting, consider enabling additional logging parameters based on your scenario. Scenario Parameter Suggested value Notes Slow query analysis log_min_duration_statement 1000 Logs statements slower than 1 second. Short controlled test log_min_duration_statement 0 Logs every statement. Use carefully. Lock troubleshooting log_lock_waits on Helps identify lock waits. Temporary file analysis log_temp_files 0 Logs all temporary files. Autovacuum visibility log_autovacuum_min_duration 0 Useful during focused analysis. Useful parameters include: Recommended logging parameters log_lock_waits = on log_temp_files = 0 log_autovacuum_min_duration = 0 To capture query durations, configure: Duration logging log_min_duration_statement = 1000 This logs statements that run longer than 1000 milliseconds. For short test runs, you can temporarily use: Short test run only log_min_duration_statement = 0 Caution: Use log_min_duration_statement = 0 carefully on busy production servers. It logs every statement and can generate a large volume of logs. Duration matters: If duration logging is not enabled, pgBadger can still show query counts and events, but slowest-query, total duration, average duration, and maximum duration sections will be limited or empty. Common mistakes and quick fixes Symptom Likely cause Fix Report is empty Prefix mismatch Match --prefix with log_line_prefix . No duration data Duration logging was not enabled Set log_min_duration_statement before collecting logs. No files visible Server logs disabled or retention expired Enable capture and check retention. pgBadger command fails pgBadger is not in the current folder or path Run pgbadger -V to confirm installation. Common troubleshooting FAQs 1. Report is created but empty This usually means the pgBadger prefix did not match the actual log format. Check the first few lines: Copy and run head -5 ./$FOLDER/*.log Make sure the pgBadger --prefix matches the server’s log_line_prefix . 2. Report shows queries but no duration PostgreSQL logged statements but did not log durations. Enable one of the following, collect fresh logs, and rerun pgBadger: Parameter options log_min_duration_statement = 1000 # or temporarily for testing log_min_duration_statement = 0 3. No .log files are visible Confirm that Server logs are enabled: Portal setting Capture logs for download Also check the retention period. If the retention period has expired, older logs may no longer be available for download. 4. pgBadger command fails Confirm that pgBadger is available in the current folder or installed in your path. Copy and run pgbadger -V If you are running pgBadger from the local folder, use: Copy and run perl -X ./pgbadger Summary For customers already using Azure Database for PostgreSQL Flexible Server logs, the pgBadger workflow is straightforward: Install pgBadger. Configure log_line_prefix . Enable Server logs for download. Download the .log files. Place them in a local date-based folder. Run pgBadger with the matching prefix. Open pgbadger-report.html . Bottom line: Server logs give you the shortest path from Azure Database for PostgreSQL Flexible Server logs to a pgBadger report. Download the native .log files, run pgBadger with the matching prefix, and open the generated HTML report. References pgBadger - source and documentation GitHub pgBadger - project site Azure - Download server logs from the portal Flexible Server Azure - Logging concepts Flexible Server Azure - Configure server parameters via the portal PostgreSQL - log_line_prefix and logging parametersFrom RAG to agents: Build AI pipelines inside Azure HorizonDB
By Abe Omorogbe, Navya Teja Gajula, Binnur Gorer, B Harsha Kashyap, Krishnakumar Ravi (KK) from Microsoft PostgreSQL AI team If you’ve ever shipped a RAG app, this will feel familiar. Your data lives in Postgres. But the pipeline that turns that data into vectors lives somewhere else, spread across external services, queues, and retry logic. And when the embedding API hiccups mid-batch? That’s a 2 a.m. production incident. You didn’t set out to build your own embedding service. You just wanted to search your documents. And RAG is only the beginning. The moment AI works on your data: extraction, summarization, reranking, keeping embeddings fresh, or powering agent, you’re back to stitching together more services, queues, and glue code, all outside the database. AI pipelines in Azure HorizonDB (Preview) removes that entire stack. Define your workflows steps like chunking, embeding, extracting, and generating in SQL, and HorizonDB runs them as AI pipelines next to your data. No orchestrator. No glue code. Just Postgres. In this post we'll cover: The external-orchestrator issue that every AI on Postgres team eventually hits What AI pipelines are, and the four-part anatomy that makes them click Use cases worth trying: semantic search, knowledge extraction, content generation, smarter reranking, and always-fresh embeddings How to watch your pipelines run as live graphs in VS Code How to spin up HorizonDB and run your first pipeline today 🚀 Try it on Azure HorizonDB. AI pipelines are built into Microsoft's new PostgreSQL cloud service, no extra infrastructure to stand up. Write ai.create_pipeline(...), call ai.run(...), and it runs. Get started in HorizonDB → AI preprocessing runs outside the database, far from your data The standard way to get data into a vector store looks reasonable on a whiteboard: a service reads source rows, calls an embedding API, and writes chunks back to Postgres. However, some interesting issues often occur in production. The embedding API fails mid-batch, and there's no shared checkpoint showing which rows were completed. You rerun the job, and the extra API calls increases cost. A worker crashes after writing chunks but before flipping the parent row's processed flag. Now your embeddings are quietly inconsistent, and nobody knows. Every one of these is the same missing primitive: durable, checkpointed execution that lives where your data lives. External orchestrators can do it, but now you're operating a second service just to feed the first one. AI pipelines move that logic into HorizonDB itself. The source, the steps, the sink, and the full run history are all SQL protected by the same transactions, backups, and point-in-time restore your data already has. The database is already where your data commits. It's a natural place for the pipeline to live too. Anatomy of an AI pipeline in HorizonDB are optional and can be adjusted as needed. A pipeline has four parts: Source: where rows come from. A table_source(...) over a HorizonDB table, optionally with an incremental_column so the pipeline skips rows it already processed. Steps: the AI operations that transform each row, in order. Each step appends columns to the in-flight batch. Sink: where results land, ready for use by your AI apps or agent. Trigger: 'on_change' (run automatically when source rows change) or 'manual' (run only when you call ai.run()). Those four parts give the pipeline its shape. The steps are where you define the AI work itself, using composable building blocks: Step What it does ai.chunk() Split long text into overlapping chunks ai.embed() Generate vector embeddings ai.extract() Pull structured fields out of text with an LLM ai.generate() Generate text from a prompt (i.e content generation, classify, summarize and more) ai.rank() Score documents against a query How the pieces fit together. The ai.* API gives you the AI pipeline shape: sources define where data comes from, steps define the AI work to perform, sinks define where results land, and triggers define when the pipeline runs. Under the covers, HorizonDB turns that definition into a durable execution graph, where each step can be checkpointed, retried, and resumed if something fails. Built on open source. That durability isn't magic, every AI pipeline compiles down to a graph that runs on pg_durable, Microsoft's open-source durable-execution engine for PostgreSQL (built on the duroxide Rust runtime). The ai.* API is the AI-shaped surface (sources, steps, sinks, triggers) and pg_durable is the general-purpose engine underneath that handles checkpointing, retries, and crash recovery. So, your pipelines stand on a transparent, inspectable foundation you can read, and run on any Postgres 17 & 18. No black box, no lock-in. Use case 1: Semantic search over your data This is one of the most popular use cases. Turn a table of documents into searchable vectors, durably, and keep them fresh as the data changes. That last part matters: in production, documents are edited, added, and deleted constantly, and every change needs the right chunks and embeddings updated without reprocessing the entire corpus or leaving stale vectors behind. With AI pipelines, HorizonDB can track those incremental updates for you. Chunk the body, embed each chunk, and land the result in a DiskANN-indexed table. -- Define the pipeline: source -> chunk -> embed -> sink. SELECT ai.create_pipeline( name => 'rag_pipeline', source => ai.table_source(table_name => 'documents'), steps => ARRAY[ ai.chunk(input => 'content', chunk_size => 512, overlap => 64), ai.embed(model => 'default-embedding', input => 'chunk_text', dimensions => 1536) ], trigger => 'on_change', -- re-embed automatically as rows change sink => ai.table_sink('rag_pipeline_output') ); -- Run it SELECT ai.run('rag_pipeline'); -- Search your data SELECT chunk_text, embedding <=> azure_openai.create_embeddings('text-embedding-3-small', 'how does vector search work?')::vector AS distance FROM rag_pipeline_output ORDER BY distance LIMIT 3; 📘 Read more details in the AI Pipelines documentation That's the entire ingestion layer; chunking, embedding, checkpointing, retries, and sink writes in one definition. Because trigger => 'on_change', the pipeline updates embeddings whenever source rows change, processing only what is new or modified instead of redoing the whole corpus. Your vectors stay in sync with your data, and your ingestion work stays efficient as the dataset grows. Point a query at the DiskANN index and you've got production semantic search without a single line of application glue. That's the whole loop: define, run, inspect. The embedding service you were about to build the queue, the workers, the retry logic, the checkpoint table, the 2 a.m. production incident doesn't happens. Why it's better than an external service: a failure in ai.embed() never re-runs ai.chunk(), each step is a durable node. If the database restarts mid-run, it resumes from the last checkpointed batch, not row zero. Use case 2: Turn unstructured text into structured metadata Support tickets, contracts, product reviews, research papers are full of structure that's locked inside unstructured documents. ai.extract() pulls named fields out of text and merges them into the metadata JSONB column, so you can filter and aggregate on things an LLM read for you. SELECT ai.create_pipeline( name => 'extraction_pipeline', source => ai.table_source(table_name => 'documents'), steps => ARRAY[ ai.chunk(input => 'content'), ai.extract( input => 'chunk_text', data => ARRAY['topics: string - the main topics discussed', 'entities: string - named people, products, or places'] model => 'my-gpt' -- optional, the default model when AI model management is activate ) ], sink => ai.table_sink('extraction_pipeline_output') ); SELECT ai.run('extraction_pipeline'); -- Now query the structured fields the LLM extracted: SELECT doc_id, metadata->'topics' AS topics, metadata->'entities' AS entities FROM extraction_pipeline_output; 📘 Read more details in the AI Pipelines documentation You describe each field as a label: description string in the ai.extract step, and HorizonDB does the rest durably, in bulk, with the same retry-and-resume guarantees. Each field is a label, either a bare name like product, or the detailed form name: type - description (for example `sentiment: number - sentiment score from 1 to 5`). HorizonDB does the rest, durably, in bulk, with the same retry-and-resume guarantees. Use case 3: Summarize and rewrite content at scale ai.generate() runs an LLM prompt against every row, perfect for bulk summarization, classification, tone normalization, or generating titles. Because it's a pipeline, "summarize 4 million documents" becomes a job that survives restarts instead of a script you have to monitor overnight. SELECT ai.create_pipeline( name => 'summary_pipeline', source => ai.table_source(table_name => 'documents'), steps => ARRAY[ ai.chunk(input => 'content'), ai.generate( input => 'chunk_text', system_prompt => 'Create a concise summary in 50 words or fewer.' model => 'my-gpt' -- optional, the default model when AI model management is activate ) ], sink => ai.table_sink('generation_pipeline_output') ); SELECT ai.run('summary_pipeline'); -- Now query the generated text: SELECT doc_id, left(generated_text, 100) AS summary_preview FROM generation_pipeline_output WHERE generated_text IS NOT NULL LIMIT 5; 📘 Read more details in the AI Pipelines documentation Swap the system_prompt and the same shape becomes a classifier ("Label this ticket as billing, bug, or feature request"), a translator, or a headline generator. The instruction goes in system_prompt; the result lands in generated_text. Use case 4: Keep embeddings fresh, and re-embed cleanly when the model changes This is where AI pipelines become especially useful. In a real AI app, two things change constantly: your data and your model. AI pipelines are designed to handle both changes directly. Your data changes. Set incremental_column and an on_change trigger, and the pipeline only embeds new or changed rows, automatically, forever, until you pause or drop it. SELECT ai.create_pipeline( name => 'rag_pipeline', source => ai.table_source( table_name => 'documents', incremental_column => 'updated_at' -- only process what changed ), steps => ARRAY[ ai.chunk(input => 'content'), ai.embed(model => 'default-embedding', input => 'chunk_text', dimensions => 1536) ], trigger => 'on_change', sink => ai.table_sink('rag_pipeline_output') ); Your model changes. Bump the model or the dimensions, then run a single, resumable backfill, no migration script, no babysitting: TRUNCATE rag_pipeline_output; SELECT ai.backfill('rag_pipeline'); 📘 Read more details in the AI Pipelines documentation The backfill runs as one durable instance. If the database restarts mid-backfill, it picks up from the last checkpointed batch instead of starting over. The painful "re-embed everything" migration becomes a one-liner you can actually trust. Watch your pipelines run as live graphs in VS Code A pipeline you can see is a pipeline you can trust. Install the PostgreSQL extension for VS Code, connect to HorizonDB, then right-click your database and open Pipelines & Workflows → AI Pipelines. Select any run and the center pane renders the execution as a color-coded graph: Blue 🔵 : source and sink (where data enters and exits) Green 🟢 : processing steps (chunk, embed, extract, generate, rank) Pink 🟣 : external model and service calls For each run you can read the status (completed, running, failed), the run ID for traceability, start time and duration for performance, and a link back to the pipeline definition. When a run fails, open the graph and jump straight to the step where execution stopped, no log spelunking. Get Started: Try It Now We have a few demoes of AI pipelines in action: Resource Link Microsoft Build AI Pipeline Demo Simplify app dev with cloud-native PostgreSQL in Azure HorizonDB | DEM364 Microsoft Build AI Pipeline GitHub AI Pipelines Demo GitHub Repo | DEM364 Microsoft Mechanic Demo AI Pipeline Demo on Microsoft Mechanic Documentation AI pipelines on HorizonDB Enabling AI pipelines takes minutes: enable to azure_ai, pg_durable, vector and pg_diskann extensions and you can get started. -- On Azure HorizonDB — the extensions are built in. CREATE EXTENSION IF NOT EXISTS pg_durable; CREATE EXTENSION IF NOT EXISTS azure_ai; CREATE EXTENSION IF NOT EXISTS vector; CREATE EXTENSION IF NOT EXISTS pg_diskann; That's it, your PostgreSQL database can now run AI pipelines Learn more MS Learn AI pipelines on HorizonDB: Azure HorizonDB Preview pg_durable on GitHub (open source) MS Learn Durable Functions on HorizonDB Scalable vector search with DiskANN PostgreSQL extension for VS CodeTLS Certificate Pinning and Best Practices in Azure Database for PostgreSQL
TLS certificate pinning in Azure Database for PostgreSQL Transport Layer Security (TLS) encrypts data in transit between client applications and the server and authenticates the service endpoint in client-server authentication. Azure Database server certificates are issued by well-known trusted public Certificate Authorities (CAs), including Microsoft-issued certificates, and are validated by clients during the TLS handshake. Customers do not manage certificates on the server side. Certificate pinning is a client-side security technique where an application restricts trust to a specific certificate, for example by thumbprint, public key, or CA, rather than relying solely on the default OS or platform trust store. The trust store contains pre-installed root CAs and may also include additional certificates configured by the client. During standard TLS validation, the client will trust any server certificate that chains to one of those root CAs. Why detecting TLS certificate pinning is not possible by design Certificate pinning is entirely client-side logic. The server has no visibility into whether pinning is configured on the client. From the server’s perspective, the client either completes the TLS handshake or aborts it. The server never sees: Which certificate(s) the client trusts Whether the client is comparing root CA, intermediate CA, leaf certificate or SPKI hash Whether the trust decision was static or dynamic What the server can see is TLS handshake failure patterns, TLS protocol, and cipher negotiation. Why certificate pinning is risky While certificate pinning was historically used to reduce the risk of man-in-the-middle attacks, it introduces significant operational fragility in cloud environments, particularly during certificate rotations. Server certificates and certificate authorities (CAs) must be rotated periodically to maintain security and compliance. In Azure Database for PostgreSQL, when certificate pinning is used, clients bind trust to a specific certificate or CA. As a result, any change to the server certificate chain—including CA updates—can cause connection failures, even when the new certificates are fully valid and secure. One of the most common complications during certificate rotations is certificate pinning. Recommended TLS certificate trust model for Azure PostgreSQL Instead of pinning, adopt a CA‑based trust model that allows certificates to change safely. Trust root CAs, not individual certificates. Configure clients to use standard TLS validation against Azure-documented root CAs, rather than restricting trust to specific certificates or a narrowly scoped set of certificate authorities. Avoid configurations that effectively implement certificate pinning—such as trusting only a single certificate, public key, or limited CA set—unless explicitly required. Maintain a flexible and up-to-date trust store Clients rely on a trust store, key store, or equivalent certificate bundle to validate server certificates during TLS negotiation. Include the appropriate root and intermediate certificate authorities (CAs) required to validate the server certificate chain Ensure that trust stores are periodically reviewed and updated in line with provider guidance and announced certificate authority changes For the current TLS certificates visit the Azure Database for PostgreSQL documentation. Use certificate validation modes that rely on standard CA-based trust rather than pinning For PostgreSQL client configurations, prefer: sslmode=verify-ca Validates the server certificate chain against trusted CAs sslmode=verify-full Verifies CA and hostname match These modes ensure that clients validate the server certificate chain against trusted CAs, and in stricter modes, verify hostname identity. They do not imply certificate pinning by themselves. They rely on standard CA-based trust. Configurations only become rigid when trust is narrowly restricted, such as to a single certificate or limited CA set, often through custom or overly constrained trust stores. This effectively introduces certificate pinning. When properly configured, these modes authenticate the service endpoint and protect against spoofing, while remaining resilient to certificate rotations. Maintain a combined CA during certificate rotations Azure may rotate root or intermediate CAs over time. When Azure announces a CA rotation: Add newly required root CAs to the client trust store before the rotation begins. Retain existing trusted root CAs until the transition is fully complete. Avoid removing older root certificates prematurely. If specific rotation guidance includes updates related to intermediate CAs, follow the service-specific instructions provided for that rotation. This combined CA approach, using both the current and upcoming certificate authorities during the transition window, allows clients to continue validating the server certificate chain without interruption. As you review your current client configurations, ensure your applications rely on CA-based trust, avoid overly restrictive certificate configurations such as certificate pinning, and are prepared to handle routine certificate rotations without disruption. For a deeper dive, see the full article: TLS Certificate Pinning in PostgreSQL and MySQL: Risks, Rotations, and Best Practices.Multitude builds resilient banking platform with PostgreSQL and MySQL on Azure
Expanding into new markets is usually a sign that things are going well. For digital banking platforms, however, growth brings a different kind of challenge - more customers, more data, and stricter expectations around availability, security, and regulatory compliance. At Multitude, we operate across 17 countries and deliver digital banking, credit services, payment processing, and regulatory reporting through a platform composed of more than 400 microservices. Each service encapsulates a defined business capability, including onboarding, risk assessment, collections, and compliance workflows. Historically, our services relied on on-premises PostgreSQL and MySQL environments deployed within our own data centers, where capacity scaled vertically on shared compute and storage resources. This model created contention between unrelated workloads and limited their ability to scale independently. Expanding capacity required adding or upgrading physical hardware, which involved demand forecasting, procurement, delivery coordination, and installation within the data center. Over time, continued growth amplified these architectural constraints. The database engines themselves remained reliable, but the surrounding infrastructure limited elasticity and domain-level isolation. As a result, sustained growth began to expose structural limits in the underlying infrastructure. "In a regulated financial environment, those constraints carried broader implications. Frameworks such as DORA and GDPR require predictable availability, controlled recovery procedures, and governed access to sensitive data. As workload demands increased, sustaining both growth and compliance required structural changes at the database layer. We decided that redesigning our data architecture was necessary to improve workload isolation, scalability, and governance alignment. Rearchitecting data boundaries with Azure Databases We initiated our architectural redesign by migrating database workloads to Microsoft Azure and standardizing on Azure Database for PostgreSQL and Azure Database for MySQL for core application services. Central to this redesign was the adoption of bounded contexts. Each bounded context represents a logical business domain and encapsulates the services and schemas required to support that capability. Each domain is owned and managed by a single team, aligning technical boundaries with team responsibility and accountability. Rather than maintaining a small number of large, shared database instances, we provisioned dedicated database instances aligned to defined business domains, establishing domain-level isolation at the database layer. Today, approximately 35 database instances support more than 400 microservices across the platform. Each instance may host multiple schemas serving related services within the same domain, while cross-domain database dependencies are intentionally avoided. This structure limits the blast radius of configuration changes or workload spikes and allows scaling adjustments to be applied within clearly defined domain boundaries. While the bounded context model was a strategic architectural decision, leveraging managed database services helped us implement it by drastically reducing the operational overhead of provisioning, scaling, and maintaining independent instances across domains. Azure Database for PostgreSQL and Azure Database for MySQL provide the managed capabilities required to sustain this model. Instances are provisioned according to the performance and storage requirements of each domain and can be adjusted as workload characteristics evolve. Compute and storage resources are scaled at the instance level, allowing capacity changes to be applied to a specific bounded context without affecting unrelated domains. Altogether, these architectural decisions balance domain-level isolation with operational manageability. A database-per-microservice pattern would significantly increase provisioning, monitoring, and lifecycle overhead without materially improving data ownership boundaries. By grouping related services within bounded contexts, we maintain clear domain alignment while keeping the number of database instances practical to operate. As a result, data boundaries, scaling behavior, and operational controls remain consistent with business domain structures across the platform. Operationalizing high availability and backup strategy To support availability, we deploy Azure Database for PostgreSQL and Azure Database for MySQL with zone-redundant high availability, placing primary and standby replicas in separate availability zones within the same Azure region. Replication preserves transactional consistency, and zone separation reduces exposure to localized infrastructure failures. We periodically exercise failover procedures as part of operational validation to confirm recovery behavior under defined conditions. Availability controls are complemented by a layered backup strategy. Azure Database for PostgreSQL and Azure Database for MySQL provide automated backups with a retention window of up to 35 days and point-in-time restore capabilities. These features allow us to restore a database to a specific timestamp within the retention window, supporting recovery from application-level errors or unintended data modifications without custom snapshot orchestration. Together, operational backups and governed archival retention address both short-term recovery and long-term compliance obligations. Restore operations require documented justification and follow established approval workflows, ensuring that recovery actions remain controlled, traceable, and auditable. We also enforce consistency through lifecycle management. Azure’s managed service model standardizes engine patching and version updates across environments, reducing configuration drift and minimizing manual coordination. By operating within the managed service boundary, the database team can focus on workload analysis, performance tuning, and capacity planning. For migration and synchronization scenarios, we use Azure Data Migration Service to orchestrate controlled cutovers between database environments. Engineers validate configuration and readiness before initiating synchronization, after which Azure-managed replication then maintains data alignment until final switchover. Provisioning decisions and structural modifications remain subject to internal governance approvals to preserve change control and oversight. By combining zone-redundant availability, structured recovery workflows, governed retention policies, and standardized lifecycle management, we operate a database layer engineered for resilience, auditability, and regulatory alignment at scale. Compliance as an architectural property For us, governance is embedded directly into how the platform operates, beginning at the identity layer. Access to Azure Database for PostgreSQL and Azure Database for MySQL integrates with Microsoft Entra ID, aligning database authentication with centrally managed corporate identities. Role-based access control is enforced through enterprise identity policies, providing centralized visibility into access assignments and authentication events across environments. These controls extend into production access management. Privileged access is approval-based and time-bound, and administrative roles are not permanently assigned. Access requests follow defined workflows, and all privileged actions are logged for review under established oversight procedures, ensuring traceability of operational interventions. Database isolation reinforces these identity controls. By aligning database instances with bounded contexts, each business domain maintains a discrete data boundary at the database layer. This structure limits lateral access across domains and confines sensitive data to clearly defined ownership scopes, simplifying monitoring and audit review. In a regulated financial environment, these architectural controls also support compliance requirements under frameworks such as DORA and GDPR. By embedding identity integration, domain isolation, and lifecycle controls directly into the platform architecture, governance becomes an operational property of the system rather than a separate procedural layer. The simplicity of this architecture is a strong driver for both auditability and security of the whole platform. Measurable impact across engineering teams and business outcomes Beyond improved stability, our ability to respond to growth has changed significantly since moving to Azure. In the past, expanding database capacity meant procuring hardware and planning installation in the data center. Now, capacity adjustments happen directly within Azure and can be applied to individual databases instances, allowing us to scale in near real time as workload demands change. Maintenance effort has also decreased. Managed patching, version alignment, and automated backups have reduced the need for manual coordination and reactive capacity management. Infrastructure-level tasks that once required continuous oversight are now handled within the managed service boundary. Our DBAs are now focused on improving performance and stability. We spend far less time maintaining the basics. Resilience by design The structural changes behind these results reflect a deliberate long-term strategy. Our database architecture now aligns with the operating model we expect to sustain over the next five years and beyond. Bounded contexts define discrete data domains, while Azure Database for PostgreSQL and Azure Database for MySQL provide managed high availability, scaling controls, and standardized lifecycle management across those domains. Identity integration and governed recovery procedures operate consistently across environments. With this architecture in place, Multitude scales responsibly in regulated markets while maintaining strict governance and availability standards. Expanding into new markets still means more customers and more data - but now our platform is designed to handle that success.Building an Azure architecture that’s ready for every signature
At Exclaimer, we help organizations manage email signatures at scale, so every message can carry a consistent, compliant, on-brand signature without IT teams manually updating thousands of mailboxes. This is more difficult than it may seem, especially when you're doing it for more than 80,000 customers, around 9.6 million seats, and more than 21 billion emails a year. Every signature must show up in the right place, with the right details, for the right sender, recipient, device, and business rule. Behind that are constantly changing employee records, customer-specific policies, email chains, recipient lists, regional disclaimers, and brand requirements. Because our platform sits directly in the email flow, availability is critical. And because many of our customers operate in regulated industries, they also need confidence that data stays in-region and configured signatures are applied consistently. To support that level of scale and reliability, we’ve spent the last several years evolving our architecture on Microsoft Azure. Today, Azure Kubernetes Service (AKS), Azure SQL Database, Azure Database for PostgreSQL, Azure Cosmos DB, Azure Data Explorer, and Azure Databricks help us run a global platform that’s more responsive, more resilient, and more cost-efficient. Reading the signs that our architecture needed to change In the beginning, our cloud product ran more like a multi-server, on-premises product hosted on Azure Virtual Machines (VMs). The platform was split into a smaller number of core services, and the team relied heavily on VM-based infrastructure to keep those services running. As Exclaimer grew, our architecture had to keep pace with higher volumes, more regions, and more complex customer requirements. Regional demand shifted throughout the day, but scaling infrastructure up and down still relied on scripts, pre-baked VMs, and operational coordination. That created more risk during maintenance and failover. We run parallel data centers in regional pairs so we can move traffic away from one site when needed. But when traffic moves, the receiving environment has to be ready to handle the full load. In the VM world, that meant someone or something had to remember to scale up standby resources at the right moment. At the same time, our product was becoming more service-oriented. We were moving away from a smaller set of larger services toward well over 100 microservices. Every new service created more conversations about VM sizing, images, patching, and operational overhead. It was time for a model that could scale faster, run more efficiently, and reduce the amount of infrastructure work required to ship and operate the product. Signing on to AKS for faster, more efficient scaling By moving many workloads to Linux containers on AKS, we gained a smaller footprint, faster startup times, and a more consistent way to package and deploy services. AKS also gave us a managed Kubernetes foundation for running those containers at global scale, with autoscaling capabilities that better matched our traffic patterns. With Horizontal Pod Autoscaler, services can react to load in seconds rather than minutes. With Cluster Autoscaler, we can add or remove node capacity based on what the platform actually needs. That means we can pack workloads onto nodes more efficiently, scale down during quiet periods, and scale up quickly when demand returns. The operational difference is just as important. During an incident, maintenance event, or regional failover, our teams have fewer manual steps to think about. If traffic shifts, the platform can scale with it. That takes away one more thing for engineers to worry about when they should be focused on keeping the customer experience steady. The move to containers and a more streamlined CI/CD workflow also improved our deployment cadence by making it easier to build, test, and deploy changes across the platform. In 2021, we deployed 285 changes, features, and fixes to production over the course of the entire year. Today, we deploy that many every few days. Cost has improved, too. Since 2024, when the bulk of our migration to containerized services took place, we’ve reduced our average cost per user by about 39 percent, even as the product has grown more complex and we’ve added more capabilities for customers. We achieved that through a combination of containerized architecture, AKS autoscaling, and expanded reservations across compute and storage technologies. Choosing the right database for the right kind of data We started with a strong Microsoft SQL Server foundation, and Azure SQL Database remains core to our platform today. It stores critical customer configuration data and continues to give us the reliability, replication, resizing flexibility, and regional scale we need. But not every workload belongs in the same database. Customer configuration, relational service data, key-value storage, usage events, and business intelligence (BI) all have different access patterns. That principle led us to Azure Database for PostgreSQL flexible server for one of our most important migrations. We had used Azure Table storage for a core service that needed to retrieve customer data quickly. It was cost-effective and stable for a long time, but as the product evolved, the data became more relational, and we found ourselves adding complexity in application code that a relational database could handle more naturally. Azure Database for PostgreSQL gave us that relational model with low management overhead, fast read replicas, reserved instances for predictable workloads, and a path to future scale. After the migration, average request time for a critical service dropped from 18.6 milliseconds to 1.79 milliseconds. That’s a 90 percent improvement across a service that handles around 9 billion requests each month. Azure Cosmos DB plays a different role, supporting key-value and document storage where we need scale, availability, low latency, encryption at rest, and straightforward dev/test support. Optimized for unstructured data and high-performance reads and writes, it gives us a highly scalable foundation for workloads that don't fit a traditional relational model. We use it to store customer assets for signatures and video branding, high-volume metadata for internal message-processing operations, audit events that help customers track account changes, and tokens used to collect data from third-party systems on behalf of customers. It also gives us a clean way to keep data and services aligned. Azure Data Explorer solved another scaling challenge: usage and billing data. We need to be able to audit the number of messages we process for our customers so we can bill accurately, and at more than 20 billion emails a year, our previous SQL-based usage pipeline became difficult to manage. With Azure Data Explorer, we can ingest massive volumes of event data at low storage cost, connect to Azure Event Hubs, and avoid maintaining custom plumbing. That move reduced the cost of the system by around 70 percent. Azure Databricks rounds out the picture as our BI and data platform, giving our teams a shared foundation for transformations, analysis, and reporting across product and business data. Keeping every region ready for business Our customers are everywhere, so our platform has to be, too. Exclaimer runs in seven distinct geographic locations: Australia, Canada, Europe, Germany, the United Arab Emirates, the United Kingdom, and the United States. That global footprint helps us meet customer expectations around availability and data residency. Many organizations want their data to stay in-region, and Azure gives us the coverage we need to support that. Availability is especially important because our platform is part of a live communication flow. When someone sends an email, they expect it to keep moving. Our Azure architecture helps us support that expectation across the stack. AKS lets compute scale with regional demand. Azure SQL and Azure Database for PostgreSQL support critical relational workloads. Azure Cosmos DB gives us scalable, low-latency storage for document and key-value patterns. Azure Data Explorer handles very high-volume usage ingestion without the complexity of our former custom pipeline. Across the board, these managed Azure services reduce the amount of operational work our engineers have to carry. We can spend less time maintaining the basics and more time tuning performance, improving stability, and building the capabilities our customers need next. Building for the future on a stronger foundation The biggest sign that our architecture is working may be how little we have to reinvent when we build something new. As we develop upcoming product capabilities, we already have many of the foundational pieces in place: AKS for compute, Azure Cosmos DB for state, and Azure Service Bus for messaging. We also have Azure SQL for core data, Azure Database for PostgreSQL where relational service data needs room to scale, Azure Data Explorer for high-volume event analysis, and Azure Databricks for BI tooling. Together, these services make our platform faster, more efficient, and more resilient. Email signatures may look simple on the surface. Behind every one, there’s a set of decisions about performance, scale, data, availability, and trust. With Azure, we’ve built an architecture that helps us keep every signature moving, wherever our customers do business. About the authors Phil Vetter started in engineering at Exclaimer as a developer at the start of 2013, and now sits at the helm as VP of Engineering. Lee Jones started at Exclaimer in 2013 in the IT department, and now serves as Director of Platform Engineering, managing the infrastructure and resilience of Exclaimer Cloud.Monitoring and using pg_repack in Azure Database for PostgreSQL flexible server
In this post: we will walk through how to configure and use the pg_repack extension in Azure Database for PostgreSQL flexible server. We will also cover how to run pg_repack on a table, monitor the progress during execution, and validate the results after the repack operation is completed. Why Monitor pg_repack During Execution? While running pg_repack is straightforward, administrators often need visibility into what is happening behind the scenes, especially when working with large tables in production environments. Monitoring the operation provides several benefits: Verify that pg_repack is actively running and has not stalled. Identify the current phase of the operation, such as table copying, index rebuilding, or final table swap. Understand resource usage and the impact on the database. Before you start: Before performing this lab, ensure the following prerequisites are met: Azure Resources: An active Azure subscription An Azure Database for PostgreSQL flexible server instance Database Requirements: A table with a PRIMARY KEY or UNIQUE NOT NULL index (required by pg_repack) Linux Machine: I have used an Ubuntu Linux Virtual Machine SSH connectivity to the VM using PuTTY Configuring and using pg_repack in Azure Database for PostgreSQL Step 1: Allow list and create the pg_repack Extension Before using pg_repack, the extension must be allowlisted and created in the target database. Navigate to your Azure Database for PostgreSQL flexible server and add pg_repack to the allow list of extensions. Once the server configuration is updated, connect to the database and create the extension. Step 2: Create and connect to a Linux Virtual Machine Since pg_repack is a client-side utility, a Linux virtual machine was created to install and run the pg_repack client against Azure Database for PostgreSQL flexible server. Step 3: Connect to the Linux Virtual Machine Before running pg_repack, I connected to the Linux virtual machine that would be used to install and execute the pg_repack client. In the Azure portal, navigate to the Linux Virtual Machine. Open PuTTY and enter the VM's IP Address. Select SSH (Port 22) as the connection type and click Open. Enter the VM username and password when prompted. After successful authentication, a terminal session is established The following output confirms that the connection was successful and that the Ubuntu operating system is ready for further configuration. Step 4: Update Package Repositories on the Linux Virtual Machine Before installing the pg_repack client, update the package repositories on the Ubuntu virtual machine to ensure the latest package information is available. Please run the following command and provide the password for the linux virtual machine when prompted. sudo apt update Step 5: Download the pg_repack Important: If you face any version mismatch issue or errors then you can use the below command to resolve After preparing the test environment and generating table bloat, the next step was to download the pg_repack source code to the Linux virtual machine. The git clone command downloads the pg_repack source code from the official GitHub repository to the Linux virtual machine. This source code is later used to build and install the pg_repack client utility required to perform table reorganization operation. After downloading the repository, the cd pg_repack command changes the current directory to the downloaded project folder. git clone https://github.com/reorg/pg_repack.git cd pg_repack Step 6: Install PostgreSQL Client Packages After updating the package repositories, the next step was to install the PostgreSQL client packages on the Linux virtual machine. The PostgreSQL package installs the PostgreSQL client tools, including psql, which is used to connect to Azure Database for PostgreSQL flexible server. sudo apt install postgresql postgresql-contrib When the command is executed, Ubuntu displays a summary of the packages that will be installed along with their dependencies. To proceed with the installation, type Y and press Enter. Step 7: Connect to Azure Database for PostgreSQL flexible server After installing the PostgreSQL client packages on the Linux virtual machine, the next step is to establish a connection to the Azure Database for PostgreSQL flexible server using the psql client. Please update the following command with your server details and execute it and enter your PostgreSQL server user password to establish the connection. Note: Make sure your Linux machine network is allowed to connect on your Azure Database for PostgreSQL flexible server. psql -h <Hostname> -p 5432 -U <username> postgres Step 8: Create a Test Database After successfully connecting to Azure Database for PostgreSQL flexible server, a dedicated database was created to perform the pg_repack lab activities as shown below: Step 9: Connect to the Newly Created Database After creating the repack_lab database, connect to it before proceeding with the pg_repack activities. \c repack_lab Step 10: Create a Sample Table for pg_repack Testing After connecting to the repack_lab database, I created a sample table that would be used throughout the lab to test the functionality of pg_repack. CREATE TABLE test_table ( id SERIAL PRIMARY KEY, name TEXT, created_at TIMESTAMP DEFAULT NOW() ); Step 11: Insert Sample Data into the Test Table After creating the test_table, the next step was to populate it with sample data. This helps simulate a realistic workload and provides enough records to demonstrate how pg_repack works. The following command was used to insert 100,000 rows into the table: INSERT INTO test_table(name) SELECT md5(random()::text) FROM generate_series(1,100000); Step 12: Create an Index on the Test Table After populating the test_table with 100,000 records, an index was created as shown below: CREATE INDEX idx_test_name ON test_table(name); Step 13: Check the Initial Table Size Before generating table, bloat and running pg_repack, it is useful to capture the current size of the table. This serves as a baseline for comparing storage consumption before and after the repack operation. SELECT pg_size_pretty(pg_total_relation_size('test_table')); Step 14: Generate Table Bloat Using UPDATE Operations To demonstrate how pg_repack reorganizes a table and reclaims unused space, the next step was to generate table bloat by repeatedly updating all rows in the table. The following command was executed multiple times: UPDATE test_table SET name = md5(random()::text); Step 15: Disable Autovacuum on the Test Table To clearly observe table bloat and demonstrate the effectiveness of pg_repack, autovacuum was temporarily disabled on the test table. This prevents Azure Database for PostgreSQL flexible server from automatically cleaning up dead tuples generated by the previous UPDATE and DELETE operations. ALTER TABLE test_table SET (autovacuum_enabled = false); Step 16: Analyze Live and Dead Tuples Before Running pg_repack After generating table bloat through multiple UPDATE and DELETE operations and disabling autovacuum, the next step was to measure the number of live and dead tuples in the table. Step 17: Execute pg_repack to Reorganize the Table The pg_repack utility was executed against the test table to reclaim unused space and reorganize the table structure. The pg_repack utility reorganizes tables and indexes online while minimizing locking and application downtime. Unlike VACUUM FULL, pg_repack performs the reorganization in the background and requires only a brief lock during the final table swap operation. Command executed: pg_repack \ --host=myflexibleserver.postgres.database.azure.com \ --port=5432 \ --username=dbadmin \ --dbname=repack_lab \ --table=test_table \ --jobs=2 \ --no-kill-backend \ --no-superuser-check Monitoring pg_repack Execution Once the pg_repack operation was initiated, the next step was to monitor its execution and identify the activities being performed by the utility in the background: To track active pg_repack sessions, the following query was executed: SELECT pid, usename, application_name, state, wait_event_type, wait_event, now() - query_start AS running_for, query FROM pg_stat_activity WHERE application_name ILIKE '%repack%' OR query ILIKE '%repack%' ORDER BY query_start; After starting the pg_repack operation, I monitored the active sessions by querying the pg_stat_activity system view. This helped me understand the current stage of the operation and verify that the process was executing successfully. The query returned multiple sessions created by pg_repack, indicating that the utility was actively processing the table. Session 1 - Lock Acquisition LOCK TABLE public.test_table IN SHARE UPDATE EXCLUSIVE MODE This session acquired a SHARE UPDATE EXCLUSIVE lock on the target table. This lock prevents conflicting schema changes while still allowing normal read and write operations during most of the repack process. Session 2 - Temporary Repack Table Creation SELECT 'repack.table_24861'::regclass::oid At this stage, pg_repack was working with an internal temporary table created to hold the reorganized data. This table acts as a replacement for the original table during the repack operation. Session 3 - Creating Primary Key Index CREATE UNIQUE INDEX index_24869 ON repack.table_24861 USING btree(id) This session shows pg_repack rebuilding the primary key index on the new table structure. Session 4 - Creating Secondary Index CREATE INDEX index_24873 ON repack.table_24861 USING btree(name) This indicates that additional indexes are being recreated on the temporary table to match the original table definition. Based on the output, the operation had successfully moved past the initialization phase and was actively rebuilding indexes on the temporary table. This is one of the final stages before pg_repack performs the table swap and completes the reorganization process. Conclusion In summary, monitoring pg_repack execution is essential for ensuring a smooth and efficient table reorganization process. Proper visibility into progress and resource consumption helps administrator complete maintenance tasks confidently while maintaining optimal database performance and availability. References Optimize by using pg_repack - Azure Database for PostgreSQL | Microsoft Learn PostgreSQL: Documentation: 18: 27.4. Progress Reporting pg_repack 1.5.3 -- Reorganize tables in PostgreSQL databases with minimal locks
