Azure SQL Scaling Stuck? Investigating CDC and Transaction Log Pressure
Azure SQL Database makes scaling compute straightforward. But troubleshooting becomes more complicated when several things happen simultaneously: A scale operation remains IN_PROGRESS. A cancellation remains CANCEL_IN_PROGRESS. Transaction-log usage is growing. Change Data Capture (CDC) is enabled. Log truncation reports REPLICATION. Long-running transactions are present. The temptation is to connect all these observations immediately: "CDC is preventing Azure SQL from scaling." That's a hypothesis, not yet a diagnosis. CDC reads changes from the transaction log, and CDC capture latency can delay log truncation under certain conditions. Microsoft specifically documents scenarios where CDC combined with long-running transactions can contribute to continued transaction-log growth. However, that doesn't establish a universal rule that: CDC causes scale operations to become stuck. REPLICATION as a log reuse wait means scaling is blocked. Disabling CDC will resolve a scale operation. A CDC problem and a long-running scale operation necessarily share the same root cause. The safest investigation separates the evidence into layers. 1. Start With the Scale Operation A scale operation changes the resources available to Azure SQL Database. Depending on the type of scaling operation, Azure SQL might need to: Create a compute instance with the requested configuration. Move or copy data when necessary. Redirect connections to the new environment. Terminate existing connections during the final switch. Microsoft documents that Azure SQL Database normally remains online during much of this process, although the final connection switch can cause a short interruption. Long-running transactions can increase the time required for recovery when connections are terminated. So, begin with the operation itself. 2. Check the Operation State Azure SQL Database exposes database-operation status using: sys.dm_operation_status This DMV reports operations such as ALTER and their state. Run the query from the logical server's master database: SELECT session_activity_id, major_resource_id AS database_name, operation, state_desc, percent_complete, start_time, last_modify_time, error_code, error_desc FROM sys.dm_operation_status ORDER BY start_time DESC; Possible states include: PENDING IN_PROGRESS COMPLETED FAILED CANCEL_IN_PROGRESS CANCELLED An important detail is that percent_complete shouldn't automatically be interpreted like a traditional progress bar. For many active operations: 50 = operation in progress and: 100 = operation complete Restore operations have more granular progress values. So: An ALTER operation remaining at 50% doesn't necessarily mean exactly half of the work is complete. 3. Build an Operation Timeline Capture: Scale request time. Requested service objective. Operation start time. Current state. last_modify_time. Cancellation request time, when applicable. Subsequent scale requests. Application impact. Database availability. Workload running during the operation. For example: Time Observation T0 Scale requested T1 Operation enters IN_PROGRESS T2 Transaction-log utilization rises T3 log_reuse_wait_desc reports REPLICATION T4 Cancellation requested T5 Operation still CANCEL_IN_PROGRESS This timeline establishes correlation. It doesn't yet establish causation. 4. Don't Keep Submitting Scale Requests When a scale or cancellation appears slow, avoid repeatedly submitting new modifications without understanding the state of the existing operation. Instead: Preserve the current operation ID. Record the operation state. Record timestamps. Capture current database workload. Investigate the transaction log. Investigate CDC independently. Escalate if the operation remains persistently unable to reach a terminal state. The goal is to reduce variables rather than introduce new ones. 5. Measure Transaction-Log Utilization Now move to the database being investigated. sys.dm_db_log_space_usage is supported in Azure SQL Database and reports how much transaction-log space is currently occupied. SELECT total_log_size_in_bytes / 1024.0 / 1024.0 AS total_log_size_mb, used_log_space_in_bytes / 1024.0 / 1024.0 AS used_log_space_mb, used_log_space_in_percent FROM sys.dm_db_log_space_usage; Capture this repeatedly. You're looking for a trend such as: T1 25% T2 41% T3 58% T4 73% T5 88% versus: T1 60% T2 63% T3 56% T4 61% T5 58% The first pattern suggests sustained net log growth. The second shows that log space is being reused. One snapshot can't tell you which behavior is occurring. 6. Ask Why the Log Can't Be Reused One of the most useful checks is: SELECT name, log_reuse_wait_desc FROM sys.databases WHERE database_id = DB_ID(); The value describes the current condition preventing transaction-log reuse. Microsoft also exposes the same concept through sys.dm_db_log_stats() as log_truncation_holdup_reason. A more complete diagnostic query is: SELECT database_id, total_log_size_mb, active_log_size_mb, total_vlf_count, active_vlf_count, log_truncation_holdup_reason, log_since_last_checkpoint_mb FROM sys.dm_db_log_stats(DB_ID()); sys.dm_db_log_stats() is documented as applicable to Azure SQL Database and returns summary-level information for diagnosing transaction-log health. 7. What Does REPLICATION Mean When CDC Is Enabled? This point requires careful interpretation. CDC uses the transaction log as the source for captured changes. Inserts, updates, and deletes are recorded in the transaction log before CDC processes those changes. Microsoft documents an important interaction: CDC is enabled. Transactions produce log records. CDC must capture relevant log information. Capture latency can hold back log truncation. log_reuse_wait_desc can consequently report REPLICATION. Therefore, if you see: REPLICATION in a CDC-enabled Azure SQL Database, CDC-related log processing is a legitimate investigation path. But don't jump from: "log_reuse_wait_desc = REPLICATION" to: "CDC caused the scaling operation to become stuck." That's a different claim requiring separate evidence. 8. Confirm Whether CDC Is Enabled Azure SQL Database exposes CDC status through sys.databases. SELECT name, is_cdc_enabled FROM sys.databases WHERE database_id = DB_ID(); If: is_cdc_enabled = 1 CDC is enabled for the database. Microsoft documents CDC support in Azure SQL Database and notes that the database must first be CDC-enabled before individual source tables can be tracked. 9. Inventory CDC Capture Instances Rather than using SQL Server Agent job queries, inspect Azure SQL Database CDC metadata. SELECT ct.capture_instance, OBJECT_SCHEMA_NAME(ct.source_object_id) AS source_schema, OBJECT_NAME(ct.source_object_id) AS source_table, ct.start_lsn, ct.create_date FROM cdc.change_tables AS ct ORDER BY source_schema, source_table; This helps answer: Which tables are tracked? How many capture instances exist? When were they created? Has the CDC scope changed recently? An important Azure SQL-specific detail is: Azure SQL Database doesn't depend on SQL Server Agent CDC capture and cleanup jobs. Instead, Azure SQL Database uses a built-in CDC scheduler that automatically runs capture and cleanup within the database. That distinction is critical when adapting SQL Server CDC troubleshooting procedures to Azure SQL Database. 10. Check Whether CDC Is Advancing A simple high-level check is: SELECT sys.fn_cdc_get_min_lsn( ct.capture_instance ) AS minimum_lsn, sys.fn_cdc_get_max_lsn() AS maximum_lsn, ct.capture_instance, OBJECT_SCHEMA_NAME(ct.source_object_id) AS source_schema, OBJECT_NAME(ct.source_object_id) AS source_table FROM cdc.change_tables AS ct; Take repeated samples. The objective isn't to derive a universal threshold. The objective is to determine whether the observable CDC boundary is: Advancing. Intermittently advancing. Apparently static over a meaningful period. Again: Trend matters more than a single snapshot. 11. Investigate Long-Running Transactions This is particularly important when CDC and log growth appear together. Microsoft specifically documents transaction-log growth scenarios involving CDC and long-running transactions. Start by examining database transactions: SELECT dt.transaction_id, dt.database_transaction_begin_time, dt.database_transaction_type, dt.database_transaction_state, dt.database_transaction_log_record_count, CAST ( dt.database_transaction_log_bytes_used / 1024.0 / 1024.0 AS decimal(18,2) ) AS log_mb_used, CAST ( dt.database_transaction_log_bytes_reserved / 1024.0 / 1024.0 AS decimal(18,2) ) AS log_mb_reserved FROM sys.dm_tran_database_transactions AS dt WHERE dt.database_id = DB_ID() ORDER BY dt.database_transaction_log_bytes_used DESC; sys.dm_tran_database_transactions is supported in Azure SQL Database and provides transaction-level database information, including transaction start time and log bytes consumed. 12. Connect Transactions Back to Sessions A transaction ID is useful, but identifying the application session makes it actionable. SELECT st.session_id, at.transaction_id, at.transaction_begin_time, at.transaction_type, at.transaction_state, dt.database_transaction_begin_time, CAST ( dt.database_transaction_log_bytes_used / 1024.0 / 1024.0 AS decimal(18,2) ) AS log_mb_used, s.login_name, s.host_name, s.program_name, r.status, r.command, r.wait_type, r.blocking_session_id FROM sys.dm_tran_session_transactions AS st JOIN sys.dm_tran_active_transactions AS at ON st.transaction_id = at.transaction_id JOIN sys.dm_tran_database_transactions AS dt ON st.transaction_id = dt.transaction_id LEFT JOIN sys.dm_exec_sessions AS s ON st.session_id = s.session_id LEFT JOIN sys.dm_exec_requests AS r ON st.session_id = r.session_id WHERE dt.database_id = DB_ID() ORDER BY at.transaction_begin_time; sys.dm_tran_active_transactions and sys.dm_tran_database_transactions are documented for Azure SQL Database. Look for: Very old transactions. Transactions consuming large amounts of log. Sleeping sessions with open transactions. Blocking. Large write activity. Don't terminate transactions merely because they're old. First determine: Which application owns them. Whether rollback would cause more impact. Whether they're expected. Whether they're actually contributing to the observed log condition. 13. Correlate With Log-Write Pressure Azure SQL Database also exposes recent transaction-log write utilization through: sys.dm_db_resource_stats It provides roughly one hour of resource history with approximately 15-second samples. SELECT TOP (240) end_time, avg_cpu_percent, avg_data_io_percent, avg_log_write_percent, avg_memory_usage_percent FROM sys.dm_db_resource_stats ORDER BY end_time DESC; Now correlate: Scale-operation start time. Log-space growth. log_reuse_wait_desc. CDC progression. Long-running transactions. avg_log_write_percent. This can reveal two very different scenarios. Scenario A Log is large. Log-write rate is currently modest. CDC continues advancing. No unusually long transaction is present. This weakens the hypothesis that current transaction-log pressure is the primary explanation for the ongoing scale behavior. Scenario B Log usage continually rises. REPLICATION remains the reuse wait. CDC progression appears delayed. Large transactions remain active. Log-write pressure is high. This strengthens the case for investigating the log/CDC workload further. It still doesn't prove CDC is blocking scaling. 14. Test Correlation Without Inventing Causation Build a timeline like this: Time Scale state Log used Reuse wait CDC progress Long transaction T1 IN_PROGRESS 45% NOTHING Advancing No T2 IN_PROGRESS 62% REPLICATION Slower Yes T3 IN_PROGRESS 78% REPLICATION Limited Yes T4 IN_PROGRESS 66% NOTHING Advancing No Then ask: Evidence supporting a relationship Problem begins at the same time. Log pressure remains sustained. CDC progression is materially delayed. Long transactions contribute substantial log activity. Reducing unnecessary write pressure produces measurable improvement. Evidence weakening the relationship CDC progresses normally. Log space is actively reused. Scale remains IN_PROGRESS despite normal log behavior. Scale problems occur independently of CDC pressure. Log pressure clears but operation state doesn't change. This distinction protects the investigation from a common mistake: Correlation is an investigation path, not proof of root cause. 15. Don't Disable CDC as a First Step Disabling CDC is a substantial configuration change. It can affect: ETL pipelines. Integration platforms. Analytics workloads. Incremental processing. Downstream synchronization. CDC history. Azure SQL Database provides documented procedures for enabling and disabling CDC, but those procedures should be used because changing CDC is an intentional design decision, not simply because a scale operation is taking longer than expected. Even if disabling CDC changes the symptom, that fact alone doesn't prove: CDC was the root cause. Other conditions may have changed simultaneously. 16. Use a Safe Mitigation Order When a scale operation, CDC, and transaction-log pressure overlap, consider this sequence. Step 1: Stop introducing new variables Avoid repeatedly submitting duplicate or conflicting scale operations. Step 2: Preserve operation evidence Capture: Operation ID. Start time. State. Last modification time. Errors. Step 3: Measure log health Capture: Log size. Used log percentage. Truncation holdup reason. Trend over time. Step 4: Validate CDC Confirm: CDC enabled. Tracked tables. CDC progression. Recent CDC configuration changes. Step 5: Investigate long-running transactions Determine: Age. Session. Application owner. Log consumption. Business impact of intervention. Step 6: Reduce avoidable write pressure When operationally safe, reduce nonessential: Bulk updates. Large batches. ETL writes. Maintenance jobs. Excessive write concurrency. Step 7: Treat CDC changes as controlled interventions Only disable CDC when: Downstream impact is understood. The change is justified. The recovery/reinitialization path is understood. Evidence supports testing it. 17. Validate Recovery Don't stop when the scale operation finally completes. Confirm all layers. Scale operation Verify that the operation reached: COMPLETED FAILED CANCELLED rather than remaining indefinitely in an intermediate state. Transaction log Verify: Log usage stabilizes. Log reuse resumes. log_truncation_holdup_reason changes appropriately. CDC Verify: Change capture continues. Expected tracked tables remain enabled. Downstream consumers remain healthy. Application Verify: Connections recover. Workload resumes normally. No unexpected backlog remains. 18. Know When to Escalate Escalation becomes appropriate when you've established that: The scale or cancellation operation remains persistently nonterminal. No meaningful state changes are occurring. Database workload evidence has been captured. Log health is understood. CDC status is documented. Long transactions have been investigated. Business impact continues. Microsoft provides sys.dm_operation_status specifically for monitoring operations performed on Azure SQL databases, including pending, in-progress, failed, cancellation-in-progress, and canceled states. A useful escalation package includes: Sanitized operation history. Service objective before and after the requested scale. sys.dm_operation_status evidence. Log-space trend. log_truncation_holdup_reason. CDC state. CDC progress observations. Long-running transaction evidence. Resource metrics. Mitigations attempted and their results. Remaining application/business impact. 19. Practical Decision Tree When an Azure SQL scale operation appears stuck: Is the operation still nonterminal? No Validate workload, CDC, and log health. Continue monitoring. Yes Record its state and timeline. Is the transaction log under sustained pressure? No Don't make CDC the primary hypothesis solely because it's enabled. Yes Continue. What is preventing log reuse? ACTIVE_TRANSACTION Investigate long-running transactions. REPLICATION with CDC enabled Investigate CDC progress and transaction activity. Another value Investigate that truncation holdup independently. Is CDC demonstrating progress? Yes A completely stalled CDC process becomes a weaker hypothesis. Continue investigating log generation and the scale operation separately. No, or inconclusive Gather repeated samples. Check transactions and workload pressure. Did log pressure resolve while scaling remained stuck? Yes The evidence weakens the hypothesis that log/CDC pressure alone explains the operation. No Continue correlation and consider escalation. Final Takeaway When Azure SQL scaling, CDC, and transaction-log pressure happen together, don't begin with: "CDC blocked the scale operation." Begin with four independent questions: What is the actual state of the scale operation? Is the transaction log genuinely under sustained pressure? Why is log reuse being delayed? Is CDC demonstrating forward progress? Azure SQL Database provides documented tools for each layer: sys.dm_operation_status for operation state. sys.dm_db_log_space_usage for log-space utilization. sys.dm_db_log_stats() for transaction-log health and truncation holdup. sys.dm_tran_database_transactions for transaction/log consumption. sys.dm_db_resource_stats for recent CPU, I/O, and log-write pressure. Azure SQL Database's built-in CDC scheduler and CDC metadata for capture investigation. Most importantly: Don't disable CDC just because two symptoms occurred together. Preserve evidence, reduce unnecessary workload pressure, address demonstrated transaction problems, and escalate persistent nonterminal scaling operations with a clear timeline. Because: The safest diagnosis separates correlation from causation before changing the workload. Further reading Change Data Capture (CDC) with Azure SQL Database Scale single database resources in Azure SQL Database sys.dm_operation_status sys.dm_db_log_space_usage sys.dm_db_log_stats Transaction log growth with long-running transactions and CDCPart 2: Safely Cleaning Orphaned Records in Change Tracking Side Tables
Applies to: Azure SQL Database (Change Tracking enabled) Recap (Part 1) In Part 1, we covered how to detect “orphaned” records in Change Tracking (CT) side tables — rows whose sys_change_xdes_id no longer has a matching transaction entry in the commit table (sys.syscommittab). This situation often leads to unexpected CT growth and “stuck cleanup” symptoms because the mapping data required for normal cleanup is missing. Part 1 link: Identifying Orphaned Records in Change Tracking Side Tables (Read‑Only Health Check) Why Part 2 is needed A common “root pattern” we see in the field is: Side-table cleanup attempts to delete expired metadata Some side-table deletions fail (locks/timeouts/errors) Commit-table cleanup proceeds anyway (or a custom workflow deletes from commit table without validating side-table deletes) Remaining side-table rows now reference xdes_id values that no longer exist in sys.syscommittab → orphans Microsoft Learn also emphasizes that syscommittab cleanup depends on side-table cleanup — commit-table cleanup should happen only after side tables are cleaned. This Part 2 script focuses on removing orphaned rows from side tables (and does not touch sys.syscommittab), so cleanup logic can stabilize again. Important prerequisites & constraints (read this first) 1) Internal table access on Azure SQL Database In Azure SQL Database, customers may not be able to access certain internal CT artifacts directly (even when attempting DAC-style workflows). In the related case discussions, internal testing noted that self‑service cleanup against internal tables can be infeasible. 2) CHECKPOINT note (why it’s in the script) sys.dm_tran_commit_table exposes commit-table data and is backed by sys.syscommittab. Microsoft Learn notes that read-only users may not see live changes until a CHECKPOINT occurs. That’s why your script includes the optional CHECKPOINT comment before reading commit-table state. 3) Supported guidance vs. custom remediation Microsoft Learn provides official troubleshooting/mitigation guidance for CT cleanup issues (including checking dbo.MSChange_tracking_history, assessing stale rows, and using sp_flush_commit_table_on_demand for commit-table cleanup). Your script is a targeted remediation pattern for a specific failure mode (orphaned side-table rows). Use it carefully, test first, and follow organizational approval processes. What this script does (high-level) T-SQL script is essentially Part 1 detection + optional targeted delete generation: Compute the “safe cleanup point” from CT retention (wall-clock → CSN) using sp_changetracking_time_to_csn — the same concept used in Part 1. Enumerate CT side tables via sys.internal_tables where internal_type = 209 (CT side tables). For each side table, identify candidate orphaned transaction IDs (sys_change_xdes_id) that: are older than a computed boundary (@minXdesId derived from sys.dm_tran_commit_table), and have no matching xdes_id in sys.syscommittab at/before the cleanup point Print orphan counts per side table using RAISERROR … WITH NOWAIT (operator-friendly, streaming output). Safety cross-check: abort if any “orphan” unexpectedly exists in sys.syscommittab (defensive sanity gate) Generate a DELETE statement for the current side table (execution is commented out) The script (Part 2) — “detect + generate delete” Below is T-SQL script. I kept the delete step disabled by default, so it remains safe to share. (You can enable execution only after approvals/testing.) -- use <[DBName]> -- switch to the right database -- run checkpoint first to ensure all in-memory commit table data is persisted to disk (syscommittab) -- checkpoint SET NOCOUNT ON -- find the invalid clean version based on configured retention DECLARE time DATETIME, @csn BIGINT = 0, @minCleanupPoint BIGINT = 0 DECLARE @retention_period INT, @retention_period_units NVARCHAR(10) SELECT @retention_period = retention_period, @retention_period_units = retention_period_units FROM sys.change_tracking_databases where database_id = DB_ID() SELECT time = CASE WHEN @retention_period_units = 1 then DATEADD(minute, (-1 * @retention_period), GETUTCDATE()) WHEN @retention_period_units = 2 then DATEADD(hour, (-1 * @retention_period), GETUTCDATE()) ELSE DATEADD(day, (-1 * @retention_period), GETUTCDATE()) END EXEC sp_changetracking_time_to_csn time = time, @csn = @csn OUTPUT SELECT @minCleanupPoint = @csn SELECT @minCleanupPoint as minCsn -- 688118 -- iterate over all the change tracking side tables DECLARE @sideTable SYSNAME; DECLARE ct_cursor CURSOR FAST_FORWARD FOR SELECT name FROM sys.internal_tables WHERE internal_type = 209; -- internal_type = 209 is for change tracking side tables OPEN ct_cursor; FETCH NEXT FROM ct_cursor INTO @sideTable; WHILE @@FETCH_STATUS = 0 BEGIN -- find the minimum expired xdes id declare @minXdesId BIGINT SELECT @minXdesId = min(xdes_id) FROM sys.dm_tran_commit_table where commit_ts <= @minCleanupPoint -- SELECT @minXdesId as minXdes -- create temp table for storing orphaned xdes id DROP TABLE IF EXISTS #OrphanedXdes; CREATE TABLE #OrphanedXdes ( sys_change_xdes_id BIGINT NOT NULL ); DECLARE @sql NVARCHAR(MAX); SET @sql = N' INSERT INTO #OrphanedXdes(sys_change_xdes_id) SELECT ct.sys_change_xdes_id FROM sys.' + QUOTENAME(@sideTable) + N' AS ct WHERE ct.sys_change_xdes_id < @minXdesId AND NOT EXISTS ( SELECT 1 FROM sys.syscommittab AS s WHERE s.xdes_id = ct.sys_change_xdes_id AND s.commit_ts <= @minCleanupPoint );'; EXEC sys.sp_executesql @sql, N'@minXdesId BIGINT, @minCleanupPoint BIGINT', @minXdesId = @minXdesId, @minCleanupPoint = @minCleanupPoint; DECLARE @orphanedIdsCount BIGINT; SET @sql = N' SELECT @cnt = COUNT_BIG(sys_change_xdes_id) FROM #OrphanedXdes; '; EXEC sys.sp_executesql @sql, N'@cnt BIGINT OUTPUT', @cnt = @orphanedIdsCount OUTPUT; -- Raise error if any orphaned xdes exists IF (@orphanedIdsCount > 0) BEGIN DECLARE @msg NVARCHAR(4000) = @sideTable + N' : ' + CONVERT(NVARCHAR(30), @orphanedIdsCount); RAISERROR (@msg, 16, 1) WITH NOWAIT; DECLARE @newLine NVARCHAR(10) = CHAR(13) + CHAR(10) PRINT (@newLine) END -- Cross-check that no xdes should exist in syscommittab -- !!!IMPORTANT!!! RAISE an error and stop the cleanup if it does SET @sql = N' DECLARE @nonMatchingXdesCount BIGINT; SELECT @nonMatchingXdesCount = COUNT_BIG(*) FROM #OrphanedXdes AS ct WHERE EXISTS ( SELECT 1 FROM sys.syscommittab AS s WHERE s.xdes_id = ct.sys_change_xdes_id ); -- SELECT @nonMatchingXdesCount as nonMatchingXdesCount IF (COALESCE(@nonMatchingXdesCount, 0) > 0) BEGIN TRY DECLARE @msg NVARCHAR(1024); SET @msg = N''Cleanup aborted: orphan cross-check failed for side table [' + @sideTable + N'].''; RAISERROR(@msg, 16, 1) WITH NOWAIT; RETURN; END TRY BEGIN CATCH THROW; END CATCH '; EXEC sys.sp_executesql @sql; IF (@orphanedIdsCount > 0) BEGIN -- Prepare the query to delete the orphaned rows from the side table SET @sql = N'DELETE ct FROM sys.' + @sideTable + N' ct WHERE EXISTS (SELECT 1 FROM #OrphanedXdes AS o WHERE o.sys_change_xdes_id = ct.sys_change_xdes_id);'; SELECT @sql -- validate the delete query is correctly generated -- Sample delete statement: DELETE ct FROM sys.change_tracking_1221579390 ct WHERE EXISTS (SELECT 1 FROM #OrphanedXdes AS o WHERE o.sys_change_xdes_id = ct.sys_change_xdes_id); -- NOTE: Uncomment the below query to execute the delete statement and remove the orphaned records -- EXEC sys.sp_executesql @sql; END DROP TABLE IF EXISTS #OrphanedXdes; FETCH NEXT FROM ct_cursor INTO @sideTable; END CLOSE ct_cursor; DEALLOCATE ct_cursor; SET NOCOUNT OFF Why the “cross-check abort” is a good idea Notice the safety gate: You first define orphans as those not existing in sys.syscommittab (for the cleanup horizon). Then you re-check: “If any of these appear in syscommittab, abort.” This prevents an accidental delete if: the cleanup horizon math is wrong, the environment has unexpected visibility differences, or the temp-table contents are not what you expect. That defensive posture aligns well with the general principle documented in Microsoft Learn: commit-table cleanup should only occur after side-table cleanup, and troubleshooting should be data-driven and cautious. How to interpret the output If you see no RAISERROR lines, the script did not find orphaned rows under the defined criteria. If you see: change_tracking_<id> : <count> that indicates <count> orphaned transaction references in that CT side table. This is the same style used in Part 1 for long-running, streaming progress. Next steps (recommended order) Confirm CT configuration (retention + AUTO_CLEANUP status) using official guidance. Run Part 1 / Part 2 detection to quantify scope (which side tables, how many). If you need to remediate: Prefer supported mitigations where possible (for example, disable/enable CT on a table to purge tracking metadata for that table is listed as a “quickest remedy” in Microsoft Learn for certain cleanup lock conflict scenarios). If table-level disable/enable isn’t acceptable, use an approval-driven approach for targeted cleanup. Closing Orphaned CT side-table records are one of those “silent growth” conditions that can be easy to miss until storage or CHANGETABLE performance becomes painful. Part 1 helps you spot the issue early; Part 2 helps you prepare a safe, targeted cleanup workflow — with explicit safety gates and a delete step that remains disabled by default. References Identifying Orphaned Records in Change Tracking Side Tables (Read‑Only Health Check) Troubleshoot change tracking auto cleanup issues sys.dm_tran_commit_table (Change Tracking)Cannot enable Change Data Capture (CDC) on Azure SQL Database: Msg 22830 + Error 40529 (SUSER_SNAME)
Issue While enabling CDC at the database level on Azure SQL Database using: EXEC sys.sp_cdc_enable_db; GO the operation fails. Error The customer observed the following failure when running sys.sp_cdc_enable_db: Msg 22830: Could not update the metadata that indicates the database is enabled for Change Data Capture. Failure occurred when executing drop user cdc Error 40529: "Built-in function 'SUSER_SNAME' in impersonation context is not supported in this version of SQL Server." What we checked (quick validation) Before applying any changes, we confirmed CDC wasn’t partially enabled and no CDC artifacts were created: -- Is CDC enabled for this database? SELECT name, is_cdc_enabled FROM sys.databases WHERE name = DB_NAME(); -- Does CDC schema exist? SELECT name FROM sys.schemas WHERE name = 'cdc'; -- Does CDC user exist? SELECT name FROM sys.database_principals WHERE name = 'cdc'; These checks were used during troubleshooting in the SR. (Also note: Microsoft Learn documents that enabling CDC creates the cdc schema/user and requires exclusive use of that schema/user.) Cause In this case, the failure aligned with a known Azure SQL Database CDC scenario: enabling CDC can fail if there is an active database-level trigger that calls SUSER_SNAME(). To identify active database-level triggers, we used: SELECT name, object_id FROM sys.triggers WHERE parent_class_desc = 'DATABASE' AND is_disabled = 0; Resolution / Workaround The customer resolved the issue by: Identifying the active database-level trigger. Disabling the trigger temporarily. Enabling CDC at the database level and then enabling CDC on the required tables. Re-enabling the trigger after CDC was successfully enabled. Post-resolution verification After enabling CDC, you can validate the state using: -- Confirm CDC enabled at DB level SELECT name, is_cdc_enabled FROM sys.databases WHERE name = DB_NAME(); And for table-level tracking, Microsoft Learn recommends checking the is_tracked_by_cdc column in sys.tables. Notes / Requirements To enable CDC for Azure SQL Database, db_owner is required. Azure SQL Database uses a CDC scheduler (instead of SQL Server Agent jobs) for capture/cleanup.Performance Tuning for CDC: Managing Replication Lag in Azure Database for PostgreSQL with Debezium
Written By: Shashikant Shakya, Ashutosh Bapat, and Guangnan Shi The Problem Picture this: your CDC pipeline is running smoothly, streaming changes from PostgreSQL to Kafka. Then, a bulk update hits millions of rows. Suddenly, Kafka queues pile up, downstream systems lag; dashboards go stale. Why does replication lag spike during heavy operations? And what can you do about it? Why This Matters Change Data Capture (CDC) powers real-time integrations, pushing row-level changes from OLTP systems into event streams, data lakes, caches, and microservices. Debezium is a leading open-source CDC engine for PostgreSQL, and many teams successfully run Debezium against Azure Database for PostgreSQL to keep downstream systems synchronized. However, during large DML operations (bulk updates, deletes) or schema changes (DDL), replication lag can occur because: Debezium consumes WAL slower than the database produces it Kafka throughput dips Consumers fall behind This article explains why lag happens, grounded in logical decoding internals, and shows how to diagnose it quickly and what to tune across the database, Azure, and connector layers to keep pipelines healthy under heavy load. CDC Basics CDC streams incremental changes (INSERT/UPDATE/DELETE) from your source database to downstream systems in near real-time. In PostgreSQL, CDC is typically implemented using logical decoding and logical replication: PostgreSQL records every change in the Write-Ahead Log (WAL) WALSender reads WAL and decodes it into change events The pgoutput extension formats those changes, while Debezium subscribes and publishes them to Kafka topics Benefits of CDC: Low latency Lower source overhead than periodic full extracts Preserves transactional ordering for consumers The Internals: Why Lag Happens Replication lag during heavy operations isn’t random, it’s rooted in how PostgreSQL handles logical decoding. To understand why, let’s look at the components that process changes and what happens when they hit resource limits. Logical Decoding & ReorderBuffer Logical decoding reconstructs transaction-level changes so they can be delivered in commit order. The core component enabling this is the ReorderBuffer. What ReorderBuffer does: Reads WAL and groups changes per transaction, keeping them in memory until commit If memory exceeds logical_decoding_work_mem , PostgreSQL spills decoded changes to disk in per-slot spill files On commit, it reads back spilled data and emits changes to the client (via pgoutput → Debezium) Disk Spill Mechanics (Deep Dive) When a transaction is too large for memory: PostgreSQL writes decoded changes to spill files under pg_replslot/<slot_name>/ Wait events like ReorderBufferWrite and ReorderBufferRead dominate during heavy load Spills to disk increase latency because disk I/O is far slower than memory access Analogy: Think of ReorderBuffer as a warehouse staging floor: Small shipments move quickly in memory A huge shipment forces workers to move boxes offsite (spill-to-disk), then bring them back later, slowing everything down Why Multiple Slots Amplify the Impact The WAL is shared by all slots Each slot decodes the entire WAL stream because filtering happens after decoding Result: A single large transaction affects every slot, multiplying replication lag Recommendation: Minimize the number of slots/connectors Remember: logical_decoding_work_mem applies per slot, not globally Impact Snapshot: Scenario Spill Size I/O Impact 1 Slot 1 GB 1× I/O 5 Slots 1 GB × 5 5× I/O Lifecycle: WAL → ReorderBuffer → Memory → Spill to Disk → Read Back → Send to Client How to Detect Spills and Lag Detection should be quick and repeatable. Start by confirming slot activity and LSN distance (how far producers are ahead of consumers), then check walsender wait events to see if decoding is stalling, and finally inspect per-slot spill metrics to quantify memory overflow to disk. 1. Active slots and lag Use this to measure how far each logical slot is behind the current WAL. A large lsn_distance indicates backlog. If restart_lsn is far behind, the server must retain more WAL on disk, increasing storage pressure. SELECT slot_name, active_pid, confirmed_flush_lsn, restart_lsn, pg_current_wal_lsn(), pg_size_pretty((pg_current_wal_lsn() - confirmed_flush_lsn)) AS lsn_distance FROM pg_replication_slots; Interpretation: Focus on slots with the largest lsn_distance . If active_pid is NULL, the slot isn’t currently consuming; investigate connector health or connectivity. 2. Wait events for walsender Check whether the WAL sender backends are stalled on decoding or I/O. ReorderBuffer-related waits typically point to spill-to-disk conditions or slow downstream consumption. SELECT pid, backend_type, application_name, wait_event FROM pg_stat_activity WHERE backend_type = 'walsender' ORDER BY backend_start; Interpretation: Frequent ReorderBufferWrite / ReorderBufferRead suggests large transactions are spilling. 3. Spill stats Quantify how often and how much each slot spills from memory to disk. Rising spill_bytes and spill_count during heavy DML are strong signals to increase logical_decoding_work_mem , reduce transaction size, or tune connector throughput. SELECT slot_name, spill_txns, spill_count, pg_size_pretty(spill_bytes) AS spill_bytes, total_txns, pg_size_pretty(total_bytes) AS total_bytes, stats_reset FROM pg_stat_replication_slots; Interpretation: Compare spill_bytes across slots; if many slots spill simultaneously, aggregate I/O multiplies. Consider reducing the number of active slots or batching large DML. Fixing the Lag: Practical Strategies Once you’ve identified replication lag and its root causes, the next step is mitigation. Solutions span across the database configuration, Azure infrastructure, and the Debezium connector layer. These strategies aim to reduce I/O overhead, optimize memory usage, and ensure smooth data flow under heavy workloads. Database & Azure Layer At the database and infrastructure level, focus on reducing unnecessary overhead and ensuring resources are scaled for peak demand. Here’s what you can do: Avoid REPLICA IDENTITY FULL : prefer PRIMARY KEY; or add a unique index and set REPLICA IDENTITY USING INDEX Use appropriately scaled IO-capable storage / right SKU for higher IOPS Right-size logical_decoding_work_mem considering multiple slots Break up large DML: batch updates/deletes (10k–50k rows/commit) Schedule/throttle maintenance: stagger VACUUM/REINDEX/DDL Network placement: use Private Endpoint and co-locate Debezium/Kafka within the same region/VNet Debezium Connector Layer Connector-level tuning ensures that Debezium can keep pace with PostgreSQL WAL generation and Kafka throughput. Key adjustments include: Tune throughput & buffering: increase max.batch.size , max.queue.size , reduce poll.interval.ms Offset flush tuning: reduce offset.flush.interval.ms Heartbeats: introduce heartbeat events to detect staleness and prevent WAL buildup Conclusion Managing replication lag in Azure Database for PostgreSQL with Debezium isn’t just about tweaking parameters; it’s about understanding logical decoding internals, anticipating workload patterns, and applying proactive strategies across the entire solution. Key Takeaways: Monitor early, act fast: Use diagnostic queries to track lag, wait events, and spill activity Minimize complexity: Fewer replication slots and well-tuned connectors reduce redundant work Plan for scale: Batch large DML operations, right-size memory settings Leverage Azure capabilities: Optimize IOPS tiers, network placement for predictable performance By combining these best practices with continuous monitoring and operational discipline, you can keep your CDC pipelines healthy, even under heavy load, while ensuring downstream systems stay in sync with minimal latency. Further Reading Azure Database for PostgreSQL Flexible Server Overview PostgreSQL Logical Replication Debezium PostgreSQL Connector
Impact of CDC on SQL Server
Hi, We are testing performance impact of CDC on SQL Server. There are two identical databases (KST_S001, KST_002) on SQL Server 2017 which is running in linux container. They both have CDC enabled for 180 tables and a data generator that is doing mostly updates on these tables. The data generators are doing around 300k DML operations per minute on each database (600k total). The configuration of CDC jobs is kept default (500 transactions, 5s polling interval for capture job, I think three days retention period on cleaning job) The host machine is Azure Standard_E8-4as_v4: 4 vCPU, 64 GB RAM, 12800 IOPS, 128GB SSD. After 1 hour of running the setup with CDC enabled, the average time of 1000 updates is 73ms. With CDC disabled the average time of 1000 updates after 1 hour is 15ms. I've attached two grafana screenshots showing the difference in metrics when the CDC is disabled versus enabled on these databases. We are trying to understand the underlying mechanism that is contributing to the impact visible from the metrics. Why is memory consumption so much higher? Why do DML operations take longer with CDC enabled? CDC disabled: CDC enabled: Thanks for answers and suggestions. PeterLesson Learned #352: Could not update the metadata that indicates database enabling CDC.
Today, we got a error message while trying to enable cdc for a database using the sqladmin user. Our customer got the error message: Msg 22830, Level 16, State 1, Procedure sys.sp_cdc_enable_db_internal, Line 283 [Batch Start Line 0] Could not update the metadata that indicates database XYZ is enabled for Change Data Capture. The failure occurred when executing the command 'SetCDCTracked(Value = 1)'. The error returned was 33171: 'Only active directory users can impersonate other active directory users.'. Use the action and error to determine the cause of the failure and resubmit the request.Capture Changed Data From your Cosmos DB analytical store (Preview)
Making it super-easy to create efficient and fast ETL processing the cloud, Azure Data Factory has invested heavily in change data capture features. Today, we are super-excited to announce that Azure Cosmos DB analytics store now supports Change Data Capture (CDC), for Azure Cosmos DB API for NoSQL, and Azure Cosmos DB API for Mongo DB in public preview!
