Troubleshooting Azure SQL Data Sync Failure: SQL Error 8106 During Bulk Insert
Azure SQL Data Sync is widely used to maintain consistency across distributed databases in hub–member topologies. However, synchronization may occasionally fail due to schema mismatches between participating databases — even when everything appears correctly configured at first glance. In this post, we’ll walk through a real-world troubleshooting scenario involving a Data Sync failure caused by a schema inconsistency related to an IDENTITY column, and how it was mitigated. Sample Error: sync_7726d6cb22124c0f901192c434f49106bd618f8ab16343b2adc03250f8367ff4\3953fb7d-1dba-4656-8150-83153d5d019b.batch. See the inner exception for more details. Inner exception: Failed to execute the command 'BulkInsertCommand' for table 'schema.table_name'; the transaction was rolled back. Ensure that the command syntax is correct. Inner exception: SqlException ID: e19b3677-d67e-4c8e-bc49-13d3df61ad0e, Error Code: -2146232060 - SqlError Number:8106, Message: SQL error with code 8106 For more information, provide tracing ID ‘92e76130-f80a-4372-9a48-ec0ede8b0288’ to customer support." Scenario Overview A synchronization operation began failing for a specific table within an Azure SQL Data Sync group. The failure was observed during the sync process when applying changes using a batch file. The error surfaced as part of a failed BulkInsertCommand execution on a synced table, causing the transaction to roll back. Further investigation revealed the following SQL exception: SqlError Number: 8106 Table does not have the identity property. Cannot perform SET operation. Initial Troubleshooting Steps Before identifying the root cause, the following actions were taken: The affected table was removed from the sync group. A sync operation was triggered. The table was re-added to the sync group. Sync was triggered again. Despite performing these steps, the issue persisted with the same error. This indicated that the failure was not related to sync metadata or temporary configuration inconsistencies. Root Cause Analysis After reviewing the table definitions across the sync topology, it was discovered that: The synchronized table had an IDENTITY column defined on one side of the topology (Hub or Member) but not on the other. This schema mismatch led to the sync service attempting to apply SET IDENTITY_INSERT operations during the bulk insert phase — which failed on the database where the column lacked the identity property. Azure SQL Data Sync relies on consistent schema definitions across all participating databases. Any deviation — particularly involving identity columns — can interrupt data movement operations. Mitigation Approach To resolve the issue, the following corrective steps were applied: Remove the affected table from the sync group and save the configuration. Refresh the sync schema. Recreate the table to include the appropriate IDENTITY property. Add the corrected table back to the sync group. Trigger a new sync operation. These steps ensured that the table definitions were aligned across all sync participants, allowing the synchronization process to proceed successfully. Best Practices to Avoid Similar Issues To prevent identity-related sync failures in Azure SQL Data Sync: ✅ Ensure table schemas are identical across all participating databases before onboarding them into a sync group. ✅ Pay special attention to: IDENTITY properties Primary keys Data types Nullable constraints ✅ Always validate schema consistency when: Adding new tables to a sync group Modifying existing table definitions Final Thoughts Schema mismatches — especially those involving identity columns — are a common but often overlooked cause of Data Sync failures. By ensuring consistent table definitions across your hub and member databases, you can significantly reduce the risk of synchronization errors and maintain reliable data movement across regions.Understanding and Monitoring Class 2 Transactions in Azure SQL Database
During a recent customer engagement, we investigated sustained transaction log growth in Azure SQL Database without obvious large user transactions. The customer was familiar with PostgreSQL diagnostics and wanted to understand how similar insights can be obtained in Azure SQL Database—especially around Class 2 (system) transactions. This post summarizes what we discussed, explains why Azure SQL behaves differently, and walks through practical DMV‑based monitoring patterns you can use today. Azure SQL Database vs. PostgreSQL: Diagnostic Model Differences One of the first clarifications we made is that Azure SQL Database does not expose diagnostic settings equivalent to PostgreSQL’s system‑level log diagnostics. Azure SQL Database is a fully managed PaaS service, and many internal operations—such as checkpoints, version store cleanup, and background maintenance—are abstracted from direct control. Instead of low‑level engine logs, Azure SQL provides cumulative Dynamic Management Views (DMVs) that expose the effects of system activity rather than the internal implementation. What Are Class 2 Transactions? In Azure SQL Database, Class 2 transactions generally refer to system‑generated transactions, not directly initiated by user workloads. These commonly include: Checkpoint operations Version store cleanup Ghost record cleanup Background metadata maintenance Although they are not user‑driven, these transactions still generate transaction log activity, which can be surprising when log usage grows steadily without large user transactions. Key DMVs to Monitor Class 2 Activity 1. Transaction Log Usage SELECT * FROM sys.dm_db_log_space_usage; This DMV provides: Total log size Used log space Used log percentage If log usage grows steadily without large user transactions, it is often a signal that background system activity (Class 2 transactions) is responsible. Checkpoint Activity SELECT * FROM sys.dm_exec_requests WHERE command = 'CHECKPOINT'; Frequent checkpoints result in: More frequent log flushes Increased system log writes In Azure SQL Database, checkpoint frequency is system‑managed and cannot be tuned through configuration or diagnostic settings. Version Store Usage (Common Class 2 Contributor) SELECT * FROM sys.dm_tran_version_store_space_usage; High version store usage often leads to: Background cleanup tasks Increased system transactions Additional transaction log generation This is especially common in workloads using: Snapshot Isolation Read Committed Snapshot Isolation (RCSI) Long‑running transactions or readers Automating Monitoring with Azure Elastic Jobs Because these DMVs are cumulative, capturing them over time is key. During the call, we discussed automating data collection using Azure Elastic Jobs. Elastic Jobs allow you to: Schedule DMV snapshots Store historical trends Correlate spikes with workload patterns Microsoft provides full guidance on creating and managing Elastic Jobs using T‑SQL here: Create and manage Elastic Jobs using T‑SQL Index Management and Class 2 Impact Index maintenance can indirectly increase Class 2 activity by: Increasing version store usage Triggering additional background cleanup Instead of manual index tuning, we recommended enabling Query Performance Insight – Index recommendations in the Azure Portal. This allows Azure SQL Database to automatically: Suggest index creation Suggest index removal based on real workload patterns. Why Checkpoints Cannot Be Tuned A common question is whether checkpoint frequency can be reduced to lower system log activity. In Azure SQL Database: Checkpoints are engine‑managed There is no diagnostic or configuration setting to control their frequency This design ensures platform stability and predictable recovery behavior As a result, monitoring—not tuning—is the correct approach. Practical Takeaways From this case, the key lessons are: Not all transaction log growth is user‑driven Class 2 transactions are a normal part of Azure SQL Database DMVs provide the best visibility into system behavior Trend‑based monitoring is more valuable than point‑in‑time checks Automation via Elastic Jobs is essential for long‑term analysis Conclusion Class 2 transactions are often misunderstood because they operate quietly in the background. By using the right DMVs and collecting data over time, you can clearly distinguish expected system behavior from genuine workload issues. If you’re coming from PostgreSQL or on‑prem SQL Server, the key mindset shift is this: Azure SQL Database exposes outcomes, not internals—and that’s by design.Understanding Azure SQL Data Sync Firewall Requirements
Why IP Whitelisting Is Required and What Customers Should Know Azure SQL Data Sync is commonly used to synchronize data between on‑premises SQL Server databases and Azure SQL Database. While the setup experience is generally straightforward, customers sometimes encounter connectivity or configuration issues that are rooted in network security and firewall behavior. This blog explains why Azure SQL Data Sync requires firewall exceptions, what type of IP addresses may appear in audit logs, and how to approach this topic from a security and documentation standpoint—based on real troubleshooting discussions within the Azure SQL Data Sync ecosystem. The Scenario: Sync Agent Configuration Fails Despite Valid Setup A frequently reported issue occurs when the Azure SQL Data Sync Agent (installed on an on‑premises server) fails to save its configuration. The error typically indicates that a valid agent key is required—even when: The agent key was freshly generated from the Azure SQL Data Sync portal Connection tests succeed The agent has been reinstalled or the server restarted New sync groups were created Despite these efforts, synchronization does not proceed until a specific public IP address is allowed through the Azure SQL Database firewall. Why Firewall Rules Matter for Azure SQL Data Sync Azure SQL Database is protected by a server‑level firewall that blocks all inbound traffic by default. Any external client—including the Data Sync Agent—must be explicitly allowed to connect. In Azure SQL Data Sync: The Data Sync Agent runs on‑premises It connects outbound over TCP port 1433 It uses the public endpoint of the Azure SQL logical server The Azure SQL firewall must allow the public IP address used by the agent If this IP is not allowed, the agent cannot complete configuration or perform synchronization operations—even if authentication and permissions are otherwise correct. Identifying the Required IP Address In the referenced discussion, the required IP address was identified by reviewing Azure SQL audit logs, which revealed connection attempts being blocked at the firewall layer. Once this IP address was added to the Azure SQL server firewall rules, synchronization completed successfully. This highlights an important point: Audit logs can be a reliable way to identify which IP address must be whitelisted when Data Sync connectivity fails. Is This IP Address Owned by Microsoft? Can It Change? A natural follow‑up question is whether the observed IP address is Microsoft‑owned, and whether it can change. From the discussion: Azure SQL Data Sync relies on Microsoft‑managed service infrastructure Some outbound connectivity may originate from Azure service IP ranges Microsoft publishes official IP ranges and service tags for transparency However, documentation does not guarantee that a single static IP will always be used. Customers should therefore treat firewall configuration as a network security requirement, not a one‑time exception. Related Microsoft Resources While Azure SQL Data Sync documentation focuses on setup and troubleshooting, firewall requirements are often implicit rather than explicitly called out. The following Microsoft resources were referenced in the discussion to help customers understand Azure service IP ownership and ranges: Gateway IP addresses – Azure Synapse Analytics Download Azure IP Ranges and Service Tags – Public Cloud These resources can help security teams validate Microsoft‑owned IPs and plan firewall policies accordingly. Key Takeaways for Customers ✅ Azure SQL Data Sync requires firewall access to Azure SQL Database ✅ The public IP used by the Data Sync Agent must be explicitly allowed ✅ Audit logs are useful for identifying blocked IPs ✅ IP addresses may belong to Microsoft infrastructure and can change over time ✅ Firewall configuration is a security prerequisite, not an optional step Closing Thoughts Azure SQL Data Sync operates securely by design, leveraging Azure SQL Database firewall protections. While this can introduce configuration challenges, understanding the network flow and firewall requirements can significantly reduce setup friction and troubleshooting time. If you're implementing Azure SQL Data Sync in a locked‑down network environment, we recommend involving your network and security teams early and validating firewall rules as part of the initial deployment checklist.Troubleshooting Azure SQL Data Sync Groups Stuck in Progressing State
Azure SQL Data Sync is commonly used to synchronize data across Azure SQL Databases and on‑premises SQL Server environments. While the service works well in many scenarios, customers may occasionally encounter a situation where a Sync Group remains stuck in a “Progressing” state and cannot be started, stopped, or refreshed. This blog walks through a real-world troubleshooting scenario, highlights the root cause, and outlines practical remediation steps based on actual support investigation and collaboration. Problem Overview In this scenario, the customer reported that: The Sync Group was stuck in “Progressing” for multiple days Sync operations could not be started or stopped Tables could not be refreshed or reconfigured Azure Activity Logs showed operations as Succeeded, yet sync never progressed Our backend telemetry showed the Sync Group as Active, while hub and member databases were in Reprovisioning state The last successful sync occurred on XX day, after which the sync pipeline stopped making progress. Initial Investigation Findings During the investigation, several key observations were made: 1. High DATA IO Utilization Telemetry and backend checks revealed that DATA IO utilization was pegged at 100% on one of the sync member databases starting XX day. Despite no noticeable change in application workload, the database was under sustained IO pressure, which directly impacted Data Sync operations. 2. Deadlocks During Sync Processing Our backend telemetry showed repeated deadlock errors: Transaction was deadlocked on lock resources with another process and has been chosen as the deadlock victim. These deadlocks were observed for multiple Sync Member IDs starting the same day IO saturation began. This aligned with the hypothesis that resource contention, not a Data Sync service failure, was the underlying issue. 3. Metadata Database Was Healthy The Sync metadata database was running on a serverless Azure SQL Database (1 vCore) and showed healthy resource usage, ruling it out as a bottleneck. Recommended Troubleshooting Steps Based on the findings, the following steps were recommended and validated: ✅ Step 1: Address Database Resource Constraints First Before attempting to recreate or reset the Sync Group, the focus was placed on resolving DATA IO saturation on the affected database. Actions included: Scaling up the database (DTUs / vCores) Monitoring IO utilization after scaling Ensuring sufficient headroom for sync operations This was identified as the primary remediation step. ✅ Step 2: Use the Azure SQL Data Sync Health Checker The Azure SQL Data Sync Health Checker was recommended to validate: Sync metadata integrity Table-level configuration issues Agent and connectivity status GitHub tool: AzureSQLDataSyncHealthChecker ✅ Step 3: Validate Sync Group and Agent State via PowerShell PowerShell was used to confirm: Sync Group state Last successful sync time On‑premises Sync Agent connectivity Example commands used: Get-AzureRmSqlSyncGroup ` -ResourceGroupName "ResourceGroup01" ` -ServerName "Server01" ` -DatabaseName "Database01" | Format-List Get-AzureRmSqlSyncAgent ` -ResourceGroupName "ResourceGroup01" ` -ServerName "Server01" | Select ResourceGroupName, SyncState, LastSyncTime Resolution After the customer increased the database size, DATA IO utilization dropped, sync operations resumed normally, and the customer confirmed that the issue was resolved.Why Long-Term Retention (LTR) Backups Don’t Attach After a PITR Restore in Azure SQL Database
Summary Customers sometimes expect that after performing a Point‑in‑Time Restore (PITR) and renaming the restored database back to its original name, existing Long‑Term Retention (LTR) backups will automatically appear and continue from where they left off. This behavior may have worked in older or legacy environments, but in modern Azure SQL Database deployments—especially across new servers or subscriptions—this expectation can lead to confusion. This article explains why LTR backups do not attach to restored databases, even if the database name is reused, and what customers should expect instead. The Scenario The discussion originated from a common migration pattern: A customer has an Azure SQL Database with LTR policies configured (for example, monthly backups retained for 10 years). The customer performs a Point‑in‑Time Restore (PITR) of that database. After the restore, the database is renamed to match the original database name. The customer expects the existing LTR backups to appear under the restored database. In legacy environments, this behavior appeared to work. However, in newer Azure SQL Database deployments, the LTR backups are not visible after the restore and rename process. Key Technical Detail: LTR Is Not Based on Database Name The most important concept to understand is this: LTR backups are associated with the database’s logical database ID—not the database name. Each Azure SQL Database is assigned a unique logical database ID at creation time. When a PITR restore is performed: A new database is created It receives a new logical database ID Even if you rename the database to match the original name, the logical ID remains different As a result, the restored database is treated as a completely new database from an LTR perspective, and it does not inherit the historical LTR backup chain. Why Renaming the Database Does Not Help Renaming a database only changes its display name. It does not change: The logical database ID The internal association used by the LTR system Because LTR configuration and backup visibility are tied to the logical database ID, renaming alone cannot reattach historical LTR backups. Subscription Boundaries Matter Another important clarification raised in the discussion: LTR backups are scoped to the subscription where the database was created While you can restore LTR backups to a different server within the same subscription, you cannot carry historical LTR backups across subscriptions If a customer migrates to a new subscription, the historical LTR chain from the old subscription cannot be reused or reattached. Only new LTR backups created after the move will exist in the new subscription. What Customers Will Observe After a PITR restore and rename: ✅ The database is successfully restored ✅ LTR policies can be configured again ❌ Historical LTR backups from the original database are not visible ❌ The restored database does not inherit old LTR backups, even if the name matches This is expected behavior and aligns with the current Azure SQL Database architecture. How to Validate LTR Backups Correctly To avoid confusion caused by portal caching or UI expectations, customers can list LTR backups programmatically using PowerShell or Azure CLI, as documented in Microsoft Learn: Azure SQL Database: Manage long-term backup retention Azure SQL Database: Manage long-term backup retention - Azure SQL Database | Microsoft Learn This confirms whether LTR backups exist for a specific logical database ID. Best Practices and Recommendations Do not rely on database renaming to preserve LTR history. Treat any PITR restore as a new database from an LTR perspective. If historical LTR backups must remain accessible: Keep the original database intact Restore LTR backups directly from the original database when needed Plan migrations carefully, especially when moving across subscriptions, as LTR history cannot be migrated. Final Thoughts LTR backups are a powerful compliance and recovery feature in Azure SQL Database, but they are intentionally designed to be immutable and identity‑based, not name‑based. Understanding that logical database ID—not database name—controls LTR association helps set correct expectations and avoids surprises during restores or migrations. Frequently Asked Questions (FAQ) Q1: Why don’t my existing LTR backups appear after I restore a database using PITR? Because a Point‑in‑Time Restore (PITR) creates a new database with a new logical database ID. Long‑Term Retention (LTR) backups are associated with the database’s logical ID—not its name—so the restored database does not inherit the historical LTR backup chain. Q2: If I rename the restored database to the original name, shouldn’t the LTR backups reappear? No. Renaming a database only changes its display name. It does not change the logical database ID, which is what LTR uses to associate backups. As a result, renaming does not reattach existing LTR backups. Q3: This used to work in our legacy environment—why is it different now? In older environments, the behavior may have appeared to work due to differences in platform implementation. In current Azure SQL Database architecture, LTR association is strictly identity‑based, which ensures immutability, compliance, and predictable backup behavior. Q4: Can I attach historical LTR backups to a restored database manually? No. LTR backups are immutable and cannot be reattached or reassigned to a different logical database ID. This behavior is by design. Q5: What happens if I move my database to a new subscription? LTR backups are scoped to the subscription where the database was created. If you migrate to a new subscription: Historical LTR backups from the old subscription cannot be carried over Only new LTR backups created after the move will exist in the new subscription Q6: Can I still restore from my old LTR backups? Yes. As long as the original database (or its logical identity) still exists in the original subscription, you can restore directly from those LTR backups—even if a newer database with the same name exists elsewhere. Q7: How can I verify which LTR backups actually exist? The most reliable way is to list LTR backups programmatically using Azure PowerShell or Azure CLI, which queries backups by logical database ID rather than relying solely on portal views. Refer to the official documentation: Azure SQL Database – Manage long‑term backup retention Q8: What is the recommended approach if we need long‑term recoverability after PITR? Treat every PITR restore as a new database from an LTR perspective Keep the original database intact if historical LTR backups must remain accessible Plan subscription migrations carefully, as LTR history cannot be migratedLesson Learned #345: How to get the most from Azure SQL Database - Session Delivered
This last April 27 we had a great opportunity and honor to deliver the session how to get the most of your Azure SQL database and Azure SQL Managed Instance with my colleague Sabrin Alsahsah Support Engineer in Azure SQL Database EMEA team.
Lesson Learned #341: Index Disabled vs Performance issues
Today, we got a very interesting service request where our customer reported an issue about performance, where they mentioned that they have an execution plan where in staging environment the filter is using Index Seek and production environment is using Index Scan and they need to know why SQL Server is not using the Index Seek operator when the structure is the same.
