Cost-effective alternatives to control table for processed files in Azure Synapse
Hello, good morning.In Azure Synapse Analytics, I want to have a control table for the files that have already been processed by the bronze or silver layers. For this, I wanted to create a dedicated pool, but I see that at the minimum performance level it charges 1.51 USD per hour (as I show in the image), so I wanted to know what other more economical alternatives I have, since I will need to do inserts and updates to this control table and with a serverless option this is not possible.QTip: Getting timeouts and exceptions using Azure SQL DB Query Store
Requirements: Azure SQL Database SQL Server Management Studio (SSMS) Tables testcodes and testinvoice used for this demo 1-Configure Query Store to collect data and save as soon as possible to be able to review it * Data flush interval is the time to send information to disk ** Statistics collection interval is the time range to collect data , query store is not collecting one record by execution is collecting statistics of each execution and you can get max, min, avg but not excecution by excecution *** Query store capture mode ALL will collect information of all executions query store will not make any kind of filter 2-Create table testcodes CREATE TABLE [dbo].[testcodes]( [Code] [char](5) NULL, [Description] [nchar](10) NULL ) ON [PRIMARY] GO 3-Create table testinvoice CREATE TABLE [dbo].[testinvoice]( [code] [char](5) NULL, [pieces] [int] NULL ) ON [PRIMARY] GO 4-Add some codes to be used 5-Add some records to invoice Exception ... 6-Run query to get information from invoices and get description from codes select *,description=(select testcodes.Code from testcodes where testcodes.code=invoice.code) from testinvoice invoice Is possible to see execution without error 7-Now add in codes a second code BBB to force exception 8-Run query from point 6 again Result is an exception 9-Run query to see queries with exceptions or timeouts starting 2 days ago (declare @datestart as datetime = dateadd(D,-2,getdate());) declare @datestart as datetime = dateadd(D,-2,getdate()); declare @datefinish as datetime = getdate(); /* if you want to set to specific time */ --set @datestart = '2025-04-09 00:00:00'; --set @datefinish = '2025-04-09 23:59:59'; select rs.last_execution_time, rs.execution_type_desc, qt.query_sql_text, q.query_id, CONVERT(VARCHAR(1000), q.query_hash, 1) as strqueryhash, p.plan_id, rs.last_cpu_time, rs.last_duration, rs.count_executions, rs.last_rowcount, rs.last_logical_io_reads, rs.last_physical_io_reads, rs.last_query_max_used_memory, rs.last_tempdb_space_used, rs.last_dop, p.is_forced_plan, p.last_force_failure_reason, p.last_force_failure_reason_desc FROM sys.query_store_query_text AS qt JOIN sys.query_store_query AS q ON qt.query_text_id = q.query_text_id JOIN sys.query_store_plan AS p ON q.query_id = p.query_id JOIN sys.query_store_runtime_stats AS rs ON p.plan_id = rs.plan_id where rs.last_execution_time>= @datestart and rs.last_execution_time<=@datefinish and (rs.execution_type=3 or rs.execution_type=4) -- 3 timeout, 4 error --and qt.query_sql_text like '%actual%' --and q.query_hash=0x009C458D20394C37 --and p.plan_id=12 ORDER BY rs.last_execution_time DESC 10-In query there are some commented lines that you can use to add more filters or modify them To see all records comment line 20 declare @datestart as datetime = dateadd(D,-2,getdate()); declare @datefinish as datetime = getdate(); /* if you want to set to specific time */ --set @datestart = '2025-04-09 00:00:00'; --set @datefinish = '2025-04-09 23:59:59'; select rs.last_execution_time, rs.execution_type_desc, qt.query_sql_text, q.query_id, CONVERT(VARCHAR(1000), q.query_hash, 1) as strqueryhash, p.plan_id, rs.last_cpu_time, rs.last_duration, rs.count_executions, rs.last_rowcount, rs.last_logical_io_reads, rs.last_physical_io_reads, rs.last_query_max_used_memory, rs.last_tempdb_space_used, rs.last_dop, p.is_forced_plan, p.last_force_failure_reason, p.last_force_failure_reason_desc FROM sys.query_store_query_text AS qt JOIN sys.query_store_query AS q ON qt.query_text_id = q.query_text_id JOIN sys.query_store_plan AS p ON q.query_id = p.query_id JOIN sys.query_store_runtime_stats AS rs ON p.plan_id = rs.plan_id where rs.last_execution_time>= @datestart and rs.last_execution_time<=@datefinish --and (rs.execution_type=3 or rs.execution_type=4) -- 3 timeout, 4 error --and qt.query_sql_text like '%actual%' --and q.query_hash=0x009C458D20394C37 --and p.plan_id=12 ORDER BY rs.last_execution_time DESC Now you can see all records 11-Reproduce error several times within 1 minute to see value in execution count (number of executions inside of statistics collection interval) Timeout ... 12-Configure command timeout different to 0 for this demo use 10 (seconds) in parameters previous to connect Second option 12-Use query below that will run 1000 times and at some point will fail INSERT INTO [dbo].[testinvoice] SELECT * FROM [dbo].[testinvoice] GO 1000 13-Run query from point 9 to see data in query store Now you can reproduce and get data about all excecutions , exceptions and timeouts good luck!
SQL Server - SOS_SCHEDULER_YIELD and Unbalanced NUMA Node Usage
Hi everyone, we've faced an unusual behavior on our SQL Server and would like to understand the cause and how to manage it properly. Our DB Server is a physical server with 64 cores (128 logical CPUs) and 4 NUMA nodes. During a peak in application requests/session, we noticed a significant increase in SOS_SCHEDULER_YIELD waits, with an unusual distribution of CPU load: only 2 NUMA nodes were heavily saturated, while the others remained underutilized. Our main questions are: At what point does SQL Server assign the execution NUMA node to a session/process? During connection establishment? When the statement starts executing? If processor affinity is left at its default setting (not manually configured at the instance level), what metrics can we monitor to better understand and manage this behavior? Any insights or experiences with similar cases would be greatly appreciated. Thanks in advance!
Enhancements for protecting hosted SQL servers across clouds and hybrid environments
Introduction We are releasing an architecture upgrade for the Defender for SQL Servers on Machines plan. This upgrade is designed to simplify the onboarding experience and improve protection coverage. In this blog post, we will discuss details about the architecture upgrade and the key steps customers using the Defender for SQL Servers on Machine plan should take to adopt an optimal protection strategy following this update. Overview of Defender for Cloud database security and the Defender for SQL Servers on Machines plan Databases are an essential part of building modern applications. Microsoft Defender for Cloud, a Cloud Native Application Protection Platform (CNAPP), provides comprehensive database security capabilities to assist security and infrastructure administrators in identifying and mitigating security posture risks, and help Security Operation Center (SOC) analysts detect and respond to database cyberattacks. As organizations advance their digital transformation, a comprehensive database security strategy that covers hybrid and multicloud scenarios is essential. The Defender for SQL Servers on Machines plan delivers this by protecting SQL Server instances hosted on Azure, AWS, GCP, and on-premises machines. It provides database security posture management capabilities and threat protection capabilities to help you start secure and stay secure when building applications. More specifically, it helps to: Centralize discovery of managed and shadow databases across clouds and hybrid environments. Reduce database risks using risk-based recommendations and attack path analysis. Detect and respond to database threats including SQL injections, access anomaly, and suspicious queries. SOC teams can also detect and investigate attacks on databases using built-in integration with Microsoft Defender XDR. Benefits of the agent upgrade for the Defender for SQL Servers on Machine plan Starting from April 28, 2025, we began a gradual rollout of an upgraded agent architecture for the Defender for SQL Servers on Machines plan. This upgraded architecture is designed to simplify the onboarding process and improve protection coverage. This upgrade will eliminate the Azure Monitor framework dependency and replace it with a proven, native SQL extension infrastructure. Azure SQL VMs and Azure Arc-enabled SQL Servers will automatically migrate to the updated architecture. Actions required after the upgrade Although the agent architecture upgrade will be automatic, customers the have enabled the Defender for SQL Servers on Machines plan before April 28th, will need to take action to ensure they adopt optimal plan configurations to help detect and protect unregistered SQL Servers. 1) Update the Defender for SQL Servers on Machines plan configuration for optimal protection coverage To automatically discover unregistered SQL Servers, customers are required to update the plan configurations using this guide. This will ensure Defender for SQL Servers on Machines plan can detect and protect all SQL Server instances. Click the Enable button to update the agent configuration setting: 2) Verify the protection status of SQL virtual machines or Arc-enabled SQL servers Defender for Cloud provides a recommendation titled "The status of Microsoft SQL Servers on Machines should be protected” to help customers assess the protection status of all registered SQL Servers hosted on Azure, AWS, GCP, and on-premises machines within a specified Azure subscription and presents the protection status of each SQL Server instance. Technical context on the architecture upgrade Historically, the Defender for SQL Servers on Machines plan relied on the Azure Monitor agent framework (MMA/AMA) to deliver its capabilities. However, this architecture has proven to be sensitive to diverse customer environmental factors, often introducing friction during agent installation and configuration. To address these challenges, we are introducing an upgraded agent architecture designed to reduce complexity, improve reliability, and streamline onboarding across varied infrastructures. Simplifying enablement with a new agent architecture The SQL extension is a management tool that is available on all Azure SQL virtual machines and SQL servers connected through Azure Arc. It plays a key role in helping simplify the migration process to Azure, enabling large-scale management of your SQL environments and enhancing the security posture of your databases. With the new agent architecture, Defender for SQL utilizes the SQL extension as a backchannel to streamline the data from SQL server instances to the Defender for Cloud portal. Product performance implications Our assessments confirm that the new architecture does not negatively impact performance. For more information, please refer to Common Questions - Defender for Databases. Learn more To learn more about the Defender for SQL Servers on Machines architecture upgrade designed to simplify the onboarding experience and enhance protection coverage, please visit our documentation and review the actions needed to adopt optimal plan configurations after the agent upgrade.
SQL Server Config Manager Error "MMC could not create the snap-in"
Hi, I have seen this error elsewhere online. I have gone to mmc to enable the snap in and I still have had no fix. My computer is running Windows Server 2022, SQL Server Express 2022, and SSMS. I have reinstalled, repaired, and all of the other tricks. Help!
Setting up SQL Managed Instance Link to an Availability Group
On a recent case, a customer was trying to set up SQL Managed Instance Link to partner with an on premise Always On Availability Group (AG). Using the public documentation will work, but this will only be active against the primary node and a failover will cause the database on the managed instance side to stop syncing. This post will be using steps from the following documents. Prepare your environment for a link - Azure SQL Managed Instance Configure link with scripts - Azure SQL Managed Instance The steps to set this up are below Create a database master key on your AG nodes Enabling trace flags on your AG nodes (Optional) Testing network connectivity between your SQL Managed Instance and the AG Create certificates on your AG nodes Import AG certificate public keys to your SQL Managed Instance Import the certificate public key of your SQL Managed Instance to your AG nodes Import Azure-trusted root certificate authority keys to your AG nodes Alter the mirroring endpoint on your AG nodes Create a distributed availability group on your AG Set up the managed Instance Link Create a database master key You first need to create a master encryption key on all nodes of your AG if it does not already exist. You can check if it exists by running this query. -- Run on SQL Server USE master; GO SELECT * FROM sys.symmetric_keys WHERE name LIKE '%DatabaseMasterKey%'; If no results are returned, run this query, keep note of the passwords in a confidential and secure place. -- Run on SQL Server -- Create a master key USE master; GO CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<strong_password>'; Enabling trace flags on your AG nodes (Optional) To optimise the performance of your link, we recommend enabling the trace flags T1800 and T9567 on all of your nodes. Instructions on this are here and will require a restart of the service. Testing network connectivity between your SQL Managed Instance and the AG Instructions on how to run the connectivity tests can be found here, and can be done by SSMS or T-SQL. For these tests, your SQL Managed Instance will need to be able to connect to your Mirror endpoint using the IP of your availability group. All nodes of your AG will need to be able to connect to ports 5022 and 11000-11999 on the subnet of your SQL Managed Instance. Create certificates on your AG nodes On each node, run the query below to create a certificate. You can alter your @cert_expiry_date variable to a date that suits you. -- Create the SQL Server certificate for the instance link USE MASTER -- Customize SQL Server certificate expiration date by adjusting the date below DECLARE @cert_expiry_date AS varchar(max)='03/30/2025' -- Build the query to generate the certificate DECLARE @sqlserver_certificate_name NVARCHAR(MAX) = N'Cert_' + @@servername + N'_endpoint' DECLARE @sqlserver_certificate_subject NVARCHAR(MAX) = N'Certificate for ' + @sqlserver_certificate_name DECLARE @create_sqlserver_certificate_command NVARCHAR(MAX) = N'CREATE CERTIFICATE [' + @sqlserver_certificate_name + '] ' + char (13) + ' WITH SUBJECT = ''' + @sqlserver_certificate_subject + ''',' + char (13) + ' EXPIRY_DATE = '''+ @cert_expiry_date + ''''+ char (13) IF NOT EXISTS (SELECT name from sys.certificates WHERE name = @sqlserver_certificate_name) BEGIN PRINT (@create_sqlserver_certificate_command) -- Execute the query to create SQL Server certificate for the instance link EXEC sp_executesql @stmt = @create_sqlserver_certificate_command END ELSE PRINT 'Certificate ' + @sqlserver_certificate_name + ' already exists.' GO Import AG certificate public keys to your SQL Managed Instance The public keys of your AG certificates need to be imported into your SQL Managed Instance. Run this query on each node to get the data required -- Run on SQL Server -- Show the name and the public key of generated SQL Server certificate USE MASTER GO DECLARE @sqlserver_certificate_name NVARCHAR(MAX) = N'Cert_' + @@servername + N'_endpoint' DECLARE @PUBLICKEYENC VARBINARY(MAX) = CERTENCODED(CERT_ID(@sqlserver_certificate_name)); SELECT @sqlserver_certificate_name as 'SQLServerCertName' SELECT @PUBLICKEYENC AS SQLServerPublicKey; You will get a result similar to below You can now import these into your SQL Managed Instance using Azure Cloud Shell. Log in using this script, replacing . # Run in Azure Cloud Shell (select PowerShell console) # Enter your Azure subscription ID $SubscriptionID = "<SubscriptionID>" # Login to Azure and select subscription ID if ((Get-AzContext ) -eq $null) { echo "Logging to Azure subscription" Login-AzAccount } Select-AzSubscription -SubscriptionName $SubscriptionID Then run this script, replacing the values of $CertificateName, $PublicKeyEncoded and $ManagedInstanceName. # Run in Azure Cloud Shell (select PowerShell console) # =============================================================================== # POWERSHELL SCRIPT TO IMPORT SQL SERVER PUBLIC CERTIFICATE TO SQL MANAGED INSTANCE # ===== Enter user variables here ==== # Enter the name for the server SQLServerCertName certificate – for example, "Cert_sqlserver1_endpoint" $CertificateName = "<SQLServerCertName>" # Insert the certificate public key blob that you got from SQL Server – for example, "0x1234567..." $PublicKeyEncoded = "<SQLServerPublicKey>" # Enter your managed instance short name – for example, "sqlmi" $ManagedInstanceName = "<ManagedInstanceName>" # ==== Do not customize the below cmdlets==== # Find out the resource group name $ResourceGroup = (Get-AzSqlInstance -InstanceName $ManagedInstanceName).ResourceGroupName # Upload the public key of the authentication certificate from SQL Server to Azure. New-AzSqlInstanceServerTrustCertificate -ResourceGroupName $ResourceGroup -InstanceName $ManagedInstanceName -Name $CertificateName -PublicKey $PublicKeyEncoded Repeat this step for all AG nodes. Import the certificate public key of your SQL Managed Instance to your AG nodes Using the same Azure Cloud Shell, run this script to get the public key of your SQL Managed Instance certificate, replacing the value of the variable $ManagedInstanceName. # Run in Azure Cloud Shell (select PowerShell console) # =============================================================================== # POWERSHELL SCRIPT TO EXPORT MANAGED INSTANCE PUBLIC CERTIFICATE # ===== Enter user variables here ==== # Enter your managed instance short name – for example, "sqlmi" $ManagedInstanceName = "<ManagedInstanceName>" # ==== Do not customize the following cmdlet ==== # Find out the resource group name $ResourceGroup = (Get-AzSqlInstance -InstanceName $ManagedInstanceName).ResourceGroupName # Fetch the public key of the authentication certificate from Managed Instance. Outputs a binary key in the property PublicKey. Get-AzSqlInstanceEndpointCertificate -ResourceGroupName $ResourceGroup -InstanceName $ManagedInstanceName -EndpointType "DATABASE_MIRRORING" | out-string On all nodes of your AG you can then run this query, replacing with the value of the PublicKey output in the previous step. -- Run on SQL Server USE MASTER CREATE CERTIFICATE [<ManagedInstanceFQDN>] FROM BINARY = <PublicKey> Import Azure-trusted root certificate authority keys to your AG nodes Use the steps here to import the Microsoft PKI root-authority certificate and DigiCert PKI root-authority certificate onto all nodes of your AG. Alter the mirroring endpoint on your AG nodes The mirroring endpoint on your AG nodes will also need to be updated to allow your Managed Instance to authenticate with your AG using the newly created certificate. This can be done by navigating to Server Objects > Endpoints > Database Mirroring. Right click on the endpoint and select Script Endpoint as > CREATE To > New Query Editor Window. You will need to add this command under AUTHENTICATION, changing the cert_name value and change the CREATE ENDPOINT to ALTER ENDPOINT. CERTIFICATE [cert_name] Once updated, execute the query. Create a distributed availability group on your AG Use this script to set up the distributed availability group, updating the values - Your choice of name - AG Name already created : - The IP of your AG listener and the endpoint listener port - Your choice of name - FQDN of your instance - Just the instance name -- Run on SQL Server -- Create a distributed availability group for the availability group and database -- ManagedInstanceName example: 'sqlmi1' -- ManagedInstanceFQDN example: 'sqlmi1.73d19f36a420a.database.windows.net' USE MASTER CREATE AVAILABILITY GROUP [<DAGName>] WITH (DISTRIBUTED) AVAILABILITY GROUP ON N'<AGNameOnSQLServer>' WITH ( LISTENER_URL = 'TCP://<SQLServerIP>:<EndpointPort>', AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT, FAILOVER_MODE = MANUAL, SEEDING_MODE = AUTOMATIC, SESSION_TIMEOUT = 20 ), N'<AGNameOnSQLMI>' WITH ( LISTENER_URL = 'tcp://<ManagedInstanceFQDN>:5022;Server=[<ManagedInstanceName>]', AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT, FAILOVER_MODE = MANUAL, SEEDING_MODE = AUTOMATIC ); GO Set up the managed Instance Link The final step is to set up the link, you can do this by running the script here. Please ensure that the $SQLServerIP is the IP of your AG listener and not the IP of the primary node. The in the link includes updated parameters from Az.Sql 6.0.0. This version is not currently available in Azure Cloud Shell. If you are using Azure Cloud Shell, replace the final section of the command... ...with the below New-AzSqlInstanceLink -InstanceName $ManagedInstanceName -Name $DAGName -PrimaryAvailabilityGroupName $AGNameOnSQLServer -ResourceGroupName $ResourceGroup -SecondaryAvailabilityGroupName $AGNameOnSQLMI -SourceEndpoint $SourceIP -TargetDatabase $DatabaseName Once this is executed, your MI Link should be up and running and failing over to another node on your AG will not stop your MI Link from syncing. Disclaimer Please note that products and options presented in this article are subject to change. This article reflects MI Link and the documentation in January 2025. I hope this article was helpful for you, please feel free to share your feedback in the comments section.
Creating LinkedServer between On-Prem SQL to Azure SQL MI
Hi, is it possible to use security option (Be made using the login's current security context )with lnikedserver from on-prem SQL to MI ? Our MI is enabled for windows auth and we are successfully connect MI in SSMS using windows auth from on-prem client. when I use this option in linkedserver and try to access linked server with windows auth I get following error. Login failed for user '<token-identified principal>'. Reason: Could not find a user matching the name provided
Price reduction and upcoming features for Azure confidential ledger!
Effective March 1, 2025, you can keep your records in Azure confidential ledger (ACL) at the reduced price of ~$3/day per instance! The reduced price is for the computation and the ledger use. The price of any additional storage used will remain unchanged. To tamper protect your records: Automatically create hash (e.g. MD5 or SHA256) of your blob storage data and keep those in Azure confidential ledger. For forensics, you can verify the integrity of the data against the signature in ACL. Imagine doing this as you are migrating data from one system to another, or when you restore archived records from cold storage. It is also valuable when there is a need to protect from insider/administrator risks and confidently report to authorities. If you keep your data in Azure SQL database, you can use their security ledger feature to auto generate record digests and store them in confidential ledger for integrity protection and safeguarding. You can use the SQL stored procedure to verify that no tampering or administrator modifications occurred to your SQL data! In addition, we are announcing the preview of User Defined Functions for Azure confidential ledger. Imagine doing a schema validation before writing data to the Ledger or using pattern matching to identify sensitive information in log messages and perform data massaging to mask it. To increase your awareness, request access for this preview via the sign-up form. Get started by reading our documentation and trying out confidential ledger yourself! _____________________________________________________________________________________________________ What is Azure confidential ledger and what is the change? It is a tamper protected and auditable data store backed by a Merkle tree blockchain structure for sensitive records that require high levels of integrity protection and/or confidentiality. While customers from AI, financial services, healthcare, and supply chain continue to use the ledger for their business transaction’s archival needs and confidential data’s unique identifiers for audit purposes, we are acting on their feedback for scaling ledgers to more of their workloads with a more competitive price! How can I use Azure confidential ledger? - Azure SQL database ledger customers can enable confidential ledger as its trusted digest store to uplevel integrity and security protection posture - Azure customers who use blob storage have found value in migrating their workloads to Azure with a tamper protection check via the Azure confidential ledger Marketplace App. - Azure customers who use data stores and databases (e.g. Kusto, Cosmos, and Log Analytics) may benefit from auditability and traceability of logs being kept in the confidential ledger with new compliance certifications in SOC 2 Type 2 and ISO27001. How much does Azure confidential ledger cost? - Approximately $3/day/ledger _____________________________________________________________________________________________________ Resources Explore the Azure confidential ledger documentation Read the blog post on: Integrity protect blob storage Read the blog post on: How to choose between ledger in Azure SQL Database and Azure Confidential Ledger Read the blog post on: Verify integrity of data transactions in Azure confidential ledger View our recent webinar in the Security Community Recent case studies: HB Antwerp & BeekeeperAI
