Preview of multiparty analytics with Azure Confidential Clean Rooms
Today, we are excited to announce the preview of multiparty analytics feature of Azure Confidential Clean Rooms, a fully managed service that allows customers and their partners to securely analyze privacy-sensitive datasets from multiple parties. It uses confidential compute enabled Apache Spark-based big-data analytics (Spark SQL) which helps protect their raw data from other collaborators and from the Azure operator by performing computations in a Trusted Execution Environment (TEE). Privacy-sensitive datasets include personally identifiable information (PII), protected health information (PHI) and cryptographic secrets. Organizations across industries are increasingly looking to supplement their data with data from business partners, to build a complete view of their business. For example, brands, publishers, and their partners need to collaborate using datasets containing Intellectual Property (IP) to improve the relevance of their campaigns. Confidential data clean rooms help solve this challenge by enabling organizations to share and analyze granular datasets in a secure environment that helps prevent raw data exfiltration—protecting intellectual property, preserving customer privacy, and addressing concerns around regulatory compliance. You can sign up for the preview here Key Features Fully Managed: Azure takes care of the infrastructure provisioning and scaling with no user intervention. This significantly reduces your onboarding effort allowing you to focus on the queries and insights, not on infra management. Confidential Spark SQL: Spark SQL allows you to query large datasets and run complex queries in a distributed computing environment. In the confidential computing enabled version, the Spark driver and executors are fully attested policy-governed enclaves running as virtual nodes on confidential Azure Container Instances (ACI) which helps prevent exfiltration of collaborators’ data during query execution. Governance: Helps manage membership to cleanrooms, enables and verifies approval for queries from relevant collaborators before executing them and verifies consent to access sensitive collaborator data. It also helps generate tamper-resistant audit trails containing salient clean room events. This is made possible with the help of an implementation of the Confidential Consortium Framework (CCF). Telemetry: Throughout every clean-room run, detailed logs are streamed out in real time to monitor performance, troubleshoot issues, and keep the analytics healthy — all without ever exposing the collaborators’ data at any time. Verifiable trust: Cryptographic remote attestation viz. full attestation based on confidential hardware reports allows independent verification of the TEE along with along with all components that are part of it, without just trusting the cloud provider, before sensitive data and decryption keys are made available to the TEE. Open-source containers: All Microsoft provided cleanroom containers and sidecars are open-sourced here and can be verified for provenance and integrity guarantees using GitHub artifact attestation. Use Cases Multi-party confidential big-data analytics unlocks value in scenarios where data sensitivity, regulatory pressure, or competitive concerns previously blocked collaboration. These are some early scenarios that can benefit from this. Media & Advertising Collaboration of advertiser CRM data with publisher data for audience targeting and segment activation. Collaboration of audience data with measurement partners for measurement and attribution. Banking & Finance Collaboration between banks and insurance firms to upsell relevant products to existing bank customers without sharing raw data from either side Collaboration with retailers to generate customized offers for bank customers, without exposing either party’s underlying data. Government & Public Sector Secure collaboration of data across government departments to deliver better citizen welfare outcomes. Secure collaboration between government and private enterprises on shared-interest workloads such as traffic monitoring and weather systems. Healthcare Enable healthcare firms — including biopharma organizations — to combine their data with third-party institutions to accelerate clinical development, like identifying eligible participants for a clinical trial, without exposing underlying patient data. Combine patient datasets across hospitals to study disease patterns or outcomes without exposing sensitive protected health information. Beyond Spark SQL Realizing other multi-party scenarios like custom analytics, ML training and inferencing on Azure Confidential Clean Rooms is in our roadmap. If you have such a scenario to be realized, you can fill in and submit the preview signup form with the details of your scenario and we’ll get back to you. Learn More · Signup for the preview of Azure Confidential Clean Rooms for Analytics · Confidential Consortium Framework (CCF) · Virtual Nodes on Azure Container Instances
Creating Azure SQL VM with same name as VM
Hi, Currently we have a resource group, which contains a Virtual Machine and SQL Virtual Machine (and a few other resources). The VM and SQL VM has the same name: I want to move this resource group (and its resources) to another subscription. We tried using Resource Mover, but we couldn't as there are backups. I tried the steps in the following link: https://petri.com/copy-azure-vm-using-managed-disk-snapshots/ I was successful in moving 5 of the 6 resources to the new subscription, but was unable to move the SQL virtual machine: I tried to create a SQL virtual machine, but it says I can't create one with the same name: I found that in order to create a VM and SQL VM's with the same name, in the above image, I have to choose an image that has SQL Server and Windows. But this doesn't allow me to use the original managed disk (it only has an OS disk) and it also creates a couple of more disks, which are not in the original resource group. I was wondering if there are recommendations on how to create a SQL VM that has the same name as the VM in the same resource group and are also linked to each other. Jason
Cost-effective alternatives to control table for processed files in Azure Synapse
Hello, good morning.In Azure Synapse Analytics, I want to have a control table for the files that have already been processed by the bronze or silver layers. For this, I wanted to create a dedicated pool, but I see that at the minimum performance level it charges 1.51 USD per hour (as I show in the image), so I wanted to know what other more economical alternatives I have, since I will need to do inserts and updates to this control table and with a serverless option this is not possible.
Price reduction and upcoming features for Azure confidential ledger!
Effective March 1, 2025, you can keep your records in Azure confidential ledger (ACL) at the reduced price of ~$3/day per instance! The reduced price is for the computation and the ledger use. The price of any additional storage used will remain unchanged. To tamper protect your records: Automatically create hash (e.g. MD5 or SHA256) of your blob storage data and keep those in Azure confidential ledger. For forensics, you can verify the integrity of the data against the signature in ACL. Imagine doing this as you are migrating data from one system to another, or when you restore archived records from cold storage. It is also valuable when there is a need to protect from insider/administrator risks and confidently report to authorities. If you keep your data in Azure SQL database, you can use their security ledger feature to auto generate record digests and store them in confidential ledger for integrity protection and safeguarding. You can use the SQL stored procedure to verify that no tampering or administrator modifications occurred to your SQL data! In addition, we are announcing the preview of User Defined Functions for Azure confidential ledger. Imagine doing a schema validation before writing data to the Ledger or using pattern matching to identify sensitive information in log messages and perform data massaging to mask it. To increase your awareness, request access for this preview via the sign-up form. Get started by reading our documentation and trying out confidential ledger yourself! _____________________________________________________________________________________________________ What is Azure confidential ledger and what is the change? It is a tamper protected and auditable data store backed by a Merkle tree blockchain structure for sensitive records that require high levels of integrity protection and/or confidentiality. While customers from AI, financial services, healthcare, and supply chain continue to use the ledger for their business transaction’s archival needs and confidential data’s unique identifiers for audit purposes, we are acting on their feedback for scaling ledgers to more of their workloads with a more competitive price! How can I use Azure confidential ledger? - Azure SQL database ledger customers can enable confidential ledger as its trusted digest store to uplevel integrity and security protection posture - Azure customers who use blob storage have found value in migrating their workloads to Azure with a tamper protection check via the Azure confidential ledger Marketplace App. - Azure customers who use data stores and databases (e.g. Kusto, Cosmos, and Log Analytics) may benefit from auditability and traceability of logs being kept in the confidential ledger with new compliance certifications in SOC 2 Type 2 and ISO27001. How much does Azure confidential ledger cost? - Approximately $3/day/ledger _____________________________________________________________________________________________________ Resources Explore the Azure confidential ledger documentation Read the blog post on: Integrity protect blob storage Read the blog post on: How to choose between ledger in Azure SQL Database and Azure Confidential Ledger Read the blog post on: Verify integrity of data transactions in Azure confidential ledger View our recent webinar in the Security Community Recent case studies: HB Antwerp & BeekeeperAI
How to execute SQL script via Azure Pipeline, please help
How to execute SQL script via Azure Pipeline, please help, I don't want to do it via powershell. Please suggest. For example I will be creating an Azure pipeline.yml where under a stage/job will be having steps to execute an SQL script file(having multiple CRUD operation)
Confidential Data Clean Rooms – The evolution of sensitive data collaboration
Secure data collaboration between multiple parties has the potential to revolutionize societies, businesses and industries for the better. Collaborating on sensitive data assets facilitates innovation to unlock new value for organizations.Frictionless Collaborative Analytics and AI/ML on Confidential Data
Secure enclaves protect data from attack and unauthorized access, but confidential computing presents significant challenges and obstacles to performing analytics and machine learning at scale across teams and organizational boundaries. In this article, we'll explore the Opaque platform and describe how it can enable multiple parties to easily collaborate and analyze shared data while keeping it fully confidential.
Trying to Set Up Advanced SQL Tracking
I work for a corporation, and we are trying to set up Application Insights for one of our products. We have the instrumentation key and everything integrated into the code where necessary, but we still can't see all the information we want... The goal is for our screen to look like this: https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp-net-dependencies We've been using the above link to guide us. We have installed the Microsoft.Data.SQLClient NuGet Package, and the XML line, but the advanced SQL data still does not show. We are running things locally on IIS Express, and our project runs, and when we interact with the web app we see spikes in our performance graph, so we know it is collecting some metrics. We apparently need to be running SDK Version 'rddp'. We are using the version 'rddf' and 'rdddsd' which indicates dependencies are collected via DiagnosticSource or EventSource callbacks, and hence full SQL query aren't being captured. We got the information about the SDK Version by running this command in Logs in App Insights: dependencies | where timestamp > ago(1d) | summarize count() by cloud_RoleInstance, sdkVersion
Hosted SQL advice please
I’m looking to migrate away from our on prem DC and separate SQL server into Azure. My plan is to join our Windows 10 clients to our existing Azure domain (AAD), continue to use Exchange online for email and move files and folders to OneDrive. All of the above I’m happy with so far. The bit I’m unsure about is our SQL databases which our client software uses. I know I can create a SQL instance in Azure and potentially migrate our data across to it but am concerned about latency between the client app and the azure sql instance. We have a 100mb/100mb leased line as our internet link. any advice on this please
