Analytics on Azure Blog articles

Migrating Azure Cosmos DB Across Regions

Rafia_Aqil — Fri, 16 Oct 2026 04:38:18 GMT

What this article delivers? A decision-oriented, implementation-ready walkthrough based on a hands-on Azure Cosmos DB for NoSQL lab. It corrects the online copy sequence, includes the required Azure CLI snippets, and separates migration duration from application cutover downtime.

Regional migration in Azure Cosmos DB is not a single operation. The correct pattern depends on whether the priority is the simplest supported move, the smallest application outage, or a durable multi-region architecture.

The Three Migration Patterns at a Glance

Pattern	Best fit	App impact	Destination	Primary caution
1. Cross-Region PITR restore	Simplest regional move	Stop writes during final cutover	New account created by restore	Target-region capacity and restore-point readiness
2. Online copy	Minimal downtime	Source stays online until cutover	Pre-created account and empty target containers	Preview prerequisites, RBAC, RU limits, one job per container
3. Provisioned multi-region	HA/DR and predictable throughput	Online while adding region; controlled failover	Same account, second region	Higher ongoing cost; conversion back to serverless is not a standard self-service path

Before you migrate: establish the decision inputs

A reliable migration plan starts by separating total data-movement time from business downtime. Online copy can run for hours or days without requiring the application to stop. Restore may complete quickly, but the application still needs a controlled write freeze, validation, endpoint change, and restart. Define both measures explicitly.

Capacity is a migration dependency: A restore that creates a new account still requires the target region to accept the deployment. Validate region availability early, keep an escalation path ready, and avoid treating a successful small-environment test as a capacity guarantee for production.

Current Environment Setup

Cosmos DB Environment with Capacity Mode= Serverless:

Current CosmoDB Topology:

Data Count in current database pre-migration:

Approach 1: Cross-region point-in-time restore

This is the preferred first option when the customer can accept a short write outage and is comfortable moving to a new Cosmos DB account. The restore operation creates the target account; you do not pre-create the destination account as you do for online container copy.

Conceptual Plan:

Execution flow

Confirm the source account uses continuous backup and that the required restore point is available.
Validate target-region readiness and prepare the target account name, resource group, networking, identity, and application configuration plan.
Schedule the cutover and stop application writes.
Start the restore into the target region. Azure creates the new account.
Validate account state, databases, containers, policies, item counts, networking, private endpoints, and data-plane RBAC.
Update the application endpoint and credentials or managed-identity assignments.
Start the application and validate reads and writes.
Keep the source account unchanged for an agreed rollback window, then retire it after sign-off.

Pre-checks and validation snippets

RG="CosmoDB_Demo"
SRC_ACCOUNT="demotestingenvironment"

az cosmosdb show \

--resource-group "$RG" \

--name "$SRC_ACCOUNT" \

--query "{Account:name,BackupPolicy:backupPolicy.type,Locations:locations[].locationName,State:provisioningState}" \

-o json

Review BackupPolicy Type:

Note Follow Restore an Account That Uses Continuous Backup Mode - Azure Cosmos DB | Microsoft Learn to update to restore option

In the Azure Portal: Cosmos DB → testingenvironmentfordemo → Backup & Restore. Periodic → Continuous

Referesh the page and fill in Restore requirements. Cross region restore incurs additional charges on top of the normal point-in-time restores, the price depends on the regions that you are restore from and to. Learn more

New Account is created, with data:

Testing and Validation

Check if the deployment is successful:

Count of data across both databases are the same:

Approach 2: Online container copy with minimal downtime

Online container copy is a container-by-container migration. The application continues reading and writing to the source during the initial copy and incremental synchronization. At cutover, the team stops source writes, waits for the job to catch up, calls the completion operation, validates the target, and then switches the application. The Azure CLI command group and the feature are preview capabilities, so production adoption should include preview acceptance, current limitation review, and a tested fallback.

Reference topology used in the lab

Component	Source	Destination
Account	demotestingenvironment	demotestingenvironmentrestore
Database	demotestingDB	demotestingDB
Container	demotestingDBID	demotestingDBID-copy
Partition key	/category	/category
Mode	Serverless	Serverless in the lab

Step 1: Define reusable variables

RG="CosmoDB_Demo"

SRC_ACCOUNT="demotestingenvironment"

DEST_ACCOUNT="demotestingenvironmentrestore"

SRC_DB="demotestingDB"

SRC_CONTAINER="demotestingDBID"

DEST_DB="demotestingDB"

DEST_CONTAINER="demotestingDBID-copy"

JOB="online-copy-hls-final-04"

Install the copy extension

az extension add --name cosmosdb-preview --upgrade

Step 2: Verify source

az cosmosdb show -g "$RG" -n "$SRC_ACCOUNT" --query "{Backup:backupPolicy.type,Capabilities:capabilities[].name,State:provisioningState}" -o json

Confirm that Backup is Continuous. Confirm that the source has All Versions and Deletes change feed enabled, and review compatibility limitations before applying the capability to an existing production account.

Step 3: Create the empty destination container

az cosmosdb sql container create -g "$RG" -a "$DEST_ACCOUNT" -d "$DEST_DB" -n "$DEST_CONTAINER" -p "/category"

Configure source identity and destination access

az cosmosdb identity assign -g "$RG" -n "$SRC_ACCOUNT"

SRC_PRINCIPAL=$(az cosmosdb show -g "$RG" -n "$SRC_ACCOUNT" --query "identity.principalId" -o tsv)

az cosmosdb update -g "$RG" -n "$SRC_ACCOUNT" --default-identity "SystemAssignedIdentity"

az cosmosdb sql role assignment create -g "$RG" -a "$DEST_ACCOUNT" --role-definition-id "00000000-0000-0000-0000-000000000002" --principal-id "$SRC_PRINCIPAL" --scope "/"

Step 4: Start Online Copy

az cosmosdb copy create --resource-group "$RG" --job-name "$JOB" --src-account "$SRC_ACCOUNT" --dest-account "$DEST_ACCOUNT" --src-nosql database="$SRC_DB" container="$SRC_CONTAINER" --dest-nosql database="$DEST_DB" container="$DEST_CONTAINER" --mode Online

Monitor

az cosmosdb copy show -g "$RG" -a "$DEST_ACCOUNT" -n "$JOB" --query "{Status:status,Processed:processedCount,Total:totalCount,Error:error}" -o table

Query the destination container

SELECT *
FROM c
WHERE c.id = "HLS-ONLINE-COPY-001"

Also review via CLI

The record appearing in the destination proves both the initial copy and the incremental-change path. Also compare source and target counts, but do not rely on count equality alone; validate representative documents, partition-key values, updates, deletes, TTL behavior, and application queries.

Note: Perform the production cutover correctly

Stop all writes to the source container.
Wait for the online job to catch up and confirm Error is null.
Run the complete operation to finalize the online copy job.
Validate target counts, representative records, updates/deletes, indexing behavior, networking, identity, and application connectivity.
Switch the application endpoint to the destination account and container.
Run read/write smoke tests and monitor 429s, 403s, latency, and error rates.
Keep the source unchanged for the approved rollback window, then retire it.

Approach 3: Convert to provisioned throughput and use multi-region replication

This pattern changes the question from “How do I copy data into a second account?” to “How do I extend the existing account into a second region and fail over?” It is appropriate when the workload needs predictable throughput, stronger regional resiliency, or serverless limits make copy-based approaches impractical.

Convert the existing serverless account to provisioned throughput.
Review and right-size the initial manual throughput assigned to each container or database.
Add the target region as an additional region on the same account.
Wait for replication and validate application connectivity.
Change failover priority or initiate the approved failover method so the target region becomes the write region.
Decide whether to retain the original region for HA/DR or remove it after validation.

Convert: Azure Portal → Cosmos DB → demotestingenvironment → Overview → Change capacity mode to provisioned throughput and confirm. Refresh and the capacity mode should change.

Under Settings, select Replicate data globally, Select West US 2 as the existing region and under Additional regions, select North Central US.

Change the write region:

Validate current write region

Decision framework

Decision question	Restore	Online copy	Provisioned multi-region
Can the app stop writes briefly?	Best fit	Not required during bulk copy	Not required while adding region
Must the target be an existing account?	No; restore creates a new account	Yes; pre-create target account/container	Same account
Need to remain serverless?	Supported if target deployment is accepted	Possible, but throughput can slow copy	No; account becomes provisioned
Need built-in regional HA after migration?	Single target account unless redesigned	Single target account unless redesigned	Best fit
Main production risk	Target capacity and cutover validation	Preview prerequisites, RBAC, 429s, copy duration	Cost, throughput sizing, irreversible architecture choice

References

Container Copy Jobs - Azure Cosmos DB

Azure CLI: az cosmosdb copy

Restore an Azure Cosmos DB account using continuous backup

Migrate an account to continuous backup mode

Configure data-plane role-based access control

Change from serverless to provisioned throughput

Compare provisioned throughput and serverless

Azure Cosmos DB pricing

Optimizing Oracle-to-Power BI Performance and the Path to Modern Analytics

Rafia_Aqil — Fri, 16 Oct 2026 04:12:58 GMT

Organizations frequently rely on Oracle databases for large-scale operational workloads while using Microsoft Power BI as their enterprise analytics and visualization platform. Although connecting Power BI directly to Oracle is fully supported, performance can become challenging as datasets grow into the millions of rows.

A recent enterprise scenario involving millions of Oracle records illustrates an important principle: Oracle-to-Power BI performance should not be attributed to a single component without first isolating the entire retrieval path.

This article provides a practical framework for optimizing that architecture today, while also examining how Microsoft Fabric Mirroring can provide a different data-access pattern for future analytics workloads.

The Performance Challenge

Consider an Oracle environment containing approximately 15 million records.

Testing showed substantially different behavior depending on the client used to retrieve the same Oracle data:

A direct Oracle query executed through TOAD returned millions of rows in about eight minutes.

Tableau connected to the same Oracle table from the same laptop and refreshed millions of rows in 20 to 30 minutes.

Power BI Desktop, however, continued processing for multiple hours before the operation was terminated by network or database controls.

Importantly, the Power BI query did not contain additional Power Query transformations. The customer was directly executing SQL and loading the resulting data. This is an extremely useful diagnostic comparison because it demonstrates that Oracle is capable of returning the underlying dataset considerably faster than the Power BI test. It does not, however, prove that the Power BI Oracle connector itself is the single root cause.

Performance Is a Pipeline, Not a Single Component

Oracle-to-Power BI performance involves several layers:

A bottleneck at any one of these layers can affect overall refresh or query performance. Therefore, the first architectural principle is: Measure every layer before attempting to tune the entire solution. A troubleshooting exercise should evaluate:

Oracle server-side execution time
SQL execution plan
Oracle provider/client behavior
FetchSize behavior
Network transfer
Power Query evaluation
Query folding
Power BI model processing
Report design
Gateway behavior when the Power BI Service is introduced

This approach avoids prematurely concluding that Oracle, Power BI, the gateway, or the connector is inherently responsible for the slowdown.

Start With the Oracle Provider

Power BI currently provides two important approaches for Oracle connectivity.

Built-in Oracle provider

Modern versions of Power BI Desktop include Microsoft's bundled Oracle Managed ODP.NET provider.

This reduces the dependency on separately installing Oracle client software.

For environments using the built-in provider, testing should be performed with a current Power BI Desktop release because provider behavior and defaults can change between releases.

Oracle Client for Microsoft Tools

Oracle Client for Microsoft Tools, or OCMT, remains another supported option.

OCMT configures Oracle Data Provider for .NET and can therefore be tested independently from the built-in provider.

This gives architects a useful isolation mechanism: Test A: Built-in Oracle provider vs. Test B: OCMT provider. Use the same:

workstation

SQL statement

Oracle database

table

credentials

network path

expected row count

Changing several variables simultaneously makes the results considerably harder to interpret.

Understand FetchSize

One of the most important areas identified during troubleshooting is FetchSize. FetchSize controls how much data the Oracle provider retrieves during each fetch operation. For large datasets, a small FetchSize can require many retrieval operations, increasing overhead. Increasing FetchSize allows the provider to retrieve more data at a time and may improve throughput.

However, FetchSize tuning should be tested independently from the choice of Oracle provider. Switching to OCMT, for example, does not by itself demonstrate whether FetchSize tuning improves performance. Benchmark the same SQL and dataset using consistent test conditions and compare elapsed time and throughput.

FetchSize controls how much Oracle data the provider pulls back in each retrieval batch. Think of FetchSize like the size of a delivery truck. Imagine Power BI needs to retrieve a very large dataset from Oracle.

Small FetchSize:

Oracle → 📦 → Power BI
Oracle → 📦 → Power BI
Oracle → 📦 → Power BI
Oracle → 📦 → Power BI
...many retrieval operations

Larger FetchSize:

Oracle → 📦📦📦📦📦 → Power BI
Oracle → 📦📦📦📦📦 → Power BI
...fewer retrieval operations

The idea is simple: FetchSize determines how much data the Oracle provider fetches at a time.

Examine Oracle Data Types

FetchSize is not the only provider consideration. Schema matters. A troubleshooting package should include the table DDL rather than only a list of column names. Pay particular attention to large-object types such as:

BLOB

CLOB

Large-object columns may materially change retrieval behavior and can make simply increasing FetchSize insufficient. Therefore capture:

complete table DDL

column names

Oracle data types

row width

presence of BLOB/CLOB columns

expected total result size

Do not optimize FetchSize without understanding the shape of the data being transferred.

Filter Rows and Columns as Early as Possible

Probably the most universally applicable Power Query optimization is also one of the simplest: Do not retrieve data you do not need. Instead of retrieving every row and column and filtering afterward, reduce the dataset at Oracle whenever possible.

Good Oracle SQL should therefore:

select only required columns

apply restrictive predicates at the source

aggregate at Oracle where appropriate

avoid unnecessary data transfer

This matters substantially when Power BI and Oracle are separated by a network boundary.

Preserve Query Folding

Power Query query folding allows Power BI transformations to be translated back into SQL executed by the relational source. For relational sources such as Oracle, maintaining folding can substantially reduce the amount of work performed by the Power Query mashup engine. The preferred pattern is:

Where practical:

filter early

remove unnecessary columns early

avoid transformations that unexpectedly break folding

inspect where folding stops

push relational operations back into Oracle

For DirectQuery, folding is particularly important because the source remains part of the interactive report-query path.

Be Careful With Native SQL and Incremental Refresh

Native SQL can be extremely valuable for troubleshooting. Running the same targeted SELECT statement through Power BI that is executed directly against Oracle creates a useful A/B test. It helps isolate questions such as:

Is Oracle itself taking a long time?

Is Power BI generating a different query?

Is additional processing happening after Oracle returns the data?

Is the retrieval/provider layer adding significant latency?

However, architecture matters.

Power BI guidance warns that later Power Query steps cannot necessarily fold after a native SQL query, and native SQL has important implications for incremental refresh. Therefore, native SQL is useful both as an optimization technique in some architectures and as a diagnostic isolation technique, but it should not automatically be substituted for a well-folded Power Query design.

Prefer Import for Traditional Oracle BI Workloads

If the business does not require near-real-time data, Import should generally be evaluated before DirectQuery for large Oracle analytical workloads. With Import:

The latency of Oracle, networking, and the provider is mostly experienced during refresh rather than repeatedly during user interaction. This usually provides a more predictable interactive experience. DirectQuery produces a different dependency chain:

As a result, source latency can become report latency. Use DirectQuery when its architectural benefits are required, rather than simply because the source database supports it.

Make Incremental Refresh Truly Incremental

Import does not necessarily mean repeatedly loading the entire Oracle dataset. For large fact tables, incremental refresh should be evaluated so that only required partitions are refreshed. However, there is an important condition: RangeStart and RangeEnd filtering needs to be pushed back to Oracle.

If partition filters do not fold, Power BI can end up retrieving considerably more source data than expected before applying the filtering. Therefore incremental refresh testing should verify that the Oracle query generated by Power BI contains the expected predicates.

Treat the Gateway as Its Own Performance Layer

Once the workload moves into the Power BI Service, however, the on-premises data gateway becomes another measurable component. For production deployments examine:

gateway version

Oracle provider/client version

memory

network proximity to Oracle

refresh concurrency

DirectQuery concurrency

competing workloads

Where necessary, large refresh workloads and latency-sensitive DirectQuery workloads can be isolated onto different gateway resources. Do not diagnose gateway performance from a Power BI Desktop test where the gateway is not involved.

Optimize DirectQuery If It Cannot Be Avoided

Sometimes Import is not viable because users require data with very low latency. If Oracle must remain behind DirectQuery, reduce unnecessary work at every level. At Oracle

index frequently filtered columns appropriately

review execution plans

select only necessary columns

reduce returned rows

avoid unnecessarily expensive calculations

In Power Query

maintain query folding

push filtering to Oracle

minimize transformations requiring local evaluation

In the semantic model

use efficient dimensional modeling

avoid unnecessary high-cardinality columns

consider aggregation strategies where appropriate

In reports

limit visual density

use selective filters

avoid generating unnecessary simultaneous source queries

test with realistic user concurrency

DirectQuery performance needs to be assessed as an end-to-end architecture, rather than as an individual SQL benchmark.

Collect Diagnostics Before Declaring a Root Cause

For difficult Oracle performance cases, establish a repeatable diagnostic package. Capture:

Oracle

database version

execution plan

SQL execution duration

table DDL

data types

row count

approximate returned data size

Provider

built-in provider versus OCMT

provider/client version

FetchSize configuration

Power BI

Power BI Desktop version

generated SQL

Query Diagnostics

Mashup traces

Applied Steps or M code

total execution time

Network

Oracle-to-client path

throughput

latency

connection interruptions or timeouts

Comparison clients

If Tableau or another Oracle client appears dramatically faster, reproduce the same SQL, same columns, same records, same database and preferably the same machine. Only then does the comparison become a useful engineering benchmark.

Future Architecture: Oracle Mirroring in Microsoft Fabric

Optimizing the direct Oracle connection solves one class of problem. Microsoft Fabric Mirroring provides a different architecture altogether. Instead of repeatedly placing Oracle in the analytical query path, Fabric can continuously replicate Oracle changes into OneLake. Conceptually:

Fabric Mirroring uses Oracle LogMiner to capture changes and maintain an analytical copy in OneLake. This separates many downstream analytical reads from the operational Oracle database. The key architectural advantage isn't simply "a faster connector." The advantage is decoupling analytics from the operational database.

Recommended Architecture Decision Tree

Requirement: traditional BI with scheduled freshness

Use: Oracle → Import → Power BI semantic model. Then optimize:

FetchSize

provider

query folding

source-side filtering

incremental refresh

Requirement: operational or near-real-time reporting where Oracle must remain the source

Evaluate: Oracle → DirectQuery → Power BI. But aggressively optimize:

Oracle SQL

execution plans

query folding

visual density

provider configuration

concurrency

gateway architecture

Requirement: enterprise analytics with frequent Oracle changes and growing BI/AI requirements

Evaluate: Oracle → Fabric Mirroring → OneLake → Fabric / Power BI. This reduces repeated analytical pressure against the operational Oracle environment and creates a reusable analytical data foundation for additional workloads.

A Practical Troubleshooting Sequence

A useful Oracle-to-Power BI performance investigation can be organized into five phases.

Phase 1: Establish the Oracle baseline

Execute exactly the same query directly against Oracle. Record: SQL → rows → bytes → execution time

Phase 2: Isolate the provider

Run the same dataset through: Built-in Oracle provider and OCMT. Record the results independently.

Phase 3: Test FetchSize

Test controlled FetchSize values while holding every other variable constant.

Measure throughput rather than relying on subjective impressions.

Phase 4: Optimize Power Query

Validate:

query folding

source filtering

column projection

transformations

incremental refresh behavior

Phase 5: Decide whether the architecture itself should change

If extremely large Oracle datasets must continually feed analytical workloads, consider whether repeatedly querying Oracle remains the appropriate long-term design.

At that point, Fabric Mirroring becomes an architectural discussion rather than simply another troubleshooting step.

Final Takeaway

Oracle-to-Power BI performance problems should not immediately be categorized as an Oracle problem, a Power BI problem, or a gateway problem.

They are usually best investigated as a data retrieval pipeline. For existing Power BI architectures, the strongest optimization sequence is: Efficient Oracle SQL

For the longer term, Microsoft Fabric Mirroring changes the architecture: Oracle becomes the operational system of record while OneLake becomes the analytical data plane. That separation can be particularly valuable as organizations expand from traditional Power BI reporting into broader Fabric analytics, data engineering and AI workloads.

Azure Databricks Cost and Workload Assessment: a Local Web UI

varghesejoji — Wed, 30 Sep 2026 03:26:49 GMT

Introduction
1. What you can assess
2. The five-step workflow
3. Reopen saved assessments
4. Architecture and collection flow
5. Get started

Introduction

To review an Azure Databricks environment, you need to know what it costs, how its compute is used, and which jobs need attention. That information is spread across Azure billing and Databricks.

The Assessment & Optimization Workbench is a local web application that brings these details together. Select your workspaces and dates, check access, and run an assessment. You can then explore the results in your browser and download a report to discuss with your team. The walkthrough below shows each step.

1. What you can assess

The assessment covers costs, workload behavior, and selected configuration checks:

Area	What you can inspect
Cost	Actual and Amortized costs, top drivers, attribution gaps, and commitment scenarios with eligible hourly inputs.
Efficiency	CPU/memory samples, idle observations, sizing candidates, query duration, and queueing.
Job health	Failures, retries, failure notifications, and network observations.
Inventory and posture	Workspace assets, IP access list settings, and Unity Catalog metastore assignment.
Evidence	Collection coverage, snapshots, raw-data imports, and offline re-analysis.
Handoff	Prioritized findings, optional review decisions, formatted reports, and Excel exports.

If some data could not be collected, the UI shows what is missing. Recommendations are suggestions for you to review and test; the assessment does not apply them automatically. Actual savings must be measured after you make the recommended changes.

2. The five-step workflow

The UI guides you from choosing what to assess to downloading a report. First, select your workspaces and check access. Then start the assessment, explore the results, and export what you need. The walkthrough below shows each step.

Configure → Validate → Run analysis → Visualize results → Review & export

Configure

This Configure step defines what the assessment will cover: which workspaces to review, the date range, and the data to collect. These choices prepare the assessment; they do not start collection.

The GIF shows the scope selection, warehouse choices, and date settings before moving to validation.

Select the scope: choose the subscriptions, resource groups, and Databricks workspaces to include.
Set the dates and cost basis: choose the period to assess. ActualCost shows charges as recorded; AmortizedCost spreads eligible commitment costs over time.
Choose the collection profile: use Standard for core assessment data, Extended to include workspace asset metadata, or Custom to select optional data and analysis modules.
Select SQL Warehouses: choose a warehouse for each workspace that needs system-table queries. Selection does not start compute. Queries can incur charges, so warehouse use requires approval in Validate.

Next: Select Validate configuration.

Validate

Validation checks your setup and access to the selected Azure and Databricks data. It helps you identify missing permissions or approvals before starting the assessment.

Select Run validation, approving warehouse auto-start if required. The UI shows which checks are running, which have passed, and what needs attention. The GIF shows an approval issue being resolved, followed by the access checks.

Fix any blockers that prevent the assessment from running, and read warnings about data that may be limited or unavailable. If permissions are missing, the permission panel provides commands for an authorized administrator to run. Validation itself does not grant access.

Next: Once validation allows you to proceed, select Continue to run below the permission panel. You will review the setup and start the assessment separately.

Run analysis

This step collects data from your selected Azure and Databricks workspaces, analyzes it, and creates the assessment report. You can follow the progress and see which sources returned data.

Review the selected workspaces and date range, then select Start read-only assessment. The screen shows:

Progress: the current stage, from initial checks and collection through analysis and report generation, plus elapsed time.
Collection sources: cards grouped into Azure data and Databricks workspaces. Each card shows its status and the number of items collected, such as cost records, inventory, or billing data.
Console: detailed messages to help explain delays or collection problems.

When the run finishes, green means collection and analysis succeeded, not that every workload is healthy. Red means the run failed or some data needs attention. A skipped source was not collected, often because an optional check was not selected.

The report and collected data are saved locally, including any reported gaps.

Next: Select Visualize results to explore the saved assessment.

Visualize results

This step turns the collected data into charts, tables, and findings you can explore. Start with the overall cost and collection summary, then look at individual workspaces, compute resources, and jobs to understand what needs attention. You are viewing saved results, so switching tabs does not collect data again.

The GIF follows the results from cost and workload details through findings, data quality, and a proposed roadmap. The tabs are grouped into Overview, Technical, and Decisions.

Overview: understand the cost

Executive summary: see total cost, spending mapped to workspaces, optimization candidates, and collection status. Check whether the detailed costs match the billing total and how much supporting data is available.
Cost analysis: use Cost evidence to compare Actual and Amortized trends. Break down spending by Service, Meter category, SKU, Resource group, Workspace, or Owner tag, then inspect the largest cost drivers and unmapped spend. Commitment opportunities lets you enter a proposed number of committed nodes and calculate a scenario when the required hourly usage and pricing data is available. It does not purchase a commitment.

Technical: inspect resources and workloads

Compute and SQL has five views:
- Inventory: cluster and warehouse settings, including node types, worker counts, Photon, and automatic shutdown, plus job run and failure summaries.
- Utilization: CPU, memory, idle observations, and sample counts. Select a resource and a driver or worker instance to view its CPU and memory chart.
- Sizing: current node and worker configuration, with candidates to benchmark before resizing.
- Job health: runs, failures, notification settings, tasks, and retry policies.
- Network: data sent and received by nodes, plus CPU-wait measurements. These are traffic observations, not billed network charges.
Queries: switch between Individual queries, Warehouse summaries, and User summaries. Review durations, queue times, and failures; search and sort to find queries that need investigation.
Posture: inspect IP access list settings and Unity Catalog metastore assignment. Each check shows its observed value and outcome; this is not a full security audit.
Assets: browse collected repositories, notebook metadata, MLflow experiments, serving endpoints, SQL alerts, Genie spaces, and Unity Catalog volumes. These appear only when the relevant optional data was collected.

Decisions: check findings and plan the next steps

Findings: browse recommendations by category, status, and confidence. Select a row to read what was observed, the recommended next step, supporting records, and any limitations.
Evidence quality: see coverage by workspace, missing data, failed or skipped sources, and records excluded from scope. Open Inspect effective rules / create another analysis to review or change analysis settings, then Create child analysis to rerun them on saved data without changing the original snapshot.
Roadmap: review proposed work across Days 0-30, 31-60, and 61-90, including owners and dependencies. The measurement plan explains the baseline for checking savings after a change; the roadmap is not an approved implementation schedule.

Use Filters to narrow findings by subscription, resource group, workspace, workload, category, confidence, or status. Scope selections also apply to supported technical tables, which have their own search, sorting, and paging controls. Select a resource name to inspect its saved details.

If data is unavailable, check Evidence quality before drawing a conclusion. A missing measurement does not mean that a resource was unused.

Next: Select Review & export to read the report or download the results.

Review & export

This final step lets you read the assessment report and download files to discuss with your team. You can also record decisions on individual findings. Review is optional, so you do not need to approve every finding before exporting.

The GIF shows a finding being reviewed, the formatted report and supporting files being opened, and an Excel workbook being generated and downloaded.

Read the report: select Preview report to move directly to the formatted report in your browser. Use its contents links to jump to sections, and open supporting evidence links to inspect the saved files.
Download the report: select Download report to save the original Markdown version. Downloading a file completes this workflow step, but does not approve findings or apply changes.
Record a decision: expand Record a decision, choose a finding and decision, then enter the reviewer and an optional note. Select Save review decisions to retain the changes. A decision other than pending requires a reviewer.
Inspect supporting files: use Run artifacts to preview supported files or download individual outputs. Reports appear as formatted pages; CSV and JSON previews show the file contents as text.
Create an Excel workbook: choose the modules under Excel workbook, then select Generate workbook artifact and download the generated file. It includes summary, rules, quality, findings, saved review decisions, and the selected module sheets. It covers the full saved assessment, not just the rows currently filtered in the UI.

Save or discard pending review edits before generating a workbook. Exports include saved decisions only. Check the files before sharing them because supporting evidence can contain sensitive environment details.

Optional dashboard publication is a separate cloud action for publishing coverage counts, not the full assessment. It requires a destination workspace and warehouse, a preview of the publication plan, and explicit approval of the write and possible warehouse charges.

3. Reopen saved assessments

A snapshot is a locally saved assessment, including the selected scope, collected data, findings, reports, and review decisions. It lets you return to earlier results, continue a review, or download files later without querying Azure or Databricks again.

Reopen an assessment: choose a run from Saved snapshots. Entries show the run date, customer, status, and run ID, with the newest first. You can then explore its results or continue in Review & export.
Try different analysis settings: open Evidence quality, expand Inspect effective rules / create another analysis, adjust the settings, and select Create child analysis. This creates a separate assessment using the same saved data. The original remains unchanged, and the new findings need their own review.
Remove old assessments: use Manage snapshots to delete individual snapshots or clear the saved history. Deletion requires confirmation and permanently removes the selected runs, including their evidence, reports, and review decisions. Back up important runs first.

Snapshots show what was collected at the time, not the current environment. To get newer data or collect a missing source, start a new assessment.

4. Architecture and collection flow

The workbench runs on your machine, with a browser interface connected to a local Python server at http://127.0.0.1:8765. Azure and Databricks supply the data; collection scripts, analysis, and saved results stay local. No Azure-hosted application is needed.

The diagram follows an assessment through seven stages:

Configure: the React and TypeScript browser UI captures your selected workspaces, dates, and collection options.
Coordinate: the Python API receives requests from the browser, launches assessment processes, and tracks progress.
Check access: validation checks the selected scope, permissions, and required approvals. These checks use live access, but full collection starts only when you select Start read-only assessment.
Collect: PowerShell scripts read Azure resource inventory and billing data, plus Databricks APIs and system tables. Requests to these external sources use HTTPS.
Save evidence: each run stores the returned data, configuration, source outcomes, and logs in a local folder.
Analyze: Python combines the saved data, checks costs against billing totals, identifies data gaps, and applies rules to produce findings and reports.
Review and export: the API returns saved results to the browser, where you explore findings, record decisions, and download reports or data files.

Keep in mind: the assessment does not apply recommendations. SQL Warehouse queries can incur charges, and exported files should be checked for sensitive details before sharing.

5. Get started

Try it

Prerequisites: PowerShell 7+, Python 3, Node.js 22.12+/npm for the build, and Azure CLI signed in. Collection needs Azure Reader/Cost Management Reader and the required Databricks source access.

From the repository root:

Set-Location .\ui npm ci npm run build Set-Location .. .\ui\Start-AssessmentUi.ps1

Open http://127.0.0.1:8765 and keep the launcher running. If already built, only the final command is needed.

Learn more

Contribute

Report issues in the repository with reproduction steps and redacted source statuses. Never include credentials or raw customer evidence.

Publishing check: redact environment identifiers in GIFs and upload media when posting outside the repository. Displayed timings and amounts are not benchmarks or savings claims.

Learn What to Do When You Hit Capacity in Azure Databricks!

Rafia_Aqil — Fri, 18 Sep 2026 14:58:25 GMT

Microsoft's Cloud Architects: Manu Mehta manumehta, Chris Walk cwalk, Eduardo Dos Santos eduardomdossantos, Maria Hito mariahito, Kiran Raja Ch KiranRaja, Paul Singh PaulSingh, Aladdin Alchalabi AladdinAlchalabi and Rafia Aqil Rafia_Aqil

Start Here: Engage Microsoft

Capacity constraints in Azure Databricks are not an Azure Databricks product issue. Azure Databricks does not own or reserve compute, it dynamically provisions VMs from Azure when clusters are created or scaled. This means cluster creation, autoscaling, or job execution can stall when the underlying VM SKUs are constrained at the regional level. The fastest path to resolution is a structured conversation with your Microsoft account team, who can engage the Azure capacity intake process on your behalf.

Create a Quota Support Ticket via Microsoft Support and bring the following to your account team with your Support Ticket Number. Each field maps directly to what capacity intake teams will ask for: missing fields slow the request.

What to Prepare Before You Reach Out Your Account Team

Field	What Capacity Intake Needs	Example
Subscription IDs	The exact Azure subscriptions that will host the workspaces and clusters	7ebee83d-7923-426c-8449-59fd4dff25ab
Region(s)	Primary region, plus any acceptable alternates	East US 2
VM family / SKU	Specific series and version requested	Eadsv5, ESv4, DSv4, DSv2
Core count / new limit	Total vCPU or core count per SKU	10,000 cores for Eadsv5
Workload characteristic	CPU-bound vs. memory/shuffle-heavy vs. IO-heavy; batch vs. streaming vs. SQL	“Memory-intensive ETL with large joins and shuffles”
Scale and timing	When you need it, ramp profile, peak vs. steady state	“Need by month-end; ramp from 2,000 to 9,650 cores over Q3”
Business context	Business use case	“Migration off AWS”

What “Capacity” Really Means: A Layered Mental Model

Before diving into fixes, it is important to understand what is actually happening behind the scenes. Capacity constraints can occur at three distinct layers, and solving them requires addressing each one.

Layer 1: Azure Infrastructure

This is the layer most teams underestimate. Capacity here is governed by:

VM SKU availability in the region. D-series and E-series: the two most common Databricks worker families: have repeatedly hit capacity constraints across multiple Azure regions, causing cluster creation failures, autoscale stalls, and provisioning delays.

Regional supply constraints, which are dynamic and shared across all Azure tenants.

vCPU quotas and limits per subscription, which are separate from regional supply. Quota is your subscription’s limit to deploy resources (like a credit card limit); regional capacity is the underlying infrastructure available. Both must be sufficient.

Mechanism	Guarantees Capacity	Costs Money when idle	Discount
vCPU quota	No	No	N/A
Instance pool	Best effort	Yes (VM only, no DBU)	No
Reserved Instance	No	N/A	Yes
Savings Plan	No	N/A	Yes
CRG	Yes, within SLA	Yes	No, but RI/SP can apply
Serverless	Platform-Managed	No	N/A

Layer 2: Azure Databricks Platform

The Azure Databricks control plane has its own published ceilings that your architecture must proactively respect. Key limits from the official Azure Databricks resource limits documentation:

Resource	Limit	Scope
Jobs created per hour	10,000	Workspace
Tasks running simultaneously	2,000	Workspace (Run Job and For Each parent tasks excluded)
Parent tasks running simultaneously (Run Job / For Each)	750	Workspace
SQL warehouses	1,000	Workspace
Attached notebooks or execution contexts	145	Cluster
Virtual machines	25,000	Per subscription per region

Note: For limits marked as non-fixed in the official documentation, you can request an increase through your Azure Databricks account team. Reference: https://learn.microsoft.com/en-us/azure/databricks/resources/limits

Layer 3: Workload (Spark Execution)

Even when both lower layers cooperate, Spark’s own execution model can produce capacity-like symptoms:

Parallelism and task distribution, which dictate how many cores a job can usefully consume.

Memory pressure from joins, shuffles, and skewed keys.

IO demand and caching behavior, including Delta cache effectiveness and Spark cache misuse.

Understanding these layers is critical. Retries sometimes succeed because capacity is dynamic: as other workloads complete, nodes are released back to Azure and briefly become available.

Recognizing When You’ve Hit Capacity

Capacity issues rarely present as a single clean error. Instead, they appear as inconsistent behaviors:

Clusters stuck in Pending state

Autoscaling fails or never reaches the desired size

Jobs intermittently fail to start

Retry attempts sometimes succeed

These inconsistencies occur because capacity is shared across Azure tenants and fluctuates throughout the day. Running workloads outside peak business hours in the impacted region’s time zone is one of the most effective short-term mitigations. Inconsistent symptoms are not the same as unknowable ones. Before escalating, confirm what you are actually looking at. Several very different problems produce the symptoms above, and only one of them is a regional capacity shortage.

1. Where to look first

- Start with the cluster's termination reason and event log in the Azure Databricks workspace.
- Then cross-check the Azure Activity Log for the workspace's managed resource group over the same time window, which shows the VM allocation attempt and its result.

2. Match signal to the clause

What you observe	Points to Review	What to do
Cluster-provider launch or stockout failure	Regional capacity for that VM size	Immediate Actions, below
Quota, core, or vCPU limit referenced	Subscription quota, not capacity	Request a quota increase — capacity may be fine
VM size unavailable in the region or zone	Availability restriction, not a transient shortage	Switch VM SKU or Family below, retrying will not help
Pool returns INSTANCE_POOL_MAX_CAPACITY_FAILURE	A Databricks pool ceiling you configured	Raise the pool's maximum capacity
Cluster starts normally but jobs run slowly, spill, or OOM	Workload design, not capacity	Why Adding more Nodes is Not Always the Answer, below

- Only the first row is a genuine Azure capacity constraint. The others are resolved without any capacity conversations

3. Check quota before you conclude capacity

- Quota and capacity fail in similar ways but are resolved through entirely different paths.
- Compare current usage against the limit for the VM series and region in question. If usage is below the limit and allocation still fails, the constraint is regional capacity. If usage it at the limit, it is quota and an increase may resolve it outright.

Immediate Actions: How to Unblock Your Workloads

When you are actively hitting capacity constraints, speed matters. Please reach out to your Microsoft Account team and try these mitigations that are ordered from quickest to most involved.

Retry and Run During Off-Peak Hours

- Capacity availability changes throughout the day as workloads complete and release VMs. Running outside peak business hours for the impacted region significantly improves success rates.
- Retrying is bounded, not unlimited. As a rule of thumb, retry two or three times across different hours, including at least one off-peak window in the impacted region's time zone. If the same VM size and region fall consistently across a full business day, stop treating it as transient; open a support ticket, engage your account team, and evaluate VM families in parallel.
- If the workload is production critical with a fixed deadline, or if failures are blocking a migration or cutover already in flight, escalate immediately without waiting for the retry window.

Switch VM SKU or Family

If a specific VM SKU is constrained, switching to another can immediately unblock provisioning.

- Move within the same family (for example, DSv4 → DSv5)

- Or switch families entirely (for example, D-series → F-series or L-series)

Choosing the Right VM Family

Most Databricks environments default to D-series (general purpose) and E-series (memory optimized). These are also the most heavily used and most capacity-constrained VM families. Consider alternatives based on your workload:

VM Family	Best For	When to Use	Trade-off
D-series	General workloads	Default choice	Often constrained in high-demand regions
E-series	Memory-heavy Spark jobs	Joins, shuffles, analytics	High demand; higher cost
F-series	CPU-intensive jobs	Parsing, transformations	Lower memory per core
L-series	IO-heavy workloads	Delta caching, large datasets	Higher cost; large local NVMe

Practical decision framework:

- Memory-bound workloads (joins, shuffles): Move from E-series to L-series. Similar memory per core, plus large local NVMe for Delta caching.

- CPU-bound workloads: Move from D-series to F-series. Higher CPU performance at lower cost.

- IO-heavy or cache-sensitive workloads: L-series can significantly improve performance and reduce shuffle pressure.

Implement Regional Diversity in your Databricks workload

- As Azure capacity constraints are region and SKU-specific, it is important to build architectural flexibility into your Databricks deployments.

- For critical or large-scale workloads, consider deploying multiple Databricks workspaces across different Azure regions to reduce dependency on any single region’s capacity.

This approach enables:

- - improved resilience to regional capacity constraints

- - greater flexibility in workload placement

- - Important: Multi-region deployment requires deliberate architecture, including deploying separate workspaces and replicating data and configurations across regions; it is not automatic.

Why Adding More Nodes Is Not Always the Answer

When jobs slow down, the instinct is to scale compute. With Spark, more nodes do not always solve the problem.

Common workload issues that masquerade as capacity problems:

Data skew

Excessive shuffle operations

Inefficient partitioning

Overuse of UDFs

In some workloads, shuffle operations can grow significantly larger than the original input data, placing substantial pressure on compute, memory, disk I/O, and network resources. Because shuffle workloads are distributed across the cluster, adding nodes can improve performance by increasing parallelism. However, that benefit reaches a limit when the bottleneck is caused by data skew, oversized shuffle partitions, network-intensive data movement, or data explosion from joins and aggregations. In these scenarios, the workload becomes constrained by the shuffle pattern itself, and simply adding more nodes does not address the root cause. Instead, the shuffle strategy, partitioning approach, or query design should be optimized.

Smarter optimization strategies:

Reduce shuffle through repartitioning and query optimization

Enable Photon for faster execution

Optimize Delta tables using Z-ordering and compaction

Leverage caching strategically (not just Spark cache: use the Delta/disk cache)

These optimizations can reduce your dependency on scarce VM capacity altogether. Review optimization strategies: The Complete Guide to Azure Databricks Cost Optimization | Microsoft Community Hub.

What to Do When Your Capacity Is Approved

Once Azure approves your capacity request, retaining it requires active steps. Because Azure capacity is dynamic and shared, approved capacity is held only while compute remains actively deployed and running. This is especially important in highly constrained regions. Microsoft recommends the following:

Configure an Instance Pool

For workloads that cannot yet use serverless compute, configure an Azure Databricks Instance Pool with a minimum number of idle nodes aligned to your production requirements.

An instance pool pre-allocates and maintains a set of idle, ready-to-use VM instances. When a cluster is created from the pool, it draws from these warm nodes: eliminating the need to request new VMs from the regional Azure capacity pool between job runs.

Key behaviors:

The pool holds a minimum number of nodes continuously, keeping them warm and immediately available.

Clusters attached to the pool pull from warm nodes, avoiding re-acquisition from Azure between runs.

No DBU charges apply while nodes are idle in the pool.

Azure VM infrastructure costs do apply for all minimum idle instances. Size the pool conservatively: aligned to production need only: to balance capacity retention against ongoing cost.

Important:

Instance pools hold idle nodes on a best-effort basis. Periodic platform events can recycle pool nodes, briefly causing the pool to fall below its configured minimum idle count while Azure re-acquires replacement nodes. Pools significantly improve availability and startup latency, but they do not change the fact that the underlying VMs are still requested from Azure on demand. They are not a hard reservation. Reference: https://learn.microsoft.com/en-us/azure/databricks/compute/pools
You can launch a pool's instances against an Azure capacity reservation group by setting the capacity_reservation_group field in the pool's azure_attributes to the group's resource ID. Configure it through the Instance Pools API or the Azure Databricks SDKs. The same requirements apply as for clusters: on-demand instances only, and only workspaces that use VNet injection.

Designing for Resilience: Long-Term Best Practices

To avoid repeated capacity issues, your architecture needs to evolve beyond reactive mitigations.

Plan Ahead with Azure Capacity Reservation Groups

For organizations running mission-critical Azure Databricks workloads, Azure Capacity Reservation Groups (CRGs) can provide additional predictability by reserving VM capacity in advance for your Databricks compute resources. Rather than competing for available regional capacity during periods of high demand, reserved capacity helps ensure that the required VM families are available when clusters need to scale or start, Reference: Databricks Clusters API documentation.

Note: Before you commit to a reservation, know three things:

1. Auto-termination stops saving VM costs. When a cluster terminates, its reserved capacity returns to an unused state and continues billing at the full VM rate.
2. A pool backed by a reservation is not billed twice. If a team already pays for minimum idle pool nodes in a constrained region, a reservation at comparable spend converts best-effort capacity into SLA-backed capacity.
3. Confirm that the cluster is actually using the reservation. Creating the reservation proves Azure set capacity aside, but it does not prove Databricks is drawing on it. Start a cluster, then check that allocated instances on the reservation rose by the expected node count. If it stays at zero, the usual causes are a VM size mismatch, availability not set to ON_DEMAND_AZURE, a workspace on the Databricks-managed VNet, or the RBAC actions never granted. Note also that omitting capacity_reservation_group when editing an instance pool silently clears it.

Step-by-Step Instructions:

Attaching a CRG to Databricks is done only through the Clusters/Instance Pools API or the Databricks SDKs, it is not available in the compute UI.

Prerequisites

- VNet-injected workspace only. The workspace must be deployed into your own VNet. Workspaces on the default Databricks-managed VNet cannot use a CRG.

- On-demand instances only. The cluster/pool must use ON_DEMAND_AZURE availability. Spot and serverless are not eligible.

- Same region. Create the CRG in the same Azure region as the workspace.

- Matching VM size. Reserve the exact VM SKU(s) your cluster uses (driver and workers).

- Sufficient subscription quota for that SKU and core count.

- Go to the CRG resource -> Access Control -> Add role assignment and add the below roles to the workspace (i.e. databricks-login-prod) Enterprise Application:
  - Microsoft.Compute/capacityReservationGroups/read
  - Microsoft.Compute/capacityReservationGroups/deploy/action
  - Microsoft.Compute/capacityReservationGroups/capacityReservations/read
  - Microsoft.Compute/capacityReservationGroups/capacityReservations/deploy/action

Step 1: Create the CRG and reservation in Azure

az group create -l eastus -g myResourceGroup az capacity reservation group create \ -n myCapacityReservationGroup -l eastus -g myResourceGroup --zones 1 2 3 az capacity reservation create \ -c myCapacityReservationGroup -n myCapacityReservation \ -l eastus -g myResourceGroup --sku Standard_D2s_v3 --capacity 5 --zone 1

Note: If you want to create the CRG from the Azure Portal you can do the following: Set Subscription, Resource group, Name, and Region (use the same region as your Databricks workspace). Optionally pick Availability zones.

Add one or more reservations: Reservation name, Instances (quantity), and VM size (match your cluster's driver/worker SKU). Example here: reservation-eadsv5, 5 × Standard_D4s_v3. Confirm the summary (price, basics, reservations), then click Create.

Step 2 Attach the CRG to the cluster (Clusters API or SDK)

This would be the Azure Databricks compute cluster, the Spark cluster you create inside your Azure Databricks workspace (Compute → Create compute, or a job cluster).

You add an azure_attributes block to the cluster definition. The snippet below is a fragment that goes inside the cluster's JSON, alongside the normal cluster fields. You provide the CRG resource ID; Azure picks a matching reservation within the group.

databricks clusters edit --json '{ "cluster_id": "<existing-cluster-id>", "spark_version": "15.4.x-scala2.12", "node_type_id": "Standard_D4s_v3", "num_workers": 4, "azure_attributes": { "availability": "ON_DEMAND_AZURE", "capacity_reservation_group": "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Compute/capacityReservationGroups/<crg-name>" } }'

The cluster's node_type_id (VM SKU) has to be the same VM size you reserved in the CRG (Step 1). If the reservation is Standard_D4s_v3, the cluster's node type must also be Standard_D4s_v3, or it won't draw from the reservation.

For instance pools, set the same capacity_reservation_group field via the Instance Pools API or SDK (If you omit the field when editing a pool, Databricks clears any CRG already configured on it).

Plan for Capacity Early

Understand VM quotas and limits before you need them: not after a constraint occurs.

Avoid designing a single SKU. Build flexibility into cluster configurations so you can switch families without re-engineering jobs.

Standardize Compute Configurations

Consistent, policy-driven environments make it easier to adapt when capacity constraints occur. Use Databricks Cluster Policies to constrain cluster creation to approved, available VM families: this prevents teams from inadvertently requesting constrained SKUs. Also, consider enforcing the CRG setting through a Databricks compute policy, so teams launch only against approved, reserved capacity.

Move Toward Serverless Where Possible

Serverless compute abstracts capacity management away from the customer. As the Databricks platform expands serverless support, migrating eligible workloads is the most durable long-term strategy. Azure continues to expand infrastructure capacity, but there are no guaranteed timelines for relief in constrained regions.

Note: If your workload supports serverless compute, Databricks recommends using serverless compute instead of pools or classic VM-backed clusters. Serverless removes dependency on specific VM SKUs and regional capacity: scaling is managed by the platform with significantly improved availability. Reference: https://learn.microsoft.com/en-us/azure/databricks/serverless-compute. For eligible workloads: including Databricks Jobs (automated workflows), Databricks SQL Warehouses, and Delta Live Tables: serverless compute eliminates VM SKU dependency entirely. Configuration guidance is available in the Azure Databricks deployment guide, Development Section, Step 9.

Multi-Region Strategy for Critical Workloads

For the most critical workloads, evaluate a multi-region deployment as part of your business's continuity planning. This is a significant architectural investment: see the FAQ for the full scope: but it is the only approach that provides true regional redundancy. Coordinate this with your Microsoft account team. Reference: Azure Databricks & Microsoft Fabric Disaster Recovery: The Complete Better‑Together Strategy for Cloud Architects

Know the Difference: Azure Capacity Reservations vs. Reserved Instances vs. Savings Plans

When planning Azure infrastructure, it is important to separate capacity assurance from cost optimization. Although these options are sometimes discussed together, they solve different problems.

On-Demand Capacity Reservations (ODCR) are designed to reserve compute capacity for workloads that need to run now. They are useful when an organization needs capacity for an eligible VM size in a specific region or availability zone. ODCRs generally offer flexibility because they do not require a long-term commitment and can be canceled when no longer needed.
Future Capacity Reservations (FCR) support planned capacity requirements for a future need-by date. They are useful for migrations, major launches, seasonal events, and other predictable workload ramps that require advance capacity planning.
In comparison, Azure Reserved VM Instances and Azure Savings Plans for Compute are primarily commercial constructs. Reserved Instances provide discounts for predictable, consistently running workloads through a one-year or three-year commitment. Savings Plans offer broader flexibility by applying discounts to eligible compute usage in exchange for an hourly spending commitment.

The key takeaway is simple: capacity reservations address infrastructure availability, while Reserved Instances and Savings Plans address pricing. Organizations can use them together, pairing a capacity reservation with an applicable pricing benefit to improve both workload readiness and cost efficiency.

Final Takeaways

Capacity issues are infrastructure-level constraints, not Databricks product failures

VM family selection is critical: do not rely solely on D-series and E-series

Workload optimization can reduce dependency on scarce resources before requesting more capacity
Serverless compute is Microsoft’s preferred long-term recommendation for eligible workloads
Architectural flexibility: multi-SKU, multi-region awareness is your best defense against future constraints

FAQ

Why do retries work?

Capacity in Azure regions is shared across all tenants and fluctuates throughout the day as workloads complete and release VMs. A retry succeeds when capacity temporarily frees up. Retrying during off-peak hours improves success rates significantly.

Why does capacity fluctuate during the day?

Capacity is a function of regional supply and concurrent demand. As workloads complete, nodes are released back to Azure. Peak business hours in the impacted region’s time zone tend to be the tightest windows.

Why are instance pools not a hard reservation?

Pools hold a minimum number of nodes on a best-effort basis. Periodic platform events recycle pool nodes, so a pool can briefly fall below its configured minimum idle count while Azure re-acquires replacement nodes. Setting minimum idle to 0 avoids paying for idle VMs at the cost of slower acquisition time. Pools significantly improve availability and startup latency but do not guarantee capacity at the Azure infrastructure level.

Why does serverless behave differently from classic clusters?

Serverless compute removes customer control over individual VM SKUs. Databricks manages the underlying capacity across a shared pool. SKU-swap and pool-based mitigations do not apply. Customer-side levers reduce to retry and off-peak scheduling. The trade-off is that serverless is the simplest and most reliable option when the workload supports it.

Why is changing regions a last resort?

Region changes require redeployment of the Azure Databricks workspace and migration of all dependent artifacts: jobs, clusters, libraries, networking (private endpoints, VNet injection), Unity Catalog assignments, identities, and source data. The destination region must be validated for the same SKU and zonal configuration. For these reasons, region change should always be coordinated with the Microsoft account team and attempted only after preferred mitigations have been exhausted.

Why does VM family selection matter so much for capacity?

Different VM families have different supply curves. D-series and E-series are the most requested Databricks worker families and the ones most frequently constrained. Choosing a SKU based on whether the workload is memory/shuffle-heavy, CPU-bound, or IO-heavy improves both performance and the probability that capacity is available. The capacity team often steers customers toward newer-generation alternatives when supply differs by generation version.

What does the Microsoft account team actually do?

They route the request into the Azure capacity intake process, advise alternate SKUs and regions, surface zonal vs. regional considerations, and provide forward visibility into known constraints. The customer’s job is to bring a complete, accurate workload profile so the account team can advocate effectively.

It is also recommended to open an Azure Support ticket. This will save time later, as the capacity planning teams would like to track issues and requests via a support ticket. Once an Azure Support ticket is opened, the ticket number should be shared to the Microsoft Account Team, at a minimum to the Customer Success Account Manager (CSAM), if one is assigned to your organization.

Complete Guide to Azure Databricks Cost Optimization for Data Engineers

Rafia_Aqil — Sat, 03 Oct 2026 04:08:41 GMT

Co-Authored by: Aladdin Alchalabi AladdinAlchalabi, Sanjeev Nair Sanjeev Nair and Rafia Aqil Rafia_Aqil

This guide walks through a proven approach to Azure Databricks cost optimization, structured in three phases: 1. Discovery, 2. Cluster/Data/Code Best Practices, and 3. Team Alignment & Next Steps.

Phase 1: Discovery

Assessing Your Current State

The following questions are designed to guide your initial assessment and help you identify areas for improvement. Documenting answers to each will provide a baseline for optimization and inform the next phases of your cost management strategy.

Environment & Organization

Cluster Management

Cost Optimization

Data Management

Performance Monitoring

Future Planning

What is the current scale of your Databricks environment?

How many workspaces do you have?

How are your workspaces organized (e.g., by environment type, region, use case)?

How many clusters are deployed?

How many users are active?

What are the primary use cases for Databricks in your organization?

Data engineering
Data science
Machine learning
Business intelligence

How are clusters currently managed?

Manual configuration
Automated scripts
Databricks REST API
Cluster policies

What is the average cluster uptime?

Hours per day
Days per week

What is the average cluster utilization rate?

CPU usage
Memory usage

What is the current monthly spend on Databricks?

Total cost
Breakdown by workspace
Breakdown by cluster

What cost management tools are currently in use?

Azure Cost Management
Third-party tools

Are there any existing cost optimization strategies in place?

Reserved instances
Spot instances
Cluster auto-scaling

What is the current data storage strategy?

Data lake
Data warehouse
Hybrid

What is the average data ingestion rate?

GB per day
Number of files

What is the average data processing time?

ETL jobs
Machine learning models

What types of data formats are used in your environment?

Delta Lake
Parquet
JSON
CSV
Other formats relevant to your workloads

What performance monitoring tools are currently in use?

Databricks Ganglia
Azure Monitor
Third-party tools

What are the key performance metrics tracked?

Job execution time
Cluster performance
Data processing speed

Are there any planned expansions or changes to the Databricks environment?

New use cases
Increased data volume
Additional users

What are the long-term goals for Databricks cost optimization?

Reducing overall spend
Improving resource utilization & cost attribution
Enhancing performance

Understanding Databricks Cost Structure

Total Cost = Cloud Cost + DBU Cost

- Cloud Cost: Compute (VMs, networking, IP addresses), storage (ADLS, MLflow artifacts), other services (firewalls), cluster type (serverless compute, classic compute)

- DBU Cost: Workload size, cluster/warehouse size, photon acceleration, compute runtime, workspace tier, SKU type (Jobs, Delta Live Tables, All Purpose Clusters, Serverless), model serving, queries per second, model execution time

Diagnose Cost and Issues

Effectively diagnosing cost and performance issues in Databricks requires a structured approach. Use the following steps and metrics to gain visibility into your environment and uncover actionable insights.

1. Identify Costly Workloads

- Account Console Usage Reports: Review usage reports to identify usage breakdowns by product, SKU name, and custom tags.
- - Usage Breakdown by Product and SKU: Helps you understand which services and compute types (clusters, SQL warehouses, serverless options) are consuming the most resources.
  - Custom Tags for Attribution: Tags allow you to attribute costs to teams, projects, or departments, making it easier to identify high-cost areas.
  - Workflow and Job Analysis: By correlating usage data with workflows and jobs, you can pinpoint long-running or resource-heavy workloads that drive costs.
1. 1. Focus on Long-Running Workloads: Examine workloads with extended runtimes or high resource utilization.

- - Key Question: Which pipelines or workloads are driving the majority of your costs?

- Governance Hub: is a centralized, account-level UI for monitoring and managing governance across Databricks. Note this is a beta feature, we will update this article as we get more information and use-cases on this feature.

Now That You’ve Identified Long-Running Workloads, Review These Key Areas:

2. Review Cluster Metrics

- CPU Utilization: Track guest, iowait, idle, irq, nice, softirq, steal, system, and user times to understand how compute resources are being used.
- Memory Utilization: Monitor used, free, buffer, and cached memory to identify over- or under-utilization.

- Key Question: Is your cluster over- or under-utilized? Are resources being wasted or stretched too thin?

3. Review SQL Warehouse Metrics

1. Live Statistics: Monitor warehouse status, running/queued queries, and current cluster count.
2. Time Scale Filter: Analyze query and cluster activity over different time frames (8 hours, 24 hours, 7 days, 14 days).
3. Peak Query Count Chart: Identify periods of high concurrency.
4. Completed Query Count Chart: Track throughput and query success/failure rates.
5. Running Clusters Chart: Observe cluster allocation and recycling events.
6. Query History Table: Filter and analyze queries by user, duration, status, and statement type.

- Key Question: Is your SQL Warehouse over- or under-utilized? Are resources being wasted or stretched too thin?

4. Review Spark UI

- Stages Tab: Look for skewed data, high input/output, and shuffle times. Uneven task durations may indicate data skew or inefficient data handling.

- Jobs Timeline: Identify long-running jobs or stages that consume excessive resources.

- Stage Analysis: Determine if stages are I/O bound or suffering from data skew/spill.

- Executor Metrics: Monitor memory usage, CPU utilization, and disk I/O. Frequent garbage collection or high memory usage may signal the need for better resource allocation.

4.1. Spark UI: Storage & Jobs Tab

- Storage Level: Check if data is stored in memory, on disk, or both.

- Size: Assess the size of cached data.

- Job Analysis: Investigate jobs that dominate the timeline or have unusually long durations. Look for gaps caused by complex execution plans, non-Spark code, driver overload, or cluster malfunction.

4.2. Spark UI: Executor Tab

- Storage Memory: Compare used vs. available memory.

- Task Time (Garbage Collection): Review long tasks and garbage collection times.

- Shuffle Read/Write: Measure data transferred between stages.

5. Additional Diagnostic Methods

- System Tables in Unity Catalog:
  - Query system tables for cost attribution and resource usage trends.
  - Cost Observability Queries
- Tagging Analysis:
  Use tags to identify which teams or projects consume the most resources.
- Dashboards & Alerts:
  Set up cost dashboards and budget alerts for proactive monitoring.

Phase 2: Cluster/Code/Data Best Practices Alignment

Cluster UI Configuration and Cost Attribution

Effectively configuring clusters/workloads in Databricks is essential for balancing performance, scalability, and cost. Tunning settings and features when used strategically can help organizations maximize resource efficiency and minimize unnecessary spending.

Key Configuration Strategies

1. Reduce Idle Time: Clusters to incur costs even when not actively processing workloads. To avoid paying for unused resources:

- Enable Auto-Terminate: Set clusters automatically shut down after a period of inactivity. This simple setting can significantly reduce wasted spending.

Enable Autoscaling: Workloads fluctuate in size and complexity. Autoscaling allows clusters to dynamically adjust the number of nodes based on demand:

- Automatic Resource Adjustment: Scale up for heavy jobs and scale down for lighter loads, ensuring you only pay for what you use.

- - It significantly enhances cost efficiency and overall performance.
  - For serverless and streaming, using Delta Live Tables with autoscaling is recommended.
  - This approach leads to better resource management and reliability.

Use Spot Instances: For batch processing and non-critical workloads, spot instances offer substantial cost savings:

- Lower VM Costs: Spot instances are typically much cheaper than standard VMs. However, they are not recommended for jobs requiring constant uptime due to potential interruptions.
- Considerations:
  - Azure Spot VMs are intended for non-critical, fault-tolerant tasks.
  - They can be evicted without notice, risking production stability.
  - No SLA guarantees mean potential downtime for critical applications.
  - Using Spot VMs could lead to reliability issues in production environments.

Leverage Photon Engine: Photon is Databricks’ high-performance, vectorized query engine:

- Accelerate Large Workloads: Photon can dramatically reduce runtime for compute-intensive tasks, improving both speed and cost efficiency.

Keep Runtimes Up to Date: Using the latest Databricks runtime ensures optimal performance and security:

- Benefit from Improvements: Regular updates include performance enhancements, bug fixes, and new features.

Apply Cluster Policies: Cluster policies help standardize configurations and enforce cost controls across teams:

- Governance and Consistency: Policies can restrict certain settings, enforce tagging, and ensure clusters are created with cost-effective defaults.

Optimize Storage: type impacts both performance and cost:

- Switch from HDDs to SSDs: SSDs provide faster caching and shuffle operations, which can improve job efficiency and reduce runtime.

Tag Clusters for Cost Attribution: Tagging clusters enables granular tracking and reporting:

- Visibility and Accountability: Use tags to attribute costs to specific teams, projects, or environments, supporting better budgeting and chargeback processes.

Select the Right Cluster Type: Different workloads require different cluster types, see table below for Serverless vs Classic Compute:

Feature	Classic Compute	Serverless Compute
Control	Full control over config & network	Minimal control, fully managed by Databricks
Startup Time	Slower (unless pre-warmed)	Instant
Cost Model	Hourly, supports reservations	Pay-per-use, elastic scaling
Security	VNet injection, private endpoints	NCC-based private connectivity
Best For	Heavy ETL, ML, compliance workloads	Interactive queries, unpredictable demand

- Job Clusters: Ideal for scheduled jobs and Delta Live Tables.
- All-Purpose Clusters: Suited for ad-hoc analysis and collaborative work.
- Single-Node Clusters: Efficient for simple exploratory data analysis or pure Python tasks.
- Serverless Compute: Scalable, managed workloads with automatic resource management.

11. Monitor and Adjust Regularly: review cluster metrics and query history:

- Continuous Optimization: Use built-in dashboards to monitor usage, identify bottlenecks, and adjust cluster size or configuration as needed.

Code Best Practices

Avoid Reprocessing Large Tables

- Use a CDC (Change Data Capture) architecture with Delta Live Tables (DLT) to process only new or changed data, minimizing unnecessary computation.

Ensure Code Parallelizes Well

- Write Spark code that leverages parallel processing. Avoid loops, deeply nested structures, and inefficient user-defined functions (UDFs) that can hinder scalability.

Reduce Memory Consumption

- Tweak Spark configurations to minimize memory overhead. Clean out legacy or unnecessary settings that may have carried over from previous Spark versions.

Prefer SQL Over Complex Python

- Use SQL (declarative language) for Spark jobs whenever possible. SQL queries are typically more efficient and easier to optimize than complex Python logic.

Modularize Notebooks

- Use %run to split large notebooks into smaller, reusable modules. This improves maintainability.

Use LIMIT in Exploratory Queries

- When exploring data, always use the LIMIT clause to avoid scanning large datasets unnecessarily.

Monitor Job Performance

- Regularly review Spark UI to detect inefficiencies such as high shuffle, input, or output. Review the below table for optimization opportunities:

For more guidance review: Spark stage high I/O - Azure Databricks | Microsoft Learn

Databricks Code Performance Enhancements & Data Engineering Best Practices

By enabling the below features and applying best practices, you can significantly lower costs, accelerate job execution, and build Databricks pipelines that are both scalable and highly reliable. For more guidance review: Comprehensive Guide to Optimize Data Workloads | Databricks.

Feature / Technique	Purpose / Benefit	How to Use / Enable / Key Notes
Disk Caching	Accelerates repeated reads of Parquet files	Set spark.databricks.io.cache.enabled = true
Dynamic File Pruning (DFP)	Skips irrelevant data files during queries, improves query performance	Enabled by default in Databricks
Low Shuffle Merge	Reduces data rewriting during MERGE operations, less need to recalculate ZORDER	Use Databricks runtime with feature enabled
Adaptive Query Execution (AQE)	Dynamically optimizes query plans based on runtime statistics	Available in Spark 3.0+, enabled by default
Deletion Vectors	Efficient row removal/change without rewriting entire Parquet file	Enable in workspace settings, use with Delta Lake
Materialized Views	Faster BI queries, reduced compute for frequently accessed data	Create in Databricks SQL
Optimize	Compacts Delta Lake files, improves query performance	Run regularly, combine with ZORDER on high-cardinality columns
ZORDER	Physically sorts/co-locates data by chosen columns for faster queries	Use with OPTIMIZE, select columns frequently used in filters/joins
Auto Optimize	Automatically compacts small files during writes	Enable optimizeWrite and autoCompact table properties
Liquid Clustering	Simplifies data layout, replaces partitioning/ZORDER, flexible clustering keys	Recommended for new Delta tables, enables easy redefinition of clustering keys
File Size Tuning	Achieve optimal file size for performance and cost	Set delta.targetFileSize table property
Broadcast Hash Join	Optimizes joins by broadcasting smaller tables	Adjust spark.sql.autoBroadcastJoinThreshold and spark.databricks.adaptive.autoBroadcastJoinThreshold
Shuffle Hash Join	Faster join alternative to sort-merge join	Prefer over sort-merge join when broadcasting isn’t possible, Photon engine can help
Cost-Based Optimizer (CBO)	Improves query plans for complex joins	Enabled by default, collect column/table statistics with ANALYZE TABLE
Data Spilling & Skew	Handles uneven data distribution and excessive shuffle	Use AQE, set spark.sql.shuffle.partitions=auto, optimize partitioning
Data Explosion Management	Controls partition sizes after transformations (e.g., explode, join)	Adjust spark.sql.files.maxPartitionBytes, use repartition() after reads
Delta Merge	Efficient upserts and CDC (Change Data Capture)	Use MERGE operation in Delta Lake, combine with CDC architecture
Data Purging (Vacuum)	Removes stale data files, maintains storage efficiency	Run VACUUM regularly based on transaction frequency

Phase 3: Team Alignment and Next Steps

Implementing Cost Observability and Taking Action

Effective cost management in Databricks goes beyond configuration and code—it requires robust observability, granular tracking, and proactive measures. Below outlines how your teams can achieve this using system tables, tagging, dashboards, and actionable scripts.

Cost Observability with System Tables

Databricks Unity Catalog provides system tables that store operational data for your account. These tables enable historical cost observability and empower FinOps teams to analyze spend independently.

- System Tables Location: Found inside the Unity Catalog under the “system” schema.
- Key Benefits: Structured data for querying, historical analysis, and cost attribution.
- Action: Assign permissions to FinOps teams so they can access and analyze dedicated cost tables.

Enable Tags for Granular Tracking

Tagging is a powerful feature for tracking, reporting, and budgeting at a granular level.

- Classic Compute: Manually add key/value pairs when creating clusters, jobs, SQL Warehouses, or Model Serving endpoints. Use cluster policies to enforce custom tags.

- Serverless Compute: Create budget policies and assign permissions to teams or members for serverless workloads.

- Action: Tag all compute resources to enable detailed cost attribution and reporting.

Track Costs with Dashboards and Alerts

Databricks offers prebuilt dashboards and queries for cost forecasting and usage analysis.

- Dashboards: Visualize spend, usage trends, and forecast future costs.
- Prebuilt Queries: Use top queries with system tables to answer meaningful cost questions.
- Budget Alerts: Set up alerts in the Account Console (Usage > Budget) to receive notifications when spend approaches defined thresholds.

Build Culture of Efficiency

To go beyond technical fixes and build a culture of efficiency, by focusing on the below strategic actions:

- Collaborate with Internal Engineers:
  Spend time with engineering teams to understand workload patterns and optimization opportunities.
- Peer Reviews and Code Audits:
  Conduct regular code review sessions and peer reviews to ensure best practices are followed for Spark jobs, data pipelines, and cluster configurations.
- Create Internal Best Practice Documentation:
  Develop clear guidelines for writing optimized code, managing data, and maintaining clusters. Make these resources easily accessible for all teams.
- Implement Observability Dashboards:
  Use Databricks’ built-in features to create dashboards that track spend, monitor resource utilization, and highlight anomalies.
- Set Alerts and Budgets:
  Configure alerts for long-running workloads and establish budgets using prebuilt Databricks capabilities to prevent cost overruns.

5. Azure Reservations and Azure Savings Plan

When optimizing Databricks costs on Azure, it’s important to understand the two main commitment-based savings options: Azure Reservations and Azure Savings Plans. Both can help you reduce compute costs, but they differ in flexibility and how savings are applied.

- Which Should You Choose?
  - Reservations are ideal if you have stable, predictable Databricks workloads and want maximum savings.
  - Savings Plans are better if you expect your compute needs to change, or if you want a simpler, more flexible way to save across multiple services.

Pro Tip: You can combine both options—use Reservations for your baseline, always-on Databricks clusters, and Savings Plans for bursty, variable, or new workloads.

Summary Table: Action Steps

It’s critical to monitor costs continuously and align your teams with established best practices, while scheduling regular code review sessions to ensure efficiency and consistency.

Area	Best Practice / Action
System Tables	Use for historical cost analysis and attribution
Tagging	Apply to all compute resources for granular tracking
Dashboards	Visualize spend, usage, and forecasts
Alerts	Set budget alerts for proactive cost management
Scripts/Queries	Build custom analysis tools for deep insights
Cluster/Data/Code Review & Align	Regularly review best practices, share findings, and align teams on optimization
Save on your Usage	Consider Azure Reservations and Azure Savings Plan

Enabling the Compliance Security Profile (CSP) for HIPAA on Azure Databricks

Rafia_Aqil — Fri, 02 Oct 2026 13:43:37 GMT

Microsoft Architect's: Aladdin Alchalabi AladdinAlchalabi, Kiran Raja KiranRaja, Peter Lenges PeterLenges, Jessica Reece jareece, Benjamin Coughtry bcoughtry, Anishek Kamal anishekkamal, Tayo Akigbogun takigbogun, Eric Kwashie ekwashie, Peter Lo PeterLo and Rafia Aqil Rafia_Aqil

Peer Reviewed: Ted Kim tedkim and Arvind Periyasamy ArvindPeriyasamy

Purpose and WHY

Azure Databricks has put in place controls to meet the unique compliance needs of highly regulated industries. The requirement for the compliance security profile (CSP) is a joint effort between Microsoft and Databricks for Azure-Databricks workspaces. The value proposition of the compliance security profile is that it provides Customers significantly more hardening and security features.

Mandatory Deadline: The Compliance Security Profile (CSP) becomes mandatory for processing HIPAA, HITRUST, and IRAP regulated data on Azure-Databricks by September 1, 2026.
Key Dates: Enable the Compliance Security Profile and select HIPAA by September 1, 2026. Prepare for Azure Virtual Network encryption enforcement beginning February 1, 2027.

Compliance Responsibility: Enabling CSP supports applicable technical controls but does not, by itself, establish HIPAA compliance. Compliance is a shared responsibility among the customer, Microsoft, and Databricks. Customers must evaluate their administrative, physical, and technical safeguards and confirm that an applicable Microsoft Business Associate Agreement is in place.

Enabling CSP on Workspaces

These requirements are checked and enforced on new workspaces today, with enforcement on existing workspaces expected in the future; where prerequisites are missing, clusters may fail to start.

Prerequisite	Requirement
Costs	There is a 10% cost of the Azure Databricks product spend within each workspace where CSP is enabled. **Review with your account team for any grace period during which the Enhanced Security & Compliance (ESC) add-on is available at no charge. After the grace period ends, a 10% DBU upcharge applies.
Enhanced Security & Compliance add-on	For existing workspaces: From Azure portal, click the Settings > Security & compliance on an existing Azure Databricks workspace: **Review Note #2 below
Azure VNet encryption	Azure Virtual Network encryption must be enabled on the Azure Databricks workspace VNet. Infrastructure as code: Update the encryption block on your VNet resource. In Terraform, that's azurerm_virtual_network. Azure portal: Toggle encryption on the VNet (Overview → Properties → Encryption). Command line: Enable it with the Azure CLI or PowerShell. **Review Note #4 below
Supported VM instance types	Use a VM series that supports VNet encryption and verify compatibility before enabling the profile. ***This does not apply to serverless compute.*

NOTE:

Confirm your workspace is using Premium Pricing tier.
The profile can be enabled when a workspace is created or on an existing workspace, through the Azure portal, the Azure CLI, PowerShell, an ARM template, or Terraform.
Only the Public Preview, Private Preview, and Beta features listed in this section are supported for workspaces with the compliance security profile enabled: Compliance security profile - Azure Databricks | Microsoft Learn
Currently, the compliance security profile checks and enforces only the use of specific VM instance types, not the enablement of Azure Virtual Network encryption. Enforcement of the Azure Virtual Network encryption requirement begins on February 1, 2027, including on workspaces that already have the compliance security profile enabled. This flexibility shall allow customers more time to set up VNET encryption. This has been updated in documentation today (See ‘Important’ box).
Regarding rollback, CSP can be reversed via a support ticket, if no regulated data has been processed on a particular workspace.
Plan for possible effects on cluster startup, networking, feature availability, maintenance operations, and cost.

A closer look at VNet encryption

CSP is enabled per Databricks workspace, but VNet encryption is applied at the VNet level. Enabling it for a Databricks workload therefore affects every resource within that VNet, not just the workspace. A common approach in a hub-and-spoke design is to leave the hub VNet unencrypted and encrypt only the spoke VNet. The hub typically holds shared services such as the DNS resolver, while the spoke hosts the Databricks workspaces that require CSP.

What does it mean for a VNet to be encrypted?
An encrypted VNet is a security measure that protects VM-to-VM traffic. Data is encrypted in transit through a DTLS tunnel. This is platform-level encryption, applied automatically to traffic within your VNet and across peered VNets. It requires no changes to your operating system or applications.

What happens to my VM-to-VM traffic?
Qualifying VM-to-VM traffic is encrypted. Traffic involving unqualified instances simply keeps flowing unencrypted. The only enforcement available today is AllowUnencrypted.

Important clarifications

- Encrypting the VNet does not guarantee all traffic within it will be encrypted. The only traffic that gets encrypted is VM-to-VM traffic where both the source and destination VMs are (1) on a supported SKU and (2) have Accelerated Networking enabled on the network interface.
- Encrypting the VNet does not drop or break traffic from unsupported SKUs. The only supported GA setting today is to allow unencrypted traffic, so non-qualifying traffic is still permitted; it just isn't encrypted. A future DropUnencrypted setting will drop that traffic instead for further hardening. It isn't available yet, and it's currently unknown whether it will become a required setting for CSP.

Review the following recommended steps

The steps below represent a validated implementation pattern. The exact network design can vary by environment, but the same prerequisite, isolation, and end-to-end validation principles should be applied.

Validated implementation step	Recommended approach and expected outcome
Isolated sandbox workspace	Enable CSP first in a representative non-production workspace. This avoids irreversible changes to DEV or production while the network topology, dependencies, VM compatibility, and operational behavior are validated.
Enable CSP and select HIPAA	Enable the Compliance Security Profile and select HIPAA under Settings > Security & compliance before processing PHI after September 1, 2026.
Enable VNet encryption	Enable VNet encryption. **Review Azure Virtual Network encryption limitations: What is Azure Virtual Network encryption? - Azure Virtual Network \| Microsoft Learn
Start a classic cluster	Confirm that a classic cluster starts successfully after CSP and VNet encryption prerequisites are applied. This validates that the selected compute path and VM types remain operational.
Validate storage connectivity	Confirm storage connectivity continue to work.
Confirm rollout readiness	Proceed to DEV and production only after the complete private connectivity path, cluster startup, storage access, DNS resolution, data pipelines, and performance have been validated from end to end.

Things to Review

Enablement is permanent

Enabling the compliance security profile, or adding a compliance standard, is intended to be a permanent change. You cannot remove the profile or an individual standard from a workspace that has ever processed regulated data; to revert, you must delete the workspace and create a new one. Validate the configuration in an isolated, representative non-production workspace before enabling DEV or production.

Inventory and Assessment

- - Identify Regulated Workspaces: Catalogue all existing Azure-Databricks workspaces. Determine which ones currently process, or are planned to process, data subject to HIPAA, HITRUST, or IRAP.

- - Review Data Pipelines: Map out all data ingress and egress points for these identified workspaces, including connections to on-premises data sources, other cloud services, and external APIs. This helps identify potential network impacts.

Verify Prerequisites Before Rollout: Confirm that selected VM instance types support VNet encryption and that every required CSP and networking setting is in place, because missing prerequisites can prevent clusters from starting.

- - Enablement Method: Choose the appropriate tooling for enablement of Azure Portal, Azure CLI, PowerShell, ARM templates, or Terraform to ensure consistency and automation.

Keep sensitive data out of customer-defined fields

You are solely responsible for ensuring that PHI or other sensitive information is never entered into customer-defined input fields. These include workspace names, compute and resource names, tags, job names, job run names, network names, credential names, storage account names, and Git repository IDs or URLs, all of which may be stored, processed, or accessed outside the compliance boundary.

What Changes After Enabling Compliance Security Profile

On CSP-enabled workspaces, Partner-powered AI features are disabled by default and some assistive features such as Genie Code are also disabled; a workspace admin can re-enable them if required. In addition, only the specific preview features listed in the compliance security profile documentation are supported. No other Public Preview, Private Preview, or Beta feature may be used to process regulated data.

Compliance Security Profile (CSP) enhances the security posture of Azure Databricks by enabling a hardened compute image, enhanced security monitoring, and automatic cluster updates. With automatic cluster updates enabled, classic compute resources are periodically updated and may restart during configured maintenance windows, so production schedules should be planned accordingly. Enhanced security monitoring deploys security monitoring agents on supported compute resources and generates logs that security teams can ingest and analyze. When deploying through ARM templates, CSP, enhancedSecurityMonitoring, and automaticClusterUpdate are configurable security and compliance settings that can be specified as part of the workspace deployment.

Why GPU-accelerated compute may be affected

Azure Databricks supports several GPU families, but Azure VNet encryption currently documents a narrower GPU list. Please review the list here: What is Azure Virtual Network encryption? - Azure Virtual Network | Microsoft Learn

What happens if a customer does not enable Azure Databricks CSP?

Azure Databricks has implemented additional controls to support the security and compliance requirements of highly regulated industries. CSP should not be characterized solely as a Microsoft initiative; it is part of the Azure Databricks security and compliance offering delivered by Microsoft and Databricks. CSP provides additional platform hardening and security capabilities, including a CIS Level 1 hardened compute image, automatic cluster updates, enhanced security monitoring, and TLS 1.2 or higher for relevant communications.

Beginning September 1, 2026, CSP is required for Azure Databricks workspaces processing data subject to applicable standards, including HIPAA, HITRUST, and IRAP. If a customer chooses not to enable CSP while processing data subject to one of these standards, the workspace would not meet the documented Azure Databricks configuration requirements for that regulated workload. This does not, by itself, determine the customer's overall legal or regulatory compliance; customers should assess their obligations with their legal, compliance, and audit teams.

References

Compliance security profile: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/security-profile
Configure enhanced security and compliance settings: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/enhanced-security-compliance
HIPAA, Azure Databricks, Microsoft Learn: https://learn.microsoft.com/en-us/azure/databricks/security/privacy/hipaa
What is Azure Virtual Network encryption: https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-encryption-overview
Create a Virtual Network with encryption: https://learn.microsoft.com/en-us/azure/virtual-network/how-to-create-encryption?tabs
Hashicorp azurerm_virtual_network: azurerm_virtual_network | Resources | hashicorp/azurerm | Terraform | Terraform Registry

Meet the IQ's: How Microsoft is Creating Context-Aware AI

Rafia_Aqil — Tue, 22 Sep 2026 20:59:54 GMT

Microsoft Architect's: Allison Rose allisonrose, Lavanya Sreedhar LavanyaSreedhar, Tom Dinh Tom-Dinh, Oviya Soundararajan oviyasound and Rafia Aqil Rafia_Aqil

The AI era demands more than powerful language models. It demands context a deep understanding of what enterprise data means, how it connects, and how AI systems can reason and act on it intelligently. Microsoft has been building the foundational intelligence layer that makes this possible: a family of capabilities collectively known as the IQ Platform.

The Microsoft IQ Platform is not a single product but a set of complementary intelligence layers: Work IQ, Fabric IQ, Foundry IQ and Web IQ each designed to inject rich contextual understanding into a different part of the enterprise technology stack. Together, they represent Microsoft’s strategic vision for how AI can move beyond isolated answers and become a true operating system for organizational intelligence.

This article unpacks each IQ, explains the problems they solve, and explores how they work together to power the next generation of AI-driven enterprise workflows.

How the IQs Work Together?

Work IQ, Fabric IQ, Foundry IQ, and Web IQ are not competing products or overlapping investments. They are complementary intelligence layers designed to operate across different contexts within the enterprise, and they are most powerful when combined.

Work IQ brings the intelligence of Microsoft 365 to every agent and Copilot experience- connecting people, conversations, documents, and organizational signals into a semantic layer that understands how work happens.
Fabric IQ brings the intelligence of enterprise data and business context- teaching AI not just what the data says, but what it means in the language of your business: entities, relationships, rules, and governed actions.
Foundry IQ brings the infrastructure intelligence that enables all of this to scale- eliminating the undifferentiated plumbing of agentic AI and letting teams focus on building the workflows that actually differentiate their business.
Web IQ brings fresh, external web context, helping agents consider current and relevant public information alongside the knowledge and business context inside the organization. Microsoft Web IQ grounds most of the top AI platforms, including Copilot and ChatGPT.

Together, the IQ platform represents Microsoft’s answer to one of the defining challenges of the AI era: not just making AI more capable, but making AI contextually aware-grounded in the real knowledge, relationships, and intent of your organization.

Fabric IQ: Teaching AI the Language of Business

Microsoft Fabric is an end-to-end, unified data analytics platform centered on OneLake- a centralized data lake that stores all analytical and operational business data in open Delta format. Because every Fabric compute experience (Data Engineering, Data Warehouse, Data Factory, Power BI, and Real-Time Intelligence) natively reads from OneLake, organizations gain a single source of truth without copying or duplicating data. OneLake also provides mirroring and shortcut capabilities so existing data can be accessed in place, wherever it lives.

Most organizations have made significant progress consolidating their data. The harder challenge is giving AI- and the people who use it-the ability to reason about that data in business terms, not technical ones. Outside of data professionals, businesses do not talk about tables or schemas. They talk about entities that matter to them.

Fabric organizes data. Fabric IQ teaches AI what that data means.

Three Layers of Business Context

Fabric IQ introduces three intelligence layers that together create a unified, contextually rich environment for enterprise AI:

Unified Data Layer: Delivered through OneLake and the OneLake Catalog, this provides a single source of truth for all structured and unstructured data across the organization.

Business Intelligence Layer: Delivered through Power BI Semantic Models, this layer provides curated measures, hierarchies, dimensions, and trusted KPIs- translating raw data into the analytical language of your business.

Operational Intelligence Layer: This is where Fabric IQ’s most distinctive capability lives: Ontology. An Ontology is a model of your business- a graph of entities (such as Patient, Provider, Product, or Account), the relationships between them, the business rules that govern them, and the actions AI agents can take. It functions as the brain that enables AI to understand business context and act on it in a governed, explainable way.

Together, these three layers create shared context across all business data stored in OneLake-enabling modern businesses, people, and AI to operate as one unified system.

A Real-World Example: Healthcare

Consider a care management executive asking: “Which diabetic patients discharged in the last 30 days are at high risk of readmission because they missed follow-up appointments, had medication adherence issues, and recently visited the Emergency Department?”

Without Fabric IQ, answering this requires analysts to manually join EHR data, appointment systems, pharmacy records, and ED utilization data- writing SQL across multiple datasets and validating business logic with clinicians. It is slow, brittle, and error-prone. Semantic models can curate data for reporting and analysis, but they do not provide enterprise-scale context integration.

With Fabric IQ, an Ontology can be created with entities like Patient, Encounter, Provider, Medication, Diagnosis, Appointment, and Care Plan- each bound to Lakehouse tables, Eventhouse tables, or Materialized Views. Relationships describe how patients connect to their diagnoses, medications, appointments, and treating providers. Business rules enforce data quality, identifying missed follow-ups, recent Emergency visits, and medication gaps.

The result is a shift from siloed analytics to true system-level intelligence- an organization where data, AI, and people operate from a shared understanding of the business.

Foundry IQ: From Infrastructure to Intelligence

Building production-grade AI agents has traditionally meant writing a significant amount of undifferentiated plumbing, custom retrieval pipelines, memory systems, ranking logic, and orchestration code just to enable core RAG and agentic capabilities. While powerful, this approach often leads to complex, hard-to-maintain codebases that distract from the real goal: solving domain-specific problems. With Foundry IQ, Microsoft is fundamentally changing that model by turning these underlying capabilities into managed platform services, allowing teams to shift from building infrastructure to focusing on intelligent workflows.

Foundry IQ acts as part of Microsoft's managed platform, enabling agents to use agentic reasoning to access, process, and act on knowledge from anywhere. It is Microsoft Foundry’s way of turning the undifferentiated plumbing behind a RAG agent, such as retrieval, ranking, citations, memory, and personalization, into managed, server-side services that you provision once and call through clean interfaces.

Foundry IQ allows you to remove the infrastructure you never wanted to own in the first place.

What This Means in Practice

Instead of stitching together retrieval pipelines, embedding logic, ranking strategies, and memory mechanisms, Foundry IQ centralizes these capabilities into a single, opinionated platform layer that agents can directly consume. Developers no longer design and maintain each component individually.

The knowledge base becomes the centerpiece of the workflow. Rather than coordinating multiple services and response handlers, applications make a single call to retrieve grounded context. Vector-semantic-hybrid querying, query planning, semantic ranking, and citation generation are all encapsulated within the provisioned knowledge base-with no retrieval or embedding logic to maintain in the client application.

Memory follows the same pattern of abstraction. Instead of multiple classes and helper utilities to manage storage, user profiles, summarization, and context reconstruction, Foundry IQ replaces this entire layer with a single memory provider backed by a service-managed store with built-in capabilities for chat summarization and user-profile extraction.

A Real-World Example: Clinical Workflows

Consider building an AI-powered clinical workflow application. Previously, features like agent memory, knowledge base retrieval for grounding, and personalization all had to be written as custom logic and wired manually into the application. This resulted in thousands of lines of code, numerous helper functions, and brittle architecture that was difficult to evolve.

With Foundry IQ, that same solution can be reimagined. A single provisioning script now stands up all required services and executes the data-plane steps to create a memory store, build the search index, and provision a Foundry IQ knowledge base for agentic retrieval. Because the top-level router agent carries its own memory, it can directly answer recalled context without relying on confidence thresholds, rule-based branching, or forced workflow paths. Conversation history is handled automatically at ingress- no custom thread management system required.

What remains is only what was always worth building: domain-specific logic. Citation validation against grounded evidence. Hallucination checking using LLM-as-a-judge patterns. Agent revision loops. Everything else- retrieval, ranking, memory, user profiles, conversation management- is provisioned once and consumed as a platform capability.

The result: a dramatically reduced surface area for bugs, significantly less code to maintain, and teams freed to focus entirely on the work that differentiates their product.

Work IQ: Making Microsoft 365 Data Meaningful

For years, Microsoft has given organizations API access to their Microsoft 365 data through the Microsoft Graph- emails, calendar events, OneDrive files, Teams conversations, and more. While valuable, this access essentially treated M365 as a structured database: query an endpoint, retrieve an artifact, parse the metadata. The problem was volume and context. With thousands of signals generated every day across the organization, customers needed a way to extract not just data but meaning.

In the past year, Microsoft introduced a semantic index built on top of that raw M365 data- a layer that understands not just what exists in your ecosystem, but how everything relates to one another. This intelligence layer is Work IQ, and in an increasingly agent-driven world, it fundamentally changes what AI can do for your organization.

In an AI-first world, the advantage is not simply in a model’s ability to reason- it’s in the richness of the context it can reason over.

The Contrast in Action

Consider asking an agent a simple question: “What’s the latest on Customer Contoso?”

With the Microsoft Graph API alone, the agent must stitch together multiple endpoint queries- Teams chats, SharePoint documents, email threads and attempt to piece the results into a coherent answer. It lacks any connective tissue. It doesn’t know what’s relevant, what’s meaningful, or how these isolated data sources relate to each other. The burden of reasoning falls entirely on the agent.

With Work IQ, that same prompt taps into a semantic layer that has already done the connecting. The agent knows Contoso-related details span a specific SharePoint folder, identifies the active Teams channel for progress tracking, and surfaces the key people involved. The response is grounded in a web of contextual relationships not just retrieved data.

Three Core Components

Work IQ is enabled by three powerful components:

Data: Unifies signals from files, emails, meetings, chats, and other M365 business systems to capture how work actually gets done across your organization.

Memory: Enables persistent context about how people and teams work: details inferred from past conversations, explicit memories stored with Copilot, and custom instructions you’ve configured. Each interaction allows Copilot to learn more about your priorities, preferences, and working style.

Inference: Brings together skills, models, and tools to move work forward. It goes beyond understanding your work to deciding what should happen next.

Data captures and indexes your M365 knowledge. Memory builds a personalized understanding of how you work. Inference translates this into action. Think of Work IQ as a specialized brain trained on who you are at work within the full context of what your organization knows.

Web IQ: Bringing Real-World Intelligence into Agent Context

An agent can understand your business and still miss information that matters outside it. For teams researching an account, investigating a supply-chain issue, or comparing products, internal knowledge is only part of the picture. The missing context may be on a company website, in a public announcement, or in newly published guidance.

Web IQ is the part of Microsoft IQ that brings in outside information. It's a set of web search and grounding APIs that give agents current evidence from the web, beyond what the model learned in training and beyond a company's internal systems.

Web IQ gives developers a way to ground AI applications and agents in fresh web content through Microsoft-hosted REST and MCP interfaces. An application can retrieve current, external information when it needs it, rather than relying only on the knowledge captured during model training. Web IQ supplies the grounding content; the application determines how to combine it with other context and use it in a response.

This adds a distinct dimension to the IQ story:

Information beyond the organization. The goal is not to replace trusted internal knowledge, but to help people examine it alongside current, relevant external evidence. Source links keep that evidence available for review, so users can check the information behind an answer.

A Real-World Example: Sales Preparation

Consider a seller preparing for a customer conversation. Work IQ might explain the relationship, outstanding commitments, and business priorities, and Web IQ could contribute public context, such as a recent company announcement. Bringing these inputs together in an agent could help the seller identify a more timely question to ask, while preserving the distinction between internal account information and external evidence.

Get Started

Whether you’re exploring how to ground your AI applications in richer organizational context, looking to reduce the infrastructure burden of building intelligent agents, seeking to make your enterprise data more actionable, or aiming to ground agents in current web information, the Microsoft IQ Platform offers a path forward. We encourage you to explore the Microsoft Fabric documentation, Azure AI Foundry resources, the Microsoft 365 developer platform, and the Web IQ documentation to learn more about how each IQ capability can fit into your architecture. 

Build Microsoft IQ powered agents, this cookbook walks through it step by step: files → Web IQ → Work IQ → Fabric IQ → MCP endpoint: https://lnkd.in/edrjG99F
Select Microsoft IQ in your Copilot agent settings, follow step by step instructions here: Bring your enterprise data to every agent conversation

We’d love to hear how you’re thinking about context-aware AI in your organization. Share your thoughts and questions in the comments below.

Streaming and Batch Data Architectures with Microsoft Fabric to Azure Databricks

Rafia_Aqil — Fri, 02 Oct 2026 02:23:51 GMT

Author's: Aladdin Alchalabi AladdinAlchalabi, Oscar Alvarado oscaralvarado and Rafia Aqil Rafia_Aqil

Note: This article describes a solution idea. Your cloud architect can use this guidance to help visualize the major components for a typical implementation. Use this article as a starting point to design a well-architected solution that aligns with your workload’s specific requirements.

3 Most Common Approaches for integrating into Azure Databricks

As organizations adopt Microsoft Fabric as their unified analytics platform, it has become a leading path for ingesting both streaming and batch data into Azure Databricks. This article covers integration approaches -via Microsoft Fabric- and details the five Fabric-specific paths that connect OneLake/ADLS and Databricks for end-to-end data processing.

Medallion Architecture

The following data flow corresponds to the architecture diagram:

Data is ingested through Microsoft Fabric (via Mirroring, RTI, or Data Factory) lands data into OneLake/ADLS. With the medallion pattern, consisting of Bronze, Silver, and Gold storage layers, organizations have flexible access and extendable data processing:

Bronze – Raw data entry point. Data arrives in its source format and is converted to the open, transactional Delta Lake format.
Silver – Optimized for BI and data science. ETL and stream processing tasks filter, clean, transform, join, and aggregate Bronze data into curated datasets using SQL, Python, R, or Scala.
Gold – Enriched data ready for analytics and reporting. Analysts use Power BI, PySpark, SQL, or Excel for insights and queries.

Fabric Integration Paths

Figure: Microsoft Fabric integration paths into Azure Databricks

Note: This architecture establishes a complete loop-back between Microsoft Fabric and Azure Databricks, enabling Gold layer tables to be seamlessly mirrored back to Microsoft Fabric for dashboarding through Azure Databricks Mirroring.

The following five paths connect Microsoft Fabric to Azure Databricks:

Fabric Mirroring to OneLake – A low-cost, low-latency turnkey solution that creates a replica of data from operational sources (SQL Server, Azure Cosmos DB, Oracle) in OneLake. Handles the initial load and ongoing CDC changes automatically, keeping data continuously up to date.
Fabric RTI to OneLake – Fabric Real-Time Intelligence ingests streaming event data into OneLake with sub-second latency, enabling real-time analytics on live event streams.
Fabric Data Factory to OneLake – Orchestrates ingestion from diverse sources not covered by Mirroring (such as Sybase or REST APIs) and lands data in OneLake, ensuring complete source coverage.
OneLake to Azure Databricks – Unity Catalog connections to OneLake, secured via Managed Identities from Microsoft Entra ID, allow Databricks to query OneLake data items as a native catalog without data duplication.
Fabric Data Factory to Azure Databricks (direct) – Orchestrates ingestion from diverse sources directly into Azure Data Lake Storage (ADLS), where Azure Databricks picks up the data for medallion architecture processing.

Design Considerations

Area	Updated guidance
Direct RTI-to-Databricks integration	There is still no broad GA direct integration where Fabric RTI and Databricks operate as one native real-time runtime. Integration should be positioned through open protocols, Event Hubs/Kafka-style patterns, OneLake, Delta, and federation.
OneLake federation in Azure Databricks	OneLake federation in Azure Databricks is now the key integration story. It allows Databricks Unity Catalog to query Fabric Lakehouse and Warehouse data in OneLake without copying it. Access is read-only and depends on Fabric tenant settings, workspace permissions, and Databricks Unity Catalog setup.
RTI data availability to Databricks	Data ingested through Fabric RTI can be made available to Databricks by landing or exposing the data into OneLake-backed items, especially Lakehouse/Warehouse patterns. Eventhouse data can be made available in OneLake in Delta format through OneLake availability, but Databricks OneLake federation should be validated against the specific Fabric item type and access path.
Existing Databricks customers	Existing Databricks customers do not need to abandon Databricks. They can use Fabric RTI as the event ingestion, real-time detection, operational alerting, and business action layer, while continuing to use Databricks for engineering, ML, advanced analytics, and Unity Catalog-governed access.
Activator and business action	Fabric Activator is the cleanest business-user action layer. It can monitor streaming events and trigger Teams messages, email, Power Automate flows, Fabric pipelines, notebooks, Spark jobs, Dataflows, UDFs, and other downstream actions. This is a strong differentiator because it lets business users act on events without waiting for batch analytics.
Operations Agents	Operations Agents are in preview and should be positioned carefully. They monitor real-time data from Eventhouse or ontology sources, surface insights, recommend actions, and can connect to Activator/Power Automate action paths. They are not simply a pre-ingestion decision engine before data lands anywhere; they work from configured Fabric knowledge/data sources.
Before landing in Lakehouse	For decisioning before Lakehouse persistence, use Eventstream processing and Activator rules on streams. For AI-assisted operational recommendations, use Operations Agents once the relevant data is available in Eventhouse or ontology.

Requirement-Specific Notes

Data Ingestion

Microsoft Fabric Mirroring currently supports SQL Server, Azure Cosmos DB, and Oracle as source systems. For sources not yet supported by Mirroring—such as Sybase or REST APIs—use Fabric Data Factory pipelines to ensure full coverage across all data systems.

Once data is in the landing zone with the correct format, Mirroring’s CDC replication starts automatically and manages the complexity of merging changes (updates, inserts, and deletes) into Delta tables, keeping data in Fabric continuously up to date. Learn more about open mirroring

Storage Format and Time Travel

OneLake supports Delta tables, enabling schema evolution and time travel across all data stored in the lakehouse. Learn more about OneLake and Delta tables

Security

Encryption at rest: OneLake automatically encrypts all data at rest using Microsoft-managed keys, compliant with FIPS 140-2 standards. Learn more

Encryption in transit: All data in transit is encrypted using TLS 1.2 or higher, securing data movement between Fabric, OneLake, and Azure Databricks. Learn more

Data Governance

OneLake can be registered and scanned by Microsoft Purview, enabling cataloging of stored metadata and data quality profiling. This protects sensitive information, including PHI and PII, across ingestion and analytics workflows. Learn more about Purview with Fabric Lakehouse

Operations and Monitoring

Use the Fabric monitor hub to track pipeline health, Spark application performance, and ingestion job status across all Fabric workloads. Learn more about the Fabric monitor hub

Scenario Details

This architecture applies to any organization that needs to unify streaming and batch data at scale. Common characteristics include:

Multiple operational data sources (databases, SaaS applications, event streams)
A requirement to process both real-time and historical data in the same platform
Governance and compliance requirements for sensitive data (PHI, PII, financial records)
Analytics consumers spanning BI (Power BI), data science (Databricks notebooks), and ML workloads

Potential Use Cases

Healthcare and life sciences – PHI/PII protection via Purview; real-time patient telemetry + batch EHR analytics
Financial services – Real-time fraud detection streams + batch regulatory reporting
Retail and e-commerce – Streaming clickstream analytics + batch inventory and supply chain processing
Energy and utilities – IoT sensor telemetry streaming + batch consumption analytics

Next Steps

Designing Reliable Data Platforms: Centralized Failure Logging Framework with Azure Monitor

Sally_Dabbah — Mon, 08 Jun 2026 10:31:33 GMT

Introduction

Modern data platforms are no longer just about moving and transforming data. In production, what really matters is reliability and how quickly you can understand and react when something breaks.

If you’re using Azure Synapse/ADF/Microsoft Fabric, you already have built-in monitoring. You can see pipeline runs, error messages.

But it doesnt show you activity level errors, Pipeline errors works well when you’re debugging a single failure.

But it doesn’t scale.

Once you have dozens of pipelines running across multiple environments, failures become harder to track. You find yourself jumping between pipeline runs, scanning activity outputs, and trying to piece together what actually happened.

And suddenly, simple questions become difficult to answer:

Which datasets are failing most often?
Are failures concentrated in Bronze, Silver, or Gold?
Is this a one-off issue or a recurring pattern?
What changed between yesterday and today?

At that point, pipeline-level monitoring is no longer enough. You need something more structured.

P.S the framework can be implemented across both Synapse and Microsoft Fabric environments with minimal changes.

Why we need a custom logging framework

The core issue is that pipeline failures are treated as runtime events, not as data.

They live inside pipeline output and are tied to a specific run. This makes them hard to query across time, aggregate across pipelines, correlate across environments, or understand which activities failed inside the pipeline or integrate into alerting and dashboards in a consistent way.

Pipeline Failures are visible but activity failures are not , and they’re not operationalized, what’s missing is a central place where all failures are captured in a consistent, structured format, regardless of which pipeline or dataset produced them including Activity level logs.

That’s where a custom logging framework comes in, instead of relying only on built-in monitoring, we introduce a layer that captures failures as structured events, standardizes the payload across pipelines, and sends it to Log Analytics where it can be queried using KQL.

This shifts the model from checking a pipeline when it fails to treating failures as a dataset that can be analyzed, monitored, and improved over time.

Once you make that shift, you can build alerts based on patterns instead of reacting to single failures, track reliability across datasets or domains, and identify recurring issues instead of dealing with incidents one by one.

It also changes who can use the data, visibility is no longer limited to engineers digging into pipeline runs it becomes accessible at the platform level for leads and stakeholders.

This framework doesn’t replace Synapse monitoring. It complements it by adding a proper observability layer on top.

Architecture

When a pipeline fails in Synapse, the failure is intercepted through a dedicated failure path. At this stage, we don’t just log the error as-is we pass it through a custom logging framework that transforms the failure into a structured payload.

This payload includes key context such as pipeline name, activity, environment, dataset, layer (Bronze/Silver/Gold), error details, and correlation identifiers.

The important part here is consistency every pipeline emits the same schema, regardless of its logic.

Once the payload is constructed, it is sent to Azure Monitor using the Logs Ingestion API, this API acts as the entry point into the monitoring system and decouples the pipelines from the underlying storage implementation.

A Data Collection Rule (DCR) sits behind the ingestion layer and defines how incoming data is handled. It acts as a contract for the payload schema and optionally applies transformations before the data is persisted.

Finally, the logs are stored in a custom Log Analytics table, where they become fully query-able using KQL, at this point, failures are no longer tied to a single pipeline run they are part of a centralized dataset that can be analyzed across time, environments, and domains.

Setting up Log Analytics

Before integrating the logging framework with Synapse, we first need to set up the destination for our logs, this includes creating a Log Analytics workspace, defining a custom table, and configuring the ingestion path using a Data Collection Rule (DCR).

The goal is to create a pipeline where structured failure events can be received, validated, and stored in a consistent format.

P.S all steps mentioned in this blog can be automated with ARM templates.

1. Create a Log Analytics workspace

Start by creating a Log Analytics workspace. This will act as the central store for all failure logs across your data platform.

In the Azure Portal:

Navigate to Azure Monitor → Log Analytics workspaces
Create a new workspace in your target subscription and region
Choose a meaningful name (for example: log-analytics-data-domain)

This workspace becomes the single place where all pipeline failures will be collected and queried.

2. Create a custom table for pipeline failures

Instead of relying on generic tables, we define a dedicated custom table to store pipeline failure events.

From the Log Analytics workspace:

Go to Tables → Create → Custom table (DCR-based)
Define a table name such as:
DataDomain_SynapsePipelineErrors_CL [it has to end with CL suffix]

At this stage, you’ll define the schema that represents your logging payload. Typical fields include:

TimeGenerated
PipelineName
PipelineRunId
ActivityName
ActivityType
Status
ErrorCode
ErrorMessage
Severity
Environment
Layer
DatasetName
PartitionDate
WorkspaceName
CorrelationId

The key here is consistency, this schema will be reused across all pipelines, so take the time to define it properly.

3. Create a Data Collection Rule (DCR)

The Data Collection Rule defines how incoming data is ingested into Log Analytics. It acts as both a schema contract and a routing mechanism.

In Azure Portal:

Go to Azure Monitor → Data Collection Rules
Create a new DCR and associate it with your Log Analytics workspace

Within the DCR:

Define a custom stream (for example: DataDomain_SynapsePipelineErrors_CL)
Map this stream to your custom table
Optionally define transformations using KQL (for example, renaming fields or enforcing types)

This step is critical because it decouples your pipelines from the storage layer. If the schema evolves later, you can adjust it here without changing pipeline logic.

4. Configure the Logs Ingestion endpoint

Once the DCR is created, Azure generates an ingestion endpoint that will be used by your pipelines.

The endpoint follows this pattern:

https://<dce>.<region>.ingest.monitor.azure.com/dataCollectionRules/<dcrId>/streams/<streamName>?api-version=2023-01-01

This endpoint is what your Synapse pipeline will call using a Web Activity.

At this point, you should also:

Enable Managed Identity authentication
Grant the Synapse workspace permission to send data to the DCR

This ensures secure ingestion without using secrets.

5.RBAC for Managed Identity

The Managed Identity used by Synapse or Microsoft Fabric must have the following Azure RBAC role:

Monitoring Metrics Publisher

This role allows the identity to send data through the Azure Monitor Logs Ingestion API.

The role should be assigned on the:

Data Collection Rule (DCR) resource

In Azure Portal:

Data Collection Rule (DCR) → Access Control (IAM) → Add Role Assignment → Monitoring Metrics Publisher → Select Synapse/Fabric Managed Identity

Without this role assignment, requests to the Logs Ingestion API will fail with authorization errors such as HTTP 403.

6. Validate the setup

Before integrating with pipelines, it’s a good idea to validate that ingestion works.

You can send a test payload (via Postman or a simple script) and then query your table in Log Analytics:

DataDomain_SynapsePipelineErrors_CL | take 10

If everything is configured correctly, you should see your test records appear.

Integrating with Synapse pipelines

Now that the ingestion layer is ready, the next step is connecting Synapse pipelines, so failures are logged automatically instead of sending manual test payloads.
The idea is simple:
whenever a pipeline activity fails, we capture the failure details, transform them into a structured payload, and send them directly to the Logs Ingestion API, this turns pipeline failures into centralized operational events.

1. Add a failure handling path

Inside your Synapse pipeline, add an On Failure dependency from the activities you want to monitor.

Typically, this includes critical activities such as:

Copy Activities
Notebook executions
Stored Procedures
Data Flows
Web Activities

Instead of allowing the pipeline to fail silently, the failure path redirects execution into a dedicated logging step , in most production environments, this is implemented as a reusable child pipeline such as:

pipeline name : Customized Logs API

This keeps logging logic centralized and avoids duplicating the same implementation across dozens of pipelines.

2. Pass failure metadata as parameters

The logging pipeline should receive operational context from the parent pipeline.

Typical parameters include:

Pipeline name
Pipeline run ID
Activity name
Activity type
Error code
Error message
Environment
Layer (Bronze/Silver/Gold)
Dataset name
Severity
Correlation ID

This metadata becomes the foundation of the structured logging payload.

The more operational context you capture here, the easier troubleshooting becomes later.

3. Construct the logging payload

Inside the logging pipeline [Customized Logs API], use a dynamic content expression to construct a JSON payload matching the Log Analytics schema.

Example payload:

concat( '[{"TimeGenerated":"', utcNow(), '","PipelineName":"POC_Test"', ',"PipelineRunId":"', pipeline().RunId, '","PipelineStatus":"Failed"', ',"ActivityName":"TestActivity"', ',"ActivityType":"Web"', ',"ActivityStatus":"Failed"', ',"ErrorCode":"TEST"', ',"ErrorMessage":"POC test"', ',"Severity":"Warning"', ',"Environment":"Test"', ',"Layer":"Bronze"', ',"ExecutionStage":"POC"', ',"DatasetName":"TestDataset"', ',"PartitionDate":"', utcNow(), '","WorkspaceName":"', pipeline().DataFactory, '","TriggerName":"Manual"', ',"TriggerTimeUtc":"', utcNow(), '","DurationMs":1000', ',"RetryCount":0', ',"Compute":"Synapse"', ',"CorrelationId":"', pipeline().RunId, '","Payload":{"source":"test","target":"loganalytics"}}]' )

The important part is schema consistency.

Every pipeline should emit the same payload structure regardless of which activity failed.

This makes downstream querying and dashboarding significantly easier.

4. Send logs using a Web Activity

After constructing the payload, use a Web Activity to send the data to the Logs Ingestion API endpoint configured earlier.

Typical configuration:

URL:

https://<data-collection-endpoint>.<region>.ingest.monitor.azure.com/dataCollectionRules/<dcr-id>/streams/<stream-name>?api-version=2023-01-01

Method
POST

Authentication
Managed Identity

Resource
https://monitor.azure.com

Headers

{ "Content-Type": "application/json" }

Body
Dynamic JSON payload generated in the previous step.

I highly recommend using Managed Identity avoids storing secrets or credentials inside Synapse pipelines and keeps authentication fully managed by Azure.

5. Validate end-to-end ingestion

Once the pipeline is connected, trigger a controlled failure and verify that the event appears in Log Analytics.

Run:

DataDomain_SynapsePipelineErrors_CL | sort by TimeGenerated desc | take 20

You should now see real pipeline failures arriving automatically from Synapse.

At this point, the framework becomes fully operational.

Failures are no longer isolated runtime events buried inside activity outputs they are centralized, queryable operational records that can be analyzed across the entire platform.

Future Steps

Now that we have a centralized logging framework in place, we can take observability one step further by building operational dashboards in Power BI or Microsoft Fabric to analyze reliability trends across the entire data platform. Instead of reacting to isolated pipeline failures, we can aggregate logs across pipelines, datasets, environments, and medallion layers to identify what is actually causing instability over time. This allows engineering teams to detect recurring error patterns, identify unstable datasets, measure platform reliability, analyze failure spikes after deployments, and understand where operational bottlenecks are concentrated. By transforming pipeline failures into structured operational telemetry, the framework evolves beyond simple logging into a true observability platform that supports proactive reliability engineering, helping teams move from reactive firefighting to data-driven operational improvements based on measurable reliability KPIs such as failure trends, MTTR, SLA compliance, severity distribution, and pipeline health scoring.

Building AI apps and agents with Azure Databricks, Copilot Studio, and GitHub Copilot

Jason_Pereira — Tue, 02 Jun 2026 16:15:00 GMT

A workspace-wide Genie MCP endpoint for Copilot Studio

Genie is Azure Databricks’ AI agent that lets any employee chat with their data and get trusted answers instantly. Genie Spaces are curated, business‑domain workspaces for teams to find strategic insights for their targeted use cases. Until now, connecting Azure Databricks Genie to Microsoft Copilot Studio meant adding each Genie Space as a separate tool. This works and adds value for customers wanting to integrate a specific Genie Space with Copilot Studio, but the per-space MCP server added overhead when trying to connect multiple Genie spaces to one Copilot Studio agent. The workspace-wide MCP endpoint changes that. One endpoint per workspace gives a Copilot Studio agent access to every connected Genie space and Unity Catalog dataset, and the curated context inside each Genie space stays in place.

Key capabilities:

Natural-language access across the workspace. Copilot Studio agents can route questions across every connected Genie Space and Unity Catalog dataset without losing the curation that keeps answers accurate.
Unity Catalog governance. Access controls are enforced at query time, so existing data permissions extend to every agent built in Copilot Studio.
Beyond a single domain. Move from a finance agent or a supply chain agent to a workspace-aware agent that follows users wherever the data lives.

Lakebase branching with GitHub Copilot agent mode

Production AI agents fail on real-data edge cases that synthetic or mocked environments do not catch. But giving a developer direct production access to investigate is not a realistic option in most enterprises. Lakebase branching, now integrated with GitHub Copilot agent mode, gives you a way to debug against real data without ever connecting to the production database.

Key capabilities:

Copy-on-write branching. Create a full-fidelity branch of a Lakebase production database in seconds. No data is moved and no production records are altered.
Native GitHub Copilot agent debugging. Point GitHub Copilot agent mode at the branch endpoint to reproduce, root-cause, and resolve data-dependent issues with AI assistance.
Azure-native end-to-end workflow. The full loop runs across GitHub, Azure Databricks, and Lakebase. No third-party tools or custom infrastructure required.
Compliance built in. Fixes ship through the standard Git-based deployment and compliance workflows already in place, so debug cycles compress from hours to minutes.

What this unlocks for AI agent teams

Together, the two capabilities cover both halves of the agent lifecycle on Azure:

Author Copilot Studio agents that reason over an entire Azure Databricks workspace through one MCP connection.
Debug production AI agents against real Lakebase data using GitHub Copilot agent mode, reducing production data risk.
Keep Unity Catalog governance and existing compliance controls in place from authoring through deployment.
Standardize the data, agent, and developer toolchain on GitHub, Azure Databricks, and Microsoft 365.

Get started

Both features are available in public preview on June 2, 2026, directly in Azure Databricks workspaces.

Secure Medallion Architecture Pattern on Azure Databricks (Part II)

mscagliola — Fri, 15 May 2026 08:48:58 GMT

Disclaimer: The views in this article are my own and do not represent Microsoft or Databricks.

This article is part of a series focused on deploying a secure Medallion Architecture. The series follows a top-down approach , beginning with a high-level architectural perspective and gradually drilling down into implementation details using repeatable, code.

In this part we will discuss the implementation of the pattern using GitHub Copilot

If you have missed, please read first the first part of this blog series. It can be found at: Secure Medallion Architecture Pattern on Azure Databricks (Part I).

I waited a while before publishing this article. Partly due to other priorities, but also because I wanted to experiment with deploying infrastructure and data pipelines using agents. At that point, I was looking to leverage agents with a spec-driven approach, and through using GitHub Copilot, I learned what skills are and how I can use them to achieve my scope.

In this blog I'll share what I learned using GitHub Copilot for spec-driven development.

I'll use the content from my previous article, Secure Medallion Architecture Pattern on Azure Databricks (Part I) , as a technical specification to extract implementation details and generate two outputs:

Terraform code for infrastructure, platform configuration, and deployment
Databricks Declarative Automation Bundles for jobs, pipelines, and other deployment-ready workload resources

I've tried not to overfit the prompts within the skills I've developed, so they remain portable to other technical articles, not just the one mentioned in this blog.

Separate the platform from the workload

When I started the design, I decided to modularise the automation scripts by separating the platform from the actual data platform workloads. I assigned networking, storage, identities, secret scopes, and workspace configuration to Terraform, while Databricks notebook runs, job clusters, pipelines, and environment-specific deployments were developed within Databricks Declarative Automation Bundles (formerly known as Databricks Asset Bundles).

That may sound obvious, but it's exactly where generated code often goes wrong. Without explicit instructions, AI tools tend to blur these boundaries and produce one oversized block of configuration. That's why my Copilot skill needs to enforce a clear contract by:

Infer the architecture from the article
Identify what is explicit and what is assumed
Emit Terraform only for infrastructure concerns
Emit bundle files only for workload concerns
Leave placeholders for anything the article does not specify

That last point is critical. A blog post or low-level technical specification is not a source of truth for account IDs, hostnames, catalog names, secret values, or subnet IDs. Good automation should never fabricate those values. Instead, I decided to produce a starter implementation with TODO markers wherever environment-specific values are required.

Skills are a great way to get more consistent, repeatable output across runs, so I decided to use them for this project. I could have used one of the tools listed in the table below, but I chose to go my own way, into developing a Spec-Driven Development (SDD) framework which I hope it will carryon improve with time.

Tool	Creator	Type	Link	Description
GitHub Spec Kit	GitHub	Open source	github/spec-kit	Turns feature ideas into specs, plans, and task lists before any code is written. Works with multiple AI coding agents. Specification first, code as generated output.
BMAD Method	BMad Code LLC	Open source	bmad-code-org/BMAD-METHOD	An AI-driven agile framework with specialised agents covering the full lifecycle from ideation to deployment. Scale-adaptive — adjusts planning depth from a bug fix to an enterprise system.
OpenSpec	Fission AI	Open source	Fission-AI/OpenSpec	Lightweight spec layer that sits above your existing AI tools. Each change gets a proposal, specs, design, and task list. No rigid phase gates, no IDE lock-in.

What are skills, and why are they a good fit?

Skills are essentially reusable prompt modules that aim to force LLMs to produce repeatable answers. Within a skill, I define the behavior and then attach supporting resources or scripts so Copilot can perform the task consistently. That means a skill can do more than just "write some code."

A skill can define a repeatable workflow like this:

Fetch the blog URL
Extract headings, paragraphs, and code snippets
Normalize the article into a lightweight implementation spec
Decide what belongs in Terraform
Decide what belongs in the Databricks bundle
Generate files in a predictable project structure
Produce a TODO.md file for unresolved values

This approach turns Copilot from a generic assistant into a specialized code-conversion tool.

However, there are some constraints I had to be mindful of when developing skills:

Context window limits. The model has limited space to read instructions, process input, and generate output. Long prompts can cause files to be cut off or steps to be skipped.
Non-determinism. Output may vary between runs, even with strict instructions. I always lint, validate, and review the diff before committing.
Boundary leakage. Models may invent plausible but incorrect values. The TODO.md pattern must be enforced as a rule, not a suggestion.
Model and tool drift. Copilot's model and tool surface change over time. I use example inputs and outputs as repeatable sanity checks.
Maintainability. A skill is code-as-prompt and will age with the platforms it targets. I keep skills narrowly scoped so they stay easy to update.

I'll explain the TODO.md file in more detail later in this post.

The GitHub repo

The repository can be found at the link MarcoScagliola/CopilotBlogToCode

Below you will find a function I have added that, when invoked, deletes all the files produced by the skills, so you can test the repo from a clean state.

python .github/skills/blog-to-databricks-iac/scripts/reset_generated.py --force;

If you want to tried it out, please clone and try it on your copy. In GitHub Copilot, I usually keep:

Model as Auto
Foer the configure tools I keep just the built-in tools selected.

Below you can find the prompt that I use to run the skills and have the blog analysed.

Use the blog-to-databricks-iac skill on this article:
https://techcommunity.microsoft.com/blog/analyticsonazure/secure-medallion-architecture-pattern-on-azure-databricks-part-i/4459268

Inputs:

workload: blg

environment: dev

azure_region: uksouth

github_environment:

To make this more repeatable and less manual, I've added a prompt file at run-blogToDatabricksIac-selected-tools.prompt.md, which can be run directly from VS Code by opening the file and clicking the run button at the top. Feel free to experiment with it and let me know what you think.

Further instructions on how to use the repo are available READ_FIRST.md.

Following you will find the exact repository setup I used for this workflow, starting with my initial configuration and ending with the final directory structure and files.

1. Create a new GitHub repository and clone it locally

I started by creating a new repository on GitHub, then cloned it to my local machine so I could add the Copilot skill, Terraform scaffolding, and Databricks bundle files in a centralized location.

git clone https://github.com/YOUR-ORG/blog-to-databricks-iac.git cd blog-to-databricks-iac

This approach keeps the workflow organised from the start: the repository exists on GitHub first, and the local clone becomes the working directory for all subsequent setup steps.

2. Create the GitHub skill folder structure (first iteration)

GitHub Copilot skills are file-based and centered on a SKILL.md file inside a skill folder. GitHub's current pattern places these under .github/skills/.

I used the script below to create the folder hierarchy for my initial integration.

mkdir -p .github/skills/blog-to-databricks-iac/scripts mkdir -p .github/skills/blog-to-databricks-iac/templates mkdir -p infra/terraform mkdir -p databricks-bundle/resources mkdir -p databricks-bundle/src

This script generates the structure depicted below.

3. Add the main skill definition

Next, I created the SKILL.md file at .github/skills/blog-to-databricks-iac/. The orchestrator decides what happens and in what order, while each specialist decides what its own file should contain (as an example the Terraform specialist owns the Terraform, the bundle specialist owns the bundle, and so on).

In practice, SKILL.md turns Copilot from a general assistant into a domain-specific generator for this repo. GitHub documents this SKILL.md-based structure as the foundation of agent skills. My first iteration of .github/skills/blog-to-databricks-iac/SKILL.md> was very simple and can be found here.

4. Add a script to fetch and normalize the blog article

Next, I created a Python script that the main orchestrator SKILL.md invokes to read the blog article. This script is stored at .github/skills/blog-to-databricks-iac/scripts/ and named fetch_blog.py. Within SKILL.md, the script is invoked as shown below.

### 1. Fetch article ```bash python .github/skills/blog-to-databricks-iac/scripts/fetch_blog.py "<url>" ``` If fetch fails, stop and return the fetch error output. Do not retry; surface the error to the user and wait for guidance.</url>

The script validates the URL, fetches the HTML with a 30-second timeout, and uses a spoofed Mozilla User-Agent to avoid being blocked by CDNs (Content Delivery Networks). It reads through the HTML one tag at a time, flagging when it enters relevant sections like paragraphs, headings, or code blocks, and buffering text until the tag closes. Before storing anything, it cleans the text by decoding HTML objects, collapsing whitespace, and trimming edges.

As it parses, the script also scans for cloud platform keywords: AWS, S3, Azure, ADLS, GCP, Google Cloud. The first match wins; if none are found, it returns unknown. This is a quick heuristic, not authoritative.

Finally, it outputs clean JSON with the extracted data: title, headings, paragraphs, code blocks, and cloud hint, capped at reasonable sizes to keep the output manageable. If anything goes wrong, such as a network error, timeout, bad HTML, or empty content, the script exits cleanly with a structured error message, making it easy to integrate into larger workflows without surprises.

The Python scrip can be found here.

5. The output and output contract

Now I needed to think about the output I wanted GitHub Copilot to deliver through the skills. To reiterate, I needed the following:

File Name	Description
README.md	This is the operator-facing runbook that turns the generated artifacts into a working deployment. It contains no unresolved placeholders and no embedded credentials. The header summarizes the architecture and links back to the source blog. A prerequisites section lists required Azure access, Entra permissions, GitHub Environment setup, and local CLI versions. It includes tables of always-required GitHub secrets and variables, plus conditional ones based on deployment mode. Step-by-step numbered sections walk through bootstrapping the deployment principal and populating the GitHub Environment. Workflow blocks describe each Terraform validation, infrastructure deployment, and DAB deployment step, including file paths, triggers, and outputs. A commands section lists the exact Terraform and Databricks bundle sequences to run. Finally, assumption notes point the operator to `TODO.md` and `SPEC.md` for context.
TODO.md	The operator's checklist of remaining tasks. It uses a strict five-section format (Heading, What this is, Why deferred, Source, Resolution, Done looks like) with no commands or code, only concepts and decisions. Each section captures a different layer of post-deployment work, pre-deployment tasks like RBAC roles and GitHub secrets, deployment-time inputs like region and environment, post-infrastructure setup like Key Vault secrets and external locations, post-DAB work like Unity Catalog grants and job schedules, and architectural choices the orchestrator couldn't make (network posture, schemas, partitioning). Every entry comes from something the article left unstated, plus the universal post-deploy work for any Databricks deployment. The operator works through `TODO.md` sequentially, resolving each item before the system is production-ready.
SPEC.md	The structured, source-faithful read of the blog article, organized by checklist. Every item is marked as a stated value, inferred from code or diagrams, or "not stated in article." It includes architecture details, Azure services configuration, Databricks setup, data model, security and identity requirements, and observations. `SPEC.md` is the single source of truth that Terraform and DAB generators read from, `TODO.md` is populated from every "not stated" entry, and `README.md` references it for assumptions. This ensures the deployment is built on documented decisions, not hidden assumptions.

Together, these files create a clear boundary:

SPEC.md answers what the blog says,
TODO.md captures what's missing or must be decided,
README.md tells you exactly how to deploy.

This split is enforced by validation rules that fail if any content duplicates across the three files.

To make these files as repeatable as possible, I needed two things:

Two templates, one for README.md and one for TODO.md, that the orchestrator fills in from SPEC.md at generation time.
A broader delivery contract, output-contract.md, which lists the five files the orchestrator must produce. README.md and TODO.md are two of those five, and the templates are how they get produced.

The output-contract.md file defines a strict, ordered format that the agent must follow when transforming a blog article about Databricks-on-Azure architecture into a runnable repository.

The first commit was deliberately minimal, as you can see from the file available here. No leaf-skill routing, no repo-context.md, no GitHub Actions workflows, no validation rules, no entry-field templates for TODO.md. That commit's single job was to lock down the shape of the output: what gets produced and in what order. Every commit since has refined how to produce that shape without changing what gets produced.

Putting the contract in the very first commit gave every later change a fixed reference point. Every leaf skill, generator script, and validation rule I've added since has fit into one of its five sections. The pipeline has changed; the deliverables haven't.

The structure of the GitHub repo at commit 17ab443 can be see in the pictorial below.

6. The `README.md` and `TODO.md` templates

After iteratively working on the orchestrator, a clear pattern emerged, the code-generation paths were kind of stable, but the documentation outputs weren't. Every run produced README.md and TODO.md from scratch in free-form Markdown. Across runs, the same content kept drifting.

Section ordering changed between runs and the explanation of GitHub Environments was rewritten with subtle wording differences. RBAC roles appeared sometimes as lists, sometimes in prose, sometimes split across sections. Universal post-deploy actions (create the secret scope, populate the vault, set up Unity Catalog grants) were re-derived every time, occasionally with steps missing. The root cause was that the orchestrator was treating durable, universal content as if it were per-run content.

So I've decided to add two templates: README.md.template and TODO.md.template. Templates separate universal content (RBAC, TODO sections, GitHub setup) in the template from per-workload content (catalog names, credentials) substituted from SPEC.md. This delivers consistency across runs. The README and TODO are structurally identical, so readers can navigate them intuitively. Universal content is correct by construction; I write it once, review carefully, and every run inherits that quality.

Validation also becomes more precise, and the agent's job shrinks from open-ended writing to mechanical substitution, which is easier to validate and maintain. Templates introduce clear vocabulary: {placeholder} is filled by the orchestrator at generation time, by the deployer at run time. Finally, templates enforce traceability: every "not stated in article" entry in SPEC.md automatically becomes a TODO entry via the from SPEC.md slot, making this an automatically-enforced rule.

I'm invoking the templates in the orchestrator as shown below. The Git commit with this code can be found at this link.

### 3.1 Generate README from template Load the template: `.github/skills/blog-to-databricks-iac/templates/README.md.template` ### 3.2 Generate TODO from template Load the template: `.github/skills/blog-to-databricks-iac/templates/TODO.md.template`

7. The output of the `fetch_blog.py` file and the interaction with the orchestrator

When the orchestrator invokes fetch_blog.py, the script produces a JSON output and passes it back to the orchestrator. The orchestrator then reads the JSON document into its working context and maps each field onto an analysis checklist. The title and meta description establish the article identity and scope. Headings with their levels reveal the structure, helping the agent locate sections about architecture, security, data flow, and naming. Paragraphs provide evidence for stated values like regions, resource types, and RBAC models. Code blocks become the source of inferred values. As an example, a Terraform snippet might reveal SKU choices or naming patterns not mentioned in the text.

These inferred values get tagged "inferred from code snippet" when recorded. The cloud hint acts as a sanity check that the article actually describes an Azure architecture.

For every checklist item, the agent records either an extracted value or the literal string "not stated in article". This becomes SPEC.md, the single source of truth for everything downstream. SPEC.md drives every subsequent step. Steps 3 through 7 (the Terraform module, workflows, and Databricks bundle generators) read architectural decisions from it. Step 8 then produces TODO.md by converting every "not stated in article" entry into a TODO item the operator must resolve before deployment.

What I find worth pointing out is how little the output contract has actually moved since that very first commit. The implementation underneath has changed completely. Leaf skills emerged, generator scripts came in, validation rules got added, a soft-delete state machine showed up to handle Key Vault recovery. None of those existed at the start. But what the orchestrator delivers, the list of files it puts on disk, has stayed exactly the same.

We have a much larger SKILL.md today that still mirrors the initial five-item output list. The contract itself has changed by exactly one line: the addition of "Design of the architecture" to section 5.

SPEC.md: the structured, source-faithful read of the article, organised by the analysis checklist ( link )
TODO.md: the operator's checklist of everything the article didn't specify, plus the universal post-deploy actions ( link )
Terraform code under infra/terraform/: the platform layer with networking, storage, identities, Key Vault, workspace ( link )
Databricks Asset Bundle under databricks-bundle/: the workload layer with jobs, entry points, environment configuration ( link )
README.md: the operator runbook, with the architecture design diagram embedded ( link )

If the JSON contains an error, the orchestrator stops immediately. Per the skill rule "If fetch fails, stop and return the fetch error output. Do not retry," the error surfaces to the user rather than propagating downstream.

So the script's output is the raw evidence pack: title, structure, prose, code, cloud hint. The agent uses it to fill the architecture spec, which parameterises every generated artifact.

At this point the fetch_blog.py output is sent to Step 2 of the orchestrator, as shown in the code snippet below.

### 2. Analyse article Analyse the fetched article against the structured checklist in `.github/skills/blog-to-databricks-iac/references/blog-analysis-checklist.md`. The analysis covers the article text, diagrams, screenshots, and code snippets.

And, much later in the orchestrator, Step 8 closes the loop by turning everything that's been recorded into the two operator-facing documents:

### 8. Generate README and TODO from templates Use the templates in `.github/skills/blog-to-databricks-iac/templates/`: - `README.md.template` -> `README.md` - `TODO.md.template` -> `TODO.md`

8. How this actually came together

What I've described so far is how the orchestrator works currently. The reality of building it was much cumbersome , but also fun.

I got from the first version to the current one by iterating. Rerun the orchestrator, find the defect, identify the rule that would have caught it, add the rule to the skill that owns the artifact, rerun.

The reason I'm calling this out now, before walking through the rest of the pipeline, is that everything from this point on is a story about a specific lesson learned that way. The leaf skills exist because a single SKILL.md got too dense. The restricted-tenant guardrails exist because the deployment failed against a tenant that couldn't read Microsoft Graph. The validation harness exists because prose rules weren't catching the regressions that mattered. The soft-delete state machine exists because the same vault name kept colliding with a previous deploy. None of these rules were present from day-one.

So in the next sections I'll walk through how the pipeline actually matured: how the single skill split into a graph, what the inner regenerate-fix loop felt like in practice, the day the project pivoted to support restricted tenants, the bugs that became rules, and the Key Vault soft-delete state machine that closed the project out.

9. From a single skill to a skill graph

When I started, everything lived inside a single SKILL.md. It was simpler that way, and to be honest, at that point I didn't yet know which rules would actually matter. But as I kept rerunning the orchestrator on the article, a pattern emerged. Each rerun produced something that broke in a slightly different way, and the fix always belonged to a very specific concern: Terraform authoring, bundle structure, workflow generation, or the orchestration logic itself. Stuffing the rules for all of them into one file was making the orchestrator unreadable and, worse, was silently dropping rules when the context window got tight.

So I split it. The orchestrator stayed at the top, kept routing the work and validating the result, and each concern got promoted to its own leaf skill. The Databricks bundle skill itself ended up needing one more split a few days later, it had got too dense, so I broke it into two leaves:

databricks-yml-authoring ( link )
Python-entrypoints ( link )

The diagram below shows the shape the repo has today.

The orchestrator now does almost no authoring. It owns the sequence of steps, the contract, and the validation gates, while everything else is delegated. This was the single biggest readability win. I wish I'd done it earlier.

The REPO_CONTEXT.md is one extra node in that diagram that I want to call out But I'll come back to later in section 12.

10. The inner loop: rerun, fail, fix the skill

If I had to describe the middle of this project in one sentence, it would be: every commit was a regeneration. I'd run the orchestrator end-to-end against the article, inspect the generated Terraform, the bundle, the workflows. I'd find a defect, identify the rule that would have prevented it, add that rule to the skill that owns the artifact, then rerun. As shown in the image below.

This loop is what I think people miss when they treat AI-generated infrastructure code as a one-shot. The first run is never the deliverable. The deliverable is the skill that produces good runs. The generated files are disposable and can always be reproduced. The skill is what carries the knowledge forward.

I had to actively resist the temptation to fix bugs in the generated code directly. Patching infra/terraform/main.tf by hand fixes today's run but not tomorrow's, because the rule that would prevent the bug doesn't exist anywhere. So I made it a discipline: never edit the output, always edit the skill, then regenerate.

11. Restricted-tenant compatibility

The bug was simple to describe and brutal to fix: the deployment principal in the target tenant couldn't read Microsoft Graph. Any Terraform data source that resolved an Entra name to an object ID at plan time (e.g., azuread_user, azuread_group, azuread_service_principal) blew up at terraform plan.

My first instinct was to think "I just give the principal Graph permissions". But in a lot of real environments this is not possible. The principal that runs your IaC is governed by a security team, the team has a policy, and the policy says no Graph reads.

The pivot was getting the skill to produce Terraform that never reads Graph. Object IDs are inputs, not lookups. They come in as trusted secrets, the workflow exports them as TF_VAR_*, and Terraform consumes them as variables. No data "azuread_*" block is allowed in the generated code, ever.

I thought this was a simple fix. It wasn't. It cascaded into about six other things:

App Registration vs Service Principal object IDs. The workflow was being given the wrong one. Role assignments need the Enterprise Application (Service Principal) object ID, not the App Registration object ID. The two are different objects in Entra with different IDs. I encoded the distinction in the skill as *_SP_OBJECT_ID (the Service Principal) versus *_CLIENT_ID (the App Registration's application ID). Naming carries the meaning now, so the wrong value is hard to pass.
Single-principal mapping. In some tenants you only have one principal and it has to play both deployment and runtime roles. The skill grew a layer_sp_mode = existing input so the generator stops trying to create a new Service Principal and reuses the deployment one instead.
Key Vault access policies, gone. Access policies were Graph-touching, and not all tenants support them anyway. The skill switched fully to RBAC role assignments (Key Vault Secrets User, and so on). A few cascading bugs followed, but this was the right call.

It took some time to harden the Terraform skill against everything the restricted tenant was throwing back. Each iterations had the same shape, each orchestrator runs, hits a fresh provider error, I add the rule, run again, hit the next one. The commit subjects from that run are basically a transcript of the conversation I was having with the platform.

12. The bugs that became rules

There are three bugs that I believe are worth telling the story of, because they each illustrate a slightly different lesson.

The HCL trim() arity bug. The generator emitted trim(var.something) in a validation block. HCL's trim() takes two arguments, not one. The function I actually wanted was trimspace(). This is the kind of bug that any human would catch in a code review in two seconds, and which the model produced confidently because the shape of the call looked right. I added the rule to the Terraform skill ("for whitespace trimming use trimspace, never trim") and the bug never came back.

Lesson: even for trivial syntactic mistakes, the fix belongs in the skill.

The variable shadowing bug. The deploy workflow had a job-level env: block that set TF_VAR_key_vault_recover_soft_deleted to a static value. A detection step earlier in the workflow was supposed to compute the right value at runtime and write it via $GITHUB_ENV. The problem is that GitHub Actions resolves job-level environment variables before $GITHUB_ENV writes take effect, so the static value always won and the dynamic one was silently ignored.

The fix was to never set the recovery flag at job level. It must be written in the detection step, on every code path, including the trivial "no recovery needed" path.

Lesson: state must be explicit, not inherited. If a flag has three possible meanings, three code paths must each write it.

The hardcoded -platform suffix. The workflow had a shell-side suffix that someone (let's be honest, the model) had invented to make the resource group name "look right". When recovery logic started running and the workflow looked for the canonical resource group, it looked for -platform instead of whatever the Terraform locals.tf actually emitted. The result was that the recovery handler was happily reaching past the real resource group and into a different one. I made it a rule in the orchestrator: workflow-invented suffixes are not permitted. Naming is owned by Terraform's locals.tf.

There are seventeen more defects in the catalogue, and the pattern is the same in every case. The bug surfaces, the rule gets written, the rule lives in the skill that owns the affected artifact.

There is no implementation-learnings.md in the repo. There used to be, but I've deleted it because a tracked log of past bugs, sitting next to a skill that's already supposed to encode the lessons from those bugs, is a duplication waiting to drift. I believe that if the rule is in the skill, the log is redundant. If the rule isn't in the skill, the log is an evidence that I haven't finished the work. Either way, the right place for bug history is git log.

13. Splitting "the skill" from "this repo's defaults"

I then wanted the orchestrator to be portable, but every run kept needing the same handful of decisions. Which Azure region by default? Which environment names? Which catalog naming convention? These weren't part of the article. They weren't part of the Terraform skill either. They were specific to this repository's opinion about how things should be deployed.

If I baked them into the orchestrator, the orchestrator stopped being portable. If I left them out, every run produced unhelpful "not stated in article" entries for the same five universal decisions.

The answer was a new file called REPO_CONTEXT.md stored in the repo root. It's read by the orchestrator before generation and it carries the defaults that are owned by the repo, not by the skill. The split looks like this in practice:

SKILL.md answers the question "how do I turn an article into a runnable repo?" It is portable.
REPO_CONTEXT.md answers the question "what does this repo default to when the article doesn't say?" It is local.

Cloning the orchestrator into another GitHub project is now a clean operation. You take the skill, you write your own REPO_CONTEXT.md, and the same generator produces output appropriate to your environment.

14. The Validations

Most of the rules I'd written into the skills were prose. "Don't invent suffixes." "Object IDs are inputs, not lookups." "Every required Terraform variable must have a matching TF_VAR_* in the workflow." The model is good at following prose rules most of the time.

So a few of the most regression-prone rules became executable. The most important one is scripts/validate_workflow_parity.sh. Every variable declared in infra/terraform/variables.tf must appear as a TF_VAR_* export in the deploy workflow. The script greps both files, diffs the sets, and exits non-zero if they don't match. It is run at the end of generation. If it fails, the run failed, even if everything else looks fine.

This caught real bugs. The most embarrassing was a variable I'd added to variables.tf and forgot to wire through the workflow. Terraform plan would prompt interactively for it on a non-interactive runner, and the run would hang.

The rule of thumb I've ended up with is: prose rules are the default, but if a rule has been violated more than twice, it gets promoted to an executable check. There's a short list of those checks now, and it's the load-bearing one.

15. Key Vault soft-delete state machine

Key Vaults in Azure have soft delete on by default. When you delete a vault, it sticks around for ninety days in a "soft-deleted" state. If you try to create a vault with the same name in the same subscription during that window, the deploy fails. The right behaviour is to recover the soft-deleted vault, not create a new one.

The first version of my recovery handler covered exactly one case: if the vault is soft-deleted, recover it. This worked the first time I ran it.

The second time, the recovered vault came back into the previous resource group, not the new one I had just created. Terraform then tried to create a new vault in the correct resource group and failed because the name was already taken globally. The handler had no concept of "the recovered vault is in the wrong resource group." So I added that case.

The third time, the previous resource group itself was gone, and the handler was looking for it to verify the move. So I added that case too.

By the end, the state machine had three distinct cases and two preconditions, as shown in the diagram below.

The reason I keep coming back to this state machine is that it captures something that I think is generally true about agent-generated infrastructure code. The happy path is easy and meaningless, while the value is in the failure modes. The first version that worked on a clean tenant was about ten lines of bash. The version that works on a tenant that has been deployed-into and partially-torn-down five times is six times longer, and every additional line of it corresponds to a real environmental condition that I had to learn the hard way.

16. What I've learned so far

I'm not going to pretend the full list of principles below was clear to me on day one. Every single one of these was learned by getting it wrong first. Looking back at the history, though, they are the ones that survived contact with reality.

The contract precedes the implementation. output-contract.md was committed before any generator existed. Locking the shape of the deliverable first meant every later change had a fixed reference point.
Generators, not stencils. Workflows are produced by Python scripts that take parameters and emit YAML. When restricted-tenant logic and the soft-delete state machine arrived, they needed conditional structure that a static template can't express.
Every bug becomes a rule. Patching the generated code is a tax on tomorrow's run. While patching the skill is an investment.
Each concern has a clear owner. The orchestrator routes, the leaves author, and the repo context holds the local defaults.
Restricted-tenant compatibility is non-negotiable. No Microsoft Graph reads from generated Terraform. Object IDs are trusted inputs. Single-principal mapping is supported.
Naming is owned by Terraform. No suffixes invented in shell. The validation harness enforces this.
State must be explicit, not inherited. Every workflow run writes its own flags. No reliance on env defaults from a previous step or a previous run.
Validation is executable when a rule has been violated more than twice. Prose rules are the default. Promotion to a script is earned.
Operator docs describe concepts, not commands. Command syntax ages out, while conceptual descriptions don't. The TODO template enforces this rule.
Add strong testing at the end of the process, once all the files are generated. Each run may produce slightly different output and introduce bugs, even if the previous run was successful.
End-to-end runs against dirty tenants are the truth. The acceptance test isn't a clean-room deploy. It's a deploy into a tenant that has soft-deleted vaults, lingering RGs, and existing role assignments. Until that works, the project isn't done.
From time to time, skills need to be reviewed and consolidated.

The summary above of the journey is the one I find most useful to share when people ask whether this approach actually goes anywhere. From an empty repo to a generator that produces a deployable, restricted-tenant-compatible infrastructure-as-code repository from a blog URL, with executable validation and a recovery state machine that survives a previously-deployed environment.

The first commit was an empty workspace. The last commit was the one where the same orchestrator, run against the same blog, against a tenant carrying state from five previous runs, deployed cleanly with no manual intervention.

That is what I what I was aiming to achieve when I started!

Thanks for reading.

Resilient by Design: Azure Databricks Disaster Recovery Strategy

KonstantinaF — Wed, 06 May 2026 16:28:58 GMT

Introduction: From Recovery Plans to Resilience Strategy

As organizations increasingly rely on Azure Databricks for mission-critical analytics and data engineering workloads, the need for robust disaster recovery (DR) strategies becomes paramount. These platforms are no longer just analytics engines, they power real-time decisions, AI models, and core business operations. Yet many organizations still approach Disaster Recovery (DR) as a reactive safeguard rather than a strategic capability. Resilience today is not about “if something fails,” but about ensuring continuity, trust, and performance under any condition.

A modern DR strategy must therefore evolve beyond backup configurations and failover scripts. It must align with business priorities, regulatory requirements, risk tolerance, and operational maturity to become a core pillar of the enterprise data platform. In this context, organizations are increasingly adopting architecture patterns that enable cross-region resilience for the Azure Databricks Lakehouse. This pattern includes synchronizing Unity Catalog objects—catalogs, schemas, tables, views, function, models, and volumes—across regions, combined with scalable data movement mechanisms and secure data access approaches such as Delta Sharing and high-performance transfer tools.

To help organizations operationalize this approach today, we have defined a structured strategy for synchronizing Unity Catalog objects and associated data across regions, enabling a resilient-by-design Azure Databricks architecture. This post focuses on that approach, outlining the key architectural patterns, strategic considerations, and practical implementation steps required to design and enable cross-region resilience.

In October 2025, Databricks announced a Managed Disaster Recovery solution, developed in collaboration with Capital One, which includes managed replication, customer-specified failover, and read-only secondary capabilities. The approach outlined in this post serves as a complementary, customer-managed pattern, providing a practical and production-ready path for organizations to achieve robust disaster recovery and business continuity while Databricks continues to expand its native DR capabilities.

Why Disaster Recovery for Azure Databricks is Different

Traditional Disaster Recovery approaches do not fully apply to modern Lakehouse platforms.

In Azure Databricks, resilience must account for:

Tight coupling between data, compute, and metadata (Unity Catalog)

Distributed pipelines (batch, streaming, ML)

Decentralized workspace ownership and rapid platform growth

This makes disaster recovery not just an infrastructure concern, but a data platform design challenge.

Figure 1. Main Disaster Recovery Considerations

Understanding the Fundamentals: RTO, RPO, and DR Trade-offs

Before defining a disaster recovery strategy, it is essential to understand the core concepts that drive design decisions.

Recovery Time Objective (RTO) defines how quickly a system must be restored after a disruption; while Recovery Point Objective (RPO) defines how much data loss is acceptable. These two metrics directly influence the architecture, cost, and complexity of any DR solution.

As illustrated in Figure 1, there is a clear trade-off between cost and recovery performance:

Active-active (hot) architectures, minimize downtime and data loss but come at a higher cost.

Warm standby provides a balance between cost and recovery time.

Cold DR is cost-efficient but results in longer recovery times and higher data loss risk.

Understanding these trade-offs is critical to aligning DR strategy with business expectations.

Understanding the Fundamentals: RTO, RPO, and DR Trade-offs

Before defining a disaster recovery strategy, it is essential to understand the core concepts that drive design decisions.

As illustrated in Figure 1, there is a clear trade-off between cost and recovery performance:

Active-active (hot) architectures, minimize downtime and data loss but come at a higher cost.

Warm standby provides a balance between cost and recovery time.

Cold DR is cost-efficient but results in longer recovery times and higher data loss risk.

Understanding these trade-offs is critical to aligning DR strategy with business expectations.

Designing for Resilience: A Phased Disaster Recovery Approach

Disaster recovery has evolved beyond a one-time setup into a structured, lifecycle-driven capability. Leading organizations design resilience intentionally, implement it systematically, and continuously validate it to ensure ongoing effectiveness.

The framework outlined below provides a practical and strategic approach to operationalizing disaster recovery in Azure Databricks environments, bridging the gap between architectural intent and true operational readiness.

Figure 2. Different Phases of Azure Databricks Disaster Recovery

Phase 1: Discovery & Assessment

A resilient disaster recovery strategy starts with clarity—yet in many Azure Databricks environments, that clarity is often missing. As platforms evolve, clusters multiply, jobs are duplicated, and data assets grow, making it increasingly difficult to answer a simple question: what do we actually have, and how critical is it?

The Discovery phase addresses this by establishing a single, authoritative view of the platform. By consolidating all assets, dependencies, and usage patterns into a structured baseline, organizations can move from fragmented visibility to informed decision-making.

This approach aligns closely with the concepts outlined in “From Chaos to Clarity: Your Databricks Workspace on a Single Pane of Glass”, where establishing a comprehensive inventory becomes the foundation for governance, optimization, and ultimately resilience.

This foundation enables teams to identify what matters most, define appropriate RTO and RPO targets, and understand the dependencies that will ultimately shape their disaster recovery strategy.

Outcome

A clear, data-driven baseline of the environment—enabling confident workload prioritization and effective disaster recovery design.

Phase 2: Strategy & Design

Once visibility is established, the next step is making deliberate design choices—balancing resilience, cost, and complexity. At this stage, organizations define how their platform should behave under failure. This typically starts with selecting a multi-site deployment pattern, in which two primary approaches are commonly adopted:

Active–Active, where both regions are fully operational and serve live workloads

Active–Passive (Warm Standby), where a secondary region is pre-provisioned and activated only during failover

Active–active architectures provide near-zero downtime and minimal data loss but come with increased cost and architectural complexity. Active–passive patterns offer a more cost-efficient alternative, with slightly higher recovery times depending on how failover is orchestrated. Beyond selecting the deployment pattern, a key architectural decision is how data is replicated across the Medallion architecture (Bronze, Silver, Gold). Our approach introduces a set of practical scenarios that allow organizations to tailor resilience based on both workload criticality and recovery requirements.

A common starting point is aligning the DR strategy to workload tiers, such as:

Tier 1 (Mission-critical): Active–Active with full replication

Tier 2 (Business-critical) : Active–Passive with partial replication

Building on this, organizations can further refine their approach by defining how data is replicated across the Medallion layers:

Full replication (Bronze, Silver, Gold) , i.e. fastest recovery at highest cost;

Bronze-only replication, lower cost, with re-computation required during recovery;

Gold-only replication, optimized for consumption-focused use cases.

This combination of workload tiering and Medallion replication strategies enables a flexible, fit-for-purpose approach to disaster recovery, which balances performance, cost, and operational complexity.

Below we demonstrate, as an example, two representative patterns: (a) Active–Active architecture, where data pipelines operate in continuous trigger mode across regions, enabling near real-time synchronization; and (b) Active–Passive architecture, where all layers are replicated using a clone-based approach and activated on demand during failover.

These scenarios highlight how organizations can balance recovery performance and cost by adjusting both the deployment model and the depth of data replication.

Figure 3. Active - Active Scenario - Continuous Trigger Mode

Within the active–passive model, multiple variations can be applied, ranging from full replication of all medallion layers to more selective approaches (such as replicating only Bronze or Gold layers). This flexibility allows organizations to further balance recovery performance, cost, and operational complexity.

Figure 4. Active - Passive Scenario - Clone All Layers Mode

Phase 3: Disaster Recovery Implementation & Enablement

With the strategy defined, the focus shifts to translating design into a repeatable and operational solution. At this stage, resilience is no longer conceptual, it is embedded into the platform through automation, data replication, and standardized deployment patterns.

From Strategy to Architecture

At a high level, the DR architecture spans both the primary and secondary Azure regions, ensuring that all critical components can be either replicated or recreated:

Control plane synchronization: Users, groups, and workspace assets are replicated using SCIM, Terraform, and CI/CD pipelines.

Workspace and metadata portability: Jobs, notebooks, and configurations are defined as code and deployed consistently across regions.

Data layer replication: Managed data, external data, and streaming checkpoints are synchronized using deep clone operations.

This layered approach ensures that the platform can be reconstructed end-to-end, not just partially recovered.

Unity Catalog-Driven Replication

A critical aspect of the implementation is the replication of Unity Catalog metadata and associated data assets. This includes:

Synchronizing catalogs, schemas, tables, views, functions, and volumes

Using Delta Sharing to expose datasets across regions

Leveraging deep clone and storage replication to ensure data availability

Recreating external and managed locations in the target region

By combining metadata synchronization with data replication, the target environment becomes a fully functional mirror of the source.

Figure 5. Unity Catalog Focused DR Mechanisms

Operationalizing with a DR Pipeline

To make this repeatable, the architecture is supported by a DR pipeline that orchestrates the process end-to-end:

Synchronize schemas and Unity Catalog structures

Perform deep clone of Delta tables

Recreate views and dependent objects

Provision volumes and copy associated data

Ensure consistency across storage layers (e.g., ADLS via AzCopy)

This pipeline can operate either continuously or on demand, depending on the selected DR pattern.

Figure 6. Azure Databricks DR Replication Workflow

Outcome

A fully implemented disaster recovery solution where data, metadata, and platform components are consistently synchronized, enabling rapid and reliable activation of workloads in a secondary region.

Phase 4: DR Drill: Validation, Operations & Continuous Improvement

A disaster recovery strategy is only valuable if it works when needed. This phase focuses on validating, operating, and continuously improving the DR solution to ensure it meets business expectations.

Failover & Failback in Practice

In a real failure scenario, the transition to the secondary region must be simple, predictable, and fast.

A typical failover process includes:

Detecting primary region unavailability

Executing a final synchronization (if possible)

Redirecting connections to the DR workspace

Resuming operations without requiring code changes

Equally important is failback, once the primary region is restored:

Re-synchronizing data from DR to primary

Switching pipelines and configurations back

Gradually restoring normal operations

Because infrastructure and metadata are standardized, this process becomes operational rather than reactive.

Operating DR as a Continuous Capability

Beyond failover, DR must be actively managed as part of daily operations:

Monitoring & Alerting: Track job failures, performance bottlenecks, and system health

Governance & Change Management: Maintain consistency between environments using IaC and version-controlled pipelines

Continuous Optimization: Adjust replication strategies, scaling, and performance as workloads evolve

This ensures the DR solution remains aligned with both technical and business changes over time.

Ensuring Performance, Integrity, and Security

A production-ready DR solution must also guarantee:

Performance & Scalability: Optimize compute, autoscaling, and data transfer to handle recovery scenarios efficiently

Data Integrity & Consistency: Validate schema synchronization, monitor replication jobs, and ensure parity between regions

Security & Compliance: Enforce consistent access controls, secure credentials, and enable audit logging across environments

Outcome

A validated and continuously evolving DR capability—where recovery processes are tested, monitored, and improved over time, providing confidence to both technical teams and business stakeholders.

Key Takeaways and Closing Thoughts

Resilience in modern data platforms is no longer defined by how quickly systems can recover, but by how effectively they are designed to withstand disruption in the first place. Azure Databricks, as a core engine for data, analytics, and AI, requires a disaster recovery approach that extends beyond infrastructure—one that treats data, metadata, pipelines, and governance as a unified system. By combining a structured discovery phase, a strategy aligned to workload criticality, and automated, repeatable implementation patterns, organizations can move from reactive recovery to resilience by design. This not only reduces risk, but also ensures that critical data workloads remain available, trusted, and performant when it matters most.

The approach outlined in this post provides a practical and flexible way to enable cross-region resilience today, while also complementing the managed disaster recovery capabilities expected to be introduced by Databricks. As we anticipate the availability of these native features, this approach offers a production-ready foundation that can extend and integrate with future platform capabilities. In a world where disruption is inevitable, the objective is no longer simply to recover—but to maintain continuity of data, decisions, and business operations with confidence.

Special thank you to Vasilis Zisiadis, Dimitris Kotanis who contributed their expertise to create this material and bring it to life.

Thank You Antony Bitar, Collin Brian and Jason Pereira for their support in reviewing the content.

From Chaos to Clarity: Your Databricks Workspace on a Single Pane of Glass

AmitDamle — Wed, 06 May 2026 16:25:47 GMT

The question that never stays answered — until now

As Azure Databricks workspaces evolve, complexity creeps in unnoticed. Every Azure Databricks conversation with customers eventually lands on the same question: “What do we actually have in this workspace?” Over time, clusters multiply, jobs get cloned, warehouses are spun up for one-off demos and forgotten, and Unity Catalog keeps expanding until it’s hard to reason about. In most enterprises, each business or data science team operates its own workspace, while the central platform or operations team has little to no visibility into what’s being created or why. Teams often spend days—or weeks—trying to piece together what exists, who owns it, and the business purpose behind it, only to realize they still don’t have the full picture. And when the same question comes up next quarter, the cycle starts all over again.

To address this, we built a utility that helps customers answer exactly that—by providing a single pane of glass for all Databricks assets through comprehensive cataloging and usage analysis. The utility works in two phases: Discovery and Analysis. This post focuses on the first step—the Discovery phase, where we establish a clear, authoritative inventory of everything that exists in the workspace.

What the Discovery Phase delivers?

Think of the Discovery phase as a workspace health assessment. Once configured against a target workspace, the utility runs in a selected mode and consolidates all discovered assets into a centralized, Delta-based repository. The result is a structured, queryable, and dashboard-ready metadata store.

Behind the scenes, ten purpose-built scanners run in a tiered and parallelized architecture, enabling a fast yet comprehensive scan of the entire workspace.

Scanner	What is Cataloged
Clusters	Interactive, job, SQL — configs, policies, pools
Jobs	Workflows, schedules, tasks, run history
Warehouses	SQL endpoints, sizes, serverless settings
Pipelines	Delta Live Tables and their state
Unity Catalog	Catalogs, schemas, tables, volumes
Workspace Objects	Notebooks, repos, ML experiments, serving endpoints, alerts, Genie spaces
Security	Identity, network, data protection settings
Billing	30–180 days of DBU usage by SKU and product
Utilization	Real CPU, memory, runtime patterns (deep scan)
Spark Job Optimizer (plugin)	Skew, spill, small files, broadcast hints (deep scan)

Design Overview

#	Block	Role	Contents / Flow
1	Source	Starting point — the Databricks environments being discovered.	One or more Azure Databricks workspaces. Auth via OAuth. Outputs an authenticated WorkspaceClient to the Orchestrator.
2	Orchestrator	The brain of the utility — coordinates scanning, concurrency, retries, timing.	Tiered thread-pool executor, scan config (mode, billing window, UC depth, max workers). Dispatches scanners in controlled waves.
3	Tier 1 Scanners	Lightweight, high-concurrency scans. Run first for quick signal.	Clusters, Warehouses, Pipelines, Security. Up to 12 workers, 10-min timeout. Artifacts flow to the Centralized Repository.
4	Tier 2 Scanners	High-volume scans. Controlled concurrency to avoid API throttling.	Jobs, Workspace Objects (notebooks, repos, experiments, serving, alerts, Genie), Unity Catalog, Billing (30–180 days DBU). 1/2 workers, 30-min timeout.
5	Tier 3 Scanners	Sequential, analysis-grade scans (deep scan only).	Utilization (CPU, memory, SQL usage patterns) and Spark Job Optimizer plugin (skew, spill, small files, broadcast hints). Runs after Tiers 1 & 2.
6	Centralized Repository	The catalog of truth — where all output lands, timestamped and queryable.	Unity Catalog Delta tables (dashboard-ready) plus portable JSON and CSV exports for offline sharing or downstream tools.
7	Single Pane of Glass	The user-facing view — insight at a glance.	Pre-built Lakeview dashboard: KPI strip, inventory charts, and week-over-week trends. Refresh to see current workspace state.

Why users love the view — visualization that earns its keep

This is where the Discovery phase stops being just a scan and starts becoming a decision-making tool. Because everything is consolidated into a single, Unity Catalog–backed source of truth, the Lakeview dashboard delivers a genuine single pane of glass for the entire Databricks workspace.

At a glance, you get:

KPI strip at the top — total clusters, active jobs, UC tables, SQL warehouses, DLT pipelines, workspace objects. One glance, one number each.

Inventory charts — clusters by type, jobs by schedule, warehouses by size, tables by catalog. The shape of your workspace becomes obvious.

The “that doesn’t look right” moments — The idle SQL warehouse with zero queries, the cluster running the wrong runtime, the notebook floating outside any repo. These surface instantly, without hunting. Change over time — because every scan is timestamped, you can literally see your platform grow (or sprawl) week over week.

In the first customer walkthrough, the platform team identified an always-on SQL warehouse with zero queries and three jobs running on the wrong compute tier—all within the first 30 minutes. That single view paid for the project.

Sample Item Catalog

Closing thoughts

The Discovery phase isn’t about governance for governance’s sake—it’s about clarity. Before teams can optimize costs, improve performance, or enforce standards, they first need a reliable answer to a basic question: what actually exists today? By giving platform and operations teams a single, authoritative view of all Databricks assets—grounded in data, not tribal knowledge—Discovery turns guesswork into informed decisions. In the next phase, Analysis, that foundation is used to go deeper: identifying inefficiencies, risks, and opportunities to simplify and optimize the platform. But it all starts here—by finally knowing what you have.

Special thank you to Antony Bitar, Collin Brian and Jason Pereira for their support in reviewing the content.

From Manual Backfills to Autonomous Pipelines: Building an LLM-Powered Backfill Agent on Azure

Sally_Dabbah — Tue, 05 May 2026 10:16:19 GMT

Introduction:

Data backfills are a common operational requirement in modern data platforms. Missing partitions, upstream delays, or failed pipeline runs often require engineers to manually identify gaps, determine the appropriate recovery window, and trigger reprocessing. This approach does not scale well it introduces operational overhead, increases the risk of human error, and requires deep knowledge of data dependencies and pipeline behavior.

In this post, I describe how to build a backfill agent using Azure AI Foundry, Model Context Protocol (MCP), Azure Functions, Synapse, and ADX. The goal is to automate the decision-making process while keeping execution controlled, observable, and governed.

The design separates responsibilities across three layers:

Decision layer: an LLM-based agent determines whether a backfill is required and defines the recovery scope (e.g., which dates, datasets, or layers)
Execution layer: an MCP server hosted on Azure Functions exposes controlled operations such as triggering pipelines and querying system state
State layer: ADX tables maintain backfill control metadata, data availability signals, and execution history

This separation keeps the system flexible while ensuring that all actions are traceable, auditable, and policy-driven.

Importantly, this pattern is not limited to a single dataset or pipeline. It can be applied across all datasets and across all layers of a medallion architecture Bronze, Silver, and Gold with layer-specific validation rules and backfill strategies. For example, Bronze may focus on completeness of ingestion, while Silver and Gold can enforce data quality and business logic constraints before initiating recovery.

The key benefit of a backfill agent is that it shifts backfilling from a manual, reactive process to an automated, intelligent, and consistent workflow. Instead of engineers investigating incidents and triggering reruns, the agent continuously evaluates data state, identifies gaps, and initiates-controlled recovery actions. This reduces operational burden, improves reliability, and ensures faster recovery from data issues while maintaining governance, observability, and strict control over execution.

Architecture Overview

The solution is designed as a controlled orchestration pattern that separates decision-making, execution, and state management. This allows backfill operations to be automated without compromising governance or observability.

The architecture consists of four main components.

Logic Apps Trigger

The workflow is initiated using a Logic App. The trigger can be scheduled or invoked on demand, depending on operational requirements.

It provides the input context required for the backfill evaluation, such as dataset name, processing layer, and scope constraints (for example, maximum number of dates to process).

Azure AI Foundry Agent (Decision Layer)

The Azure AI Foundry agent acts as the decision layer.

It evaluates the request and determines whether a backfill is required, and if so, what scope should be applied. The agent does not interact directly with data systems. Instead, it invokes predefined tools exposed through the MCP server.

This ensures that decision logic is flexible, while execution remains controlled.

Azure Function App – MCP Server (Execution Layer)

The Azure Function App hosts the MCP server and exposes a set of operations to the agent.

These operations include querying missing partitions, triggering Synapse pipelines, retrieving execution status, and updating control tables.

All interactions with external systems (Synapse and ADX) are handled within this layer. It is responsible for input validation, authorization, and enforcing execution rules.

This abstraction ensures that infrastructure actions are not directly performed by the agent.

Synapse Pipelines (Processing Layer)

Backfill execution is handled by a parameterized Synapse pipeline.

The pipeline follows a consistent pattern:

Data is first written to a staging table
Validation is performed
Data is promoted to the main table only if validation succeeds

This approach ensures data quality and prevents partial or invalid data from being published.

Azure Data Explorer(State and Observability Layer)

ADX is used as the central state store.

It maintains control and execution tables that track expected partitions, missing data, pipeline runs, and execution outcomes.

This enables:

Detection of missing partitions
Idempotent execution (avoiding duplicate processing)
Full traceability of backfill operations

The agent relies on this state, accessed via the MCP server, to make decisions.

End-to-End Flow

The Logic App triggers the workflow and passes the request context.
The Foundry agent evaluates the request.
The agent invokes an MCP tool to retrieve missing partitions from ADX.
Based on the result, the agent determines whether a backfill is required.
If required, the agent invokes an MCP tool to trigger the Synapse pipeline.
The pipeline executes the backfill using a staging and validation pattern.
Execution details are written to ADX.
The agent returns a summary of the operation.

Analytics Layer:

Azure Synapse Analytics:

In the Synapse workspace, I created a generic parameterized pipeline that has three steps:
1.Copy data from upstream and ingest it to ADX staging table

2. Run Data validation

3 ingest staging data to main dataset table.

the pipeline gets as a parameter dataset name, partitioning date, isbackfill flag and layer and ingest dataset into kusto table.

values for layer : Bronze,Silver or Gold.

Kusto:

In Kusto, the solution relies on the following tables:

Dataset tables for example, the Customers table in this demo [the same pattern can be extended to support multiple datasets.]
BackfillControl:
its the central configuration and decision input for the backfill process. It defines which dataset partitions require backfill and provides the metadata needed for the agent to make execution decisions.
each row in this table represents a specific dataset partition (for example, a given date in a specific layer) and its current backfill state.
BackfillExecutionLog :

this table is used to track the execution of backfill operations. It provides a complete record of when backfills were triggered, their outcome, and the associated pipeline runs, while the BackfillControl table defines what should be processed, the BackfillExecutionLog captures what actually happened.

Code for Creating the tables:
.create table BackfillExecutionLog ( ExecutionId: string, DatasetName: string, Layer: string, PartitionDate: datetime, PipelineName: string, PipelineRunId: string, TriggeredAt: datetime, TriggeredBy: string, ExecutionStatus: string, Reason: string ) .create table BackfillControl ( DatasetName: string, Layer: string, PartitionDate: datetime, BackfillRequired: bool, Status: string, DQStatus: string, RetryCount: int, MaxRetryCount: int, Reason: string )

Output examples:

In this demo, Logic Apps, Synapse, and Kusto are treated as existing systems; the focus is how to expose controlled MCP tools from an Azure Function App and connect them to Azure AI Foundry agent.

Microsoft’s Azure Functions MCP extension lets a Function App expose functions as MCP tools, and Foundry can connect to the deployed MCP endpoint.

Steps:

Step1: Create the local Function App project

in VS code, run the command:

mkdir backfill-kusto-mcp cd backfill-kusto-mcp func init . --worker-runtime python --python

Step2: Implement the MCP tools

add requirements to requirements.txt file:
azure-functions>=1.24.0 azure-identity azure-kusto-data requests python-dotenv
The host.json file defines runtime-level behavior for the Azure Function App.
In this implementation, it is used to configure the MCP extension, logging, and extension bundles.
{ "version": "2.0", "extensions": { "mcp": { "system": { "webhookAuthorizationLevel": "Anonymous" } } }, "logging": { "applicationInsights": { "samplingSettings": { "isEnabled": true, "excludedTypes": "Request" }, "enableLiveMetricsFilters": true } }, "extensionBundle": { "id": "Microsoft.Azure.Functions.ExtensionBundle.Experimental", "version": "[4.*, 5.0.0)" } }
The local.settings.json file is used to define environment-specific configuration for the Azure Function App during local development.

It contains application settings (environment variables) that are read by the Function App at runtime. These settings are not checked into source control and are replaced by App Settings in Azure after deployment.
For example:
{ "IsEncrypted": false, "Values": { "AzureWebJobsStorage": "UseDevelopmentStorage=true", "FUNCTIONS_WORKER_RUNTIME": "python", "KUSTO_CLUSTER": "https://<ClusterName>.<Region>.kusto.windows.net", "KUSTO_DATABASE": "<DatabaseName>", "BACKFILL_CONTROL_TABLE": "BackfillControl", "BACKFILL_EXECUTION_LOG_TABLE": "BackfillExecutionLog", "SYNAPSE_WORKSPACE": "<SynapseWorkspaceName>", "SYNAPSE_PIPELINE": "<PipelineName>", "AUTH_MODE": "az_login", "AZURE_CLIENT_ID": "", "DEFAULT_DATASET_NAME": "Customers", "DEFAULT_LAYER": "Bronze", "MAX_DATES_DEFAULT": "5" } }
For local development, AUTH_MODE is set to az_login.

Before deploying to Azure Functions, change AUTH_MODE to MANAGED_IDENTITY in the Function App application settings.
The function_app.py defines the main implementation of MCP server

ir:
- Exposes MCP tools (find_backfill_candidates, trigger_backfill, run_backfill_agent, get_backfill_execution_log)
- Reads configuration from environment variables
- Authenticates using Azure CLI (local) or Managed Identity (Azure)
- Queries BackfillControl in Kusto to identify missing partitions
- Triggers Synapse pipelines for backfill
- Writes execution results to BackfillExecutionLog
- Enforces idempotency by checking if a partition was already triggered
Code:

import os import uuid import json import logging from datetime import datetime, timezone from urllib.parse import quote import requests import azure.functions as func from azure.identity import ManagedIdentityCredential, AzureCliCredential from azure.kusto.data import KustoClient, KustoConnectionStringBuilder app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS) logging.basicConfig(level=logging.INFO) AUTH_MODE = os.getenv("AUTH_MODE", "MANAGED_IDENTITY").lower() KUSTO_CLUSTER = os.getenv("KUSTO_CLUSTER") KUSTO_DATABASE = os.getenv("KUSTO_DATABASE") CONTROL_TABLE = os.getenv("BACKFILL_CONTROL_TABLE", "BackfillControl") EXECUTION_LOG_TABLE = os.getenv("BACKFILL_EXECUTION_LOG_TABLE", "BackfillExecutionLog") SYNAPSE_WORKSPACE = os.getenv("SYNAPSE_WORKSPACE") SYNAPSE_PIPELINE = os.getenv("SYNAPSE_PIPELINE", "Customer Dataset") DEFAULT_DATASET_NAME = os.getenv("DEFAULT_DATASET_NAME", "Customers") DEFAULT_LAYER = os.getenv("DEFAULT_LAYER", "Bronze") def utc_now() -> str: return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") def log_event(event: str, **properties): logging.info( "MCP_BACKFILL %s", json.dumps( { "event": event, "timestamp_utc": utc_now(), **properties, }, default=str, ), ) def require_setting(name: str, value: str | None): if not value: raise ValueError(f"Missing required app setting: {name}") def escape_kusto_string(value: str | None) -> str: if value is None: return "" return str(value).replace("\\", "\\\\").replace('"', '\\"') def get_credential(): if AUTH_MODE == "az_login": return AzureCliCredential() managed_identity_client_id = os.getenv("AZURE_CLIENT_ID") if managed_identity_client_id: log_event( "using_user_assigned_managed_identity", client_id=managed_identity_client_id, ) return ManagedIdentityCredential(client_id=managed_identity_client_id) log_event("using_system_assigned_managed_identity") return ManagedIdentityCredential() def get_kusto_client() -> KustoClient: require_setting("KUSTO_CLUSTER", KUSTO_CLUSTER) if AUTH_MODE == "az_login": kcsb = KustoConnectionStringBuilder.with_az_cli_authentication(KUSTO_CLUSTER) else: managed_identity_client_id = os.getenv("AZURE_CLIENT_ID") if managed_identity_client_id: kcsb = KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication( KUSTO_CLUSTER, client_id=managed_identity_client_id, ) else: kcsb = KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication( KUSTO_CLUSTER ) return KustoClient(kcsb) def execute_kusto_query(query: str): require_setting("KUSTO_DATABASE", KUSTO_DATABASE) client = get_kusto_client() response = client.execute(KUSTO_DATABASE, query) return response.primary_results[0] def execute_kusto_command(command: str): require_setting("KUSTO_DATABASE", KUSTO_DATABASE) client = get_kusto_client() return client.execute_mgmt(KUSTO_DATABASE, command) def find_backfill_candidates_core( dataset_name: str, layer: str, max_dates: int, ) -> list[dict]: dataset = escape_kusto_string(dataset_name) layer_name = escape_kusto_string(layer) query = f""" {CONTROL_TABLE} | where DatasetName == "{dataset}" | where Layer == "{layer_name}" | where BackfillRequired == true | where RetryCount < MaxRetryCount | where Status in ("Missing", "Failed") or DQStatus == "Failed" | top {int(max_dates)} by PartitionDate asc | project DatasetName, Layer, PartitionDate, Status, DQStatus, RetryCount, MaxRetryCount, Reason """ rows = execute_kusto_query(query) return [ { "DatasetName": row["DatasetName"], "Layer": row["Layer"], "PartitionDate": str(row["PartitionDate"])[:10], "Status": row["Status"], "DQStatus": row["DQStatus"], "RetryCount": row["RetryCount"], "MaxRetryCount": row["MaxRetryCount"], "Reason": row["Reason"], } for row in rows ] def was_backfill_already_triggered( dataset_name: str, layer: str, partition_date: str, ) -> bool: dataset = escape_kusto_string(dataset_name) layer_name = escape_kusto_string(layer) query = f""" {EXECUTION_LOG_TABLE} | where DatasetName == "{dataset}" | where Layer == "{layer_name}" | where PartitionDate == datetime({partition_date}) | where ExecutionStatus == "Triggered" | summarize Count = count() """ rows = list(execute_kusto_query(query)) return bool(rows and rows[0]["Count"] > 0) def write_execution_log( execution_id: str, dataset_name: str, layer: str, partition_date: str, pipeline_name: str, pipeline_run_id: str, execution_status: str, reason: str, ): command = f""" .set-or-append {EXECUTION_LOG_TABLE} <| print ExecutionId = "{escape_kusto_string(execution_id)}", DatasetName = "{escape_kusto_string(dataset_name)}", Layer = "{escape_kusto_string(layer)}", PartitionDate = datetime({partition_date}), PipelineName = "{escape_kusto_string(pipeline_name)}", PipelineRunId = "{escape_kusto_string(pipeline_run_id)}", TriggeredAt = datetime({utc_now()}), TriggeredBy = "FoundryMCPBackfillAgent", ExecutionStatus = "{escape_kusto_string(execution_status)}", Reason = "{escape_kusto_string(reason)}" """ execute_kusto_command(command) def trigger_synapse_pipeline( dataset_name: str, layer: str, partition_date: str, ) -> str: require_setting("SYNAPSE_WORKSPACE", SYNAPSE_WORKSPACE) require_setting("SYNAPSE_PIPELINE", SYNAPSE_PIPELINE) credential = get_credential() token = credential.get_token("https://dev.azuresynapse.net/.default").token encoded_pipeline_name = quote(SYNAPSE_PIPELINE, safe="") url = ( f"https://{SYNAPSE_WORKSPACE}.dev.azuresynapse.net" f"/pipelines/{encoded_pipeline_name}/createRun" f"?api-version=2020-12-01" ) payload = { "DatasetName": dataset_name, "Layer": layer, "PartitionDate": partition_date, "IsBackfill": True, } response = requests.post( url, headers={ "Authorization": f"Bearer {token}", "Content-Type": "application/json", }, json=payload, timeout=30, ) log_event( "synapse_create_run_response", status_code=response.status_code, body=response.text[:2000], ) response_json = {} try: response_json = response.json() except Exception: pass if "runId" in response_json: return response_json["runId"] raise Exception( f"Synapse trigger failed. " f"StatusCode={response.status_code}. " f"Body={response.text}" ) def trigger_backfill_core( dataset_name: str, layer: str, partition_date: str, ) -> dict: execution_id = str(uuid.uuid4()) log_event( "trigger_backfill_started", execution_id=execution_id, dataset_name=dataset_name, layer=layer, partition_date=partition_date, ) try: if was_backfill_already_triggered(dataset_name, layer, partition_date): return { "ExecutionId": execution_id, "DatasetName": dataset_name, "Layer": layer, "PartitionDate": partition_date, "Status": "Skipped", "Reason": "Backfill was already triggered for this partition.", } pipeline_run_id = trigger_synapse_pipeline( dataset_name=dataset_name, layer=layer, partition_date=partition_date, ) write_execution_log( execution_id=execution_id, dataset_name=dataset_name, layer=layer, partition_date=partition_date, pipeline_name=SYNAPSE_PIPELINE, pipeline_run_id=pipeline_run_id, execution_status="Triggered", reason="Triggered by Foundry MCP backfill agent", ) return { "ExecutionId": execution_id, "DatasetName": dataset_name, "Layer": layer, "PartitionDate": partition_date, "PipelineName": SYNAPSE_PIPELINE, "PipelineRunId": pipeline_run_id, "Status": "Triggered", } except Exception as ex: error_message = str(ex) log_event( "trigger_backfill_failed", execution_id=execution_id, dataset_name=dataset_name, layer=layer, partition_date=partition_date, error=error_message, ) try: write_execution_log( execution_id=execution_id, dataset_name=dataset_name, layer=layer, partition_date=partition_date, pipeline_name=SYNAPSE_PIPELINE or "", pipeline_run_id="", execution_status="FailedToTrigger", reason=error_message, ) except Exception as log_ex: log_event( "failed_to_write_execution_log", execution_id=execution_id, original_error=error_message, log_error=str(log_ex), ) return { "ExecutionId": execution_id, "DatasetName": dataset_name, "Layer": layer, "PartitionDate": partition_date, "Status": "FailedToTrigger", "Error": error_message, } def run_backfill_agent_core( dataset_name: str, layer: str, max_dates: int, ) -> list[dict]: log_event( "run_backfill_agent_started", dataset_name=dataset_name, layer=layer, max_dates=max_dates, ) candidates = find_backfill_candidates_core( dataset_name=dataset_name, layer=layer, max_dates=max_dates, ) results = [] for candidate in candidates: result = trigger_backfill_core( dataset_name=candidate["DatasetName"], layer=candidate["Layer"], partition_date=candidate["PartitionDate"], ) results.append(result) log_event( "run_backfill_agent_completed", dataset_name=dataset_name, layer=layer ) return results def get_execution_log_core( dataset_name: str, limit: int, ) -> list[dict]: dataset = escape_kusto_string(dataset_name) query = f""" {EXECUTION_LOG_TABLE} | where DatasetName == "{dataset}" | top {int(limit)} by TriggeredAt desc | project ExecutionId, DatasetName, Layer, PartitionDate, PipelineName, PipelineRunId, TriggeredAt, TriggeredBy, ExecutionStatus, Reason """ rows = execute_kusto_query(query) return [ { "ExecutionId": row["ExecutionId"], "DatasetName": row["DatasetName"], "Layer": row["Layer"], "PartitionDate": str(row["PartitionDate"])[:10], "PipelineName": row["PipelineName"], "PipelineRunId": row["PipelineRunId"], "TriggeredAt": str(row["TriggeredAt"]), "TriggeredBy": row["TriggeredBy"], "ExecutionStatus": row["ExecutionStatus"], "Reason": row["Reason"], } for row in rows ] @app.mcp_tool() @app.mcp_tool_property(arg_name="dataset_name", description="Dataset name, for example Customers.") @app.mcp_tool_property(arg_name="layer", description="Layer name, for example Bronze.") @app.mcp_tool_property(arg_name="max_dates", description="Maximum number of dates to return.") def find_backfill_candidates( dataset_name: str = DEFAULT_DATASET_NAME, layer: str = DEFAULT_LAYER, max_dates: int = 5, ) -> list[dict]: return find_backfill_candidates_core(dataset_name, layer, max_dates) @app.mcp_tool() @app.mcp_tool_property(arg_name="dataset_name", description="Dataset name, for example Customers.") @app.mcp_tool_property(arg_name="layer", description="Layer name, for example Bronze.") @app.mcp_tool_property(arg_name="partition_date", description="Partition date in yyyy-MM-dd format.") def trigger_backfill( dataset_name: str, layer: str, partition_date: str, ) -> dict: return trigger_backfill_core(dataset_name, layer, partition_date) @app.mcp_tool() @app.mcp_tool_property(arg_name="dataset_name", description="Dataset name, for example Customers.") @app.mcp_tool_property(arg_name="layer", description="Layer name, for example Bronze.") @app.mcp_tool_property(arg_name="max_dates", description="Maximum number of dates to trigger.") def run_backfill_agent( dataset_name: str = DEFAULT_DATASET_NAME, layer: str = DEFAULT_LAYER, max_dates: int = 5, ) -> list[dict]: return run_backfill_agent_core(dataset_name, layer, max_dates) @app.mcp_tool() @app.mcp_tool_property(arg_name="dataset_name", description="Dataset name, for example Customers.") @app.mcp_tool_property(arg_name="limit", description="Maximum number of execution log rows to return.") def get_backfill_execution_log( dataset_name: str = DEFAULT_DATASET_NAME, limit: int = 10, ) -> list[dict]: return get_execution_log_core(dataset_name, limit)

Step3: . Run locally
1. Activate virtual environment:

python -m venv .sally-env .\.sally-env\Scripts\activate

2. Install dependencies :

pip install -r requirements.txt npm install -g azurite

3. Open 2 terminals, in one terminal run:

azurite

4. in the second terminal:

az login

start the function app:

func start

P.S make sure to change auth in local.settings.json file to

"AUTH_MODE": "az_login"

Step4: Create Azure resources and deploy

# LOGIN az login # VARIABLES $RG="rg-backfill-kusto-mcp-demo" $LOCATION="westeurope" $STORAGE="stbackfillmcp$((Get-Random -Minimum 10000 -Maximum 99999))" $FUNCAPP="func-backfill-kusto-mcp-$((Get-Random -Minimum 10000 -Maximum 99999))" # CREATE RESOURCE GROUP az group create --name $RG --location $LOCATION # CREATE STORAGE ACCOUNT az storage account create ` --name $STORAGE ` --resource-group $RG ` --location $LOCATION ` --sku Standard_LRS # CREATE FUNCTION APP az functionapp create ` --resource-group $RG ` --consumption-plan-location $LOCATION ` --runtime python ` --runtime-version 3.11 ` --functions-version 4 ` --name $FUNCAPP ` --storage-account $STORAGE ` --os-type Linux # ENABLE MANAGED IDENTITY az functionapp identity assign ` --resource-group $RG ` --name $FUNCAPP # GET PRINCIPAL ID $FUNC_PRINCIPAL_ID = az functionapp identity show ` --resource-group $RG ` --name $FUNCAPP ` --query principalId ` --output tsv Write-Host "Function App Principal ID: $FUNC_PRINCIPAL_ID" # CONFIGURE APP SETTINGS az functionapp config appsettings set ` --resource-group $RG ` --name $FUNCAPP ` --settings ` AUTH_MODE=MANAGED_IDENTITY ` KUSTO_CLUSTER="https://<ClusterName>.<Region>.kusto.windows.net" ` KUSTO_DATABASE="<DatabaseName>" ` BACKFILL_CONTROL_TABLE="BackfillControl" ` BACKFILL_EXECUTION_LOG_TABLE="BackfillExecutionLog" ` SYNAPSE_WORKSPACE="<SynapseWorkspaceName>" ` SYNAPSE_PIPELINE="<PipelineName>" ` DEFAULT_DATASET_NAME="Customers" ` DEFAULT_LAYER="Bronze" ` MAX_DATES_DEFAULT="5" # DEPLOY FUNCTION APP (RUN FROM PROJECT FOLDER) func azure functionapp publish $FUNCAPP # GET MCP ENDPOINT $MCP_ENDPOINT="https://$FUNCAPP.azurewebsites.net/runtime/webhooks/mcp" Write-Host "MCP Endpoint: $MCP_ENDPOINT" # GET MCP KEY $MCP_KEY = az functionapp keys list ` --resource-group $RG ` --name $FUNCAPP ` --query "systemKeys.mcp_extension" ` --output tsv Write-Host "MCP Key: $MCP_KEY" # TEST MCP TOOL $body = @{ jsonrpc = "2.0" id = "1" method = "tools/call" params = @{ name = "find_backfill_candidates" arguments = @{ dataset_name = "Customers" layer = "Bronze" max_dates = 1 } } } | ConvertTo-Json -Depth 10 Invoke-RestMethod ` -Uri $MCP_ENDPOINT ` -Method POST ` -Headers @{ Accept = "application/json, text/event-stream" "x-functions-key" = $MCP_KEY } ` -ContentType "application/json" ` -Body $body

After deployment, the Function App’s managed identity must be granted the appropriate permissions in both Kusto and Synapse with Function app principal id , this allows the Function App to query Kusto tables and trigger Synapse pipelines without issues.

Step5: . Connect the MCP server to Azure AI Foundry

Go to Azure AI Foundry portal
Navigate to your Project
Open your Agent
Add MCP as a tool :
- Go to Tools
- Click Add Tool
- Select:
Custom → Model Context Protocol (MCP)

Configure custom MCP and click on Save
MCP endpoint:
https://<function-app-name>.azurewebsites.net/runtime/webhooks/mcp

Step6: Define Agent instructions

You are a Backfill Reliability Agent. You MUST use the backfill_agent MCP tool. Do NOT ask the user for candidate dates. When asked to run backfill: Find dataset name 1. Call find_backfill_candidates with dataset_name layer max_dates 2. Then call run_backfill_agent with dataset_name, layer= max_dates Return the PipelineRunId.

Note:
The instructions are very generic; you need to modify it based on your business scenario.

Step7: Test prompt

Now in Synapse Monitor:
Search for PipelineRunId: df1b1920-09dd-415b-bbe9-d810d8505f58:

Future Enhancements:

The backfill agent automates recovery by detecting missing or failed data and triggering controlled reprocessing via MCP.

It can scale across all datasets and medallion layers (Bronze, Silver, Gold) with layer-specific rules.

The design can evolve into a multi-agent workflow for example, if backfill fails multiple times, a notification agent can automatically send emails or create incidents for upstream teams.

Overall, this shift backfilling from a manual, reactive task to an automated, governed, and intelligent data operations process.

Links:
Tutorial: Host an MCP server on Azure Functions | Microsoft Learn

Quickstart: Set up Microsoft Foundry resources - Microsoft Foundry | Microsoft Learn

Quickstart: Connect Azure Data Explorer to an Azure Synapse Analytics workspace - Azure Synapse Analytics | Microsoft Learn

Would love to hear your Feedback: Sally Dabbah | LinkedIn

Step by Step Guide to Ontology and Plan for Financial Service

NaufalPrawironegoro — Sun, 19 Apr 2026 15:06:24 GMT

What We Will Build

In this guide, we will construct a complete Fabric IQ solution that accomplishes the following:

First, a Lakehouse that ingests publicly available data including bank financials, P2P lending statistics, borrower demographics, and licensing information. Second, a Semantic Model that defines the analytical layer with proper dimensions, measures, and relationships. Third, an Ontology that elevates these tables into business entities such as Bank, P2P Platform, Borrower, and Loan, connected by meaningful relationships and governed by regulatory rules. Fourth, a Planning sheet that enables supervisors to forecast enforcement workloads, allocate examination budgets, and model scenarios based on live data.

Step 1: Preparing the Data Foundation in Fabric Lakehouse

Every Fabric IQ solution begins with data. Before we can model business semantics or build planning sheets, we need a well structured Lakehouse that holds our source data in a governed and queryable format.

Creating the Lakehouse

Navigate to your Fabric workspace and create a new Lakehouse. In this example, we have named it P2PLendingLH, housed within the workspace P2P Lending CrossSector Demo. The Lakehouse serves as the Bronze and Silver layer of our medallion architecture, storing both raw ingested data and transformed analytical tables.

Data Sources and Tables

The Lakehouse is populated with data from publicly available publications. The table structure follows a dimensional modeling pattern with clear separation between dimension tables (prefixed with dim_) and relationship tables (prefixed with rel_). The following tables form the foundation of our model:

Table Name	Description
dim_bank	Bank profiles including KBMI tier, total assets, CAR, NPL, channeling exposure percentage
dim_borrower	Borrower demographics with credit score, employment type, province, and risk segment
dim_p2p_platform	Licensed P2P lending operators with TWP90 rate, outstanding balance, and total borrowers
dim_loan	Individual loan records with amount, tenure, interest rate, and repayment status
dim_supervisor_team	supervisory teams and their regional assignments
dim_channeling_agreement	Bank to P2P channeling contracts and exposure limits

In addition to dimension tables, several relationship tables capture the connections between entities. These include rel_bank_channels_platform (which bank funds which P2P platform), rel_borrower_takes_loan (linking borrowers to their loans), rel_loan_funded_by_bank (tracing the funding chain), rel_platform_issues_loan (connecting platforms to the loans they originate), and rel_supervisor_oversees_platform and rel_supervisor_oversees_bank (mapping supervisory responsibility).

Step 2: Creating the Semantic Model

With data in the Lakehouse, the next step is to create a Semantic Model that defines the analytical interface. The Semantic Model is a Power BI construct that organizes your tables into a star schema with proper relationships, hierarchies, and measures. More importantly for our purpose, this Semantic Model will later serve as the blueprint from which we generate our Ontology.

Generating the Model from Lakehouse

From within the Lakehouse, click on "New semantic model" in the toolbar. A dialog appears allowing you to name your model and select which tables to include. In our case, we select all dimension and relationship tables to ensure the Ontology will have full visibility into the data landscape.

Figure 1. Creating a new Direct Lake semantic model from the P2PLendingLH Lakehouse, selecting dimension and relationship tables for inclusion.

Notice that the dialog shows the workspace name (P2P Lending CrossSector Demo) and provides a searchable list of all available tables. The Direct Lake mode is automatically selected, which means the Semantic Model will query data directly from the Lakehouse parquet files without importing a copy. This is important for our use case because it ensures that when regulator publishes updated monthly statistics and the Lakehouse is refreshed, the Semantic Model and subsequently the Ontology will reflect the latest data.

Configuring Relationships and Properties

After creation, the Semantic Model opens in the editing view where you can configure relationships, add calculated measures, and define display properties. The model view shows the entity cards with their fields and the lines connecting related tables.

Figure 2. The Semantic Model editor showing entity cards for dim_bank and dim_borrower, with relationship lines and the full table listing in the Data panel.

In the screenshot above, you can see two of the core dimension tables. The dim_bank table contains fields such as bank_id, bank_type, channeling_exposure_pct, channeling_total, name, regulator_team, and total_assets. The dim_borrower table holds borrower_id, credit_score, employment_type, name, province, and risk_segment. The Data panel on the right reveals the complete set of tables available in this model, including all the relationship tables that define the connections between entities.

At this stage, you should verify that all necessary relationships are correctly established. For example, dim_bank should connect to rel_bank_channels_platform through bank_id, and dim_p2p_platform should connect to rel_platform_issues_loan through platform_id. These relationships are what enable the Ontology to reason across domains in the next step.

You may also want to add calculated measures at this point, such as a weighted average TWP90 across all platforms funded by a specific bank, or a total channeling exposure as a percentage of the bank's total assets. These measures will be carried forward into the Ontology and can be used by AI agents for natural language querying.

Step 3: Generating the Ontology

This is the step where the magic of Fabric IQ truly comes alive. The Ontology transforms your Semantic Model from a reporting layer into an intelligence layer. While the Semantic Model answers the question "what does the data look like," the Ontology answers the question "what does the data mean."

What the Ontology Does

An Ontology in Fabric IQ is a machine understandable vocabulary of your business. It consists of entity types (the things in your environment, such as Bank, Borrower, or P2P Platform), properties (the facts about those entities, such as a bank's NPL ratio or a platform's TWP90 rate), and relationships (the ways entities connect, such as a Bank channels funding to a P2P Platform). Beyond static modeling, the Ontology also supports rules and constraints that can trigger automated actions when business conditions are met.

Generating from the Semantic Model

To create the Ontology, open your Semantic Model and look for the "Generate Ontology" button in the toolbar. Clicking it opens the generation dialog, which presents three key value propositions:

Unify models into a semantic layer allows you to align concepts across domains and modeling paradigms, bringing banking data and P2P lending data into a shared vocabulary.

Model expressively enables you to capture complex relationships, domain specific rules, and actions that drive business workflows, such as triggering an alert when a P2P platform's TWP90 crosses the 5 percent regulatory threshold.

Reason over events and temporal patterns means that the Ontology can use sequences and trends to inform decisions and automation, such as detecting three consecutive months of TWP90 deterioration.

Figure 3. The Ontology generation dialog, creating a new Ontology named NewP2P from the existing Semantic Model within the P2P Lending CrossSector Demo workspace.

In the dialog, you specify the workspace (P2P Lending CrossSector Demo) and give your Ontology a name (in this example, NewP2P). After clicking Create, Fabric IQ analyzes the Semantic Model's structure, identifies entity types from dimension tables, infers relationships from the foreign key connections, and generates a navigable graph that represents your business domain.

Enriching the Ontology with Rules

Once the Ontology is generated, you can enrich it with business rules that reflect regulatory requirements. For the P2P lending use case, the following rules are particularly relevant:

Rule Name	Condition	Action
Elevated TWP90	P2P Platform TWP90 exceeds 5 percent	Flag platform as high risk and alert PVML supervisor
Contagion Risk	Bank channeling exposure to flagged P2P platform exceeds 10 percent of portfolio	Alert Banking supervisor and recommend joint examination
Youth Overleveraged	Borrowers aged 19 to 34 represent more than 60 percent of a platform's portfolio AND TWP90 is above average	Trigger consumer protection review and education program allocation
CAR Threshold	Bank CAR drops below 10 percent while having active P2P channeling agreements	Escalate to Kepala Eksekutif Pengawas Perbankan

These rules integrate with Fabric Activator, enabling the Ontology to automatically initiate business processes through alerts and automated actions. This means that when new monthly P2P statistics are ingested and a platform's TWP90 crosses the threshold, the system does not wait for an analyst to discover it manually. The rule fires, the alert is sent, and the supervisory workflow begins.

Querying with Natural Language

One of the most powerful capabilities enabled by the Ontology is the ability to query across domains using natural language through a Data Agent. Because the Ontology defines the business vocabulary and binds it to real data, a supervisor can ask questions like:

"Which banks have channeling agreements with P2P platforms whose TWP90 is currently above 5 percent, and what is their total exposure?"

The Data Agent resolves this query by traversing the Ontology graph: from the Bank entity through the channels_funding_to relationship to P2P Platform, filtering by the TWP90 property, and aggregating the channeling_total measure.

Step 4: Setting Up Planning Sheets

While the Ontology tells you what is happening in your business right now, the Plan item in Fabric IQ helps you decide what should happen next. Planning in Fabric IQ brings budgeting, forecasting, and scenario modeling directly into the same environment where your data lives, eliminating the disconnect between analytical insights and forward looking decisions.

Creating a Planning Sheet

To create a Plan, navigate to your workspace and select New Item followed by Plan (preview). After naming the plan and connecting it to your Semantic Model, you can begin building Planning sheets that pull dimensions and measures directly from the same data that powers your Ontology.

In the screenshot below, we see a Planning sheet named "Planning P2P" that presents a tabular view of all P2P lending platforms alongside their key risk metrics.

Figure 4. The Planning sheet showing P2P lending platforms with their TWP90 rates, total outstanding balances (in trillions of Rupiah), total borrower counts (in thousands), and risk categories.

The Planning sheet is structured with the platform name and risk_category as row dimensions, and three critical measures as values: Sum of twp90_rate, Sum of total_outstanding (displayed in trillions of Rupiah), and Sum of total_borrowers (displayed in thousands). The risk_category column provides an immediate visual classification of each platform's health status, with categories such as Elevated and Very High clearly indicating where supervisory attention should be directed.

Looking at the data, several insights emerge immediately. DanaBijak and DanaCepat both carry a Very High risk category, with TWP90 rates of 18.77 and 17.79 respectively. CashWagon ID shows an Elevated risk designation despite a comparatively modest TWP90 of 8.26, likely due to its substantial outstanding balance of 144.97 thousand borrowers. The aggregate row at the top reveals the industry total: a combined TWP90 of 365.38 (this is a sum across all platforms), total outstanding of 29.86 trillion Rupiah, and 7,281.55 thousand borrowers across the monitored universe.

Using Planning for Supervisory Resource Allocation

The real power of the Planning sheet becomes apparent when supervisors begin using it for forward looking decisions. Consider the following scenarios that can be modeled directly within the Planning interface:

Enforcement Forecasting: Based on the current data showing multiple platforms in the Very High risk category, supervisors can forecast the expected volume of warning letters and administrative sanctions for the coming quarter. If historical patterns show that each Very High platform typically receives two to three rounds of correspondence before resolution, the planning sheet can project staffing requirements for the enforcement team.

Budget Allocation: The Planning sheet can incorporate budget dimensions alongside risk metrics. If the current quarterly examination budget allows for on site visits to 15 platforms, the risk category column helps prioritize which platforms should be visited first. The forecast capability can then project whether the budget is sufficient given the current risk trajectory, or whether a reallocation request should be submitted.

Microsoft Fabric Operations Agent Step by Step Walkthrough

NaufalPrawironegoro — Sat, 18 Apr 2026 07:00:41 GMT

Fabric Capacity and Workspace

You need a Microsoft Fabric workspace backed by a paid capacity. Trial capacities are not supported for Operations Agent. Your capacity must be provisioned in a supported region. As of April 2026, Operations Agent is available in all Microsoft Fabric regions except South Central US and East US. If your capacity is outside the US or EU, you will also need to enable cross geo processing and storage for AI through the tenant settings.

Your workspace must contain an Eventhouse with at least one KQL database. The Eventhouse is the telemetry backbone, and the KQL database holds the tables the agent will monitor. In the screenshot below, you can see a workspace named OperationAgent-WS that contains an Eventhouse (ops_eventhouse), two KQL databases (ops_db and ops_eventhouse), and a Lakehouse (ops_lakehouse). This is the environment used throughout this guide.

Figure 1. Workspace contents showing the Eventhouse, KQL databases, and Lakehouse ready for the Operations Agent.

Enabling the Operations Agent in the Admin Portal

A Fabric administrator must enable the Operations Agent preview toggle in the Admin Portal before anyone in the organization can create an agent. Navigate to the Admin Portal, locate the section for Real Time Intelligence, and find the setting labeled Enable Operations Agents (Preview). Toggle it to Enabled for the entire organization or for specific security groups depending on your governance requirements.

In addition to this toggle, ensure that Microsoft Copilot and Azure OpenAI Service are also enabled at the tenant level. The Operations Agent relies on Azure OpenAI to generate its playbook and to reason about data when conditions are met.

Figure 2. The Admin Portal showing the Enable Operations Agents (Preview) toggle set to Enabled for the entire organization.

Note that messages sent to Operations Agents are processed through the Azure AI Bot Service. If your capacity is outside the EU Data Boundary, data may be processed outside your geographic or national cloud boundary. Be sure to communicate this to your compliance stakeholders before enabling the feature in production tenants.

Microsoft Teams Account

Every person who will receive recommendations from the agent must have a Microsoft Teams account. The Operations Agent delivers its findings and action suggestions through a dedicated Teams app called Fabric Operations Agent. You can install this app from the Teams app store by searching for its name. Once installed, the agent will be able to send messages containing data summaries and recommended actions directly to the designated recipients.

Creating and Configuring the Operations Agent

With your prerequisites in place, you are ready to create the Operations Agent. The following steps walk you through the entire configuration process using the Fabric portal.

Step 1: Create a New Operations Agent

Open the Microsoft Fabric portal and navigate to your workspace. On the Fabric home page, select the ellipsis icon and then select Create. In the Create pane, scroll to the Real Time Intelligence section and select Operations Agent. A dialog will appear asking you to name your agent and select the target workspace. Choose a descriptive name that reflects the agent’s purpose. In this guide, the agent is named OperationsAgent_1 and is deployed to the OperationAgent-WS workspace.

Step 2: Define Business Goals and Agent Instructions

Once the agent is created, you are taken to the Agent Setup page. This page is divided into two halves. On the left side, you configure the agent’s behavior. On the right side, you see the generated Agent Playbook after saving.

The first field is Business Goals, where you describe the high level objective the agent should accomplish. Write this in clear, outcome oriented language. In this demo, the business goal is set to:

“Monitor data pipeline execution and alert on failures.”

The second field is Agent Instructions, where you provide more specific guidance on how the agent should reason about the data. Think of this as a brief you would hand to an analyst who will be watching your systems overnight. Be explicit about the table name, the column to watch, and the condition that constitutes an alert. In this demo, the instruction reads:

“Monitor pipeline_runs table. Alert when status is failed.”

Together, the business goals and instructions give the underlying large language model enough context to generate an accurate playbook. The more specific your instructions, the more reliable the agent’s behavior will be.

Figure 3. The Agent Setup page showing business goals, agent instructions, and the generated playbook on the right.

On the right side of the screen, you can see the Agent Playbook that was generated after saving. The playbook includes a Business Term Glossary, which shows the business objects the agent inferred from your goals and data. In this case, it identified an object called PipelineRun, mapped to the pipeline_runs table, with two properties: status (the pipeline run status from the status column) and runId (the unique identifier from the run_id column). It also displays the Rules section, which contains the conditions the agent will evaluate.

Review the playbook carefully. Since it is generated by an AI model, there may be occasional misinterpretations. Verify that every property maps to the correct column and that the rules reflect your intended thresholds. If something is off, update your goals or instructions and save again to regenerate the playbook.

Step 3: Add a Knowledge Source

Scroll down on the Agent Setup page to find the Knowledge section. This is where you connect the agent to the data it will monitor. When you first open this section, it will display a message indicating that no knowledge source has been added yet.

Figure 4. The Knowledge section before any data source has been added.

Select the Add Data button to browse the available data sources. A panel will appear listing the KQL databases and Eventhouses accessible within your Fabric environment. In this demo, three sources are available: ops_db in the OperationAgent-WS workspace, wms_eventhouse in the WMS-CDC-Demo workspace, and ops_eventhouse in the OperationAgent-WS workspace. Select the database that contains the table you want the agent to monitor. For this guide, select ops_db, which holds the pipeline_runs table referenced in the agent instructions.

Figure 5. Selecting the knowledge source from available KQL databases and Eventhouses.

Once the knowledge source is connected, the agent will be able to query this database at regular intervals (approximately every five minutes) to evaluate its rules. Make sure the table in your selected database is actively receiving data, especially if you plan to demonstrate the agent detecting a condition in real time.

Step 4: Define Actions

Actions are the responses the agent can recommend when it detects a condition that matches its rules. Scroll further down the Agent Setup page to find the Actions section. Select the Add Action button to define a new custom action.

A dialog titled New Custom Action will appear. It has three fields. The Action Name is a short, descriptive label for the action. The Action Description explains the purpose of the action and gives the agent context about when to use it. The Parameters section allows you to define input fields that pass dynamic values (such as names, dates, or identifiers) into the Power Automate flow that will be triggered.

Figure 6. The New Custom Action dialog where you define the action name, description, and optional parameters.

In this demo, the action is named Send Email Alert with a description indicating that it should send an email notification when a pipeline failure is detected. Once created, you can see the action listed in the Actions section with a green status indicator showing that the action is successfully connected.

Figure 7. The Actions section showing the Send Email Alert action with a connected status.

Step 5: Configure the Custom Action with Power Automate

After creating the action, you need to configure it by linking it to an activator item and a Power Automate flow. Select the action you just created to open the Configure Custom Action pane.

In this pane, you will see several fields. First, select the Workspace where the activator item resides. In this demo, the workspace is OperationAgent-WS. Next, select the Activator, which is the Fabric item that bridges the Operations Agent and Power Automate. Here, the activator is named Email_Alert_Activator.

Once the connection is created, a Connection String is generated. This string is a unique identifier that links the Operations Agent to the Power Automate flow. Select the Copy button to copy this connection string to your clipboard. You will need it in the next step.

Below the connection string, you will find the Open Flow Builder button. Select this to launch the Power Automate flow designer where you will build the email notification flow.

Figure 8. The Configure Custom Action pane showing the workspace, activator, connection string, and the button to open the flow builder.

Step 6: Build the Power Automate Flow

When you select Open Flow Builder, a new browser tab opens with the Power Automate designer. The flow is pre-configured with a trigger called When an Activator Rule is Triggered. This trigger fires whenever the Operations Agent approves an action.

In the Parameters tab of the trigger, you will see a field labeled Connection String. Paste the connection string you copied from the previous step into this field. This is the critical link that connects the Power Automate flow back to your Operations Agent. If this string is incorrect or missing, the flow will not fire when the agent recommends the action.

Figure 9. The Power Automate flow builder with the activator trigger and the Connection String field.

Below the trigger, you can add any actions your workflow requires. For an email alert scenario, add an Office 365 Outlook action to send an email to the operations team. You can use dynamic content from the trigger to include details such as the pipeline run ID, the failure status, and any parameters passed through from the Operations Agent.

Save the flow and return to the Fabric portal. Your action is now fully configured and ready to be triggered by the agent.

Step 7: Generate the Playbook and Start the Agent

With all configuration complete (business goals, instructions, knowledge source, and actions), select Save on the Agent Setup page. Fabric will use the underlying large language model to generate the agent’s playbook. The playbook is a structured summary of everything the agent knows: its goals, the properties it monitors, and the rules it evaluates.

You can also select Generate Playbook at the top of the page to regenerate the playbook if you have made changes. Review the playbook one final time to confirm that properties map correctly to your table columns and that rules reflect the exact conditions you want to monitor.

When you are satisfied, select Start in the toolbar at the top of the page. The agent will begin actively monitoring your data. It queries the knowledge source approximately every five minutes, evaluating the playbook rules against the latest data. If a condition is met, the agent uses the LLM to summarize the data, generate a recommendation, and send a message to the designated recipients through Microsoft Teams.

To pause the agent at any time, select Stop. This is useful during demos when you want to control the timing of the demonstration.

How the Agent Operates at Runtime

Once started, the Operations Agent follows a continuous loop. Every five minutes, it queries the connected KQL database to evaluate the rules defined in the playbook. If no conditions are met, it continues silently. If a condition is matched (for example, a pipeline run with a status of "failed" appears in the pipeline_runs table), the agent proceeds through the following sequence.

First, the agent uses the large language model to analyze the data that triggered the condition. It summarizes the context, identifies the relevant business object (such as a specific pipeline run), and determines which action to recommend.

Second, the agent sends a message to the designated recipients through Microsoft Teams. This message contains a summary of the detected insight, the data context that triggered it, and a suggested action. Recipients can approve the action by selecting Yes or reject it by selecting No. If parameters are included (such as a run ID or a severity level), they can be reviewed and adjusted before final approval.

Third, if the recipient approves the action, the agent executes it on behalf of the creator using the creator’s credentials. In this demo, approving the action would trigger the Power Automate flow that sends an email alert.

It is important to note that if a recommendation is not responded to within three days, the operation is automatically canceled. After cancellation, the action can no longer be approved or interacted with.

Azure Managed Redis & Azure Databricks: Real-time Feature Serving for Low-Latency Decisions

Jason_Pereira — Wed, 25 Mar 2026 16:00:00 GMT

This blog content has been a collective collaboration between the Azure Databricks and Azure Managed Redis Product and Product Marketing teams.

Executive summary

Modern decisioning systems, fraud scoring, payments authorization, personalization, and step-up authentication, must return answers in tens of milliseconds while still reflecting the most recent behavior. That creates a classic tension: lakehouse platforms excel at large-scale ingestion, feature engineering, governance, training, and replayable history, but they are not designed to sit directly on the synchronous request path for high-QPS, ultra-low-latency lookups. This guide shows a pattern that keeps Azure Databricks as the primary system for building and maintaining features, while using Azure Managed Redis as the online speed layer that serves those features at memory speed for real-time scoring.

The result is a shorter and more predictable critical path for your application: the Payment API (or any online service) reads features from Azure Managed Redis and calls a model endpoint; Azure Databricks continuously refreshes features from streaming and batch sources; and your authoritative systems of record (for example, account/card data) remain durable and governed. You get real-time responsiveness without giving up data correctness, lineage, or operational discipline.

What each service does

Azure Databricks is a first-party analytics and AI platform on Azure built on Apache Spark and the lakehouse architecture. It is commonly used for batch and streaming pipelines, feature engineering, model training, governance, and operationalization of ML workflows. In this architecture, Azure Databricks is the primary data and AI platform environment where features are defined, computed, validated, published, as well as where governed history is retained.

Azure Managed Redis is a Microsoft‑managed, in‑memory data store based on Redis Enterprise, designed for low‑latency, high‑throughput access patterns. It is commonly used for traditional and real‑time caching, counters, and session state, and increasingly as a fast state layer for AI‑driven applications. In this architecture, Azure Managed Redis serves as the online feature store and speed layer: it holds the most recent feature values and signals required for real‑time scoring and can also support modern agentic patterns such as short‑ and long‑term memory, vector lookups, and fast state access alongside model inference.

Business story: real-time fraud scoring as a running example

Consider a payment system that must decide to approve, decline, or step-up authentication in tens of milliseconds—faster than a blink of an eye! The decision depends on recent behavioral signals, velocity counters, device changes, geo anomalies, and merchant patterns, combined with a fraud model. If the online service tries to compute or retrieve those features from heavy analytics systems on-demand, the request path becomes slower and more variable, especially at peak load. Instead, Azure Databricks pipelines continuously compute and refresh those features, and Azure Managed Redis serves them instantly to the scoring service. Behavioral history, profiles, and outcomes are still written to durable Azure datastores such as Delta tables, and Azure Cosmos DB so fraud models can be retrained with governed, reproducible data.

The pattern: online feature serving with a speed layer

The core idea is to separate responsibilities. Azure Databricks owns “building” features, ingest, join, aggregate, compute windows, and publish validated governed results. Azure Managed Redis owns “serving” features, fast, repeated key-based access on the hot path. The model endpoint then consumes a feature payload that is already pre-shaped for inference. This division prevents the lakehouse from becoming an online dependency and lets you scale online decisioning independently from offline compute.

Pseudocode: end-to-end flow (online scoring + feature refresh)

The pseudocode below intentionally reads like application logic rather than a single SDK. It highlights what matters: key design, pipelined feature reads, conservative fallbacks, and continuous refresh from Azure Databricks.

# ---------------------------- # Online scoring (critical path) # ---------------------------- function handleAuthorization(req): schemaV = "v3" keys = buildFeatureKeys(schemaV, req) # card/device/merchant + windows feats = redis.MGET(keys) # single round trip (pipelined) feats = fillDefaults(feats) # conservative, no blocking payload = toModelPayload(req, feats) score = modelEndpoint.predict(payload) # Databricks Model Serving or an Azure-hosted model endpoint decision = policy(score, req) # approve/decline/step-up emitEventHub("txn_events", summarize(req, score, decision)) # async emitMetrics(redisLatencyMs, modelLatencyMs, missCount(feats)) return decision # ----------------------------------------- # Feature pipeline (async): build + publish # ----------------------------------------- function streamingFeaturePipeline(): events = readEventHubs("txn_events") ref = readCosmos("account_card_reference") # system of record lookups feats = computeFeatures(events, ref) # windows, counters, signals writeDelta("fraud_feature_history", feats) # ADLS Delta tables (lakehouse) publishLatestToRedis(feats, schemaV="v3") # SET/HSET + TTL (+ jitter) # ----------------------------------- # Training + deploy (async lifecycle) # ----------------------------------- function trainAndDeploy(): hist = readDelta("fraud_feature_history") labels = readCosmos("fraud_outcomes") # delayed ground truth model = train(joinPointInTime(hist, labels)) register(model) deployToDatabricksModelServing(model)

Why it works

This architecture works because each layer does the job it is best at. The lakehouse and feature pipelines handle heavy computation, validation, lineage, and re-playable history. The online speed layer handles locality and frequency: it keeps the “hot” feature state close to the online compute so requests do not pay the cost of re-computation or large fan-out reads. You explicitly control freshness with TTLs and refresh cadence, and you keep clear correctness boundaries by treating Azure Managed Redis as a serving layer rather than the authoritative system of record, with durable, governed feature history and labels stored in Delta tables and Azure data stores such as Azure Cosmos DB.

Design choices that matter

Cost efficiency and availability start with clear separation of concerns. Serving hot features from Azure Managed Redis avoids sizing analytics infrastructure for high‑QPS, low‑latency SLAs, and enables predictable capacity planning with regional isolation for online services. Azure Databricks remains optimized for correctness, freshness, and re-playable history while the online tier scales independently by request rate and working set size.

Freshness and TTLs should reflect business tolerance for staleness and the meaning of each feature. Short velocity windows need TTLs slightly longer than ingestion gaps, while profiles and reference features can live longer. Adding jitter (for example ±10%) prevents synchronized expirations that create load spikes.

Key design is the control plane for safe evolution and availability. Include explicit schema version prefixes and keep keys stable by entity and window. Publish new versions alongside existing ones, switch readers, and retire old versions to enable zero‑downtime rollouts.

Protect the online path from stampedes and unnecessary cost. If a hot key is missing, avoid triggering widespread re-computation in downstream systems. Use a short single‑flight mechanism and conservative defaults, especially for risk‑sensitive decisions.

Keep payloads compact so performance and cost remain predictable. Online feature reads are fastest when values are small and fetched in one or two round trips. Favor numeric encodings and small blobs, and use atomic writes to avoid partial or inconsistent reads during scoring.

Reference architecture notes (regional first, then global)

Start with a single-region deployment to validate end-to-end freshness and latency. Co-locate the Payment API compute, Azure Managed Redis, the model endpoint, and the primary data sources for feature pipelines to minimize round trips. Once the pattern is proven, extend to multi-region by deploying the online tier and its local speed layer per region, while keeping a clear strategy for how features are published and reconciled across regions (often via regional pipelines that consume the same event stream or replicated event hubs).

Operations and SRE considerations

Layer	What to Monitor	Why It Matters	Typical Signals / Metrics
Online service (API / scoring)	End‑to‑end request latency, error rate, fallback rate	Confirms the critical path meets application SLAs even under partial degradation	p50/p95/p99 latency, error %, step‑up or conservative decision rate
Azure Managed Redis (speed layer)	Feature fetch latency, hit/miss ratio, memory pressure	Indicates whether the working set fits and whether TTLs align with access patterns	GET/MGET latency, miss %, evictions, memory usage
Model serving	Inference latency, throughput, saturation	Separates model execution cost from feature access cost	Inference p95 latency, QPS, concurrency utilization
Azure Databricks feature pipelines	Streaming lag, job health, data freshness	Ensures features are being refreshed on time and correctness is preserved	Event lag, job failures, watermark delay
Cross‑layer boundaries	Correlation between misses, latency spikes, and pipeline lag	Helps identify whether regressions originate in serving, pipelines, or models	Redis miss spikes vs pipeline delays vs API latency

Monitor each layer independently, then correlate at the boundaries. This makes it clear whether an SLA issue is caused by online serving pressure, model inference, or delayed feature publication, without turning the lakehouse into a synchronous dependency.

Putting it all together

Adopt the pattern incrementally. First, publish a small, high-value feature set from Azure Databricks into Azure Managed Redis and wire the online service to fetch those features during scoring. Measure end-to-end impact on latency, model quality, and operational stability. Next, extend to streaming refresh for near-real-time behavioral features, and add controlled fallbacks for partial misses. Finally, scale out to multi-region if needed, keeping each region’s online service close to its local speed layer and ensuring the feature pipelines provide consistent semantics across regions.

Sources and further reading

Azure Databricks documentation: https://learn.microsoft.com/en-us/azure/databricks/

Azure Managed Redis documentation (overview and architecture): https://learn.microsoft.com/azure/redis/

Azure Architecture Center: Stream processing with Azure Databricks: https://learn.microsoft.com/azure/architecture/reference-architectures/data/stream-processing-databricks

Databricks Feature Store / feature engineering docs (Azure Databricks): https://learn.microsoft.com/azure/databricks/

Announcing the New Home for the Azure Databricks Blog

AnaviNahar — Wed, 18 Mar 2026 18:44:42 GMT

We’re excited to share that the Azure Databricks blog has moved to a new address on Microsoft Tech Community Hub!

Azure Databricks | Microsoft Community Hub

Our new blog home is designed to make it easier than ever for you to discover the latest product updates, deep technical insights, and real-world best practices directly from the Azure Databricks product team. Whether you're a data engineer, data scientist, or analytics leader, this is your go-to destination for staying informed and inspired.

What You’ll Find on the New Blog

At our new address, you can expect:

Latest Announcements – Stay up to date with new features, capabilities, and releases
Best Practice Guidance – Learn proven approaches for building scalable data and AI solutions
Technical Deep Dives – Explore detailed walkthroughs and architecture insights
Customer Stories – See how organizations are driving impact with Azure Databricks

Why the Move?

This new blog gives us the flexibility to deliver a better reading experience, improved navigation, and richer content dedicated to Azure Databricks. It also allows us to bring you more frequent updates and more in-depth resources tailored to your needs.

Stay Connected

We encourage you to bookmark the new blog and check back regularly. Even better—follow along so you never miss an update. By staying connected, you’ll be among the first to hear about new features, performance improvements, and expert recommendations to help you get the most out of Azure Databricks.

👉 Follow the new Azure Databricks blog today and stay ahead with the latest announcements and best practices.

We’re looking forward to continuing this journey with you—now at our new home!

Check out the latest blogs if you haven’t already:

• Introducing Lakeflow Connect Free Tier, now available on Azure Databricks | Microsoft Community Hub

•Near–Real-Time CDC to Delta Lake for BI and ML with Lakeflow on Azure Databricks | Microsoft Community Hub

Smart Pipelines Orchestration: Designing Predictable Data Platforms on Shared Spark

Sally_Dabbah — Sun, 08 Feb 2026 09:21:51 GMT

Introduction

In mature data platforms, scaling compute is rarely the primary challenge. Shared, elastic Spark pools already provide sufficient processing capacity for most workloads. The harder problem is achieving predictable execution when multiple pipelines compete for the same resources.

In Azure Synapse, Spark pools are commonly shared across pipelines to optimize cost and utilization. While this model is efficient, it introduces a key limitation: execution order is determined by scheduling behavior, not business priority.

This post describes an orchestration pattern that makes priority explicit, allowing critical workloads to run predictably on shared Spark compute without modifying Spark code, configuration, or cluster capacity.

Goal

This work does not aim to optimize Spark performance.
Its goal is to ensure that, when pipelines share a Spark pool:

latency-sensitive workloads run first
heavy backfills do not delay critical pipelines
execution order is deterministic under contention

All of this needed to be achieved without changes to Spark configuration, notebook logic, or cluster size.

Why This Problem Occurs

In a naïve orchestration model, pipelines are triggered in parallel. From Spark’s perspective:

all jobs are equivalent
all jobs attempt to acquire executors at the same time
scheduling decisions are based on availability and timing

As a result, priority is implicit and often incorrect. A heavy workload may acquire executors before a lightweight but critical one simply because it requests more resources earlier.

This behavior is expected from Spark. The issue lies in orchestration, not in compute.

Core Concept: Priority as Execution Ordering

In shared Spark platforms, priority is enforced through execution ordering, not compute tuning.

The orchestration layer controls when workloads are admitted to shared compute. Once execution begins, Spark processes each workload normally.

This preserves Spark’s execution model while providing deterministic workload ordering.

Step 1: Workload Classification

In the demo presented in this blog, workloads are classified during configuration based on business impact:

Category	Description	Priority example
Light (critical)	SLA sensitive dashboard and downstream consumers	High priority , low resource weight(data volume)
Medium (High)	Core reporting workloads	Medium priority
Heavy(Best Effort)	Backfills and historical computes	Low priority, high resource weight(data volume)

This classification is external to Spark and external to code. It represents business intent, not implementation.

As a future phase, classification can be automated,for example, an agent may adjust priority based on observed failure rates or execution stability.
Workload classification is expressed as orchestration metadata, for example:

[ {"name":"ExecDashboard","pipeline":"PL_Light_ExecDashboard","weight":1,"tier":"Critical"}, {"name":"FinanceReporting","pipeline":"PL_Medium_FinanceReporting","weight":3,"tier":"High"}, {"name":"Backfill","pipeline":"PL_Heavy_Backfill","weight":8,"tier":"BestEffort"} ]

What Runs in Each Workload Category

All pipelines execute on the same shared Spark pool, but the work they perform differs in scope, data volume, and sensitivity to contention.

Light workloads power SLA-sensitive dashboards and downstream consumers. Their notebooks perform targeted reads with strong filtering, limited joins, and small aggregations. Execution time is short, and overall pipeline duration is dominated by executor availability rather than computation.

Medium workloads represent core reporting and analytics logic. These notebooks process larger datasets, perform joins across multiple sources, and apply aggregations that are more expensive than Light workloads but still time-bounded and business-critical.

Heavy workloads are best-effort pipelines such as backfills and historical recomputation. Their notebooks scan large data volumes, apply expensive transformations, and are optimized for throughput rather than responsiveness. These workloads tolerate delay but place significant pressure on shared compute when admitted concurrently.

All workloads use the same Spark pool, executor configuration, and runtime. The distinction reflects business intent and execution characteristics, not Spark tuning.

Example notebooks for each category are available in the accompanying GitHub repository.

Step 2: Naïve Orchestration (Baseline)

The following pipeline run illustrates the baseline behavior when all workloads are triggered in parallel against a shared Spark pool.

All Light, Medium, and Heavy pipelines are admitted concurrently. Executor acquisition and execution order depend on timing rather than business priority, resulting in non-deterministic behavior under contention.

Although Light workloads require minimal compute, they are delayed by executor contention caused by Medium and Heavy pipelines entering the Spark pool at the same time.

Step 3: Smart Orchestration (Priority-Aware)

Orchestration Model

The same child pipelines and notebooks are reused.
The parent pipeline enforces admission order:

Light (Critical)
Medium (High)
Heavy (Best Effort)

Dependencies control admission to the Spark pool. Parallelism is preserved within a priority class.

Effect on Shared Spark

Light workloads enter the Spark pool without contention
Medium workloads run after Light completes
Heavy workloads are intentionally delayed
Executor acquisition aligns with business priority

Light pipelines execute first and complete before medium pipelines are admitted. Heavy workloads run last by design. No Spark configuration changes are introduced.

The Spark pool, notebooks, and executor configuration are identical to the naïve run. Only the orchestration graph differs.

Step 4: Impact on Light Workloads

Light workloads are particularly sensitive to orchestration because their runtime is dominated by queueing time, not computation.

Comparing the naïve and priority-aware runs shows that Spark execution time is unchanged, but pipeline duration improves due to earlier admission to the Spark pool and immediate executor access

Naïve Execution

Spark execution time: short and unchanged
Pipeline duration: minutes under contention
Delay caused by executor unavailability

Smart Execution

Spark execution time: unchanged
Pipeline duration closely matches compute time
Immediate access to executors

The improvement comes from removing admission contention, not from increasing resources.

Results and Performance

Compared to naïve orchestration, priority-aware orchestration ensures that Light workloads complete in minutes rather than tens of minutes under contention, while Spark execution time itself remains unchanged. Heavy workloads no longer delay latency-sensitive pipelines, and execution order is deterministic across runs. These improvements are achieved solely by controlling admission to the shared Spark pool, without modifying Spark configuration, notebook logic, or cluster capacity.

Next Steps:

1. Optimizing Heavy Workloads

Once heavy workloads are isolated by priority, they can be optimized independently:

retries with backoff
tolerance for transient failures
increased executor counts or larger pools

Without admission control, these optimizations increase contention, with smart orchestration, they do not impact critical pipelines.

2. Moving Beyond Static Classification

In this implementation, workload classification is static and configuration-driven, which is sufficient for stabilization.

A next phase is adaptive classification:

collect execution metrics and failure rates
detect unstable pipelines
reclassify pipelines that exceed thresholds (e.g., >20% failures in a rolling window)

This prevents unstable workloads from impacting critical execution paths and makes the pipeline reliable with minimal maintenance.

3. Assisted Classification with Copilot agent

At scale, priority decisions benefit from automation. A Copilot-style agent can use historical execution data to recommend classification changes, grounding decisions in observed behavior while keeping engineers in control.

Example: Changing workload classification from Light to Medium

Consider a pipeline initially classified as Light because it powers an SLA-sensitive dashboard and typically executes quickly with minimal resource usage.

Over time, execution telemetry shows a change in behavior:

The pipeline fails in 4 of the last 10 runs due to transient Spark errors
Average duration has increased by 3×, even when admitted early
Retry attempts amplify contention for other Light workloads

Based on these signals, an automated agent flags the workload as unstable and recommends reclassifying it from Light to Medium.

After reclassification:

The pipeline is admitted after Light workloads but before Heavy workloads
It no longer blocks latency-critical paths when retries occur
Execution remains predictable, while instability is isolated from critical workloads

The notebook logic and Spark configuration remain unchanged, only the workload’s admission priority is updated via orchestration metadata.

This approach allows the platform to adapt to changing workload characteristics while preserving deterministic execution for critical pipelines.

Conclusion

Parallel execution is a default, not a strategy. In shared environments, orchestration must explicitly encode business intent rather than relying on scheduler behavior.

Enforcing priority at the orchestration layer restores predictability without sacrificing efficiency and provides a foundation for adaptive, policy-driven execution as platforms evolve.

Building a Reliable Real Time Data Pipeline with Microsoft Fabric

NaufalPrawironegoro — Tue, 27 Jan 2026 01:34:03 GMT

The Two Pillars That Determine Success

Data Quality Cannot Be an Afterthought

The most common mistake we see is treating data quality as something to address after the pipeline is running. This approach creates technical debt that compounds over time and erodes trust in your data.

Your CDC pipeline will ingest millions of events daily. Without proper validation at each layer, small issues become major problems. A single source system changing a column from integer to string can silently corrupt downstream analytics for days before anyone notices.

What you need to implement from day one:

Validation at the Bronze layer should focus on structural integrity. Every record landing in your raw layer needs verification that required fields exist, timestamps are valid, and CDC operation types are recognized.
The Silver layer is where business validation happens. Here you check referential integrity, apply domain specific rules, and flag anomalies. A customer ID that does not exist in your customer master table needs to be caught here.
Schema drift detection deserves special attention. Source systems change without warning. Your pipeline needs to detect these changes before they break downstream processes.
The quality score approach works well in practice. Rather than binary pass or fail checks, calculate a quality score for each batch. A score above 95 percent proceeds normally. Between 90 and 95 percent sends a warning. Below 90 percent halts processing.

Replication Lag Requires Active Management

The second pillar is understanding and managing replication lag. In a real time pipeline, the value of data degrades rapidly with age. A five minute delay might be acceptable for daily reporting but catastrophic for fraud detection or inventory management.

Lag accumulates at multiple points in your pipeline. There is capture lag between when a change occurs in the source database and when the CDC mechanism detects it. Processing lag occurs within Eventstream as events are transformed and routed. Ingestion lag happens between Eventstream and your destination tables. Each component adds latency, and under load, these delays compound.

Building effective lag management:

Monitor each stage independently. Knowing your total lag is useful, but knowing where lag accumulates is actionable.
Establish baselines before setting alerts. Collect at least two weeks of baseline metrics before configuring alert thresholds.
Implement automatic recovery procedures. When lag exceeds acceptable thresholds, your system should respond without waiting for human intervention.

Operational Foundations You Cannot Skip

Capacity Planning Prevents Expensive Surprises

Microsoft Fabric uses a capacity unit model where all workloads draw from a shared pool. Underprovisioning leads to throttling and failed jobs. Overprovisioning wastes budget. Start with realistic estimates based on your data volumes.

The F4 SKU handles most development and small production workloads comfortably. Medium deployments with 10 to 25 sources typically need F8. Large enterprise deployments should start at F16 and scale based on observed utilization. Watch for sustained utilization above 70 percent as a signal to consider scaling up.

Network Security Shapes Your Architecture

For production deployments handling sensitive data, network isolation is not optional. Private endpoints keep traffic on the Microsoft backbone network, eliminating exposure to the public internet. Plan your network architecture before building pipelines. Retrofitting private connectivity into an existing deployment is significantly more complex than designing it from the start.

Logging Enables Troubleshooting

When something goes wrong at 2 AM, your ability to diagnose the problem depends entirely on what information you captured beforehand. Centralized logging using Eventhouse gives you a queryable record of everything that happened across your pipeline. Log more than you think you need initially. Storage is inexpensive compared to the cost of troubleshooting without adequate information.

Key Decisions You Need to Make

Before proceeding with implementation, your team should align on several important decisions.

Data retention requirements affect storage costs and query performance. How long do you need to keep Bronze layer data versus aggregated Gold layer data?
Recovery time objectives determine how you architect for resilience. If the pipeline can be down for four hours without business impact, your approach differs from a scenario where even 15 minutes causes significant problems.
Who owns data quality shapes how you design validation and alerting. If source system teams are responsible, your pipeline detects and reports. If your team owns quality, you implement correction and enrichment.

Moving Forward

The patterns and practices in this guide reflect lessons learned from real implementations. Every organization has unique requirements, but the fundamentals of data quality, lag management, capacity planning, and operational readiness apply universally.

Start with the foundations. A pipeline that handles one source reliably is more valuable than one that theoretically handles fifty but fails unpredictably. Build observability from day one. Automate responses to common problems. Document what you learn.

Your data is a strategic asset. The pipeline that delivers it reliably deserves the same careful engineering you would apply to any critical business system.

Analytics on Azure Blog articles

Migrating Azure Cosmos DB Across Regions

The Three Migration Patterns at a Glance

Before you migrate: establish the decision inputs

Current Environment Setup

Approach 1: Cross-region point-in-time restore

Approach 2: Online container copy with minimal downtime

Approach 3: Convert to provisioned throughput and use multi-region replication

Decision framework

References

Optimizing Oracle-to-Power BI Performance and the Path to Modern Analytics

The Performance Challenge

Performance Is a Pipeline, Not a Single Component

Start With the Oracle Provider

Understand FetchSize

Examine Oracle Data Types

Filter Rows and Columns as Early as Possible

Preserve Query Folding

Be Careful With Native SQL and Incremental Refresh

Prefer Import for Traditional Oracle BI Workloads

Make Incremental Refresh Truly Incremental

Treat the Gateway as Its Own Performance Layer

Optimize DirectQuery If It Cannot Be Avoided

Collect Diagnostics Before Declaring a Root Cause

Future Architecture: Oracle Mirroring in Microsoft Fabric

Recommended Architecture Decision Tree

A Practical Troubleshooting Sequence

Final Takeaway

Azure Databricks Cost and Workload Assessment: a Local Web UI

Table of Contents

Introduction

1. What you can assess

2. The five-step workflow

Configure

Validate

Run analysis

Visualize results

Overview: understand the cost

Technical: inspect resources and workloads

Decisions: check findings and plan the next steps

Review & export

3. Reopen saved assessments

4. Architecture and collection flow

5. Get started

Try it

Learn more

Contribute

Learn What to Do When You Hit Capacity in Azure Databricks!

Start Here: Engage Microsoft

What “Capacity” Really Means: A Layered Mental Model

Recognizing When You’ve Hit Capacity

Immediate Actions: How to Unblock Your Workloads

Why Adding More Nodes Is Not Always the Answer

What to Do When Your Capacity Is Approved

Designing for Resilience: Long-Term Best Practices

Final Takeaways

FAQ

Complete Guide to Azure Databricks Cost Optimization for Data Engineers

Phase 1: Discovery

Assessing Your Current State

Understanding Databricks Cost Structure

Diagnose Cost and Issues

1. Identify Costly Workloads

Now That You’ve Identified Long-Running Workloads, Review These Key Areas:

2. Review Cluster Metrics

3. Review SQL Warehouse Metrics

4. Review Spark UI

4.1. Spark UI: Storage & Jobs Tab

4.2. Spark UI: Executor Tab

5. Additional Diagnostic Methods

Phase 2: Cluster/Code/Data Best Practices Alignment

Cluster UI Configuration and Cost Attribution

Key Configuration Strategies

Code Best Practices

Databricks Code Performance Enhancements & Data Engineering Best Practices

Phase 3: Team Alignment and Next Steps

Implementing Cost Observability and Taking Action

Cost Observability with System Tables

Enable Tags for Granular Tracking

Track Costs with Dashboards and Alerts