ACID Transactions on Azure Databricks: What Breaks Without Retry Logic
A warehouse has always guaranteed that updating two tables together either both succeed or both roll back. A lakehouse, historically, has not. Multi-table, multi-statement transactions reaching general availability on Unity Catalog close that gap directly on Delta Lake and Apache Iceberg tables, without a separate transactional system sitting in front of the lake. Transactions on Azure Databricks reached general availability on October 16, 2026. I covered the headline already in a shorter piece. This is the expanded version: both transaction modes in full, the four documented conflict scenarios, a staging-and-validation ETL pattern end to end, JDBC and Python client code with retry logic added on top of what the documentation shows, and the specific limitations worth knowing before a production pipeline leans on this. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload a diagram showing two tables being updated inside one atomic boundary, with a commit arrow joining both, or a screenshot of a BEGIN ATOMIC block running in the SQL Editor. Suggested alt text: "Two tables updated inside a single atomic transaction boundary on Azure Databricks".] 1. Why this used to need two separate systems Moving money between two accounts and writing an audit row look like three small operations, but a lakehouse without native transactions only gave you ACID guarantees per statement, per table. If the audit insert failed after both balance updates had already landed, nothing rolled the first two back automatically. Teams built custom reconciliation jobs to detect and repair that drift, or they ran the whole workflow on a separate warehouse system and synced results back into the lake afterward, duplicating both the data and the governance policy that protected it. Multi-statement, multi-table transactions remove that second system. The requirement is narrow but firm: every table written to must be a Unity Catalog managed table, Delta Lake or Iceberg, with catalog commits enabled. ALTER TABLE main.default.accounts SET TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported'); My take: that single table property is doing more work than its one line suggests. It is the same catalog commits feature that cross-engine ABAC depends on for external reads, which means a table you enable for transactions is, incidentally, already a step closer to being safely readable from outside Azure Databricks too. Worth enabling deliberately on tables you expect to matter for governance later, not only the ones with an immediate transactional need. 2. Two modes, and two different conflict-detection granularities Non-interactive transactions use a `BEGIN ATOMIC ... END;` block. Azure Databricks commits automatically if every statement succeeds, and rolls back automatically if any statement fails. This mode runs on any SQL warehouse, serverless compute, or a cluster on Databricks Runtime 18.0 and above, and it accepts SQL, PySpark `spark.sql()` calls, and Scala `spark.sql()` calls alike: BEGIN ATOMIC UPDATE accounts SET balance = balance - 100 WHERE id = 1; UPDATE accounts SET balance = balance + 100 WHERE id = 2; INSERT INTO audit_log VALUES (1, 2, 100, current_timestamp()); END; Interactive transactions use `BEGIN TRANSACTION`, followed by an explicit `COMMIT` or `ROLLBACK`, and they only run on a SQL warehouse: BEGIN TRANSACTION; UPDATE accounts SET balance = balance - 100 WHERE id = 1; UPDATE accounts SET balance = balance + 100 WHERE id = 2; INSERT INTO audit_log VALUES (1, 2, 100, current_timestamp()); COMMIT; The two modes are not interchangeable conveniences, they detect conflict at a different granularity. Non-interactive transactions detect conflict at the row level: two transactions touching different rows in the same table do not conflict with each other. Interactive transactions detect conflict at the table level, except for unconditional appends. If your workload has many small transactions running concurrently against a shared table, that difference alone decides whether most of them succeed or most of them fail and retry. 3. Optimistic concurrency, consistent snapshots, and four ways to conflict Neither mode locks a row while a transaction is open. Both use optimistic concurrency: the transaction runs freely, and Azure Databricks checks for conflicting changes only at commit time. If another transaction modified the same data after yours began, your commit fails and the transaction needs to be retried against fresh data. The first read of a table inside a transaction fixes a snapshot, and every later read of that same table inside the same transaction uses that same snapshot, even if another session commits a change to it in between: BEGIN TRANSACTION; SELECT * FROM products WHERE product_id = 1001; -- another session now runs, and commits: -- UPDATE products SET price = 29.99 WHERE product_id = 1001; SELECT * FROM products WHERE product_id = 1001; -- still returns the original price, the snapshot was fixed -- at the first read, not re-taken on the second one COMMIT; Four distinct scenarios can cause a commit to fail: Write-write conflicts: two transactions update or delete the same rows. Write-read conflicts: another transaction modified rows that yours read, on a table your transaction also wrote to. This applies only under Serializable isolation. Phantom read conflicts: another transaction inserted new rows matching a predicate your transaction read. This applies under both WriteSerializable and Serializable isolation. Metadata conflicts: another transaction changed the table's schema or properties mid-transaction. Uncommitted changes inside an interactive transaction are visible only to your own session. A second session querying the same table sees nothing until you run `COMMIT`, which is the isolation guarantee that keeps one session from reading data another session might still roll back. 4. Hands-on: a staging, validate, and merge pattern The pattern that benefits most from this feature is the one almost every ETL pipeline already has informally: land data in staging, check it, then merge it into production only if it passes. Written as one atomic unit, a failed check rolls back the staging load too, instead of leaving half-validated rows sitting in a staging table for the next run to trip over: BEGIN ATOMIC -- Stage today's incoming rows INSERT INTO staging_customers SELECT * FROM external_customers WHERE ingest_date = current_date(); -- Fail the whole batch if any row looks malformed IF (SELECT COUNT(*) FROM staging_customers WHERE ingest_date = current_date() AND email NOT LIKE '%@%') > 0 THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Malformed email addresses in today''s batch'; END IF; -- Only reached if validation passed MERGE INTO customers AS target USING staging_customers AS source ON target.customer_id = source.customer_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *; END; `SIGNAL` is what makes this work as a hard stop rather than a warning: raising it inside a `BEGIN ATOMIC` block triggers the same automatic rollback as any other statement failure, which means the staging insert from the first step also gets undone, not just the merge that never ran. 5. Error handling you control: SIGNAL, ROLLBACK, and GET DIAGNOSTICS Interactive transactions give you the same decision point without forcing it into a single block. You can insert, inspect the result with an ordinary query, and only then decide whether to commit or discard: BEGIN TRANSACTION; UPDATE inventory SET quantity = quantity - 50 WHERE product_id = 2001; -- inspect before deciding SELECT quantity FROM inventory WHERE product_id = 2001; -- if the number above is negative, discard everything: ROLLBACK; -- otherwise: -- COMMIT; For non-interactive transactions driven by business logic rather than manual inspection, combine `SIGNAL` with `GET DIAGNOSTICS` to retrieve how many rows the most recent statement actually touched, which is useful for asserting that an update affected exactly the rows you expected before letting the transaction proceed: BEGIN ATOMIC DECLARE rows_touched INT; UPDATE inventory SET quantity = quantity - 50 WHERE product_id = 2001; GET DIAGNOSTICS rows_touched = ROW_COUNT; IF rows_touched = 0 THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Product 2001 not found, nothing to deduct'; END IF; END; 6. Transactions from an external client, with retry logic the docs leave to you JDBC clients opt into interactive transactions by turning off autocommit explicitly: Connection conn = DriverManager.getConnection(jdbcUrl, properties); try { conn.setAutoCommit(false); Statement stmt = conn.createStatement(); stmt.executeUpdate("INSERT INTO orders (order_id, total) VALUES (1001, 500.00)"); stmt.executeUpdate("UPDATE customers SET last_order = CURRENT_DATE() WHERE customer_id = 5001"); conn.commit(); } catch (SQLException e) { conn.rollback(); throw e; } finally { conn.close(); } The Databricks SQL Connector for Python follows the same pattern with `autocommit=False` on the connection: from databricks import sql with sql.connect( server_hostname="dbc-a1b2345c-d6e7.cloud.databricks.com", http_path="sql/1.0/warehouses/abc123def456", access_token="your-access-token", autocommit=False ) as connection: with connection.cursor() as cursor: cursor.execute("INSERT INTO orders (order_id, total) VALUES (1001, 500.00)") cursor.execute("UPDATE customers SET last_order = CURRENT_DATE() WHERE customer_id = 5001") connection.commit() Neither example above retries on conflict, and under optimistic concurrency, a commit failing because another transaction got there first is not an edge case, it is the expected cost of allowing concurrency without locks. A client built on this feature needs retry logic as a first-class concern, not an afterthought: from databricks import sql import time def run_with_retry(connection_factory, statements, max_attempts=3): for attempt in range(1, max_attempts + 1): with connection_factory() as connection: with connection.cursor() as cursor: try: for statement in statements: cursor.execute(statement) connection.commit() return except Exception as exc: connection.rollback() if attempt == max_attempts: raise # Back off before retrying against fresh data, # the conflict was detected at commit time, so the # transaction must be re-run from the start, not resumed. time.sleep(0.5 * attempt) My take: treat that retry wrapper as mandatory infrastructure, not optional polish, for any application code that writes through an interactive transaction under real concurrency. A transaction that silently never retries will look correct in every single-user test and then fail intermittently the day two users actually collide, which is exactly the kind of bug that is hard to reproduce on demand and easy to misdiagnose as something else. 7. What this doesn't solve The scope has real edges that matter before redesigning a pipeline around this feature: DDL operations, `CREATE TABLE`, `ALTER TABLE`, `DROP TABLE`, must run outside any transaction. Metadata operations that enumerate objects, `SHOW TABLES`, `SHOW DATABASES`, `DatabaseMetaData` calls over JDBC, ODBC catalog functions, and queries against system tables, are not supported inside a transaction on any protocol. Time travel cannot be used inside a transaction. A single transaction can read from or write to up to 100 tables combined, and read from up to 100 views, with up to 100 intermediate commits per table. Every transaction rolls back automatically after 48 hours total duration. Interactive transactions additionally roll back after 10 minutes of inactivity, and still require an explicit `ROLLBACK` afterward to clear the session's transaction state if the session itself is still connected. Row-level concurrency for `MERGE` is not available on AWS GovCloud or on single-user clusters, where `MERGE` falls back to table-level concurrency regardless of which transaction mode you use. Reading a non-transactional external source, Parquet, Avro, CSV, JSON, or a federated JDBC table, inside a transaction requires the explicit `allow_nontransactional_read` hint, and that read is not repeatable: concurrent changes to the source during the transaction can produce an inconsistent read. That last point deserves its own code, because the failure mode if you skip the hint is a hard error, not a silent gap: BEGIN TRANSACTION; INSERT INTO transactional_table SELECT col1, col2 FROM external_json_table WITH (allow_nontransactional_read = true); COMMIT; 8. My perspective on this The detail I keep coming back to is that non-interactive and interactive transactions are not two ways to write the same thing, they are two different concurrency models wearing similar syntax. Row-level conflict detection under `BEGIN ATOMIC` and table-level detection under `BEGIN TRANSACTION` will behave identically in a demo with one writer and completely differently under real concurrent load. Picking between them by which syntax feels more familiar, rather than by which conflict granularity the workload actually needs, is the single most likely way to end up confused about why a transaction "randomly" fails in production when it never failed in testing. I would also flag the metadata-conflict scenario as the one most teams forget to test for. Write-write and phantom-read conflicts are intuitive, two things touching the same data. A schema change landing mid-transaction and invalidating an in-flight commit is a rarer trigger, but it is exactly the kind of event a migration or a parallel DDL job can cause without anyone connecting it to a transaction failing somewhere else at the same time. And the 10-minute idle timeout on interactive transactions is worth building defensively around rather than discovering in an incident: any interactive session that pauses for user input, a manual approval step, a human reading a validation result before deciding whether to commit, needs either a much shorter human-review window than 10 minutes or a design that does not leave a transaction open while waiting on that human at all. Conclusion Multi-table, multi-statement transactions do not make Azure Databricks a warehouse, they remove the specific reason a workload needed a separate warehouse system just to get atomicity across two related writes. The two modes, the four conflict scenarios, and the retry logic a real client needs to add on top of the SDK examples are the parts most likely to matter the first time this runs under genuine concurrency rather than in an isolated test. Build the retry path in from the start, choose the transaction mode based on the conflict granularity your workload actually needs, and treat the limitations list as a design constraint, not fine print to revisit only after something breaks. Official references Microsoft Learn, "Transactions - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/transactions/) Microsoft Learn, "Transaction modes - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/transactions/transaction-modes) Microsoft Learn, "Tutorial: Coordinate transactions across tables - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/transactions/tutorial) Microsoft Learn, "Catalog commits - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/tables/features/catalog-commits) Microsoft Learn, "Isolation levels and write conflicts - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/optimizations/isolation/) Microsoft Learn, "Row-level concurrency - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/optimizations/isolation/row-level-concurrency)Cross-Engine ABAC on Azure Databricks: Governing External Spark Reads
Unity Catalog's row filters, column masks, and ABAC policies have always assumed the query comes from inside Azure Databricks. Cross-engine ABAC removes that assumption, and it does it by routing the external engine's read through a Databricks-managed enforcement layer rather than by trusting the external engine to respect the policy on its own. On October 1, 2026, cross-engine attribute-based access controls (ABAC) reached general availability on Azure Databricks. I covered the headline in a shorter post already. This article goes a layer deeper: the exact requirements, the Spark configuration for both Delta and Iceberg, the one concurrency caveat worth testing before you trust this in production, and how to watch what it costs. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload a screenshot of the Unity Catalog metastore Details tab with the "External data access" toggle visible, or a diagram showing an external Spark client, the Azure Databricks serverless enforcement layer, and the underlying Delta/Iceberg table. Suggested alt text: "External Spark client querying a Unity Catalog managed table through the Azure Databricks serverless ABAC enforcement layer".] 1. Why ABAC used to stop at the Databricks engine door Every previous expansion of Unity Catalog's ABAC surface, metastore-wide policies, identity attributes from Microsoft Entra ID, context attributes that distinguish an agent from a person, explicit DENY, policies on views, shared one assumption: the engine reading the data is Azure Databricks itself. That assumption holds for most workspaces, but it breaks down the moment a genuinely multi-engine estate has an external Apache Spark job, a Flink pipeline, or a BI tool reading the same Unity Catalog managed table directly. Before cross-engine ABAC, the only way to protect that external read was to duplicate the policy logic inside the external engine itself, or to accept that the external path saw unfiltered data. Neither option scales once the policy changes on one side and someone forgets to mirror it on the other. 2. The mechanism: a serverless layer sits between the external engine and the table Cross-engine ABAC does not ask the external engine to evaluate the policy. Instead, when an external engine queries a table with an ABAC policy, a row filter, or a column mask attached, Azure Databricks routes the read through a specialized serverless compute layer. That layer evaluates the policy, filters and masks the data server-side, and only then returns the sanitized result to the external engine. The external engine never sees the unfiltered rows, and it does not need to know the policy exists. This is deliberately narrow in scope today: the only supported external engine is Apache Spark, reading either Delta tables through the Unity REST API or Iceberg tables through the Unity Catalog Iceberg REST endpoint. 3. Turning it on: metastore access, grants, and catalog commits Four things have to be true before an external Spark job can read a single filtered row. Enable external data access on the metastore This is disabled by default, on purpose, to prevent unauthorized external access. A metastore admin has to turn it on explicitly, and the option only appears for sufficiently privileged users: In a workspace attached to the metastore, open Catalog. Click the gear icon at the top of the Catalog pane and select Metastore. On the Details tab, enable External data access. Grant EXTERNAL USE SCHEMA explicitly This privilege is intentionally excluded from ALL PRIVILEGES, and a schema owner does not have it by default, both specifically to prevent accidental data exfiltration through an external path nobody was watching. Only the parent catalog owner can grant it, and it must always be granted explicitly: GRANT EXTERNAL USE SCHEMA ON SCHEMA catalog_name.schema_name TO `email address removed for privacy reasons` The querying principal also needs the ordinary SELECT on the table, USE CATALOG on its parent catalog, and USE SCHEMA on its parent schema, exactly as it would for an internal read. If the workload creates external tables rather than only reading managed ones, it additionally needs CREATE TABLE on the schema and both CREATE EXTERNAL TABLE and EXTERNAL USE LOCATION on the target external location, and EXTERNAL USE LOCATION carries the same explicit-grant-only rule as EXTERNAL USE SCHEMA: GRANT EXTERNAL USE LOCATION ON EXTERNAL LOCATION external_location_name TO `email address removed for privacy reasons` Enable catalog commits on the table Fine-grained enforcement for external reads only works against a managed table with catalog commits enabled, which requires Databricks Runtime 16.4 or above for a brand-new table, or Databricks Runtime 18.0 or above to turn it on for a table that already exists. Creating a new table with catalog commits enabled from the start: CREATE TABLE catalog_name.schema_name.table_name (id INT, name STRING) USING delta TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported'); Upgrading a table that already exists: ALTER TABLE catalog_name.schema_name.table_name SET TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported'); Only after this step can you attach an ABAC policy, a row filter, or a column mask to the table and expect an external Spark read to respect it. 4. Connecting Apache Spark: Delta and Iceberg take different configuration The two paths share the same authorization model but need different Spark packages and session configuration, so keep them separate rather than guessing at a hybrid. Delta tables, via the Unity REST API This path needs Delta-Spark 4.1 or above and Unity Catalog Spark connector 0.4 or above. Using OAuth machine-to-machine authentication, which handles credential and token renewal automatically and is the right choice for a long-running job: "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension", "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog", "spark.sql.catalog.uc_catalog": "io.unitycatalog.spark.UCSingleCatalog", "spark.sql.catalog.uc_catalog.uri": "https://adb-1234567890123456.12.azuredatabricks.net", "spark.sql.catalog.uc_catalog.auth.type": "oauth", "spark.sql.catalog.uc_catalog.auth.oauth.uri": "https://adb-1234567890123456.12.azuredatabricks.net/oidc/v1/token", "spark.sql.catalog.uc_catalog.auth.oauth.clientId": "<oauth-client-id>", "spark.sql.catalog.uc_catalog.auth.oauth.clientSecret": "<oauth-client-secret>", "spark.sql.catalog.uc_catalog.ServerSidePlanning.enabled": "true", "spark.sql.defaultCatalog": "uc_catalog", "spark.jars.packages": "io.delta:delta-spark_2.13:4.1.0,io.delta:delta-iceberg_2.13:4.1.0,io.unitycatalog:unitycatalog-spark_2.13:0.4.0,org.apache.hadoop:hadoop-azure:3.4.2" Setting ServerSidePlanning.enabled to true is not optional decoration, it is the specific switch that routes the read through the enforcement layer instead of letting Spark plan the scan on its own. Miss that one line and the external job will read the table, just without any of the row filters or column masks applied. Iceberg tables, via the Unity Catalog Iceberg REST endpoint This path needs Iceberg-Spark 1.11 or above and Apache Spark 4.0 or above: "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.uc_catalog": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.uc_catalog.type": "rest", "spark.sql.catalog.uc_catalog.uri": "https://adb-1234567890123456.12.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest/", "spark.sql.catalog.uc_catalog.credential": "<oauth-client-id>:<oauth-client-secret>", "spark.sql.catalog.uc_catalog.oauth2-server-uri": "https://adb-1234567890123456.12.azuredatabricks.net/oidc/v1/token", "spark.sql.catalog.uc_catalog.warehouse": "uc_catalog", "spark.sql.catalog.uc_catalog.cache-enabled": "false", "spark.sql.defaultCatalog": "uc_catalog" Once either configuration is in place, the query itself is ordinary Spark SQL, the enforcement is invisible from the client side: SELECT * FROM uc_catalog.schema_name.table_name; 5. What actually gets enforced, and where the hard edges are ABAC policies, standalone row filters, and column masks are all enforced on the external read, dynamically, the same way they would be for a query running inside Azure Databricks. That is the entire value proposition. But the current scope has real edges, not just minor rough patches: Only reads are supported from an external engine under fine-grained access control. To write, the writing principal must be exempted from the ABAC policy entirely, there is no write-path enforcement yet. Dynamic views are not supported. Projecting a VARIANT column is not supported. Filtering on a BINARY column is not supported. A column-masking function whose return type differs from the original column's type is not supported. Large aggregations can see real performance degradation, since every row still passes through the serverless enforcement layer before it reaches the external engine. And one warning from the documentation is worth repeating verbatim rather than paraphrasing away its force: concurrent writes during query planning can cause the same table to be read from different table snapshots within a single self-join or multi-scan query, which can produce incorrect results. If your external workload runs a self-join or anything that scans the same table more than once inside one query, and the table sees concurrent writes, test that specific pattern under write load before trusting its output in production. This is not a theoretical edge case, it is a named, documented failure mode. 6. Watching what it costs The enforcement layer is serverless compute, and Azure Databricks charges for it under its own billing line, separate from ordinary warehouse or job compute. Anyone with access to the billing system table can see exactly how much a given user's external reads are costing, broken down by SKU and by the identity that ran the query: SELECT usage_date, sku_name, identity_metadata.run_as, SUM(usage_quantity) AS dbus_consumed_by_cross_engine_abac FROM system.billing.usage WHERE usage_date BETWEEN '2026-06-01' AND '2026-07-01' AND billing_origin_product = 'EXTERNAL_COMPATIBILITY' GROUP BY 1, 2, 3 ORDER BY 1; If an external Spark job reading a heavily policy-protected table runs more often than its consumers actually need fresh data, that EXTERNAL_COMPATIBILITY line is where the cost of that habit shows up first, not in the external engine's own compute bill. 7. My perspective on this Every ABAC expansion I have covered before this one assumed the reader was inside the platform. This is the first one that assumes the opposite, and the caution in the scope reflects that: read-only, one external engine, a documented snapshot-consistency warning stated plainly rather than buried. That restraint is the right call for a first general-availability release of a capability whose entire failure mode is "the external engine saw data it should not have seen." The detail I would not skip past is ServerSidePlanning.enabled. It is one line in a long configuration block, easy to leave out by copying an older Delta-Spark config that predates this feature, and leaving it out does not produce an error, it produces an unfiltered read that looks identical to a correctly filtered one until someone notices the wrong person saw the wrong row. Treat that flag as a required field, not an optional tuning parameter, and verify it is present in code review for any external Spark job that touches a governed table. I would also budget real test time for the concurrency warning before this goes anywhere near a production pipeline that both writes to and externally reads from the same table. A silent correctness bug that only appears under concurrent writes during self-joins is exactly the kind of failure that passes every test run in a quiet environment and then shows up the first time two jobs actually overlap in production. Conclusion Cross-engine ABAC does not ask external Spark to respect your governance policy, it makes respecting that policy the external engine's only option, by interposing a serverless enforcement layer it cannot see around. The setup cost is real: a metastore-level toggle only an admin can flip, an explicit grant that intentionally does not come bundled with broader privileges, and a table property that most existing managed tables do not have yet. For a genuinely multi-engine estate, that cost is still far lower than the alternative, which was duplicating policy logic in every external engine and hoping nobody let the two copies drift apart. Official references Microsoft Learn, "Cross-engine attribute-based access controls (ABAC) - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/external-access/cross-engine-abac) Microsoft Learn, "Access Databricks tables from Delta clients - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/external-access/unity-rest) Microsoft Learn, "Enable external data access to Unity Catalog - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/external-access/admin) Microsoft Learn, "Catalog commits - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/tables/features/catalog-commits) Microsoft Learn, "Row filters and column masks - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/filters-and-masks/) Microsoft Learn, "Authorize service principal access to Azure Databricks with OAuth" (learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m)Azure Databricks Cost Governance: Budgets, Tags, and Alerts Before the Invoice Arrives
Cost visibility on Azure Databricks does not have to wait for the monthly invoice, and for Azure customers specifically, the attribution trail can run straight into Azure Cost Management, not just Databricks system tables. On October 7, 2026, Databricks published a practical guide on setting budgets and alerts for cloud data warehouse costs. The original post is written in general, cross-cloud terms. This article walks through the same problem from the Azure Databricks side specifically, using the account console, the REST API, and the system tables as they actually behave on Azure today, including two capabilities that go further than what the original post describes: per-user Unity Gateway budgets with usage blocking, and serverless usage policies whose tags propagate directly into Azure's own cost analysis tools. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload a screenshot of the Azure Databricks account console Budgets tab (Usage > Budgets), or if the editor accepts an external URL, use a relevant Microsoft Learn budgets screenshot. Suggested alt text: "Azure Databricks account console showing the Budgets tab with monthly threshold and burn-rate chart".] 1. Why the bill always arrives as a surprise The pattern repeats across data teams on any cloud: usage is not attributed to a team or a query, no team owns a budget number, the overspend only surfaces a billing cycle later, and manual spreadsheet review does not scale past a handful of warehouses. None of this is a tooling gap specific to Databricks, it is what happens when cost tracking is bolted on after the fact instead of built into how compute gets created in the first place. Azure Databricks addresses this with four layers that build on each other: tag-based attribution, the system.billing.usage table as the source of truth, budgets scoped to workspaces or tags in the account console, and alerts that run on a schedule instead of waiting for a human to check a dashboard. 2. Attribution: tags, and the Azure-native alternative to tagging by hand The baseline mechanism is the same one most clouds use: key-value tags on a SQL warehouse, set at creation or edit time, that later show up in the custom_tags column of the billing system table. Tagging a warehouse through the REST API curl -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/warehouses" \ -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "finance-analytics", "cluster_size": "Small", "warehouse_type": "PRO", "tags": { "custom_tags": [ { "key": "team", "value": "finance" }, { "key": "cost-center", "value": "1234" } ] } }' Two details matter when you actually run this. The warehouse tagging limit is under 45 tags per warehouse, and if a custom tag key collides with one of Databricks' own default tags (Vendor, ClusterId, SqlEndpointId, Creator), the custom one gets an x_ prefix during propagation rather than overwriting the default. Plan tag key names with that collision rule in mind. Query tags: tagging at the session or statement level (Public Preview) Warehouse tags attribute an entire warehouse to a team, but a shared warehouse serving multiple teams needs finer granularity than that. Query tags (Public Preview) solve this by letting you tag individual sessions or individual statements, independent of which warehouse they run on. From the SQL editor, a notebook, or a dashboard, set tags for the rest of the session with the SET QUERY_TAGS statement: -- Tag every subsequent query in this session SET QUERY_TAGS['team'] = 'marketing', QUERY_TAGS['cost_center'] = '701'; -- Inspect what's currently set SET QUERY_TAGS; -- Clear a tag when the session changes purpose SET QUERY_TAGS['cost_center'] = UNSET; Session-level tags apply to every statement that follows until changed. Statement-level tags are a separate, narrower mechanism: pass a query_tags array in the request body of the Statement Execution API, and only that one statement carries the tag, with the session reverting to its own tags afterward: curl -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/statements" \ -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "warehouse_id": "abc123", "statement": "SELECT * FROM samples.nyctaxi.trips LIMIT 10", "query_tags": [ { "key": "team", "value": "engineering" }, { "key": "env", "value": "prod" } ] }' Tags land in the query_tags column of system.query.history, which needs its own access grant from an account admin if you do not already have it. Query it directly to see spend by query rather than by whole warehouse: SELECT statement_id, query_tags, executed_by, start_time FROM system.query.history WHERE MAP_CONTAINS_KEY(query_tags, 'team') AND query_tags['team'] = 'marketing' ORDER BY start_time DESC LIMIT 100 The limits are tight enough to plan around: 20 user-specified tags per query, 10 KB of tags per session, 128 characters per key or value, and tag keys cannot contain commas, colons, hyphens, slashes, equals signs, or periods. A key starting with @@ is reserved, that is the prefix dbt and Power BI use for their own automatically applied tags (@@dbt_model_name, @@powerbi_report_id, and similar). Set a tag using a session configuration parameter instead of SQL and an invalid key gets silently dropped with a tag_invalid: true sentinel; set it through SET QUERY_TAGS and an invalid key fails the statement outright at execution time, which is the behavior worth relying on if you want bad tags to be loud rather than quiet. The Azure-specific upgrade: serverless usage policies Tagging every warehouse by hand does not scale, and nothing stops a team from forgetting it on the one warehouse that ends up costing the most. For serverless compute specifically, Azure Databricks has a more structural answer: serverless usage policies (Public Preview). A policy bundles a set of tags and gets assigned to users; any serverless notebook, job, pipeline, serving endpoint, or app that user creates inherits the policy's tags automatically, with no manual tagging step per resource. The detail that matters most for an Azure-focused reader: serverless usage policy tags propagate directly into the billing records emitted to Azure Cost Analysis, not only into Databricks' own system.billing.usage table. That means the same tag you use to scope a Databricks budget is also what your FinOps team sees inside the Azure portal's own cost views, without a separate mapping step. Known limitations worth planning around: existing notebooks, jobs, and pipelines do not retroactively pick up a policy, you have to assign it manually to each existing asset; classic (non-serverless) compute is not covered at all; and a pipeline triggered by a job does not inherit the job's policy, the pipeline needs its own. 3. The source of truth: system.billing.usage Every budget, every alert, and every dashboard ultimately reads from one Unity Catalog system table: system.billing.usage. Records typically land within 12 hours. The schema includes, among other columns, custom_tags (a map, populated by both manual warehouse tags and serverless usage policy tags), usage_metadata.warehouse_id, usage_quantity (in the unit given by usage_unit, typically DBU), and record_type, which can be ORIGINAL, RETRACTION, or RESTATEMENT when Databricks issues a billing correction. Note that query tags from the previous section live in system.query.history, not here, custom_tags on this table only reflects warehouse-level and policy-level tags. The two tables are separate for this reason, use system.query.history for query-level cost investigation and this table for warehouse and account-level aggregation. That last detail is not cosmetic. A naive SUM(usage_quantity) without accounting for retractions can overcount. The pattern that handles it correctly: SELECT usage_metadata.job_id, usage_start_time, usage_end_time, SUM(usage_quantity) AS usage_quantity FROM system.billing.usage GROUP BY ALL HAVING usage_quantity != 0 Aggregating before filtering lets the retraction record cancel out the original it corrects, so a corrected job's usage does not get double-counted. Applied to the attribution problem directly, two queries worth keeping in a cost dashboard: -- SQL warehouse spend by team, last 30 days SELECT custom_tags['team'] AS team, usage_metadata.warehouse_id, SUM(usage_quantity) AS dbus_consumed FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND usage_date >= current_date() - INTERVAL 30 DAYS GROUP BY ALL ORDER BY dbus_consumed DESC -- Unattributed SQL usage: the attribution gap, should trend to zero SELECT usage_date, SUM(usage_quantity) AS unattributed_dbus FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND (custom_tags['team'] IS NULL OR custom_tags['team'] = '') GROUP BY usage_date ORDER BY usage_date DESC For a dollar figure rather than a DBU count, join against the system.billing.list_prices system table on sku_name and the price's effective date range. I am not reproducing that join here because I have not verified its exact column names against the current schema, and getting a pricing join subtly wrong is worse than not showing it. Check the list prices system table reference on Microsoft Learn (learn.microsoft.com/en-us/azure/databricks/admin/system-tables/pricing) before writing that query yourself. 4. Budgets in the account console: more than a threshold and an email Budgets live in the Azure Databricks account console (accounts.azuredatabricks.net), under Usage > Budgets. A budget can scope to specific workspaces, to custom tags, or to both, and if you leave both empty it tracks your entire account. What the Azure documentation describes goes further than a single alert value: Up to four monthly thresholds per budget, each with its own email recipient list. Up to 1,000 budgets per account. Budgets scoped to Unity Gateway (the routing layer in front of Pay-Per-Token foundation model inference, batch ai_query, and external model provider usage) support a per-user spend threshold, on top of the shared thresholds. Cross a per-user threshold and that specific user's requests can optionally be blocked, with the option to grant named users or groups a higher override limit. Budget visibility and editing is governed by a dedicated permission role, not just account admin status, so a team lead can see their own budget without account-wide access. The important caveat, stated plainly in the documentation: budgets are a monitoring mechanism, not a hard spending cap, except specifically for Unity Gateway and Genie budgets, where the optional "Block usage" setting does enforce a stop. Even there, enforcement is approximate, based on a near-real-time cost estimate, so a user can be blocked slightly before or after the literal threshold. For standard SQL and compute budgets, nothing stops the spend, the budget's job is to get a human's attention fast. A layered pattern, adapted for Azure, that tends to hold up in practice: a BI Serving + Production budget at $8,000/month scoped to UseCase:BIServing and Env:Prod, alerting the platform team; an Analytics Ad-Hoc budget at $3,000/month scoped to UseCase:AdHoc, alerting the analytics manager; a Marketing Analytics budget at $2,500/month scoped to Team:Marketing, alerting the marketing data lead; and an account-wide safety net at $25,000/month with no tag scope, alerting the CTO and FinOps. Team-level budgets for accountability, plus one account-wide budget as the net that catches whatever slips past every tag filter. The values above are illustrative, size them against your own baseline. 5. Alerts that run before the threshold, not after Databricks SQL alerts run a query on a schedule and evaluate a condition against the result, independent of whether anyone opens a dashboard. On Azure Databricks, the current alert engine resolves to one of three states: OK, TRIGGERED, or ERROR, a simplification from the legacy alert system's extra UNKNOWN state. A daily threshold alert, scheduled once a day, built on the team-attribution query above: SELECT custom_tags['team'] AS team, SUM(usage_quantity) AS dbus_today FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND usage_date = current_date() GROUP BY ALL HAVING dbus_today > 500 -- adjust to your own per-team daily baseline A threshold alert only catches a problem on the day it happens. A pacing alert catches it earlier, by projecting month-end spend from the days elapsed so far: WITH month_to_date AS ( SELECT SUM(usage_quantity) AS mtd_dbus, DAY(current_date()) AS days_elapsed, DAY(LAST_DAY(current_date())) AS days_in_month FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND usage_date >= DATE_TRUNC('MONTH', current_date()) ) SELECT mtd_dbus, mtd_dbus / days_elapsed * days_in_month AS projected_month_end_dbus FROM month_to_date HAVING projected_month_end_dbus > 15000 -- adjust to your monthly DBU budget Schedule the pacing alert every four hours and route it to Slack or email through the alert's notification settings, the same pattern the original Databricks post recommends. Running it more often than that mostly adds noise, since system.billing.usage itself only refreshes every few hours. One cost that neither alert catches on its own: materialized view refreshes bill as serverless pipeline usage, under billing_origin_product = 'DLT', not as warehouse usage. If your team only watches SQL spend, a materialized view refreshed far more often than its source data actually changes can run invisibly. Align refresh frequency to real data freshness, and include DLT in your cost queries, not only SQL. 6. Where this still needs a human None of the above replaces judgment on the two levers that move cost the most: warehouse type and warehouse size. Azure Databricks documentation still points to SQL Serverless as the default starting point for most workloads, since it starts in seconds, scales to zero between queries, and bills only for execution time, and to predictive optimization (learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-optimization), enabled by default on accounts created after November 11, 2024, to keep OPTIMIZE, VACUUM, and clustering key maintenance running automatically on managed Unity Catalog tables, instead of a scheduled job someone has to remember to run. Auto-stop is worth checking explicitly rather than trusting the default: Pro and Classic warehouses default to 45 minutes of idle time before stopping, serverless defaults to 10 minutes, and serverless can be brought down to as low as 1 minute through the API, below what the UI exposes. For a warehouse that serves a handful of scheduled queries a day, 45 idle minutes can be the single biggest line item. 7. My perspective on this The feature set here is more capable on Azure specifically than the general cross-cloud framing suggests, and the detail that stands out to me is the Azure Cost Analysis propagation for serverless usage policy tags. That is not a Databricks convenience feature, it closes the exact gap that usually breaks cost governance in a real organization: the Databricks admin owns one view of spend, the cloud FinOps team owns another, and reconciling the two by hand is where attribution efforts actually die. If the same tag shows up in both without a manual export step, that reconciliation work goes away. I would start with serverless usage policies before hand-tagging every warehouse, specifically because the policy approach survives a new team member forgetting to tag something, while manual tagging does not. The known limitation that existing assets do not retroactively inherit a new policy is real friction, budget time for a one-time pass over your current notebooks, jobs, and pipelines when you adopt this, it will not happen automatically. And treat every budget as a monitoring signal, not a spending cap, with the explicit exception of Unity Gateway and Genie budgets where blocking is actually available. Assuming the wrong one of those two behaviors for a given budget is how a team ends up either silently overspending or unexpectedly blocked mid-production. Conclusion Cost governance on Azure Databricks is not a single feature, it is tags feeding a system table, a system table feeding budgets and alerts, and budgets feeding back into how the next warehouse gets created. The Azure-specific pieces, Unity Gateway per-user budgets with blocking and serverless usage policy tags flowing into Azure Cost Analysis, are worth adopting even if you came from the general cross-cloud version of this guide expecting less. Official references Databricks Blog, "Set Budgets and Alerts for Cloud Data Warehouse Costs" (www.databricks.com/blog/set-budgets-and-alerts-cloud-data-warehouse-costs) Microsoft Learn, "Create and monitor budgets - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/account-settings/budgets) Microsoft Learn, "Attribute usage with serverless usage policies - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/usage/budget-policies) Microsoft Learn, "Billable usage system table reference - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/system-tables/billing) Microsoft Learn, "Databricks SQL alerts - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/sql/user/alerts/) Microsoft Learn, "Use tags to attribute and track usage - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags) Microsoft Learn, "Query tags - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-tags) Microsoft Learn, "Governance Hub - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/governance-hub/)LTAP Direct Writes: Loading Terabytes into Lakebase Postgres Without Touching Your Live Application
A new beta capability that removes the classic bottleneck of bulk-loading data into a live Postgres database on Azure Databricks. On October 8, 2026, Databricks published details on LTAP Direct Writes, a Beta capability of the LTAP (Lake Transactional/Analytical Processing) architecture that accelerates large data loads into Lakebase Postgres. Until now, bulk-loading billions of rows into a live transactional Postgres database meant competing directly with the application for the same CPU and memory. Every write, regardless of its origin, has to pass through a single primary instance. One Databricks customer loading close to 1 billion rows per day through Synced Tables needed more than 8 hours per load, and had to overprovision OLTP compute just to absorb that peak without degrading the application. LTAP Direct Writes attacks this problem by moving the heavy construction work out of the primary instance entirely. Databricks reports loading 1 TB of data in under 5 minutes with this approach, up to 147 times faster, without consuming resources from the live application. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload the file, or if the editor accepts an external URL, use: https://www.databricks.com/sites/default/files/blog_images/load-tbs-in-minutes-into-postgres-blog-img-6.png . Suggested alt text: "LTAP architecture diagram showing Lakebase compute streaming the WAL to a storage layer that transcodes data into columnar Parquet, readable through Delta and Iceberg".] 1. Why a single Postgres primary cannot scale bulk loads linearly In any standard Postgres deployment, every write, no matter where it originates, passes through the primary instance. This is not a configuration limitation, it is the engine's transactional guarantee: the primary has to coordinate commit order, maintain consistency, and manage the write-ahead log as a single sequence. Throwing more parallelism at the client side does not help much when the bottleneck sits in the one process that can actually write. This is why bulk loads of billions of rows have historically competed directly with the production application for the same CPU and memory on the primary, forcing a choice between load throughput and application latency. 2. What LTAP Direct Writes changes LTAP separates compute from storage in Lakebase. As data is materialized to object storage, it is transcoded from Postgres's row-oriented representation into columnar Parquet, readable through Delta or Iceberg. This creates a single logical copy of the data, served by two specialized engines: Postgres for transactional workloads, an analytical engine for OLAP, with no external CDC pipeline to keep in sync. Synced Tables, the capability that serves lakehouse data back into Lakebase, is one concrete implementation of this architecture, and it is exactly where LTAP Direct Writes comes in to accelerate the initial load. The mechanism moves data construction out of the primary and runs it in parallel, outside of it: Parallel page building: Each Spark executor runs an isolated, sandboxed Postgres instance in binary-upgrade mode, the same mechanism `pg_upgrade` uses internally. Each sandbox reuses the destination's catalog OIDs, and the driver assigns non-overlapping OID ranges so concurrent objects cannot collide. Frozen binary COPY: Inside each sandbox, a binary `COPY` with `FREEZE` builds that executor's slice of the heap. Frozen tuples are treated as already committed, so the sandbox does not need to maintain transaction history. Per-page checksums: Each worker computes a checksum for its own constructed page. Lakebase's pageserver validates this on ingest, so a corrupted page never becomes authoritative. Index building without downloading the full heap: Each worker exports key columns and tuple IDs to object storage. A custom table access method feeds these into Postgres's standard B-tree builder, shifting each tuple pointer by the cumulative size of earlier slices. The full heap never has to be downloaded back to build the index. Atomic handoff: The Spark driver writes a manifest and calls a single SQL function. The primary records the import as a compact WAL entry, carrying only the import's description rather than the full data volume. The pageserver claims the files, warms them on local SSD, and the primary atomically swaps the staged data into the user-visible table. Until that commit, the imported table stays isolated. The team behind this is explicit that the technique uses a Postgres extension and a custom table access method, without modifying Postgres core. 3. What this unlocks for Azure Databricks users specifically For teams already running Lakebase on Azure Databricks, this closes a gap that used to force an uncomfortable choice: either accept a multi-hour load window with degraded application performance, or keep bulk analytical datasets out of Lakebase entirely and pay the cost of a separate reverse-ETL pipeline just to avoid that degradation. With Direct Writes, feeding a Lakebase-backed application with a large batch of enriched lakehouse data (a nightly scoring table, a freshly computed feature set, a reference dataset rebuilt from scratch) no longer means scheduling that load for a maintenance window. It becomes a background operation that the live application does not feel. 4. Hands-on: creating a Synced Table with Direct Writes The capability that benefits from Direct Writes today is the creation of a Synced Table. Direct Writes itself is currently a checkbox in the Create synced table dialog in the UI, not yet an explicit field documented in the CLI or SDK JSON spec, so the examples below show the general synced table creation flow that Direct Writes accelerates. Creating the synced table with the Databricks CLI databricks postgres create-synced-table main.sales.orders_sync \ --json '{ "spec": { "source_table_full_name": "main.sales.orders", "branch": "projects/my-project/branches/production", "primary_key_columns": ["order_id"], "scheduling_policy": "SNAPSHOT", "postgres_database": "mydb", "create_database_objects_if_missing": true } }' Creating the synced table with the Python SDK from databricks.sdk import WorkspaceClient from databricks.sdk.service.postgres import ( SyncedTable, SyncedTableSyncedTableSpec, SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy, ) w = WorkspaceClient() synced_table = w.postgres.create_synced_table( synced_table=SyncedTable(spec=SyncedTableSyncedTableSpec( source_table_full_name="main.sales.orders", branch="projects/my-project/branches/production", primary_key_columns=["order_id"], scheduling_policy=SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy.SNAPSHOT, postgres_database="mydb", create_database_objects_if_missing=True, )), synced_table_id="main.sales.orders_sync", ).wait() print(f"Synced table created: {synced_table.name}") Querying the loaded data directly from Postgres Once the load completes, applications query the result with a standard Postgres driver, no different from any other table: SELECT * FROM sales.orders_sync WHERE order_id = 48213; 5. Checking load status programmatically For a load that might take anywhere from seconds to minutes depending on volume, polling status through the SDK is more practical than watching the UI: from databricks.sdk import WorkspaceClient w = WorkspaceClient() table = w.postgres.get_synced_table("synced_tables/main.sales.orders_sync") print(f"State: {table.status.detailed_state}") print(f"Last sync: {table.status.last_sync_time}") print(f"Message: {table.status.message}") This is useful to wire into a Lakeflow Job as a dependency gate: downstream tasks can wait for `detailed_state` to report a completed sync before reading from the table, instead of assuming a fixed load duration. 6. What Direct Writes accelerates, and what it does not yet LTAP Direct Writes accelerates the initial load in every sync mode. For subsequent syncs, the acceleration applies specifically to Snapshot mode, because that mode performs a full refresh on every cycle. In Triggered and Continuous modes, later updates are already incremental through Change Data Feed, not bulk loads, so Direct Writes only matters for their initial load. The team behind the feature is upfront about its current scope: The 147x benchmark measures heap-building time only. Index building is not parallelized the same way yet, and the team describes this as active, ongoing work. LTAP Direct Writes is Beta, and requires a Lakebase project running Postgres 16, 17, or 18. It can only be selected when a synced table is created, not added to an existing one. To use it with an existing synced table, you delete it and create a new one. A table with many secondary indexes will likely not see the same proportional gain as the simple table used in the published benchmark, since index construction is the part still being parallelized. 7. My perspective on this change The performance number is the headline, but the architectural detail that matters more is where the heavy work moved to. The load no longer competes with your live application inside the primary instance, it becomes isolated Spark work that happens alongside it. That is the problem the cited customer actually had: not "slow load" in isolation, but "slow load competing for resources with production." I would test this first on a table without complex secondary indexes before assuming the 147x gain repeats identically on any schema, given the team's own disclosure that index building is still catching up to the heap-building speedup. For Azure Databricks teams already committed to Lakebase as their operational Postgres, this is one more argument for keeping large, bulk-refreshed datasets inside Lakebase instead of routing them through a separate reverse-ETL tool just to avoid the old load-time penalty. Conclusion LTAP Direct Writes is not an isolated feature, it is separating compute from storage applied all the way down into the bulk-load process itself. For teams already using Synced Tables and feeling the weight of large loads on the primary, it is worth testing the Beta before assuming overprovisioning the OLTP side is the only way out. Official references Databricks Blog, "Load terabytes of data in minutes into Lakebase Postgres" Microsoft Learn, "LTAP architecture - Azure Databricks" Microsoft Learn, "Serve lakehouse data with synced tables" Microsoft Learn, "Lakebase architecture - Azure Databricks" Databricks Blog, "Lakebase LTAP: rethinking database storage"Multiple Triggers in Azure Databricks Lakeflow Jobs: Simplifying Data Orchestration
A new feature that simplifies event-driven pipeline execution in Azure Databricks. On October 5, 2026, Databricks announced support for Multiple Triggers in Lakeflow Jobs, currently available in Beta. Until now, a Job could have only one configured trigger. This created an interesting limitation in data engineering projects: what happens when the same pipeline needs to run under different conditions? Imagine a data ingestion pipeline that needs to run when a new file arrives in storage, when a Delta table is updated, or every day at a specific time, regardless of those events. Previously, a common solution was to create separate Jobs, each with its own trigger, while reusing or duplicating the processing logic. Another option was to create intermediate Jobs responsible for calling the main pipeline. Both approaches work, but they introduce additional components and increase maintenance effort. With Multiple Triggers, we can now configure different execution conditions directly within the same Job. 1. How did it work before? Let's consider a pipeline responsible for processing sales data. This pipeline has three requirements: - Process new files arriving in a Unity Catalog Volume. - Reprocess data whenever a source Delta table is updated. - Run every day at 8 AM to ensure scheduled processing. Before Multiple Triggers, each Job supported only one trigger configuration. One possible approach was to create three separate Jobs, each responsible for a specific trigger type, pointing to the same processing logic. Although notebooks and tasks could be shared across these Jobs, we still needed to manage separate Job definitions, permissions, parameters, and execution settings. In enterprise environments, this complexity can grow as the number of pipelines and dependencies increases. 2. What changed with Multiple Triggers? Azure Databricks now allows us to configure up to five triggers within a single Lakeflow Job, including combinations of different trigger types. For example: Scheduled Trigger: Runs the Job at a predefined time. File Arrival Trigger: Runs when new files are detected. Table Update Trigger: Runs when specific tables are updated. Model Update Trigger: Runs based on updates to models registered in Unity Catalog. The execution logic is straightforward: when any active trigger meets its condition, it can initiate a Job run. This means we no longer need to maintain separate Jobs just to support multiple execution conditions. Another useful feature is the ability to pause or resume each trigger independently. For example, during maintenance, we can temporarily pause the Table Update Trigger while keeping the daily schedule active. 3. Practical example: Configuring Multiple Triggers Let's create a Job responsible for processing sales data. In this example, we will configure three execution conditions: Trigger 1: New files arriving in a Unity Catalog Volume. Trigger 2: Updates to the bronze.sales.orders table. Trigger 3: Daily execution at 8 AM, using the São Paulo time zone. We can configure these triggers through the Azure Databricks interface by opening the Job and adding triggers under Schedules & Triggers. We can also use the Jobs API to automate this configuration. Example using the Jobs API The following JSON illustrates a Job configuration using the triggers array. { "name": "sales_processing_job", "max_concurrent_runs": 1, "tasks": [ { "task_key": "process_sales", "notebook_task": { "notebook_path": "/Shared/pipelines/process_sales" }, "existing_cluster_id": "<CLUSTER_ID>" } ], "triggers": [ { "file_arrival": { "url": "/Volumes/bronze/sales/landing" }, "pause_status": "UNPAUSED" }, { "table_update": { "table_names": [ "bronze.sales.orders" ], "condition": "ANY_UPDATED" }, "pause_status": "UNPAUSED" }, { "schedule": { "quartz_cron_expression": "0 0 8 * * ?", "timezone_id": "America/Sao_Paulo" }, "pause_status": "UNPAUSED" } ] } In this example, we have a single Job with three independent execution conditions. The triggers field contains an array of trigger configurations, allowing multiple triggers to be defined without relying on the legacy top-level schedule, trigger, or continuous fields. The example uses an existing cluster for simplicity. In production environments, the compute configuration should be selected according to workload requirements and organizational policies. Creating the Job with Python We can submit the configuration using Python and the requests library. import os import json import requests workspace_url = os.environ["DATABRICKS_HOST"].rstrip("/") token = os.environ["DATABRICKS_TOKEN"] with open("sales_job.json", encoding="utf-8") as file: job_settings = json.load(file) response = requests.post( f"{workspace_url}/api/2.1/jobs/create", headers={ "Authorization": f"Bearer {token}", "Content-Type": "application/json" }, json=job_settings, timeout=30 ) response.raise_for_status() print("Job created:", response.json()["job_id"]) The sales_job.json file contains the JSON configuration shown earlier. For enterprise implementations, I recommend using service principals and secure authentication mechanisms instead of embedding credentials directly in the code. 4. What happens when two triggers fire at the same time? This is an important architectural consideration. Imagine a new file arriving at exactly 8 AM, when the scheduled execution is also expected to run. Both triggers may request an execution of the same Job. However, this does not necessarily mean that two runs will execute simultaneously. The behavior depends on the Job's concurrency settings. By default, a Job allows only one active run at a time. Therefore, we need to consider concurrency limits and how additional runs are handled. Another important consideration is idempotency. In simple terms, running the same pipeline multiple times against the same data should not produce duplicate records or inconsistent results. In Azure Databricks, we can use Delta Lake features and MERGE operations to implement idempotent processing strategies. Multiple Triggers improves execution flexibility, but it does not replace proper pipeline design. 5. Monitoring Jobs using System Tables One capability I find particularly useful is the possibility of monitoring trigger configurations through Databricks System Tables. The system.lakeflow.jobs table provides information about Jobs, including the trigger_type and triggers fields. When a Job has multiple triggers, the trigger_type field can contain the value MULTIPLE. We can identify these Jobs using SQL: WITH current_jobs AS ( SELECT workspace_id, job_id, name, trigger_type, triggers, ROW_NUMBER() OVER ( PARTITION BY workspace_id, job_id ORDER BY change_time DESC ) AS rn FROM system.lakeflow.jobs ) We can also expand the triggers array to inspect the individual configurations associated with each Job. WITH current_jobs AS ( SELECT workspace_id, job_id, name, triggers, ROW_NUMBER() OVER ( PARTITION BY workspace_id, job_id ORDER BY change_time DESC ) AS rn FROM system.lakeflow.jobs ) SELECT workspace_id, job_id, name, trigger FROM current_jobs LATERAL VIEW EXPLODE(triggers) AS trigger WHERE rn = 1; This approach can support governance and observability dashboards, especially in environments with hundreds or thousands of Jobs. 6. Limitations and considerations Since Multiple Triggers is currently in Beta, there are some important limitations and requirements: Each Job can have up to five triggers. The Multiple Triggers feature must be enabled in the workspace Preview settings. Each trigger can be paused or resumed independently. Continuous triggers cannot be combined with other trigger types. When updating the triggers array through the Jobs API, the entire existing trigger list is replaced. The last point is particularly important when working with infrastructure automation or CI/CD pipelines. When updating trigger configurations through the API, we need to make sure that all triggers we want to preserve are included in the new definition. 7. My perspective on this change Based on my experience working with data architecture and data engineering projects, one recurring challenge is avoiding unnecessary complexity in orchestration. The problem is not always related to data processing itself. Sometimes, it comes from the number of components we need to maintain just to execute relatively simple logic. Multiple Triggers does not change how we build tasks within a Job, but it improves how we define when those tasks should run. I see significant value in this feature for incremental ingestion, event-driven processing, and pipelines that need to combine data updates with scheduled execution. I also appreciate the ability to manage triggers independently, which can make maintenance and operational control easier. It may seem like a relatively small configuration change, but it can lead to meaningful architectural simplification in certain scenarios. Conclusion With Multiple Triggers support, Azure Databricks Lakeflow Jobs provides greater flexibility for configuring different execution conditions within a single Job. This feature reduces the need for auxiliary Jobs or duplicated Job definitions, making orchestration architectures simpler and easier to maintain. We still need to consider concurrency, idempotency, governance, and observability, but we now have another option for building event-driven pipelines without introducing unnecessary components. For anyone working with Azure Databricks, this feature is worth exploring to identify opportunities to simplify existing orchestration workflows. Official references Databricks: Automate jobs with schedules and triggers Databricks Release Notes, October 2026 Databricks Jobs API Databricks System Tables: Jobs
Engineering an Azure Data Layer for Odoo Analytics Without Changing the ERP
One thing I have learned from working on ERP-driven data projects is that the ERP is rarely the place where you want to solve every analytics problem. We saw this firsthand in a manufacturing engagement where Odoo was being used across finance, inventory, procurement, and manufacturing, but reporting still involved substantial manual extraction and Excel-based consolidation. The interesting engineering question wasn't “How do we replace the ERP?” It was: “How do we build a reliable analytical layer around it without changing the transactional system?” That shaped the architecture. We used Odoo APIs and Azure Data Factory for ingestion, landing the data in Azure Data Lake Storage Gen2. From there, the data was organized into Bronze, Silver, and Gold layers: Bronze: Source-aligned Odoo data was retained with minimal transformation. Silver: Data was cleansed, standardized, and prepared across finance, inventory, procurement, and manufacturing domains. Gold: Analytics-ready data was prepared for downstream reporting and consumption. Azure Databricks with Spark handled transformation and KPI standardization, while Synapse/SQL layers provided structured analytical models for reporting. Power BI datasets consumed the prepared data, with Row-Level Security applied for role-based access. Why did this separation matter for the project? The important architectural boundary was: Odoo → transactional operations Azure data platform → ingestion, storage, transformation, standardization Power BI → analytical consumption That separation meant the reporting layer did not have to repeatedly extract and reshape data directly from the ERP. It also gave us a dedicated place to handle historical data and standardize information across business domains before it reached the reporting layer. What changed? The implementation automated 15+ Odoo-based reports and provided governed Power BI dashboards to 60+ business users. More importantly, teams moved away from repeated ERP extraction and spreadsheet consolidation toward a reusable analytical data layer. For our team, the bigger takeaway was that ERP modernization does not necessarily mean modifying the ERP. For teams building Azure data platforms around ERP systems, how are you drawing the boundary between API ingestion, historical storage, transformation, and analytical modeling?Lakeflow in Azure Databricks
Anyone who has ever built a data engineering stack from scratch knows the pattern: one ingestion tool to pull data from CRM and ERP, a separate orchestrator to stitch together the dependencies between jobs, notebooks or scripts doing the actual transformation, and a fourth system just to check whether all of that ran correctly last night. Each piece comes from a different vendor, with its own authentication, its own logs, and its own SLA. When something breaks at three in the morning, the first job isn't fixing the problem, it's figuring out which of the four tools it lives in. Lakeflow attacks that fragmentation head-on: it brings ingestion, transformation, and orchestration together into a single surface inside Azure Databricks, with Unity Catalog providing end-to-end governance. This isn't marketing repackaging of three products that already existed separately, it's the promise that the dependency between ingestion and transformation, column-level lineage, and access control stop being reconciled by hand across systems and start existing natively in the same graph. The three pieces that make up Lakeflow Lakeflow Connect covers ingestion, with point-and-click connectors for SaaS applications (Salesforce, Workday, ServiceNow), databases (SQL Server), and messaging, plus Zerobus Ingest, a serverless direct-write API for anyone with events coming from their own applications who doesn't want to stand up a message bus in the middle. Spark Declarative Pipelines is the transformation layer: instead of hand-writing the orchestration of streaming tables and materialized views, you declare the target table and the business logic, and the engine infers the dependency graph on its own, builds the DAG, and takes care of operational details like backfill and versioning. The name has changed over time (anyone who has followed the platform for a while will recognize it as the direct evolution of what used to be Delta Live Tables), but the underlying mechanism, declaring the desired result instead of the procedural step-by-step, remains the central idea. Lakeflow Jobs orchestrates all of it as a unified DAG: SQL workloads, Python code, declarative pipelines, dashboards, and external systems live in the same job definition, with data-driven triggers (a table was updated, a file landed in a volume), control-flow tasks, and no-code backfill runs. Hands-on: a simple end-to-end pipeline To make it concrete, here's how the three pieces fit together in a basic ingestion and declarative transformation pipeline: import dlt from pyspark.sql.functions import col # Streaming table: incremental ingestion via Auto Loader @dlt.table( comment="Raw order ingestion via Auto Loader" ) def orders_bronze(): return ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .load("/Volumes/sales/bronze/orders_raw") ) # Materialized view: validation and enrichment @dlt.table( comment="Validated orders, with a quality filter" ) @dlt.expect_or_drop("positive_amount", "total_amount > 0") def orders_silver(): return ( dlt.read_stream("orders_bronze") .withColumn("total_amount", col("total_amount").cast("decimal(10,2)")) .filter(col("customer_id").isNotNull()) ) This snippet doesn't run on its own, it becomes a task inside a Lakeflow Job that can also kick off an AI/BI dashboard as soon as the "orders_silver" table is updated, using a data-driven trigger instead of a fixed time-based schedule. That binding between ingestion, transformation, and the next step in the flow, all in the same place, is what eliminates much of the glue work that today lives in a separate Airflow or Azure Data Factory. Governance without manual reconciliation The point that usually goes unnoticed by anyone looking only at the product surface is what happens underneath with Unity Catalog. Because ingestion, transformation, and orchestration share the same catalog, data lineage is captured end to end automatically, from the source table in SQL Server all the way to the final dashboard, without relying on manual annotation or on a third-party lineage system trying to infer relationships from logs. My take: this unification of governance is, in my view, Lakeflow's strongest argument, stronger even than the convenience of having a ready-made Workday connector. I've seen more than one data engineering project fail an audit not because the pipeline was wrong, but because nobody could prove with confidence where a specific number had come from, and how many transformations sat in between. Having that guaranteed structurally, rather than as a manual documentation process that is always out of date, changes the kind of conversation you have with the compliance team. System Tables come in as the observability piece: instead of building your own monitoring dashboard by querying each separate tool's API, you can build alerts and health reports directly on top of native SQL tables, with retention and schema standardized by Databricks itself. Cost: where the promise requires your own verification Databricks publishes some impressive cost-reduction figures, including a case of up to 83% lower ETL cost and up to 90x performance improvement reported by a specific customer. Those numbers deserve the standard skepticism any vendor benchmark does: they usually reflect a specific migration, from a poorly optimized stack to a new, well-tuned configuration. In practice, I'd test the real savings by running the same representative workload from your own environment on old vs. new, with equivalent cluster policies and compute mode (Performance vs. Standard on serverless compute), before using the marketing number in a budget justification to leadership. The mechanism behind the savings is real and makes technical sense: serverless compute with automatic optimization eliminates cold starts between sequential tasks in the same job, and granular per-task resource control avoids over-provisioning a cluster for a lightweight stage of the pipeline. That's different from guaranteeing that any migration will hit an 83% reduction. What this doesn't solve Consolidating ingestion, transformation, and orchestration into one platform doesn't remove the need for well-thought-out data modeling, nor does it replace architecture decisions about partitioning and clustering large tables. It also doesn't, on its own, solve the problem for teams that already have a heavy investment in Airflow with custom plugins that are hard to port, migrating legacy orchestration is still manual rewrite work, even with the dbt task and third-party integrations easing part of the path. And Lakeflow Connect's point-and-click connectors cover a specific set of sources, legacy systems or proprietary APIs without a native connector still depend on custom ingestion via Auto Loader or your own API. Bottom line Lakeflow doesn't invent a new data engineering paradigm, it removes the friction of coordinating three different tools to do what was always conceptually a single thing: bring data in from outside, transform it with confidence, and deliver it in the right shape to the next consumer. In practice: for anyone on Azure Databricks who already feels the pain of maintaining Data Factory, a third-party orchestrator, and a separate lineage catalog, it's worth piloting on a real pipeline before promising Databricks' cost-reduction number to anyone. References Databricks Blog, "Modernize your Data Engineering Platform with Lakeflow on Azure Databricks": https://www.databricks.com/blog/modernize-your-data-engineering-platform-lakeflow-azure-databricks Databricks Docs, "Get started: Build an ETL pipeline": https://docs.databricks.com/aws/en/getting-started/data-pipeline-get-started Microsoft Learn, "Tutorial: Build an ETL pipeline with Lakeflow pipelines - Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/getting-started/data-pipeline-get-started Microsoft Learn, "System tables": https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/Not Every AI Decision Needs a Chat: Running a Pure Decision Model Straight from SQL
Anyone who has ever pointed a large language model at a bulk classification job, ticket triage, content moderation, queue routing, knows the discomfort: you're paying for a model that can write poetry just to choose between "urgent" and "not urgent". A general reasoning model is great when the question is open-ended, but when the right answer already sits among a handful of known options, it turns into expensive, slow overkill. A newer class of model attacks exactly that problem: instead of generating free text, it takes a discrete set of options and returns a calibrated decision, complete with a probability per option, at a fraction of the cost and latency of a chat model. And the detail that matters to anyone working with data: you can query this kind of model directly from SQL, against tables already governed by Unity Catalog, without writing a single line of agent orchestration code. What changes when the model only needs to decide, not converse A decision model isn't a smaller version of an LLM, it's a different category built for a different purpose. It doesn't hold conversation state, doesn't generate a natural-language explanation by default, and isn't trying to be useful for just any question. It solves a narrow problem: given a fixed set of possible labels, which one is most likely for this input, and with how much confidence. That matters because most real-world classification work in production, moderation, routing, triage, risk scoring, is already fundamentally this kind of closed decision. Running a general reasoning model for that kind of task is like hiring a senior consultant to stamp a form: it works, but the cost per decision stops making sense at scale. ai_query: the bridge between the model and data that already lives in Unity Catalog Azure Databricks exposes "ai_query", a general-purpose AI Function that queries any supported model directly from SQL or Python, no separate inference pipeline required. What sets it apart from task-specific AI Functions ("ai_classify", "ai_summarize", and similar) is control: "ai_query" lets you choose the model, the prompt, and the parameters manually, exactly what you need to point at a custom decision model hosted on your own Model Serving endpoint. Two requirements worth checking before you try this: "ai_query" doesn't run on Classic SQL warehouses, it needs Serverless; and the minimum Databricks Runtime is 15.4 LTS, with 18.2 or above recommended for the best performance. Hands-on: querying a custom decision model A custom decision model, in the Jev family, typically doesn't expose a standard chat-completions interface, it's served as a regular Model Serving endpoint, and the "ai_query" call uses the custom-model syntax, with explicit "endpoint", "request", and "returnType": SELECT ticket_id, description, ai_query( endpoint => "support-triage-decision", request => named_struct( "text", description, "channel", source_channel, "customer_plan", plan ), returnType => "STRUCT<category: STRING, urgency: STRING, confidence: DOUBLE>" ) AS decision FROM support.open_tickets WHERE status = 'new'; The structured return ("category", "urgency", "confidence") is already shaped to become a table column, no free-text parsing or regex needed to pull a field out of a chat response. It's the same pattern the official documentation uses for a traditional ML model (their example is a spam classifier), just swapping the endpoint for the decision model. My take: the win here isn't just cost per call, it's architectural. Putting the decision inside SQL itself means the result is born as a column in a governed table, right where Unity Catalog's lineage, access permission, and change history already apply. Compared to a separate agent pipeline that writes back to the lakehouse afterward, that's one less failure surface. Three ways to host the model, three levels of operational effort "ai_query" accepts three categories of model, and the choice between them changes how much infrastructure someone ends up maintaining: Platform-hosted model ("system.ai.*"), like the Claude, Llama, and Qwen families already offered as a service. Nothing to provision, it scales on its own, and it's the recommended option for anyone who just needs batch inference without customizing model weights. Provisioned-throughput model, for anyone who has already fine-tuned a foundation model or needs reserved capacity. Here the Model Serving endpoint has to be created and sized manually, but "ai_query" still handles the parallelization and retries of the batch query, it doesn't use the endpoint's own capacity for that. Custom or external model, the category a decision model like Jev falls into: a traditional ML model, trained outside the foundation-model ecosystem, or hosted outside Azure Databricks via an external model endpoint. It's the option with the most control, and also the one that demands the most infrastructure decisions on your own, creating and maintaining the serving endpoint is on whoever operates it. For anyone deciding where to invest platform effort, that list already works as a priority order: start with what the platform hosts for free, and only move to the custom option once the specific task justifies the extra work of operating the endpoint. Unity Gateway governance, with a real caveat A call to "ai_query" against a hosted model ("system.ai.*") is automatically routed through Unity Gateway, but only a subset of gateway functionality applies to that route: usage tracking and budget integration (including hard spend caps) work normally, but guardrails via service policy, inference tables, tracing tables, rate limits, and fallback between providers don't apply to a call routed through "ai_query". It's easy to assume that's "already included" and find out otherwise in production. What this doesn't solve on its own "ai_query" solves how the call gets there, not the quality of the decision. A custom decision model still depends on training curation and evaluation against ground truth, the documentation itself recommends using Agent Evaluation to measure batch accuracy before trusting the result in production. And unlike the platform's natively hosted "system.ai" models, a fine-tuned or fully custom model still requires provisioning and maintaining the Model Serving endpoint yourself, without the automatic autoscaling that natively hosted models get for free. Bottom line Not every AI problem is a conversation problem. When the output is a decision within a closed set of options, a decision model queried via "ai_query" tends to cost less, respond faster, and integrate better with data that already lives in Unity Catalog than standing up one more conversational agent to do the same triage. In practice: the question worth asking before writing the next classification prompt is simple, is this a decision among known options, or is it genuinely an open-ended question? The answer completely changes which kind of model is worth paying for. References Databricks Blog, "Running open-Jev in SQL on Databricks": https://www.databricks.com/blog/running-open-jev-sql-databricks Microsoft Learn, "Use ai_query - Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/large-language-models/ai-query Microsoft Learn, "Structured outputs on Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/structured-outputs Databricks Docs, "ai_query function": https://docs.databricks.com/aws/en/sql/language-manual/functions/ai_queryA New Model Costs 60 Percent More by Default: Isolating the Financial Risk of Testing Frontier AI
Every AI model launch comes with the same marketing promise, "frontier model", and the same question with no obvious answer behind it: genuinely better, or just more expensive with better talking points? The most common response is binary, lock down access until someone validates it calmly, which slows everyone down, or open it up to the whole team to test, which has already proven expensive in practice. When one pilot ran open access to a new model with no spend control, the average developer spent 60% more than before, with no quality gain proven at that point. Databricks documented the internal process that resolved this dilemma, not by choosing between fast and safe, but by splitting the two into phases with isolated financial responsibility: experimental access on launch day for more than 12,000 employees, with cost risk contained before the quality evaluation even finishes. The problem neither open nor locked-down access solves Locking down access until full validation sounds prudent, but it pushes the decision to a committee with no real urgency to respond quickly, and in the meantime the team loses the actual advantage of testing a new model early. Opening it up with no control solves the speed problem, but overshoots on the financial side, because nobody yet knows whether that specific model delivers a quality gain proportional to the higher cost that usually comes with a new launch. The blind spot common to both extremes is treating "granting access" and "controlling spend" as the same decision, when in practice they're two problems on different timelines, access can be immediate, cost control needs to last until the real evaluation is done. Phase 1: immediate experimental access via Unity Gateway The new model enters Unity Gateway, the company's central AI governance hub, and the configuration rolls out to more than 12,000 employees via the Unity Gateway CLI, installed on laptops through mobile device management. In the interface, the model shows up flagged as "Experimental", a simple visual signal that already tells the user this isn't a validated choice yet, without hiding the option or breaking the workflow for anyone who'd rather stick with the model they already know. Phase 2: isolating financial risk with layered budgets The part that avoids repeating the earlier pilot's problem is the layered budget design, with four tiers per user: an overall monthly cap, a daily safety limit against accidental overspend, adjustable via Slack when needed, a slice reserved for a premium model two to three times more expensive, only for tasks that actually justify it, and a separate slice exclusively for the new experimental model. The key point is that the newly launched model only draws from the experimental slice, isolating the financial risk from the rest of the person's budget. The official Unity Gateway documentation details how this budget is actually configured: a budget is scoped by workspace and resource type, accepts an optional resource tag to restrict it to a specific team, and supports up to four shared limits and twenty per-user limits, each able to trigger an email alert, usage blocking, or both. Usage blocking is applied approximately, based on a near-real-time estimate, so a burst of usage can exceed the configured limit before the block actually kicks in on the next call. My take: the detail that stands out most here isn't the number of budget tiers, it's the documentation being upfront about the imprecision of the blocking mechanism itself. An "approximate" block looks like a weakness at first glance, but it's exactly the kind of limitation no team notices until they trust the number too much and get surprised by a bill slightly above the configured cap. It's worth treating this kind of budget as a safety net against gross overspend, not as a mathematical guarantee of an absolute ceiling. Hands-on: creating a Unity Gateway budget for the experimental group A budget for the team that's going to test the experimental model, with both a shared and an individual limit, and both alert and block actions configured: databricks account budgets create --json '{ "budget_configuration": { "display_name": "experimental-model-platform-team", "filter": { "workspace_id": ["all"], "tags": {"team": ["ml-platform"]} }, "resource_types": ["UNITY_AI_GATEWAY"], "alert_configurations": [ { "scope": "SHARED", "monthly_threshold_usd": 1000, "alert_emails": ["email address removed for privacy reasons"], "action": "ALERT" }, { "scope": "PER_USER", "monthly_threshold_usd": 100, "alert_emails": ["email address removed for privacy reasons"], "action": "BLOCK_USAGE" } ] } }' The practical point of this design is that the shared limit works as an early alarm for the whole team, while the per-user block is the safety net that keeps a single person from burning through the entire group's budget alone, without requiring manual review of every request. Phase 3: three signals decide promotion or discard within days The decision to keep the new model as the default or drop it combines three signals, none sufficient on its own. The first is controlled benchmarking, internal tests covering both offline evaluation, like document reasoning and in-workspace search, and side-by-side comparison on a real task, like generating a pull request. The second is qualitative feedback from power users, collected via Slack and surveys, on how the model behaves in practice compared to the previous one. The third is cost tracking via OpenTelemetry, with all Unity Gateway traffic logged in a unified trace format, normalized per session and stratified by usage pattern, so early adopters with a different usage profile than the average user don't get compared apples-to-oranges. Combining the three signals, two models tested in this cycle confirmed a real per-session cost gain over their predecessor, a 29% drop in one case and 48% in the other, enough to promote them from experimental access to standard availability within three days. What this architecture doesn't solve on its own None of this replaces already having a mature internal evaluation suite capable of consistently running offline tests and real-task side-by-side comparisons, without that the three signals become two, formal benchmarking drops out, and the decision ends up leaning on subjective opinion more than it should. Normalizing cost by usage pattern is also a modeling choice, not an objective fact, if early adopters have a systematically different usage profile than the rest of the company, the per-session cost comparison carries a bias that needs reviewing, not just blind acceptance. And the budget block being approximate by design means this architecture protects against gross overspend, not against a small, recurring overshoot that never trips the configured alert. Is this design worth adopting? The real gain here isn't releasing a new model faster by itself, it's managing to do that without repeating the uncontrolled pilot's mistake, where access speed and financial accountability ended up decoupled from each other. For a company that already tests new models frequently, splitting budget into tiers and tying promotion to a combined signal, not an isolated benchmark, is a replicable architecture even outside the Databricks ecosystem. What actually requires real investment before you can copy that three-day result is the part the documentation takes for granted, already having a mature internal benchmark and cost-tracking infrastructure in place before the next frontier model shows up. References Databricks Blog, "How Databricks rolls out frontier models to 12,000 employees on Day 1": https://www.databricks.com/blog/how-databricks-rolls-out-frontier-models-14000-employees-day-1 Microsoft Learn, "Manage budgets for Unity Gateway - Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/budgets Databricks Docs, "Manage budgets for Unity AI Gateway": https://docs.databricks.com/aws/en/ai-gateway/budgetsOne Command Opens Claude Code, Another Opens Codex: How Azure Databricks Governs Coding Agents
The best frontier model changes every few days, and that creates a structural dilemma for any company trying to give its whole team access to a coding agent. Locking into a single provider saves governance headaches, but leaves you stuck with a six-month-old choice right when something better shows up. Letting each team choose freely solves the stagnation problem, but scatters API keys, spend limits, and audit logs across a pile of disconnected systems, every coding agent with its own account, its own access secret pasted onto the developer's machine. Azure Databricks' Unity Gateway CLI attacks that dilemma without inventing a new permission system: it reuses the same Unity Catalog that already governs the rest of the company's data, and gives the developer the simplest possible experience, one command that opens the approved agent already authenticated, with no provider key pasted anywhere on the local machine. What the single command actually solves The Unity Gateway CLI (ug) is the single entry point for running coding agents against Unity Gateway: it handles OAuth, writes each agent's config file, and routes all traffic through whatever model or MCP server is already registered in the workspace. ug claude opens Claude Code, ug codex opens Codex CLI, ug gemini opens Gemini CLI, and the same CLI also covers OpenCode, GitHub Copilot CLI, and Pi, with ug --help listing the full supported set. Registering a tool follows the same simplicity pattern: ug mcp add connects a native Azure Databricks MCP server, Unity Catalog function, AI Search, SQL warehouse, or external connection already discovered in the workspace, without the developer having to wire that integration by hand. ug usage returns a seven-day consumption summary straight in the terminal, no separate dashboard needed for a quick check. The structural insight here isn't the command-line interface itself, it's what it avoids: without this central point, every team ends up pasting a provider's API key straight into each tool's local config, and the company loses aggregate visibility exactly when it needs it most, once coding-agent spend starts to scale. How the admin governs access and spend The detail that stands out most to anyone who already administers Azure Databricks: governing a coding agent doesn't require learning a new system. A model service is governed like any other protected Unity Catalog object, through the same three mechanisms as always. First, permission: in Catalog Explorer, granting EXECUTE on the model service to a user or group, under the Permissions tab, decides who can use that model through a coding agent. Second, rate limiting: still in Catalog Explorer, on the model service's Overview tab, setting a queries-per-minute (QPM) or tokens-per-minute (TPM) limit, for everyone or per user. Third, budget: an account admin creates a budget that aggregates spend across every Unity Gateway endpoint, with a configurable threshold to alert on, or block, usage before cost exceeds what was agreed. Once configured, you can confirm traffic is actually flowing through Unity Gateway and being recorded by querying the usage system table: SELECT service_name, requester, status_code, COUNT(*) AS calls FROM system.ai_gateway.usage WHERE service_type = 'MODEL_SERVICE' GROUP BY service_name, requester, status_code ORDER BY calls DESC; That means changing who has access to which model, or tightening a specific team's spend limit, is just another Unity Catalog permission change, not a parallel configuration the platform team has to learn from scratch. Smart Routing: per-task savings, still in Beta Smart Routing automatically picks the lowest-cost model capable of handling each task, so the developer doesn't have to choose a model for every prompt. The feature is in Beta and enabled per account: an account admin has to turn the preview on before any user can use it. Activation is per agent, via the CLI's own flag: ug codex --enable-smart-routing ug claude --enable-smart-routing The setting persists across sessions, and turning it off follows the same pattern (--disable-smart-routing). Two restrictions matter in practice: Smart Routing only chooses among model services prefixed system.ai, and the user needs EXECUTE on every candidate model in the routing pool, otherwise the request fails, naming exactly which model service is missing permission, not a generic error. Routing across different agents (say, from Claude Code to Codex) requires Omnigent, version 0.8.0 or later; within a single agent, routing already works on its own via the CLI. Tracking: from the usage table to OpenTelemetry Beyond querying system.ai_gateway.usage directly, Azure Databricks also exposes a ready-made dashboard: under Govern, in the top right of the Unity Gateway page, the Usage Dashboard tab includes a section dedicated to coding agents, with usage and cost charts per tool, no query needed. For anyone who wants finer granularity than the aggregate usage table, Azure Databricks also supports exporting metrics and logs via OpenTelemetry straight into a Unity Catalog managed Delta table, with the metric and log schema already defined for a CREATE TABLE. Claude Code, for example, exports this by setting CLAUDE_CODE_ENABLE_TELEMETRY and a set of OTEL_EXPORTER_OTLP_* variables pointing at the workspace's /api/2.0/otel/v1/metrics endpoint, but that path requires turning on the OpenTelemetry preview in Azure Databricks first. Hands-on: from zero to configured governance Installation requires Python 3.12 or later and uv, done once per device: uv tool install git+https://github.com/databricks/unity-gateway With that installed, the developer opens the approved agent, and on first use the CLI asks for the workspace URL and authenticates on its own: ug claude # Claude Code ug codex # Codex CLI ug gemini # Gemini CLI ug opencode # OpenCode ug copilot # GitHub Copilot CLI On the admin side, with no new API required: grant EXECUTE on the model service to a group in Catalog Explorer, set QPM or TPM on the same screen, and create an account budget covering the Unity Gateway endpoints. All three actions use the interface any Unity Catalog admin already knows, the coding agent just becomes one more consumer of the same model the platform already serves. What this doesn't solve on its own Smart Routing is still Beta, enabled per account, and restricted to regions with Unity Gateway support, so not every workspace has access today. Routing within a single agent only works if the user has EXECUTE on every candidate model, missing permission on just one candidate already breaks the request. Routing across different agents isn't native to the CLI, it depends on a specific Omnigent version. And fine-grained tracking via OpenTelemetry, while documented, requires turning on a separate preview, it doesn't come enabled by default alongside the CLI. Beyond that, not every client is covered via ug: Cursor IDE, for example, still depends on manual configuration (base URL and key pointed straight at Unity Gateway in the editor's own settings), the CLI only covers terminal-based agents. Is it worth adopting this way? The strongest point here isn't the CLI itself, it's the fact that governing a coding agent doesn't require a parallel permission system: it's the same Unity Catalog EXECUTE, the same per-model-service rate limit, the same account budget any Azure Databricks admin already uses for another workload. For a company already feeling the classic symptom, scattered provider keys, spend invisible until the bill arrives, reusing this mechanism lowers the cost of adoption precisely because it asks nobody to learn new governance, just apply what already exists to one more type of consumer. Smart Routing's payoff is real, but conditional on being available in your region and your account already having the preview turned on, so it's worth confirming that before promising the team routing savings. References Databricks Blog, "Deploy and manage coding agents at scale with the Unity Gateway CLI": https://www.databricks.com/blog/deploy-and-manage-coding-agents-scale-unity-gateway-cli Microsoft Learn, "Integrate with coding agents - Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/coding-agent-integration-model-services Microsoft Learn, "Tutorial: Govern a coding agent's model access with Unity Gateway - Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/govern-coding-agent-models Microsoft Learn, "Smart Routing for coding agents - Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/smart-routing Unity Gateway, official repository: https://github.com/databricks/unity-gateway
