Environment Variables for Serverless Jobs on Azure Databricks
Anyone who has maintained a production notebook knows the bad pattern: a variable at the top of the notebook deciding whether this run is test or production, switched manually before execution, or worse, an API token pasted directly into the code because "it's just for this one run." That works fine until someone forgets to switch the variable back, or until the notebook with the token leaks into a repository. Environment variables for serverless jobs, still in Beta in Lakeflow Jobs, attack exactly that problem by separating configuration from code. The core idea itself is not new, every mature execution platform has something similar, what is worth understanding is how Azure Databricks specifically designed the scope and precedence of this mechanism. The mechanism: a named entry at the job level, selected explicitly per task An environment variable entry is defined at the job level, not the task level, but each task chooses exactly one entry to use. An entry has a unique key and contains inline variables, a path to a .env file, or both. A task that does not declare an environment_variables_key simply receives no variables at all. The detail that most separates this design from a naive implementation: entries never merge, and one entry does not inherit from another. If two tasks in the same job need nine of the same ten variables plus one that differs, the only option is to duplicate almost the entire entry twice, not compose a base entry with an extension. This is a deliberate choice of simplicity over convenience, likely to avoid the kind of subtle bug that cascading configuration inheritance tends to produce. My take: the lack of inheritance might look like an annoying limitation at first glance, but in practice it avoids an entire class of hard-to-trace bug, "this task is using a value I never explicitly declared for it, where did it come from." In exchange for a bit of repetition in configuration, you get full predictability about exactly what each task actually receives. Hands-on: configuring it through the Jobs API Configuring this through the API makes the link between task and entry explicit. A job with two tasks, each with its own set of variables: { "name": "pipeline-with-environment-variables", "tasks": [ { "task_key": "process_orders", "notebook_task": { "notebook_path": "/Workspace/Shared/process-orders" }, "environment_key": "default", "environment_variables_key": "production" }, { "task_key": "generate_summary", "spark_jar_task": { "main_class_name": "com.company.summary.Main" }, "environment_key": "summary_environment", "environment_variables_key": "summary_vars" } ], "environments": [ { "environment_key": "default", "spec": { "environment_version": "5", "dependencies": [] } }, { "environment_key": "summary_environment", "spec": { "environment_version": "5" } } ], "environment_variables": [ { "environment_variables_key": "production", "spec": { "variables": { "APP_ENV": "production", "REGION": "us-west-2", "SERVICE_TOKEN": "{{secrets/production/service-token}}" }, "files": ["/Volumes/company/default/prod/execution.env"] } }, { "environment_variables_key": "summary_vars", "spec": { "variables": { "APP_ENV": "production", "LOG_LEVEL": "INFO" } } } ] } Notice the SERVICE_TOKEN field: it uses the {{secrets/<scope>/<name>}} syntax to reference a secret that already exists, instead of pasting the value directly into the job configuration. The value is only resolved when the task starts running, using the job's "Run as" identity, and the job configuration stores only the reference, never the resolved value. The .env file has its own format, and it is deliberately strict For variables loaded from a file, the accepted format looks simple on the surface but does not interpret anything beyond the basics: one variable per line in the format KEY=VALUE, no space before the equals sign, no quote interpretation, no escaping, no variable substitution such as ${OTHER_VARIABLE}. If you write APP_ENV="production" with quotes, the quotes become a literal part of the value. # execution.env file APP_ENV=production REGION=us-west-2 GREETING=hello world OPTIONAL_VALUE= When the same variable appears in more than one place, the precedence rule is straightforward: within a single file, the last definition wins; across files, a file listed later overrides an earlier one; and an inline variable always overrides anything coming from a file. This precedence order matters in practice because it is common to have a shared .env file across several jobs, with just one or two job-specific inline variables overriding the default. What this does not solve The list of limitations is short, but each item matters for anyone planning to adopt this at scale. User-defined functions (UDFs) cannot read any environment variable at all, because a UDF runs inside Spark's execution logic, not inside the task process, so configuration that needs to reach a UDF still depends on the older pattern of passing an explicit parameter. Only workspace-level secrets are supported for now, Unity Catalog secrets are still "on the way" with no confirmed date. And the limit of ten environment variable entries per job, twenty inline variables per entry, means a job with very granular configuration needs can hit these ceilings sooner than expected. One caution worth flagging: the documentation is honest about a point that deserves extra attention, secret redaction in logs is "best effort," not a guarantee. If the application code prints the secret value or writes it to a log, the value can appear in plain text in the compute's execution log, even though notebook output shows [REDACTED]. This means replacing a hardcoded credential with an environment variable solves the problem of credentials living in source code, but it does not by itself solve the problem of a credential leaking into logs if the application code is not careful about that separately. Is it worth adopting? Yes, especially for anyone currently maintaining multiple copies of a notebook just to switch one environment setting, or for anyone who has already been caught off guard by a forgotten token sitting inside a notebook cell. The lack of inheritance between entries will feel repetitive at first, but it is a small price for the certainty that every task runs with exactly the configuration you declared for it, nothing more and nothing less. References Microsoft Learn, "Configure environment variables for serverless jobs - Azure Databricks" (https://learn.microsoft.com/en-us/azure/databricks/jobs/environment-variables) Databricks Docs (AWS), "Environment variables" (https://docs.databricks.com/aws/en/jobs/environment-variables) Microsoft Learn, "Secret management - Azure Databricks" (https://learn.microsoft.com/en-us/azure/databricks/security/secrets/)Simplify private connectivity for Azure Databricks Serverless with Private Network Gateway
Recently, we announced the public preview of Private Network Gateway (PNG) for Azure Databricks - a simpler way to connect Azure Databricks serverless workloads to resources accessible from your Azure Virtual Network. With a single managed connection into your network, PNG helps reduce connectivity complexity while letting you control which resources serverless workloads can access. In this post, we explain what PNG means for private connectivity and how to get started. PNG brings the serverless experience together with the Azure networking environment and controls organizations already use, helping make serverless practical for a broader set of enterprise workloads. One connection to your network Azure Databricks serverless compute runs in an Azure Databricks managed network. This removes infrastructure management overhead, but private resources still need an explicit network path from serverless compute. Today, connecting serverless compute to resources you manage such as databases, Kafka clusters, and internal APIs could require separate private endpoints, load balancers, and proxy VMs for individual resources. That complexity grows as workloads need to reach more services across virtual networks, peered networks, and on-premises environments. Private Network Gateway introduces a simpler model: one gateway to your network instead of separately creating connectivity infrastructure for each resource. With Private Network Gateway on Azure Databricks, you can: Simplify private data ingestion. Use Lakeflow Connect on serverless compute to ingest from supported databases across your virtual network and connected on-premises environments. Connect AI agents to private services. An agent built on Azure Databricks can call a private MCP server you host. Use private package repositories. Install Python libraries from a private PyPI, Artifactory, or Nexus repository inside your network. Route egress through your existing firewall. Send traffic through your firewall or content inspection service so egress can be filtered and logged using your own tools. Use your own source IP. SaaS applications such as Salesforce can see a source IP belonging to your organization. Getting started is straightforward Private Network Gateway is designed to simplify the setup needed to bring serverless compute into your network. You delegate one subnet in your Azure Virtual Network to Azure Databricks. Private Network Gateway then establishes a managed, encrypted gateway from serverless compute into that subnet. The gateway is configured within a Network Connectivity Configuration (NCC), the account-level object used for Azure Databricks serverless networking. You can attach the NCC to your workspaces, and supported serverless workloads can then reach the private destinations you allow. And because the gateway is managed for you, there are no gateway VMs for you to deploy, patch, or scale in your network. Keep control of your Azure network Simplifying connectivity doesn’t mean giving up control of your network boundary. Private Network Gateway provides a dedicated path into your network rather than a shared appliance carrying other customers’ traffic. The gateway applies workspace-level access controls before traffic leaves, and isolation is enforced through encapsulation and network security rules. Once traffic reaches your subnet, your existing Azure networking controls continue to determine what serverless workloads can reach. This includes your route tables, network security groups, DNS resolvers, and security appliances. You also control how much traffic travels through the gateway: Specific destinations: Send only the fully qualified domain names you specify through Private Network Gateway. Other traffic retains its normal path. All traffic: Route outbound traffic through your network, except traffic matching a more specific Private Link or service-endpoint path. This can be used when you want broader egress inspection through your firewall or want traffic to leave through your own IP addresses. Private Network Gateway is also additive to your existing connectivity architecture. It sits alongside the private connectivity and egress controls you already use and can be enabled or removed without changing the existing setup. Bringing serverless to more of your Azure Databricks workloads Azure Databricks combines Databricks capabilities with native integration across the Azure ecosystem. Private Network Gateway extends that Azure Databricks experience to private networking for serverless workloads. Instead of building connectivity infrastructure for every resource, you can establish one managed gateway from serverless compute to your network and keep using your existing Azure network controls to decide where traffic goes. The result is a simpler approach to connecting serverless workloads to private enterprise resources while keeping the network boundary in your control. Get started today Private network gateway - Azure Databricks | Microsoft Learn Introducing Private Network Gateway | Databricks Blog
Optimize selective analytics reads in Azure Data Lake Storage with ABFS Vectored Read
ABFS Vectored Read helps compatible analytics engines reduce request overhead when reading selected portions of columnar files in Azure Data Lake Storage. By coordinating multiple noncontiguous byte ranges through the Azure Blob File System (ABFS) driver, it can combine nearby ranges into fewer storage requests and execute those requests asynchronously. For data engineers working with selective Parquet queries, this brings the storage read pattern closer to what the query needs: a coordinated set of ranges rather than a collection of independent small reads. When the Parquet reader supports vectored I/O and the runtime includes a compatible ABFS client, Spark workloads that use higher-level APIs can enable this capability through configuration - without rewriting queries, changing application logic, or migrating data. Why selective reads need coordinated I/O Consider a query that selects a few columns from a Parquet dataset and filters out most row groups. The reader can skip much of the data the query doesn't need, but the data it needs may be spread across different parts of the file. Reading each range independently can create many small storage requests. Each request carries network and processing overhead, so selecting less data doesn't necessarily remove the cost of coordinating those reads. Vectored Read lets a compatible reader provide the ranges together, giving the client an opportunity to plan the I/O as a group. How ABFS Vectored Read works A compatible reader submits the file ranges it needs together. ABFS validates and sorts the ranges, combines nearby ranges when appropriate, and divides the resulting work to fit its read buffers. It then queues the storage requests asynchronously and returns the requested bytes to the corresponding range buffers. Illustrative request pattern, not a benchmark. Actual grouping depends on range layout and client configuration; coalesced storage requests can include bytes between the requested ranges. This is a client-side optimization. It does not change the file format or rewrite your stored data. The engine still receives the data for the ranges it requested. What this means for your workloads For compatible selective-read workloads, Vectored Read offers: Less request overhead: Combining nearby ranges can reduce separate storage requests and network round trips. Coordinated I/O: Asynchronous reads allow the client to work on multiple requested ranges without treating every range as an isolated operation. Configuration-based adoption: Compatible Spark workloads using the Hadoop Parquet and ABFS path can use the feature without rewriting their queries. The best starting point is a workload that reads selected portions of Parquet files through abfs:// or abfss://. Full sequential scans may see less benefit, and engines that bypass the Hadoop/ABFS path do not use this capability. Support depends on the Hadoop, hadoop-azure, and parquet-hadoop libraries included in the runtime, not only on the Spark version. See Availability and resources for the minimum versions and verification guidance. The benefit depends on the number, size, and spacing of requested ranges, the coalescing configuration, storage and network latency, concurrency and caching. Combining ranges can reduce requests and round trips, but it can also read additional bytes between noncontiguous requested ranges. Compare query duration, storage transactions, bytes read, and resource utilization against a controlled baseline using representative data and concurrency. Get started First, verify that your runtime contains Apache Hadoop 3.3.5 or later, including the hadoop-azure module, and parquet-hadoop 1.12.3 or later. The presence of an ABFS URI alone does not establish that the optimized implementation is installed. For Spark, enable the Parquet setting before the application starts, using the spark.hadoop. prefix: spark-submit \ --conf spark.hadoop.parquet.hadoop.vectored.io.enabled=true \ <other-options> \ <application> Replace the command placeholders with your normal submission options and application. Set the property explicitly rather than relying on a default, which can vary by packaged Parquet version. For managed services, use the service's supported cluster or session configuration mechanism. Availability and resources Azure Databricks does not currently support ABFS Vectored Read because its runtime ships an ABFS version that does not include this capability. In addition, Photon does not use the ABFS driver for data access, even when a workload uses an abfs:// or abfss:// URI. Therefore, the presence of an ABFS URI alone does not indicate that Vectored Read is available. For other packaged or managed runtimes, verify that the environment includes Apache Hadoop 3.3.5 or later with the hadoop-azure module, and parquet-hadoop 1.12.3 or later. Start with one representative selective-read workload and compare it with your baseline. For background, see the ABFS driver overview.ACID Transactions on Azure Databricks: What Breaks Without Retry Logic
A warehouse has always guaranteed that updating two tables together either both succeed or both roll back. A lakehouse, historically, has not. Multi-table, multi-statement transactions reaching general availability on Unity Catalog close that gap directly on Delta Lake and Apache Iceberg tables, without a separate transactional system sitting in front of the lake. Transactions on Azure Databricks reached general availability on October 16, 2026. I covered the headline already in a shorter piece. This is the expanded version: both transaction modes in full, the four documented conflict scenarios, a staging-and-validation ETL pattern end to end, JDBC and Python client code with retry logic added on top of what the documentation shows, and the specific limitations worth knowing before a production pipeline leans on this. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload a diagram showing two tables being updated inside one atomic boundary, with a commit arrow joining both, or a screenshot of a BEGIN ATOMIC block running in the SQL Editor. Suggested alt text: "Two tables updated inside a single atomic transaction boundary on Azure Databricks".] 1. Why this used to need two separate systems Moving money between two accounts and writing an audit row look like three small operations, but a lakehouse without native transactions only gave you ACID guarantees per statement, per table. If the audit insert failed after both balance updates had already landed, nothing rolled the first two back automatically. Teams built custom reconciliation jobs to detect and repair that drift, or they ran the whole workflow on a separate warehouse system and synced results back into the lake afterward, duplicating both the data and the governance policy that protected it. Multi-statement, multi-table transactions remove that second system. The requirement is narrow but firm: every table written to must be a Unity Catalog managed table, Delta Lake or Iceberg, with catalog commits enabled. ALTER TABLE main.default.accounts SET TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported'); My take: that single table property is doing more work than its one line suggests. It is the same catalog commits feature that cross-engine ABAC depends on for external reads, which means a table you enable for transactions is, incidentally, already a step closer to being safely readable from outside Azure Databricks too. Worth enabling deliberately on tables you expect to matter for governance later, not only the ones with an immediate transactional need. 2. Two modes, and two different conflict-detection granularities Non-interactive transactions use a `BEGIN ATOMIC ... END;` block. Azure Databricks commits automatically if every statement succeeds, and rolls back automatically if any statement fails. This mode runs on any SQL warehouse, serverless compute, or a cluster on Databricks Runtime 18.0 and above, and it accepts SQL, PySpark `spark.sql()` calls, and Scala `spark.sql()` calls alike: BEGIN ATOMIC UPDATE accounts SET balance = balance - 100 WHERE id = 1; UPDATE accounts SET balance = balance + 100 WHERE id = 2; INSERT INTO audit_log VALUES (1, 2, 100, current_timestamp()); END; Interactive transactions use `BEGIN TRANSACTION`, followed by an explicit `COMMIT` or `ROLLBACK`, and they only run on a SQL warehouse: BEGIN TRANSACTION; UPDATE accounts SET balance = balance - 100 WHERE id = 1; UPDATE accounts SET balance = balance + 100 WHERE id = 2; INSERT INTO audit_log VALUES (1, 2, 100, current_timestamp()); COMMIT; The two modes are not interchangeable conveniences, they detect conflict at a different granularity. Non-interactive transactions detect conflict at the row level: two transactions touching different rows in the same table do not conflict with each other. Interactive transactions detect conflict at the table level, except for unconditional appends. If your workload has many small transactions running concurrently against a shared table, that difference alone decides whether most of them succeed or most of them fail and retry. 3. Optimistic concurrency, consistent snapshots, and four ways to conflict Neither mode locks a row while a transaction is open. Both use optimistic concurrency: the transaction runs freely, and Azure Databricks checks for conflicting changes only at commit time. If another transaction modified the same data after yours began, your commit fails and the transaction needs to be retried against fresh data. The first read of a table inside a transaction fixes a snapshot, and every later read of that same table inside the same transaction uses that same snapshot, even if another session commits a change to it in between: BEGIN TRANSACTION; SELECT * FROM products WHERE product_id = 1001; -- another session now runs, and commits: -- UPDATE products SET price = 29.99 WHERE product_id = 1001; SELECT * FROM products WHERE product_id = 1001; -- still returns the original price, the snapshot was fixed -- at the first read, not re-taken on the second one COMMIT; Four distinct scenarios can cause a commit to fail: Write-write conflicts: two transactions update or delete the same rows. Write-read conflicts: another transaction modified rows that yours read, on a table your transaction also wrote to. This applies only under Serializable isolation. Phantom read conflicts: another transaction inserted new rows matching a predicate your transaction read. This applies under both WriteSerializable and Serializable isolation. Metadata conflicts: another transaction changed the table's schema or properties mid-transaction. Uncommitted changes inside an interactive transaction are visible only to your own session. A second session querying the same table sees nothing until you run `COMMIT`, which is the isolation guarantee that keeps one session from reading data another session might still roll back. 4. Hands-on: a staging, validate, and merge pattern The pattern that benefits most from this feature is the one almost every ETL pipeline already has informally: land data in staging, check it, then merge it into production only if it passes. Written as one atomic unit, a failed check rolls back the staging load too, instead of leaving half-validated rows sitting in a staging table for the next run to trip over: BEGIN ATOMIC -- Stage today's incoming rows INSERT INTO staging_customers SELECT * FROM external_customers WHERE ingest_date = current_date(); -- Fail the whole batch if any row looks malformed IF (SELECT COUNT(*) FROM staging_customers WHERE ingest_date = current_date() AND email NOT LIKE '%@%') > 0 THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Malformed email addresses in today''s batch'; END IF; -- Only reached if validation passed MERGE INTO customers AS target USING staging_customers AS source ON target.customer_id = source.customer_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *; END; `SIGNAL` is what makes this work as a hard stop rather than a warning: raising it inside a `BEGIN ATOMIC` block triggers the same automatic rollback as any other statement failure, which means the staging insert from the first step also gets undone, not just the merge that never ran. 5. Error handling you control: SIGNAL, ROLLBACK, and GET DIAGNOSTICS Interactive transactions give you the same decision point without forcing it into a single block. You can insert, inspect the result with an ordinary query, and only then decide whether to commit or discard: BEGIN TRANSACTION; UPDATE inventory SET quantity = quantity - 50 WHERE product_id = 2001; -- inspect before deciding SELECT quantity FROM inventory WHERE product_id = 2001; -- if the number above is negative, discard everything: ROLLBACK; -- otherwise: -- COMMIT; For non-interactive transactions driven by business logic rather than manual inspection, combine `SIGNAL` with `GET DIAGNOSTICS` to retrieve how many rows the most recent statement actually touched, which is useful for asserting that an update affected exactly the rows you expected before letting the transaction proceed: BEGIN ATOMIC DECLARE rows_touched INT; UPDATE inventory SET quantity = quantity - 50 WHERE product_id = 2001; GET DIAGNOSTICS rows_touched = ROW_COUNT; IF rows_touched = 0 THEN SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'Product 2001 not found, nothing to deduct'; END IF; END; 6. Transactions from an external client, with retry logic the docs leave to you JDBC clients opt into interactive transactions by turning off autocommit explicitly: Connection conn = DriverManager.getConnection(jdbcUrl, properties); try { conn.setAutoCommit(false); Statement stmt = conn.createStatement(); stmt.executeUpdate("INSERT INTO orders (order_id, total) VALUES (1001, 500.00)"); stmt.executeUpdate("UPDATE customers SET last_order = CURRENT_DATE() WHERE customer_id = 5001"); conn.commit(); } catch (SQLException e) { conn.rollback(); throw e; } finally { conn.close(); } The Databricks SQL Connector for Python follows the same pattern with `autocommit=False` on the connection: from databricks import sql with sql.connect( server_hostname="dbc-a1b2345c-d6e7.cloud.databricks.com", http_path="sql/1.0/warehouses/abc123def456", access_token="your-access-token", autocommit=False ) as connection: with connection.cursor() as cursor: cursor.execute("INSERT INTO orders (order_id, total) VALUES (1001, 500.00)") cursor.execute("UPDATE customers SET last_order = CURRENT_DATE() WHERE customer_id = 5001") connection.commit() Neither example above retries on conflict, and under optimistic concurrency, a commit failing because another transaction got there first is not an edge case, it is the expected cost of allowing concurrency without locks. A client built on this feature needs retry logic as a first-class concern, not an afterthought: from databricks import sql import time def run_with_retry(connection_factory, statements, max_attempts=3): for attempt in range(1, max_attempts + 1): with connection_factory() as connection: with connection.cursor() as cursor: try: for statement in statements: cursor.execute(statement) connection.commit() return except Exception as exc: connection.rollback() if attempt == max_attempts: raise # Back off before retrying against fresh data, # the conflict was detected at commit time, so the # transaction must be re-run from the start, not resumed. time.sleep(0.5 * attempt) My take: treat that retry wrapper as mandatory infrastructure, not optional polish, for any application code that writes through an interactive transaction under real concurrency. A transaction that silently never retries will look correct in every single-user test and then fail intermittently the day two users actually collide, which is exactly the kind of bug that is hard to reproduce on demand and easy to misdiagnose as something else. 7. What this doesn't solve The scope has real edges that matter before redesigning a pipeline around this feature: DDL operations, `CREATE TABLE`, `ALTER TABLE`, `DROP TABLE`, must run outside any transaction. Metadata operations that enumerate objects, `SHOW TABLES`, `SHOW DATABASES`, `DatabaseMetaData` calls over JDBC, ODBC catalog functions, and queries against system tables, are not supported inside a transaction on any protocol. Time travel cannot be used inside a transaction. A single transaction can read from or write to up to 100 tables combined, and read from up to 100 views, with up to 100 intermediate commits per table. Every transaction rolls back automatically after 48 hours total duration. Interactive transactions additionally roll back after 10 minutes of inactivity, and still require an explicit `ROLLBACK` afterward to clear the session's transaction state if the session itself is still connected. Row-level concurrency for `MERGE` is not available on AWS GovCloud or on single-user clusters, where `MERGE` falls back to table-level concurrency regardless of which transaction mode you use. Reading a non-transactional external source, Parquet, Avro, CSV, JSON, or a federated JDBC table, inside a transaction requires the explicit `allow_nontransactional_read` hint, and that read is not repeatable: concurrent changes to the source during the transaction can produce an inconsistent read. That last point deserves its own code, because the failure mode if you skip the hint is a hard error, not a silent gap: BEGIN TRANSACTION; INSERT INTO transactional_table SELECT col1, col2 FROM external_json_table WITH (allow_nontransactional_read = true); COMMIT; 8. My perspective on this The detail I keep coming back to is that non-interactive and interactive transactions are not two ways to write the same thing, they are two different concurrency models wearing similar syntax. Row-level conflict detection under `BEGIN ATOMIC` and table-level detection under `BEGIN TRANSACTION` will behave identically in a demo with one writer and completely differently under real concurrent load. Picking between them by which syntax feels more familiar, rather than by which conflict granularity the workload actually needs, is the single most likely way to end up confused about why a transaction "randomly" fails in production when it never failed in testing. I would also flag the metadata-conflict scenario as the one most teams forget to test for. Write-write and phantom-read conflicts are intuitive, two things touching the same data. A schema change landing mid-transaction and invalidating an in-flight commit is a rarer trigger, but it is exactly the kind of event a migration or a parallel DDL job can cause without anyone connecting it to a transaction failing somewhere else at the same time. And the 10-minute idle timeout on interactive transactions is worth building defensively around rather than discovering in an incident: any interactive session that pauses for user input, a manual approval step, a human reading a validation result before deciding whether to commit, needs either a much shorter human-review window than 10 minutes or a design that does not leave a transaction open while waiting on that human at all. Conclusion Multi-table, multi-statement transactions do not make Azure Databricks a warehouse, they remove the specific reason a workload needed a separate warehouse system just to get atomicity across two related writes. The two modes, the four conflict scenarios, and the retry logic a real client needs to add on top of the SDK examples are the parts most likely to matter the first time this runs under genuine concurrency rather than in an isolated test. Build the retry path in from the start, choose the transaction mode based on the conflict granularity your workload actually needs, and treat the limitations list as a design constraint, not fine print to revisit only after something breaks. Official references Microsoft Learn, "Transactions - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/transactions/) Microsoft Learn, "Transaction modes - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/transactions/transaction-modes) Microsoft Learn, "Tutorial: Coordinate transactions across tables - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/transactions/tutorial) Microsoft Learn, "Catalog commits - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/tables/features/catalog-commits) Microsoft Learn, "Isolation levels and write conflicts - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/optimizations/isolation/) Microsoft Learn, "Row-level concurrency - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/optimizations/isolation/row-level-concurrency)Cross-Engine ABAC on Azure Databricks: Governing External Spark Reads
Unity Catalog's row filters, column masks, and ABAC policies have always assumed the query comes from inside Azure Databricks. Cross-engine ABAC removes that assumption, and it does it by routing the external engine's read through a Databricks-managed enforcement layer rather than by trusting the external engine to respect the policy on its own. On October 1, 2026, cross-engine attribute-based access controls (ABAC) reached general availability on Azure Databricks. I covered the headline in a shorter post already. This article goes a layer deeper: the exact requirements, the Spark configuration for both Delta and Iceberg, the one concurrency caveat worth testing before you trust this in production, and how to watch what it costs. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload a screenshot of the Unity Catalog metastore Details tab with the "External data access" toggle visible, or a diagram showing an external Spark client, the Azure Databricks serverless enforcement layer, and the underlying Delta/Iceberg table. Suggested alt text: "External Spark client querying a Unity Catalog managed table through the Azure Databricks serverless ABAC enforcement layer".] 1. Why ABAC used to stop at the Databricks engine door Every previous expansion of Unity Catalog's ABAC surface, metastore-wide policies, identity attributes from Microsoft Entra ID, context attributes that distinguish an agent from a person, explicit DENY, policies on views, shared one assumption: the engine reading the data is Azure Databricks itself. That assumption holds for most workspaces, but it breaks down the moment a genuinely multi-engine estate has an external Apache Spark job, a Flink pipeline, or a BI tool reading the same Unity Catalog managed table directly. Before cross-engine ABAC, the only way to protect that external read was to duplicate the policy logic inside the external engine itself, or to accept that the external path saw unfiltered data. Neither option scales once the policy changes on one side and someone forgets to mirror it on the other. 2. The mechanism: a serverless layer sits between the external engine and the table Cross-engine ABAC does not ask the external engine to evaluate the policy. Instead, when an external engine queries a table with an ABAC policy, a row filter, or a column mask attached, Azure Databricks routes the read through a specialized serverless compute layer. That layer evaluates the policy, filters and masks the data server-side, and only then returns the sanitized result to the external engine. The external engine never sees the unfiltered rows, and it does not need to know the policy exists. This is deliberately narrow in scope today: the only supported external engine is Apache Spark, reading either Delta tables through the Unity REST API or Iceberg tables through the Unity Catalog Iceberg REST endpoint. 3. Turning it on: metastore access, grants, and catalog commits Four things have to be true before an external Spark job can read a single filtered row. Enable external data access on the metastore This is disabled by default, on purpose, to prevent unauthorized external access. A metastore admin has to turn it on explicitly, and the option only appears for sufficiently privileged users: In a workspace attached to the metastore, open Catalog. Click the gear icon at the top of the Catalog pane and select Metastore. On the Details tab, enable External data access. Grant EXTERNAL USE SCHEMA explicitly This privilege is intentionally excluded from ALL PRIVILEGES, and a schema owner does not have it by default, both specifically to prevent accidental data exfiltration through an external path nobody was watching. Only the parent catalog owner can grant it, and it must always be granted explicitly: GRANT EXTERNAL USE SCHEMA ON SCHEMA catalog_name.schema_name TO `email address removed for privacy reasons` The querying principal also needs the ordinary SELECT on the table, USE CATALOG on its parent catalog, and USE SCHEMA on its parent schema, exactly as it would for an internal read. If the workload creates external tables rather than only reading managed ones, it additionally needs CREATE TABLE on the schema and both CREATE EXTERNAL TABLE and EXTERNAL USE LOCATION on the target external location, and EXTERNAL USE LOCATION carries the same explicit-grant-only rule as EXTERNAL USE SCHEMA: GRANT EXTERNAL USE LOCATION ON EXTERNAL LOCATION external_location_name TO `email address removed for privacy reasons` Enable catalog commits on the table Fine-grained enforcement for external reads only works against a managed table with catalog commits enabled, which requires Databricks Runtime 16.4 or above for a brand-new table, or Databricks Runtime 18.0 or above to turn it on for a table that already exists. Creating a new table with catalog commits enabled from the start: CREATE TABLE catalog_name.schema_name.table_name (id INT, name STRING) USING delta TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported'); Upgrading a table that already exists: ALTER TABLE catalog_name.schema_name.table_name SET TBLPROPERTIES ('delta.feature.catalogManaged' = 'supported'); Only after this step can you attach an ABAC policy, a row filter, or a column mask to the table and expect an external Spark read to respect it. 4. Connecting Apache Spark: Delta and Iceberg take different configuration The two paths share the same authorization model but need different Spark packages and session configuration, so keep them separate rather than guessing at a hybrid. Delta tables, via the Unity REST API This path needs Delta-Spark 4.1 or above and Unity Catalog Spark connector 0.4 or above. Using OAuth machine-to-machine authentication, which handles credential and token renewal automatically and is the right choice for a long-running job: "spark.sql.extensions": "io.delta.sql.DeltaSparkSessionExtension", "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.delta.catalog.DeltaCatalog", "spark.sql.catalog.uc_catalog": "io.unitycatalog.spark.UCSingleCatalog", "spark.sql.catalog.uc_catalog.uri": "https://adb-1234567890123456.12.azuredatabricks.net", "spark.sql.catalog.uc_catalog.auth.type": "oauth", "spark.sql.catalog.uc_catalog.auth.oauth.uri": "https://adb-1234567890123456.12.azuredatabricks.net/oidc/v1/token", "spark.sql.catalog.uc_catalog.auth.oauth.clientId": "<oauth-client-id>", "spark.sql.catalog.uc_catalog.auth.oauth.clientSecret": "<oauth-client-secret>", "spark.sql.catalog.uc_catalog.ServerSidePlanning.enabled": "true", "spark.sql.defaultCatalog": "uc_catalog", "spark.jars.packages": "io.delta:delta-spark_2.13:4.1.0,io.delta:delta-iceberg_2.13:4.1.0,io.unitycatalog:unitycatalog-spark_2.13:0.4.0,org.apache.hadoop:hadoop-azure:3.4.2" Setting ServerSidePlanning.enabled to true is not optional decoration, it is the specific switch that routes the read through the enforcement layer instead of letting Spark plan the scan on its own. Miss that one line and the external job will read the table, just without any of the row filters or column masks applied. Iceberg tables, via the Unity Catalog Iceberg REST endpoint This path needs Iceberg-Spark 1.11 or above and Apache Spark 4.0 or above: "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", "spark.sql.catalog.uc_catalog": "org.apache.iceberg.spark.SparkCatalog", "spark.sql.catalog.uc_catalog.type": "rest", "spark.sql.catalog.uc_catalog.uri": "https://adb-1234567890123456.12.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest/", "spark.sql.catalog.uc_catalog.credential": "<oauth-client-id>:<oauth-client-secret>", "spark.sql.catalog.uc_catalog.oauth2-server-uri": "https://adb-1234567890123456.12.azuredatabricks.net/oidc/v1/token", "spark.sql.catalog.uc_catalog.warehouse": "uc_catalog", "spark.sql.catalog.uc_catalog.cache-enabled": "false", "spark.sql.defaultCatalog": "uc_catalog" Once either configuration is in place, the query itself is ordinary Spark SQL, the enforcement is invisible from the client side: SELECT * FROM uc_catalog.schema_name.table_name; 5. What actually gets enforced, and where the hard edges are ABAC policies, standalone row filters, and column masks are all enforced on the external read, dynamically, the same way they would be for a query running inside Azure Databricks. That is the entire value proposition. But the current scope has real edges, not just minor rough patches: Only reads are supported from an external engine under fine-grained access control. To write, the writing principal must be exempted from the ABAC policy entirely, there is no write-path enforcement yet. Dynamic views are not supported. Projecting a VARIANT column is not supported. Filtering on a BINARY column is not supported. A column-masking function whose return type differs from the original column's type is not supported. Large aggregations can see real performance degradation, since every row still passes through the serverless enforcement layer before it reaches the external engine. And one warning from the documentation is worth repeating verbatim rather than paraphrasing away its force: concurrent writes during query planning can cause the same table to be read from different table snapshots within a single self-join or multi-scan query, which can produce incorrect results. If your external workload runs a self-join or anything that scans the same table more than once inside one query, and the table sees concurrent writes, test that specific pattern under write load before trusting its output in production. This is not a theoretical edge case, it is a named, documented failure mode. 6. Watching what it costs The enforcement layer is serverless compute, and Azure Databricks charges for it under its own billing line, separate from ordinary warehouse or job compute. Anyone with access to the billing system table can see exactly how much a given user's external reads are costing, broken down by SKU and by the identity that ran the query: SELECT usage_date, sku_name, identity_metadata.run_as, SUM(usage_quantity) AS dbus_consumed_by_cross_engine_abac FROM system.billing.usage WHERE usage_date BETWEEN '2026-06-01' AND '2026-07-01' AND billing_origin_product = 'EXTERNAL_COMPATIBILITY' GROUP BY 1, 2, 3 ORDER BY 1; If an external Spark job reading a heavily policy-protected table runs more often than its consumers actually need fresh data, that EXTERNAL_COMPATIBILITY line is where the cost of that habit shows up first, not in the external engine's own compute bill. 7. My perspective on this Every ABAC expansion I have covered before this one assumed the reader was inside the platform. This is the first one that assumes the opposite, and the caution in the scope reflects that: read-only, one external engine, a documented snapshot-consistency warning stated plainly rather than buried. That restraint is the right call for a first general-availability release of a capability whose entire failure mode is "the external engine saw data it should not have seen." The detail I would not skip past is ServerSidePlanning.enabled. It is one line in a long configuration block, easy to leave out by copying an older Delta-Spark config that predates this feature, and leaving it out does not produce an error, it produces an unfiltered read that looks identical to a correctly filtered one until someone notices the wrong person saw the wrong row. Treat that flag as a required field, not an optional tuning parameter, and verify it is present in code review for any external Spark job that touches a governed table. I would also budget real test time for the concurrency warning before this goes anywhere near a production pipeline that both writes to and externally reads from the same table. A silent correctness bug that only appears under concurrent writes during self-joins is exactly the kind of failure that passes every test run in a quiet environment and then shows up the first time two jobs actually overlap in production. Conclusion Cross-engine ABAC does not ask external Spark to respect your governance policy, it makes respecting that policy the external engine's only option, by interposing a serverless enforcement layer it cannot see around. The setup cost is real: a metastore-level toggle only an admin can flip, an explicit grant that intentionally does not come bundled with broader privileges, and a table property that most existing managed tables do not have yet. For a genuinely multi-engine estate, that cost is still far lower than the alternative, which was duplicating policy logic in every external engine and hoping nobody let the two copies drift apart. Official references Microsoft Learn, "Cross-engine attribute-based access controls (ABAC) - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/external-access/cross-engine-abac) Microsoft Learn, "Access Databricks tables from Delta clients - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/external-access/unity-rest) Microsoft Learn, "Enable external data access to Unity Catalog - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/external-access/admin) Microsoft Learn, "Catalog commits - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/tables/features/catalog-commits) Microsoft Learn, "Row filters and column masks - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/filters-and-masks/) Microsoft Learn, "Authorize service principal access to Azure Databricks with OAuth" (learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-m2m)Azure Databricks Cost Governance: Budgets, Tags, and Alerts Before the Invoice Arrives
Cost visibility on Azure Databricks does not have to wait for the monthly invoice, and for Azure customers specifically, the attribution trail can run straight into Azure Cost Management, not just Databricks system tables. On October 7, 2026, Databricks published a practical guide on setting budgets and alerts for cloud data warehouse costs. The original post is written in general, cross-cloud terms. This article walks through the same problem from the Azure Databricks side specifically, using the account console, the REST API, and the system tables as they actually behave on Azure today, including two capabilities that go further than what the original post describes: per-user Unity Gateway budgets with usage blocking, and serverless usage policies whose tags propagate directly into Azure's own cost analysis tools. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload a screenshot of the Azure Databricks account console Budgets tab (Usage > Budgets), or if the editor accepts an external URL, use a relevant Microsoft Learn budgets screenshot. Suggested alt text: "Azure Databricks account console showing the Budgets tab with monthly threshold and burn-rate chart".] 1. Why the bill always arrives as a surprise The pattern repeats across data teams on any cloud: usage is not attributed to a team or a query, no team owns a budget number, the overspend only surfaces a billing cycle later, and manual spreadsheet review does not scale past a handful of warehouses. None of this is a tooling gap specific to Databricks, it is what happens when cost tracking is bolted on after the fact instead of built into how compute gets created in the first place. Azure Databricks addresses this with four layers that build on each other: tag-based attribution, the system.billing.usage table as the source of truth, budgets scoped to workspaces or tags in the account console, and alerts that run on a schedule instead of waiting for a human to check a dashboard. 2. Attribution: tags, and the Azure-native alternative to tagging by hand The baseline mechanism is the same one most clouds use: key-value tags on a SQL warehouse, set at creation or edit time, that later show up in the custom_tags column of the billing system table. Tagging a warehouse through the REST API curl -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/warehouses" \ -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "name": "finance-analytics", "cluster_size": "Small", "warehouse_type": "PRO", "tags": { "custom_tags": [ { "key": "team", "value": "finance" }, { "key": "cost-center", "value": "1234" } ] } }' Two details matter when you actually run this. The warehouse tagging limit is under 45 tags per warehouse, and if a custom tag key collides with one of Databricks' own default tags (Vendor, ClusterId, SqlEndpointId, Creator), the custom one gets an x_ prefix during propagation rather than overwriting the default. Plan tag key names with that collision rule in mind. Query tags: tagging at the session or statement level (Public Preview) Warehouse tags attribute an entire warehouse to a team, but a shared warehouse serving multiple teams needs finer granularity than that. Query tags (Public Preview) solve this by letting you tag individual sessions or individual statements, independent of which warehouse they run on. From the SQL editor, a notebook, or a dashboard, set tags for the rest of the session with the SET QUERY_TAGS statement: -- Tag every subsequent query in this session SET QUERY_TAGS['team'] = 'marketing', QUERY_TAGS['cost_center'] = '701'; -- Inspect what's currently set SET QUERY_TAGS; -- Clear a tag when the session changes purpose SET QUERY_TAGS['cost_center'] = UNSET; Session-level tags apply to every statement that follows until changed. Statement-level tags are a separate, narrower mechanism: pass a query_tags array in the request body of the Statement Execution API, and only that one statement carries the tag, with the session reverting to its own tags afterward: curl -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/statements" \ -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "warehouse_id": "abc123", "statement": "SELECT * FROM samples.nyctaxi.trips LIMIT 10", "query_tags": [ { "key": "team", "value": "engineering" }, { "key": "env", "value": "prod" } ] }' Tags land in the query_tags column of system.query.history, which needs its own access grant from an account admin if you do not already have it. Query it directly to see spend by query rather than by whole warehouse: SELECT statement_id, query_tags, executed_by, start_time FROM system.query.history WHERE MAP_CONTAINS_KEY(query_tags, 'team') AND query_tags['team'] = 'marketing' ORDER BY start_time DESC LIMIT 100 The limits are tight enough to plan around: 20 user-specified tags per query, 10 KB of tags per session, 128 characters per key or value, and tag keys cannot contain commas, colons, hyphens, slashes, equals signs, or periods. A key starting with @@ is reserved, that is the prefix dbt and Power BI use for their own automatically applied tags (@@dbt_model_name, @@powerbi_report_id, and similar). Set a tag using a session configuration parameter instead of SQL and an invalid key gets silently dropped with a tag_invalid: true sentinel; set it through SET QUERY_TAGS and an invalid key fails the statement outright at execution time, which is the behavior worth relying on if you want bad tags to be loud rather than quiet. The Azure-specific upgrade: serverless usage policies Tagging every warehouse by hand does not scale, and nothing stops a team from forgetting it on the one warehouse that ends up costing the most. For serverless compute specifically, Azure Databricks has a more structural answer: serverless usage policies (Public Preview). A policy bundles a set of tags and gets assigned to users; any serverless notebook, job, pipeline, serving endpoint, or app that user creates inherits the policy's tags automatically, with no manual tagging step per resource. The detail that matters most for an Azure-focused reader: serverless usage policy tags propagate directly into the billing records emitted to Azure Cost Analysis, not only into Databricks' own system.billing.usage table. That means the same tag you use to scope a Databricks budget is also what your FinOps team sees inside the Azure portal's own cost views, without a separate mapping step. Known limitations worth planning around: existing notebooks, jobs, and pipelines do not retroactively pick up a policy, you have to assign it manually to each existing asset; classic (non-serverless) compute is not covered at all; and a pipeline triggered by a job does not inherit the job's policy, the pipeline needs its own. 3. The source of truth: system.billing.usage Every budget, every alert, and every dashboard ultimately reads from one Unity Catalog system table: system.billing.usage. Records typically land within 12 hours. The schema includes, among other columns, custom_tags (a map, populated by both manual warehouse tags and serverless usage policy tags), usage_metadata.warehouse_id, usage_quantity (in the unit given by usage_unit, typically DBU), and record_type, which can be ORIGINAL, RETRACTION, or RESTATEMENT when Databricks issues a billing correction. Note that query tags from the previous section live in system.query.history, not here, custom_tags on this table only reflects warehouse-level and policy-level tags. The two tables are separate for this reason, use system.query.history for query-level cost investigation and this table for warehouse and account-level aggregation. That last detail is not cosmetic. A naive SUM(usage_quantity) without accounting for retractions can overcount. The pattern that handles it correctly: SELECT usage_metadata.job_id, usage_start_time, usage_end_time, SUM(usage_quantity) AS usage_quantity FROM system.billing.usage GROUP BY ALL HAVING usage_quantity != 0 Aggregating before filtering lets the retraction record cancel out the original it corrects, so a corrected job's usage does not get double-counted. Applied to the attribution problem directly, two queries worth keeping in a cost dashboard: -- SQL warehouse spend by team, last 30 days SELECT custom_tags['team'] AS team, usage_metadata.warehouse_id, SUM(usage_quantity) AS dbus_consumed FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND usage_date >= current_date() - INTERVAL 30 DAYS GROUP BY ALL ORDER BY dbus_consumed DESC -- Unattributed SQL usage: the attribution gap, should trend to zero SELECT usage_date, SUM(usage_quantity) AS unattributed_dbus FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND (custom_tags['team'] IS NULL OR custom_tags['team'] = '') GROUP BY usage_date ORDER BY usage_date DESC For a dollar figure rather than a DBU count, join against the system.billing.list_prices system table on sku_name and the price's effective date range. I am not reproducing that join here because I have not verified its exact column names against the current schema, and getting a pricing join subtly wrong is worse than not showing it. Check the list prices system table reference on Microsoft Learn (learn.microsoft.com/en-us/azure/databricks/admin/system-tables/pricing) before writing that query yourself. 4. Budgets in the account console: more than a threshold and an email Budgets live in the Azure Databricks account console (accounts.azuredatabricks.net), under Usage > Budgets. A budget can scope to specific workspaces, to custom tags, or to both, and if you leave both empty it tracks your entire account. What the Azure documentation describes goes further than a single alert value: Up to four monthly thresholds per budget, each with its own email recipient list. Up to 1,000 budgets per account. Budgets scoped to Unity Gateway (the routing layer in front of Pay-Per-Token foundation model inference, batch ai_query, and external model provider usage) support a per-user spend threshold, on top of the shared thresholds. Cross a per-user threshold and that specific user's requests can optionally be blocked, with the option to grant named users or groups a higher override limit. Budget visibility and editing is governed by a dedicated permission role, not just account admin status, so a team lead can see their own budget without account-wide access. The important caveat, stated plainly in the documentation: budgets are a monitoring mechanism, not a hard spending cap, except specifically for Unity Gateway and Genie budgets, where the optional "Block usage" setting does enforce a stop. Even there, enforcement is approximate, based on a near-real-time cost estimate, so a user can be blocked slightly before or after the literal threshold. For standard SQL and compute budgets, nothing stops the spend, the budget's job is to get a human's attention fast. A layered pattern, adapted for Azure, that tends to hold up in practice: a BI Serving + Production budget at $8,000/month scoped to UseCase:BIServing and Env:Prod, alerting the platform team; an Analytics Ad-Hoc budget at $3,000/month scoped to UseCase:AdHoc, alerting the analytics manager; a Marketing Analytics budget at $2,500/month scoped to Team:Marketing, alerting the marketing data lead; and an account-wide safety net at $25,000/month with no tag scope, alerting the CTO and FinOps. Team-level budgets for accountability, plus one account-wide budget as the net that catches whatever slips past every tag filter. The values above are illustrative, size them against your own baseline. 5. Alerts that run before the threshold, not after Databricks SQL alerts run a query on a schedule and evaluate a condition against the result, independent of whether anyone opens a dashboard. On Azure Databricks, the current alert engine resolves to one of three states: OK, TRIGGERED, or ERROR, a simplification from the legacy alert system's extra UNKNOWN state. A daily threshold alert, scheduled once a day, built on the team-attribution query above: SELECT custom_tags['team'] AS team, SUM(usage_quantity) AS dbus_today FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND usage_date = current_date() GROUP BY ALL HAVING dbus_today > 500 -- adjust to your own per-team daily baseline A threshold alert only catches a problem on the day it happens. A pacing alert catches it earlier, by projecting month-end spend from the days elapsed so far: WITH month_to_date AS ( SELECT SUM(usage_quantity) AS mtd_dbus, DAY(current_date()) AS days_elapsed, DAY(LAST_DAY(current_date())) AS days_in_month FROM system.billing.usage WHERE billing_origin_product = 'SQL' AND usage_date >= DATE_TRUNC('MONTH', current_date()) ) SELECT mtd_dbus, mtd_dbus / days_elapsed * days_in_month AS projected_month_end_dbus FROM month_to_date HAVING projected_month_end_dbus > 15000 -- adjust to your monthly DBU budget Schedule the pacing alert every four hours and route it to Slack or email through the alert's notification settings, the same pattern the original Databricks post recommends. Running it more often than that mostly adds noise, since system.billing.usage itself only refreshes every few hours. One cost that neither alert catches on its own: materialized view refreshes bill as serverless pipeline usage, under billing_origin_product = 'DLT', not as warehouse usage. If your team only watches SQL spend, a materialized view refreshed far more often than its source data actually changes can run invisibly. Align refresh frequency to real data freshness, and include DLT in your cost queries, not only SQL. 6. Where this still needs a human None of the above replaces judgment on the two levers that move cost the most: warehouse type and warehouse size. Azure Databricks documentation still points to SQL Serverless as the default starting point for most workloads, since it starts in seconds, scales to zero between queries, and bills only for execution time, and to predictive optimization (learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-optimization), enabled by default on accounts created after November 11, 2024, to keep OPTIMIZE, VACUUM, and clustering key maintenance running automatically on managed Unity Catalog tables, instead of a scheduled job someone has to remember to run. Auto-stop is worth checking explicitly rather than trusting the default: Pro and Classic warehouses default to 45 minutes of idle time before stopping, serverless defaults to 10 minutes, and serverless can be brought down to as low as 1 minute through the API, below what the UI exposes. For a warehouse that serves a handful of scheduled queries a day, 45 idle minutes can be the single biggest line item. 7. My perspective on this The feature set here is more capable on Azure specifically than the general cross-cloud framing suggests, and the detail that stands out to me is the Azure Cost Analysis propagation for serverless usage policy tags. That is not a Databricks convenience feature, it closes the exact gap that usually breaks cost governance in a real organization: the Databricks admin owns one view of spend, the cloud FinOps team owns another, and reconciling the two by hand is where attribution efforts actually die. If the same tag shows up in both without a manual export step, that reconciliation work goes away. I would start with serverless usage policies before hand-tagging every warehouse, specifically because the policy approach survives a new team member forgetting to tag something, while manual tagging does not. The known limitation that existing assets do not retroactively inherit a new policy is real friction, budget time for a one-time pass over your current notebooks, jobs, and pipelines when you adopt this, it will not happen automatically. And treat every budget as a monitoring signal, not a spending cap, with the explicit exception of Unity Gateway and Genie budgets where blocking is actually available. Assuming the wrong one of those two behaviors for a given budget is how a team ends up either silently overspending or unexpectedly blocked mid-production. Conclusion Cost governance on Azure Databricks is not a single feature, it is tags feeding a system table, a system table feeding budgets and alerts, and budgets feeding back into how the next warehouse gets created. The Azure-specific pieces, Unity Gateway per-user budgets with blocking and serverless usage policy tags flowing into Azure Cost Analysis, are worth adopting even if you came from the general cross-cloud version of this guide expecting less. Official references Databricks Blog, "Set Budgets and Alerts for Cloud Data Warehouse Costs" (www.databricks.com/blog/set-budgets-and-alerts-cloud-data-warehouse-costs) Microsoft Learn, "Create and monitor budgets - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/account-settings/budgets) Microsoft Learn, "Attribute usage with serverless usage policies - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/usage/budget-policies) Microsoft Learn, "Billable usage system table reference - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/system-tables/billing) Microsoft Learn, "Databricks SQL alerts - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/sql/user/alerts/) Microsoft Learn, "Use tags to attribute and track usage - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags) Microsoft Learn, "Query tags - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-tags) Microsoft Learn, "Governance Hub - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/governance-hub/)LTAP Direct Writes: Loading Terabytes into Lakebase Postgres Without Touching Your Live Application
A new beta capability that removes the classic bottleneck of bulk-loading data into a live Postgres database on Azure Databricks. On October 8, 2026, Databricks published details on LTAP Direct Writes, a Beta capability of the LTAP (Lake Transactional/Analytical Processing) architecture that accelerates large data loads into Lakebase Postgres. Until now, bulk-loading billions of rows into a live transactional Postgres database meant competing directly with the application for the same CPU and memory. Every write, regardless of its origin, has to pass through a single primary instance. One Databricks customer loading close to 1 billion rows per day through Synced Tables needed more than 8 hours per load, and had to overprovision OLTP compute just to absorb that peak without degrading the application. LTAP Direct Writes attacks this problem by moving the heavy construction work out of the primary instance entirely. Databricks reports loading 1 TB of data in under 5 minutes with this approach, up to 147 times faster, without consuming resources from the live application. [INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload the file, or if the editor accepts an external URL, use: https://www.databricks.com/sites/default/files/blog_images/load-tbs-in-minutes-into-postgres-blog-img-6.png . Suggested alt text: "LTAP architecture diagram showing Lakebase compute streaming the WAL to a storage layer that transcodes data into columnar Parquet, readable through Delta and Iceberg".] 1. Why a single Postgres primary cannot scale bulk loads linearly In any standard Postgres deployment, every write, no matter where it originates, passes through the primary instance. This is not a configuration limitation, it is the engine's transactional guarantee: the primary has to coordinate commit order, maintain consistency, and manage the write-ahead log as a single sequence. Throwing more parallelism at the client side does not help much when the bottleneck sits in the one process that can actually write. This is why bulk loads of billions of rows have historically competed directly with the production application for the same CPU and memory on the primary, forcing a choice between load throughput and application latency. 2. What LTAP Direct Writes changes LTAP separates compute from storage in Lakebase. As data is materialized to object storage, it is transcoded from Postgres's row-oriented representation into columnar Parquet, readable through Delta or Iceberg. This creates a single logical copy of the data, served by two specialized engines: Postgres for transactional workloads, an analytical engine for OLAP, with no external CDC pipeline to keep in sync. Synced Tables, the capability that serves lakehouse data back into Lakebase, is one concrete implementation of this architecture, and it is exactly where LTAP Direct Writes comes in to accelerate the initial load. The mechanism moves data construction out of the primary and runs it in parallel, outside of it: Parallel page building: Each Spark executor runs an isolated, sandboxed Postgres instance in binary-upgrade mode, the same mechanism `pg_upgrade` uses internally. Each sandbox reuses the destination's catalog OIDs, and the driver assigns non-overlapping OID ranges so concurrent objects cannot collide. Frozen binary COPY: Inside each sandbox, a binary `COPY` with `FREEZE` builds that executor's slice of the heap. Frozen tuples are treated as already committed, so the sandbox does not need to maintain transaction history. Per-page checksums: Each worker computes a checksum for its own constructed page. Lakebase's pageserver validates this on ingest, so a corrupted page never becomes authoritative. Index building without downloading the full heap: Each worker exports key columns and tuple IDs to object storage. A custom table access method feeds these into Postgres's standard B-tree builder, shifting each tuple pointer by the cumulative size of earlier slices. The full heap never has to be downloaded back to build the index. Atomic handoff: The Spark driver writes a manifest and calls a single SQL function. The primary records the import as a compact WAL entry, carrying only the import's description rather than the full data volume. The pageserver claims the files, warms them on local SSD, and the primary atomically swaps the staged data into the user-visible table. Until that commit, the imported table stays isolated. The team behind this is explicit that the technique uses a Postgres extension and a custom table access method, without modifying Postgres core. 3. What this unlocks for Azure Databricks users specifically For teams already running Lakebase on Azure Databricks, this closes a gap that used to force an uncomfortable choice: either accept a multi-hour load window with degraded application performance, or keep bulk analytical datasets out of Lakebase entirely and pay the cost of a separate reverse-ETL pipeline just to avoid that degradation. With Direct Writes, feeding a Lakebase-backed application with a large batch of enriched lakehouse data (a nightly scoring table, a freshly computed feature set, a reference dataset rebuilt from scratch) no longer means scheduling that load for a maintenance window. It becomes a background operation that the live application does not feel. 4. Hands-on: creating a Synced Table with Direct Writes The capability that benefits from Direct Writes today is the creation of a Synced Table. Direct Writes itself is currently a checkbox in the Create synced table dialog in the UI, not yet an explicit field documented in the CLI or SDK JSON spec, so the examples below show the general synced table creation flow that Direct Writes accelerates. Creating the synced table with the Databricks CLI databricks postgres create-synced-table main.sales.orders_sync \ --json '{ "spec": { "source_table_full_name": "main.sales.orders", "branch": "projects/my-project/branches/production", "primary_key_columns": ["order_id"], "scheduling_policy": "SNAPSHOT", "postgres_database": "mydb", "create_database_objects_if_missing": true } }' Creating the synced table with the Python SDK from databricks.sdk import WorkspaceClient from databricks.sdk.service.postgres import ( SyncedTable, SyncedTableSyncedTableSpec, SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy, ) w = WorkspaceClient() synced_table = w.postgres.create_synced_table( synced_table=SyncedTable(spec=SyncedTableSyncedTableSpec( source_table_full_name="main.sales.orders", branch="projects/my-project/branches/production", primary_key_columns=["order_id"], scheduling_policy=SyncedTableSyncedTableSpecSyncedTableSchedulingPolicy.SNAPSHOT, postgres_database="mydb", create_database_objects_if_missing=True, )), synced_table_id="main.sales.orders_sync", ).wait() print(f"Synced table created: {synced_table.name}") Querying the loaded data directly from Postgres Once the load completes, applications query the result with a standard Postgres driver, no different from any other table: SELECT * FROM sales.orders_sync WHERE order_id = 48213; 5. Checking load status programmatically For a load that might take anywhere from seconds to minutes depending on volume, polling status through the SDK is more practical than watching the UI: from databricks.sdk import WorkspaceClient w = WorkspaceClient() table = w.postgres.get_synced_table("synced_tables/main.sales.orders_sync") print(f"State: {table.status.detailed_state}") print(f"Last sync: {table.status.last_sync_time}") print(f"Message: {table.status.message}") This is useful to wire into a Lakeflow Job as a dependency gate: downstream tasks can wait for `detailed_state` to report a completed sync before reading from the table, instead of assuming a fixed load duration. 6. What Direct Writes accelerates, and what it does not yet LTAP Direct Writes accelerates the initial load in every sync mode. For subsequent syncs, the acceleration applies specifically to Snapshot mode, because that mode performs a full refresh on every cycle. In Triggered and Continuous modes, later updates are already incremental through Change Data Feed, not bulk loads, so Direct Writes only matters for their initial load. The team behind the feature is upfront about its current scope: The 147x benchmark measures heap-building time only. Index building is not parallelized the same way yet, and the team describes this as active, ongoing work. LTAP Direct Writes is Beta, and requires a Lakebase project running Postgres 16, 17, or 18. It can only be selected when a synced table is created, not added to an existing one. To use it with an existing synced table, you delete it and create a new one. A table with many secondary indexes will likely not see the same proportional gain as the simple table used in the published benchmark, since index construction is the part still being parallelized. 7. My perspective on this change The performance number is the headline, but the architectural detail that matters more is where the heavy work moved to. The load no longer competes with your live application inside the primary instance, it becomes isolated Spark work that happens alongside it. That is the problem the cited customer actually had: not "slow load" in isolation, but "slow load competing for resources with production." I would test this first on a table without complex secondary indexes before assuming the 147x gain repeats identically on any schema, given the team's own disclosure that index building is still catching up to the heap-building speedup. For Azure Databricks teams already committed to Lakebase as their operational Postgres, this is one more argument for keeping large, bulk-refreshed datasets inside Lakebase instead of routing them through a separate reverse-ETL tool just to avoid the old load-time penalty. Conclusion LTAP Direct Writes is not an isolated feature, it is separating compute from storage applied all the way down into the bulk-load process itself. For teams already using Synced Tables and feeling the weight of large loads on the primary, it is worth testing the Beta before assuming overprovisioning the OLTP side is the only way out. Official references Databricks Blog, "Load terabytes of data in minutes into Lakebase Postgres" Microsoft Learn, "LTAP architecture - Azure Databricks" Microsoft Learn, "Serve lakehouse data with synced tables" Microsoft Learn, "Lakebase architecture - Azure Databricks" Databricks Blog, "Lakebase LTAP: rethinking database storage"Multiple Triggers in Azure Databricks Lakeflow Jobs: Simplifying Data Orchestration
A new feature that simplifies event-driven pipeline execution in Azure Databricks. On October 5, 2026, Databricks announced support for Multiple Triggers in Lakeflow Jobs, currently available in Beta. Until now, a Job could have only one configured trigger. This created an interesting limitation in data engineering projects: what happens when the same pipeline needs to run under different conditions? Imagine a data ingestion pipeline that needs to run when a new file arrives in storage, when a Delta table is updated, or every day at a specific time, regardless of those events. Previously, a common solution was to create separate Jobs, each with its own trigger, while reusing or duplicating the processing logic. Another option was to create intermediate Jobs responsible for calling the main pipeline. Both approaches work, but they introduce additional components and increase maintenance effort. With Multiple Triggers, we can now configure different execution conditions directly within the same Job. 1. How did it work before? Let's consider a pipeline responsible for processing sales data. This pipeline has three requirements: - Process new files arriving in a Unity Catalog Volume. - Reprocess data whenever a source Delta table is updated. - Run every day at 8 AM to ensure scheduled processing. Before Multiple Triggers, each Job supported only one trigger configuration. One possible approach was to create three separate Jobs, each responsible for a specific trigger type, pointing to the same processing logic. Although notebooks and tasks could be shared across these Jobs, we still needed to manage separate Job definitions, permissions, parameters, and execution settings. In enterprise environments, this complexity can grow as the number of pipelines and dependencies increases. 2. What changed with Multiple Triggers? Azure Databricks now allows us to configure up to five triggers within a single Lakeflow Job, including combinations of different trigger types. For example: Scheduled Trigger: Runs the Job at a predefined time. File Arrival Trigger: Runs when new files are detected. Table Update Trigger: Runs when specific tables are updated. Model Update Trigger: Runs based on updates to models registered in Unity Catalog. The execution logic is straightforward: when any active trigger meets its condition, it can initiate a Job run. This means we no longer need to maintain separate Jobs just to support multiple execution conditions. Another useful feature is the ability to pause or resume each trigger independently. For example, during maintenance, we can temporarily pause the Table Update Trigger while keeping the daily schedule active. 3. Practical example: Configuring Multiple Triggers Let's create a Job responsible for processing sales data. In this example, we will configure three execution conditions: Trigger 1: New files arriving in a Unity Catalog Volume. Trigger 2: Updates to the bronze.sales.orders table. Trigger 3: Daily execution at 8 AM, using the São Paulo time zone. We can configure these triggers through the Azure Databricks interface by opening the Job and adding triggers under Schedules & Triggers. We can also use the Jobs API to automate this configuration. Example using the Jobs API The following JSON illustrates a Job configuration using the triggers array. { "name": "sales_processing_job", "max_concurrent_runs": 1, "tasks": [ { "task_key": "process_sales", "notebook_task": { "notebook_path": "/Shared/pipelines/process_sales" }, "existing_cluster_id": "<CLUSTER_ID>" } ], "triggers": [ { "file_arrival": { "url": "/Volumes/bronze/sales/landing" }, "pause_status": "UNPAUSED" }, { "table_update": { "table_names": [ "bronze.sales.orders" ], "condition": "ANY_UPDATED" }, "pause_status": "UNPAUSED" }, { "schedule": { "quartz_cron_expression": "0 0 8 * * ?", "timezone_id": "America/Sao_Paulo" }, "pause_status": "UNPAUSED" } ] } In this example, we have a single Job with three independent execution conditions. The triggers field contains an array of trigger configurations, allowing multiple triggers to be defined without relying on the legacy top-level schedule, trigger, or continuous fields. The example uses an existing cluster for simplicity. In production environments, the compute configuration should be selected according to workload requirements and organizational policies. Creating the Job with Python We can submit the configuration using Python and the requests library. import os import json import requests workspace_url = os.environ["DATABRICKS_HOST"].rstrip("/") token = os.environ["DATABRICKS_TOKEN"] with open("sales_job.json", encoding="utf-8") as file: job_settings = json.load(file) response = requests.post( f"{workspace_url}/api/2.1/jobs/create", headers={ "Authorization": f"Bearer {token}", "Content-Type": "application/json" }, json=job_settings, timeout=30 ) response.raise_for_status() print("Job created:", response.json()["job_id"]) The sales_job.json file contains the JSON configuration shown earlier. For enterprise implementations, I recommend using service principals and secure authentication mechanisms instead of embedding credentials directly in the code. 4. What happens when two triggers fire at the same time? This is an important architectural consideration. Imagine a new file arriving at exactly 8 AM, when the scheduled execution is also expected to run. Both triggers may request an execution of the same Job. However, this does not necessarily mean that two runs will execute simultaneously. The behavior depends on the Job's concurrency settings. By default, a Job allows only one active run at a time. Therefore, we need to consider concurrency limits and how additional runs are handled. Another important consideration is idempotency. In simple terms, running the same pipeline multiple times against the same data should not produce duplicate records or inconsistent results. In Azure Databricks, we can use Delta Lake features and MERGE operations to implement idempotent processing strategies. Multiple Triggers improves execution flexibility, but it does not replace proper pipeline design. 5. Monitoring Jobs using System Tables One capability I find particularly useful is the possibility of monitoring trigger configurations through Databricks System Tables. The system.lakeflow.jobs table provides information about Jobs, including the trigger_type and triggers fields. When a Job has multiple triggers, the trigger_type field can contain the value MULTIPLE. We can identify these Jobs using SQL: WITH current_jobs AS ( SELECT workspace_id, job_id, name, trigger_type, triggers, ROW_NUMBER() OVER ( PARTITION BY workspace_id, job_id ORDER BY change_time DESC ) AS rn FROM system.lakeflow.jobs ) We can also expand the triggers array to inspect the individual configurations associated with each Job. WITH current_jobs AS ( SELECT workspace_id, job_id, name, triggers, ROW_NUMBER() OVER ( PARTITION BY workspace_id, job_id ORDER BY change_time DESC ) AS rn FROM system.lakeflow.jobs ) SELECT workspace_id, job_id, name, trigger FROM current_jobs LATERAL VIEW EXPLODE(triggers) AS trigger WHERE rn = 1; This approach can support governance and observability dashboards, especially in environments with hundreds or thousands of Jobs. 6. Limitations and considerations Since Multiple Triggers is currently in Beta, there are some important limitations and requirements: Each Job can have up to five triggers. The Multiple Triggers feature must be enabled in the workspace Preview settings. Each trigger can be paused or resumed independently. Continuous triggers cannot be combined with other trigger types. When updating the triggers array through the Jobs API, the entire existing trigger list is replaced. The last point is particularly important when working with infrastructure automation or CI/CD pipelines. When updating trigger configurations through the API, we need to make sure that all triggers we want to preserve are included in the new definition. 7. My perspective on this change Based on my experience working with data architecture and data engineering projects, one recurring challenge is avoiding unnecessary complexity in orchestration. The problem is not always related to data processing itself. Sometimes, it comes from the number of components we need to maintain just to execute relatively simple logic. Multiple Triggers does not change how we build tasks within a Job, but it improves how we define when those tasks should run. I see significant value in this feature for incremental ingestion, event-driven processing, and pipelines that need to combine data updates with scheduled execution. I also appreciate the ability to manage triggers independently, which can make maintenance and operational control easier. It may seem like a relatively small configuration change, but it can lead to meaningful architectural simplification in certain scenarios. Conclusion With Multiple Triggers support, Azure Databricks Lakeflow Jobs provides greater flexibility for configuring different execution conditions within a single Job. This feature reduces the need for auxiliary Jobs or duplicated Job definitions, making orchestration architectures simpler and easier to maintain. We still need to consider concurrency, idempotency, governance, and observability, but we now have another option for building event-driven pipelines without introducing unnecessary components. For anyone working with Azure Databricks, this feature is worth exploring to identify opportunities to simplify existing orchestration workflows. Official references Databricks: Automate jobs with schedules and triggers Databricks Release Notes, October 2026 Databricks Jobs API Databricks System Tables: Jobs
Engineering an Azure Data Layer for Odoo Analytics Without Changing the ERP
One thing I have learned from working on ERP-driven data projects is that the ERP is rarely the place where you want to solve every analytics problem. We saw this firsthand in a manufacturing engagement where Odoo was being used across finance, inventory, procurement, and manufacturing, but reporting still involved substantial manual extraction and Excel-based consolidation. The interesting engineering question wasn't “How do we replace the ERP?” It was: “How do we build a reliable analytical layer around it without changing the transactional system?” That shaped the architecture. We used Odoo APIs and Azure Data Factory for ingestion, landing the data in Azure Data Lake Storage Gen2. From there, the data was organized into Bronze, Silver, and Gold layers: Bronze: Source-aligned Odoo data was retained with minimal transformation. Silver: Data was cleansed, standardized, and prepared across finance, inventory, procurement, and manufacturing domains. Gold: Analytics-ready data was prepared for downstream reporting and consumption. Azure Databricks with Spark handled transformation and KPI standardization, while Synapse/SQL layers provided structured analytical models for reporting. Power BI datasets consumed the prepared data, with Row-Level Security applied for role-based access. Why did this separation matter for the project? The important architectural boundary was: Odoo → transactional operations Azure data platform → ingestion, storage, transformation, standardization Power BI → analytical consumption That separation meant the reporting layer did not have to repeatedly extract and reshape data directly from the ERP. It also gave us a dedicated place to handle historical data and standardize information across business domains before it reached the reporting layer. What changed? The implementation automated 15+ Odoo-based reports and provided governed Power BI dashboards to 60+ business users. More importantly, teams moved away from repeated ERP extraction and spreadsheet consolidation toward a reusable analytical data layer. For our team, the bigger takeaway was that ERP modernization does not necessarily mean modifying the ERP. For teams building Azure data platforms around ERP systems, how are you drawing the boundary between API ingestion, historical storage, transformation, and analytical modeling?Lakeflow in Azure Databricks
Anyone who has ever built a data engineering stack from scratch knows the pattern: one ingestion tool to pull data from CRM and ERP, a separate orchestrator to stitch together the dependencies between jobs, notebooks or scripts doing the actual transformation, and a fourth system just to check whether all of that ran correctly last night. Each piece comes from a different vendor, with its own authentication, its own logs, and its own SLA. When something breaks at three in the morning, the first job isn't fixing the problem, it's figuring out which of the four tools it lives in. Lakeflow attacks that fragmentation head-on: it brings ingestion, transformation, and orchestration together into a single surface inside Azure Databricks, with Unity Catalog providing end-to-end governance. This isn't marketing repackaging of three products that already existed separately, it's the promise that the dependency between ingestion and transformation, column-level lineage, and access control stop being reconciled by hand across systems and start existing natively in the same graph. The three pieces that make up Lakeflow Lakeflow Connect covers ingestion, with point-and-click connectors for SaaS applications (Salesforce, Workday, ServiceNow), databases (SQL Server), and messaging, plus Zerobus Ingest, a serverless direct-write API for anyone with events coming from their own applications who doesn't want to stand up a message bus in the middle. Spark Declarative Pipelines is the transformation layer: instead of hand-writing the orchestration of streaming tables and materialized views, you declare the target table and the business logic, and the engine infers the dependency graph on its own, builds the DAG, and takes care of operational details like backfill and versioning. The name has changed over time (anyone who has followed the platform for a while will recognize it as the direct evolution of what used to be Delta Live Tables), but the underlying mechanism, declaring the desired result instead of the procedural step-by-step, remains the central idea. Lakeflow Jobs orchestrates all of it as a unified DAG: SQL workloads, Python code, declarative pipelines, dashboards, and external systems live in the same job definition, with data-driven triggers (a table was updated, a file landed in a volume), control-flow tasks, and no-code backfill runs. Hands-on: a simple end-to-end pipeline To make it concrete, here's how the three pieces fit together in a basic ingestion and declarative transformation pipeline: import dlt from pyspark.sql.functions import col # Streaming table: incremental ingestion via Auto Loader @dlt.table( comment="Raw order ingestion via Auto Loader" ) def orders_bronze(): return ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .load("/Volumes/sales/bronze/orders_raw") ) # Materialized view: validation and enrichment @dlt.table( comment="Validated orders, with a quality filter" ) @dlt.expect_or_drop("positive_amount", "total_amount > 0") def orders_silver(): return ( dlt.read_stream("orders_bronze") .withColumn("total_amount", col("total_amount").cast("decimal(10,2)")) .filter(col("customer_id").isNotNull()) ) This snippet doesn't run on its own, it becomes a task inside a Lakeflow Job that can also kick off an AI/BI dashboard as soon as the "orders_silver" table is updated, using a data-driven trigger instead of a fixed time-based schedule. That binding between ingestion, transformation, and the next step in the flow, all in the same place, is what eliminates much of the glue work that today lives in a separate Airflow or Azure Data Factory. Governance without manual reconciliation The point that usually goes unnoticed by anyone looking only at the product surface is what happens underneath with Unity Catalog. Because ingestion, transformation, and orchestration share the same catalog, data lineage is captured end to end automatically, from the source table in SQL Server all the way to the final dashboard, without relying on manual annotation or on a third-party lineage system trying to infer relationships from logs. My take: this unification of governance is, in my view, Lakeflow's strongest argument, stronger even than the convenience of having a ready-made Workday connector. I've seen more than one data engineering project fail an audit not because the pipeline was wrong, but because nobody could prove with confidence where a specific number had come from, and how many transformations sat in between. Having that guaranteed structurally, rather than as a manual documentation process that is always out of date, changes the kind of conversation you have with the compliance team. System Tables come in as the observability piece: instead of building your own monitoring dashboard by querying each separate tool's API, you can build alerts and health reports directly on top of native SQL tables, with retention and schema standardized by Databricks itself. Cost: where the promise requires your own verification Databricks publishes some impressive cost-reduction figures, including a case of up to 83% lower ETL cost and up to 90x performance improvement reported by a specific customer. Those numbers deserve the standard skepticism any vendor benchmark does: they usually reflect a specific migration, from a poorly optimized stack to a new, well-tuned configuration. In practice, I'd test the real savings by running the same representative workload from your own environment on old vs. new, with equivalent cluster policies and compute mode (Performance vs. Standard on serverless compute), before using the marketing number in a budget justification to leadership. The mechanism behind the savings is real and makes technical sense: serverless compute with automatic optimization eliminates cold starts between sequential tasks in the same job, and granular per-task resource control avoids over-provisioning a cluster for a lightweight stage of the pipeline. That's different from guaranteeing that any migration will hit an 83% reduction. What this doesn't solve Consolidating ingestion, transformation, and orchestration into one platform doesn't remove the need for well-thought-out data modeling, nor does it replace architecture decisions about partitioning and clustering large tables. It also doesn't, on its own, solve the problem for teams that already have a heavy investment in Airflow with custom plugins that are hard to port, migrating legacy orchestration is still manual rewrite work, even with the dbt task and third-party integrations easing part of the path. And Lakeflow Connect's point-and-click connectors cover a specific set of sources, legacy systems or proprietary APIs without a native connector still depend on custom ingestion via Auto Loader or your own API. Bottom line Lakeflow doesn't invent a new data engineering paradigm, it removes the friction of coordinating three different tools to do what was always conceptually a single thing: bring data in from outside, transform it with confidence, and deliver it in the right shape to the next consumer. In practice: for anyone on Azure Databricks who already feels the pain of maintaining Data Factory, a third-party orchestrator, and a separate lineage catalog, it's worth piloting on a real pipeline before promising Databricks' cost-reduction number to anyone. References Databricks Blog, "Modernize your Data Engineering Platform with Lakeflow on Azure Databricks": https://www.databricks.com/blog/modernize-your-data-engineering-platform-lakeflow-azure-databricks Databricks Docs, "Get started: Build an ETL pipeline": https://docs.databricks.com/aws/en/getting-started/data-pipeline-get-started Microsoft Learn, "Tutorial: Build an ETL pipeline with Lakeflow pipelines - Azure Databricks": https://learn.microsoft.com/en-us/azure/databricks/getting-started/data-pipeline-get-started Microsoft Learn, "System tables": https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/
