Azure Databricks Cost Governance: Budgets, Tags, and Alerts Before the Invoice Arrives

Cost visibility on Azure Databricks does not have to wait for the monthly invoice, and for Azure customers specifically, the attribution trail can run straight into Azure Cost Management, not just Databricks system tables.

On October 7, 2026, Databricks published a practical guide on setting budgets and alerts for cloud data warehouse costs. The original post is written in general, cross-cloud terms. This article walks through the same problem from the Azure Databricks side specifically, using the account console, the REST API, and the system tables as they actually behave on Azure today, including two capabilities that go further than what the original post describes: per-user Unity Gateway budgets with usage blocking, and serverless usage policies whose tags propagate directly into Azure's own cost analysis tools.

[INSERT IMAGE HERE using the editor's own Insert Image button, not pasted markup. Upload a screenshot of the Azure Databricks account console Budgets tab (Usage > Budgets), or if the editor accepts an external URL, use a relevant Microsoft Learn budgets screenshot. Suggested alt text: "Azure Databricks account console showing the Budgets tab with monthly threshold and burn-rate chart".]

1. Why the bill always arrives as a surprise

The pattern repeats across data teams on any cloud: usage is not attributed to a team or a query, no team owns a budget number, the overspend only surfaces a billing cycle later, and manual spreadsheet review does not scale past a handful of warehouses. None of this is a tooling gap specific to Databricks, it is what happens when cost tracking is bolted on after the fact instead of built into how compute gets created in the first place.

Azure Databricks addresses this with four layers that build on each other: tag-based attribution, the system.billing.usage table as the source of truth, budgets scoped to workspaces or tags in the account console, and alerts that run on a schedule instead of waiting for a human to check a dashboard.

2. Attribution: tags, and the Azure-native alternative to tagging by hand

The baseline mechanism is the same one most clouds use: key-value tags on a SQL warehouse, set at creation or edit time, that later show up in the custom_tags column of the billing system table.

Tagging a warehouse through the REST API

curl -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/warehouses" \
  -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "finance-analytics",
    "cluster_size": "Small",
    "warehouse_type": "PRO",
    "tags": {
      "custom_tags": [
        { "key": "team", "value": "finance" },
        { "key": "cost-center", "value": "1234" }
      ]
    }
  }'

Two details matter when you actually run this. The warehouse tagging limit is under 45 tags per warehouse, and if a custom tag key collides with one of Databricks' own default tags (Vendor, ClusterId, SqlEndpointId, Creator), the custom one gets an x_ prefix during propagation rather than overwriting the default. Plan tag key names with that collision rule in mind.

Query tags: tagging at the session or statement level (Public Preview)

Warehouse tags attribute an entire warehouse to a team, but a shared warehouse serving multiple teams needs finer granularity than that. Query tags (Public Preview) solve this by letting you tag individual sessions or individual statements, independent of which warehouse they run on.

From the SQL editor, a notebook, or a dashboard, set tags for the rest of the session with the SET QUERY_TAGS statement:

-- Tag every subsequent query in this session
SET QUERY_TAGS['team'] = 'marketing', QUERY_TAGS['cost_center'] = '701';
-- Inspect what's currently set
SET QUERY_TAGS;
-- Clear a tag when the session changes purpose
SET QUERY_TAGS['cost_center'] = UNSET;

Session-level tags apply to every statement that follows until changed. Statement-level tags are a separate, narrower mechanism: pass a query_tags array in the request body of the Statement Execution API, and only that one statement carries the tag, with the session reverting to its own tags afterward:

curl -X POST "https://${DATABRICKS_HOST}/api/2.0/sql/statements" \
  -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "warehouse_id": "abc123",
    "statement": "SELECT * FROM samples.nyctaxi.trips LIMIT 10",
    "query_tags": [
        { "key": "team", "value": "engineering" },
        { "key": "env", "value": "prod" }
      ]
  }'

Tags land in the query_tags column of system.query.history, which needs its own access grant from an account admin if you do not already have it. Query it directly to see spend by query rather than by whole warehouse:

SELECT statement_id, query_tags, executed_by, start_time
FROM system.query.history
WHERE MAP_CONTAINS_KEY(query_tags, 'team')
  AND query_tags['team'] = 'marketing'
ORDER BY start_time DESC
LIMIT 100

The limits are tight enough to plan around: 20 user-specified tags per query, 10 KB of tags per session, 128 characters per key or value, and tag keys cannot contain commas, colons, hyphens, slashes, equals signs, or periods. A key starting with @@ is reserved, that is the prefix dbt and Power BI use for their own automatically applied tags (@@dbt_model_name, @@powerbi_report_id, and similar). Set a tag using a session configuration parameter instead of SQL and an invalid key gets silently dropped with a tag_invalid: true sentinel; set it through SET QUERY_TAGS and an invalid key fails the statement outright at execution time, which is the behavior worth relying on if you want bad tags to be loud rather than quiet.

The Azure-specific upgrade: serverless usage policies

Tagging every warehouse by hand does not scale, and nothing stops a team from forgetting it on the one warehouse that ends up costing the most. For serverless compute specifically, Azure Databricks has a more structural answer: serverless usage policies (Public Preview). A policy bundles a set of tags and gets assigned to users; any serverless notebook, job, pipeline, serving endpoint, or app that user creates inherits the policy's tags automatically, with no manual tagging step per resource.

The detail that matters most for an Azure-focused reader: serverless usage policy tags propagate directly into the billing records emitted to Azure Cost Analysis, not only into Databricks' own system.billing.usage table. That means the same tag you use to scope a Databricks budget is also what your FinOps team sees inside the Azure portal's own cost views, without a separate mapping step.

Known limitations worth planning around: existing notebooks, jobs, and pipelines do not retroactively pick up a policy, you have to assign it manually to each existing asset; classic (non-serverless) compute is not covered at all; and a pipeline triggered by a job does not inherit the job's policy, the pipeline needs its own.

3. The source of truth: system.billing.usage

Every budget, every alert, and every dashboard ultimately reads from one Unity Catalog system table: system.billing.usage. Records typically land within 12 hours. The schema includes, among other columns, custom_tags (a map, populated by both manual warehouse tags and serverless usage policy tags), usage_metadata.warehouse_id, usage_quantity (in the unit given by usage_unit, typically DBU), and record_type, which can be ORIGINAL, RETRACTION, or RESTATEMENT when Databricks issues a billing correction.

Note that query tags from the previous section live in system.query.history, not here, custom_tags on this table only reflects warehouse-level and policy-level tags. The two tables are separate for this reason, use system.query.history for query-level cost investigation and this table for warehouse and account-level aggregation.

That last detail is not cosmetic. A naive SUM(usage_quantity) without accounting for retractions can overcount. The pattern that handles it correctly:

SELECT
  usage_metadata.job_id,
  usage_start_time,
  usage_end_time,
  SUM(usage_quantity) AS usage_quantity
FROM system.billing.usage
GROUP BY ALL
HAVING usage_quantity != 0

Aggregating before filtering lets the retraction record cancel out the original it corrects, so a corrected job's usage does not get double-counted.

Applied to the attribution problem directly, two queries worth keeping in a cost dashboard:

-- SQL warehouse spend by team, last 30 days
SELECT
  custom_tags['team'] AS team,
  usage_metadata.warehouse_id,
  SUM(usage_quantity) AS dbus_consumed
FROM system.billing.usage
WHERE billing_origin_product = 'SQL'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL
ORDER BY dbus_consumed DESC
-- Unattributed SQL usage: the attribution gap, should trend to zero
SELECT
  usage_date,
  SUM(usage_quantity) AS unattributed_dbus
FROM system.billing.usage
WHERE billing_origin_product = 'SQL'
  AND (custom_tags['team'] IS NULL OR custom_tags['team'] = '')
GROUP BY usage_date
ORDER BY usage_date DESC

For a dollar figure rather than a DBU count, join against the system.billing.list_prices system table on sku_name and the price's effective date range. I am not reproducing that join here because I have not verified its exact column names against the current schema, and getting a pricing join subtly wrong is worse than not showing it. Check the list prices system table reference on Microsoft Learn (learn.microsoft.com/en-us/azure/databricks/admin/system-tables/pricing) before writing that query yourself.

4. Budgets in the account console: more than a threshold and an email

Budgets live in the Azure Databricks account console (accounts.azuredatabricks.net), under Usage > Budgets. A budget can scope to specific workspaces, to custom tags, or to both, and if you leave both empty it tracks your entire account.

What the Azure documentation describes goes further than a single alert value:

• Up to four monthly thresholds per budget, each with its own email recipient list.
• Up to 1,000 budgets per account.
• Budgets scoped to Unity Gateway (the routing layer in front of Pay-Per-Token foundation model inference, batch ai_query, and external model provider usage) support a per-user spend threshold, on top of the shared thresholds. Cross a per-user threshold and that specific user's requests can optionally be blocked, with the option to grant named users or groups a higher override limit.
• Budget visibility and editing is governed by a dedicated permission role, not just account admin status, so a team lead can see their own budget without account-wide access.

The important caveat, stated plainly in the documentation: budgets are a monitoring mechanism, not a hard spending cap, except specifically for Unity Gateway and Genie budgets, where the optional "Block usage" setting does enforce a stop. Even there, enforcement is approximate, based on a near-real-time cost estimate, so a user can be blocked slightly before or after the literal threshold. For standard SQL and compute budgets, nothing stops the spend, the budget's job is to get a human's attention fast.

A layered pattern, adapted for Azure, that tends to hold up in practice: a BI Serving + Production budget at $8,000/month scoped to UseCase:BIServing and Env:Prod, alerting the platform team; an Analytics Ad-Hoc budget at $3,000/month scoped to UseCase:AdHoc, alerting the analytics manager; a Marketing Analytics budget at $2,500/month scoped to Team:Marketing, alerting the marketing data lead; and an account-wide safety net at $25,000/month with no tag scope, alerting the CTO and FinOps. Team-level budgets for accountability, plus one account-wide budget as the net that catches whatever slips past every tag filter. The values above are illustrative, size them against your own baseline.

5. Alerts that run before the threshold, not after

Databricks SQL alerts run a query on a schedule and evaluate a condition against the result, independent of whether anyone opens a dashboard. On Azure Databricks, the current alert engine resolves to one of three states: OK, TRIGGERED, or ERROR, a simplification from the legacy alert system's extra UNKNOWN state.

A daily threshold alert, scheduled once a day, built on the team-attribution query above:

SELECT
  custom_tags['team'] AS team,
  SUM(usage_quantity) AS dbus_today
FROM system.billing.usage
WHERE billing_origin_product = 'SQL'
  AND usage_date = current_date()
GROUP BY ALL
HAVING dbus_today > 500  -- adjust to your own per-team daily baseline

A threshold alert only catches a problem on the day it happens. A pacing alert catches it earlier, by projecting month-end spend from the days elapsed so far:

WITH month_to_date AS (
  SELECT
    SUM(usage_quantity) AS mtd_dbus,
    DAY(current_date()) AS days_elapsed,
    DAY(LAST_DAY(current_date())) AS days_in_month
  FROM system.billing.usage
  WHERE billing_origin_product = 'SQL'
    AND usage_date >= DATE_TRUNC('MONTH', current_date())
)
SELECT
  mtd_dbus,
  mtd_dbus / days_elapsed * days_in_month AS projected_month_end_dbus
FROM month_to_date
HAVING projected_month_end_dbus > 15000  -- adjust to your monthly DBU budget

Schedule the pacing alert every four hours and route it to Slack or email through the alert's notification settings, the same pattern the original Databricks post recommends. Running it more often than that mostly adds noise, since system.billing.usage itself only refreshes every few hours.

One cost that neither alert catches on its own: materialized view refreshes bill as serverless pipeline usage, under billing_origin_product = 'DLT', not as warehouse usage. If your team only watches SQL spend, a materialized view refreshed far more often than its source data actually changes can run invisibly. Align refresh frequency to real data freshness, and include DLT in your cost queries, not only SQL.

6. Where this still needs a human

None of the above replaces judgment on the two levers that move cost the most: warehouse type and warehouse size. Azure Databricks documentation still points to SQL Serverless as the default starting point for most workloads, since it starts in seconds, scales to zero between queries, and bills only for execution time, and to predictive optimization (learn.microsoft.com/en-us/azure/databricks/optimizations/predictive-optimization), enabled by default on accounts created after November 11, 2024, to keep OPTIMIZE, VACUUM, and clustering key maintenance running automatically on managed Unity Catalog tables, instead of a scheduled job someone has to remember to run.

Auto-stop is worth checking explicitly rather than trusting the default: Pro and Classic warehouses default to 45 minutes of idle time before stopping, serverless defaults to 10 minutes, and serverless can be brought down to as low as 1 minute through the API, below what the UI exposes. For a warehouse that serves a handful of scheduled queries a day, 45 idle minutes can be the single biggest line item.

7. My perspective on this

The feature set here is more capable on Azure specifically than the general cross-cloud framing suggests, and the detail that stands out to me is the Azure Cost Analysis propagation for serverless usage policy tags. That is not a Databricks convenience feature, it closes the exact gap that usually breaks cost governance in a real organization: the Databricks admin owns one view of spend, the cloud FinOps team owns another, and reconciling the two by hand is where attribution efforts actually die. If the same tag shows up in both without a manual export step, that reconciliation work goes away.

I would start with serverless usage policies before hand-tagging every warehouse, specifically because the policy approach survives a new team member forgetting to tag something, while manual tagging does not. The known limitation that existing assets do not retroactively inherit a new policy is real friction, budget time for a one-time pass over your current notebooks, jobs, and pipelines when you adopt this, it will not happen automatically.

And treat every budget as a monitoring signal, not a spending cap, with the explicit exception of Unity Gateway and Genie budgets where blocking is actually available. Assuming the wrong one of those two behaviors for a given budget is how a team ends up either silently overspending or unexpectedly blocked mid-production.

Conclusion

Cost governance on Azure Databricks is not a single feature, it is tags feeding a system table, a system table feeding budgets and alerts, and budgets feeding back into how the next warehouse gets created. The Azure-specific pieces, Unity Gateway per-user budgets with blocking and serverless usage policy tags flowing into Azure Cost Analysis, are worth adopting even if you came from the general cross-cloud version of this guide expecting less.

Official references

Databricks Blog, "Set Budgets and Alerts for Cloud Data Warehouse Costs" (www.databricks.com/blog/set-budgets-and-alerts-cloud-data-warehouse-costs)
Microsoft Learn, "Create and monitor budgets - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/account-settings/budgets)
Microsoft Learn, "Attribute usage with serverless usage policies - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/usage/budget-policies)
Microsoft Learn, "Billable usage system table reference - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/system-tables/billing)
Microsoft Learn, "Databricks SQL alerts - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/sql/user/alerts/)
Microsoft Learn, "Use tags to attribute and track usage - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/account-settings/usage-detail-tags)
Microsoft Learn, "Query tags - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/sql/user/queries/query-tags)
Microsoft Learn, "Governance Hub - Azure Databricks" (learn.microsoft.com/en-us/azure/databricks/admin/governance-hub/)