By Amit Damle, RK Iyer
The question that never stays answered — until now
As Azure Databricks workspaces evolve, complexity creeps in unnoticed. Every Azure Databricks conversation with customers eventually lands on the same question: “What do we actually have in this workspace?” Over time, clusters multiply, jobs get cloned, warehouses are spun up for one-off demos and forgotten, and Unity Catalog keeps expanding until it’s hard to reason about. In most enterprises, each business or data science team operates its own workspace, while the central platform or operations team has little to no visibility into what’s being created or why. Teams often spend days—or weeks—trying to piece together what exists, who owns it, and the business purpose behind it, only to realize they still don’t have the full picture. And when the same question comes up next quarter, the cycle starts all over again.
To address this, we built a utility that helps customers answer exactly that—by providing a single pane of glass for all Databricks assets through comprehensive cataloging and usage analysis. The utility works in two phases: Discovery and Analysis. This post focuses on the first step—the Discovery phase, where we establish a clear, authoritative inventory of everything that exists in the workspace.
What the Discovery Phase delivers?
Think of the Discovery phase as a workspace health assessment. Once configured against a target workspace, the utility runs in a selected mode and consolidates all discovered assets into a centralized, Delta-based repository. The result is a structured, queryable, and dashboard-ready metadata store.
Behind the scenes, ten purpose-built scanners run in a tiered and parallelized architecture, enabling a fast yet comprehensive scan of the entire workspace.
|
Scanner |
What is Cataloged |
|
Clusters |
Interactive, job, SQL — configs, policies, pools |
|
Jobs |
Workflows, schedules, tasks, run history |
|
Warehouses |
SQL endpoints, sizes, serverless settings |
|
Pipelines |
Delta Live Tables and their state |
|
Unity Catalog |
Catalogs, schemas, tables, volumes |
|
Workspace Objects |
Notebooks, repos, ML experiments, serving endpoints, alerts, Genie spaces |
|
Security |
Identity, network, data protection settings |
|
Billing |
30–180 days of DBU usage by SKU and product |
|
Utilization |
Real CPU, memory, runtime patterns (deep scan) |
|
Spark Job Optimizer (plugin) |
Skew, spill, small files, broadcast hints (deep scan) |
Design Overview
|
# |
Block |
Role |
Contents / Flow |
|
1 |
Source |
Starting point — the Databricks environments being discovered. |
One or more Azure Databricks workspaces. Auth via OAuth. Outputs an authenticated WorkspaceClient to the Orchestrator. |
|
2 |
Orchestrator
|
The brain of the utility — coordinates scanning, concurrency, retries, timing. |
Tiered thread-pool executor, scan config (mode, billing window, UC depth, max workers). Dispatches scanners in controlled waves. |
|
3 |
Tier 1 Scanners |
Lightweight, high-concurrency scans. Run first for quick signal. |
Clusters, Warehouses, Pipelines, Security. Up to 12 workers, 10-min timeout. Artifacts flow to the Centralized Repository. |
|
4 |
Tier 2 Scanners |
High-volume scans. Controlled concurrency to avoid API throttling. |
Jobs, Workspace Objects (notebooks, repos, experiments, serving, alerts, Genie), Unity Catalog, Billing (30–180 days DBU). 1/2 workers, 30-min timeout. |
|
5 |
Tier 3 Scanners |
Sequential, analysis-grade scans (deep scan only). |
Utilization (CPU, memory, SQL usage patterns) and Spark Job Optimizer plugin (skew, spill, small files, broadcast hints). Runs after Tiers 1 & 2. |
|
6 |
Centralized Repository |
The catalog of truth — where all output lands, timestamped and queryable. |
Unity Catalog Delta tables (dashboard-ready) plus portable JSON and CSV exports for offline sharing or downstream tools. |
|
7 |
Single Pane of Glass |
The user-facing view — insight at a glance. |
Pre-built Lakeview dashboard: KPI strip, inventory charts, and week-over-week trends. Refresh to see current workspace state. |
Why users love the view — visualization that earns its keep
This is where the Discovery phase stops being just a scan and starts becoming a decision-making tool. Because everything is consolidated into a single, Unity Catalog–backed source of truth, the Lakeview dashboard delivers a genuine single pane of glass for the entire Databricks workspace.
At a glance, you get:
- KPI strip at the top — total clusters, active jobs, UC tables, SQL warehouses, DLT pipelines, workspace objects. One glance, one number each.
- Inventory charts — clusters by type, jobs by schedule, warehouses by size, tables by catalog. The shape of your workspace becomes obvious.
- The “that doesn’t look right” moments — The idle SQL warehouse with zero queries, the cluster running the wrong runtime, the notebook floating outside any repo. These surface instantly, without hunting. Change over time — because every scan is timestamped, you can literally see your platform grow (or sprawl) week over week.
In the first customer walkthrough, the platform team identified an always-on SQL warehouse with zero queries and three jobs running on the wrong compute tier—all within the first 30 minutes. That single view paid for the project.
Sample Item Catalog
Closing thoughts
The Discovery phase isn’t about governance for governance’s sake—it’s about clarity. Before teams can optimize costs, improve performance, or enforce standards, they first need a reliable answer to a basic question: what actually exists today? By giving platform and operations teams a single, authoritative view of all Databricks assets—grounded in data, not tribal knowledge—Discovery turns guesswork into informed decisions. In the next phase, Analysis, that foundation is used to go deeper: identifying inefficiencies, risks, and opportunities to simplify and optimize the platform. But it all starts here—by finally knowing what you have.
Special thank you to Antony Bitar, Collin Brian and Jason Pereira for their support in reviewing the content.
Microsoft