Azure Databricks Unity Catalog connector - Metadata

For the Azure Databricks Unity Catalog connector, I believe Microsoft Purview scans and collects the following technical metadata:

• Table name
• View name
• Column names
• Catalog name
• Schema name
• Metastore information
• Table and column descriptions from Databricks comments
• Unity Catalog tags, which are stored within Purview properties
• Table, view and column lineage where supported

Storage location information may also appear within the technical properties of the asset, depending on the Databricks object. However, Microsoft does not appear to clearly document storage location as guaranteed metadata from every scan.

Can anybody explain this further or confirm whether they have seen different behaviour?

Where the metadata is stored or managed in Azure Databricks

Metadata collected by Microsoft Purview

Where it is stored or managed in Azure Databricks

Metastore

Unity Catalog Metastore

Catalog name

Unity Catalog

Schema name

Unity Catalog Schema

Table name

Unity Catalog Table metadata

View name

Unity Catalog View metadata

Column names

Table schema in Unity Catalog

Table description

Unity Catalog Table Comment

Column description

Unity Catalog Column Comment

Unity Catalog tags

Tags applied to Unity Catalog objects such as tables, views and columns

Table and column lineage

Unity Catalog lineage metadata and system lineage tables

Storage location

Table metadata where applicable, particularly for external tables

I believe table and column descriptions are stored in Databricks as comments against the Unity Catalog object.

These comments can then be scanned into Microsoft Purview and displayed as descriptions.

The Microsoft documentation appears clearer for column comments than it does for table comments, so I am planning to test this myself.

Does anybody have practical experience of this behaviour?

Unity Catalog tags

Tags are stored as metadata against Unity Catalog securable objects.

Tag information can also be queried through Unity Catalog INFORMATION_SCHEMA, including TABLE_TAGS and the corresponding column tag metadata.

The core Unity Catalog structure is:

• Unity Catalog Metastore
• Catalog
• Schema
• Table or View
• Columns

Additional metadata can include:

• Comments
• Tags
• Lineage
• Storage location where applicable

What Microsoft currently documents as collected by the Microsoft Purview Azure Databricks Unity Catalog connector

Databricks Unity Catalog metadata

Collected into Purview?

Notes

Metastore

Yes

Represented in the Purview hierarchy

Catalog name

Yes

Unity Catalog catalog

Schema name

Yes

Unity Catalog schema

Table name

Yes

Creates/discovers the table data asset

View name

Yes

Views are supported

Column names

Yes

Schema/column metadata is collected

Table comment / description

Yes

Your observation is correct: Databricks table comments/descriptions can appear as the Purview Data Asset description

Column comments / descriptions

Yes

Microsoft explicitly states that Databricks column comments are displayed as column descriptions in Purview

Unity Catalog tags

Yes

Microsoft explicitly states that Unity Catalog tags are scanned into Purview Properties

Table/view lineage

Yes, subject to prerequisites

Extracted from Unity Catalog system lineage tables

Column lineage

Yes, subject to prerequisites

Supported, although there are lineage limitations

External tables

Metadata yes

External table metadata is supported; lineage for external tables is not supported

Microsoft's connector documentation currently lists the following core hierarchy and metadata as being extracted by the scan:

• Metastore
• Catalogs
• Schemas
• Tables and Columns
• Views and Columns
• Unity Catalog Tags

Data Asset descriptions

This behaviour appears to be genuine.

Microsoft specifically documents that a Databricks column comment can be used as the description displayed in Microsoft Purview.

Microsoft also notes that setting a Databricks table column comment to an empty string prevents the column description from being displayed in Purview.

For practical purposes, the relationship is:

• Databricks comment or description
• Purview scan
• Microsoft Purview description

This is explicitly documented for column comments.

Table comments also appear to be surfaced as the Data Asset description through the connector, based on observed behaviour, although Microsoft's documentation is less explicit about this.

Has anybody seen this working consistently for table descriptions?

Unity Catalog tags

Unity Catalog tags are supported by the current connector.

Microsoft documentation states that tags from Unity Catalog are extracted during scanning and displayed within Microsoft Purview Properties.

There is a documented exception where tags are not supported when the scan uses the Kubernetes Self-hosted Integration Runtime option.

That exception should not apply where Kubernetes SHIR is not being used.

Storage location

The official connector documentation clearly documents the Unity Catalog hierarchy and the metadata listed above. However, it does not appear to explicitly list the table storage location or storage path as a guaranteed metadata element within the supported metadata capabilities. Storage location information may therefore appear for some Databricks objects, particularly external tables, but I would not currently treat it as guaranteed metadata from every Purview Unity Catalog scan without further testing.

My current understanding is that a Microsoft Purview scan of Azure Databricks Unity Catalog can collect:

Metastore

Catalog

Schema

Table

View

Columns

Table descriptions or comments

Column descriptions or comments

Unity Catalog tags

Table and view lineage where supported

Column lineage where supported

External table metadata

Storage location may also be visible where applicable, but this appears less clearly documented as a guaranteed field? Does this sound correct based on other people's experience of Microsoft Purview scans of Azure Databricks Unity Catalog data sources?