Pinned Posts
Forum Widgets
Latest Discussions
Azure Databricks Unity Catalog connector - Metadata
For the Azure Databricks Unity Catalog connector, I believe Microsoft Purview scans and collects the following technical metadata: Table name View name Column names Catalog name Schema name Metastore information Table and column descriptions from Databricks comments Unity Catalog tags, which are stored within Purview properties Table, view and column lineage where supported Storage location information may also appear within the technical properties of the asset, depending on the Databricks object. However, Microsoft does not appear to clearly document storage location as guaranteed metadata from every scan. Can anybody explain this further or confirm whether they have seen different behaviour? Where the metadata is stored or managed in Azure Databricks Metadata collected by Microsoft Purview Where it is stored or managed in Azure Databricks Metastore Unity Catalog Metastore Catalog name Unity Catalog Schema name Unity Catalog Schema Table name Unity Catalog Table metadata View name Unity Catalog View metadata Column names Table schema in Unity Catalog Table description Unity Catalog Table Comment Column description Unity Catalog Column Comment Unity Catalog tags Tags applied to Unity Catalog objects such as tables, views and columns Table and column lineage Unity Catalog lineage metadata and system lineage tables Storage location Table metadata where applicable, particularly for external tables I believe table and column descriptions are stored in Databricks as comments against the Unity Catalog object. These comments can then be scanned into Microsoft Purview and displayed as descriptions. The Microsoft documentation appears clearer for column comments than it does for table comments, so I am planning to test this myself. Does anybody have practical experience of this behaviour? Unity Catalog tags Tags are stored as metadata against Unity Catalog securable objects. Tag information can also be queried through Unity Catalog INFORMATION_SCHEMA, including TABLE_TAGS and the corresponding column tag metadata. The core Unity Catalog structure is: Unity Catalog Metastore Catalog Schema Table or View Columns Additional metadata can include: Comments Tags Lineage Storage location where applicable What Microsoft currently documents as collected by the Microsoft Purview Azure Databricks Unity Catalog connector Databricks Unity Catalog metadata Collected into Purview? Notes Metastore Yes Represented in the Purview hierarchy Catalog name Yes Unity Catalog catalog Schema name Yes Unity Catalog schema Table name Yes Creates/discovers the table data asset View name Yes Views are supported Column names Yes Schema/column metadata is collected Table comment / description Yes Your observation is correct: Databricks table comments/descriptions can appear as the Purview Data Asset description Column comments / descriptions Yes Microsoft explicitly states that Databricks column comments are displayed as column descriptions in Purview Unity Catalog tags Yes Microsoft explicitly states that Unity Catalog tags are scanned into Purview Properties Table/view lineage Yes, subject to prerequisites Extracted from Unity Catalog system lineage tables Column lineage Yes, subject to prerequisites Supported, although there are lineage limitations External tables Metadata yes External table metadata is supported; lineage for external tables is not supported Microsoft's connector documentation currently lists the following core hierarchy and metadata as being extracted by the scan: Metastore Catalogs Schemas Tables and Columns Views and Columns Unity Catalog Tags Data Asset descriptions This behaviour appears to be genuine. Microsoft specifically documents that a Databricks column comment can be used as the description displayed in Microsoft Purview. Microsoft also notes that setting a Databricks table column comment to an empty string prevents the column description from being displayed in Purview. For practical purposes, the relationship is: Databricks comment or description Purview scan Microsoft Purview description This is explicitly documented for column comments. Table comments also appear to be surfaced as the Data Asset description through the connector, based on observed behaviour, although Microsoft's documentation is less explicit about this. Has anybody seen this working consistently for table descriptions? Unity Catalog tags Unity Catalog tags are supported by the current connector. Microsoft documentation states that tags from Unity Catalog are extracted during scanning and displayed within Microsoft Purview Properties. There is a documented exception where tags are not supported when the scan uses the Kubernetes Self-hosted Integration Runtime option. That exception should not apply where Kubernetes SHIR is not being used. Storage location The official connector documentation clearly documents the Unity Catalog hierarchy and the metadata listed above. However, it does not appear to explicitly list the table storage location or storage path as a guaranteed metadata element within the supported metadata capabilities. Storage location information may therefore appear for some Databricks objects, particularly external tables, but I would not currently treat it as guaranteed metadata from every Purview Unity Catalog scan without further testing. My current understanding is that a Microsoft Purview scan of Azure Databricks Unity Catalog can collect: Metastore Catalog Schema Table View Columns Table descriptions or comments Column descriptions or comments Unity Catalog tags Table and view lineage where supported Column lineage where supported External table metadata Storage location may also be visible where applicable, but this appears less clearly documented as a guaranteed field? Does this sound correct based on other people's experience of Microsoft Purview scans of Azure Databricks Unity Catalog data sources?
How can we identify the size of a Microsoft Purview DLP policy and individual DLP rules?
Hello Microsoft Community, According to the Microsoft Purview DLP policy reference, the following platform limits apply: Maximum size of a DLP policy: 100 KB Maximum size of an individual DLP rule: 100 KB or 102,400 characters Maximum number of rules within a policy: Limited by the overall policy size I would like to understand how these limits are calculated and how administrators can monitor the current size of an existing DLP policy or rule. Could someone please clarify the following? What exactly is included when calculating the 100 KB DLP policy size? Does the policy size include only policy-level configuration, such as locations, users, groups, administrative units, inclusions and exclusions, or does it also include the combined definitions of all rules associated with the policy? What exactly is included in the 100 KB individual rule size? Sensitive information types and their GUIDs Condition groups Instance-count and confidence-level configurations Exceptions Endpoint DLP restrictions User notifications and policy-tip messages Override options Alert and incident-report settings Recipient lists Advanced rule configuration For example, does it include: The documentation describes the rule limit as “100 KB (102,400 characters).” Is the limit based on: The number of characters The UTF-8 or Unicode byte size The serialized JSON/XML representation An internally generated policy payload Is there a supported method in the Microsoft Purview portal, Security & Compliance PowerShell, Microsoft Graph, or another API to display the current size of: A DLP policy Each DLP rule The remaining available policy capacity Can the size be estimated by exporting the output from Get-DlpCompliancePolicy and Get-DlpComplianceRule? If so, which properties should be included in the calculation, and what encoding or serialization format should be used? Does the 100 KB policy limit represent the combined size of the policy and all its rules, or are the policy and rule limits evaluated independently? What error or warning is generated when a policy or rule approaches or exceeds the limit? Is there any notification before the limit is reached? We are designing global Microsoft Purview DLP policies containing multiple Sensitive Information Types, condition groups and workload-specific rules. We need a reliable method to measure policy and rule sizes during design and ongoing policy governance. Any official guidance, supported script, API property, or calculation method would be appreciated.
Acessos pastas SharePoint / SharePoint folder access
É possível identificar somente acessos a pastas do SharePoint? Algumas informações internas da companhia foram vazadas e precisamos saber quem acessou a pastas do Sharepoint onde estavam essas informações. Outra situação, é que conseguimos identificar que nessa mesma pasta alguns arquivos foram baixados, mas no relatório o usuário aparece "app@sharepoint". Alguém saberia explicar? It is possible to identify access to SharePoint folders? Some of the company's international information was leaked, and we need to know who accessed the SharePoint folders containing that data. Another issue is that we identified files being downloaded from that same folder, but the user appears as "app@sharepoint". Could anyone explain this?
Purview archive shipped. Delete policy is still armed.
Roadmap 561208 went GA over the last few weeks with very little noise. Purview retention policies in Data Lifecycle Management can now move inactive OneDrive and SharePoint content into Microsoft 365 Archive. The framing is storage cost. I want to talk about something else, because I think there's a gap here that's going to catch people out, and I'd like to know whether anyone else is seeing it. Archive is a storage tier. It is not a retention state. An archived SharePoint site stays fully in scope of every retention policy that applied to it while it was active. Nothing about archiving protects content. Now layer that onto conflict resolution. Retention beats deletion, so a site policy set to delete after 2 years goes dormant while a label is retaining the item. Dormant, not cancelled. So picture this: A file carries a retention label with a 4-year retention period. The site it lives on has a delete-after-2-years retention policy. The label's end-of-retention action is deactivate retention settings. For four years, everything looks correct. The label holds. The delete policy sits there doing nothing. Then the label reaches end of retention and deactivates. The shield is gone. The delete instruction wakes up, unopposed, two years overdue, and the file goes. Not to archive. Just gone. The reflex here is "explicit wins over implicit." It doesn't apply. That principle only arbitrates between competing delete actions. A label that expires with no delete action of its own brings nothing to the contest. There is no explicit action to outrank the policy with. "Do nothing" sounds like a safe default. It's actually a decision to stop defending the file. Right now you cannot set a label to archive at end of retention. The options are deactivate, delete automatically, disposition review, or hand it to a Power Automate flow. Archive isn't on the list. Microsoft shipped archive at the policy level and left the label level to flows and webhooks. That reads to me like they know the gap exists and haven't decided how to close it. And here's my suspicion about why this shipped now. Archived content is excluded from Copilot indexing. That's Microsoft's own framing in the message center post. Which suggests this feature is going to evolve on Copilot's roadmap rather than records management's. The moment "archive" becomes the button you press to clean up Copilot's answers, people will press it constantly, at scale, without once checking what policy is sitting underneath. Meanwhile Exchange has nothing. Purview retention still can't move mail into an archive mailbox, and we're all still running MRM tags next to Purview years after being told MRM was legacy. Has anyone hit the deactivate-into-delete scenario in a live tenant, or is this still theoretical? And does anyone read the label-level gap differently than I do? Genuinely open to being told I've got this wrong.Published Glossary Terms Not Appearing When Editing Microsoft Purview Data Assets
Help and Support for All Problem Published glossary terms were not available for selection when editing Data Map assets in Microsoft Purview. Although the glossary terms had been created and published within Governance Domains, and the user had the appropriate permissions, the Glossary Terms dropdown remained empty. This prevented glossary terms from being assigned to data assets or individual columns. Symptoms The following behaviour was observed: Published glossary terms existed within Governance Domains in the Unified Catalog. User permissions had been verified and were correctly configured. The Glossary Terms field was visible when editing a Data Map asset. The dropdown opened successfully but did not display any available glossary terms. Glossary terms could not be associated with data assets or columns. Root Cause The issue occurred because Asset Curation had not been enabled in Microsoft Purview. Although glossary terms had been created and published within Governance Domains in the Unified Catalog, the Data Map asset curation experience had not been configured to use Unified Catalog glossary terms. As a result, the Glossary Terms field was displayed when editing an asset, but no terms were available for selection. Microsoft Purview requires Asset Curation to be enabled and configured before Unified Catalog glossary terms can be assigned to Data Map assets and columns. Resolution The following steps resolved the issue: Confirmed that no Classic Glossaries, Classic Glossary Terms or Term Templates existed in the Purview account. Enabled Asset Curation (Preview). Configured Microsoft Purview to use Unified Catalog Glossary Terms. Completed the Asset Curation configuration wizard. Edited a Data Map asset and verified that published glossary terms were now displayed. Successfully assigned glossary terms to both Data Map assets and individual columns. Because no Classic Glossary content existed, there was no glossary content that required migration. Outcome After Asset Curation was enabled and configured, published Unified Catalog glossary terms became available in the Glossary Terms dropdown when editing Data Map assets. Glossary terms could then be successfully assigned to both assets and columns. Key Takeaway Creating and publishing glossary terms within Governance Domains is not sufficient on its own. To use Unified Catalog glossary terms when curating Data Map assets and columns, Asset Curation must also be enabled and configured to use Unified Catalog Glossary Terms.Microsoft Purview Unified Catalog; Governance Domains and Business Concepts
I've been using the attached artefacts for some time to help explain the knowledge exchange aspects of Microsoft Purview Unified Catalog, particularly how Governance Domains and Business Concepts work together to provide business context, ownership, stewardship and operational insights. They have been useful in workshops with data architects, governance professionals, product owners and business stakeholders to demonstrate how concepts fit together within a governance domain and contribute towards trusted information and better business outcomes. I'm interested in hearing from the wider Purview community: Do these artefacts accurately represent the intent and capabilities of Governance Domains within Microsoft Purview? Are there any concepts that you feel are missing, over-emphasised, or could be represented more clearly? How are others explaining Governance Domains and Business Concepts to non-technical stakeholders? Any feedback, suggestions, or alternative approaches would be greatly appreciated. I'm always looking to refine these materials and make them more useful for organisations adopting Purview Unified Catalog. #MicrosoftPurview #DataGovernance #DataManagement #Metadata #DataProducts #MicrosoftData #Purview #DataArchitecture #UnifiedCatalogPurview Data Map – Proposed Domain & Collection Structure
Microsoft Purview Data Map – Proposed Domain & Collection Structure This proposed Microsoft Purview Data Map domain and collection structure ensures that users responsible for specific data assets can be granted precisely scoped permissions—particularly for updating metadata—by mapping Business Units, Departments, Teams, and environments in a clear hierarchy that allows RBAC inheritance to assign the right level of access to the right people. Domain Name Data Catalogue (Short, clear, governance-aligned name to avoid UI truncation and scripting issues.) Collection Path Data Catalogue → Business Units → Departments → Teams → [Prod | Non-Prod] Level 1: Business Units Level 2: Departments (within each Business Unit) Level 3: Teams (within each Department) Optional: Environment segregation under Teams (Prod / Non-Prod) Reasons & Requirements 1. Domain Naming Short, clear name avoids UI truncation and scripting issues. Detailed descriptions stored in metadata; name remains simple for automation and future-proofing. 2. Structure Alignment Alignment with organisational charts and unified governance hierarchy: Business Units → Departments → Teams Provides intuitive navigation and meaningful context for users. 3. Hierarchy Depth Limited to 4–5 levels for usability and RBAC inheritance. Avoids unnecessary complexity while maintaining clarity. 4. Environment Handling Prod / Non-Prod split under Teams for simplicity. Additional environments only if governance differs significantly. 5. RBAC & Ownership Permissions align with organisational roles. Supports the principle of least privilege. 6. Scanning & Policy Scans assigned at Team level for precise governance. Policies inherit from higher levels for consistency. Selective scanning preferred for cost efficiency. 7. Best Practice Compliance Matches Microsoft guidance: short names, shallow hierarchy, environment segregation. Clear distinction between governance path and technical hierarchy. Role Assignment in Collections Data Curator Role Designed for users who: Edit and update metadata. Manage business context for assets within the collection. Assign to: Data Owners (Directorate level). Data Stewards (Team level). Data Product Owners / Asset Managers (for their own assets). Why at Collection Level? RBAC in Purview inherits down the collection hierarchy: Assign at Team collection → edit metadata for all assets in that Team. Assign at Group or Directorate level → edit metadata for all child collections. Ensures least privilege and ownership-based editing. Best Practice Read-only roles (Data Reader) applied broadly for transparency. Data Curator scoped to the lowest level where the user has responsibility (usually Team). Avoid assigning Data Curator at the root unless absolutely necessary.Secure your data—Microsoft Purview at Ignite 2025
Security is a core focus at Microsoft Ignite this year, with the Security Forum on November 17, deep dive technical sessions, theater talks, and hands-on labs designed for security leaders and practitioners. Join us in San Francisco, November 17–21, or online, November 18–20, to learn what’s new and what’s next across data security, compliance, and AI. This year’s sessions and labs will help you prevent data exfiltration, manage insider risks, and enable responsible AI adoption across your organization. Featured sessions: BRK250: Preventing data exfiltration with a layered protection strategy Learn how Microsoft Purview enables a layered approach to data protection, including AI and non-AI apps, devices, browsers, and networks. BRK257: Drive secure Microsoft 365 Copilot adoption using Microsoft Purview Discover built-in safeguards to prevent data loss and insider risks as you scale Copilot and agentic AI. LAB548: Prevent data exposure in Copilot and AI apps with DLP Configure DLP policies to protect sensitive data across Microsoft 365 services and AI scenarios. Explore and filter the full security catalog by topic, format, and role: aka.ms/Ignite/SecuritySessions. Why attend: Ignite is your chance to see the latest Purview features, connect with product experts, and get hands-on with new compliance and data protection tools. Microsoft will also preview future enhancements for agentic AI and unified data governance. Security Forum (November 17): Kick off with an immersive, in‑person pre‑day focused on strategic security discussions and real‑world guidance from Microsoft leaders and industry experts. Select Security Forum during registration. Connect with peers and security leaders through these signature security experiences: Security Leaders Dinner—CISOs and VPs connect with Microsoft leaders. CISO Roundtable—Gain practical insights on secure AI adoption. Secure the Night Party—Network in a relaxed, fun setting. Register for Microsoft Ignite >October 16 | What’s New in Copilot in Microsoft Purview
Speaker: Patrick David, Principal Product Manager, CxE CAT Compliance Join us for an insider’s look at the latest innovations in Microsoft Purview —where alert triage agents for DLP and IRM are transforming how we respond to sensitive data risks and improve investigation depth and speed. We’ll also dive into powerful new capabilities in Data Security Posture Management (DSPM) with Security Copilot, designed to supercharge your security insights and automation. Whether you're driving compliance or defending data, this session will give you the edge. Register now. Check out the rest of the Security Copilot Skilling Series here.Purview YouTube Show and Podcast
I am a Microsoft MVP who co-hosts All Things M365 Compliance with Ryan John Murphy from Microsoft. The show focuses on Microsoft 365 compliance, data security, and governance. Our episodes cover: Microsoft Purview features and updates Practical guidance for improving compliance posture Real-world scenarios and expert discussions Recent episodes include: Mastering Records Management in Microsoft Purview: A Practical Guide for AI-Ready Governance Teams Private Channel Messages: Compliance Action Required by 20 Sept 2025 Microsoft Purview DLP: Best Practices for Successful Implementation Shadow AI, Culture Change, and Compliance: Securing the Future with Rafah Knight 📺 Watch on YouTube: All Things M365 Compliance - YouTube 🎧 Listen on your favourite podcast platform: All Things M365 Compliance | Podcast on Spotify If you’re responsible for compliance, governance, or security in Microsoft 365, this is for you. 👉 Subscribe to stay up to date – and let us know in the comments what topics you’d like us to cover in future episodes!
