Custom SITs fine tunned in MIP
Hello Everyone, Currently working on MS Purview Solutions greenfield deployment project for one of the customer for on-premise data and M365 data. I have created few custom SITs classifiers with regex pattern in the MIP portal almost 3 months ago and it's classifying the data as expected but with some false positives. All of them are fine-tuned to prevent false positives. It's scanning and classifying the newly created M365 data as expected. However it's not reclassifying the previously classified false positive data. How can I forcefully rescan/reclassify the false positive M365 data. I just want to reclassify the data with fine-tuned custom SITs to correctly classify the data before labelling. One more question related to on-prem scanners. I have started the on-prem scanners to scan all the SharePoint sites and Fileshares for any sensitive information. Initially its ran full scan and later it's started as incremental scan. Above scan started before creating the custom SITs and labels. Now I want run a full scan just to classify the data with recommending the labels based on the sensitive data instead of enforcing and applying the label. Can someone throw some light which options need to be select for just recommending the label instead of applying the label. Current configuration as shown below: Any help really appreciated. Regards Anand Sunka
eDiscovery for email attachment with encrypted sensitivity labels
We are currently testing encrypted sensitivity labels in conjunction with eDiscovery. We applied an encrypted label to a document, and eDiscovery was able to successfully search for the content in both OneDrive and SharePoint. However, the same functionality does not appear to work for email attachments—the content of encrypted attachments is not searchable. Are there any specific settings or configurations that need to be enabled to support encrypted email attachments in eDiscovery? Thanks
Default Label and Justification Suddenly Stopped Working
Hi, Sometime last week, default labels for documents suddenly stopped working, it still works for emails. Also, there is a configuration where users have to provide a justification to lower a sensitivity label, that stopped working as well. This has all been in place since May and have always worked but just suddenly stopped working last week. I created a new label with the exact configuration to test, but that works perfectly. I have tried recreating the labels that do not work anymore, but nothing changed. Has anyone experienced this and how did you go about it. Thanks, AishatPurview YouTube Show and Podcast
I am a Microsoft MVP who co-hosts All Things M365 Compliance with Ryan John Murphy from Microsoft. The show focuses on Microsoft 365 compliance, data security, and governance. Our episodes cover: Microsoft Purview features and updates Practical guidance for improving compliance posture Real-world scenarios and expert discussions Recent episodes include: Mastering Records Management in Microsoft Purview: A Practical Guide for AI-Ready Governance Teams Private Channel Messages: Compliance Action Required by 20 Sept 2025 Microsoft Purview DLP: Best Practices for Successful Implementation Shadow AI, Culture Change, and Compliance: Securing the Future with Rafah Knight 📺 Watch on YouTube: All Things M365 Compliance - YouTube 🎧 Listen on your favourite podcast platform: All Things M365 Compliance | Podcast on Spotify If you’re responsible for compliance, governance, or security in Microsoft 365, this is for you. 👉 Subscribe to stay up to date – and let us know in the comments what topics you’d like us to cover in future episodes!New blog post: Is Your Data Ready for Microsoft 365 Copilot?
Is Your Data Ready for Microsoft 365 Copilot? Microsoft 365 Copilot is a game-changer for productivity, but here’s the catch: Copilot surfaces what users already have access to. If your governance isn’t in order, sensitive data could be exposed. In my latest blog, I share: ✅ How to prevent oversharing in Teams & SharePoint ✅ Why sensitivity labels are critical for Copilot ✅ How to monitor usage and avoid shadow AI ✅ Why you don’t need perfect governance to start 📖 Read the full blog: Microsoft 365 Copilot Data Readiness Checklist 👉 What’s your biggest challenge with Copilot readiness? Drop your thoughts below!
Firewall/Proxy allow-list for onboarding devices and Microsoft Purview
We are enabling Microsoft Purview in our Microsoft 365 tenant and onboarding clients. We need to finalize our network firewall/proxy allow-list. Please hlep us confirm the required ports, IP ranges, service tags, and FQDNs for client(onboarding device)-to-Microsoft Purview Service communication?
Managing Multi-Tenant Azure/365: Workarounds for Cross-Tenant Limitations in Purview and Fabric
I am working in a Microsoft Azure/365 multi-tenant setting due to some constraints. I am using Purview (Tenant1) and Fabric (Tenant2), M365 in (Tenant 2). I'm facing issues with various solutions due to cross tenant limitation for eg: Data Quality Connection, Metadata ingestion, lineage, etc. To overcome this I am exploring various workarounds. Key Question: 1. Are there proven workarounds or solutions to manage data estate in this scenario? (Can't merge /migrate tenants)Java MIP SDK 1.17.154: commitAsync() TemplateNotFoundError (C# OK; Java fails Win & Ubuntu)
TL;DR Java SDK 1.17.154: calling setLabel() then commitAsync() fails with TemplateNotFoundError (TemplateId=2ea3c830-...). Same label/code works on Java 1.16.x and C# 1.17.154. Policy cache cleared, templates/labels verified, token/tenant checked—issue persists. Environment SDK (Java): 1.16.x (OK), 1.17.154 (FAIL) SDK (C#): 1.17.154 (OK) OS (Java): Windows 10/11 (win32 build), Ubuntu 20.04 / 22.04 / 24.04 Java: Corretto/OpenJDK 17.0.16 (x64) Service/Tenant: Microsoft Purview Information Protection Auth: (e.g., user delegated token / app-only token) Code Snippet (Java) // Label apply options LabelingOptions labelingOptions = new LabelingOptions(); labelingOptions.setAssignmentMethod(AssignmentMethod.PRIVILEGED); labelingOptions.setDowngradeJustified(true); labelingOptions.setJustificationMessage("Label Apply"); // Get label Label label = fileEngine.getLabelById(labelId); // Apply label (no explicit template handling) fileHandler.setLabel(label, labelingOptions, new ProtectionSettings()); // Commit File workFile = new File(domainFolder, UidUtil.makeUid()); CompletableFuture<Boolean> commitFuture = fileHandler.commitAsync(workFile.getAbsolutePath()); commitFuture.get(); // <-- Throws TemplateNotFoundError on 1.17.154 Stack trace excerpt: Caused by: com.microsoft.informationprotection.internal.gen.Error: TemplateNotFoundError: Could not find template with id: 2ea3c830-5a0e-4eea-b48b-c72186d453c0, BadInputError.Code=General, CorrelationId=42ffaad4-3a0f-4986-ba9d-b5a79c5fd076 (ProtectionEngine), CorrelationId=16819f70-e419-473f-9895-c756f3dd5e4b (FileHandler) at com.microsoft.informationprotection.internal.gen.SdkWrapperJNI.SwigDirector_FileHandler_Observer_OnCommitFailure(SdkWrapperJNI.java:2688) Expected Behavior setLabel() should apply the label (and its protection) and commit successfully, as it does in Java 1.16 and C# 1.17.154. Actual Behavior commitAsync() fails with TemplateNotFoundError for the GUID referenced by the label’s ApplyProtectionAction. What I’ve Tried Policy/cache refresh: Deleted %LOCALAPPDATA%\Microsoft\MSIP\ / ~/.mip/, reloaded engine. Template/label verification: Confirmed existence and publish scope in Purview portal & via PowerShell/Graph. Label actions check: policyEngine.getLabelActions(labelId) shows an ApplyProtectionAction with that GUID. Token/tenant sanity check: Correct scopes and same tenant. Rollback test: Java 1.16 works; C# 1.17.154 works. Questions Any breaking change in Java 1.17 regarding how protection templates are resolved during setLabel()? Is this a known issue specific to Java SDK 1.17.154 (win32 & Ubuntu 20/22/24 builds)? Should we now explicitly use ProtectionDescriptor / SetProtection() in Java? Can someone review the service logs using the CorrelationIds above? Happy to provide additional logs, PowerShell/Graph queries, or action dumps if needed. Thanks!Column-Level Lineage Visualization Issue for Custom Entities and Processes in Azure Purview
I’m trying to implement column-level lineage for data assets and custom transformation processes in Azure Purview using the Atlas API. I have defined custom typedefs for tables (yellowbrick_table), columns (column), and a process type (custom_data_transformation_process), and I’m uploading entities with composition relationships and detailed column mappings. Although the generated JSON and typedefs appear to be correct according to the Apache Atlas documentation, I’m unable to get the column-level lineage to display properly in the Purview UI Specific Issues: 1. Missing 'Schema' tab in custom table entities (yellowbrick_table): When navigating to the detail page of a yellowbrick_table entity, the 'Schema' tab — where the contained columns should be listed — does not appear. 2. Missing 'Switch to column lineage' button in the lineage view of custom processes: In the 'Lineage' tab of a custom_data_transformation_process entity, the side panel titled 'Columns' displays "No mapped columns found," and there is no button or option to switch to column-level lineage view. This happens even when the columnMapping attribute is correctly populated in the process entity. 3. Error message when trying to edit column mappings in the process lineage panel: If I attempt to edit the column mapping in the lineage side panel of the process, I receive the error: "Unable to map columns for this asset. It's a process type asset that doesn't have a schema." (This is expected, as processes don’t have schemas, but it confirms that the UI is not interpreting the columnMapping for visualization purposes.) Context and Steps Taken: I have followed the Apache Atlas documentation and modeling patterns for lineage: Typedef column: Defined with superTypes: ["Referenceable"]. RelationshipDef table_columns: Defined as a COMPOSITION between DataSet (extended by yellowbrick_table) and column, with cardinality: SET on the column side. Typedef yellowbrick_table: Contains an attribute columns with typeName: "array<column>" and relationshipTypeName: "table_columns". Typedef custom_data_transformation_process: Extends from Process and includes a columnMapping attribute of typeName: "array<string>". Entities Uploaded (JSON): * Table entities include complete definitions of their nested columns in the columns attribute. * Process entities include the columnMapping attribute as a list of JSON strings, where each string represents a DatasetMapping with a nested ColumnMapping that uses only the column names (e.g., "Source": "COLUMN_NAME"). * I’ve tested with different browsers Despite these efforts, the issue persists. I would like to know if there are any additional requirements or known behaviors in the Purview UI regarding column lineage visualization for custom types. Specific Questions: 1. Is there any additional attribute or configuration required in the typedefs or entities to make the 'Schema' tab appear in my custom table entities? 2. Are there any specific requirements for the qualifiedName of tables or columns that could be preventing the column-level lineage from being visualized? 3. Could there be a known issue or limitation in the Purview UI regarding column-level lineage rendering for user-defined asset types? 4. Is there any way to verify on the Purview backend that the column composition relationships and the columnMapping of processes have been correctly indexed?" Annexes: # 1. Define the 'column' type # Ensure the superType for 'column' is 'Referenceable' typedef_payload_column = { "entityDefs": [{ "category": "ENTITY", "name": "column", "description": "Columna lógica para linaje columna a columna", "typeVersion": "1.0", "superTypes": ["Referenceable"], "attributeDefs": [] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_column) # print(f"Estado typedef column: {response.status_code} ({response.text.strip()[:100]}...)") # 2. Define the explicit relationship 'table_columns' # Ensure that the cardinality of 'endDef2' (columns) is 'SET' typedef_payload_table_columns_relationship = { "relationshipDefs": [ { "category": "RELATIONSHIP", "name": "table_columns", "description": "Relación entre una tabla y sus columnas", "typeVersion": "1.0", "superTypes": ["AtlasRelationship"], "endDef1": { "type": "DataSet", "name": "parentTable", "isContainer": True, "cardinality": "SINGLE", "isLegacyAttribute": False }, "endDef2": { "type": "column", "name": "columns", "isContainer": False, "cardinality": "SET", "isLegacyAttribute": False }, "relationshipCategory": "COMPOSITION", "attributeDefs": [] } ] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_table_columns_relationship) # print(f"Estado relationshipDef table_columns: {response.status_code} ({response.text.strip()[:100]}...)") # 3. Modify the table's typedef to use this relationship # Ensure the 'columns' attribute on the table points to 'table_columns' typedef_payload_yellowbrick_table = { "entityDefs": [{ "category": "ENTITY", "name": "yellowbrick_table", "description": "Tabla en Yellowbrick", "typeVersion": "1.0", "superTypes": ["DataSet"], "attributeDefs": [ { "name": "columns", "typeName": "array<column>", "isOptional": True, "cardinality": "LIST", "valuesMinCount": 0, "valuesMaxCount": -1, "isUnique": False, "isIndexable": False, "includeInNotification": True, "relationshipTypeName": "table_columns" } ] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_yellowbrick_table) # print(f"Estado typedef yellowbrick_table: {response.status_code} ({response.text.strip()[:100]}...)") # 4. Define the custom process type # Ensure the 'columnMapping' attribute is a string array typedef_payload_process = { "entityDefs": [{ "category": "ENTITY", "name": "custom_data_transformation_process", "description": "Proceso de transformación de datos con linaje de columna (Custom)", "typeVersion": "1.0", "superTypes": ["Process"], "attributeDefs": [ { "name": "columnMapping", "typeName": "array<string>", "isOptional": True, "cardinality": "LIST", "valuesMinCount": 0, "valuesMaxCount": -1 } ] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_process) # print(f"Estado typedef custom_data_transformation_process: {response.status_code} ({response.text.strip()[:100]}...)") example of generated json: [ { "typeName": "yellowbrick_table", "guid": "-105", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn", "name": "TBL8_CONOCIMIENTO_CLIENTE", "description": "Tabla origen: TBL8_CONOCIMIENTO_CLIENTE", "columns": [ { "typeName": "column", "guid": "-336", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#CUENTA", "name": "CUENTA", "description": "Columna CUENTA de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-338", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#TIPO_DOCUMENTO", "name": "TIPO_DOCUMENTO", "description": "Columna TIPO_DOCUMENTO de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-340", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#IDENTIFICACION", "name": "IDENTIFICACION", "description": "Columna IDENTIFICACION de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-342", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#NOMBRE_1", "name": "NOMBRE_1", "description": "Columna NOMBRE_1 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-344", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#APELLIDO_1", "name": "APELLIDO_1", "description": "Columna APELLIDO_1 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-346", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#APELLIDO_2", "name": "APELLIDO_2", "description": "Columna APELLIDO_2 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-348", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#GENERO", "name": "GENERO", "description": "Columna GENERO de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-349", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#EDAD", "name": "EDAD", "description": "Columna EDAD de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-350", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#GENERACION", "name": "GENERACION", "description": "Columna GENERACION de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } } ] } }, { "typeName": "custom_data_transformation_process", "guid": "-107", "attributes": { "qualifiedName": "linaje_process_from_tbl8_conocimiento_cliente_to_tbl_tmp_clien_conocimiento_cliente_c@yellowbrick_conn", "name": "linaje_tbl8_conocimiento_cliente_to_tbl_tmp_clien_conocimiento_cliente_c", "description": "Proceso que conecta TBL8_CONOCIMIENTO_CLIENTE a TBL_TMP_CLIEN_CONOCIMIENTO_CLIENTE_C", "inputs": [ { "guid": "-105" } ], "outputs": [ { "guid": "-106" } ], "columnMapping": [ "{\"DatasetMapping\": {\"Source\": \"DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn\", \"Sink\": \"DB_DWH_STAG.CLIENTES.TBL_TMP_CLIEN_CONOCIMIENTO_CLIENTE_C@yellowbrick_conn\"}, \"ColumnMapping\": [{\"Source\": \"CUENTA\", \"Sink\": \"CUENTA\"}, {\"Source\": \"TIPO_DOCUMENTO\", \"Sink\": \"TIPO_DOCUMENTO\"}, {\"Source\": \"IDENTIFICACION\", \"Sink\": \"IDENTIFICACION\"}, {\"Source\": \"NOMBRE_1\", \"Sink\": \"NOMBRE_1\"}, {\"Source\": \"APELLIDO_1\", \"Sink\": \"APELLIDO_1\"}, {\"Source\": \"APELLIDO_2\", \"Sink\": \"APELLIDO_2\"}, {\"Source\": \"GENERO\", \"Sink\": \"GENERO\"}, {\"Source\": \"EDAD\", \"Sink\": \"EDAD\"}, {\"Source\": \"GENERACION\", \"Sink\": \"GENERACION\"}]}" ] } } ]
