Column-Level Lineage Visualization Issue for Custom Entities and Processes in Azure Purview
I’m trying to implement column-level lineage for data assets and custom transformation processes in Azure Purview using the Atlas API. I have defined custom typedefs for tables (yellowbrick_table), columns (column), and a process type (custom_data_transformation_process), and I’m uploading entities with composition relationships and detailed column mappings. Although the generated JSON and typedefs appear to be correct according to the Apache Atlas documentation, I’m unable to get the column-level lineage to display properly in the Purview UI Specific Issues: 1. Missing 'Schema' tab in custom table entities (yellowbrick_table): When navigating to the detail page of a yellowbrick_table entity, the 'Schema' tab — where the contained columns should be listed — does not appear. 2. Missing 'Switch to column lineage' button in the lineage view of custom processes: In the 'Lineage' tab of a custom_data_transformation_process entity, the side panel titled 'Columns' displays "No mapped columns found," and there is no button or option to switch to column-level lineage view. This happens even when the columnMapping attribute is correctly populated in the process entity. 3. Error message when trying to edit column mappings in the process lineage panel: If I attempt to edit the column mapping in the lineage side panel of the process, I receive the error: "Unable to map columns for this asset. It's a process type asset that doesn't have a schema." (This is expected, as processes don’t have schemas, but it confirms that the UI is not interpreting the columnMapping for visualization purposes.) Context and Steps Taken: I have followed the Apache Atlas documentation and modeling patterns for lineage: Typedef column: Defined with superTypes: ["Referenceable"]. RelationshipDef table_columns: Defined as a COMPOSITION between DataSet (extended by yellowbrick_table) and column, with cardinality: SET on the column side. Typedef yellowbrick_table: Contains an attribute columns with typeName: "array<column>" and relationshipTypeName: "table_columns". Typedef custom_data_transformation_process: Extends from Process and includes a columnMapping attribute of typeName: "array<string>". Entities Uploaded (JSON): * Table entities include complete definitions of their nested columns in the columns attribute. * Process entities include the columnMapping attribute as a list of JSON strings, where each string represents a DatasetMapping with a nested ColumnMapping that uses only the column names (e.g., "Source": "COLUMN_NAME"). * I’ve tested with different browsers Despite these efforts, the issue persists. I would like to know if there are any additional requirements or known behaviors in the Purview UI regarding column lineage visualization for custom types. Specific Questions: 1. Is there any additional attribute or configuration required in the typedefs or entities to make the 'Schema' tab appear in my custom table entities? 2. Are there any specific requirements for the qualifiedName of tables or columns that could be preventing the column-level lineage from being visualized? 3. Could there be a known issue or limitation in the Purview UI regarding column-level lineage rendering for user-defined asset types? 4. Is there any way to verify on the Purview backend that the column composition relationships and the columnMapping of processes have been correctly indexed?" Annexes: # 1. Define the 'column' type # Ensure the superType for 'column' is 'Referenceable' typedef_payload_column = { "entityDefs": [{ "category": "ENTITY", "name": "column", "description": "Columna lógica para linaje columna a columna", "typeVersion": "1.0", "superTypes": ["Referenceable"], "attributeDefs": [] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_column) # print(f"Estado typedef column: {response.status_code} ({response.text.strip()[:100]}...)") # 2. Define the explicit relationship 'table_columns' # Ensure that the cardinality of 'endDef2' (columns) is 'SET' typedef_payload_table_columns_relationship = { "relationshipDefs": [ { "category": "RELATIONSHIP", "name": "table_columns", "description": "Relación entre una tabla y sus columnas", "typeVersion": "1.0", "superTypes": ["AtlasRelationship"], "endDef1": { "type": "DataSet", "name": "parentTable", "isContainer": True, "cardinality": "SINGLE", "isLegacyAttribute": False }, "endDef2": { "type": "column", "name": "columns", "isContainer": False, "cardinality": "SET", "isLegacyAttribute": False }, "relationshipCategory": "COMPOSITION", "attributeDefs": [] } ] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_table_columns_relationship) # print(f"Estado relationshipDef table_columns: {response.status_code} ({response.text.strip()[:100]}...)") # 3. Modify the table's typedef to use this relationship # Ensure the 'columns' attribute on the table points to 'table_columns' typedef_payload_yellowbrick_table = { "entityDefs": [{ "category": "ENTITY", "name": "yellowbrick_table", "description": "Tabla en Yellowbrick", "typeVersion": "1.0", "superTypes": ["DataSet"], "attributeDefs": [ { "name": "columns", "typeName": "array<column>", "isOptional": True, "cardinality": "LIST", "valuesMinCount": 0, "valuesMaxCount": -1, "isUnique": False, "isIndexable": False, "includeInNotification": True, "relationshipTypeName": "table_columns" } ] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_yellowbrick_table) # print(f"Estado typedef yellowbrick_table: {response.status_code} ({response.text.strip()[:100]}...)") # 4. Define the custom process type # Ensure the 'columnMapping' attribute is a string array typedef_payload_process = { "entityDefs": [{ "category": "ENTITY", "name": "custom_data_transformation_process", "description": "Proceso de transformación de datos con linaje de columna (Custom)", "typeVersion": "1.0", "superTypes": ["Process"], "attributeDefs": [ { "name": "columnMapping", "typeName": "array<string>", "isOptional": True, "cardinality": "LIST", "valuesMinCount": 0, "valuesMaxCount": -1 } ] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_process) # print(f"Estado typedef custom_data_transformation_process: {response.status_code} ({response.text.strip()[:100]}...)") example of generated json: [ { "typeName": "yellowbrick_table", "guid": "-105", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn", "name": "TBL8_CONOCIMIENTO_CLIENTE", "description": "Tabla origen: TBL8_CONOCIMIENTO_CLIENTE", "columns": [ { "typeName": "column", "guid": "-336", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#CUENTA", "name": "CUENTA", "description": "Columna CUENTA de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-338", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#TIPO_DOCUMENTO", "name": "TIPO_DOCUMENTO", "description": "Columna TIPO_DOCUMENTO de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-340", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#IDENTIFICACION", "name": "IDENTIFICACION", "description": "Columna IDENTIFICACION de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-342", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#NOMBRE_1", "name": "NOMBRE_1", "description": "Columna NOMBRE_1 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-344", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#APELLIDO_1", "name": "APELLIDO_1", "description": "Columna APELLIDO_1 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-346", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#APELLIDO_2", "name": "APELLIDO_2", "description": "Columna APELLIDO_2 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-348", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#GENERO", "name": "GENERO", "description": "Columna GENERO de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-349", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#EDAD", "name": "EDAD", "description": "Columna EDAD de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-350", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#GENERACION", "name": "GENERACION", "description": "Columna GENERACION de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } } ] } }, { "typeName": "custom_data_transformation_process", "guid": "-107", "attributes": { "qualifiedName": "linaje_process_from_tbl8_conocimiento_cliente_to_tbl_tmp_clien_conocimiento_cliente_c@yellowbrick_conn", "name": "linaje_tbl8_conocimiento_cliente_to_tbl_tmp_clien_conocimiento_cliente_c", "description": "Proceso que conecta TBL8_CONOCIMIENTO_CLIENTE a TBL_TMP_CLIEN_CONOCIMIENTO_CLIENTE_C", "inputs": [ { "guid": "-105" } ], "outputs": [ { "guid": "-106" } ], "columnMapping": [ "{\"DatasetMapping\": {\"Source\": \"DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn\", \"Sink\": \"DB_DWH_STAG.CLIENTES.TBL_TMP_CLIEN_CONOCIMIENTO_CLIENTE_C@yellowbrick_conn\"}, \"ColumnMapping\": [{\"Source\": \"CUENTA\", \"Sink\": \"CUENTA\"}, {\"Source\": \"TIPO_DOCUMENTO\", \"Sink\": \"TIPO_DOCUMENTO\"}, {\"Source\": \"IDENTIFICACION\", \"Sink\": \"IDENTIFICACION\"}, {\"Source\": \"NOMBRE_1\", \"Sink\": \"NOMBRE_1\"}, {\"Source\": \"APELLIDO_1\", \"Sink\": \"APELLIDO_1\"}, {\"Source\": \"APELLIDO_2\", \"Sink\": \"APELLIDO_2\"}, {\"Source\": \"GENERO\", \"Sink\": \"GENERO\"}, {\"Source\": \"EDAD\", \"Sink\": \"EDAD\"}, {\"Source\": \"GENERACION\", \"Sink\": \"GENERACION\"}]}" ] } } ]
Java MIP SDK 1.17.154: commitAsync() TemplateNotFoundError (C# OK; Java fails Win & Ubuntu)
TL;DR Java SDK 1.17.154: calling setLabel() then commitAsync() fails with TemplateNotFoundError (TemplateId=2ea3c830-...). Same label/code works on Java 1.16.x and C# 1.17.154. Policy cache cleared, templates/labels verified, token/tenant checked—issue persists. Environment SDK (Java): 1.16.x (OK), 1.17.154 (FAIL) SDK (C#): 1.17.154 (OK) OS (Java): Windows 10/11 (win32 build), Ubuntu 20.04 / 22.04 / 24.04 Java: OpenJDK 17.0.16 (x64) Service/Tenant: Microsoft Purview Information Protection Auth: (e.g., user delegated token / app-only token) Code Snippet (Java) // Label apply options LabelingOptions labelingOptions = new LabelingOptions(); labelingOptions.setAssignmentMethod(AssignmentMethod.PRIVILEGED); labelingOptions.setDowngradeJustified(true); labelingOptions.setJustificationMessage("Label Apply"); // Get label Label label = fileEngine.getLabelById(labelId); // Apply label (no explicit template handling) fileHandler.setLabel(label, labelingOptions, new ProtectionSettings()); // Commit File workFile = new File(domainFolder, UidUtil.makeUid()); CompletableFuture<Boolean> commitFuture = fileHandler.commitAsync(workFile.getAbsolutePath()); commitFuture.get(); // <-- Throws TemplateNotFoundError on 1.17.154 Stack trace excerpt: Caused by: com.microsoft.informationprotection.internal.gen.Error: TemplateNotFoundError: Could not find template with id: 2ea3c830-5a0e-4eea-b48b-c72186d453c0, BadInputError.Code=General, CorrelationId=42ffaad4-3a0f-4986-ba9d-b5a79c5fd076 (ProtectionEngine), CorrelationId=16819f70-e419-473f-9895-c756f3dd5e4b (FileHandler) at com.microsoft.informationprotection.internal.gen.SdkWrapperJNI.SwigDirector_FileHandler_Observer_OnCommitFailure(SdkWrapperJNI.java:2688) Expected Behavior setLabel() should apply the label (and its protection) and commit successfully, as it does in Java 1.16 and C# 1.17.154. Actual Behavior commitAsync() fails with TemplateNotFoundError for the GUID referenced by the label’s ApplyProtectionAction. What I’ve Tried Policy/cache refresh: Deleted %LOCALAPPDATA%\Microsoft\MSIP\ / ~/.mip/, reloaded engine. Template/label verification: Confirmed existence and publish scope in Purview portal & via PowerShell/Graph. Label actions check: policyEngine.getLabelActions(labelId) shows an ApplyProtectionAction with that GUID. Token/tenant sanity check: Correct scopes and same tenant. Rollback test: Java 1.16 works; C# 1.17.154 works. Questions Any breaking change in Java 1.17 regarding how protection templates are resolved during setLabel()? Is this a known issue specific to Java SDK 1.17.154 (win32 & Ubuntu 20/22/24 builds)? Should we now explicitly use ProtectionDescriptor / SetProtection() in Java? Can someone review the service logs using the CorrelationIds above? Happy to provide additional logs, PowerShell/Graph queries, or action dumps if needed. Thanks!
Purview DLP Override Email Notification for Admins
We know that our admins can go into the alerts in Purview and see if a user has chosen to override a DLP policy, but is there a way to set up email notification to individuals when a user clicks Override in the policy tip and chooses to override the block? So, in the attached "Override window.png" file, once they click the Override button, security admins are alerted via email immediately?
Auto-Apply Sensitivity Label
I am working on DLP and Information Protection feature of Purview to utilize the Sensitive Info Types. In DLP, I made a policy to send alerts when the SITs are found. In Information Protection, I have sensitivity label and configured to auto apply it once it detects the SIT. I use the same SIT on alert and in sensitivity label. Upon doing tests on several documents, I am receiving alerts via email because the SIT is found. However, the automatic application of SIT on the document does not persist, even it's been 2 weeks since I have created the document. Other information: *Upon publishing other labels, i did not configure any default labels on documents. *Tenant is US *Im working in the Philippines Here is the email i am receiving coming from the DLP policy. Here is my configuration of the sensitivity label.Java MIP SDK 1.17.154: commitAsync() TemplateNotFoundError (C# OK; Java fails Win & Ubuntu)
TL;DR Java SDK 1.17.154: calling setLabel() then commitAsync() fails with TemplateNotFoundError (TemplateId=2ea3c830-...). Same label/code works on Java 1.16.x and C# 1.17.154. Policy cache cleared, templates/labels verified, token/tenant checked—issue persists. Environment SDK (Java): 1.16.x (OK), 1.17.154 (FAIL) SDK (C#): 1.17.154 (OK) OS (Java): Windows 10/11 (win32 build), Ubuntu 20.04 / 22.04 / 24.04 Java: Corretto/OpenJDK 17.0.16 (x64) Service/Tenant: Microsoft Purview Information Protection Auth: (e.g., user delegated token / app-only token) Code Snippet (Java) // Label apply options LabelingOptions labelingOptions = new LabelingOptions(); labelingOptions.setAssignmentMethod(AssignmentMethod.PRIVILEGED); labelingOptions.setDowngradeJustified(true); labelingOptions.setJustificationMessage("Label Apply"); // Get label Label label = fileEngine.getLabelById(labelId); // Apply label (no explicit template handling) fileHandler.setLabel(label, labelingOptions, new ProtectionSettings()); // Commit File workFile = new File(domainFolder, UidUtil.makeUid()); CompletableFuture<Boolean> commitFuture = fileHandler.commitAsync(workFile.getAbsolutePath()); commitFuture.get(); // <-- Throws TemplateNotFoundError on 1.17.154 Stack trace excerpt: Caused by: com.microsoft.informationprotection.internal.gen.Error: TemplateNotFoundError: Could not find template with id: 2ea3c830-5a0e-4eea-b48b-c72186d453c0, BadInputError.Code=General, CorrelationId=42ffaad4-3a0f-4986-ba9d-b5a79c5fd076 (ProtectionEngine), CorrelationId=16819f70-e419-473f-9895-c756f3dd5e4b (FileHandler) at com.microsoft.informationprotection.internal.gen.SdkWrapperJNI.SwigDirector_FileHandler_Observer_OnCommitFailure(SdkWrapperJNI.java:2688) Expected Behavior setLabel() should apply the label (and its protection) and commit successfully, as it does in Java 1.16 and C# 1.17.154. Actual Behavior commitAsync() fails with TemplateNotFoundError for the GUID referenced by the label’s ApplyProtectionAction. What I’ve Tried Policy/cache refresh: Deleted %LOCALAPPDATA%\Microsoft\MSIP\ / ~/.mip/, reloaded engine. Template/label verification: Confirmed existence and publish scope in Purview portal & via PowerShell/Graph. Label actions check: policyEngine.getLabelActions(labelId) shows an ApplyProtectionAction with that GUID. Token/tenant sanity check: Correct scopes and same tenant. Rollback test: Java 1.16 works; C# 1.17.154 works. Questions Any breaking change in Java 1.17 regarding how protection templates are resolved during setLabel()? Is this a known issue specific to Java SDK 1.17.154 (win32 & Ubuntu 20/22/24 builds)? Should we now explicitly use ProtectionDescriptor / SetProtection() in Java? Can someone review the service logs using the CorrelationIds above? Happy to provide additional logs, PowerShell/Graph queries, or action dumps if needed. Thanks!
Client Side Auto Labelling Not Working
Hi All, I have configured some rules for client-side auto labelling, sometimes it works, sometimes it does not. It is supposed to upgrade the documents to Protected B. Is anyone else experiencing this or should i be making changes to the configurations? Please see attached.Connect to ERWIN Data Mart SaaS instance
Hello All Good morning As a part of our end-to-end data governance implementation, we are trying to setup Purview connectivity to ERWIN data mart which is setup as a SaaS instance. I had a look at documentation here and it is all specifying On Premise ERWIN implementation and setup of self hosted integration runtime etc. Can anyone please guide on how to proceed in cases where ERWIN mart is on cloud? Is this currently supported with the available ERWIN connector? Thanks in advance for any leads you may have. Regards VisakhSharing: PDF readers that support Purview labels
As I was researching on Adobe Acrobat reader and Sensitivity labels, I decided to check if the common alternative PDF readers out there are able to support Purview MIP Sensitivity labels. There is already a published documentation on this for SharePoint-Compatible PDF readers that supports Microsoft IRM: https://learn.microsoft.com/en-us/purview/sp-compatible-pdf-readers-for-irm (last updated Nov-2023) but I wanted to see if these same PDF readers supports the ability for end-users to use/ select labels similar to that of Adobe Acrobat As of 11-June-2025; atleast one of them clearly do: Nitro PDF: Yes. Documentation shows that users can see and use the sensitivity labels. PDF -X.change Editor: Yes. Documentation show that users can see and use the sensitivity labels. (check the official website, I can't hyperlink it because the site is blocked. FOX PDF editor: No. Documentation only states RMS and not clear if it show Purview labels. This is for F.O.X.I.T editor (spelled without the ".") but for some reason there is a community ban on that word and it won't allow me to post the full name PDFescape: No. Sumatra PDF: No Okular: No If there are other PDF readers that I've missed, I encourage you list it down in the comment below. Would love to grow this list.Microsoft Purview - Establishing Data Quality Connection on Azure SQL Database
I have a service principal and already using it on the Data Map Solutions to scan and register Azure SQL Database as the source. It worked. Now, I am in the Unified Catalog and I am trying to establish a data quality connection on Microsoft purview for Azure SQL Database as its source type to run data quality scans. Why can't I use the Service Principal for the credential? Is there another way to establish the connection?
