C# MIP SDK v1.17.x - AccessViolationException on creation of MIPContext in 64-bit console app
I first logged this on https://stackoverflow.com/questions/79746967/accessviolationexception-when-creating-mipcontext-after-upgrade-to-v1-17 and the responses there have indicated I should raise with Microsoft a a likely bug, but I don't see a clear route to reporting other than here so any response would be appreciated, even if just to direct me to the appropriate reporting location. I've built a simple console app that demonstrates this issue that I'm happy to provide but we're seeing an issue with the 1.17.x version of the C# MIP SDK where an AccessViolationException is being thrown when trying to create an MIP context object. This is for a .Net Framework 4.8 console app built in 64-bit configuration, deployed to a Windows Server 2016 with the latest VC++ redistributable (14.44.35211) installed (both x86 and x64 versions), though we've seen the same on Windows Server 2019 and 2022. When the same app is built in 32-bit and deployed to the same environment the exception doesn't occur. The following code is what I've used to repro the issue: MIP.Initialize(MipComponent.File); var appInfo = new ApplicationInfo { ApplicationId = string.Empty, ApplicationName = string.Empty, ApplicationVersion = string.Empty }; var diagnosticConfiguration = new DiagnosticConfiguration { IsMinimalTelemetryEnabled = true }; var mipConfiguration = new MipConfiguration(appInfo, "mip_data", LogLevel.Info, false, CacheStorageType.InMemory) { DiagnosticOverride = diagnosticConfiguration }; //Expect BadInputException here due to empty properties of appInfo //When built as part of a 64-bit console app this causes AccessViolationException instead MIP.CreateMipContext(mipConfiguration); The AccessViolationException crashes the console app, with the following logged in the Windows Event Log: Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.AccessViolationException at Microsoft.InformationProtection.Internal.SdkWrapperPINVOKE.MipContext_Create__SWIG_1(System.Runtime.InteropServices.HandleRef) at Microsoft.InformationProtection.Internal.MipContext.Create(Microsoft.InformationProtection.Internal.MipConfiguration) at Microsoft.InformationProtection.Utils.MIPHelper.CreateMipContext(Microsoft.InformationProtection.MipConfiguration) The issue doesn't occur with the latest 1.16 version (1.16.149) of the SDK but does appear to be in all versions of the 1.17 release. Library: C# MIP SDK v1.17.x Target App: .Net Framework 4.8 console app Deployed OS: Windows Server 2016, 2019 and 2022 (With .Net Framework 4.8 and latest VC++ redist installed)Column-Level Lineage Visualization Issue for Custom Entities and Processes in Azure Purview
I’m trying to implement column-level lineage for data assets and custom transformation processes in Azure Purview using the Atlas API. I have defined custom typedefs for tables (yellowbrick_table), columns (column), and a process type (custom_data_transformation_process), and I’m uploading entities with composition relationships and detailed column mappings. Although the generated JSON and typedefs appear to be correct according to the Apache Atlas documentation, I’m unable to get the column-level lineage to display properly in the Purview UI Specific Issues: 1. Missing 'Schema' tab in custom table entities (yellowbrick_table): When navigating to the detail page of a yellowbrick_table entity, the 'Schema' tab — where the contained columns should be listed — does not appear. 2. Missing 'Switch to column lineage' button in the lineage view of custom processes: In the 'Lineage' tab of a custom_data_transformation_process entity, the side panel titled 'Columns' displays "No mapped columns found," and there is no button or option to switch to column-level lineage view. This happens even when the columnMapping attribute is correctly populated in the process entity. 3. Error message when trying to edit column mappings in the process lineage panel: If I attempt to edit the column mapping in the lineage side panel of the process, I receive the error: "Unable to map columns for this asset. It's a process type asset that doesn't have a schema." (This is expected, as processes don’t have schemas, but it confirms that the UI is not interpreting the columnMapping for visualization purposes.) Context and Steps Taken: I have followed the Apache Atlas documentation and modeling patterns for lineage: Typedef column: Defined with superTypes: ["Referenceable"]. RelationshipDef table_columns: Defined as a COMPOSITION between DataSet (extended by yellowbrick_table) and column, with cardinality: SET on the column side. Typedef yellowbrick_table: Contains an attribute columns with typeName: "array<column>" and relationshipTypeName: "table_columns". Typedef custom_data_transformation_process: Extends from Process and includes a columnMapping attribute of typeName: "array<string>". Entities Uploaded (JSON): * Table entities include complete definitions of their nested columns in the columns attribute. * Process entities include the columnMapping attribute as a list of JSON strings, where each string represents a DatasetMapping with a nested ColumnMapping that uses only the column names (e.g., "Source": "COLUMN_NAME"). * I’ve tested with different browsers Despite these efforts, the issue persists. I would like to know if there are any additional requirements or known behaviors in the Purview UI regarding column lineage visualization for custom types. Specific Questions: 1. Is there any additional attribute or configuration required in the typedefs or entities to make the 'Schema' tab appear in my custom table entities? 2. Are there any specific requirements for the qualifiedName of tables or columns that could be preventing the column-level lineage from being visualized? 3. Could there be a known issue or limitation in the Purview UI regarding column-level lineage rendering for user-defined asset types? 4. Is there any way to verify on the Purview backend that the column composition relationships and the columnMapping of processes have been correctly indexed?" Annexes: # 1. Define the 'column' type # Ensure the superType for 'column' is 'Referenceable' typedef_payload_column = { "entityDefs": [{ "category": "ENTITY", "name": "column", "description": "Columna lógica para linaje columna a columna", "typeVersion": "1.0", "superTypes": ["Referenceable"], "attributeDefs": [] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_column) # print(f"Estado typedef column: {response.status_code} ({response.text.strip()[:100]}...)") # 2. Define the explicit relationship 'table_columns' # Ensure that the cardinality of 'endDef2' (columns) is 'SET' typedef_payload_table_columns_relationship = { "relationshipDefs": [ { "category": "RELATIONSHIP", "name": "table_columns", "description": "Relación entre una tabla y sus columnas", "typeVersion": "1.0", "superTypes": ["AtlasRelationship"], "endDef1": { "type": "DataSet", "name": "parentTable", "isContainer": True, "cardinality": "SINGLE", "isLegacyAttribute": False }, "endDef2": { "type": "column", "name": "columns", "isContainer": False, "cardinality": "SET", "isLegacyAttribute": False }, "relationshipCategory": "COMPOSITION", "attributeDefs": [] } ] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_table_columns_relationship) # print(f"Estado relationshipDef table_columns: {response.status_code} ({response.text.strip()[:100]}...)") # 3. Modify the table's typedef to use this relationship # Ensure the 'columns' attribute on the table points to 'table_columns' typedef_payload_yellowbrick_table = { "entityDefs": [{ "category": "ENTITY", "name": "yellowbrick_table", "description": "Tabla en Yellowbrick", "typeVersion": "1.0", "superTypes": ["DataSet"], "attributeDefs": [ { "name": "columns", "typeName": "array<column>", "isOptional": True, "cardinality": "LIST", "valuesMinCount": 0, "valuesMaxCount": -1, "isUnique": False, "isIndexable": False, "includeInNotification": True, "relationshipTypeName": "table_columns" } ] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_yellowbrick_table) # print(f"Estado typedef yellowbrick_table: {response.status_code} ({response.text.strip()[:100]}...)") # 4. Define the custom process type # Ensure the 'columnMapping' attribute is a string array typedef_payload_process = { "entityDefs": [{ "category": "ENTITY", "name": "custom_data_transformation_process", "description": "Proceso de transformación de datos con linaje de columna (Custom)", "typeVersion": "1.0", "superTypes": ["Process"], "attributeDefs": [ { "name": "columnMapping", "typeName": "array<string>", "isOptional": True, "cardinality": "LIST", "valuesMinCount": 0, "valuesMaxCount": -1 } ] }] } # response = requests.post(typedef_url, headers=headers, json=typedef_payload_process) # print(f"Estado typedef custom_data_transformation_process: {response.status_code} ({response.text.strip()[:100]}...)") example of generated json: [ { "typeName": "yellowbrick_table", "guid": "-105", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn", "name": "TBL8_CONOCIMIENTO_CLIENTE", "description": "Tabla origen: TBL8_CONOCIMIENTO_CLIENTE", "columns": [ { "typeName": "column", "guid": "-336", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#CUENTA", "name": "CUENTA", "description": "Columna CUENTA de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-338", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#TIPO_DOCUMENTO", "name": "TIPO_DOCUMENTO", "description": "Columna TIPO_DOCUMENTO de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-340", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#IDENTIFICACION", "name": "IDENTIFICACION", "description": "Columna IDENTIFICACION de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-342", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#NOMBRE_1", "name": "NOMBRE_1", "description": "Columna NOMBRE_1 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-344", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#APELLIDO_1", "name": "APELLIDO_1", "description": "Columna APELLIDO_1 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-346", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#APELLIDO_2", "name": "APELLIDO_2", "description": "Columna APELLIDO_2 de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-348", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#GENERO", "name": "GENERO", "description": "Columna GENERO de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-349", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#EDAD", "name": "EDAD", "description": "Columna EDAD de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } }, { "typeName": "column", "guid": "-350", "attributes": { "qualifiedName": "DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn#GENERACION", "name": "GENERACION", "description": "Columna GENERACION de tabla tbl8_conocimiento_cliente", "type": "string", "dataType": "string" } } ] } }, { "typeName": "custom_data_transformation_process", "guid": "-107", "attributes": { "qualifiedName": "linaje_process_from_tbl8_conocimiento_cliente_to_tbl_tmp_clien_conocimiento_cliente_c@yellowbrick_conn", "name": "linaje_tbl8_conocimiento_cliente_to_tbl_tmp_clien_conocimiento_cliente_c", "description": "Proceso que conecta TBL8_CONOCIMIENTO_CLIENTE a TBL_TMP_CLIEN_CONOCIMIENTO_CLIENTE_C", "inputs": [ { "guid": "-105" } ], "outputs": [ { "guid": "-106" } ], "columnMapping": [ "{\"DatasetMapping\": {\"Source\": \"DB_DWH_EXTRACCION.HOGARES.TBL8_CONOCIMIENTO_CLIENTE@yellowbrick_conn\", \"Sink\": \"DB_DWH_STAG.CLIENTES.TBL_TMP_CLIEN_CONOCIMIENTO_CLIENTE_C@yellowbrick_conn\"}, \"ColumnMapping\": [{\"Source\": \"CUENTA\", \"Sink\": \"CUENTA\"}, {\"Source\": \"TIPO_DOCUMENTO\", \"Sink\": \"TIPO_DOCUMENTO\"}, {\"Source\": \"IDENTIFICACION\", \"Sink\": \"IDENTIFICACION\"}, {\"Source\": \"NOMBRE_1\", \"Sink\": \"NOMBRE_1\"}, {\"Source\": \"APELLIDO_1\", \"Sink\": \"APELLIDO_1\"}, {\"Source\": \"APELLIDO_2\", \"Sink\": \"APELLIDO_2\"}, {\"Source\": \"GENERO\", \"Sink\": \"GENERO\"}, {\"Source\": \"EDAD\", \"Sink\": \"EDAD\"}, {\"Source\": \"GENERACION\", \"Sink\": \"GENERACION\"}]}" ] } } ]
Java MIP SDK 1.17.154: commitAsync() TemplateNotFoundError (C# OK; Java fails Win & Ubuntu)
TL;DR Java SDK 1.17.154: calling setLabel() then commitAsync() fails with TemplateNotFoundError (TemplateId=2ea3c830-...). Same label/code works on Java 1.16.x and C# 1.17.154. Policy cache cleared, templates/labels verified, token/tenant checked—issue persists. Environment SDK (Java): 1.16.x (OK), 1.17.154 (FAIL) SDK (C#): 1.17.154 (OK) OS (Java): Windows 10/11 (win32 build), Ubuntu 20.04 / 22.04 / 24.04 Java: OpenJDK 17.0.16 (x64) Service/Tenant: Microsoft Purview Information Protection Auth: (e.g., user delegated token / app-only token) Code Snippet (Java) // Label apply options LabelingOptions labelingOptions = new LabelingOptions(); labelingOptions.setAssignmentMethod(AssignmentMethod.PRIVILEGED); labelingOptions.setDowngradeJustified(true); labelingOptions.setJustificationMessage("Label Apply"); // Get label Label label = fileEngine.getLabelById(labelId); // Apply label (no explicit template handling) fileHandler.setLabel(label, labelingOptions, new ProtectionSettings()); // Commit File workFile = new File(domainFolder, UidUtil.makeUid()); CompletableFuture<Boolean> commitFuture = fileHandler.commitAsync(workFile.getAbsolutePath()); commitFuture.get(); // <-- Throws TemplateNotFoundError on 1.17.154 Stack trace excerpt: Caused by: com.microsoft.informationprotection.internal.gen.Error: TemplateNotFoundError: Could not find template with id: 2ea3c830-5a0e-4eea-b48b-c72186d453c0, BadInputError.Code=General, CorrelationId=42ffaad4-3a0f-4986-ba9d-b5a79c5fd076 (ProtectionEngine), CorrelationId=16819f70-e419-473f-9895-c756f3dd5e4b (FileHandler) at com.microsoft.informationprotection.internal.gen.SdkWrapperJNI.SwigDirector_FileHandler_Observer_OnCommitFailure(SdkWrapperJNI.java:2688) Expected Behavior setLabel() should apply the label (and its protection) and commit successfully, as it does in Java 1.16 and C# 1.17.154. Actual Behavior commitAsync() fails with TemplateNotFoundError for the GUID referenced by the label’s ApplyProtectionAction. What I’ve Tried Policy/cache refresh: Deleted %LOCALAPPDATA%\Microsoft\MSIP\ / ~/.mip/, reloaded engine. Template/label verification: Confirmed existence and publish scope in Purview portal & via PowerShell/Graph. Label actions check: policyEngine.getLabelActions(labelId) shows an ApplyProtectionAction with that GUID. Token/tenant sanity check: Correct scopes and same tenant. Rollback test: Java 1.16 works; C# 1.17.154 works. Questions Any breaking change in Java 1.17 regarding how protection templates are resolved during setLabel()? Is this a known issue specific to Java SDK 1.17.154 (win32 & Ubuntu 20/22/24 builds)? Should we now explicitly use ProtectionDescriptor / SetProtection() in Java? Can someone review the service logs using the CorrelationIds above? Happy to provide additional logs, PowerShell/Graph queries, or action dumps if needed. Thanks!
Fabric Warehouse is not extracting metadata like tables and its columns in Purview
Hello Team, Good Evening..! We noticed that the metadata in Datawarehouse is not extracting Schema(Tab) which contains Tables and its columns We scanned Lakehouse using Service prinicple where metadata extraction is done as expected but it is not the same in Data Warehouse after the scan completes. Are there any other options to get that metadata extracted under warehouse ? Please guide us the best possible methods to handle this situation. Regards, Naresh Inaganti
Lineage Limitation on Wide Power BI Semantic Models & Built-in Classification Rule Sets
Hello everyone, I’m evaluating the new Microsoft Purview Governance Portal for our finance data governance needs. Previously with the Azure Purview classic version we had a couple of blocking issues such as it failed to scan wide semantic models. But now we're migrating to Fabric and we'd like to try the new Microsoft Purview Governance Portal. I’d appreciate any insights or confirmation from the product team or the community. Lineage Limitation on Wide Power BI Semantic Models Background: When we first ran the Purview Data Map Scanner against our finance semantic model, it failed once the total column count across all tables exceeded roughly 500 columns. In our case, a single SAP table alone has about 450 columns—so the scan wouldn’t complete, and we couldn’t capture any lineage. Questions: Has Purview removed or raised any “hidden” column-count limits for Power BI semantic models? Is there any official documentation on maximum supported column counts (e.g. 200, 500, or otherwise)? Are there recommended workarounds for very wide models—such as splitting into sub-datasets, using incremental scans, etc. to get full lineage? 2. Built-in Classification Rule Sets Background: Purview ships with a set of Microsoft-provided “Sensitive Information Types” that appear in every scan rule set. In many of our scans these defaults aren’t needed, and they clutter the results. Questions: Can we delete or permanently disable the built-in classification rules? If not, what’s the best way to ensure they’re not applied during a full scan? Are there any APIs or PowerShell commands that let us automate the exclusion of Microsoft’s defaults from our scan rule sets? Thank you in advance for any pointers, documentation links, or best-practice advice!
Challenges adding large numbers of assets to a Data Product in Unified Catalog
The client's needs oblige us to add the tables from a SQL database into the data product; they've also got a use case for a massive file share. We are hitting a wall in finding a programmatic solution to bring these data map assets into the Product, and the portal fails when trying to add more than 100 objects at a time. Looking for a timeline for the release of APIs and tools to address this?Redirecting to Classic Portal Instead of New Portal in Microsoft Purview
I am using the following endpoint in my Python script to create new terms in the Microsoft Purview New Portal: purview_endpoint = "https://api.purview-service.microsoft.com" However, this endpoint redirects me to the Classic Portal instead of the New Portal. Has anyone encountered this issue or knows how to resolve it? Thanks in advance!Purview API: When can a DataSet have many attachedSchema?
In the definition of a DataSet, there is a relationshipDef for attachedSchema that supports more than one schema. In what scenario would there be more than one attached schema for a single data set? I am trying to extract the schema for an azure_datalake_gen2_resource_set based on a delta table. I have been just using the first attachedSchema in the returned list, but am wondering if that is safe to do. Definition of the relationship: { "name": "attachedSchema", "typeName": "array<schema>", "isOptional": true, "cardinality": "SET", "valuesMinCount": -1, "valuesMaxCount": -1, "isUnique": false, "isIndexable": false, "includeInNotification": false, "relationshipTypeName": "dataset_attached_schemas", "isLegacyAttribute": false }` Example of what is returned from the purview API (when could there be more than one entry in the "attachedSchema" list?): { "entity": { "typeName": "azure_datalake_gen2_resource_set", "relationshipAttributes": { "attachedSchema": [ { "guid": "guid", "typeName": "parquet_schema", "entityStatus": "ACTIVE", "displayText": "parquet_schema", "relationshipType": "dataset_attached_schemas", "relationshipGuid": "relid", "relationshipStatus": "ACTIVE", "relationshipAttributes": { "typeName": "dataset_attached_schemas" } } ] } } }
