Data System Wide Lineage via API Request
I'm struggling with finding a solution. My goal is to identify all existing lineage relationships for any data objects within a specific data system they belong to. I've been using the Purview REST API (Datamap Dataplane) but I haven't found an endpoint returning data system side lineage/relationships. For my scenario I have a Databricks metastore and need to know the existing lineage relationships of those data objects within Purview so I can purge them out when we are doing our scheduled lineage refresh.
Cross workspace lineage for Fabric Lakehouse tables in Purview
I’m currently exploring lineage capture in Microsoft Purview for Fabric Lakehouse tables that are spread across multiple workspaces, following medallion architecture (Bronze, Silver, Gold in separate Fabric workspaces). While reviewing the documentation, I noticed the stated limitation around cross-workspace lineage for non-power BI assets, as mentioned here: https://learn.microsoft.com/en-us/purview/data-map-lineage-fabric Is there any update or workaround planned to support cross-workspace Fabric lineage in Purview? Is this limitation on the product roadmap or actively being worked on? Until native support is available, are there any recommended design patterns to handle lineage in this scenario?
Some Fabric Lakehouse tables not appearing in Microsoft Purview after scan
Hi everyone, I’m running into an issue where several tables from a Fabric Lakehouse aren’t appearing in Microsoft Purview after a workspace scan. Here’s the situation: I scanned a Fabric workspace that contains multiple Lakehouses. For most Lakehouses, the tables appear correctly in Purview after the scan. However, for one specific Lakehouse, several tables that I know exist aren’t showing up in the scanned assets — even after adding the Lakehouse as an asset to a data product in the Unified Catalog. What I’ve tried: I rescanned the workspace and the specific Lakehouses. I verified that the tables are persistent (not temporary) and appear under the Tables section in Fabric, not only as files. I confirmed permissions for the Purview connection account. Scan results and errors: After the rescan, the tables still didn’t appear. The scan logs show several ingestion errors with messages like: Failed to ingest asset with type fabric_lakehouse and qualified name [qualified name] due to invalid data payload to data map I checked the error entries to see which assets they point to, and none of them are related to the tables in the Lakehouse in question. There were four of these errors in the last run. Additional context: Some older Lakehouses that had been archived months ago in Fabric still appeared as active in Purview before the rescan, so there may be stale metadata being retained. Notes: I’m aware Fabric scanning in Purview currently has sub-item scanning limitations where item-level metadata is prioritised, and individual tables aren’t always picked up. But given that tables from other Lakehouses appear as expected, and given the ingestion errors (even though the errors do not point to the missing tables), it feels like there may be a metadata sync or processing issue rather than a simple coverage limitation. Question: Has anyone encountered this behaviour or the “invalid data payload to data map” error before? Any guidance on further troubleshooting steps would be appreciated. Thanks in advance!
No lineage with costum defined types in Purview
To better understand costum lineage, I made following very simple model: #create a simple custom table, which only inherits from DataSet custom_type_table = EntityTypeDef( name="costum_table", superTypes=["DataSet"], #inherit from DataSet serviceType="Test Model", #the bigger group ) #create a simple custom process, which only inherits from process custom_type_process = EntityTypeDef( name="costum_process", superTypes=["Process"], #nherit from Process serviceType="Test Model", #the bigger group ) #upload the entity defenitions client.upload_typedefs( entityDefs=[custom_type_table, custom_type_process], force_update=True ) ae = AtlasEntity("Table", "DataSet","companyname://server/Table", guid=-1) ae_costum = AtlasEntity("CostumTable", "costum_table","companyname://server/CostumTable", guid=-2) aptf = AtlasProcess("Table_Measures", "Process", "companyname://server/Table_Measures", [ae], [ae], guid = -3 ) aptf_costum = AtlasProcess("CostumTable_Measures", "costum_process", "companyname://server/CostumTable_Measures", [ae_costum], [ae_costum], guid = -4 ) res = client.collections.upload_entities( [ae, ae_costum, aptf, aptf_costum], "tttttt") Lineage for the costum case doesn’t show up, for the normal case it does. Even without the processes, if we just look at the table, the DataSet one has a lineage tab, whereas the inherited DataSet entytype has none. Something I find very strange. Do you have an explanation for this? How can lineage with costum types be acchieved? Thanks in advance
