Microsoft Purview now provides an endpoint that supports collecting lineage from Apache Airflow via integration with OpenLineage, an open framework for data lineage collection and analysis. This capability is now in public preview. This endpoint accepts OpenLineage payloads (assets & lineage) in the Data Map. The capability is available via both UX and API.
Teams across Microsoft and OpenLineage community have partnered closely on this effort. The work has also laid the foundation to integrate asset and lineage information from more data sources that support OpenLineage (such as Spark, dbt etc.). We would like to thank the OpenLineage community for a great collaboration.
Get started today!
- Learn more about how to setup Airflow to extract Lineage via OpenLineage: Lineage from Airflow | Microsoft Learn
- Learn more about data lineage features in Microsoft Purview Data Catalog: Data Catalog lineage user guide | Microsoft Learn
Updated Nov 16, 2023
Version 1.0
Arpitha_Dhanapathi
Microsoft
Microsoft
