Analyze root cause and impact using ADF ETL lineage in Azure Purview

Published Feb 12 2021 06:32 AM 2,689 Views
Microsoft

Welcome to the first of the “Azure Data Factory and Azure Purview integration” blog series! 

 

Many enterprise customers and ISVs are using Azure Data Factory for their data integration and transformation needs.  We are excited to see how ADF is making modeling and operating ETL flows more efficient and scalable for organizations worldwide!  To make that experience even better, we are integrating ADF with Azure Purview to bring more customer values around lineage, policy management, and data governance. 

 

In this blog series we will share scenarios, use cases, and how-to instructions on integration between ADF and Azure Purview.  Each post will take 5-10 minutes to read. 

 

As enterprises bring into more siloed data into their data lake and enterprise data warehouse, the data integration processes become more complex.  As a result, it gets increasingly harder for users to track the freshness of the produced data and to analyze and optimize the existing ETL process For example, when user needs to modify some data sources, it is very difficult to know how much impact it has on the ETL and downstream analytical process. 

 

Azure Purview is a new cloud service for data users to centrally manage and govern data across their data estate spanning cloud and on-prem environments.  

Picture1.png

You can connect one or more data factories to an Azure Purview account, and the connection allows Data Factory to automatically publish lineage data for Copy, Data flow and SSIS package execution into Azure Purview.  Data Factory Lineage in Azure Purview provides detailed information for root cause analysis and impact analysis.

 

Scenarios 1: Root cause analysis

Data Engineers own data sources within a data estate. In situations when a data source has incorrect data due to upstream issues, the data engineers have no centralized place to identify the issues.

Using catalog lineage, the data engineers can understand upstream process failures and be informed about the reasons for their data sources discrepancy.

Picture2.png

Scenario 2: Impact analysis

If data Producers want to change and deprecate a column in a data source they own, and they want to know who are being impacted upon making such a change. There isn't a centralized place a data producer can know all the downstream consumers of their data sources.

Using catalog lineage, a data producer can easily understand the impact of the downstream assets upon changing the attributes of the data source.

Picture3.png

Next steps

%3CLINGO-SUB%20id%3D%22lingo-sub-2128396%22%20slang%3D%22en-US%22%3EAnalyze%20root%20cause%20and%20impact%20using%20ADF%20ETL%20lineage%20in%20Azure%20Purview%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2128396%22%20slang%3D%22en-US%22%3E%3CP%3EWelcome%20to%20the%26nbsp%3Bfirst%26nbsp%3Bof%20the%20%E2%80%9CAzure%20Data%20Factory%20and%20Azure%26nbsp%3BPurview%20integration%E2%80%9D%20blog%20series!%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMany%26nbsp%3Benterprise%26nbsp%3Bcustomers%26nbsp%3Band%20ISVs%26nbsp%3Bare%20using%20Azure%20Data%20Factory%26nbsp%3Bfor%20their%20data%20integration%26nbsp%3Band%20transformation%26nbsp%3Bneeds.%26nbsp%3B%26nbsp%3BWe%20are%20excited%26nbsp%3Bto%20see%20how%26nbsp%3BADF%20is%26nbsp%3Bmaking%26nbsp%3Bmodeling%20and%20operating%20ETL%20flows%20more%20efficient%20and%20scalable%26nbsp%3Bfor%20organizations%20worldwide!%26nbsp%3B%26nbsp%3BTo%20make%20that%20experience%20even%20better%2C%26nbsp%3Bwe%20are%20integrating%20ADF%20with%20Azure%20Purview%20to%26nbsp%3Bbring%20more%20customer%20values%26nbsp%3Baround%26nbsp%3Blineage%2C%20policy%20management%2C%20and%20data%20governance.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIn%20this%20blog%20series%26nbsp%3Bwe%26nbsp%3Bwill%26nbsp%3Bshare%26nbsp%3Bscenarios%2C%26nbsp%3Buse%20cases%2C%20and%26nbsp%3Bhow-to%20instructions%20on%26nbsp%3Bintegration%20between%20ADF%20and%26nbsp%3BAzure%26nbsp%3BPurview.%26nbsp%3B%20Each%20post%20will%20take%205-10%20minutes%20to%20read.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAs%26nbsp%3Benterprises%26nbsp%3Bbring%20into%20more%26nbsp%3Bsiloed%26nbsp%3Bdata%20into%20their%26nbsp%3Bdata%20lake%20and%20enterprise%20data%20warehouse%2C%26nbsp%3Bthe%20data%26nbsp%3Bintegration%26nbsp%3Bprocesses%26nbsp%3Bbecome%26nbsp%3Bmore%26nbsp%3Bcomplex.%26nbsp%3B%26nbsp%3BAs%20a%20result%2C%26nbsp%3Bit%26nbsp%3Bgets%20increasingly%26nbsp%3Bharder%20for%26nbsp%3Busers%20to%26nbsp%3Btrack%20the%20freshness%26nbsp%3Bof%20the%20produced%20data%20and%26nbsp%3Bto%26nbsp%3Banalyze%20and%20optimize%20the%20existing%26nbsp%3BETL%26nbsp%3Bprocess.%26nbsp%3B%26nbsp%3BFor%20example%2C%26nbsp%3Bwhen%26nbsp%3Buser%20needs%20to%26nbsp%3Bmodify%20some%20data%20sources%2C%20it%20is%26nbsp%3Bvery%26nbsp%3Bdifficult%26nbsp%3Bto%20know%20how%20much%20impact%20it%20has%20on%20the%26nbsp%3BETL%20and%20downstream%20analytical%26nbsp%3Bprocess.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EAzure%20Purview%20is%20a%20new%20cloud%20service%20for%20data%20users%26nbsp%3Bto%26nbsp%3Bcentrally%20manage%26nbsp%3Band%26nbsp%3Bgovern%26nbsp%3Bdata%20across%20their%20data%20estate%20spanning%20cloud%20and%20on-prem%20environments.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3EYou%20can%20connect%20one%20or%20more%20data%20factories%20to%20an%20Azure%20Purview%20account%2C%20and%20the%20connection%20allows%20Data%20Factory%20to%20automatically%20publish%20lineage%20data%20for%20Copy%2C%20Data%20flow%20and%20SSIS%20package%20execution%20into%20Azure%20Purview.%26nbsp%3B%20Data%20Factory%20Lineage%20in%20Azure%20Purview%20provides%20detailed%20information%20for%20root%20cause%20analysis%20and%20impact%20analysis.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EScenarios%201%3A%20Root%20cause%20analysis%3C%2FP%3E%3CP%3EData%20Engineers%20own%20data%20sources%20within%20a%20data%20estate.%20In%20situations%20when%20a%20data%20source%20has%20incorrect%20data%20due%20to%20upstream%20issues%2C%20the%20data%20engineers%20have%20no%20centralized%20place%20to%20identify%20the%20issues.%3C%2FP%3E%3CP%3EUsing%20catalog%20lineage%2C%20the%20data%20engineers%20can%20understand%20upstream%20process%20failures%20and%20be%20informed%20about%20the%20reasons%20for%20their%20data%20sources%20discrepancy.%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3EScenario%202%3A%20Impact%20analysis%3C%2FP%3E%3CP%3EIf%20data%20Producers%20want%20to%20change%20and%20deprecate%20a%20column%20in%20a%20data%20source%20they%20own%2C%20and%20they%20want%20to%20know%20who%20are%20being%20impacted%20upon%20making%20such%20a%20change.%20There%20isn't%20a%20centralized%20place%20a%20data%20producer%20can%20know%20all%20the%20downstream%20consumers%20of%20their%20data%20sources.%3C%2FP%3E%3CP%3EUsing%20catalog%20lineage%2C%20a%20data%20producer%20can%20easily%20understand%20the%20impact%20of%20the%20downstream%20assets%20upon%20changing%20the%20attributes%20of%20the%20data%20source.%3C%2FP%3E%3CP%3E%3C%2FP%3E%3CP%3ENext%20steps%3C%2FP%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fpurview%2Fcreate-catalog-portal%22%20rel%3D%22noopener%20noreferrer%22%20target%3D%22_blank%22%3ECreate%20an%20Azure%20Purview%20account%20now%3C%2FA%3E%26nbsp%3Band%20start%20by%20connecting%20a%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fdata-factory%2Fconnect-data-factory-to-azure-purview%22%20rel%3D%22noopener%20noreferrer%22%20target%3D%22_blank%22%3EData%20Factory%3C%2FA%3E%20Learn%20how%20to%20%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fdata-factory%2Fturorial-push-lineage-to-purview%22%20rel%3D%22noopener%20noreferrer%22%20target%3D%22_blank%22%3Epush%20and%20see%20lineage%20data%20in%20Azure%20Purview%3C%2FA%3E%20Learn%20more%20on%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fazure%2Fpurview%2Fcatalog-lineage-user-guide%22%20rel%3D%22noopener%20noreferrer%22%20target%3D%22_blank%22%3Elineage%20user%20guide%3C%2FA%3E.%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2128396%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20blog%20describes%20how%20to%20analyze%20root%20cause%20and%20impact%20using%20ADF%20ETL%20lineage%20in%20Azure%20Purview.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2128396%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Factory%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20ETL%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Purview%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Feb 12 2021 06:32 AM
Updated by: