Bootstrap ETL process by bringing Azure Purview assets into Azure Data Factory

Published Feb 19 2021 06:05 AM 3,626 Views
Microsoft

This is the second blog in the “ADF/Purview integration” series.  And today we are going to discuss integrating Purview assets into ADF to boost the ETL process. 

 

Discovering and understanding data sources and their use is a big challenge for data engineers in the organization and has been an organic process based on communal knowledge. Unless they know the location of a data source and understand their business value, it’s very difficult to discover these data sources and build efficient data workflows to orchestrate data movement and transform data at scale.  

Azure Purview is a unified data governance service that helps you manage and govern your on-premises, multi-cloud, and software-as-a-service (SaaS) data. The Purview Data Map can capture metadata about enterprise data present in analytics and operation systems on-premises and clouds 

With the integration of Azure Purview, data engineers can quickly & easily find relevant data using a search experience in Data Factory portalknow the data and understand its business value, then bring them into Data Factory as linked services or datasets. 

 

Data discovery: search datasets 

As a data engineer, the first step in establishing a data workflow is to find the required data source and understand the meaning and business value of the data. Therefore, a place that can provide this information is very useful for data engineers, which can greatly shorten the time and process of searching for data. 

To discover data registered and scanned by Azure Purview, data engineers can use the Search bar at the top center of Data Factory portal. 

lrtoyou1223_0-1613743355077.png

 

 

 

Actions that you can perform over datasets with Data Factory resources 

Data engineers can directly create Linked Service, Dataset or Dataflow over the data they search using Azure Purview and build code-free ETL workflows for orchestrating data movement and transforming. 

lrtoyou1223_1-1613743355088.png

 

 

Next steps 

%3CLINGO-SUB%20id%3D%22lingo-sub-2149194%22%20slang%3D%22en-US%22%3EBootstrap%20ETL%20process%20by%20bringing%20Azure%20Purview%20assets%20into%20Azure%20Data%20Factory%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2149194%22%20slang%3D%22en-US%22%3E%3CP%3E%3CSPAN%20data-contrast%3D%22auto%22%3EThis%20is%20the%20second%20blog%20in%20the%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3E%E2%80%9CADF%2FPurview%20integration%E2%80%9D%20series%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3E.%26nbsp%3B%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3EAnd%20today%20we%20are%20going%20to%20discuss%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3Eintegrating%20Purview%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3Eassets%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3E%26nbsp%3Binto%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3EADF%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3E%26nbsp%3Bto%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3Eboost%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22none%22%3E%26nbsp%3Bthe%20ETL%20process.%3C%2FSPAN%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22auto%22%3EDiscovering%20and%20understanding%20data%20sources%20and%20their%20use%20is%20a%20big%20challenge%20for%20data%20engineer%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Es%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Bin%20the%20organization%20and%20has%20been%20an%20organic%20process%20based%20on%20communal%20knowledge.%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3BUnless%20they%20know%20the%20location%20of%20a%20data%20source%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Band%20understand%20their%20business%20value%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%2C%20it%E2%80%99s%20very%20difficult%20to%20discover%20these%20data%20sources%20and%20build%20efficient%20data%20workflows%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eto%20orchestrate%20data%20movement%20and%20transform%20data%26nbsp%3Bat%26nbsp%3Bscale.%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22auto%22%3EAzure%20Purview%20is%20a%20unified%20data%20governance%20service%20that%20helps%20you%20manage%20and%20govern%20your%20on-premises%2C%20multi-cloud%2C%20and%20software-as-a-service%20(SaaS)%20data.%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3EThe%20Purview%20Data%20Map%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ecan%20capture%20metadata%20about%20enterprise%20data%20present%20in%20analytics%20and%20operation%20systems%20on-premises%20and%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ec%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Elouds%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E.%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22auto%22%3EWith%20the%20integration%20of%20Azure%20Purview%2C%20data%20engineers%20can%20quickly%20%26amp%3B%20easily%20find%20relevant%20data%20using%20a%20search%20experience%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ein%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3EData%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3EFactory%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Bportal%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%2C%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eknow%20the%20data%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eand%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eunderstand%20its%20business%20value%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%2C%20then%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ebring%20them%20into%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3EData%20Factory%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Bas%20linked%20services%20or%20datasets.%3C%2FSPAN%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CSPAN%20data-contrast%3D%22auto%22%3EData%20discovery%3A%20search%20datasets%3C%2FSPAN%3E%3C%2FSTRONG%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22auto%22%3EAs%20a%20data%20engineer%2C%20the%20first%20step%20in%20establishing%20a%20data%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ework%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eflow%20is%20to%20find%20the%20required%20data%20source%20and%20understand%20the%20meaning%20and%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ebusiness%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Evalue%20of%20the%20data.%20Therefore%2C%20a%20place%20that%20can%20provide%20this%20information%20is%20very%20useful%20for%20data%20engineers%2C%20which%20can%20greatly%20shorten%20the%20time%20and%20process%20of%20searching%20for%20data.%3C%2FSPAN%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22auto%22%3ETo%20discover%20data%20registered%20and%20scanned%20by%20Azure%20Purview%2C%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Edata%20engineers%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Bcan%20use%20the%20Search%20bar%20at%20the%20top%20center%20of%20Data%20Factory%20portal%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E.%3C%2FSPAN%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22lrtoyou1223_0-1613743355077.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F256022i46BF8DD74CFC2931%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22lrtoyou1223_0-1613743355077.png%22%20alt%3D%22lrtoyou1223_0-1613743355077.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CSPAN%20data-contrast%3D%22auto%22%3EActions%20that%20you%20can%20perform%20over%20datasets%20with%20Data%20Factory%20resources%3C%2FSPAN%3E%3C%2FSTRONG%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-contrast%3D%22auto%22%3EData%20engineers%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Bcan%20directly%20create%20Linked%20Service%2C%20Dataset%20or%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3ED%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eataflow%20over%20the%20data%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ethey%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Bsearch%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Eusing%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3BAzure%20Purview%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Band%26nbsp%3B%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3Ebuild%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E%26nbsp%3Bcode-free%20ETL%20workflows%20for%20orchestrating%20data%20movement%20and%20transforming%3C%2FSPAN%3E%3CSPAN%20data-contrast%3D%22auto%22%3E.%3C%2FSPAN%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-inline%22%20image-alt%3D%22lrtoyou1223_1-1613743355088.png%22%20style%3D%22width%3A%20400px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F256023iA11AE75DE528D31F%2Fimage-size%2Fmedium%3Fv%3D1.0%26amp%3Bpx%3D400%22%20role%3D%22button%22%20title%3D%22lrtoyou1223_1-1613743355088.png%22%20alt%3D%22lrtoyou1223_1-1613743355088.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CSPAN%20data-contrast%3D%22auto%22%3ENext%20steps%3C%2FSPAN%3E%3C%2FSTRONG%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FP%3E%0A%3CUL%3E%0A%3CLI%20data-leveltext%3D%22%EF%82%B7%22%20data-font%3D%22Symbol%22%20data-listid%3D%221%22%20aria-setsize%3D%22-1%22%20data-aria-posinset%3D%221%22%20data-aria-level%3D%221%22%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fpurview%2Fcreate-catalog-portal%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3E%3CSPAN%20data-contrast%3D%22none%22%3ECreate%20an%20Azure%20Purview%20account%20now%3C%2FSPAN%3E%3C%2FA%3E%3CSPAN%20data-contrast%3D%22none%22%3E%E2%80%AFand%20start%20by%20connecting%20a%E2%80%AF%3C%2FSPAN%3E%3CA%20href%3D%22https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fpurview%2Fhow-to-link-azure-data-factory%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3E%3CSPAN%20data-contrast%3D%22none%22%3EData%20Factory%3C%2FSPAN%3E%3C%2FA%3E%3CSPAN%20data-ccp-props%3D%22%7B%26quot%3B134233279%26quot%3B%3Atrue%2C%26quot%3B201341983%26quot%3B%3A0%2C%26quot%3B335559739%26quot%3B%3A160%2C%26quot%3B335559740%26quot%3B%3A259%7D%22%3E%26nbsp%3B%3C%2FSPAN%3E%3C%2FLI%3E%0A%3C%2FUL%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2149194%22%20slang%3D%22en-US%22%3E%3CP%3EThis%20blog%20describes%20how%20to%20bring%20Azure%20Purview%20assets%20into%20Azure%20Data%20Factory.%3C%2FP%3E%3C%2FLINGO-TEASER%3E%3CLINGO-LABS%20id%3D%22lingo-labs-2149194%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EAzure%20Data%20Integration%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20ETL%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EAzure%20Purview%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
Co-Authors
Version history
Last update:
‎Feb 19 2021 06:05 AM
Updated by: