This is the second blog in the “ADF/Purview integration” series. And today we are going to discuss integrating Purview assets into ADF to boost the ETL process.
Discovering and understanding data sources and their use is a big challenge for data engineers in the organization and has been an organic process based on communal knowledge. Unless they know the location of a data source and understand their business value, it’s very difficult to discover these data sources and build efficient data workflows to orchestrate data movement and transform data at scale.
Azure Purview is a unified data governance service that helps you manage and govern your on-premises, multi-cloud, and software-as-a-service (SaaS) data. The Purview Data Map can capture metadata about enterprise data present in analytics and operation systems on-premises and clouds.
With the integration of Azure Purview, data engineers can quickly & easily find relevant data using a search experience in Data Factory portal, know the data and understand its business value, then bring them into Data Factory as linked services or datasets.
Data discovery: search datasets
As a data engineer, the first step in establishing a data workflow is to find the required data source and understand the meaning and business value of the data. Therefore, a place that can provide this information is very useful for data engineers, which can greatly shorten the time and process of searching for data.
To discover data registered and scanned by Azure Purview, data engineers can use the Search bar at the top center of Data Factory portal.
Actions that you can perform over datasets with Data Factory resources
Data engineers can directly create Linked Service, Dataset or Dataflow over the data they search using Azure Purview and build code-free ETL workflows for orchestrating data movement and transforming.
Next steps
- Create an Azure Purview account now and start by connecting a Data Factory