How to use Profisee with Azure Data Factory (ADF)
A key aspect of any analytic project is the availability of high quality data. High quality data enables users to support the desired analytics required. As data are ingested from multiple data sources, it is important to make sure that the data are cleansed, and transformed to be the right shape for analytics.
Profisee MDM is a master data management platform, built on Azure, and designed to ‘combine and align’ data from multiple sources and deliver trusted, relevant, and authoritative information to drive the efficiency and effectiveness of Azure.
We are excited to share the release of Profisee pre-built templates for ADF. These templates show how you can use Profisee and Azure Data Factory together to deliver high-quality data for the organization. These Profisee ARM templates enables you to use Profisee with Azure Data Factory and various Azure Data services. In addition, a reference architecture is provided on how to read and write data using the Profisee’s REST Gateway API.
Architecture and Solution
Using the Profisee REST gateway, 3rd party services can connect with Profisee using REST API. Ingress and Egress modules enable Azure data integration for Profisee, using Azure Data Factory to enable data ingestion, and transformation. Figure 1 shows the details on using Azure Data Factory with the Profisee Master Data Management solution.
Figure 1 - Azure Data Factory and Profisee Architecture Solution
Profisee leverages the REST integration support with ADF to provide a lightweight and modern integration solution for master data. The pre-built templates enable you to:
- Load Source Data to MDM – Azure Data Factory is used to extract the data from data stored on Azure Data Lake Storage Gen2, various Azure Data sources, SaaS sources, and more. Azure Data Factory helps you transform the data to align to the master data model, and load it into the MDM repository via a REST sink.
- Master Data Management Processing – The MDM platform processes source master data through a sequence of activities to verify, standardize and enrich the data, as well as execute data quality processes. Finally, matching and survivorship is performed to find and group duplicate records and create master records. Optionally, data stewards can be issued tasks to perform data stewardship. The result is a set of high-quality, trusted master data for use in downstream analytics, machine learning and so on.
- Load Master Data for Analytics – Azure Data Factory uses its REST source to load master data from Profisee to a rich set of Azure data stores.
Azure Data Factory Templates for Profisee
In collaboration with Microsoft, Profisee has developed a set of Azure Data Factory templates that make it faster and easier to integrate Profisee with various Azure Data services. These templates use Azure Data Factory REST data source and data sink to read and write data from Profisee’s REST Gateway API. Templates are provided for both reading from and writing to Profisee.
Figure 2 - Azure Data Factory pre-built templates for Profisee
ADF REST connectors enable Profisee to write Profisee model objects into Profisee using the Profisee REST gateway. ADF enables Profisee to load data from data lakes, Dynamics 365, Salesforce and more. In addition, ADF enables customers to ingest Profisee objects, and enabling customers to perform ETL tasks on Profisee objects. Choices of integration are plenty given the ADF connector portfolio.
Click here to learn more about the Azure Data Factory templates for Profisee.