MicrosoftI have been doing meta data ETL in ADF for some time using a product I've developed for this called ChillETL. ChillETL uses meta data stored in an Azure SQL database to not only copy tables but do incremental copies, power bi refreshes, stored procedure execution all thru parameterized pipelines. It also manages the scheduling of multiple processes, both sequentially and concurrently based on dependencies. ChillETL is listed in the Azure Marketplace and was approved by Microsoft for its partner co-selling program. Here are some advantages of ChillETL vs the feature described here:
• In ChillETL the metadata is Maintained and Updated via an Excel Add-In vs SQL Scripts which is a very big timesaver. Behind the Scenes the Excel Add-In generates the SQL Scripts.
• ChillETL Handles sequencing all types of processes (Executing Procs and Queries, Refreshing Power BI Datasets, Importing from Rest APIs, etc)
• ChillETL provides Intelligent Restart on Failures, where only the process that failed and subsequent processes are executed rather than re-running all the processes again.
• ChillETL provides Logging and Real Time Reporting via Power BI
• ChillETL provides Email Notifications and integration with Azure Logic Apps
• ChillETL Excel Add-In Generates SQL Tables, Procedures, Views and updates Power BI Models
