Forum Discussion
Manage Pipelines (Start/Stop/Monitoring)
I cannot find a way to manage many pipelines for ETL. For example, in case of multiple pipelines execution, if i want to disable the execution of any pipelines - how can this be done ?
Is there a tool by Microsoft or any third party tool, which can help manage, execution and monitoring of pipelines in ADF ?
Also, are there any best practices or patterns to manage multiple pipelines ?
1 Reply
Azure Data Factory does not offer a direct "Disable Pipeline" option, but there are several ways to control pipeline execution:
- Disable the trigger if the pipeline runs on a schedule.
- Use a control table (e.g., in Azure SQL) with columns PipelineName and IsActive (0 = disabled, 1 = enabled).
- Add a Lookup activity in the pipeline to check the IsActive value; if 0, the pipeline will not execute. - Use a global parameter (e.g., PipelineName_Enabled, set to true or false).
- Add a conditional check in the pipeline to evaluate this parameter before execution. - Implement a master pipeline that conditionally triggers other pipelines.
For executing pipelines, the same methods apply, with triggers being most common approach in ADF.
For monitoring, you can:
- Use the Monitor feature in ADF Studio.
- Enable diagnostics in the Azure portal to send logs to Log Analytics, allowing alerts on pipeline execution status.
Let us know your preferred approach, and we can provide further details on implementing it!
Monitor option in ADF Studio
Diagnostics settings on Data Factory resource in the Azure portal
