Forum Discussion
Dynamically executing a child pipeline using a single Execute Pipeline activity with a variable
Goal: Create a master pipeline that:
- Retrieves metadata using a lookup.
- Calculates a value (caseValue) from the lookup result.
- Maps the value (caseValue) to a pipeline name using a JSON string (pipelineMappingJson).
- Sets the pipeline name (pipelineName) dynamically.
- Runs the correct child pipeline using the pipelineName variable.
Question: Can the Execute Pipeline activity be updated to handle dynamic child pipeline names?
4 Replies
Did you pass the parameters as part of the body of the web activity?
- Lookup Activity > fetching metadata (pipeline name + parameters)
- Output example:
{
"pipelineName": "LoadOrdersPipeline",
"parameters": {
"orderDate": "2025-03-08",
"region": "EU"
}
}
- Output example:
- Set variable
- pipelineName: @activity('LookupActivity').output.firstRow.pipelineName
- parametersJson: @string(activity('LookupActivity').output.firstRow.parameters)
- Web Activity
- Example URL: https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{dataFactoryName}/pipelines/@{variables('pipelineName')}/createRun?api-version=2018-06-01
- Method: POST
- Headers: Content-Type: application/json
- Body (dynamic content)
- {
"parameters": @variables('parametersJson')
}
- {
- Authenticate via MI or SP
- Lookup Activity > fetching metadata (pipeline name + parameters)
As far as I know, this is not possible. The pipeline must be selected at design time for validation purposes.
Workarounds:
- Use a Switch Activity to call different pipelines based on the value.
For example:
Master Pipeline
- Lookup (get caseValue)
- Switch Activity (evaluate caseValue)
- Case = "LoadCustomers" > Execute Pipeline: LoadCustomersPipeline
- Case = "LoadOrders" > Execute Pipeline: LoadOrdersPipeline
- Case = "LoadProducts" > Execute Pipeline: LoadProductsPipeline - Create a generic child pipeline that handles different logic based on a passed parameter.
For example:
MasterPipeline
- Lookup (Metadata)
- Set Variable (taskType)
- Execute Pipeline (GenericChildPipeline, passing taskType)
GenericChildPipeline (parameter: taskType)
- Switch
- Case: LoadCustomers > Customer Copy Activity
- Case: LoadOrders > Orders Copy Activity
- Case: LoadProducts > Products Copy Activity - Metadata-driven with using a Web Activity to trigger the pipeline dynamically via ADF REST API.
For example:
MasterPipeline
- Lookup (Get Metadata - includes pipeline name)
- Web Activity (Trigger Pipeline via REST API - dynamic URL) - Orchestrate externally using Logic Apps, Azure Functions, or custom scripts.
Thank you for getting back. The existing approach uses a Switch Activity to call different pipelines based on the value (like you'd mentioned)
But we are running out of cases (25 limit in Switch) and we already have 2 master pipelines with close to 50 cases (25 each). I was looking for a way to optimise it and make it singular.
Web Activity is another possibility I'm exploring but cannot pass pipeline parameters dynamically, it's failing to capture the values. Any idea why this could be so?As far as I know for Web Activity it requires several components and need to set up Global Parameters in your ADF such as subscriptionId, ResourceGroupName and factory Name - these can be set in the ADF portal under "Manage" -- "Global parameters" and also to connect to the Azure Management API uses MSI authentication.
So I think the critical issues may occurs on how are we formatting the parameters in the Web Activity body.
Storing parameters separately, you might need to build the JSON manually example:
@concat('{\"parameters\":{\"param1\":\"', variables('param1'), '\",\"param2\":\"', variables('param2'), '\"}}')
In above JSON structure, even small errors or quotes escaping or at name of parameters may cause failing to capture the values.
- Use a Switch Activity to call different pipelines based on the value.
