When building complex workflows in the cloud with Azure Data Factory and Azure Synapse Pipelines, a very common pattern is to separate different workflow branches into child pipelines. The ADF team is super-excited to announce that we've made creating workflows even simpler with this new preview feature! Now you can customize the output of your pipeline and consume that return value from the calling pipeline.
In this simple example of a parent pipeline below, I have an execute pipeline activity that is calling a child pipeline. The executed pipeline will return a custom value after it has been executed. I am then consuming that return value in the subsequent activity in my parent pipeline. The 2nd activity below is a Script activity which I am using for logging and I use the expression below to grab the return value from the previous activity.
You can set the return value to either an expression or a static value. The expression to use in the parent pipeline is available as part of the pipeline expression builder:
The ADF pipeline expression builder provides quick access to the return value (preview) that I set in the child pipeline using Set Variable. Notice that I called my return value variable "myval" so that I can then access it from the parent pipeline after the previous activity expression like this:
@activity('Execute Pipeline1').output.pipelineReturnValue.myval
The mechanism used to set the pipeline return value is from the Set Variable activity which has been enhanced to now allow an option for either a user-defined variable (previously the only option) or now you can choose "Pipeline return value (preview)". Choose a name for your return value variable and then set it either to a static value or an expression.
Resources
- Set Variable Activity - Azure Data Factory & Azure Synapse | Microsoft Learn
- Set Pipeline Return Value - Azure Data Factory & Azure Synapse | Microsoft Learn
We are always open to feedback so please let us know your thoughts in the comments below or add to our Ideas forum.