In our previous installment of Understanding Pipeline Failures and Error Handling, we introduced the difference between TRY-CATCH logic and DO-IF-ELSE logic.
Just a memory refresher, DO-IF-ELSE logic defines both Upon_Success and Upon_Failure paths
If we execute this pipeline, with WillFail activity failing, the whole pipeline will be marked as Failed, as expected
Sometimes users may want a more elegant error handling in their pipeline: despite branching activity (WillFail in this case) failed, I do not want to mark the whole pipeline as Failed, maybe because
- I do not want to block downstream dependent tumbling windows, in case of tumbling window dependency
- branching activity is used to look up a control table, and sometimes it is expected to return empty results and throw an error
To implement the desired behavior, customers may consider DO-IF-SKIP-ELSE block: adding a dummy UponSkipped to the end of the UponSuccess path, like this
If we run this pipeline, WillFail will still fail, but the pipeline will be marked as SUCCESS
In the monitoring view, you can see the difference a dummy UponSkipped activity can make. Notice that PipelineErrorHandlingSuccess is defined with the addition dummy activity, while PipeErrorHandling is not. (And the additional pipeline duration comes from the extra 5 seconds in the dummy wait activity)
Technical reasons for the difference is that, Azure Data Factory defines pipeline success and failures as follows:
- Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead
- Pipeline result is success if and only if all leaves succeed
Here is an expanded table summarizing the difference:
| Approach | Error Handling Defines | When Activity Succeeds | When Activity Fails |
| TRY-CATCH | Only Upon Failure path | Pipeline shows Success | Pipeline shows Success |
| DO-IF-ELSE |
Upon Failure paths + Upon Success path |
Pipeline shows Success | Pipeline shows Failure |
| DO-IF-SKIP-ELSE |
Upon Failure paths + Upon Success path (with a dummy Upon Skipped activity at the end)
|
Pipeline shows Success | Pipeline shows Success |