Understanding Pipeline Failures and Error Handling 2

Published Apr 29 2021 04:00 PM 4,721 Views

In our previous installment of Understanding Pipeline Failures and Error Handling, we introduced the difference between TRY-CATCH logic and DO-IF-ELSE logic.

 

Just a memory refresher, DO-IF-ELSE logic defines both Upon_Success and Upon_Failure paths

Understanding Error Message 04 WillFail Definition.png

 If we execute this pipeline, with WillFail activity failing, the whole pipeline will be marked as Failed, as expected

Understanding Error Message 01 IF-ELSE.png

 

Sometimes users may want a more elegant error handling in their pipeline: despite branching activity (WillFail in this case) failed, I do not want to mark the whole pipeline as Failed, maybe because

  • I do not want to block downstream dependent tumbling windows, in case of tumbling window dependency
  • branching activity is used to look up a control table, and sometimes it is expected to return empty results and throw an error

To implement the desired behavior, customers may consider DO-IF-SKIP-ELSE block: adding a dummy UponSkipped to the end of the UponSuccess path, like this

Understanding Error Message 03 MarkSuccess Definition.png

 

If we run this pipeline, WillFail will still fail, but the pipeline will be marked as SUCCESS

Understanding Error Message 05 IF-ELSE-Success.png

 

In the monitoring view, you can see the difference a dummy UponSkipped activity can make. Notice that PipelineErrorHandlingSuccess is defined with the addition dummy activity, while PipeErrorHandling is not. (And the additional pipeline duration comes from the extra 5 seconds in the dummy wait activity)

Understanding Error Message 02 Run Status.png

 

Technical reasons for the difference is that, Azure Data Factory defines pipeline success and failures as follows:

  • Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead
  • Pipeline result is success if and only if all leaves succeed

 

Here is an expanded table summarizing the difference:

 

Approach Error Handling Defines When Activity Succeeds When Activity Fails
TRY-CATCH Only Upon Failure path Pipeline shows Success Pipeline shows Success
DO-IF-ELSE

Upon Failure paths

+

Upon Success path

Pipeline shows Success Pipeline shows Failure
DO-IF-SKIP-ELSE

Upon Failure paths

+

Upon Success path (with a dummy Upon Skipped activity at the end)

 

Pipeline shows Success Pipeline shows Success

 

1 Comment
Co-Authors
Version history
Last update:
‎Apr 29 2021 04:01 PM
Updated by: