Understanding Pipeline Failures and Error Handling

[ChenyeCharlieZhu]

Azure Data Factory orchestration allows conditional logic and enables user to take different based upon outcomes of a previous activity. In total we allows four conditional paths: Upon Success (default pass), Upon Failure, Upon Completion, and Upon Skip. Using different paths allow users to build robust pipelines and incorporates error handling in their ETL/ELT logic.

Here are two common error handling pattern we see customers use:

1. TRY-CATCH block. Define the business logic, and only defines Upon Failure path to catch any error from previous activities 
2. DO-IF-ELSE block. Define the business logic, and depends on the outcome of the activity, enacts either Upon Success path or Upon Failure path

Both are valid ways to incorporate error handling into the pipeline. However, upon pipeline execution, they may show different outcomes. Approach #1, TRY-CATCH, shows pipeline succeeds if Upon Failure path clears, where as approach #2, DO-IF-ELSE show pipeline failed if Upon Failure path is enacted.

Technical reasons for the difference is that, Azure Data Factory defines pipeline success and failures as follows:

• Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead
• Pipeline result is success if and only if all leaves succeed

Applying the logic to previous examples. 

1. In approach #1 TRY-CATCH block:
   • when previous activity succeeds: the node activity, Upon Failure, is skipped and its parent node succeeds, so overall pipeline succeeds
   • when previous activity fails: the node activity, Upon Failure, enacted and overall pipeline succeeds if Upon Failure path succeeds
2. In approach #2 DO-IF-ELSE block:
   • when previous activity succeeds: one node activity, Upon Success, succeeded, and the other node activity, Upon Failure, is skipped and its parent node succeeds; so overall pipeline succeeds
   • when previous activity fails: one node activity, Upon Success, is skipped and its parent node failed; so overall pipeline failed

Here is a table summarizing the difference

Approach	Error Handling Defines	When Activity Succeeds	When Activity Fails
TRY-CATCH	Only Upon Failure path	Pipeline shows Success	Pipeline shows Success
DO-IF-ELSE	Upon Failure and Upon Success paths	Pipeline shows Success	Pipeline shows Failure

[joelforsyth]

Hi @ChenyeCharlieZhu, I'm following the TRY-CATCH approach inside of an If Condition and the If Condition still shows as failing. Does it not work on those activities? I also tried the approach on having the entire pipeline execution output to a single activity on failure inside of another pipeline, and the executing pipeline still shows as failing as well. Any ideas?

[ChenyeCharlieZhu]

Hi we have a part 2 to the series now, expanding the capabilities of DO-IF-ELSE block: https://techcommunity.microsoft.com/t5/azure-data-factory/understanding-pipeline-failures-and-error-.... Enjoy

[MBErickson]

We find the DO-IF-ELSE pattern very useful to create easier to maintain and read ADF pipelines, which almost produces flow-like pipeline patterns. Using a simple Azure Function App for testing, this has been extremely useful. The problem is this requires us to place a "NoOperation" activity as a skip result for every path in the pipeline. It would be extremely useful to provide a switch to ignore paths not followed.

[9rune5]

"Pipeline Shows _success_" -- does it?

I've tried for the better part of the day now, and I cannot figure out how to do the equivalent of "swallowing exceptions". My problem might be compounded since I'm executing a pipeline within my main pipeline. All I get is either "all good" or "something went wrong, but I won't tell you what".

[ChenyeCharlieZhu]

@9rune5 please take a look at the throw Error activity that we recently introduced. That’s used for surfacing error message back to main pipeline. 

[9rune5]

@ChenyeCharlieZhu Thanks, but I was trying to make the error go away.

try..catch usually implies that you can catch an error and then (optionally) continue as if nothing had happened. I am not seeing that level of control in ADF.

[chitraR20148]

@ChenyeCharlieZhu

Hi there,  where can i see this throw error activity ---is it the Fail activity u  r referring to  ? 

thanks

chitra