Introduction:
This article discusses the process of deactivating or skipping an activity in Fabric Data Factory pipelines by marking it as inactive. Fortunately, Microsoft has recently introduced an enhancement that allows data engineers to effortlessly customize their data pipelines and skip specific activities.
Prerequisites:
- Microsoft Fabric workspace
- Basic knowledge in Data Factory pipelines.
Table of Contents:
Patterns For Deactivating An Activity:
There are various scenarios in which deactivating an activity within an ADF pipeline can be useful for managing and controlling your data workflows. Here are some complex patterns where activity deactivation can be beneficial:
- Conditional Execution: Utilize deactivation for conditional execution in your pipeline. Deactivate activities based on conditions like data quality, triggers, or runtime, enabling dynamic workflow adjustments.
- Debugging and Testing: Deactivate activities during debugging and testing phases to isolate and troubleshoot issues. You can deactivate certain activities to focus on diagnosing problems in other parts of the pipeline without the interference of unnecessary processing.
- Partial Pipeline Execution: In complex pipelines, sometimes we want to run only a portion of the pipeline for testing or validation purposes. Deactivating the activities that are not relevant for the current test scenario allows us to execute and validate specific parts of the pipeline independently.
- Rollback Scenarios: when we have an error or data issue in a pipeline, we can deactivate certain activities to prevent them from further processing while we address the problem. This can be part of a rollback or recovery strategy to ensure data integrity.
- Versioning and A/B Testing: When making changes to our data pipeline, we can create a new version of the pipeline and deactivate activities in the old version while testing the new version.
- Maintenance and Upgrades: During maintenance or system upgrades, deactivate activities to prevent any unwanted processing. This ensures that no data is processed while you're performing maintenance tasks on the pipeline or its dependencies.
Demo:
In this blog, I'm employing four activities: a primary copy activity, a notebook activity set to execute upon success, a team's activity triggered by failure, and an outlook activity scheduled upon completion.
Let's examine various pipeline behaviors based on the status of the copy activity.
To do so, we need to change the activity state to 'Deactivated' and select how do we want to mark the copy activity and in all the runs below, the copy activity will be marked as 'inactive'.
Succeeded copy activity:
Mark activity as succeeded:
this will trigger the outlook and notebook activity
Skipped copy activity:
No activities will be executed because we haven't defined any actions to be taken at the skip level.
Failed copy activity:
Only the teams and outlook activities were executed, which is expected because we didn't intend for the notebook activity to run.
To conclude, deactivate an activity in fabric pipelines can be very beneficial for various complex scenarios, please check Microsoft documentation for further information.
Links:
- Deactivate an Activity - Microsoft Fabric | Microsoft Learn
- How to copy data using copy activity - Microsoft Fabric | Microsoft Learn
- Teams activity - Microsoft Fabric | Microsoft Learn
- Notifying Outlook and Teams channel/group from a Microsoft Fabric pipeline | Microsoft Fabric Blog |...
Call-To-Action:
- check MS documentation on Fabric.
- Please help us improve by sharing your valuable Microsoft Fabric Preview feedback by emailing us.
- Follow me on LinkedIn for more content: Sally Dabbah | LinkedIn