UPDATE! Improvements to the CI/CD process by only requiring changed or updated triggers to be stopped and started during deployments using the updated PowerShell is generally available as of October 14, 2022.
The CI/CD in ADF uses ARM (Azure Resource Manager) templates that contain the pipeline, dataset, linked services, triggers, etc. definition. The deployments require turning off all the triggers in the data factory using PowerShell (PrePostDeploymentScript.ps1).
Most of the time, this is okay. Still, when you had mission-critical pipelines, say an event trigger-based or a pipeline that required zero-latency start, it may cause latency or even loss of events due to the triggers being turned off during deployments. The problem gets worse when deployments take longer.
While it is essential to turn off the updated triggers for a safe deployment since those contain state information, we are improving this process by only requiring the triggers that are changed/updated that will be stopped and started during deployments using the updated PowerShell script (PrePostDeploymentScript.Ver2.ps).
It stops the triggers only if they are altered and are in a started state, an improvement to the previous script (PrePostDeploymentScript.ps1) that stops all triggers before deployment.
The updated PrePostDeploymentScript compares and only turns off/ on the triggers that need to be updated rather than turning all triggers off/ on. It would result in quicker deployments, especially for the giant data factory with 1000s of artifacts in the ARM.
We have also updated the ADF's npm utility for automated publishing (Fully automated CICD). Below, you can find the steps to update your existing release process.
How to update your exisiting release process:
1. Update the azure-pipeline.yml
2. Update the package.json file
If you’re not already using the automated publish functionality, you will need to directly update the pre/post deployment script with the PrePostDeploymentScript.Ver2.ps.
Ensure that you use the PowerShell core in ADO (Azure DevOps task).
Document references:
- Automated publishing for continuous integration and delivery - Azure Data Factory | Microsoft Docs
- Automate continuous integration - Azure Data Factory | Microsoft Docs
Thanks for reading! We love hearing your feedback. If you have any thoughts or suggestions regarding these updates, please leave a comment.