Automated publish improvement in ADF's CI/CD flow

[Abhishek Narain]

The "Automated publish" improvement takes the validate all and export Azure Resource Manager (ARM) template functionality from the ADF UI and makes the logic consumable via a publicly available npm package @microsoft/azure-data-factory-utilities. This allows you to programmatically trigger these actions instead of going to the ADF UI and do a button click 'Publish'. This will give your CI/CD pipelines a truer continuous integration experience.

Current CI/CD flow

1. Each user makes changes in their private branches.
2. Push to master is forbidden; users must create a PR to master to make changes.
3. Users must load ADF UI and click publish to deploy changes to Data Factory and generate the ARM templates in the Publish branch.
4. DevOps Release pipeline is configured to create a new release and deploy the ARM template each time a new change is pushed to the publish branch.

 

Manual step in current CI/ CD flow

In the current CI/CD flow, the ADF UI is the intermediary to create the ARM template; therefore, a user must go to ADF UI and manually click publish to start the ARM template generation and drop it in the publish branch. This manual step may be problematic for a few with different expectations around automation in the CI/CD process. 

The new CI/ CD flow

1. Each user makes changes in their private branches.
2. Push to master is forbidden; users must create a PR to master to make changes.
3. Azure DevOps pipeline build is triggered every time a new commit is made to master, validates the resources and generates an ARM template as an artefact if validation succeeds.
4. DevOps Release pipeline is configured to create a new release and deploy the ARM template each time a new build is available.

For more details on implementing the new flow, check out our documentation. 

Note: You can continue to use the existing mechanism (adf_publish branch) or use the new flow. Both mechanisms will be supported, just that the latest one removes the additional 'publish' requirement using the ADF UI and does not rely on the 'adf_publish' branch. Choose the flow which works better for you.