Automated publish improvement in ADF's CI/CD flow

Published 02-08-2021 05:40 PM 6,701 Views

The "Automated publish" improvement takes the validate all and export Azure Resource Manager (ARM) template functionality from the ADF UI and makes the logic consumable via a publicly available npm package @microsoft/azure-data-factory-utilities. This allows you to programmatically trigger these actions instead of having to go to the ADF UI and do a button click 'Publish'. This will give your CI/CD pipelines a truer continuous integration experience.


Current CI/CD flow

  1. Each user makes changes in their private branches.
  2. Push to master is forbidden; users must create a PR to master to make changes.
  3. Users must load ADF UI and click publish to deploy changes to Data Factory and generate the ARM templates in the Publish branch.
  4. DevOps Release pipeline is configured to create a new release and deploy the ARM template each time a new change is pushed to the publish branch.




Manual step in current CI/ CD flow

In current CI/CD flow, the ADF UI is the intermediary to create the ARM template; therefore a user must go to ADF UI and manually click publish to start the ARM template generation and drop it in the publish branch. This manual step may be problematic for a few with different expectations around automation in the CI/CD process. 



The new CI/ CD flow

  1. Each user makes changes in their private branches.
  2. Push to master is forbidden; users must create a PR to master to make changes.
  3. Azure DevOps pipeline build is triggered every time a new commit is made to master, validates the resources and generates an ARM template as an artefact if validation succeeds.
  4. DevOps Release pipeline is configured to create a new release and deploy the ARM template each time a new build is available.



For more details on implementing the new flow, check out our documentation


Note: You can continue to use the existing mechanism (adf_publish branch) or use the new flow. Both mechanisms will be supported, just that the latest one removes the additional 'publish' requirement using the ADF UI. Choose the flow which works better for you. 


@Abhishek Narain , I've noticed that thus utility doesn't consider managedVirtualNetworks and managedPrivateEndpoints when it creates ARM templates. Because of this, I am unable to use this method to automate publishing. Do you have some solution or work around for this issue?


Is it possible to run validate all against a data factory in a feature branch?  Would be nice to block merges that won't pass validation before they get to master.

New Contributor

@bcassell-MSFT We use a dev branch AND a master branch, and make dev branch the Collaborative branch in our dev Data Factory resource. We make feature branches off dev, and then PR to dev first. The release pipeline in Azure DevOps deploys to the Dev environment first.

I think the issue with the old guidance is that the CI/CD diagram made it look like you must only have a master branch, and it looks like the new guidance suffers the same shortsightenness. Hopefully, someone special on the DF team is reading this.


@kolangareth The issue was fixed in the latest NPM package. Please update the NPM package and try.


@Jason Kohlhoff That's fair feedback. It should be the customer's choice. The major problem we sorted with this is the automated 'publish' and 'validate', getting rid of the adf_publish branch dependency, which was updated on publish using the collaboration branch (it could be master or any other).
We will come up with better guidance on deploying feature/ dev branches across environments and the limitless possibilities customers can use the CI/CD.    


@Abhishek Narain  I am using the latest version 0.1.3. But the managedVirtualNetworks and managedPrivateEndpoints are not included in the Arm Template even with this version.



This package is not exclusive for the collaboration branch, you can generate any ARM template based on any branch.

Also, it is possible to run a build when a PR is raised and add it as a mandatory check before merging. You can do this in the branch policies section, e.g.,:




Also note that you can also run a build every time there's a new commit in master.

Senior Member

@Abhishek Narain  I have noticed, while testing the ADF NPM utility, that the deleted pipelines are not getting undeployed from ADF environment.


Does the ADF utility not support this feature?


Do we need to follow some steps apart from the utility?

Senior Member

Hello Microsoft team,


NPM Utility is removing Azure DevOps Git configuration from my Data Factory integration.


I had to set after every build. Do we have any fix for this?




About the resources not deleted:

Since the ARM template deployment is an incremental operation, the resources are not deleted by design. You need to use a Post deployment PS script in order to do so, this script is included in the output of the package (PrePostDeploymentScript.ps1). Here's more info about this script and how to implement it in your release definition.



Regarding the Git problem:


It is possible that you are including global parameters in the ARM template. Since global parameters are located in the factory entity, when they are added, the factory entity is added in the ARM template as well, however, the Git configuration is not. You can check the box to do not include them in the ARM template and use the Global Parameters post deployment script in order to update them in the target environment. This script is provided in the output of the package as well (GlobalParametersUpdateScript.ps1). Here's more info about global parameters and CICD, and here is more info about the deployment with the PS script.

Regular Visitor

Hello Microsoft team,


I have datafactory for Development environment. In this datafactory both collaborator and publish branch are kept as master.

I created a feature branch from master and added a new ADF pipeline with one set variable activity in this feature branch. After this generated the arm template using the npm utility.

The ARM template got generated successfully. I also verfiied that my new ADF pipeline is present in ARMTemplateForFactory.json.


However are performing the deployment via release pipeline using the above generated ARM template.

I am not able to find the new ADF pipeline in my DataFactory.


Can you tell me where I may be doing something wrong.

Let me know if you need more information.


Regular Visitor

Hi @Abhishek Narain @CesarBerard 

Can you help me in resolving my above query ?

Regular Visitor

Hi @Abhishek Narain @CesarBerard,


My issue is resolved.

Regular Visitor

Hi @Abhishek Narain @CesarBerard,

Since now we are able to create ARM template from this npm package, can you tell me if it is possible to disable the manually publish from Data Factory ? If yes then can you help with some pointers.


Thanks in advance !!

Occasional Visitor

Hi, is there a way of incorporating this new functionality into an Azure DevOps release pipeline? The example above and in the documentation show how to do this on the commandline, but I would like to build this step into my DevOps YAML releases.

Regular Visitor
Occasional Visitor

This has been an issue for us for a long long time. As a result we decided on a different approach and started using a slick tool based on json file deployment. It works great.


Check it out at


Occasional Visitor

Hi @Abhishek Narain ,

Something is not clear for me.

We want to use this new package to validate and test the factory before the change is merged in the collaboration branch.

So: PR to collaboration branch -> build artefact produced -> deployment back in DEV and tests -> PR approved and merged -> build artefact produced for release to int and prod.

Is this correct? I understood from the documentation that this method requires this DEV deployment (which before was not needed), as a replacement of the Publish button.

How this deployment back in dev works since dev has also git integration enabled? 

I have tried it, deployment back in dev works, but I can't identify my changes in the Factory UI. There are on the feature branch and not in collaboration branch (of course, since merge did not happen at this time), but where the changes are applied after deploying back to dev?

Thank you in advance!

Occasional Visitor

Hi ,


I have only one instance of ADF and having different folders like Dev, QA, Pre-Prod , Prod.


whatever pipelines I created in Dev , Can I still use CI/CD to create the same in QA folder




Occasional Visitor

Hi @Abhishek Narain,


This is a great utility to avoid the manual publish which has been a pan for more than an year for our team. However, this doesn't seem to be supported for the data factory pipelines in the synapse.


Could we expected the support for Synapse anytime soon?

Version history
Last update:
‎Feb 07 2021 10:41 PM
Updated by: