Automated publish improvement in ADF's CI/CD flow

Jason Kohlhoff That's fair feedback. It should be the customer's choice. The major problem we sorted with this is the automated 'publish' and 'validate', getting rid of the adf_publish branch dependency, which was updated on publish using the collaboration branch (it could be master or any other).
We will come up with better guidance on deploying feature/ dev branches across environments and the limitless possibilities customers can use the CI/CD.    

bcassell-MSFT We use a dev branch AND a master branch, and make dev branch the Collaborative branch in our dev Data Factory resource. We make feature branches off dev, and then PR to dev first. The release pipeline in Azure DevOps deploys to the Dev environment first.

I think the issue with the old guidance is that the CI/CD diagram made it look like you must only have a master branch, and it looks like the new guidance suffers the same shortsightenness. Hopefully, someone special on the DF team is reading this.

Is it possible to run validate all against a data factory in a feature branch?  Would be nice to block merges that won't pass validation before they get to master.

skammili 

About the resources not deleted:

Since the ARM template deployment is an incremental operation, the resources are not deleted by design. You need to use a Post deployment PS script in order to do so, this script is included in the output of the package (PrePostDeploymentScript.ps1). Here's more info about this script and how to implement it in your release definition.

Regarding the Git problem:

It is possible that you are including global parameters in the ARM template. Since global parameters are located in the factory entity, when they are added, the factory entity is added in the ARM template as well, however, the Git configuration is not. You can check the box to do not include them in the ARM template and use the Global Parameters post deployment script in order to update them in the target environment. This script is provided in the output of the package as well (GlobalParametersUpdateScript.ps1). Here's more info about global parameters and CICD, and here is more info about the deployment with the PS script.

Hi @Abhishek Narain @CesarBerard,

My issue is resolved.

Hi @Abhishek Narain,

This is a great utility to avoid the manual publish which has been a pan for more than an year for our team. However, this doesn't seem to be supported for the data factory pipelines in the synapse. 

https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment-improvements

Could we expected the support for Synapse anytime soon?

Hello all,

If anyone interested to build, validate and export arm templates automatically as a part of CI process - this is your friend:

https://marketplace.visualstudio.com/items?itemName=SQLPlayer.DataFactoryTools

One of the tasks uses the Microsoft NPM library behind the scenes and can validate your ADF:

Then you can use:

1) a standard approach to deployment arm_templates to the target ADF

2) or deploy from code (directly from branch) using Publish ADF task from the above Azure DevOps extension.

I'm the author of this #adftools. More info: https://sqlplayer.net/adftools/

To understand the differences between these two methods:

Two methods of deployment Azure Data Factory

AmitGhotikar having only one instance of ADF is definitely a bad idea and you ask for troubles.

You should have one ADF per environment. All objects should have the same names, only selected properties must be substituted.

Furthermore, DEV environment (with git configured) should be covered by the DevOps deployment process.

vdedkov Take a look at this page: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment-improvements#create-an-azure-pipeline

There is a whole YAML file which should be helpful.

Abhishek Narain , I've noticed that thus utility doesn't consider managedVirtualNetworks and managedPrivateEndpoints when it creates ARM templates. Because of this, I am unable to use this method to automate publishing. Do you have some solution or work around for this issue?