Azure Data Factory CI-CD using YAML template

Published Feb 22 2022 01:47 PM 2,185 Views
Microsoft

Purpose of this Blog

  • Defined  the end to end Git integration and  Deployment flow using YAML template from a project implementation perspective
  • This can be easily implemented as you just need to include this YAML template and do a few changes as this has been successfully implemented  for Deployments across some projects.

Developer Workflow(CI/CD)

Integration of Code from Data Factory UI(Continuous Integration)


1. A sandbox Data Factory is created for development of data pipelines with Datasets and Linked Services.
The Data Factory is configured with Azure Dev-ops Git.(Collaboration and publish branch) and the root folder where the data factory code is committed

Akshay_Attota_0-1643924075912.png


2. A feature branch is created based on the main/collaboration branch for development. The branch in the Data Factory UI is changed to feature branch.

Akshay_Attota_1-1643924115116.png


3. After the developer tests the pipelines and is satisfied with the changes , a pull request is raised to merge
the changes with the main/collaboration branch

Akshay_Attota_2-1643924275798.png

 

4. After the PR gets approved by the concerned product leads, the changes made in feature branch are
merged into main branch.

Akshay_Attota_3-1643924335774.png


5. The branch is changed to main branch in Data Factory UI . The changes are published to main branch by
clicking on publish button.

Akshay_Attota_9-1643925485472.png


6. The changes are reflected in ARM template located in adf_publish branch

Akshay_Attota_8-1643925443625.png


Note: If some parameters are missing in the arm template follow the steps below :

I. then we should go to manage Akshay_Attota_7-1643925098371.png in ADF UI

II. click on  the ARM template

III. click on Edit parameter configuration in the DataFactory UI and include parameters that were missing in the ARM template in the json configuration file.

IV. We need to include this json configuration file in the main branch at root folder configured in Git configuration, let us consider we keep it in  'src/DataFactory/{DataFactoryName}/arm-template-parameters-definition.json'

 

- Follow this documentation on how to use the use custom parameters with the Resource Manager template : Custom parameters in a Resource Manager template - Azure Data Factory | Microsoft Docs 

 

Akshay_Attota_5-1643924912773.png


7. A Release pipeline is built in YAML to deploy the Data Factory using ARM template(We need to give
location of ARM template file) to Dev ,QA ,UAT and Production environments

 

Automated publish of ADF ARM Template :

YAML Release pipeline(Continuous Deployment)

  • Run the release pipeline for the specified target environment.
  • This will download the previously generated ARM template. It will also download secure connection strings
    from Azure Key-Vault. .Then it will deploy to your
    Target Data Factory using ARM template deployment.
  • The code below shows how to run your release agent on a Microsoft hosted agent

 

 

 

 

stages:
- stage: Release
  displayName: Release stage
  jobs:
    - job: Release
      displayName: Release job
      pool:
        vmImage: 'Windows-2019'

 

 

 

 

 


Trigger:

  • This Pipeline will be triggered based on the changes made in  adf_publish branch

 

 

 

 

trigger:
  branches:
    include:
    - adf_publish

 

 

 

 


Resources:

  • The branch where the ARM template is located is mentioned.
    Referring to the previous point where it is mentioned that the ARM template will be picked from adf_publish (Publish) branch
  • Example:

 

 

 

 

resources:
  repositories:
  - repository: <repo name>
    type: git
    name: <repo name>
    ref: adf_publish

 

 

 

 

 

Steps:
1. Deployment Variables (variable declared)

 

 

 

 

 #Deployment variables
variables:
  KeyVaultName: <keyvaultname>
  azureSubscription: <ServiceConnection>
  SourceDataFactoryName : <Source Data Factory name from which code is published>
  DeployDataFactoryName : <Target Data Factory Name>
  DeploymentResourceGroupName : <Target Resource Group Name>

 

 

 

 


2. Keyvault task to fetch the secrets(parameters to override ARM paramters) (PowerShell Task):

 

 

 

 

   - task: AzureKeyVault@1
     inputs:
       azureSubscription: '$(azureSubscription)'
       KeyVaultName: $(KeyVaultName)
       SecretsFilter: '<secrets needed for overriding ARM parameters>'
       RunAsPreJob: true

   - checkout: <Repo Name>

 

 

 

 

 

3. Stopping the triggers of Target Data Factory before Deployment(Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5
  displayName: Stop Triggers
  inputs:
    azureSubscription: '$(azureSubscription)'
    ScriptType: 'InlineScript'
    Inline: 
      $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName   
      "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
      $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger –
      ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName   
      "$(DeployDataFactoryName)" -Name $_.name -Force }
    azurePowerShellVersion: 'LatestVersion'

 

 

 

 


4. Deploying the ARM template to the Target Data Factory by passing some values to the parameters in
Incremental Mode. (Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5
  displayName: Deploy ADF
  inputs:
    azureSubscription: '$(azureSubscription)'
    ScriptType: 'InlineScript'
    Inline: 
      'New-AzResourceGroupDeployment 
      -ResourceGroupName "$(DeploymentResourceGroupName)" 
      -TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json" 
      -TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json"
      -factoryName "$(DeployDataFactoryName)"
      #<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets
      #<parameter-overridden> : <value-to-be-overridden>
      -Mode "Incremental"'
    azurePowerShellVersion: 'LatestVersion'

 

 

 

 

 

5. Starting the Triggers after Successful Deployment(Powershell Task)
Script:

 

 

 

 

- task: AzurePowerShell@5
  displayName: Restart
  inputs:
    azureSubscription: '$(azureSubscription)'
    ScriptType: 'InlineScript'
    Inline: 
      $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName   "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
      $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force }
    azurePowerShellVersion: 'LatestVersion'

 

 

 

 

 

Full ADF CD Template:

 

 

 

 

 

name: Release-$(rev:r)

trigger:
  branches:
    include:
    - adf_publish

resources:
  repositories:
  - repository: <repo name>
    type: git
    name: <repo name>
    ref: adf_publish


variables:
  KeyVaultName: <keyvaultname>
  azureSubscription: <ServiceConnection>
  SourceDataFactoryName : <Source Data Factory name from which code is published>
  DeployDataFactoryName : <Target Data Factory Name>
  DeploymentResourceGroupName : <Target Resource Group Name>

stages:
- stage: Release
  displayName: Release stage
  jobs:
    - job: Release
      displayName: Release job
      pool:
        vmImage: 'Windows-2019'
      steps:
        - task: AzureKeyVault@1
          inputs:
            azureSubscription: '$(azureSubscription)'
            KeyVaultName: $(KeyVaultName)
            SecretsFilter: '<secrets needed for overriding ARM parameters>'
            RunAsPreJob: true

        - checkout: <repo name>

        - task: AzurePowerShell@5
          displayName: Stop Triggers
          inputs:
            azureSubscription: '$(azureSubscription)'
            ScriptType: 'InlineScript'
            Inline: 
              $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName   
              "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
              $triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger –
              ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName   
              "$(DeployDataFactoryName)" -Name $_.name -Force }
            azurePowerShellVersion: 'LatestVersion'
        - task: AzurePowerShell@5
          displayName: Deploy ADF
          inputs:
            azureSubscription: '$(azureSubscription)'
            ScriptType: 'InlineScript'
            Inline: 
              'New-AzResourceGroupDeployment 
              -ResourceGroupName "$(DeploymentResourceGroupName)" 
              -TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json" 
              -TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json"
              -factoryName "$(DeployDataFactoryName)"
              #<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets
              #<parameter-overridden> : <value-to-be-overridden>
              -Mode "Incremental"'
            azurePowerShellVersion: 'LatestVersion'
        - task: AzurePowerShell@5
          displayName: Restart Triggers
          inputs:
            azureSubscription: '$(azureSubscription)'
            ScriptType: 'InlineScript'
            Inline: 
              $triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName   "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
              $triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force }
            azurePowerShellVersion: 'LatestVersion'
       
  

 

 

 

 

 

 

 

 

Version history
Last update:
‎Mar 17 2022 03:53 AM
Updated by: