Azure Data Factory CI/CD with Github Actions - Overriding Global Parameters

Copper Contributor

After a lot of research on the internet I haven't yet found a way to use Github actions in a way that replicates the CD on Azure devOps.

When using azure devOps its possible to configure git on the dev ADF instance and then using azure-pipelines you can adhere to the CI/CD lifecycle stated here When using github actions, promoting to the TEST environment is straight forward and I can also easily override the global parameters from the workflow file. However, the complication arises when I want to to publish to the PROD environment from the TEST environment. Since we are not supposed to configure git on the TEST ADF instance, I can not include the global parameters in the ARM template which I can the override in then publish workflow. Below is my deploy workflows which promotes artefacts from dev to test:

 

name: deploy-main

on:
  push:
    branches:
    - main

permissions:
      id-token: write
      contents: read

jobs:
  build-dev:
    runs-on: ubuntu-latest
    environment:  dev
    steps:

    - uses: actions/checkout@v3
# Installs Node and the npm packages saved in your package.json file in the build
    - name: Setup Node.js environment
      uses: actions/setup-node@v3.4.1
      with:
        node-version: 14.x
        
    - name: install ADF Utilities package
      run: npm install
      working-directory: ./  # (1) provide the folder location of the package.json file

    - name: Validate and Generate ARM template
      run: npm run build export ${{github.workspace}}/ /subscriptions/${{ secrets.AZURE_SUBSCRIPTION_ID }}/resourceGroups/${{ secrets.AZURE_RESOURCE_GROUP }}/providers/Microsoft.DataFactory/factories/${{ secrets.ADF_NAME }} "ExportedArmTemplate"  # (3) The build command, as validate, needs the root folder location of your repository where all the objects are stored. And the 2nd parameter is the resourceID of the ADF instance. The 3rd parameter is the exported ARM template artifact name 
      working-directory: ./
 
# In order to leverage the artifact in another job, we need to upload it with the upload action 
    - name: upload artifact
      uses: actions/upload-artifact@v3
      with:
        name: ExportedArmTemplate # (4) use the same artifact name you used in the previous export step
        path: ./ExportedArmTemplate
        
  deploy-test:
    needs: build-dev
    runs-on: ubuntu-latest
    environment:  test
    steps:
    
 # we 1st download the previously uploaded artifact so we can leverage it later in the release job     
      - name: Download a Build Artifact
        uses: actions/download-artifact@v3.0.2
        with:
          name: ExportedArmTemplate # (5) Artifact name 


      - name: Login via Az module
        uses: azure/login@v1
        with:
          creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.AZURE_TENANT_ID }}"}'
          enable-AzPSSession: true 

      - name: data-factory-deploy
        uses: Azure/data-factory-deploy-action@v1.2.0
        with:
          resourceGroupName: ${{ secrets.AZURE_RESOURCE_GROUP }} # (6) your target ADF resource group name
          dataFactoryName: ${{ secrets.ADF_NAME }} # (7) your target ADF name
          armTemplateFile:  ARMTemplateForFactory.json # (8) ARM template file name ARMTemplateForFactory.json
          armTemplateParametersFile: ARMTemplateParametersForFactory.json # (9) ARM template parameters file name ARMTemplateParametersForFactory.json
          additionalParameters: default_properties_environment_value=tst default_properties_DZSPID_value=${{ secrets.DZSPID }} default_properties_PubSPID_value=${{ secrets.PUBSPID }} # (10) Parameters which will be replaced in the ARM template. Expected format 'key1=value key2=value keyN=value'. At the minimum here you should provide the target ADF name parameter. Check the ARMTemplateParametersForFactory.json file for all the parameters that are expected in your scenario
         

You can see on the last line that I am overriding the global parameters that were defined on the dev ADF instance to values that are needed in the test ADF instance. Below is the release workflow

 

name: deploy-release
# Controls when the workflow will run on release action
on:
  release:
    types: [published]

permissions:
      id-token: write
      contents: read

jobs:
  build-test:
    runs-on: ubuntu-latest
    environment:  test
    steps:

    - uses: actions/checkout@v3
# Installs Node and the npm packages saved in your package.json file in the build
    - name: Setup Node.js environment
      uses: actions/setup-node@v3.4.1
      with:
        node-version: 14.x
        
    - name: install ADF Utilities package
      run: npm install
      working-directory: ./  # (1) provide the folder location of the package.json file

    - name: Validate and Generate ARM template
      run: npm run build export ${{github.workspace}}/ /subscriptions/${{ secrets.AZURE_SUBSCRIPTION_ID }}/resourceGroups/${{ secrets.AZURE_RESOURCE_GROUP }}/providers/Microsoft.DataFactory/factories/${{ secrets.ADF_NAME }} "ExportedArmTemplate"  # (3) The build command, as validate, needs the root folder location of your repository where all the objects are stored. And the 2nd parameter is the resourceID of the ADF instance. The 3rd parameter is the exported ARM template artifact name 
      working-directory: ./
 
# In order to leverage the artifact in another job, we need to upload it with the upload action 
    - name: upload artifact
      uses: actions/upload-artifact@v3
      with:
        name: ExportedArmTemplate # (4) use the same artifact name you used in the previous export step
        path: ./ExportedArmTemplate
        
  release-prod:
    needs: build-test
    runs-on: ubuntu-latest
    environment:  prod
    steps:
    
 # we 1st download the previously uploaded artifact so we can leverage it later in the release job     
      - name: Download a Build Artifact
        uses: actions/download-artifact@v3.0.2
        with:
          name: ExportedArmTemplate # (5) Artifact name 


      - name: Login via Az module
        uses: azure/login@v1
        with:
          creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.AZURE_TENANT_ID }}"}'
        enable-AzPSSession: true 

      - name: data-factory-deploy
        uses: Azure/data-factory-deploy-action@v1.2.0
        with:
          resourceGroupName: ${{ secrets.AZURE_RESOURCE_GROUP }} # (6) your target ADF resource group name
          dataFactoryName: ${{ secrets.ADF_NAME }} # (7) your target ADF name
          armTemplateFile:  ARMTemplateForFactory.json # (8) ARM template file name ARMTemplateForFactory.json
          armTemplateParametersFile: ARMTemplateParametersForFactory.json # (9) ARM template parameters file name ARMTemplateParametersForFactory.json
          additionalParameters: default_properties_environment_value=prd default_properties_DZSPID_value=${{ secrets.DZSPID }} default_properties_PubSPID_value=${{ secrets.PUBSPID }} # (10) Parameters which will be replaced in the ARM template. Expected format 'key1=value key2=value keyN=value'. At the minimum here you should provide the target ADF name parameter. Check the ARMTemplateParametersForFactory.json file for all the parameters that are expected in your scenario
         

The release workflow is almost identical to the deploy workflow. The only difference is its triggered by publishing code using github release management. The issue arises on trying to override the global parameters to provide values that are need in prod. Since the test environment does not have git configured, I cannot include the global parameters in the arm template. I have seen other solutions on which they simultaneously deploy to both test and prod on PR approval. This then goes against the best practice on the CI/CD lifecycle because I need to ability to only promote code the the prod ADF instances only when UATs are completed. Any ideas on how I can accomplish this? 

0 Replies