Forum Discussion
Azure Data Factory CI/CD with Github Actions - Overriding Global Parameters
After a lot of research on the internet I haven't yet found a way to use Github actions in a way that replicates the CD on Azure devOps.
When using azure devOps its possible to configure git on the dev ADF instance and then using azure-pipelines you can adhere to the CI/CD lifecycle stated https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-delivery When using github actions, promoting to the TEST environment is straight forward and I can also easily override the global parameters from the workflow file. However, the complication arises when I want to to publish to the PROD environment from the TEST environment. Since we are not supposed to configure git on the TEST ADF instance, I can not include the global parameters in the ARM template which I can the override in then publish workflow. Below is my deploy workflows which promotes artefacts from dev to test:
name: deploy-main
on:
push:
branches:
- main
permissions:
id-token: write
contents: read
jobs:
build-dev:
runs-on: ubuntu-latest
environment: dev
steps:
- uses: actions/checkout@v3
# Installs Node and the npm packages saved in your package.json file in the build
- name: Setup Node.js environment
uses: actions/setup-node@v3.4.1
with:
node-version: 14.x
- name: install ADF Utilities package
run: npm install
working-directory: ./ # (1) provide the folder location of the package.json file
- name: Validate and Generate ARM template
run: npm run build export ${{github.workspace}}/ /subscriptions/${{ secrets.AZURE_SUBSCRIPTION_ID }}/resourceGroups/${{ secrets.AZURE_RESOURCE_GROUP }}/providers/Microsoft.DataFactory/factories/${{ secrets.ADF_NAME }} "ExportedArmTemplate" # (3) The build command, as validate, needs the root folder location of your repository where all the objects are stored. And the 2nd parameter is the resourceID of the ADF instance. The 3rd parameter is the exported ARM template artifact name
working-directory: ./
# In order to leverage the artifact in another job, we need to upload it with the upload action
- name: upload artifact
uses: actions/upload-artifact@v3
with:
name: ExportedArmTemplate # (4) use the same artifact name you used in the previous export step
path: ./ExportedArmTemplate
deploy-test:
needs: build-dev
runs-on: ubuntu-latest
environment: test
steps:
# we 1st download the previously uploaded artifact so we can leverage it later in the release job
- name: Download a Build Artifact
uses: actions/download-artifact@v3.0.2
with:
name: ExportedArmTemplate # (5) Artifact name
- name: Login via Az module
uses: azure/login@v1
with:
creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.AZURE_TENANT_ID }}"}'
enable-AzPSSession: true
- name: data-factory-deploy
uses: Azure/data-factory-deploy-action@v1.2.0
with:
resourceGroupName: ${{ secrets.AZURE_RESOURCE_GROUP }} # (6) your target ADF resource group name
dataFactoryName: ${{ secrets.ADF_NAME }} # (7) your target ADF name
armTemplateFile: ARMTemplateForFactory.json # (8) ARM template file name ARMTemplateForFactory.json
armTemplateParametersFile: ARMTemplateParametersForFactory.json # (9) ARM template parameters file name ARMTemplateParametersForFactory.json
additionalParameters: default_properties_environment_value=tst default_properties_DZSPID_value=${{ secrets.DZSPID }} default_properties_PubSPID_value=${{ secrets.PUBSPID }} # (10) Parameters which will be replaced in the ARM template. Expected format 'key1=value key2=value keyN=value'. At the minimum here you should provide the target ADF name parameter. Check the ARMTemplateParametersForFactory.json file for all the parameters that are expected in your scenario
You can see on the last line that I am overriding the global parameters that were defined on the dev ADF instance to values that are needed in the test ADF instance. Below is the release workflow
name: deploy-release
# Controls when the workflow will run on release action
on:
release:
types: [published]
permissions:
id-token: write
contents: read
jobs:
build-test:
runs-on: ubuntu-latest
environment: test
steps:
- uses: actions/checkout@v3
# Installs Node and the npm packages saved in your package.json file in the build
- name: Setup Node.js environment
uses: actions/setup-node@v3.4.1
with:
node-version: 14.x
- name: install ADF Utilities package
run: npm install
working-directory: ./ # (1) provide the folder location of the package.json file
- name: Validate and Generate ARM template
run: npm run build export ${{github.workspace}}/ /subscriptions/${{ secrets.AZURE_SUBSCRIPTION_ID }}/resourceGroups/${{ secrets.AZURE_RESOURCE_GROUP }}/providers/Microsoft.DataFactory/factories/${{ secrets.ADF_NAME }} "ExportedArmTemplate" # (3) The build command, as validate, needs the root folder location of your repository where all the objects are stored. And the 2nd parameter is the resourceID of the ADF instance. The 3rd parameter is the exported ARM template artifact name
working-directory: ./
# In order to leverage the artifact in another job, we need to upload it with the upload action
- name: upload artifact
uses: actions/upload-artifact@v3
with:
name: ExportedArmTemplate # (4) use the same artifact name you used in the previous export step
path: ./ExportedArmTemplate
release-prod:
needs: build-test
runs-on: ubuntu-latest
environment: prod
steps:
# we 1st download the previously uploaded artifact so we can leverage it later in the release job
- name: Download a Build Artifact
uses: actions/download-artifact@v3.0.2
with:
name: ExportedArmTemplate # (5) Artifact name
- name: Login via Az module
uses: azure/login@v1
with:
creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.AZURE_CLIENT_SECRET }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.AZURE_TENANT_ID }}"}'
enable-AzPSSession: true
- name: data-factory-deploy
uses: Azure/data-factory-deploy-action@v1.2.0
with:
resourceGroupName: ${{ secrets.AZURE_RESOURCE_GROUP }} # (6) your target ADF resource group name
dataFactoryName: ${{ secrets.ADF_NAME }} # (7) your target ADF name
armTemplateFile: ARMTemplateForFactory.json # (8) ARM template file name ARMTemplateForFactory.json
armTemplateParametersFile: ARMTemplateParametersForFactory.json # (9) ARM template parameters file name ARMTemplateParametersForFactory.json
additionalParameters: default_properties_environment_value=prd default_properties_DZSPID_value=${{ secrets.DZSPID }} default_properties_PubSPID_value=${{ secrets.PUBSPID }} # (10) Parameters which will be replaced in the ARM template. Expected format 'key1=value key2=value keyN=value'. At the minimum here you should provide the target ADF name parameter. Check the ARMTemplateParametersForFactory.json file for all the parameters that are expected in your scenario
The release workflow is almost identical to the deploy workflow. The only difference is its triggered by publishing code using github release management. The issue arises on trying to override the global parameters to provide values that are need in prod. Since the test environment does not have git configured, I cannot include the global parameters in the arm template. I have seen other solutions on which they simultaneously deploy to both test and prod on PR approval. This then goes against the best practice on the CI/CD lifecycle because I need to ability to only promote code the the prod ADF instances only when UATs are completed. Any ideas on how I can accomplish this?