Purpose of this Blog
Developer Workflow(CI/CD)
Integration of Code from Data Factory UI(Continuous Integration)
1. A sandbox Data Factory is created for development of data pipelines with Datasets and Linked Services.
The Data Factory is configured with Azure Dev-ops Git.(Collaboration and publish branch) and the root folder where the data factory code is committed
2. A feature branch is created based on the main/collaboration branch for development. The branch in the Data Factory UI is changed to feature branch.
3. After the developer tests the pipelines and is satisfied with the changes , a pull request is raised to merge
the changes with the main/collaboration branch
4. After the PR gets approved by the concerned product leads, the changes made in feature branch are
merged into main branch.
5. The branch is changed to main branch in Data Factory UI . The changes are published to main branch by
clicking on publish button.
6. The changes are reflected in ARM template located in adf_publish branch
Note: If some parameters are missing in the arm template follow the steps below :
I. then we should go to manage in ADF UI
II. click on the ARM template
III. click on Edit parameter configuration in the DataFactory UI and include parameters that were missing in the ARM template in the json configuration file.
IV. We need to include this json configuration file in the main branch at root folder configured in Git configuration, let us consider we keep it in 'src/DataFactory/{DataFactoryName}/arm-template-parameters-definition.json'
- Follow this documentation on how to use the use custom parameters with the Resource Manager template : Custom parameters in a Resource Manager template - Azure Data Factory | Microsoft Docs
7. A Release pipeline is built in YAML to deploy the Data Factory using ARM template(We need to give
location of ARM template file) to Dev ,QA ,UAT and Production environments
Automated publish of ADF ARM Template :
YAML Release pipeline(Continuous Deployment)
stages:
- stage: Release
displayName: Release stage
jobs:
- job: Release
displayName: Release job
pool:
vmImage: 'Windows-2019'
Trigger:
trigger:
branches:
include:
- adf_publish
Resources:
resources:
repositories:
- repository: <repo name>
type: git
name: <repo name>
ref: adf_publish
Steps:
1. Deployment Variables (variable declared)
#Deployment variables
variables:
KeyVaultName: <keyvaultname>
azureSubscription: <ServiceConnection>
SourceDataFactoryName : <Source Data Factory name from which code is published>
DeployDataFactoryName : <Target Data Factory Name>
DeploymentResourceGroupName : <Target Resource Group Name>
2. Keyvault task to fetch the secrets(parameters to override ARM paramters) (PowerShell Task):
- task: AzureKeyVault@1
inputs:
azureSubscription: '$(azureSubscription)'
KeyVaultName: $(KeyVaultName)
SecretsFilter: '<secrets needed for overriding ARM parameters>'
RunAsPreJob: true
- checkout: <Repo Name>
3. Stopping the triggers of Target Data Factory before Deployment(Powershell Task)
Script:
- task: AzurePowerShell@5
displayName: Stop Triggers
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline:
$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName
"$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger –
ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName
"$(DeployDataFactoryName)" -Name $_.name -Force }
azurePowerShellVersion: 'LatestVersion'
4. Deploying the ARM template to the Target Data Factory by passing some values to the parameters in
Incremental Mode. (Powershell Task)
Script:
- task: AzurePowerShell@5
displayName: Deploy ADF
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline:
'New-AzResourceGroupDeployment
-ResourceGroupName "$(DeploymentResourceGroupName)"
-TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json"
-TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json"
-factoryName "$(DeployDataFactoryName)"
#<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets
#<parameter-overridden> : <value-to-be-overridden>
-Mode "Incremental"'
azurePowerShellVersion: 'LatestVersion'
5. Starting the Triggers after Successful Deployment(Powershell Task)
Script:
- task: AzurePowerShell@5
displayName: Restart
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline:
$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
$triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force }
azurePowerShellVersion: 'LatestVersion'
Full ADF CD Template:
name: Release-$(rev:r)
trigger:
branches:
include:
- adf_publish
resources:
repositories:
- repository: <repo name>
type: git
name: <repo name>
ref: adf_publish
variables:
KeyVaultName: <keyvaultname>
azureSubscription: <ServiceConnection>
SourceDataFactoryName : <Source Data Factory name from which code is published>
DeployDataFactoryName : <Target Data Factory Name>
DeploymentResourceGroupName : <Target Resource Group Name>
stages:
- stage: Release
displayName: Release stage
jobs:
- job: Release
displayName: Release job
pool:
vmImage: 'Windows-2019'
steps:
- task: AzureKeyVault@1
inputs:
azureSubscription: '$(azureSubscription)'
KeyVaultName: $(KeyVaultName)
SecretsFilter: '<secrets needed for overriding ARM parameters>'
RunAsPreJob: true
- checkout: <repo name>
- task: AzurePowerShell@5
displayName: Stop Triggers
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline:
$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName
"$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
$triggersADF | ForEach-Object { Stop-AzDataFactoryV2Trigger –
ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName
"$(DeployDataFactoryName)" -Name $_.name -Force }
azurePowerShellVersion: 'LatestVersion'
- task: AzurePowerShell@5
displayName: Deploy ADF
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline:
'New-AzResourceGroupDeployment
-ResourceGroupName "$(DeploymentResourceGroupName)"
-TemplateParameterFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateParametersForFactory.json"
-TemplateFile "$(System.DefaultWorkingDirectory)/$(SourceDataFactoryName)/ARMTemplateForFactory.json"
-factoryName "$(DeployDataFactoryName)"
#<parameter-overridden> : <value-to-be-overridden> there are parameters in arm template and overriden by key vault secrets
#<parameter-overridden> : <value-to-be-overridden>
-Mode "Incremental"'
azurePowerShellVersion: 'LatestVersion'
- task: AzurePowerShell@5
displayName: Restart Triggers
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
Inline:
$triggersADF = Get-AzDataFactoryV2Trigger -DataFactoryName "$(DeployDataFactoryName)" -ResourceGroupName "$(DeploymentResourceGroupName)";
$triggersADF | ForEach-Object { Start-AzDataFactoryV2Trigger -ResourceGroupName "$(DeploymentResourceGroupName)" -DataFactoryName "$(DeployDataFactoryName)" -Name $_.name -Force }
azurePowerShellVersion: 'LatestVersion'
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.