Forum Discussion
An automation and templating question with Azure Pipelines
Hello, we are working on a project where we have to automate resources across different environments. We are using Azure Pipelines
What we have:
An original environment where we would publish straight from the collaboration branch to workspace_publish. This env contains:
- A Synapse workspace (Dedicated SQL Pool)
- Azure Functions used in the Synapse pipelines through Linked Services and also one function that works as an API and should be callable always.
- Blobstorage (StorageV2)
- Key vault
- Logic Apps (sending emails)
- A environment deploy pipeline in DevOps.
What we want:
- An automated way of deploying the Synapse Workspace, Linked Services, Pipelines, Stored Procedures and tables from the dev environment to the others (qa, stage, prod)
- Templating of connection strings, function URLs, linked services so that all connections work in each environment and any other interaction with resources outside Synapse.
- Definition of the SQL Pool to run Stored Procedures
- Database migration: If we create a table/column in dev how can we push it to prod without affecting the data?
- Templating variables and resources in Azure Functions
- Template permissions/access, specifically between Azure Functions and Database, Blob,
How far we've gotten: Following this tutorial: https://www.youtube.com/watch?v=d7wsEh8Vr34
- We have tried to expose all the parameters that need templating using a template-parameters-definition.json file placed in the main branch of the repo (CICD-Test).
- Then using override parameters in the Release pipeline in DevOps change the parameters to point at the correct resources for example:
Changing the blobstorage connection string from AccountName=stgnkdwdatalakedev to AccountName=stgnkdwdatalakeqa
How can we proceed with this?
Since we do a lot of adhoc analysis in the DW and also with the live data coming from the event hub we require at least the dev and prod environments to be running.