An automation and templating question with Azure Pipelines
Hello, we are working on a project where we have to automate resources across different environments. We are using Azure Pipelines What we have: An original environment where we would publish straight from the collaboration branch to workspace_publish. This env contains: - A Synapse workspace (Dedicated SQL Pool) - Azure Functions used in the Synapse pipelines through Linked Services and also one function that works as an API and should be callable always. - Blobstorage (StorageV2) - Key vault - Logic Apps (sending emails) - A environment deploy pipeline in DevOps. What we want: - An automated way of deploying the Synapse Workspace, Linked Services, Pipelines, Stored Procedures and tables from the dev environment to the others (qa, stage, prod) - Templating of connection strings, function URLs, linked services so that all connections work in each environment and any other interaction with resources outside Synapse. - Definition of the SQL Pool to run Stored Procedures - Database migration: If we create a table/column in dev how can we push it to prod without affecting the data? - Templating variables and resources in Azure Functions - Template permissions/access, specifically between Azure Functions and Database, Blob, How far we've gotten: Following this tutorial: https://www.youtube.com/watch?v=d7wsEh8Vr34 - We have tried to expose all the parameters that need templating using a template-parameters-definition.json file placed in the main branch of the repo (CICD-Test). - Then using override parameters in the Release pipeline in DevOps change the parameters to point at the correct resources for example: Changing the blobstorage connection string from AccountName=stgnkdwdatalakedev to AccountName=stgnkdwdatalakeqa How can we proceed with this? Since we do a lot of adhoc analysis in the DW and also with the live data coming from the event hub we require at least the dev and prod environments to be running.