An automation and templating question with Azure Pipelines

Copper Contributor

Hello, we are working on a project where we have to automate resources across different environments. We are using Azure Pipelines

 

What we have:

 

An original environment where we would publish straight from the collaboration branch to workspace_publish. This env contains:

- A Synapse workspace (Dedicated SQL Pool)
- Azure Functions used in the Synapse pipelines through Linked Services and also one function that works as an API and should be callable always.
- Blobstorage (StorageV2)
- Key vault
- Logic Apps (sending emails)

- A environment deploy pipeline in DevOps. 

 

What we want:

- An automated way of deploying the Synapse Workspace, Linked Services, Pipelines, Stored Procedures and tables from the dev environment to the others (qa, stage, prod)
- Templating of connection strings, function URLs, linked services so that all connections work in each environment and any other interaction with resources outside Synapse.
- Definition of the SQL Pool to run Stored Procedures
- Database migration: If we create a table/column in dev how can we push it to prod without affecting the data?
- Templating variables and resources in Azure Functions
- Template permissions/access, specifically between Azure Functions and Database, Blob,

 

How far we've gotten: Following this tutorial: https://www.youtube.com/watch?v=d7wsEh8Vr34

- We have tried to expose all the parameters that need templating using a template-parameters-definition.json file placed in the main branch of the repo (CICD-Test).
- Then using override parameters in the Release pipeline in DevOps change the parameters to point at the correct resources for example:
   Changing the blobstorage connection string from AccountName=stgnkdwdatalakedev to AccountName=stgnkdwdatalakeqa

 

How can we proceed with this?

 

Since we do a lot of adhoc analysis in the DW and also with the live data coming from the event hub we require at least the dev and prod environments to be running.

 

0 Replies