Forum Discussion

zmkarakas's avatar
zmkarakas
Copper Contributor
Jul 03, 2023

An automation and templating question with Azure Pipelines

Hello, we are working on a project where we have to automate resources across different environments. We are using Azure Pipelines

 

What we have:

 

An original environment where we would publish straight from the collaboration branch to workspace_publish. This env contains:

- A Synapse workspace (Dedicated SQL Pool)
- Azure Functions used in the Synapse pipelines through Linked Services and also one function that works as an API and should be callable always.
- Blobstorage (StorageV2)
- Key vault
- Logic Apps (sending emails)

- A environment deploy pipeline in DevOps. 

 

What we want:

- An automated way of deploying the Synapse Workspace, Linked Services, Pipelines, Stored Procedures and tables from the dev environment to the others (qa, stage, prod)
- Templating of connection strings, function URLs, linked services so that all connections work in each environment and any other interaction with resources outside Synapse.
- Definition of the SQL Pool to run Stored Procedures
- Database migration: If we create a table/column in dev how can we push it to prod without affecting the data?
- Templating variables and resources in Azure Functions
- Template permissions/access, specifically between Azure Functions and Database, Blob,

 

How far we've gotten: Following this tutorial: https://www.youtube.com/watch?v=d7wsEh8Vr34

- We have tried to expose all the parameters that need templating using a template-parameters-definition.json file placed in the main branch of the repo (CICD-Test).
- Then using override parameters in the Release pipeline in DevOps change the parameters to point at the correct resources for example:
   Changing the blobstorage connection string from AccountName=stgnkdwdatalakedev to AccountName=stgnkdwdatalakeqa

 

How can we proceed with this?

 

Since we do a lot of adhoc analysis in the DW and also with the live data coming from the event hub we require at least the dev and prod environments to be running.

 

No RepliesBe the first to reply

Resources