CI CD in Azure Synapse Analytics Part 3
Published Dec 18 2020 04:45 AM 15.2K Views

Here's a quick review of the road so far:


CI CD in Azure Synapse Analytics Part 1

  • Creating an Azure DevOps project
  • Linking our Azure Synapse Analytics environment to that Project via Git
  • Validating that our Azure DevOps Repo was populated with our Azure Synapse Analytics environment

CI CD in Azure Synapse Analytics Part 2:

  • Create a new branch on our Repo
  • Edit our Azure Synapse Analytics environment
    • Specifically my SQL scripts have demos all over the place and Buck Woody said I have to clean up my very messy room .... Azure Synapse Analytics environment
  • Create a Pull Request in Azure Synapse Analytics to merge our new branch with the main
  • Approve the Pull Request in Azure DevOps
  • Validate our main branch is updated in our Azure Synapse Analytics Environment

This time we will:

  • Create an Artifact pipeline
    • This is to create an Artifact we can use to deploy to another environment


First we are going to examine a very important part of our Azure Synapse Analytics environment.  The Publish button.


01 CI CD part 3.png


Why all the arrows and boxes?  Because this is important.  This publish button saves the templates that we will use to deploy our environment to another Azure Synapse Analytics workspace.  When you click publish a few messages should appear.  Publishing In progress, Publishing completed, Generating templates, and Generating templates completed.  


If you get an error do not fear, validation will occur and show you where the error is in your workspace.  I've encountered this a time or two.  Eventually I will intentionally write a blog in this series where we break things just to fixt them.  For now, let us presume that everything went just fine.


01 CI CD part 3 image 2.png

Next we will move over to our Azure DevOps Repo.  Find the folder that is the same name as your Azure Synapse Analytics workspace.  In this picture mine is bballasw.  Under that folder you will find two files, TemplateForWorkspace.json & TemplateParametersForWorkspace.json.


*NOTE - these templates are not the same templates you would use to deploy a new environment.  These are only for deploying the artifacts from one environment to another.  In part 5 we will look at generating the ARM templates needed for deploying a new environment from Azure DevOps.


01 CI CD part 3 image 3.png


We will be using these files to create our artifact build pipeline.  Also highlighted is the WorkspaceDefaultSqlServer_connectionString, this string is of the type secureString.  This is important when we reach our release pipeline in Part 4, if we do not handle this string properly the release will fail.


For this exercise I've created another Azure Synapse Analytics Environment for us to deploy to named bballaswqa in a separate resource group from bballasw.  


01 CI CD part 3 image 4.png

Right now there is nothing in bballaswqa.


01 CI CD part 3 image 5.png

Especially compared to bballasw.


01 CI CD part 3 image 6.png


With our destination of bballaswqa in mind, we begin with our build pipeline.  Moving over to Azure DevOps we want to move to Pipelines.


01 CI CD part 3 image 7.png

Click New pipeline.


01 CI CD part 3 image 8.png


At the very bottom of the page, in super tiny font you will find Use the classic editor.  Make sure to click on that lin.


01 CI CD part 3 image 9.png


This is where we configure the project, repository, and the default branch for our builds.  All of this information is correct.  We are using Azure Repos Git, we will click the Continue button.


01 CI CD part 3 image 10.png

At the very top of our next page we have different options for our pipeline template.  We will click on the Empty job link.


01 CI CD part 3 image 11.png


Now we are finally to the pipeline.  First we will rename the default name.  I use ASW in my naming convention as it stands for Azure Synapse Workspace.  We rename the pipeline to ASW Build Pipeline.  Then we click on the Triggers section.  The Triggers section is where we will configure our CI portion of our build. 


01 CI CD part 3 image 12.png


Check the Enable continuous integration check box.  Under branch filters we want main, as that is the branch we are publishing to for all of our builds.  But we need to add a Path filter.  Ever time we merge a branch to main it would cause the build pipeline to run.  I only want the pipeline to run when we publish from our Azure Synapse Analytics environment.


After this return to the tasks window.


01 CI CD part 3 image 13.png


On the Agent job 1 click the + button. We need the Copy files task.  Add that to our pipeline.


01 CI CD part 3 image 14.png


In Azure DevOps there are some reserved variables.  One of the Build variables is Build.ArtifactStagingDirectory, for a full list see this Docs article Predefined variables - Azure Pipelines | Microsoft Docs. This defines a local path on a build agent and this is where we want our build files deployed to.  In part 4 we will build a Release Pipeline and that Release will be linked to the artifacts we produce.  


We will change our Display name to Copy ARM Template Files to: $(Build.ArtifactStagingDirectory).  When you use a variable in Azure DevOps you invoke it with $(variableName).  Now click on the three ellipsis next to the Source Folder text box.


01 CI CD part 3 image 15.png


We will select the folder that has the same name as our Azure Synapse Analytics workspace that has our Template JSON files.


01 CI CD part 3 image 16.png

We finish configuring this task by setting our contents to *.json.  This will pull in only the JSON files under our folder.  We set the Target Folder to $(Build.ArtifactsStagingDirectory)/ARM.


We don't need the folder path.  At this time there are no other object in my build pipeline.  If we ever want to add them we can have additional subfolders, but for now my OCD won and I created a folder.


01 CI CD part 3 image 17.png


Now click on the + sign on Agent Job 1.  We need to add the Publish Pipeline Artifacts task.  


01 CI CD part 3 image 18.png


Our File or Directory path will be $(Build.ArtifactsStagingDirectory).  Our Artifact name will be ASW_Drop. 


Hit Save & queue.


01 CI CD part 3 image 19.png


Enter a save comment, click Save and run. 


01 CI CD part 3 image 20.png


Click on the Agent job 1 section of the page and open the build agent window.



01 CI CD part 3 image 21.png

OH WOW!  IT ALL TURNED GREEN AND WORKED!!!'s not like I did this a few 100 times failing miserably until I figured it out......  Now click on the small arrow next to the Jobs run to return to the pipeline.


01 CI CD part 3 image 22.png


Under the header Related 0 work items click on 1 published; 1 consumed. 


01 CI CD part 3 image 23.png


Expand the arrow next to ASW_Drop, ARM, and we can see our Template files.  Success we have a build artifact that we can now call in a Release pipeline.


Ok Dear Reader, it's late and we are done for today.  In our next blog we will cover the release pipeline and look at what was deployed to our QA environment!


As always, thank you for stopping by.



Version history
Last update:
‎Dec 18 2020 04:45 AM