Here's a quick review of the road so far:
CI CD in Azure Synapse Analytics Part 1
First we are going to examine a very important part of our Azure Synapse Analytics environment. The Publish button.
Why all the arrows and boxes? Because this is important. This publish button saves the templates that we will use to deploy our environment to another Azure Synapse Analytics workspace. When you click publish a few messages should appear. Publishing In progress, Publishing completed, Generating templates, and Generating templates completed.
If you get an error do not fear, validation will occur and show you where the error is in your workspace. I've encountered this a time or two. Eventually I will intentionally write a blog in this series where we break things just to fixt them. For now, let us presume that everything went just fine.
Next we will move over to our Azure DevOps Repo. Find the folder that is the same name as your Azure Synapse Analytics workspace. In this picture mine is bballasw. Under that folder you will find two files, TemplateForWorkspace.json & TemplateParametersForWorkspace.json.
*NOTE - these templates are not the same templates you would use to deploy a new environment. These are only for deploying the artifacts from one environment to another. In part 5 we will look at generating the ARM templates needed for deploying a new environment from Azure DevOps.
We will be using these files to create our artifact build pipeline. Also highlighted is the WorkspaceDefaultSqlServer_connectionString, this string is of the type secureString. This is important when we reach our release pipeline in Part 4, if we do not handle this string properly the release will fail.
For this exercise I've created another Azure Synapse Analytics Environment for us to deploy to named bballaswqa in a separate resource group from bballasw.
Right now there is nothing in bballaswqa.
Especially compared to bballasw.
With our destination of bballaswqa in mind, we begin with our build pipeline. Moving over to Azure DevOps we want to move to Pipelines.
Click New pipeline.
At the very bottom of the page, in super tiny font you will find Use the classic editor. Make sure to click on that lin.
This is where we configure the project, repository, and the default branch for our builds. All of this information is correct. We are using Azure Repos Git, we will click the Continue button.
At the very top of our next page we have different options for our pipeline template. We will click on the Empty job link.
Now we are finally to the pipeline. First we will rename the default name. I use ASW in my naming convention as it stands for Azure Synapse Workspace. We rename the pipeline to ASW Build Pipeline. Then we click on the Triggers section. The Triggers section is where we will configure our CI portion of our build.
Check the Enable continuous integration check box. Under branch filters we want main, as that is the branch we are publishing to for all of our builds. But we need to add a Path filter. Ever time we merge a branch to main it would cause the build pipeline to run. I only want the pipeline to run when we publish from our Azure Synapse Analytics environment.
After this return to the tasks window.
On the Agent job 1 click the + button. We need the Copy files task. Add that to our pipeline.
In Azure DevOps there are some reserved variables. One of the Build variables is Build.ArtifactStagingDirectory, for a full list see this Docs article Predefined variables - Azure Pipelines | Microsoft Docs. This defines a local path on a build agent and this is where we want our build files deployed to. In part 4 we will build a Release Pipeline and that Release will be linked to the artifacts we produce.
We will change our Display name to Copy ARM Template Files to: $(Build.ArtifactStagingDirectory). When you use a variable in Azure DevOps you invoke it with $(variableName). Now click on the three ellipsis next to the Source Folder text box.
We will select the folder that has the same name as our Azure Synapse Analytics workspace that has our Template JSON files.
We finish configuring this task by setting our contents to *.json. This will pull in only the JSON files under our folder. We set the Target Folder to $(Build.ArtifactsStagingDirectory)/ARM.
We don't need the folder path. At this time there are no other object in my build pipeline. If we ever want to add them we can have additional subfolders, but for now my OCD won and I created a folder.
Now click on the + sign on Agent Job 1. We need to add the Publish Pipeline Artifacts task.
Our File or Directory path will be $(Build.ArtifactsStagingDirectory). Our Artifact name will be ASW_Drop.
Hit Save & queue.
Enter a save comment, click Save and run.
Click on the Agent job 1 section of the page and open the build agent window.
OH WOW! IT ALL TURNED GREEN AND WORKED!!! .....it's not like I did this a few 100 times failing miserably until I figured it out...... Now click on the small arrow next to the Jobs run to return to the pipeline.
Under the header Related 0 work items click on 1 published; 1 consumed.
Expand the arrow next to ASW_Drop, ARM, and we can see our Template files. Success we have a build artifact that we can now call in a Release pipeline.
Ok Dear Reader, it's late and we are done for today. In our next blog we will cover the release pipeline and look at what was deployed to our QA environment!
As always, thank you for stopping by.