Here's a quick review of the road so far:
CI CD in Azure Synapse Analytics Part 1
CI CD in Azure Synapse Analytics Part 3
Select the subscription we are granting the access to the MI.
Click on Access control (IAM), click + Add, then click Add role assignment.
Select the Contributor Role and click Members.
Click +Select members, type in the name of the project workspace, select that MI, click select, then click Next.
Validate that everything looks correct and click Review + assign.
Now that our permissions are in place we will create our template, in the search bar type "Azure Synapse Analytics" and click the Azure Synapse Analytics icon and then click Create.
In this example we will create a new resource group named "AzureSynapseDeployFromTemplate".... because I like names that say what we are doing. In all reality you would be using this to create your QA environment after we built our Dev, or you could be using this to build your initial Dev environment. Whatever you are using this for add your Resource Group, RG, name new or existing.
Next we set our Managed resource group. What is a Managed resource group Dear Reader? It is the resource group that creates managed components created with the PaaS server that is our Azure Synapse Analytics Workspace. This used to be auto created now you can create it yourself, I'm naming this with the same RG as above with the extension MRG for Managed Resource Group e.g. "AzureSynapseDeployFromTemplateMRG".
Now we name the workspace and we will use the simple name "deployfromtemplate".
Next we select our region, then we add the Data Lake Storage account. We will give the account name and filesystem the same name, "adsldft". Now click next.
*Note some of these names must be unique so please substitute your own naming conventions. But keep track of these things we will need them later!
I'm going to leave the SQL Server admin login as the default, CHANGE THIS!!! Do not leave this as the default value. Make it unique per your organization.
Then click Next: Networking.
Click Enable by Managed virtual network. This is my preferred setting as it requires a private endpoint for all services interacting with Azure Synapse Analytics.
Yes for create managed private endpoint to primary storage, you used to have to do this manually it is nice to add this as a part of deployment.
Allow outbound data traffic only to approved targets, I'm selecting no for the demo but when you deploy this for your Dev, QA, Test, Prod, or any other environment should select yes and specify the subscriptions that have access to this system.
Public network access. Again this is a case of do what I say not what I do. I'm enabling this, but you will disable this. This adds a public firewall rule that leaves your workspace open to anyone in your AAD tenant. They still have to be in your AAD tenant, and you can remove this rule after deployment.
When you do your forever deployments, click disable and add the firewalls needed, or leave this disabled and use Azure Synapse Analytics Private Link Hubs and private endpoints to secure your environment. That should be another blog.... more on that later.
Finally click Next: Tags.
There are all kinds of useful tags, they can allow you to group items via PowerShell and assist with charge back. Add your tags here are some examples.
Click Next: Review + create.
NOW this is where we get our template! Instead of clicking Create we will click Download a template for automation.
Click the download link up at the top of the screen.
As you can see from my screen shot, I've done this a few times. This zip file we have downloaded contains our template and parameter JSON files. Extract these files in a folder we will be uploading them to our Azure DevOps Repo shortly.
Now browse to your DevOps Repo. Click on the three dots next to the Repo name and select New Folder. We will name this folder ARMTemplate and create a file call readme.txt. We don't need this file, but you cannot create a new folder. We will delete the readme.txt shortly, but keep it for now.
Click Commit to save the readme.txt
Click the three dots next to the ARMTempalte folder and click Upload file(s).
Click Browse... in order to browse to our file we extracted our template and parameters JSON files. Click both files and open them. Click Commit.
Now we can delete the readme.txt. Click the three dots next to the readme.txt and click Delete.
Now we can create our Build Pipeline. Navigate to Pipelines and click New pipeline.
Click Use the classic editor.
Ensure our Repo is selected and click Continue.
Click Empty job.
We should name our Pipeline something that states exactly what it does, for example "Build Pipeline ARM Deployment Synapse Environment".
Click the + next to Agent job 1, type copy files, click Add.
Set the Display name to "Copy Files to: $(Build.ArtifactStagingDirectory)/ARM".
Click the ... next to the Source Folder text box ,navigate to the ARMTemplate folder, and click OK.
Under Contents type "*.json".
Our Target Folder is using the variable to store our files on the build agent, "$(Build.ArtifactStagingDirectory)/ARM"
Click the + next to Agent job 1 and type Publish Pipeline Artifacts. Click add.
Type "$(Build.ArtifactStagingDirectory)" for the File or directory path.
We will name our Artifact name ASW_Drop for Azure Synapse Workspace Drop.
Click Save & queue.
Type something under the Save comment and click Save and run.
Validate that the build goes smoothly.
Now we create our release pipeline. Navigate to Releases and click + New and + New release pipeline.
Select Empty job.
Select our Build Pipeline, and be glad we named it something obvious!
Validate that everything is populated as expected and click Add.
Click 1 job, 0 tasks.
*Note my images says 1 job, 1 task. Ignore that yours will say 1 job, 1 task.
Click the + next to Agent job and type ARM template deployment and click Add.
We will add our Azure Resource Manager connection. (If you haven't set this up you will need this. This links your DevOps Repo to your Azure Subscription, so make sure this is in place).
After selecting the Resource Manager connection, select the subscription to which we will be deploying resources.
Type in the name of the Resource Group, aka RG, from our template. For example, AzureSynapseDeployFromTemplate.
Select our Azure Region, for example East US 2.
Click the ... next to the Template text box and navigate to our template.json file in our build package.
Click the ... next to the Template parameters text box and navigate to our parameters.json file in our build package.
Now we need to set our password for our SQL Administrative account for the Azure Synapse Workspace. This value in the parameters.json is a secure string. By nature a secure string cannot be plain text and must be encrypted. To do that we will click on the Variables pane.
Click + Add. We will name this the exact same attribute that is in the JSON file, in this case "sqlAdministratorLoginPassword". Type in the password value and then click the lock Icon. That will make this a secure string.
Also add the value system.debug with a value of TRUE. If we get any errors this should help us with detailed output.
Now in the Override template parameters type "-sqlAdministratorLoginPassword $(sqlAdministratorLoginPassword)".
We are substituting the sqlAdministratorLoginPassword value in the JSON with the -sqlAdministratorLoginPassword command. The value we are replacing this with is our varialble $(sqlAdministratorLoginPassword).
Next we need to keep in mind the MI of the DevOps project is deploying this which means it will by default have admin access to the Azure Synapse Workspace, but nothing else will. To that end we will add an Azure PowerShell script that adds an AAD group to the Admin group of the Workspace.
Select our Azure Subscription, Inline path, and copy the script below.
*Note the version of Az.Synapse PowerShell module will change over time, as of the publishing of this blog 0.7.0 currently works.
# You can write your azure powershell scripts inline here.
# You can also pass predefined and custom variables to this script using arguments
##Required for azure devops initial deployment
Install-Module Az.Synapse -RequiredVersion 0.7.0 -Scope CurrentUser -Force -AllowClobber
##ObjectID is for bballasw_WSAdmins
New-AzSynapseRoleAssignment -WorkspaceName <insert your workspace name here> -RoleDefinitionName "Workspace Admin" -ObjectID <insert your AAD Group here>
Add the Preferred Azure PowerShell Version as 3.1.0.
Click Save and OK.
Click Create release.
Click Create.
Allow this to run and validate that the resources have been deployed to our Resource Group.
Click on the link to the Azure Synapse Workspace and validate that you can access it.
My appologies on the delay and the technical issues. Thank you again Dear Reader for stopping by!
Thanks,
Brad