Azure DevOps is the service of choice when it comes to automate the provisioning of your modern data platform and to allow the continuous integration (CI) and continuous delivery (CD) to continuously test, build , and deploy your code across multiple environments.
In this particular context of Azure Synapse Analytics , you can think of CICD as a two staged process:
- A first stage, where you deal with resources such as compute pools and the workspace itself
- A second stage, where you deal with the so called artifacts (Synapse Notebooks, SQL Scripts, Pipelines, etc)
In my previous article, I demonstrated how we could easily use the Synapse Workspace Deployment extension to accomplish this second stage of the process. I’m now coming back to this topic as I realized that many of our customers were reporting difficulties in completing this second stage of their Synapse CICD process because they were failing to deploy Synapse artifacts to a Managed VNET Synapse Workspace.
In this particular scenario, the deployment was failing because their target workspace was not allowing access from public networks.
When executing the "Synapse Workspace Deployment task" the Release Pipeline would fail with the following error:
Deploy artifact failed: {"code":"PublicNetworkAccessDenied","message":"The public network interface on this Workspace is not accessible. To connect to this Workspace, use the Private Endpoint from inside your virtual network or enable public network access for this workspace."}
To address this problem, we will demonstrate how Private Endpoints and Azure DevOps Self-hosted agents can be used together to reach the target workspace privately and securely.
You can think of this as a two phased action plan :
In a first phase, we will detail the necessary configuration actions to be performed from the DevOps side of the equation
In a second phase, it will be detailed the necessary networking resource configuration steps.
Stage 1 – Configure DevOps
First things first: you need to use our own compute infrastructure that will run the Azure DevOps Release Pipeline. Remember that your Synapse workspace is blocking access from public networks, so instead of using a Microsoft-hosted that will try to reach the workspace via public endpoint, you will need to use a Self-hosted agent and sit this agent on a VNET so it can communicate privately and securely with your target Workspace.
Let’s start by creating a Self-hosted Windows agent (you can refer to this public doc to get more details about this step). In this example, I will use an Azure VM to install this agent:
- Connect to your Azure VM and open a new browser session. Navigate to https://dev.azure.com/{your_organization} and Sign in with the user account you plan to use in your Azure DevOps organization
- If you haven’t created your your personal access token (PAT) please follow the instructions provided here.
- Download and configure the Self-hosted agent following these instructions
This screenshot below reflects an example of this configuration step:
When you have finished the configuration process above, check your “Default” agent pool in DevOps. You should see your newly created Self-hosted agent with current status "Online" :
If your Self-hosted agent status is shown as "Offline", make sure to run the agent from the VM, submitting the following command from the Powershell:
C:\agent> .\run.cmd
Once you have your Self-hosted agent up and running, you will need to configure the DevOps Release Pipeline.
- Follow these steps below to configure your Release Pipeline to deploy Synapse Workspace artifacts from a source environment to a target environment:
From the left menu select “Pipelines-->Releases” and then select “New release pipeline”
Select to start with an “Empty job” when selecting your template. You can provide a new name to your Release Pipeline and to the initial stage. Click on the “+ Add” to add a new Artifact.
Configure the Pipeline source artifact by providing your Project, Source Repository and Default branch (typically the workspace_publish branch, that hosts the template JSON files resulting from "Publish" ).
Select “Add” to confirm your settings and to create the source artifact. Now it’s time to configure the Release Stage. Select the “1 job , 0 task” link on your stage.
Provide a new display name to your agent. Make sure to change the “Agent selection” from “Azure Pipelines” to “Default” (Default is the agent pool that is hosting your newly created Self-hosted agent)
Click the “+” sign on the Agent job bar to add a new task. Then search for Synapse and add the "Synapse Workspace Deployment" task to your agent.
Configure the Synapse Workspace deployment task and save your changes (in this example I’m not overriding any parameters).
At the end of this first stage you should have completed the following steps:
- Download, Install and Configure the Azure DevOps Self-hosted agent
- Create and configure a DevOps Release Pipeline using the "Synapse Workspace Deployment" task
Let’s move forward to the final stage of this article: putting all networking pieces together so the Azure DevOps service and Synapse service can communicate privately and securely.
Stage 2 – Network Setup
Navigate to your Synapse Workspace resource page and create a Private Endpoint to the Synapse Workspace development endpoint.
You must select “Dev” when choosing the “Target sub-resource”. This will refer to the Synapse Workspace Development endpoint.
Select the Virtual Network where you have hosted the DevOps Self-hosted agent VM
After the “Review and create” step, hit the “Create” button to create the Synapse Workspace Private Endpoint.
If you check your Virtual Network Private Endpoints you’ll see the newly create private endpoint:
Check your Private Endpoint DNS Configuration and look for the IP Address and FQDN values:
From the Powershell command line, "nslookup" the Synapse Workspace dev endpoint and confirm if the values from the output match the Private DNS Config values:
Before you kick-off the DevOps Release Pipeline, you need to confirm that the DevOps Connection Service Principal that is being used by Azure Synapse Deployment task has been granted the necessary Synapse RBAC permissions (if not, the deployment task will fail with a 403 authorization error)
Finally, you can create your first release to start deploying your source Workspace artifacts to your target Managed VNET Workspace:
You should now see the Synapse Workspace Deployment task succeeded!
A final word to thank my colleagues @Silas Mendes (for challenging me to address this use case) and @Sergio Fonseca for his networking expertise.