Have you ever deployed a release to production only to find out a bug has escaped your testing process and now users are being severely impacted? In this post, I’ll discuss how to deploy a fix from your development Synapse Workspace into a production Synapse Workspace without adversely affecting ongoing development projects.
This example uses Azure DevOps for CICD along with a Synapse extension for Azure DevOps: Synapse Workspace Deployment. In this example, I assume Synapse is already configured for source control with Azure DevOps Git and Build and Release pipelines are already defined in Azure DevOps. Instructions on how to apply this this can be found in the Azure Synapse documentation for continuous integration and delivery.
Let’s start by looking at an example of development activities in a Synapse workspace. During development, several things happen. The Main (aka. collaboration, trunk or master) branch in the Synapse Development environment gets branched off one or more times to support parallel feature development activities. Changes are committed within the feature branches. Periodically, the branches need to refresh their code with the latest changes that have been incorporated into Main. It is typically up to the developer working on a feature branch to merge their code from Main at whatever interval they choose. These activities can all be done through the Synapse user interface by executing a pull request in the proper direction, in this case from Main to Feature 1. It is up to the feature developers to resolve conflicts in their feature branch.
At some point, code in the Main branch is ready for deployment to the Synapse Test environment. After testing, a decision is made to release the code into Production. This is done through an Azure DevOps release pipeline outside the scope of this document. For the sake of simplicity, the following diagrams omit the Synapse Test environment and only show the Synapse Development and Production environments since we are targeting a Production “hot fix” use case. Best practice would be to have at least one intermediate environment between Development and Production.
After a Production release, development continues on with Feature 3…N, and changes are incorporated into the Main branch as shown in the diagram below.
Now to the heart of the discussion... At some point, a critical bug is discovered in Production that needs to be fixed immediately. How do we do this?
Start by getting the Commit ID of the code that was released to production and create a new branch based on the Commit ID. These steps must be done through Azure DevOps.
In Azure DevOps, navigate to Pipelines/Releases, and select the appropriate deployment pipeline for the production release. Then select the appropriate release.
Note: It is important to have the release pipeline have access to both the ARM templates created by the build process, as well as the actual source code. The ARM templates are used for deployment into later stages (Test/QA/Prod/etc). The source code will be used for hot fixes.
Copy the Commit ID (this looks like an 8 character hex string in the UX, but is actually only the first 8 characters of a longer SHA) for the released code as shown in the diagram below. Make sure you select the Commit ID for the source code and not the ARM template artifact.
Navigate to the branches section of your Azure DevOps Repo and create a new branch based on the Commit ID you copied in the last step.
Select the pulldown on “main” to get to the UX screen that allows you to search for a specific Commit:
Validate that the new branch is based on the proper Commit ID:
Now that we have a special code branch for creating the fix, we can go back into the Development Synapse Workspace, select the hot fix branch, and make our corrections.
When satisfied with the changes, commit them to the branch. In order to get changes in the hot fix branch deployed to Production without impacting the Development and Test environments, we need to create a new Azure DevOps release pipeline incorporating the Synapse Workspace Extension task. In our example, the release of the hot fix branch into Production is triggered manually in Azure DevOps.
The hot fix branch should remain in existence for the life of the Production release it is associated with, so that future hotfixes can be incorporated easily.
Let’s create a new release pipeline specifically for hot fixes following the instructions for Synapse continuous integration and delivery. We will make a couple modifications to the pipeline that is created in that document. First, the source artifact for this pipeline should have a default branch pointing to the hot fix branch for this release (ie Release3_Hotfixes in our example). Update the source alias to reflect the proper code branch. Update the stage name to “Production_Hotfix”.
Now click on the “1 job, 0 task” link under the stage name. Add a new agent job. Search for Synapse and select the “Toggle” task. Update the fields as appropriate for your Production environment. For the subscription, use a service connection for the Prod environment. You can see how to create service connections in this document.
Add another agent task. Search for Synapse and add the task called “Synapse Workspace Deployment”. Update the fields as shown in the screenshot below, replacing values as appropriate with names in your Production environment. Make sure the operation type is “Validate and Deploy”.
Add another agent task; search for Synapse and add the task called “Azure Synapse Toggle Triggers”. Update the fields as shown in the screenshot below, replacing values as appropriate with names in your Production environment. Rename the pipeline to something meaningful. Save your pipeline.
The pipeline we just created must be manually run to deploy the hot fix to Production. Go into Azure DevOps and create a new release using this pipeline when you are ready.
We have now completed the hot fix release flow shown below. Validate that your Synapse Development Main code branch and live mode UX are unaffected by the hot fix.
Don’t forget to incorporate the hot fix code changes back into the Main and Feature branches at a convenient time. As new releases are deployed, new hot fix branches based on the Commit ID of the release will need to be created. Hot fix branches associated with releases that are no longer current can be deleted.
That’s all there is! Congratulations on fixing the burning issue without interrupting ongoing development and testing.