Blog Post

Data Architecture Blog
2 MIN READ

CI CD in Azure Synapse Analytics Part 1

Bradley_Ball's avatar
Bradley_Ball
Icon for Microsoft rankMicrosoft
Dec 07, 2020

I was helping a friend earlier today with their Azure Synapse Studio CI / CD integration.  They had followed our Docs page Source control in Azure Synapse Studio and then they shared errors they were seeing in their release pipeline during deployment. 

We took a step back to discuss what they wanted to do, and it looked like they were too far in the weeds for ADO.  So I walked through creating an Azure DevOps Project, connecting Git to my Azure Synapse Studio, and then creating a branch and pushing some changes.  We'll push changes in a follow up blog post.  Today we cover the basics.

First let's navigate to Projects - Home (azure.com).  We will create a New Project and title it Azure Synapse Studio CI CD.  I'm going to mark this repo private because it's just for us.

 

Now I will click on the Repos menu.

 

 

Next I will go to the bottom of the page.  I want to select Initialize main branch with a README or gitignore.  I will click Initialize.

 

 

At this point I have a Repo that is initialized.

 

Now we can connect this to our Azure Synapse Studio.  Let us travel over to https://web.azuresynapse.net/ and log into our Azure Synapse Studio.  After we login we need to navigate to the Manage screen.  If you are not on the Git configuration page, navigate there.  

 

 

Next we want to click on Set up code repository.  You can select Azure DevOps Git or GitHub.  For this blog we will be selecting Azure DevOps Git.  Then select your organizations Azure Active Directory tenant. *a quick side note, make sure the AAD account you are using to connect to Azure DevOps is the same account that has permissions to your Azure Synapse Studio workspace.

Then click Continue.

Select the Azure DevOps Account that our organization is using. The Project and Git repository name are the same, and are the Project Name we created earlier.

My collaboration branch is main, my Publish branch is workspace_publish, my Root folder is the default, I have checked import existing resources to repository.

 

As this is my initial commit I want to commit this to my main branch.

 

Then I click Apply.

 

*Another note: your company will have a DevOps environment, and specific rules on how you want things to connect.  If I'm doing anything that makes you scream from a developer philosophy, please find me on Twitter under BuckWoody_MSFT .... also don't tell Buck I did this .... I'm not Buck.



At this point in time your Azure DevOps Git Rep should be connected.

 

If we go to our Azure DevOps Repo we should see that it is populated with objects from our Azure Synapse Studio.

 

Back in Azure Synapse Studio, we can navigate to the develop pane, create a new branch to ensure any changes I make will not be automatically deployed against my main version of Azure Synapse Studio.


This is what we will tackle next time.

Updated Dec 07, 2020
Version 1.0
  • hps2022's avatar
    hps2022
    Copper Contributor

    Thanks for responding very quick.

    It is synapse sql serverless pool.

     

    Regarding the branches, I am developing in feature branch and then commit the changes to it. Then I create the pull request in master branch to merge the changes. Then I come back to my master branch and publish the changes from there.

     

    ideally when I publish those changes , my live mode should be sync with publish/master , but live mode has different more data than master branch. 

    I have a synapse pipeline which triggers every night, which pulls new records in my lake database. Now I can see the trigger and successful pipeline run at night, but somehow the changes are not reflected in master branch, but i can see them in live mode. I repeated the process so many times. 

  • Hi hps2022 , 
        Is this an Azure Synapse Dedicated Pool or a Serverless Pool?  In the Repo the folder for the Dedicated pool will not be created.  You need to either manually create a folder structure, check in T-SQL Scripts, or use Visual Studio to create a DACPAC project (currently working on this blog), and then link and push your VS Database project to your Azure DevOps Repo.

      I would check your branches in your git configuration to validate which branch you are publishing your changes to.  For example, if you are publishing to the workspace_publish branch, then you need to perform a pull request to merge the branch to your main branch.

  • hps2022's avatar
    hps2022
    Copper Contributor

    Hello Bradley_Ball 

     

    Thanks for detailed article.

    I have SQL data base for external tables in my synapse workspace. When I connected to repo , the folder for sql database did not get created. And therefore the SQL database was not deployed in new workspace. 

    Also after connecting the azure git repo, my live mode is not being in sync with master branch and any pipeline trigger results are not being reflected in master lake database. I mean the pipelines are being triggers in live mode and new records are not being added to master branch lake database. 

    Appreciate you help with this.

     

    Thanks in advance.

     

     

  • After configure the repository I got this error on Synapse: 

     

    Please set the working branch. Previously used working branch 'developer' is not available.

     

    Set working branch

     

    Just to update the article to avoid doubts. I created an developer branch.