Introduction
In the dynamic landscape of software development, collaboration stands as a cornerstone for success. Harnessing the power of cutting-edge tools and methodologies, teams strive to streamline their processes, elevate their efficiency, and deliver unparalleled results. In this pursuit, the convergence of GIT, Azure DevOps, and Microsoft Fabric Deployment Strategies emerges as a beacon of innovation, revolutionizing the way organizations approach collaborative development and deployment.
Fabric is an end-to-end, human-centered analytics product that unifies an organization’s data and analytics in one place.
It combines the best features of Microsoft Power BI, Azure Synapse, and Azure Data Factory into a single unified software-as-a-service (SaaS) platform.
This article will expose two features of Fabric that make it very powerful for undertaking business intelligence solutions:
- Collaborative development and
- Agile application publishing in the Azure cloud.
Considering Fabric's extensive array of product items already compatible with source control and deployment pipelines, our focus lies on a flagship product empowered by Fabric: Microsoft Power BI. We firmly believe that short-term development efforts will enable most, if not all, Fabric items to benefit from these features.
In this post we delve into the integration of Fabric with Azure DevOps Repos and the possibilities of Fabric Deployment Pipelines, aimed at enhancing both collaborative development and agile release cycles of Power BI solutions. Our goal is to elucidate the practical application of these technologies through examples, emphasizing their advantages within the Fabric ecosystem, and outline key best practices for utilizing Fabric-GIT and Fabric Deployment Pipelines effectively.
Collaborative development and deployment using Microsoft Fabric.
Suppose there are two developers who must collaborate to obtain a business intelligence application where trend analysis and key performance indicators about Medicines distributed to Hospitals are displayed.
Developers want to make source control [1] of the development, so they need to share a repository. They need to deploy the solution from a DEV Stage to a TEST Stage to test the solution, and after successful testing, deploy the solution to a PRODUCTION Stage.
What is the scenario when these developers use Microsoft Fabric?
In Fabric, development and deployment tasks are done using workspaces.
Fabric Workspaces [2] are places to collaborate with colleagues to create collections of items such as lakehouses, warehouses, and reports. A workspace is like a container of the work results of the tools provided by Fabric.
We will describe the architecture of this scenario by means of Figure 1 and provide a step-by-step guide to make the Fabric and GIT integration via an Azure DevOps Repo, and to build the desired deployment pipeline.
Figure 1. Development and Deployment Scenario using Microsoft Fabric and an Azure DevOps Repo.
These are the steps to make the Fabric and GIT integration:
- Create a new project and a GIT Repo for this project in Azure DevOps Services.
- Create one or multiple workspaces within Fabric and grant access to as many developers as needed. In this case, create two workspaces named 'Workspace Dev 1' and 'Workspace Dev 2'. Additionally, create a third workspace, 'Workspace Main', intended to store the combined contents of both developers.
- Create as many branches in the ADO GIT Repo as workspaces in Fabric. Or create folders within a branch to continue maintaining files when merging to the same branch. In Figure 1, we have created the branches Dev 1, Dev 2 and Main.
- Integrate each workspace with GIT pointing to the Organization, the Project, the Branch and Folder inside the branch.
Figures 2 shows the UI to make that Fabric-GIT integration [3]. You will see how to integrate Workspace Dev 1 with GIT.
Similarly, the workspaces Dev 2 and Main can be integrated and synchronized.
Figure 2. Integrating Workspace Dev 1 with GIT.
Each developer can commit changes to their items from his workspace in Fabric. This is done by using the Source Control option of the workspace. See Figures 3 and 4.
Figure 3. Workspace Dev 1 syncing with the GIT branch of ADO Repo.
Figure 4. Workspace with its content synced.
5. Developers must create pull requests [4] in Azure DevOps to merge the commits to the main branch. See Figure 5.
6. Don’t delete any branch after merge, to keep developers maintain items independently.
Figure 5. Creating a pull request in Azure DevOps.
Figure 6. Completing the merging process.
Many developers inquire about collaborating on making changes to the same report.
If multiple developers modify the same report, merge conflicts can arise when attempting to combine their changes into the same branch. [5]
What should be done to combine both contents into a single Power BI report?
- Clone the branch locally and use VS Code to see the differences between two versions of the same file.
- Merge conflicts can be resolved more easily by installing the Pull Request Merge Conflict Microsoft extension. The extension adds special lines (commenting the conflicts) into the report.json file.
- Navigate to Azure DevOps > Repos > Files and delete all lines commenting merge conflicts.
- Commit those changes to the report.json file.
- Then, make the merge anyway. When trying to sync on Fabric, an error message will be displayed.
- When the content is synced, they may encounter a scenario where the report reflects the changes from one developer but not from the other. To ensure that all developers can view all changes, the following process must be repeated: each developer creates a pull request, the changes are merged, and then adjustments are made in the JSON file. Once the necessary modifications are made, the changes are committed to the JSON file, and synchronization in Fabric can be performed.
Figure 7. Edition of a report.json file in Azure DevOps.
To prevent merging conflicts, each developer should work on separate reports. Through deployment pipelines, the changes made by all developers can be deployed to test and production environments. A single dashboard can be created by pinning visuals from multiple reports.
Up to this point, we have shown how to obtain updated content in the Main Branch, based on the changes that developers have made in their respective workspaces.
“Main branch” is ready to be deployed now. The tools for the deployment in Fabric are the deployment pipelines.
Deployment Pipelines in Fabric.
Deployment Pipelines are a Fabric feature that can work independently of the integration of workspaces with GIT. [6]
The goal of deployment pipelines is to efficiently transfer content from one workspace to another in an agile manner.
The process followed by Fabric deployment pipelines is completely codeless.
Let's look at it.
You can create a deployment pipeline outside of all workspaces, or within one of the workspaces.
Figure 8. Creating a New Deployment pipeline out of workspaces.
The first thing is to define a meaningful name for the purpose of the pipeline.
By default, Fabric proposes to use three stages, called DEVELOPMENT, TEST and PRODUCTION.
The stages are essentially Fabric workspaces that you can either associate with existing ones or create when deploying content from the space on the left to the one on the right.
You can do inverted deploys, that is, move content from the stage on the right to the stage on the left, to undo changes.
Never has there been such agility in publishing applications in the cloud.
Currently, a Fabric pipeline can be scheduled to run at a specified time. However, this differs from the ease of creating events to automate the execution of ADO Pipelines, it is just that ADO Pipelines typically require more coding.
It is convenient to create the deployment pipeline in the space where most changes have already been synchronized, for example, in our case it would be the “Workspace Main”, but it’s possible to create a pipeline for deploying the content from any of the workspaces.
Figure 9 illustrates the creation of a deployment pipeline in Workspace Main, utilizing this workspace as the initial stage (the DEVELOPMENT Stage). See Figure 1 to see the pipeline.
However, you have the flexibility to change the assigned workspace to any stage as needed. Figure 10 shows an assignment process.
Figure 9. Creation of a deployment pipeline inside a Fabric’s workspace.
Figure 10. Assign a workspace to a stage.
TEST and PRODUCTION workspaces can either be selected from previously created workspaces or generated by Fabric when deploying content. They are assigned names based on the chosen development stage workspace, followed by either [Test] or [Production].
It’s possible to compare content [7] between stages. Figure 11 says that Workspace Dev 1 contains two less items than the workspace assigned to TEST. This is a symptom that there are missing deploys to do.
Figure 11. Compare content between stages.
You need to refresh data from PBI services to see the complete report in the target workspace to be reviewed, because Fabric only moves metadata.
Another amazing feature is the possibility to publish an app from any of the workspaces in the pipeline. You can update the content of your Power BI apps using a deployment pipeline, giving you more control and flexibility when it comes to your app's lifecycle. Create an app for each deployment pipeline stage, so that you can test each update from an end user's point of view. Use the publish or view button in the workspace card to publish or view the app in a specific pipeline stage. See Figure 12.
This opens the doors to greater participation of individuals in reviewing and exchanging criteria on specific aspects they wish to submit for review, without the need to create a complex solution to be reviewed later.
Figure 12. Publish the app from any workspace.
Summary of Good Practices.
Collaborative development and agile deployment are two key aspects for the software industry. Microsoft Fabric delivers faster, lower-code scenarios than ever before for business intelligence solutions.
Microsoft Fabric powers the use of Azure DevOps Repos and has realized a content-based application deployment strategy, allowing engineers to focus on everything that matters to the business.
To take full advantage of these benefits, we offer this best practice guide:
- Create as many branches in ADO Repo as there are developers to collaborate. The Main branch should preferably be handled by whoever oversees development direction.
- Developers should preferably deal with aspects of the solution that are relatively independent, to avoid merging conflicts over the same file.
- Create as many workspaces in Fabric as developers participate, plus the workspace where all the content will be joined. Keep the contents of each workspace synced with the corresponding Repo branch.
- Utilize the workspace that consolidates content as the initial stage in the Fabric pipeline. This approach reduces the number of workspaces required in the solution.
- Now that you have great ease and require minimal code for continuous integration, it's crucial to publish your app frequently to the cloud to gather feedback quickly. Even if it doesn't include all agreed-upon features, this approach ensures greater end-user satisfaction and commitment to the team.
Resources:
