Microsoft Fabric: Automate Artifact Deployment with Azure DevOps and Python

Microsoft

May 01, 2025

Microsoft Fabric is rapidly becoming the go-to platform for enterprise-grade analytics and reporting. However, deploying artifacts like dataflows, datasets, and reports across environments (Dev → Test → Prod) can be a manual and error-prone process.

This blog walks you through a fully automated and secure CI/CD solution that uses Azure DevOps Pipelines, Python scripting, and Microsoft Fabric REST APIs to streamline artifact deployment across Fabric workspaces. Whether you’re a DevOps engineer or Fabric administrator, this setup brings speed, security, and consistency to your deployment pipeline.

✅ The Challenge

Microsoft Fabric currently provides deployment pipeline console for promotion of artifacts. Manual promotion across environments introduces risks like:

Misconfiguration or broken dependencies
Lack of traceability or versioning
Security and audit concerns with manual artifact movement

🔧 The Solution — Python + Azure DevOps + Fabric API

This solution uses a tokenized YAML pipeline combined with a custom Python script to promote Fabric artifacts between environments using Fabric Deployment Pipelines and REST APIs.

🔑 Key Advantages & How This Helps You

✅ Zero-Touch Deployment – Automates Dev → Test artifact promotion using Fabric Deployment APIs
✅ Repeatable & Consistent – YAML pipelines enforce consistent promotion logic
✅ Secure Authentication – OAuth2 ROPC flow with service account credentials
✅ Deployment Visibility – Logs tracked via DevOps and Fabric API responses
✅ Low Overhead – Just a lightweight Python script—no external tools needed

🧩 Core Features

1️⃣ Fabric Deployment Pipeline Integration – Automates artifact promotion across Dev, Test, and Prod stages
2️⃣ Environment-Aware Deployment – Supports variable groups and environment-specific parameters
3️⃣ Flexible API Control – Granular stage control with REST API interactions
4️⃣ Real-Time Status Logging – Pipeline polls deployment status from Fabric
5️⃣ Modular YAML Architecture – Easy to plug into any existing DevOps pipeline
6️⃣ Secure Secrets Management – Credentials and sensitive info managed via DevOps variable groups

⚙️ How It Works

Define a Fabric Deployment Pipeline in your source workspace (Dev).
Configure Azure DevOps pipeline with YAML and use Python to trigger the Fabric deployment stage.
Promote artifacts (notebooks, data pipelines, semantic models, reports, lakehouses, etc.) between environments using Fabric REST APIs.
Monitor deployment in DevOps logs and optionally via Fabric’s deploymentPipelineRuns endpoint.

📌 Sample: Python API Trigger Logic

Step-by-Step Setup

1. Create a Deployment Pipeline in Fabric

Go to Microsoft Fabric Portal .
Navigate to Deployment Pipelines.
Click Create pipeline, and provide a name.
The pipeline will have default Development, Test, and Production stages.

2. Assign Workspaces to Stages

For each stage (Dev, Test), click Assign Workspace.
Choose the appropriate Fabric workspace.
Click Save and Apply.

3. Copy the Deployment Pipeline ID

Open the created pipeline.
In the browser URL, copy the ID:
Store this ID in a DevOps variable group.

4. Update Placeholder Values

Replace all placeholder values like:
tenant_id
username
client_id
deployment_pipeline_id
Use variable groups for security.

5. Create Variable Groups in Azure DevOps

a. Create a Group: fabric-secrets

Store secrets like:

tenant_id
client_id
service-acc-username
service-acc-key

b. Create a Group: fabric-ids

Store:

deployment_pipeline_id
dev_stage_id
test_stage_id

6. Get Stage IDs via API

Use this API to get stage IDs:

Or use a helper script to extract dev_stage_id and test_stage_id, then update them in your fabric-ids variable group.

7. Azure DevOps YAML Pipeline

Save the below in a .yml file in your repo:

trigger: none

variables:
- group: fabric-secrets
- group: fabric-ids

stages:
- stage: Generate_And_Deploy_Artifacts
displayName: 'Generate and Deploy Artifacts'
jobs:
- job: Generate_And_Deploy_Artifacts_Job
displayName: 'Generate and Deploy Artifacts'
steps:

- publish: $(System.DefaultWorkingDirectory)
artifact: fabric-artifacts-$(System.StageName)
displayName: 'Publish Configuration Files'

- script: |
echo "🚀 Running test.py..."
export tenant_id=$(tenant_id)
export username=$(service-acc-username)
export password=$(service-acc-key)
export client_id=$(client_id)
export deployment_pipeline_id=$(deployment_pipeline_id)
export dev_stage_id=$(dev_stage_id)
export test_stage_id=$(test_stage_id)

python fabric-artifacts-deploy/artifacts/test.py
displayName: 'Run Deployment Script (Dev → Test)' '''

8. Python Script - test.py

import json
import os
import time

tenant_id = os.getenv('tenant_id')
client_id = os.getenv('client_id')
username = os.getenv('username')
password = os.getenv('password')

deployment_pipeline_id = os.getenv('deployment_pipeline_id')
dev_stage_id = os.getenv('dev_stage_id')
test_stage_id = os.getenv('test_stage_id')
deploy_url = f'https://api.fabric.microsoft.com/v1/deploymentPipelines/{deployment_pipeline_id}/deploy'
token_url = f'https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token'

def get_access_token():
token_data = {
'grant_type': 'password',
'client_id': client_id,
'scope': 'https://api.fabric.microsoft.com/.default offline_access',
'username': username,
'password': password,
}
response = requests.post(token_url, data=token_data)
if response.status_code == 200:
return response.json().get('access_token')
else:
print("❌ Failed to authenticate")
print(response.text)
exit(1)

def poll_operation_status(location_url, headers):
while True:
response = requests.get(location_url, headers=headers)
if response.status_code in [200, 201]:
status = response.json().get("status", "Unknown")
print(f"⏳ Status: {status}")
if status == "Succeeded":
print("✅ Deployment successful!")
break
elif status == "Failed":
print("❌ Deployment failed!")
print(response.text)
exit(1)
time.sleep(5)
elif response.status_code == 202:
retry_after = int(response.headers.get("Retry-After", 5))
print(f"Waiting {retry_after} seconds...")
time.sleep(retry_after)
else:
print("❌ Unexpected response")
print(response.text)
exit(1)

def deploy(source_stage_id, target_stage_id, note, token):
payload = {
"sourceStageId": source_stage_id,
"targetStageId": target_stage_id,
"note": note
}
headers = {
'Authorization': f'Bearer {token}',
'Content-Type': 'application/json'
}
response = requests.post(deploy_url, headers=headers, data=json.dumps(payload))
if response.status_code in [200, 201]:
print("✅ Deployment completed")
elif response.status_code == 202:
location_url = response.headers.get('location')
if location_url:
poll_operation_status(location_url, headers)
else:
print("❌ Location header missing in 202 response")
else:
print("❌ Deployment failed")
print(response.text)