Automating Azure SQL DB index and statistics maintenance using Elastic Jobs.
Using as a reference of this article. Automating Azure SQL DB index and statistics maintenance using Azure Automation | Microsoft Community Hub To provide complete solution to maintain you Azure SQL DB statistics and maintenance we provide our maintenance script here What's an elastic job? Create and schedule elastic jobs that periodically run against one or many Azure SQL databases. The jobs run Transact-SQL (T-SQL) queries and perform maintenance tasks. What's required? Create an empty Azure SQL Database with the S1 tier or higher. The Elastic Job Agent will use this database to store job information, job definitions, and other related data. The database must be created on the same Azure SQL Server as the Elastic Job Agent. During the Elastic Job Agent setup, this database is used as the job database. Microsoft Entra authentication with a user-assigned managed identity (UMI) (recommended). The elastic job must be able to log in to each target server or database. Create an elastic job agent. It’s a managed service that lets you run and schedule jobs across Azure SQL databases. Creating the Azure SQL database and UMI should be fairly straightforward, but I’ve included the code below in case you’d like to follow along. # This is sample code, so feel free to replace the names and location with the values that suit your environment. # Resource Group New-AzResourceGroup -Name "rg-elasticjobs-fr" -Location "FranceCentral" # Azure SQL Database - Elastic Jobs New-AzSqlServer -ResourceGroupName "rg-elasticjobs-fr" -ServerName "srv-elasticjobs-fr" -Location "FranceCentral" -ExternalAdminName "XXXXXXXXX@msft.com" -EnableActiveDirectoryOnlyAuthentication New-AzSqlDatabase -ResourceGroupName "rg-elasticjobs-fr" -ServerName "srv-elasticjobs-fr" -DatabaseName "elasticjobs-db" -Edition "Standard" -RequestedServiceObjectiveName "S1" # Azure SQL Database - Target New-AzSqlServer -ResourceGroupName "rg-elasticjobs-fr" -ServerName "srv-target-fr" -Location "FranceCentral" -ExternalAdminName "XXXXXXXXX@msft.com" -EnableActiveDirectoryOnlyAuthentication New-AzSqlDatabase -ResourceGroupName "rg-elasticjobs-fr" -ServerName "srv-target-fr" -DatabaseName "target-db" -Edition "Standard" -RequestedServiceObjectiveName "S0" -SampleName "AdventureWorksLT" # User Managed Identity New-AzUserAssignedIdentity -ResourceGroupName "rg-elasticjobs-fr" -Name "umi_fr_user" -Location "francecentral" In the search box at the top, type Elastic Job Agent, then select it and create a new Elastic Job Agent. On the identity tab, select the UMI and then click on Review + Create Now, let's create a target group (The set of servers, pools, and databases to run a job against.) and add targets for the jobs. Elastic Jobs Agent - Jobs - Target Groups - Create, once created, then click on the name, create again and add a Target Group Member. On each of the target databases, create the required objects for the maintenance script, create a contained user linked to the UMI, and grant the user the permissions needed to run the job scripts. Note: This is sample code. Please adjust the permissions based on your requirements and apply least privilege. Next, let’s schedule a job! For an Elastic Job that runs once every week, the ISO 8601 repeating interval is: P1W Add steps to each elastic job. A target group must be chosen for each job step. If you would like to see the current status of the jobs, head to Jobs, Job Executions. If further information is required, connect to the elasticjobs-db database and run the following query: SELECT TOP (10) * FROM jobs.job_executions WHERE lifecycle <> 'Succeeded' ORDER BY create_time DESC; References: Automation in Azure SQL overview - Azure SQL Database & Azure SQL Managed Instance & Azure Synapse Analytics | Microsoft Learn Elastic Jobs Overview - Azure SQL Database | Microsoft Learn Create, configure, and manage elastic jobs - Azure SQL Database | Microsoft Learn Disclaimer Please note that products and options presented in this article are subject to change. This article reflects for Azure SQL Database in September 2026. I hope this article was helpful for you, please feel free to share your feedback in the comments section.Migrate Sentinel to Defender - Why It Is a Security Architecture Decision, Not Just a Portal Change
Microsoft will retire the Sentinel experience in Azure on March 31, 2027. Most of the conversation around this transition focuses on cost optimization and portal consolidation. That framing undersells what is actually happening. The unified Defender portal is not a new interface for the same capabilities. It is the platform foundation for a fundamentally different SOC operating model — one built on a 2-tier data architecture, graph-based investigation, and AI agents that can hunt, enrich, and respond at machine speed. Partners who understand this will help customers build security programs that match how attackers actually operate. This document covers four things: What the unified experience delivers — the security capabilities that do not exist in standalone Sentinel and why they matter against today’s threats. What the transition really involves - is not data migration, but it is a data architecture project that changes how telemetry flows, where it lives, and who queries it. Where the partner opportunity lives — a structured progression from professional services (transactional, transition execution, and advisory) to ongoing managed security services. Why does the unified experience win competitively — factual capability advantages that give partners a defensible position against third-party SIEM alternatives. The Bigger Picture: Preparing for the Agentic SOC Before getting into transition mechanics, partners need to understand where the industry is headed — because the platform decisions made during this transition will determine whether a customer’s SOC is ready for what comes next. The security industry is moving from human-driven, alert-centric workflows to an operating model built on three pillars: Intellectual Property — the detection logic, hunting hypotheses, response playbooks, and domain expertise that differentiate one security team from another. Human Orchestration — the judgment, context, and decision-making that humans bring to complex incidents. Humans set strategy, validate findings, and make containment decisions. They do not manually triage every alert. AI Agents - built agents that execute repeatable work: enriching incidents, hunting across months of telemetry, validating security posture, drafting response actions, and flagging anomalies for human review. The SOC of 2027 will not be scaled by hiring more analysts. It will be scaled by deploying agents that encode institutional knowledge into automated workflows — orchestrated by humans who focus on the decisions that require judgment. This transformation requires a platform that provides three things: Deep telemetry — agents need months of queryable data to analyze behavioral patterns, build baselines, and detect slow-moving threats. The Sentinel data lake provides this at a cost point that makes long-retention feasible. Relationship context — agents need to understand how entities connect. Which accounts share credentials? What is the blast radius of a compromised service principle? What is the attack path from a phished user to domain admin? Sentinel Graph provides this. Extensibility — partners and customers need to build and deploy their own agents without waiting for Microsoft to ship them. The MCP framework and Copilot agent architecture provide this. None of these exist in Azure experience for Sentinel. All three ship with the Defender experience. The urgency goes beyond the March 2027 deadline. Organizations are deploying AI agents, copilots, and autonomous workflows across their businesses — and every one of those creates a new attack surface. Prompt injection, data poisoning, agent hijacking, cross-plugin exploitation — these are not theoretical risks. They are in the wild today. Defending against AI-powered attacks requires a security platform that is itself AI Agent-ready. The new experience in Defender unlocks this experience. What Unified SIEM and XDR Actually Delivers The original framing — “single pane of glass for SIEM and XDR” — is accurate but insufficient. Here is what the unified platform delivers that standalone Sentinel does not. Cross-Domain Incident Correlation The Defender correlation engine does not just group alerts by time proximity. It builds multi-stage incident graphs that link identity compromise to lateral movement to data exfiltration across SIEM and XDR telemetry — automatically. Consider a token theft chain: an infostealer harvests browser session cookies (endpoint telemetry), the attacker replays the token from a foreign IP (Entra ID sign-in logs), creates a mailbox forwarding rule (Exchange audit logs), and begins exfiltrating data (DLP alerts). In standalone Sentinel, these are four separate alerts in four different tables. In the unified platform, they are one correlated incident with a visual attack timeline. 2-Tier Data Architecture The Sentinel data lake introduces a second storage tier that changes the economics and capabilities of security telemetry: Analytics Tier Data Lake Purpose Real-time detection rules, SOAR, alerting Hunting, forensics, behavioral analysis, AI agent queries Latency Sub-5-minute query and alerting Minutes to hours acceptable Cost ~$4.30/GB PAYG ingestion (~$2.96 at 100 GB/day commitment) ~$0.05/GB ingestion + $0.10/GB data processing (at least 20x cheaper) Retention 90 days default (expensive to extend) Up to 12 years at low cost Best for High-signal, low-volume sources High-volume, investigation-critical sources The architecture decision is not “which tier is cheaper.” It is “which tier gives me the right detection capability for each data source.” Analytics tier candidates: Entra ID sign-in logs, Azure activity, audit logs, EDR alerts, PAM events, Defender for Identity alerts, email threat detections. These need sub-5-minute alerting. Data lake candidates: Raw firewall session logs, full DNS query streams, proxy request logs, Sysmon process events, NSG flow logs. These drive hunting and forensic analysis over weeks or months. Dual-ingest sources: Some sources need both tiers. Entra ID sign-in logs are the canonical example — analytics tier for real-time password spray detection, Data Lake for graph-based blast radius analysis across months of authentication history. Implementation is straightforward: a single Data Collection Rule (DCR) transformation handles the split. One collection point, two routing destinations. The right framing: “Right data in the right tier = better detections AND lower cost.” Cost savings are a side effect of good security architecture, not the goal. Sentinel Graph Sentinel graph enables SOC teams and AI agents to answer questions that flat log queries cannot: What is the blast radius of this compromised account? Which service principals share credentials with the breached identity? What is the attack path from this phished user to domain admin? Which entities are connected to this suspicious IP across all telemetry sources? Graph-based investigation turns isolated alerts into context-rich intelligence. It is the difference between knowing “this account was compromised” and understanding “this account has access to 47 service principals, 3 of which have written access to production Key Vault.” Security Copilot Integration Security Copilot embedded in the defender portal helps analysts summarize incidents, generate hunting queries, explain attacker behavior, and draft response actions. For complex multi-stage incidents, it reduces the time from “I see an alert” to “I understand the full scope” from hours to minutes. With free SCUs available with Microsoft 365 E5, teams can apply AI to the highest-effort investigation work without adding incremental cost. MCP and the Agent Framework The Model Context Protocol (MCP) and Copilot agent architecture let partners and customers build purpose-built security agents. A concrete example: an MCP-enabled agent can automatically enrich a phishing incident by querying email metadata, checking the sender against threat intelligence, pulling the user’s recent sign-in patterns, correlating with Sentinel Graph for lateral risk, and drafting a containment recommendation — in under 60 seconds. This is where partner intellectual property becomes competitive advantage. The agent framework is the mechanism for encoding proprietary detection logic, response playbooks, and domain expertise into automated workflows that run at machine speed. Security Store Security Store allows partners to evolve from one‑time transition projects into repeatable, scalable offerings—supporting professional services, managed services, and agent‑based IP that align with the customer’s unified SecOps operating model As part of the transition, the Microsoft Security Store becomes the extension layer for the Defender —allowing partners to deliver differentiated agents, SaaS, and security services natively within Defender and Sentinel, instead of building and integrating in isolation The 4 Investigation Surfaces: A Customer Maturity Ladder The Sentinel Data Lake exposes four distinct investigation surfaces, each representing a step toward the Agentic SOC — and a partner service opportunity: Surface Capability Maturity Level Partner Opportunity KQL Query Ad-hoc hunting, forensic investigation Basic — “we can query” Hunting query libraries; KQL training Graph Analytics Blast radius, attack paths, entity relationships Intermediate — “we understand relationships” Graph investigation training; attack path workshops Notebooks (PySpark) Statistical analysis, behavioral baselines, ML models Advanced — “we predict behaviors” Custom notebook development; anomaly scoring Agent/MCP Access Autonomous hunting, triage, response at machine speed Agentic SOC — “we automate” Custom agent development; MCP integration The customer who starts with “help us hunt better” ends up at “build us agents that hunt autonomously.” That is the progression from professional services to managed services. What the Transition Actually Involves It is not a data migration — customers’ underlying log data and analytics remain in their existing Log Analytics workspaces. That is important for partners to communicate clearly. But partners should not set the expectation that nothing changes except the URL. Microsoft’s official transition guide documents significant operational changes — including automation rules and playbooks, analytics rule, RBAC restructuring to the new unified model (URBAC), API schema changes that break ServiceNow and Jira integrations, analytics rule transitions where the Fusion engine is replaced by the Defender XDR correlation engine, and data policy shifts for regulated industries. Most customers cannot navigate this complexity without professional help. Important: Transitioning to the Defender portal has no extra cost - estimate the billing with the new Sentinel Cost Estimator Optimizing the unified platform means making deliberate changes: Adding dual-ingest for critical sources that need both real-time detection and long-horizon hunting. Moving high-volume telemetry to the Data Lake — enabling hunting at scale that was previously cost-prohibitive. Retiring redundant data copies where Defender XDR already provides the investigation capability. Updating RBAC, automation, and integrations for the unified portal’s consolidated schema and permission structure. Training analysts on new investigation workflows, Sentinel Graph navigation, and Copilot-assisted triage. Threat Coverage: The Detection Gap Most Organizations Do Not Know They Have This transition is an opportunity to quantify detection maturity — and most organizations will not like what they find. Based on real-world breach analysis — infostealers, business email compromise, human-operated ransomware, cloud identity abuse, vulnerability exploitation, nation-state espionage, and other prevalent threat categories — organizations running standalone Sentinel with default configurations typically have significant detection gaps. Those gaps cluster in three areas: Cross-domain correlation gaps — attacks that span identity, endpoint, email, and cloud workloads. These require the Defender correlation engine because no single log source tells the complete story. Long-retention hunting gaps — threats like command-and-control beaconing and slow data exfiltration that unfold over weeks or months. Analytics-tier retention at 90 days is too expensive to extend and too short for historical pattern analysis. Graph-based analysis gaps — lateral movement, blast radius assessment, and attack path analysis that require understanding entity relationships rather than flat log queries. The unified platform with proper log source coverage across Microsoft-native sources can materially close these gaps — but only if the transition includes a detection coverage assessment, not just a portal cutover. Partners should use MITRE ATT&CK as the common framework for measuring detection maturity. Map existing detections to ATT&CK tactics and techniques before and after transition — a measurable, defensible improvement that justifies advisory fees and ongoing managed services. Partner Opportunity: Professional Services to Managed Services This transition creates a structured progression for all partner types — from professional services that build trust and surface findings, to managed security services that deliver ongoing value. The key insight most partners miss: do not jump from “transition assessment” to “managed services pitch.” Customers are not ready for that conversation until they have experienced the value of professional services. The bridge engagement — whether transactional, transition execution, or advisory — builds trust, demonstrates the expertise, and surfaces the findings that make the managed services conversation a logical next step. Professional Services (transactional + transition execution + advisory) → Managed Security Services (MSSP) The USX transition is the ideal professional services entry point because it combines a mandatory deadline (March 2027) with genuine technical complexity (analytics rule, automation behavioral changes, RBAC restructuring, API schema shifts) that most customers cannot navigate alone. Every engagement produces findings — detection gaps, automation fragility, staffing shortfalls — that are the most credible possible evidence for managed services. Professional Services Transactional Partners Offer Customer Value Key Deliverables Transition Readiness Assessment Risk-mitigated transition with clear scope Sentinel deployment inventory; Defender portal compatibility check; transition roadmap with timeline; MITRE ATT&CK detection coverage baseline Transition Execution and Enablement Accelerated time-to-value, minimal disruption Workspace onboarding; RBAC and automation updates; Dual-portal testing and validation; SOC team training on unified workflows Security Posture and Detection Optimization Better detections and lower cost Data ingestion and tiering strategy; Dual-ingest implementation for critical sources; Detection coverage gap analysis; Automation and Copilot/MCP recommendations Advisory Partners Offer Customer Value Key Deliverables Executive and Strategy Advisory Leadership alignment on why this transition matters Unified SecOps vision and business case; Zero Trust and SOC modernization alignment; Stakeholder alignment across security, IT, and leadership Architecture and Design Advisory Future-ready architecture optimized for the Agentic SOC Target-state 2-tier data architecture; Dual-ingest routing decisions mapped to MITRE tactics; RBAC, retention, and access model design Detection Coverage and Gap Analysis Measurable detection maturity improvement Current-state MITRE ATT&CK coverage mapping; Gap analysis against 24 threat patterns; Detection improvement roadmap with priority recommendations SOC Operating Model Advisory Smooth analyst adoption with clear ownership Redesigned SOC workflows for unified portal; Incident triage and investigation playbooks; RACI for detection engineering, hunting, and platform ops Agentic SOC Readiness Preparation for AI-driven security operations MCP and agent architecture assessment; Custom agent development roadmap; IP + Human Orchestration + Agent operating model design Cost, Licensing and Value Advisory Transparent cost impact with strong business case Current vs. future cost analysis; Data tiering optimization recommendations; TCO and ROI modeling for leadership The conversion to managed services is evidence-based. Every professional services engagement produces findings — detection gaps, automation fragility, staffing shortfalls. Those findings are the most credible possible case for ongoing managed services. Managed Security Services The unified platform changes the managed security conversation. Partners are no longer selling “we watch your alerts 24/7.” They are selling an operating model where proprietary AI agents handle the repeatable work — enrichment, hunting, posture validation, response drafting — and human experts focus on the decisions that require judgment. This is where the competitive moat forms. The formula: IP + Human Orchestration + AI Agents = differentiated managed security. The unified platform enables this through: Multi-tenancy — the built-in multitenant portal eliminates the need for third-party management layers. Sentinel Data Lake — agents can query months of customer telemetry for behavioral analysis without cost constraints. Sentinel Graph — agents can traverse entity relationships to assess blast radius and map attack paths. MCP extensibility — partners can build agents that integrate with proprietary tools and customer-specific systems. Partners who build proprietary agents encoding their detection logic into the MCP framework will differentiate from partners who rely on out-of-box capabilities. The Securing AI Opportunity Organizations are deploying AI agents, copilots, and autonomous workflows across their businesses at an accelerating pace. Every AI deployment creates a new attack surface — prompt injection, data poisoning, agent hijacking, cross-plugin exploitation, unauthorized data access through agentic workflows. These are not theoretical risks. They are in the wild today. Partners who can help customers secure their AI deployments while also using AI to strengthen their SOC will command premium positioning. This requires a security platform that is itself AI Agent-ready — one that can deploy defensive agents at the same pace organizations deploy business AI. The unified Defender portal is that platform. Partners who position USX as “preparing your SOC for AI-driven security operations” will differentiate from partners who position it as “moving to a new portal.” Cost and Operational Benefits Better security architecture also costs less. This is not a contradiction — it is the natural result of putting the right data in the right tier. Benefit How It Works Eliminate low-value ingestion Identify and remove log sources that are never used for detections, investigations, or hunting. Immediately lowers analytics-tier costs without impacting security outcomes. Right-size analytics rules Disable unused rules, consolidate overlapping detections, and remove automation that does not reduce SOC effort. Pay only for processing that delivers measurable security value. Avoid SIEM/XDR duplication Many threats can be investigated directly in Defender XDR without duplicating telemetry into Sentinel. Stop re-ingesting data that Defender already provides. Tier data by detection need Store high-volume, hunt-oriented telemetry in the Data Lake at at least 20x lower cost. Promote only high-signal sources to the analytics tier. Full data fidelity preserved in both tiers. Reduce operational overhead Unified SIEM+XDR workflows in a single portal reduce tool switching, accelerate investigations, simplify analyst onboarding, and enable SOC teams to scale without proportional headcount increases. Improve detection quality The Defender correlation engine produces higher-fidelity incidents with fewer false positives. SOC teams spend less time triaging noise and more time on real threats. Competitive Positioning Partners need defensible talking points when customers evaluate third-party SIEM alternatives. The following advantages are factual, sourced from Microsoft’s transition documentation and platform capabilities — not marketing claims. No extra cost for transitioning — even for non-E5 customers. Third-party SIEM migrations involve licensing, data migration, detection rewrite, and integration rebuild costs. Native cross-domain correlation across Sentinel + Defender products into multi-stage incident graphs. Third-party SIEMs receive Microsoft logs as flat events — they lack the internal signal context, entity resolution, and product-specific intelligence that powers cross-domain correlation. Custom detections across SIEM + XDR — query both Sentinel and Defender XDR tables without ingesting Defender data into Sentinel. Eliminates redundant ingestion cost. Alert tuning extends to Sentinel — previously Defender-only capability, now applicable to Sentinel analytics rules. Net-new noise reduction. Unified entity pages — consolidated user, device, and IP address pages with data from both Sentinel and Defender XDR, plus global search across SIEM and XDR. Third-party SIEMs provide entity views from ingested data only. Built-in multi-tenancy for MSSPs — multitenant portal manages incidents, alerts, and hunting across tenants without third-party management layers. Try out the new GDAP capabilities in Defender portal. Industry validation: Microsoft’s SIEM+XDR platform has been recognized as a Leader by both Forrester (Security Analytics Platforms, 2025) and Gartner (SIEM Magic Quadrant, 2025). Summary: What Partners Should Take Away Topic Key Message Framing USX is a security architecture transformation, not a portal transition. Lead with detection capability, not cost savings. Platform foundation Sentinel Data Lake + Sentinel Graph + MCP/Agent Framework = the platform for the Agentic SOC. 4 investigation surfaces KQL → Graph → Notebooks → Agent/MCP. A maturity ladder from “we can query” to “we automate at machine speed.” Architecture 2-tier data model (analytics + Data Lake) with dual-ingest for critical sources. Cost savings are a side effect of good architecture. Transition complexity Analytics rules and automation rules. API schema changes. RBAC restructuring. Most customers need professional help. Partner engagement model Professional Services (transactional + transition execution + advisory) → Managed Services (MSSP). Competitive positioning No extra cost. Native correlation. Cross-domain detections. Built-in multi-tenancy. Capabilities third-party SIEMs cannot replicate. Partner differentiation IP + Human Orchestration + AI Agents. Partners who build proprietary agents on MCP have competitive advantage. Timeline March 31, 2027. Start now — phased transition with one telemetry domain first, then scale.Integrating ServiceNow with Azure DevOps: Use Cases That Actually Show Up in Practice
I have been working in the integration space for quite some time. Though I work as a Content Manager, I am always curious to see how users handle integrations between multiple systems. Usually, the service desk runs on ServiceNow, and the build happens in Azure DevOps. But these tools aren't inherently connected, and someone has to manually relay information between them. So, the records never show consistent data. You need a way for the data to pass automatically between them. Even within this setup, a lot of scenarios come forth: from a single ServiceNow instance connecting to a single Azure DevOps org, up to companies running six or seven orgs off a shared service layer. Here's the full range of use cases I keep seeing, organized by what they're actually solving. Native Tools vs. a Dedicated Integration Layer Before the use cases, it's worth being clear on when you need more than what's built in. ServiceNow's IntegrationHub can push data through flows, and there's a spoke for looking up and creating basic Azure DevOps work items. Azure DevOps also has a native extension for pipeline-to-change-management syncing. These native options work fine when the integration is simple: one team, clean field mappings, developers on hand to maintain it. They start to struggle once you're crossing org boundaries, crossing company lines, or dealing with fields and states that don't line up between the two systems. At that point, a dedicated integration layer that handles the mapping and keeps each side's access separate is usually the better fit. Core Sync Use Cases 1. Sync ServiceNow Requests to Azure DevOps User Stories Current setup: Support or service teams work in ServiceNow. When a request needs development, someone has to get it onto an Azure DevOps board as a User Story. Problem: A person creates the work item by hand, copies the request details, and checks back later for updates. When the work item closes, the ServiceNow request often doesn't reflect that, so the requester never hears back. The two records aren't linked. Solution: A trigger, filtered by assignment group, category, or a manual send action, creates the User Story automatically. The link between the two records gets stored on both sides. Updates to the work item flow back to the request, and closing one side closes the other. 2. Sync Comments and State Back to ServiceNow Current setup: ServiceNow holds the ticket the requester sees. Azure DevOps holds the actual dev work. Problem: When the work item moves in Azure DevOps, the ServiceNow ticket's state doesn't follow. Comments the dev team adds stay in Azure DevOps, so the service team keeps pinging developers for status. Solution: As the work item's state changes, it updates the linked ServiceNow ticket. Comments from the dev side sync back too. The service team sees progress without opening Azure DevOps or chasing anyone. 3. Sync ServiceNow Catalog Items to Azure DevOps Current setup: Users order catalog items in ServiceNow. Fulfilling some of them takes real development work tracked in Azure DevOps. Problem: The catalog task sits in ServiceNow while the work happens somewhere the requester can't see. Someone bridges the two manually, and the catalog item's status stops meaning anything. Solution: The catalog item creates a matching work item in Azure DevOps with the request details attached. As the dev team works, status and comments sync back, so the catalog task shows real progress and closes when the work item does. 4. Cut Duplication Between Incidents and User Stories Current setup: Incidents land in ServiceNow. Some get tracked as User Stories in Azure DevOps so the dev team can work them alongside the rest of the backlog. Problem: The same information gets documented twice, updated twice, and the two records drift apart over time. Solution: Incidents create linked User Stories automatically, with comments and status syncing both ways. Update one side and the other reflects it, so nobody's working the same incident twice without realizing it. 5. Sync Bugs with Attachments Across Multiple Orgs Current setup: Bugs get raised in ServiceNow, often by a separate first-line support team, while product teams work across several distinct Azure DevOps orgs. Problem: Recreating bugs by hand loses attachments and context, and there's no clean way to route each bug to the right org out of several. When a bug's fixed, ServiceNow doesn't hear about it. Solution: A bug creates a matching Bug work item in the correct Azure DevOps org, attachments included. Status and comments flow back to ServiceNow when it's resolved. Each connection runs independently, so the first-line team stays in ServiceNow without needing access to the other side. 6. Project-to-Hypercare Handover Current setup: During a build, everyone works in Azure DevOps. Once it goes live and moves into maintenance, incidents and requests need to land in ServiceNow, sometimes a brand-new instance. Problem: Teams used to living in Azure DevOps now need a maintenance process in ServiceNow, with no link back to the developers who built the thing. Solution: Incidents raised during hypercare reach the dev team in Azure DevOps, and fixes flow back to ServiceNow. Because the connection is independent of either team's existing workflow, you can add it without disrupting how people already work. 7. Bidirectional Field and Lifecycle Sync Current setup: A demand owner in ServiceNow needs to track a Feature as it moves through scoping, active work, testing, and deployment in Azure DevOps. Problem: If effort, status, and assignment only live in Azure DevOps, the ServiceNow record goes stale. Fields on each side use different names and values, so a raw pass-through just creates mismatches. Solution: Key fields sync in both directions with explicit mappings. From Azure DevOps: iteration path, change reference, estimated effort, assignment, and lifecycle status. From ServiceNow: design documents, portfolio area, demand type, and requirement details. As the Feature moves through its stages, the ServiceNow record updates in real time. 8. Multiple Azure DevOps Orgs to a Central ServiceNow Instance Current setup: Each business unit runs its own Azure DevOps org, often from an acquisition or separate supplier setup. A central ServiceNow layer needs visibility across all of them without forcing a shared workflow. Problem: Without a sync, someone manually pulls data between systems and chases updates. It doesn't scale as orgs get added. Solution: Each org gets its own connection, configured independently, feeding the central ServiceNow instance. Every org keeps its own states and fields while leadership gets a single view across all of them. 9. Selective Field Sharing Between Separately Hosted Environments Current setup: One side runs cloud-hosted, the other on-prem or in a regulated environment, and neither can open full network-level access to the other. Problem: A basic integration either oversprays fields it shouldn't, or requires a network connection security won't approve. Each side wants control over exactly what leaves its environment. Solution: Outgoing and incoming rules configured independently on each side let each admin decide what leaves and what gets accepted. The two environments sync over a secured channel without either side getting direct access to the other's system. 10. Handling Closed and Read-Only Tickets Cleanly Current setup: ServiceNow access controls block updates to closed incidents. Your sync runs across tickets through their full lifecycle, including closure. Problem: A basic sync throws errors every time it tries to write to something already closed. Solution: Filter closed tickets out of the sync entirely so they never trigger a write, or let the errors surface where your team can see and review them. Which approach you pick is a team decision, not a technical constraint. 11. MSP Routing by Assignment Group Current setup: An MSP runs ServiceNow as its operational layer. Clients work in their own delivery tools, some in Azure DevOps, some elsewhere. Manual handoffs work for a couple of clients and fall apart past a handful. Problem: An engineer resolves an incident and forgets to update the client's system. A comment added on one side never reaches the other. Status shows "in progress" on one side long after it's resolved on the other. As client count grows, the gap between what your team knows and what clients see widens. Solution: An automated rule sends each incident to the right client system based on the ServiceNow assignment group. Set a filter (for example, database issues route to one client's system, network issues to another), and connections run independently with no crossover between clients. The original incident reference gets written into the client ticket, and status values map across cleanly. A couple of related patterns worth knowing: Internal notes can be filtered separately from customer-facing updates, so only what the client should see crosses over. A new client typically reuses your existing ServiceNow authentication, so onboarding is closer to registering their system and setting a routing rule than rebuilding the integration from scratch. 12. Conditional Escalation from Multiple ServiceNow Instances Current setup: Several ServiceNow instances feed a single Azure DevOps org. Only tickets that genuinely need developer attention should reach the board. Problem: Escalating everything floods the board with tickets the dev team shouldn't be handling. Solution: A conditional trigger escalates only tickets meeting a specific condition, so agents push what actually qualifies and everything else stays put. Multiple instances can each feed the same Azure DevOps org under their own trigger logic. 13. Demand Management to Feature Delivery Current setup: A demand process runs in ServiceNow end to end: submission, review, approval, and an approved Requirement. The build happens in Azure DevOps as a Feature, through scoping, active work, testing, deployment, and closure. Problem: Once a Requirement's approved, there's no automatic handoff. Someone recreates the Feature by hand and keeps both sides updated manually. The demand owner can't see effort, status, or assignment as work progresses, so the Requirement goes stale. When the Feature closes in Azure DevOps, the Requirement stays open until someone remembers to close it. Solution: A bidirectional sync connects the full flow. Feature creation: A Requirement creates a Feature automatically, carrying demand context (design sign-off, demand reference, portfolio area, stakeholders, description) so developers get what they need without opening ServiceNow. Field sync: Updates flow both ways, each field mapped to a specific direction. Azure DevOps to ServiceNow: iteration path, change reference, effort, priority, assignment, lifecycle state. ServiceNow to Azure DevOps: the demand and design fields above. Lifecycle feedback: As the Feature moves through its stages, effort, status, and assignment feed back into the Requirement in real time. Closure: When the Feature closes, the linked Requirement closes with it, and the demand gets marked fulfilled. ServiceNow stays the system of record for demand, Azure DevOps for execution, and each side keeps authority over what it owns. What Actually Makes These Work A few capabilities show up across almost every use case above: Attachment sync. Bugs and incidents depend on screenshots and logs. If attachments don't carry over, developers end up back on email. Custom field mappings. Out-of-the-box mappings only go so far. Real setups need custom fields like portfolio area or demand type mapped explicitly, with values transformed, not just copied. State mappings. ServiceNow states and Azure DevOps states don't share vocabulary. Map them explicitly, and decide upfront which side is authoritative when a state changes after creation. Selective sync. Filter by assignment group, area path, work item type, or a custom field. Fewer records crossing means less to govern and less risk of leaking internal data across a boundary. Security and access. For most Azure DevOps admins, this comes down to what gets checked in a security review anyway: Entra ID or OAuth over shared credentials, role-based access, audit logging, and real isolation between connections when multiple clients or companies are involved. For a single team with clean, one-direction mappings, a native option usually covers it. Once you're crossing orgs, crossing companies, or running a full demand-to-delivery flow with real field transformations, that's when a dedicated integration layer kind of becomes necessary. Let's discuss whether any of these match what you're dealing with, or if you've hit an edge case that isn't on this list.
The Hidden Reason Your Sentinel Playbook Won't Show Up in Automation Rules (It's Not RBAC)
If a Logic App using the native Microsoft Sentinel incident trigger doesn't show up in the "Run playbook" picker of an Automation Rule — even though permissions, region, and connection are all correct — check the internal action name of the trigger in the JSON code view. If the Logic App was created while the Azure portal was set to a non-English language, the designer may generate a localized action name instead of the expected Microsoft_Sentinel_incident, and the playbook won't be picked up. Environment Microsoft Sentinel (Log Analytics workspace) Logic Apps (Consumption plan) Trigger: native Microsoft Sentinel connector, "Incident" trigger (/incident-creation path) Authentication: system-assigned Managed Identity Same subscription, resource group, and region as the Sentinel workspace Problem Two Logic Apps, both triggered by the native Microsoft Sentinel incident trigger, both using a system-assigned managed identity with the Microsoft Sentinel Responder role granted on the workspace. Same subscription, same resource group, same region. Only one of the two appeared in the "Run playbook" dropdown when configuring an Automation Rule in Sentinel. The other was simply absent — no error message, no warning, nothing in the run history to explain it. Investigation The usual suspects were checked and ruled out one by one: API connection health — one azuresentinel connection was indeed in an "Error" state, but it turned out not to be the one referenced by the affected trigger. Managed identity RBAC — the Microsoft Sentinel Responder role was correctly assigned to the Logic App's managed identity on the Sentinel workspace. User's own RBAC/PIM role — the Logic App Contributor role (required for the account configuring the Automation Rule to even see the resource) was active via PIM at the time of testing. Region/subscription/resource group — identical for both Logic Apps. Trigger type — both used the native "Incident in Microsoft Sentinel" trigger added through the visual designer, not a generic HTTP trigger or a deprecated connector. Every documented requirement was met. The playbook was still invisible. Root Cause Comparing the raw JSON (Logic app > Development tools > Code view) revealed the actual difference. Working Logic App (visible in the picker): "triggers": { "Microsoft_Sentinel_incident": { "type": "ApiConnectionWebhook", "inputs": { "body": { "callback_url": "@{listCallbackUrl()}" }, "path": "/incident-creation" } } } Broken Logic App (invisible in the picker): "triggers": { "Incident_dans_Microsoft_Sentinel": { "type": "ApiConnectionWebhook", "inputs": { "body": { "callback_url": "@listCallbackUrl()" }, "path": "/incident-creation" } } } The connector type and the path (/incident-creation) are identical. The one meaningful structural difference is the trigger's action name: Microsoft_Sentinel_incident versus Incident_dans_Microsoft_Sentinel. The second Logic App's trigger had been added while the Azure portal was set to French. The designer generated a localized action name instead of the canonical English one. Everything points to Sentinel's playbook discovery mechanism scanning Logic App definitions for that exact canonical action name (Microsoft_Sentinel_incident) to identify a resource as a valid Sentinel playbook — rather than relying solely on the connector type or the webhook path, as one would reasonably expect. Fix Renaming the trigger's action key in the JSON to the canonical name was enough: "triggers": { "Microsoft_Sentinel_incident": { ... } } No permission, connection, or region change was required — only this rename. Takeaways The Sentinel Automation Rule playbook picker appears to depend on the exact trigger action name, not just its type or functional behavior — a dependency that isn't documented anywhere in the official Microsoft docs as of this writing. The Azure portal's display language at the time a trigger is added has a direct impact on that Logic App's compatibility with Sentinel Automation Rules. A playbook can be fully functional (running without errors when triggered manually or through an older automation mechanism) while still being invisible in the newer Automation Rule selector — which makes this particular issue easy to miss, since nothing explicitly flags it. Recommendations Temporarily switch the Azure portal to English (Portal settings > Language and region) before adding a Sentinel trigger to a Logic App, if your team usually works in another language. If an existing Logic App won't show up despite an otherwise correct configuration, check the trigger's action name in the JSON code view first — it's a 30-second check that can save hours of RBAC/connection troubleshooting. Document this in your internal runbooks if your team works in a localized portal — this kind of detail is easy to lose and can cost significant troubleshooting time down the line. Has anyone else run into similar localization-related quirks in Sentinel or Logic Apps? Would be curious to hear about other cases in the comments.The Hidden Reason Your Sentinel Playbook Won't Show Up in Automation Rules (It's Not RBAC)
If a Logic App using the native Microsoft Sentinel incident trigger doesn't show up in the "Run playbook" picker of an Automation Rule — even though permissions, region, and connection are all correct — check the internal action name of the trigger in the JSON code view. If the Logic App was created while the Azure portal was set to a non-English language, the designer may generate a localized action name instead of the expected Microsoft_Sentinel_incident, and the playbook won't be picked up. Environment Microsoft Sentinel (Log Analytics workspace) Logic Apps (Consumption plan) Trigger: native Microsoft Sentinel connector, "Incident" trigger (/incident-creation path) Authentication: system-assigned Managed Identity Same subscription, resource group, and region as the Sentinel workspace Problem Two Logic Apps, both triggered by the native Microsoft Sentinel incident trigger, both using a system-assigned managed identity with the Microsoft Sentinel Responder role granted on the workspace. Same subscription, same resource group, same region. Only one of the two appeared in the "Run playbook" dropdown when configuring an Automation Rule in Sentinel. The other was simply absent — no error message, no warning, nothing in the run history to explain it. Investigation The usual suspects were checked and ruled out one by one: API connection health — one azuresentinel connection was indeed in an "Error" state, but it turned out not to be the one referenced by the affected trigger. Managed identity RBAC — the Microsoft Sentinel Responder role was correctly assigned to the Logic App's managed identity on the Sentinel workspace. User's own RBAC/PIM role — the Logic App Contributor role (required for the account configuring the Automation Rule to even see the resource) was active via PIM at the time of testing. Region/subscription/resource group — identical for both Logic Apps. Trigger type — both used the native "Incident in Microsoft Sentinel" trigger added through the visual designer, not a generic HTTP trigger or a deprecated connector. Every documented requirement was met. The playbook was still invisible. Root Cause Comparing the raw JSON (Logic app > Development tools > Code view) revealed the actual difference. Working Logic App (visible in the picker): "triggers": { "Microsoft_Sentinel_incident": { "type": "ApiConnectionWebhook", "inputs": { "body": { "callback_url": "@{listCallbackUrl()}" }, "path": "/incident-creation" } } } Broken Logic App (invisible in the picker): "triggers": { "Incident_dans_Microsoft_Sentinel": { "type": "ApiConnectionWebhook", "inputs": { "body": { "callback_url": "@listCallbackUrl()" }, "path": "/incident-creation" } } } The connector type and the path (/incident-creation) arePower Automate - don't trust the green tick too much
There isn't actually any HTML tag in the text you pasted. The issue is most likely caused by special characters or formatting that Microsoft Tech Community's editor is interpreting incorrectly. The most suspicious part is: Request submitted -> SharePoint item created -> Approval sent -> Status updated -> Notification sent. If you're copying this from another editor, hidden formatting characters can sometimes get carried over. Also, Markdown formatting such as: Succeeded What happened? Input validation may not always paste cleanly depending on where you're posting it. For Microsoft Tech Community, I'd use a plain-text version with simple formatting: A green tick in Power Automate doesn't always mean the process succeeded. This is something I understood better after working with business-critical flows. Imagine a flow like this: Request submitted → SharePoint item created → Approval sent → Status updated → Notification sent. The flow run shows "Succeeded". Looks good. But what if the notification didn't reach the user? Or the approval is still pending after three days? Or the flow was triggered twice and created duplicate records? Or the process stopped after updating the status but before completing the next business step? From Power Automate's point of view, individual actions may have completed successfully. But from the business user's point of view, the process may still be stuck. That's where I think automation design becomes more important than simply knowing which connector or action to use. For important processes, I usually think about: Input validation • Duplicate handling • Timeout scenarios • Error handling • Retry • Logging • Notifications • Recovery • Monitoring Especially with flows running hundreds or thousands of transactions, you can't depend on someone opening the run history every time something goes wrong. The flow should give you enough information to answer: What happened? Where did it fail? Which transaction was affected? Can we safely run it again? For me, that's the difference between a flow that simply works and an automation that can actually be supported in production. What is one Power Automate issue you've faced where the run showed "Succeeded", but the business process wasn't actually completed?Syncing Multiple Azure DevOps Orgs to One ServiceNow Instance Without Forcing a Shared Workflow
If your organization runs more than one Azure DevOps org, whether from an acquisition, a spun-up subsidiary, or business units that never consolidated onto one instance, you already know the visibility gap. Central ServiceNow has no idea what's happening in any of them unless someone checks manually. Your team ends up pulling status updates by hand, chasing changes across orgs, and reconciling what got closed where. That works well for a couple of orgs, but it falls apart past that. Why a Shared Workflow Usually Creates a Bigger Problem Migrating everyone onto a single Azure DevOps org would close the visibility gap on paper. Each org's area paths, iterations, states, and processes took years to get right, and a forced migration undoes all of it. A sync layer between each Azure DevOps org and your central ServiceNow instance closes the same gap without touching how any individual org works day to day. Each org keeps its own configuration. ServiceNow ends up with a rolled-up view across all of them. Common Use Cases Post-Acquisition Org Sprawl Current Setup: A company acquires another company, or runs several business units, each with its own Azure DevOps org and its own way of working. Problem: Central ops has no single view across orgs, and checking each one by hand doesn't scale past a few teams. Solution: Connect each Azure DevOps org to the central ServiceNow instance separately, each with its own sync rules. ServiceNow gets one rolled-up view, and no org has to change how it works. Bi-Directional Status Sync Between Delivery and Support Current Setup: Support logs incidents in ServiceNow. Development tracks the corresponding work in Azure DevOps, sometimes across several orgs. Problem: Support has to ask developers for status or check Azure DevOps boards directly, and developers end up relaying the same update twice. Solution: Sync status, comments, and priority both ways, so an update in either system shows up automatically on the other side. Field-Level Control Per Org Current Setup: Each business unit or subsidiary has its own rules about what data can leave its Azure DevOps org. Problem: A single shared integration with one set of mapping rules risks exposing fields an org never agreed to share outside its own boundary. Solution: Give each org's connection its own outgoing rules, so a subsidiary decides exactly which fields leave its Azure DevOps org, field by field. Handling Closed and Read-Only Work Items Current Setup: ServiceNow blocks writes to closed incidents through ACLs, and Azure DevOps can hit a similar restriction on closed or read-only work items. Problem: A sync that keeps trying to write to a closed item throws the same error repeatedly, and the real problems get buried under the noise. Solution: Filter closed and read-only states out of the sync, or let the errors surface if operations wants visibility into them. What to Evaluate When Choosing an Approach A few criteria matter more than others once you're running this across multiple orgs. Decentralized configuration: does each Azure DevOps org get its own connection and its own rules, or does everything route through one shared setup? Filtering: can you scope the sync with something like WIQL queries on the Azure DevOps side, by area path, iteration, work item type, or tag? Field mapping: does it handle the difference between ServiceNow's field structure and Azure DevOps work item fields without dropping data? Common pairs are ServiceNow State to Azure DevOps State, ServiceNow Priority to Azure DevOps Priority, and ServiceNow Assignment Group to Azure DevOps Area Path. Custom fields usually need explicit mapping rules. Conflict handling: what happens when both sides update the same field at the same time, and what happens with closed or read-only items specifically? Security: Entra ID or OAuth authentication, PAT management per org, role-based access, audit logging, and whatever compliance certifications your security team asks for during review. Direction: bidirectional where both teams update shared fields, one-way where only one side should ever write. Technical Approaches Service Hooks and REST APIs Azure DevOps Service Hooks paired with the ServiceNow REST API give you sync in both directions. A change in Azure DevOps triggers a Service Hook, which calls the ServiceNow API to update the record, and the same flow runs in reverse. This is the most direct route if you're comfortable building and maintaining the webhook logic yourself. Custom Middleware For anything more complex, custom middleware gives you full control over field transformation, routing, and error handling. Azure Functions, Logic Apps, or a small Node.js or Python service usually does the job. The trade-off is maintenance. You own the retry logic, the error handling, and every update when either platform changes its API. Dedicated Integration Platforms Plenty of teams skip building this from scratch and use a dedicated integration platform instead. These typically come with pre-built connectors for both Azure DevOps and ServiceNow, a way to configure field mapping and filters without writing much code, and managed infrastructure so you're not hosting your own sync server. What they usually cover: Pre-configured connectors that already understand both platforms' data structures Visual or scripting configuration for field mapping and filters Managed infrastructure, so nothing runs on your own servers Built-in retry and error handling for API failures Audit logging for tracking what synced and when Support for multi-org routing and conditional logic out of the box The trade-off runs the other way: a subscription cost instead of a one-time build, less control over the exact implementation, and your data passing through a third party's infrastructure. For teams running more than 2 or 3 orgs against one ServiceNow instance, this usually ends up being less overhead than maintaining custom middleware long-term. Every org here has probably solved a version of this differently. Curious what's worked for you, especially with 3 or more Azure DevOps orgs feeding into one ServiceNow instance, and which part of the setup gave you the most trouble.Your Sentinel AMA Logs & Queries Are Public by Default - AMPLS Architectures to Fix That
When you deploy Microsoft Sentinel, security log ingestion travels over public Azure Data Collection Endpoints by default. The connection is encrypted, and the data arrives correctly — but the endpoint is publicly reachable, and so is the workspace itself, queryable from any browser on any network. For many organisations, that trade-off is fine. For others — regulated industries, healthcare, financial services, critical infrastructure — it is the exact problem they need to solve. Azure Monitor Private Link Scope (AMPLS) is how you solve it. What AMPLS Actually Does AMPLS is a single Azure resource that wraps your monitoring pipeline and controls two settings: Where logs are allowed to go (ingestion mode: Open or PrivateOnly) Where analysts are allowed to query from (query mode: Open or PrivateOnly) Change those two settings and you fundamentally change the security posture — not as a policy recommendation, but as a hard platform enforcement. Set ingestion to PrivateOnly and the public endpoint stops working. It does not fall back gracefully. It returns an error. That is the point. It is not a firewall rule someone can bypass or a policy someone can override. Control is baked in at the infrastructure level. Three Patterns — One Spectrum There is no universally correct answer. The right architecture depends on your organisation's risk appetite, existing network infrastructure, and how much operational complexity your team can realistically manage. These three patterns cover the full range: Architecture 1 — Open / Public (Basic) No AMPLS. Logs travel to public Data Collection Endpoints over the internet. The workspace is open to queries from anywhere. This is the default — operational in minutes with zero network setup. Cloud service connectors (Microsoft 365, Defender, third-party) work immediately because they are server-side/API/Graph pulls and are unaffected by AMPLS. Azure Monitor Agents and Azure Arc agents handle ingestion from cloud or on-prem machines via public network. Simplicity: 9/10 | Security: 6/10 Good for: Dev environments, teams getting started, low-sensitivity workloads Architecture 2 — Hybrid: Private Ingestion, Open Queries (Recommended for most) AMPLS is in place. Ingestion is locked to PrivateOnly — logs from virtual machines travel through a Private Endpoint inside your own network, never touching a public route. On-premises or hybrid machines connect through Azure Arc over VPN or a dedicated circuit and feed into the same private pipeline. Query access stays open, so analysts can work from anywhere without needing a VPN/Jumpbox to reach the Sentinel portal — the investigation workflow stays flexible, but the log ingestion path is fully ring-fenced. You can also split ingestion mode per DCE if you need some sources public and some private. This is the architecture most organisations land on as their steady state. Simplicity: 6/10 | Security: 8/10 Good for: Organisations with mixed cloud and on-premises estates that need private ingestion without restricting analyst access Architecture 3 — Fully Private (Maximum Control) Infrastructure is essentially identical to Architecture 2 — AMPLS, Private Endpoints, Private DNS zones, VPN or dedicated circuit, Azure Arc for on-premises machines. The single difference: query mode is also set to PrivateOnly. Analysts can only reach Sentinel from inside the private network. VPN or Jumpbox required to access the portal. Both the pipe that carries logs in and the channel analysts use to read them are fully contained within the defined boundary. This is the right choice when your organisation needs to demonstrate — not just claim — that security data never moves outside a defined network perimeter. Simplicity: 2/10 | Security: 10/10 Good for: Organisations with strict data boundary requirements (regulated industries, audit, compliance mandates) Quick Reference — Which Pattern Fits? Scenario Architecture Getting started / low-sensitivity workloads Arch 1 — No network setup, public endpoints accepted Private log ingestion, analysts work anywhere Arch 2 — AMPLS PrivateOnly ingestion, query mode open Both ingestion and queries must be fully private Arch 3 — Same as Arch 2 + query mode set to PrivateOnly One thing all three share: Microsoft 365, Entra ID, and Defender connectors work in every pattern — they are server-side pulls by Sentinel and are not affected by your network posture. Please feel free to reach out if you have any questions regarding the information provided.Building Microsoft Sentinel Connectors in Minutes with the Sentinel Connector Builder Agent
Overview We previously announced the public preview of the Microsoft Sentinel connector builder agent via VS code extension, that helps developers build Microsoft Sentinel codeless connectors faster with low-code and AI-assisted prompts. This post walks through a hands-on lab using a mock Network Log API to demonstrate how the Sentinel connector builder agent simplifies building Codeless Connector Framework (CCF) pull connectors. Instead of manually creating ingestion infrastructure and configuration files, you’ll use a guided, conversational workflow in VS Code to generate connector artifacts, test them against a live API, and deploy them into Microsoft Sentinel. The lab focuses on the end-to-end experience ranging from API setup to validated connector deployment so you can see how quickly a working integration can be produced. For additional guidance beyond this lab, refer to our MS Learn documentation. The Lab Environment This lab is built around a mock Network Log API hosted as an Azure Function App. The purpose of the lab environment is to give us a live API that we can use to build, validate, and test the Sentinel CCF connector builder agent against end to end. The API exposes 50 synthetic network activity records that look and behave like a real product data source, including web traffic, DNS requests, blocked remote access attempts, malware command-and-control blocks, VPN activity, and other common network events. That makes it a useful stand-in for the type of telemetry many teams want to onboard into Microsoft Sentinel. The API is intentionally shaped like the kind of source a customer might expose for telemetry retrieval. It uses API key authentication through the X-API-Key header, returns paginated results through a nextLink model, and provides a predictable response structure that the builder agent can map into a pull connector configuration. The repo contains everything needed for the walkthrough. There is an ARM template to deploy the Function App, reference documentation for the API, and a sample connector package showing the generated polling config, table schema, DCR, and connector definition. The end goal of the lab is straightforward: use the builder agent to generate a CCF pull connector that ingests this API into the custom NetworkLogAPIGetNetworkLogs_CL table in Sentinel. Prerequisites Before starting, make sure you have the following: Azure subscription -- with Contributor access on a resource group (for deploying the Function App) and Microsoft Sentinel Contributor access on a Sentinel-enabled workspace (for deploying the connector) Microsoft Sentinel workspace -- an existing Log Analytics workspace with Sentinel enabled. See Onboard Microsoft Sentinel to a Log Analytics workspace for more information. Azure CLI -- See How to install the Azure CLI for more information. VS Code with the Microsoft Sentinel for Visual Studio Code extension installed. GitHub Copilot -- with access to premium models. The connector builder agent requires Claude Sonnet 4.5 or 4.6, which uses Copilot premium model credits. Lab Repository -- Once the aforementioned prerequisites are met, you can access the lab repository here: Azure-Sentinel/Tools/CCF-Connector-Builder-Agent-Accelerator at master · Azure/Azure-Sentinel Deploying the Mock API The full CLI commands for this section are available in the repo. For a simpler option, you can use GitHub Copilot to handle the deployment. Enter this prompt: Follow the deployment instructions in Sentinel-CCF-Pull-Connector-Builder-Agent-Accelerator/agent-instructions.md. Let’s deploy the Network Log API and build a CCF pull connector. At a high level, the setup is four steps: clone the repo, create a resource group, ensure you have a Sentinel-enabled workspace, and deploy the Function App using the included ARM template. The template takes two parameters: an ApiKey of your choice (the secret the CCF connector will use to authenticate) and your Log Analytics workspace resource ID for Application Insights. Deployment takes about two to three minutes and outputs the FunctionAppName and endpoint URLs you will need later. Once deployed, verify the API is live: curl -s -H "X-API-Key: <your-api-key>" \ "https://<functionappname>.azurewebsites.net/api/GetNetworkLogs?page=1&pageSize=3" </functionappname></your-api-key> You should see a response like this: The API also exposes an /api/RefreshData endpoint that regenerates the 50 sample records with fresh timestamps. This is useful later in the walkthrough when you want to produce new events and trigger an immediate ingestion cycle without waiting for the next polling interval: curl -s -X POST -H "X-API-Key: <your-api-key>" \ "https://<functionappname>.azurewebsites.net/api/RefreshData" </functionappname></your-api-key> Building the Connector with the Sentinel Connector Builder Agent With the Microsoft Sentinel extension installed and GitHub Copilot running in agent mode, open a Copilot chat and enter a single prompt pointing at the API documentation file: That is the entire invocation. The agent takes it from there. It works through a structured seven-step sequence: preparation, polling config, table schema, DCR, connector definition, package validation, and summary. The agent produces four files in a sentinel-connectors/NetworkLogAPI_CCF/ output folder: NetworkLogAPI_PollingConfig.json – This is the API poller configuration. The agent reads the documentation and correctly identifies the GET /api/GetNetworkLogs endpoint, configures API Key authentication via the X-API-Key header, sets up NextPageUrl pagination using $.metadata.nextLink with a $.metadata.hasNextPage stop condition, and wires up the since query parameter for incremental delta pulls using the timestamp field. The RefreshData endpoint is correctly excluded, which the agent recognizes as a maintenance operation, not a security data stream. NetworkLogAPI_Table.json – This is the custom Log Analytics table schema for NetworkLogAPIGetNetworkLogs_CL . All 20 fields from the API response are mapped to the correct column types, with timestamp promoted to TimeGenerated as the standard Sentinel time column. NetworkLogAPI_DCR.json – This is the Data Collection Rule. This defines the stream declaration, the workspace destination, and the KQL transform that maps the raw snake_case API fields ( sourceIp , destinationIp , threatIndicator , etc.) to their PascalCase table columns. NetworkLogAPI_ConnectorDefinition.json – This is the connector UI configuration. This drives what the connector page looks like in Microsoft Sentinel: the title, description, prerequisite instructions, the BaseUrl and ApiKey input fields, sample KQL queries, and the connectivity status logic. The only point where the agent paused for input was to propose a connector description and ask for confirmation before writing it to the file. Everything else such as endpoint selection, auth type, pagination pattern, schema mapping, KQL transform, cross-file consistency was selected autonomously. To put that in perspective: without the agent, a developer building this connector from scratch would need to manually author four JSON files, understand the CCF schema for polling configs, DCRs, and connector definitions, write the KQL transform by hand, and validate that every cross-file reference lines up correctly. The agent compresses that work, typically hours of reading documentation, trial-and-error, and portal debugging, into a single prompt. Testing the Connector Before deploying anything to a Sentinel workspace, the Microsoft Sentinel connector builder agent lets you validate the generated polling config against the live API directly from your editor. Right-click the sentinel-connectors/NetworkLogAPI_CCF folder, select Microsoft Sentinel → Test Connector (Preview), and a Configuration Variables panel opens asking for the two template variables from the polling config: BaseUrl and apiKey . For other API patterns, there may be additional and different inputs. For example, apiKey input could be swapped with clientID and secret if the API supports OAUTH. Enter the Function App base URL and your API key, and the test runner connects immediately. The panel shows a live polling session. Poll #1 returns HTTP 200 with 50 events, and a countdown timer shows when the next poll will fire. Switching to the Events tab displays the ingested records in a tabular view with columns for timestamp , severity , action , bytesIn , bytesOut , category , and the rest of the mapped fields fresh from the API. Additionally, there are tabs for Headers, Payload, and Response, which can be useful for verifying that your pollerconfig.json configuration provides the expected request to your api with a working response. Data Extracted: The Test Connector feature can be used to visualize the response data in a table format to verify that data will land in a Sentinel table based on your configuration. Request from Poller: The Test Connector feature can be used to validate the request and response headers that will go out to the API based on the generated poller configuration. Request Response: The Test Connector feature shows you the live response from the API with respect to the request going to the API based on the poller configuration. This is a meaningful pre-flight check. It confirms that auth is working, the $.data events path resolves correctly, pagination is functional, and the polling interval fires as configured all before a single file is deployed to Azure. The most common connector configuration issues (wrong base URL, incorrect header name, mismatched JSON path) surface here in seconds rather than after a failed deployment and a 20-minute wait for Sentinel to attempt its first ingestion cycle. It is also the fastest way to troubleshoot if something goes wrong after deployment, far quicker than pushing changes to Azure and waiting for the connector to poll again. Deploying and Enabling the Connector With the connector tested and passing, deployment is the same right-click menu: right-click the sentinel-connectors/NetworkLogAPI_CCF folder, select Microsoft Sentinel → Deploy Connector (Preview). If you are not already signed in to Azure, the extension will prompt you to authenticate. The agent will also provide a clickbox in the chat window to invoke a connector deployment. Right Click Deploy Connector: UI Prompt Based Deploy Method: Once signed in, a workspace picker lists all available Log Analytics workspaces across your subscriptions. Select the one with Sentinel enabled and click Deploy. The extension deploys all four files to the workspace in the correct order: table schema first, then DCR, polling config, and connector definition. Once deployed, navigate to your Sentinel workspace via https://security.microsoft.com, go to Data Connectors, and find the Network Log API connector. The connector page shows the description, prerequisite notes, and the two credential fields generated by the agent: API Base URL and API Key. Enter your Function App base URL and API key and click Connect. The status updates to show the connector is connected and the deployment succeeded. Note: Data will appear in the workspace within 5 to 30 minutes depending on the polling interval. Run this query in Log Analytics to confirm ingestion. Note that the agent derives the table name from the vendor name and endpoint, so yours may differ slightly from the example below. Check the agent's summary output or the NetworkLogAPI_Table.json file for the exact name: NetworkLogAPIGetNetworkLogs_CL | sort by TimeGenerated desc | take 10 If you want to generate a fresh batch of events immediately rather than waiting for the next polling cycle, use the RefreshData endpoint to reset the sample records with new timestamps: curl -s -X POST -H "X-API-Key: " \ "https://.azurewebsites.net/api/RefreshData" Next Steps If you want to go further: Try it with your own API. The lab repo includes documentation on adapting the polling config, schema, and KQL transform to a real data source. Review the CCF connector schema documentation to understand the full range of supported configurations: pagination patterns, auth types, incremental pull strategies, and delta filter expressions. Explore the Microsoft Sentinel content hub to see how published connectors are structured and what the certification requirements look like for production submissions. Conclusion Following these steps, you saw how a working Sentinel connector can be generated, tested, and deployed in minutes rather than requiring days of manual configuration and infrastructure setup. If you are an ISV building a Sentinel integration and want hands-on support, Microsoft’s App Assure program is available to help. We partner with ISVs on connector development, validation, and deployment and provide guidance through implementation, testing, and readiness for production. You can get started by reaching out through our intake form. See our other Sentinel connector feature’s hands-on labs Building a CCF Nested API Pull Connector: A Technical Lab WalkthroughSensitivity Auto-labelling via Document Property
Why is this needed? Sensitivity labels are generally relevant within an organisation only. If a file is labelled within one environment and then moved to another environment, sensitivity label content markings may be visible, but by default, the applied sensitivity label will not be understood. This can lead to scenarios where information that has been generated externally is not adequately protected. My favourite analogy for these scenarios is to consider the parallels between receiving sensitive information and unpacking groceries. When unpacking groceries, you might sit your grocery bag on a counter or on the floor next to the pantry. You’ll likely then unpack each item, take a look at it and then decide where to place it. Without looking at an item to determine its correct location, you might place it in the wrong location. Porridge might be safe from the kids on the bottom shelf. If you place items that need to be protected, such as chocolate, on the bottom shelf, it’s not likely to last very long. So, I affectionately refer to information that hasn’t been evaluated as ‘porridge’, as until it has been checked, it will end up on the bottom shelf of the pantry where it is quite accessible. Label-based security controls, such as Data Loss Prevention (DLP) policies using conditions of ‘content contains sensitivity label’ will not apply to these items. To ensure the security of any contained sensitive information, we should look for potential clues to its sensitivity and then utilize these clues to ensure that the contained information is adequately protected - We take a closer look at the ‘porridge’, determine whether it’s an item that needs protection and if so, move it to a higher shelf in the pantry so that it’s out of reach for the kids. Effective use of Purview revolves around the use of ‘know your data’ strategies. We should be using as many methods as possible to try to determine the sensitivity of items. This can include the use of Sensitive Information Types (SITs) containing keyword or pattern-based classifiers, trainable classifiers, Exact Data Match, Document fingerprinting, etc. Matching items via SITs present in the items content can be problematic due to false positives. Keywords like ‘Sensitive’ or ‘Protected’ may be mentioned out of context, such as when referring to a classification or an environment. When classifications have been stamped via a property, it allows us to match via context rather than content. We don’t need to guess at an item’s sensitivity if another system has already established what the item’s classification is. These methods are much less prone to false positives. Why isn’t everyone doing this? Document properties are often not considered in Purview deployments. SharePoint metadata management seems to be a dying artform and most compliance or security resources completing Purview configurations don’t have this skill set. There’s also a lack of understanding of the relevance of checking for item properties. Microsoft haven’t helped as the documentation in this space is somewhat lacking and needs to be unpicked via some aligning DLP guidance (Create a DLP policy to protect documents with FCI or other properties). Many of these configurations will also be tied to regional requirements. Document properties being used by systems where I’m from, in Australia, will likely be very different to those used in other parts of the world. In the following sections, we’ll take a look at applicable use cases and walk through how to enable these configurations. Scenarios for use Labelling via document property isn’t for everyone. If your organisation is new to classification or you don’t have external partners that you collaborate with at higher sensitivity levels, then this likely isn’t for you. For those that collaborate heavily and have a shared classification framework, as is often seen across government, this is a must! This approach will also be highly relevant to multi-tenant organisations or conglomerates where information is regularly shared between environments. The following scenarios are examples of where this configuration will be relevant: 1. Migrating from 3 rd party classification tools If an item has been previously stamped by a 3 rd party classification tool, then evaluating its applied document properties will provide a clear picture of its security classification. These properties can then be used in service-based auto-labelling policies to effectively transition items from 3 rd party tools to Microsoft Purview sensitivity labels. As labels are applied to items, they will be brought into scope of label-based controls. 2. Detecting data spill Data spill is a term that is used to define situations where information that is of a higher than permitted security classification land in an environment. Consider a Microsoft 365 tenant that is approved for the storage of Official information but Top Secret files are uploaded to it. Document properties that align with higher than permitted classifications provide us with an almost guaranteed method of identifying spilled items. Pairing this document property with an auto-labelling policy allows for the application of encryption to lock unauthorized users out of the items. Tools like Content Explorer and eDiscovery can then be used to easily perform cleanup activities. If using document properties and auto-labelling for this purpose, keep in mind that you’ll need to create sensitivity labels for higher than permitted classifications in order to catch spilled items. These labels won’t impact usability as you won’t publish them to users. You will, however, need to publish them to a single user or break glass account so that they’re not ignored by auto-labelling. 3. Blocking access by AI tools If your organization was concerned about items with certain properties applied being accessed by generative AI tools, such as Copilot, you could use Auto-labelling to apply a sensitivity label that restricts EXTRACT permissions. You can find some information on this at Microsoft 365 Copilot data protection architecture | Microsoft Learn. This should be relevant for spilled data, but might also be useful in situations where there are certain records that have been marked via properties and which should not be Copilot accessible. 4. External Microsoft Purview Configurations Sensitivity labels are relevant internally only. A label, in its raw form, is essentially a piece of metadata with an ID (or GUID) that we stamp on pieces of information. These GUIDs are understood by your tenant only. If an item marked with a GUID shows up in another Microsoft 365 tenant, the GUID won’t correspond with any of that tenant’s labels or label-based controls. The art in Microsoft Purview lies in interpreting the sensitivity of items based on content markings and other identifiers, so that data security can be maintained. Document properties applied by Purview, such as ClassificationContentMarkingHeaderText are not relevant to a specific tenant, which makes them portable. We can use these properties to help maintain classifications as items move between environments. 5. Utilizing metadata applied by Records Management solutions Some EDRMS, Records or Content Management solutions will apply properties to items. If an item has been previously managed and then stamped with properties, potentially including a security classification, via one of these systems, we could use this information to inform sensitivity label application. 6. 3 rd party classification tools used externally Even if your organisation hasn’t been using 3rd party classification tools, you should consider that partner organisations, such as other Government departments, might be. Evaluating the properties applied by external organisations to items that you receive will allow you to extend protections to these items. If classification tools like Janus or Titus are used in your geography/industry, then you may want to consider checking for their properties. Regarding the use of auto-classification tools Some organisations, particularly those in Government, will have organisational policies that prevent the use of automatic classification capabilities. These policies are intended to ensure that each item is assessed by an actual person for risk of disclosure rather than via an automated service that could be prone to error. However, when auto-labelling is used to interpret and honour existing classifications, we are lowering rather than raising the risk profile. If the item’s existing classification (applied via property) is ignored, the item will be treated as porridge and is likely to be at risk. If auto-labelling is able to identify a high-risk item and apply the relevant label, it will then be within scope of Purview’s data security controls, including label-based DLP, groups and sites data out of place alerting, and potentially even item encryption. The outcome is that, through the use of auto-labelling, we are able to significantly reduce risk of inappropriate or unintended disclosure. Configuration Process Setting up document property-based auto-labelling is fairly straightforward. We need to setup a managed property and then utilize it an auto-labelling policy. Below, I've split this process into 6 steps: Step 1 – Prepare your files In order to make use of document properties, an item with the properties applied will first need to be indexed by SharePoint. SharePoint will record the properties as ‘crawled properties’, which we’ll then need to convert into ‘managed properties’ to make them useful. If you already have items with the relevant properties stored in SharePoint, then they are likely already indexed. If not, you’ll need to upload or create an item or items with the properties applied. For testing, you’ll want to create a file with each property/value combination so that you can confirm that your auto-labelling policies are all working correctly. This could require quite a few files depending on the number of properties you’re looking for. To kick off your crawled property generation though, you could create or upload a single file with the correct properties applied. For example: In the above, I’ve created properties for ClassificationContentMarkingHeaderText and ClassificationContentMarkingFooterText, which you’ll often see applied by Purview when an item has a sensitivity label content marking applied to it. I’ve also included properties to help identify items classified via JanusSeal, Titus and Objective. Step 2 – Index the files After creating or uploading your file, we then need SharePoint to index it. This should happen fairly quickly depending on the size of your environment. I'd expect to wait sometime between 10 minutes and 24 hrs. If you're not in a hurry, then I'd recommend just checking back the next day. You'll know when this has been completed when you head into SharePoint Admin > Search > Managed Search Schema > Crawled Properties and can find your newly indexed properties: Step 3 – Configure managed properties Next, the properties need to be configured as managed properties. To do this, go to SharePoint Admin > More features > Search > Managed Search Schema > Managed Properties. Create a new managed property and give it a name. Note that there are some character restrictions in naming, but you should be able to get it close to your document property name. Set the property’s type to text, select queryable and retrievable. Under ‘mappings to crawled properties’, choose add mapping, search for and select the property indexed from the file property. Note that the crawled property will have the same name as your document property, so there’s no need to browse through all of them: Repeat this so that you have a managed property for each document property that you want to look for. Step 4 – Configure Auto-labelling policies Next up, create some auto-labelling policies. You’ll need one for each label that you want to apply, not one per property as you can check multiple properties within the one auto-labelling policy. - From within Purview, head to Information Protection > Policies > Auto-labelling policies. - Create a new policy using the custom policy template. - Give your policy an appropriate name (e.g. Label PROTECTED via property). - Select the label that you want to apply (e.g. PROTECTED). - Select SharePoint based services (SharePoint and OneDrive). - Name your auto-labelling rules appropriately (e.g. SPO – Contains PROTECTED property) - Enter your conditions as a long string with property and value separated via a colon and multiple entries separated with a comma. For example: ClassificationContentMarkingHeaderText:PROTECTED,ClassificationContentMarkingFooterText:PROTECTED,Objective-Classification:PROTECTED,PMDisplay:PROTECTED,TitusSEC:PROTECTED Note that the properties that you are referencing are the Managed Property rather than the document property. This will be relevant if your managed property ended up having a different name due to character restrictions. After pasting in your string into the UI, the resultant rule should look something like this: When done, you can either leave your policy in simulation mode or save it and then turn it on from the auto-labelling policies screen. Just be aware of any potential impacts, such as accidently locking users out by automatically deploying a label with encryption configuration. You can reduce any potential impact by targeting your auto-labelling policy at a site or set of sites initially and then expanding its scope after testing. Step 5 - Test Testing your configuration will be as easy as uploading or creating a set of files with the relevant document properties in place. Once uploaded, you’ll need to give SharePoint some time to index the items and then the auto-labelling policy some time to apply sensitivity labels to them. To confirm label application, you can head to the document library where your test files are located and enable the sensitivity column. Files that have been auto-labelled will have their label listed: You could also check for auto-labelling activity in Purview via Activity explorer: Step 6 – Expand into DLP If you’ve spent the time setting up managed properties, then you really should consider capitalizing on them in your DLP configurations. DLP policy conditions can be configured in the same manner that we configured Auto-labelling in Step 3 above. The document property also gives us an anchor for DLP conditions that is independent of an item’s sensitivity label. You may wish to consider the following: DLP policies blocking external sharing of items with certain properties applied. This might be handy for situations where auto-labelling hasn’t yet labelled an item. DLP policies blocking the external sharing of items where the applied sensitivity label doesn’t match the applied document property. This could provide an indication of risky label downgrade. You could extend such policies into Insider Risk Management (IRM) by creating IRM policies that are aligned with the above DLP policies. This will allow for document properties to be considered in user risk calculation, which can inform controls like Adaptive Protection. Here's an example of a policy from the DLP rule summary screen that shows conditions of item contains a label or one of our configured document properties: Thanks for reading and I hope this article has been of use. If you have any questions or feedback, please feel free to reach out.
