rss.livelink.threads-in-node

How MS Discovery Is Empowering Scientists to Do More

sameeraman — Sun, 03 May 2026 12:05:56 GMT

Research and development has traditionally been a slow, sequential, and largely manual endeavour. Scientists formulate hypotheses, design experiments, run computations in constrained environments, and document results, each stage dependent on the last, each transition requiring human review and intervention. Knowledge is fragmented across systems, insights are bottlenecked by individual capacity, and the gap between hypothesis and actionable outcome can span weeks or months.

For organisations tackling complex scientific and operational challenges, from drug discovery to industrial process optimisation, this pace of iteration is simply no longer acceptable.

At Microsoft, we recently introduced Microsoft Discovery, a platform that I believe fundamentally changes this model. Much like Microsoft 365 transformed the way knowledge workers collaborate and create, Microsoft Discovery is designed to simplify and empower the way scientists and researchers work. It provides a unified, end-to-end platform that integrates advanced artificial intelligence, high-performance computing, and knowledge management to support the full scientific reasoning lifecycle: knowledge gathering, hypothesis generation, experiment design, simulation, results analysis, and documentation.

In this article, I want to share how we used Microsoft Discovery to automate a real-world simulation workflow for a mining organisation and what that experience taught our team about the future of AI-augmented science.

What Is Microsoft Discovery?

Microsoft Discovery is Microsoft's scientific AI platform, a solution designed to accelerate research and experimentation across the full innovation lifecycle. Rather than replacing scientific judgement, Discovery is designed to amplify human expertise, embedding AI assistance at each stage of the R&D process while maintaining governance, traceability, and scientific rigour.

From Traditional R&D to AI-Augmented Science

To appreciate what Discovery enables, it is important to understand where it fits in.

In the traditional R&D model, knowledge discovery centres on manual literature reviews and historical data analysis. Researchers individually search, read, and synthesise information which is a time-intensive process where discovery is limited by each person's capacity to locate and interpret relevant material. Hypothesis generation and experimental design are expert-led and largely manual. Computational experimentation, where it exists, runs in fixed or constrained environments with limited parallelism. Analysis and iteration follow the same sequential pattern: execute, review, document, repeat.

Microsoft Discovery changes this fundamentally. In the AI-cloud-enabled model it provides:

Knowledge synthesis at scale — Researchers can explore literature, historical experiments, and organisational knowledge through a single interface, with intelligent indexing surfacing insights faster than manual search could ever achieve.
AI-assisted hypothesis generation — Collaborative human-and-AI workflows support hypothesis exploration and feasibility assessment, while final decisions remain with the scientist.
Cloud-scale experimentation — Elastic compute and parallel processing allow simulations and experiments to run at scale, with integrated tracking and reproducibility built in.
Continuous feedback and human-in-the-loop governance — Results are analysed and compared more rapidly, enabling faster iteration, with AI-generated insights reviewed and validated by researchers before action.
Governed knowledge assets — Experiment lineage, outcomes, and best practices are captured as reusable, governed assets, supporting long-term organisational learning.

The net effect is a transition from slow, manual, and fragmented research processes to an agile, automated, and data-driven R&D model — one that improves research efficiency, increases the return on innovation investment, and enables faster, higher-impact solutions to complex challenges. In high level, the research and deveolopment loop we discussed and how Microsoft Discovery enriches it show in the following diagram.

The Real-World Problem: Screening Thousands of Molecules

To bring this to life, let me walk you through a real-world use case we worked on recently. A mining organisation needed to identify the best-performing oxidant compounds for a chemical reaction central to their operations. We will be talking about only a workflow that sits squarely in the simulation phase of the scientific loop — and it is a perfect example of the kind of work that Microsoft Discovery can strongly transform.

How Scientists Did It Before

In the traditional process, scientists would begin by selecting candidate molecules from established molecular libraries based on characteristics identified through literature review. These libraries can contain thousands of molecules, each defined in standard molecular file formats (such as XYZ or CIF files) that describe their three-dimensional atomic structures.

From there, a researcher would manually work through a multi-step pipeline:

Pre-processing and preparation: The selected molecular files are processed and prepared for quantum mechanical (QM) calculations. This involves filtering molecules based on properties like the types of metals present, electron count, and atomic weight — criteria that directly affect both the scientific relevance and the computational cost of the simulations. The output is a set of prepared input files (known as GJF files) ready for simulation.
Running quantum mechanical simulations: The prepared input files are submitted to a computational chemistry tool (Gaussian 16) to perform Density Functional Theory (DFT) calculations. These simulations compute the electronic structure and energy states of each molecule across different charge and multiplicity configurations. Crucially, each molecule requires multiple independent simulation runs, and the computational cost scales rapidly with molecular complexity. With thousands of candidate molecules, this step alone can involve thousands of individual simulation jobs.
Collecting and post-processing results: Once all simulations complete, the output log files are collected and processed. For each molecule, the lowest-energy charge and multiplicity combination is identified, and a set of quantum mechanical descriptors and classical molecular descriptors are extracted. These descriptors are then fed into a trained machine learning model to predict the redox potential of each compound, a key metric that indicates how effectively a molecule can act as an oxidant in the target reaction.
Summarisation and filtering: Finally, the predicted redox potentials and other relevant characteristics are compiled into a summary, enabling researchers to identify the most promising candidates for further investigation and experimental validation.

Every step in this pipeline required manual intervention: writing and adjusting scripts, verifying input and output files, monitoring job queues, handling failures, and stitching results together. A single researcher could easily spend days or weeks moving through this process — and any error at one stage meant going back and re-running subsequent steps.

How We Automated This with Microsoft Discovery Agents

When we looked at this workflow through the lens of Microsoft Discovery, the opportunity was clear. The scientific reasoning, selecting which molecules to test, interpreting redox potential results, deciding what to investigate next, should remain with the researcher. But the operational overhead of preparing files, submitting simulations, monitoring jobs, collecting results, and assembling summaries? That could be orchestrated by a team of AI agents.

A Team of Agents, Working Together

We designed a multi-agent architecture within Microsoft Discovery to automate this simulation workflow end to end. Here is how the team of agents operates:

Router Agent: The entry point. When a researcher submits a request for example, asking to run QM calculations on a set of candidate molecules the Router Agent interprets the intent and orchestrates the downstream workflow.

Planner Agent: Once the Router Agent identifies the task, the Planner Agent examines the input files provided by the researcher and formulates a step-by-step execution plan. It determines what needs to happen, in what order, and with what parameters, much like a project manager scoping out a piece of work.

Gaussian Prep Agent: This agent handles the preparation step. It is intelligent enough to inspect the current molecular files, apply the necessary filtering criteria, and prepare them for simulation, generating the input files that the computational chemistry tool requires. What previously involved manual scripting and file-by-file verification is now handled autonomously. We used Microsoft Discovery tools to do the underlying execution with this agent.

MPI Gaussian Agent: This is where the power of cloud-scale computing comes in. The Gaussian Agent submits the prepared simulation jobs and manages their execution using an MPI-based master-worker pattern. This approach enables massive parallel execution scaling out across the cloud to run thousands of simulations concurrently rather than sequentially. Given that the candidate molecule libraries can contain thousands of entries, and each molecule may require multiple simulation runs, this parallel execution capability is transformative. What might have taken days in a constrained local environment can now complete in a fraction of the time.

Redox Potential Agent: Once the simulations are complete, this agent takes over. It processes the simulation outputs, identifies the optimal charge and multiplicity state for each molecule, extracts the relevant QM and classical descriptors, and runs them through the trained machine learning model to predict redox potentials.

Summariser Agent: The final agent in the chain. It maps the predicted redox potentials back to the original molecules, applies any additional filtering criteria, and produces a clean, structured summary a JSON file that the researcher can immediately use to identify the most promising candidates and take them forward into the next phase of their work.

What the Researcher Experiences

From the scientist's perspective, the transformation is striking. Instead of spending days writing scripts, babysitting job queues, and manually stitching results together, they provide their input files and describe what they need. The agents take it from there planning, preparing, executing, processing, and summarising and deliver a curated output ready for scientific interpretation.

The researcher's time is freed to focus on what matters most: thinking critically about the science. Which molecules look most promising? What does the redox potential distribution tell us? Should we adjust the filtering criteria and run another round? These are the high-value questions that require human expertise and now scientists can spend their time on exactly that, rather than on operational mechanics.

The Bigger Picture: Accelerating the Entire Scientific Loop

It is important to note that this simulation workflow is just one piece of the broader scientific loop. The full cycle of scientific research, from initial knowledge gathering and literature review, through hypothesis generation, experimental design, simulation, results analysis, and documentation involves many stages, each of which can benefit from the same kind of AI-augmented approach.

Microsoft Discovery is designed to support this entire cycle. In our project, we did not stop at simulation. We also explored how agents can accelerate the knowledge gathering phase, helping researchers navigate vast bodies of literature and surface relevant prior work more efficiently. We looked at how AI can assist with hypothesis generation and evaluation, helping scientists reason about which directions are most promising before committing to expensive computations. And we examined how agents can support the analysis and reporting phases comparing results against hypotheses, generating visualisations, and even assisting with drafting research documents.

What excites me most about Microsoft Discovery is not any single capability, but the cumulative effect of embedding AI assistance across every stage of the research process. Each phase that gets faster and more efficient creates a multiplier effect on the phases that follow. When knowledge gathering takes hours instead of weeks, researchers generate better hypotheses sooner. When simulations run at cloud scale in parallel, results arrive faster. When analysis is augmented by AI, iteration cycles tighten. The entire loop accelerates.

Conclusion

The way we approach scientific research is undergoing a fundamental shift. Large language models and the AI agents built from them are not replacing scientists, they are empowering them to work at a pace and scale that was previously unimaginable.

Microsoft Discovery represents a new operating model for R&D. By combining advanced AI, high-performance cloud computing, and intelligent workflow orchestration, it enables researchers to offload the repetitive, time-consuming operational work to agents and invest their expertise where it has the greatest impact: in asking better questions, interpreting complex results, and pushing the boundaries of what we know.

In the use case I have shared here, a team of six AI agents automated a simulation pipeline that would have taken a single researcher days of manual work. They prepared molecular input files, scaled out thousands of quantum mechanical simulations in parallel across the cloud, processed the results, predicted redox potentials using machine learning, and delivered a structured summary all with minimal human intervention.

This is just the beginning. As AI agents become more capable and the tools surrounding them more mature, the potential to accelerate discovery across every scientific domain is immense. Whether you are in materials science, pharmaceuticals, energy, agriculture, or any field where complex R&D is central to progress, Microsoft Discovery offers a platform to do more, faster, and with greater confidence.

The future of science is not about working harder. It is about working smarter with AI as your partner in discovery.

My Azure DevOps Git configuration failed to load in Azure Data Factory

Plaigie — Sun, 03 May 2026 04:25:52 GMT

I have an Azure Data Factory that have been running for years and have been hooked up with DevOps Git configuration with no real automation, its just a place to store changes. Im using it for a monthly data transformation, so im not into it on a daily basis.

Suddenly yesterday the Github repository failed to load and i cant figure out why.

I have been into the app registration to see if any secrets have expired, and i have one that expires in 25 days, but that shouldnt be a problem yet.

I can also open my Github project through devops and navigate to the service connections, so i still have access to the repository, i just cant get ADF to load it properly.

Any ideas about what to do from here?

Windows 12 — Almost Ready for Its Big Reveal?

kikero_exe — Sun, 03 May 2026 13:16:59 GMT

I’ve been testing the new build of the upcoming system for two days now, and it’s clear that Microsoft is gradually addressing several key areas:

🎧 Audio pipeline

It’s finally starting to behave much more consistently.
It no longer inflates as aggressively as in previous builds (where it often jumped to 8–13 GB).
Crackling and robotic artifacts appear less frequently and stabilize much faster.
Overall audio quality is cleaner and more natural — you no longer get the feeling that some EQ or background service is distorting the sound.

⚙️ System optimization

System responsiveness, transitions, memory management, and in‑game performance have all noticeably improved.
The system feels tighter, faster, and free of unnecessary stutters.

🟢 Build stability

The build is now in such good shape that it could realistically be released to the public.
Long‑session stability (2–3 hours and more) is excellent, with no memory leaks or CPU spikes.

🎨 New UI

If you’re wondering about the new UI — it’s still locked behind cloud flags.
It doesn’t activate in this build yet, even though the system already feels prepared for it.

Modernizing Terraform Pipelines on Azure: OIDC Federation for GitHub Actions and Azure DevOps

ssinghkalra — Sat, 02 May 2026 20:34:19 GMT

The secret nobody wants to rotate

Most Terraform-on-Azure pipelines we see still authenticate the same way they did three years ago. A long-lived ARM_CLIENT_SECRET sitting in GitHub Actions or Azure DevOps, set once, copied around, and rotated only when something breaks.

It's the most ignored credential in the cloud, and statistically the most likely one to leak. A developer screenshots a variable group. A pipeline log echoes a value. A fork inherits a secret. Or the secret simply expires on a Friday evening and takes production deployments with it.

Workload Identity Federation (WIF) makes this whole class of problem go away. The pipeline mints a short-lived token at runtime, exchanges it for an Azure access token via Microsoft Entra, and never touches a secret. GitHub Actions has supported it since 2021. Azure DevOps service connections went GA with WIF in February 2024. The azurerm Terraform provider has supported it since v3.7.

This post walks through the pattern end-to-end, for both GitHub Actions and Azure DevOps, the way I've rolled it out across multiple customer estates.

How the exchange actually works

Before any YAML, it helps to picture what's happening:

The CI system (GitHub or ADO) signs a short-lived JWT describing exactly what's running- which repo, which branch, which environment, which service connection.
The pipeline sends that JWT to Microsoft Entra ID.
Entra checks it against a federated identity credential you've configured on a managed identity or app registration. The iss, sub, and aud claims must match case-sensitively.
If it matches, Entra returns an Azure access token valid for the duration of the job.
Terraform uses it. The job ends. The token expires. Nothing persists.

The token is bound to a specific subject like repo:contoso/platform:environment:prod or sc://contoso/platform/azure-prod. It can't be reused from another repo, branch, or pipeline.

Recommended Architecture

A few choices that usually hold up in production:

Decision	Choice
Identity type	User-assigned managed identity (UAMI), not app registration
Identity granularity	One UAMI per environment (not per pipeline)
Trust scope	Pinned to the environment claim, not the branch
RBAC scope	Resource group, not subscription
Remote state	OIDC + use_azuread_auth = true, shared key access disabled

Why UAMIs? They live in your subscription, don't need Application Administrator rights to manage, and follow the lifecycle of the resource group they belong to. Why one per environment? Pipeline-per-identity explodes into hundreds of identities. Environment-per-identity maps cleanly to deployment scopes.

Part 1 - GitHub Actions

Step 1: Create the identity and federate it

Two commands per environment. That's it.

az identity create -g rg-platform-identity -n id-tf-prod -l eastus az identity federated-credential create \ --name github-prod \ --identity-name id-tf-prod \ --resource-group rg-platform-identity \ --issuer https://token.actions.githubusercontent.com \ --subject repo:contoso/platform:environment:prod \ --audiences api://AzureADTokenExchange

Repeat for nonprod. No secret is created anywhere.

Step 2: Wire it up in GitHub

In repo Settings → Environments, create nonprod and prod. On prod, add required reviewers and a branch rule restricting deployments to main. Then add three environment variables (not secrets - these aren't sensitive): AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID.

The workflow itself stays small:

permissions: id-token: write contents: read jobs: apply: runs-on: ubuntu-latest environment: prod env: ARM_USE_OIDC: "true" ARM_CLIENT_ID: ${{ vars.AZURE_CLIENT_ID }} ARM_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }} steps: - uses: actions/checkout@v4 - uses: hashicorp/setup-terraform@v3 - run: terraform init && terraform apply -auto-approve

Three things make this secure:

id-token: write is the only elevated permission, and it doesn't grant write access to anything in GitHub, it just lets the runner mint a JWT.
The environment: line picks the right AZURE_CLIENT_ID and drives the sub claim. The federation refuses anything else.
No azure/login step is needed for Terraform. The azurerm provider reads GitHub's OIDC environment variables automatically.

Part 2 - Azure DevOps

The model is identical. The mechanics are different.

ADO offers two creation paths for a WIF service connection: automatic (it creates an app registration for you) and manual (you bring your own UAMI). For platform teams, manual + UAMI is almost always the better choice to ensure identity lives where governance lives.

The flow is a small dance between the two portals:

In Azure DevOps, create a new ARM service connection → choose Workload Identity Federation (manual) → fill in your UAMI's client ID, tenant ID, and subscription. Save as draft. ADO shows you an issuer URL and a subject identifier.
In Azure, on the UAMI, add a federated credential using the values ADO showed you. The subject looks like sc://contoso/platform/azure-prod.
Back in ADO, click Verify and save.

In the pipeline, the service connection only "activates" if a task in the job loads it. The simplest way is the AzureCLI@2 task:

- task: AzureCLI@2 inputs: azureSubscription: azure-prod # the WIF service connection scriptType: bash scriptLocation: inlineScript inlineScript: | terraform init && terraform apply -auto-approve env: ARM_USE_OIDC: "true" ARM_CLIENT_ID: $(AZURE_CLIENT_ID) ARM_TENANT_ID: $(AZURE_TENANT_ID) ARM_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID) ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID: $(SERVICE_CONNECTION_ID) SYSTEM_ACCESSTOKEN: $(System.AccessToken) SYSTEM_OIDCREQUESTURI: $(System.OidcRequestUri)

For teams converting dozens of legacy connections, the Azure DevOps team published a PowerShell helper that walks every ARM service connection in a project and converts them in place. There's a 7-day rollback window on each connection, which makes the migration genuinely low-risk.

Don't forget the state file

The Terraform state is your real blast radius. With OIDC, it's almost free to lock it down too. The same UAMI can read and write blob data without the storage account key:

backend "azurerm" { resource_group_name = "rg-tfstate" storage_account_name = "sttfstateprodeastus" container_name = "platform-prod" key = "platform.tfstate" use_oidc = true use_azuread_auth = true }

Grant the UAMI Storage Blob Data Contributor on the container (not the account), disable shared key access on the storage account, and you've removed the last secret in the pipeline.

RBAC and break-glass

Federation removes a credential, not a privilege. A few habits worth keeping:

Scope role assignments to resource groups, not subscriptions. The whole point of federation is that scoping is now trivially easy.
Use Role Based Access Control Administrator instead of User Access Administrator if your Terraform creates role assignments. It's a more recent, narrower role.
Have a documented break-glass. If GitHub or ADO has a token-service incident, you still need a path to ship a hotfix. A single hardware-key-protected emergency app registration in a separate identity boundary works well, audited monthly.
Monitor sign-ins. Every federated exchange shows up in Entra sign-in logs as a service principal sign-in. Pipe these to Sentinel and alert on anomalies like sign-ins outside expected hours, or from IPs outside GitHub's published ranges.

The errors you will hit (and what they really mean)

Symptom	What it actually is
AADSTS70021: No matching federated identity record found	Case-sensitive mismatch in iss, sub, or aud. Almost always a trailing slash or a capitalised character
AADSTS700016: Application not found in directory	Wrong client ID or tenant. Not a federation problem
403 on a resource even though token exchange worked	Federation is fine. Your RBAC isn't. Check the exact scope
Unable to determine OIDC token (ADO)	No task in the job loaded the service connection. Add an AzureCLI@2 step
Works on main, fails on tags	You pinned sub to a branch ref. Add a second federated credential for tags, or move to environment-based scoping

Migrating without a maintenance window

You almost never get to do this on a greenfield repo. The order that has worked for me on legacy estates:

Create the new UAMI alongside the old service principal, with the same role assignments.
Federate one canary pipeline. Verify it deploys equivalently.
Cut over pipelines in waves, lowest-risk environment first.
Once a full release cycle passes cleanly, disable the old SP's secret.
Wait another cycle. Then delete the SP entirely.
Add a CI gate that fails any new pipeline introducing ARM_CLIENT_SECRET.

The old and new auth methods coexist on the same subscription throughout. There's no hard cutover and no maintenance window, just a steady drift toward zero secrets.

Wrapping up

If you do nothing else after reading this, do one thing: search your CI variable groups for ARM_CLIENT_SECRET. Every result is an outage or a breach waiting to happen.

Federation is one of those rare changes that's both more secure and less work to operate. Once you've set it up, you stop thinking about credential rotation, secret expiry, and quarterly access reviews for service principals. The pipeline simply runs, and the audit trail is in Entra where it belongs.

That's a good trade.

Automating Azure Naming Standards using API and DevOps Pipelines

sameenamohammed — Sun, 03 May 2026 05:53:03 GMT

Introduction

In large Azure environments, one of the most overlooked yet critical governance challenges is resource naming consistency.

While organizations define naming standards, enforcing them at scale across multiple subscriptions, teams, and pipelines often becomes a manual and inconsistent process.

In real-world projects, this leads to:

Operational confusion
Difficult resource identification
Reduced traceability
Governance gaps

To address this, we implemented an API-driven naming validation approach integrated with Azure DevOps pipelines, ensuring every resource created follows organizational standards automatically.

The Problem: Inconsistent Naming Across Environments

In distributed teams and large-scale environments, naming issues commonly arise due to:

Multiple developers creating resources independently
Lack of centralized enforcement
Manual validation during deployments
No integration with CI/CD pipeline

Example (Before Automation)

Resource Type	Example Name
Resource Group	testRG1
Storage Account	mystorage123
VM	vm-prod

Problems:

No standard structure
No environment or region context
Hard to manage at scale

Goal

To ensure:

✅ Standardized naming across all resources
✅ Automated validation during deployments
✅ No manual intervention required
✅ Seamless integration with DevOps workflows

Solution Overview

We implemented a naming enforcement mechanism using:

Azure Naming Tool (or similar API-based naming service)
Azure DevOps Pipelines
Managed Identity for secure authentication

Architecture Flow

Automated Naming Validation using Naming API, Managed Identity, and DevOps Pipeline

🔍 Solution Flow Explained

Developer Commit
The process begins when a developer commits code to the repository, triggering the Azure DevOps pipeline.
Azure DevOps Pipeline Execution
The pipeline runs deployment scripts as part of the CI/CD process.
Managed Identity Authentication
The pipeline uses Managed Identity to securely authenticate and obtain an access token—eliminating the need for storing credentials.
Naming API Invocation
A request is sent to the Naming API with resource details such as:
- Resource type
- Environment
- Location
- Application name
Validation & Name Generation
The Naming API validates inputs and returns a compliant resource name based on predefined standards.
Deployment Decision
- If validation succeeds → resources are deployed
- If validation fails → deployment is blocked
Resource Deployment
Only validated, compliant resources are provisioned in Azure.

Note:
The “Azure Naming API” referenced in this blog represents an implementation pattern rather than a native Azure service.
Solutions such as Azure Naming Tool or custom APIs can be used to expose naming logic and integrate with DevOps pipelines for automated enforcement. This approach can be implemented using solutions like Azure Naming Tool, Resource Name Generator, or custom-built APIs

Implementation Details

Authentication using Managed Identity

To securely access the Naming API:

Managed Identity is used
No secrets or credentials stored in pipeline
Token retrieved dynamically

PowerShell Implementation

Below is a simplified version of what was used in implementation:

# Get access token using Managed Identity $token = (Get-AzAccessToken -ResourceUrl "api://NamingTool").Token # Call naming API $response = Invoke-RestMethod ` -Uri "https://your-namingtool-api-endpoint/api/naming" ` -Headers @{ Authorization = "Bearer $token" } ` -Method POST ` -Body @{ resourceType = "resourceGroup" environment = "prod" location = "eastus" application = "app01" } | ConvertTo-Json # Extract generated resource name $resourceName = $response.name Write-Output "Generated Name: $resourceName"

🔄 Azure DevOps Pipeline Integration

Naming validation is integrated directly into the deployment pipeline. Sample Pipeline Snippet

- task: AzureCLI@2 inputs: azureSubscription: 'ServiceConnection' scriptType: 'ps' scriptLocation: 'inlineScript' inlineScript: | Write-Output "Calling Naming API" .\scripts\Get-ResourceName.ps1

Key Benefit:

👉 Resource names are validated before deployment, preventing non-compliant resources from being created.

Security Considerations

Use Managed Identity for API authentication
Avoid storing secrets in pipelines
Ensure API access is restricted and secured

Extending This Solution

This approach can be extended to:

Enforcing tagging standards
Policy validation before deployment
Subscription vending automation
Cost governance controls

✨ Final Thoughts

Naming standards are often documented—but rarely enforced effectively.

By integrating API-based naming validation into DevOps pipelines, organizations can move from:

Guidelines → ✅ Automated Enforcement

This ensures governance is:

Scalable
Consistent
Developer-friendly

Reimagining Azure Governance with Automation & EPAC

sameenamohammed — Sat, 02 May 2026 19:26:11 GMT

🧩 The Challenge: Governance at Scale

Managing Azure environments manually introduces:

❌ Policy drift across subscriptions
❌ Inconsistent naming conventions
❌ Delays in compliance enforcement
❌ Human errors in deployments

🔍 Insight: Governance gaps are often not due to lack of policies—but lack of automation.

Solution Overview: EPAC

Enterprise Policy as Code (EPAC) to bring governance into the DevOps workflow.

EPAC helped us:

Treat policies like code
Automate deployments
Standardize governance
Maintain audit history

Architecture Overview

Our EPAC setup included:

Management Groups hierarchy for governance scope
Policy Definitions & Initiatives (JSON/Bicep)
Azure DevOps Pipeline for deployment
Managed Identity for secure execution

This is sample implementation flow of EPAC in enterprises

Flow Explanation

Azure DevOps Pipeline
- Triggers CI/CD process
- Executes deployment scripts
- Authenticates securely using Managed Identity
EPAC Framework
- Stores policies as code
- Enables version control and validation
- Acts as the central governance engine
Azure Policy Engine
- Evaluates resources against defined policies
- Enforces compliance automatically
Target Environments
- Policies applied across:
  - Management Groups
  - Subscriptions
  - Resource Groups

Why Policy-as-Code Matters

✅ Standardized governance across environments
✅ Faster onboarding of subscriptions
✅ Improved audit readiness
✅ Repeatable and reliable policy deployment
✅ Seamless DevOps integration

Security & Compliance Benefits

Enforces least privilege access
Prevents misconfigured deployments
Supports continuous compliance
Aligns with standards like ISO 27001

Implementation Walkthrough

✅ Step 1: Structuring Policy Repository

This structure ensured:

Clear separation of concerns
Reusability
Easy onboarding for new contributors

✅ Step 2: Defining Policies

We created custom policies and reused built-in ones.

Example: Restrict allowed regions

{

"if": {

"field": "location",

"notIn": ["eastus", "westeurope"]

"then": {

"effect": "deny"

}

✅ Step 3: Creating Initiatives

Instead of assigning individual policies, we grouped them into initiatives:

Security baseline
Tagging compliance
Cost optimization

👉 This reduced duplication and simplified assignments.

✅ Step 4: Pipeline Automation

We built an Azure DevOps pipeline to:

Validate policy templates
Deploy definitions to management groups
Assign initiatives automatically

Example pipeline flow:

This is example pipeline structure for deploying epac policies targeted to single environment

parameters: - name: forceDeployment displayName: 'Force deployment (ignore change detection)' type: boolean default: false - name: clearAgentCache displayName: 'Clear agent container cache (recommended for troubleshooting)' type: boolean default: true variables: # Pipeline per il deployment delle Policy in ambiente Canary/Test PAC_OUTPUT_FOLDER: ./Output PAC_DEFINITIONS_FOLDER: ./Definitions # Service connection per l'ambiente di test/canary serviceConnection: "SC-EPAC-CONTRIBUTOR-TST-001" # Environment selector per canary pacEnvironmentSelector: canary # Trigger: deploy solo manualmente o da branch specifici trigger: none # PR trigger per validazione pr: branches: include: - main - feature/* paths: include: - src/IaC/Infrastructure/epac/Definitions/* pool: name: "TST-AgentPool-01" stages: - stage: Plan displayName: "Plan Canary Environment" jobs: - job: Plan displayName: "Generate Deployment Plan" steps: - template: templates/plan.yml parameters: serviceConnection: $(serviceConnection) pacEnvironmentSelector: ${{ variables.pacEnvironmentSelector }} - stage: Deploy displayName: "Deploy to Canary (Audit Mode)" dependsOn: Plan condition: and(not(failed()), not(canceled()), or(eq('${{ parameters.forceDeployment }}', 'true'), and(eq('${{ parameters.forceDeployment }}', 'false'), or(eq(dependencies.Plan.outputs['Plan.Plan.deployPolicyChanges'], 'yes'), eq(dependencies.Plan.outputs['Plan.Plan.deployRoleChanges'], 'yes'))))) variables: PAC_INPUT_FOLDER: "$(Pipeline.Workspace)/plans-${{ variables.pacEnvironmentSelector }}" localDeployPolicyChanges: $[stageDependencies.Plan.Plan.outputs['Plan.deployPolicyChanges']] localDeployRoleChanges: $[stageDependencies.Plan.Plan.outputs['Plan.deployRoleChanges']] jobs: - deployment: DeployPolicy displayName: "Deploy Policy Changes (Audit Mode)" environment: PAC-CANARY condition: and(not(failed()), not(canceled()), or(eq('${{ parameters.forceDeployment }}', 'true'), and(eq('${{ parameters.forceDeployment }}', 'false'), eq(variables.localDeployPolicyChanges, 'yes')))) strategy: runOnce: deploy: steps: - template: templates/deploy-policy.yml parameters: serviceConnection: $(serviceConnection) pacEnvironmentSelector: ${{ variables.pacEnvironmentSelector }} forceDeployment: ${{ parameters.forceDeployment }} - deployment: DeployRoles displayName: "Deploy Role Assignments" dependsOn: DeployPolicy environment: PAC-CANARY condition: and(not(failed()), not(canceled()), eq(variables.localDeployRoleChanges, 'yes')) # Riabilitato per AMBA managed identity strategy: runOnce: deploy: steps: - template: templates/deploy-roles.yml parameters: serviceConnection: $(serviceConnection) pacEnvironmentSelector: ${{ variables.pacEnvironmentSelector }} # Stage opzionale per validazione post-deployment - stage: Validate displayName: "Validate Canary Deployment" dependsOn: Deploy condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest')) jobs: - job: ValidateCompliance displayName: "Validate Policy Compliance" steps: - task: PowerShell@2 displayName: "Check Policy Compliance Status" inputs: targetType: 'inline' script: | Write-Host "##[section]Canary deployment completed successfully" Write-Host "##[warning]Remember: All policies are in AUDIT mode - monitor compliance dashboard" Write-Host "##[task.complete result=Succeeded;]Canary validation completed"

✅ Step 5: Secure Deployment using Managed Identity

We used:

System-assigned Managed Identity
RBAC roles (Policy Contributor / Reader)

✅ Benefit:

No secrets in pipeline
Improved security posture

✅ Step 6: Policy Assignment at Scale

Policies were assigned at:

Root Management Group
Subscription level (when needed)

This ensured:

Consistent enforcement
Centralized control

Real Use Cases Implemented

Using EPAC, we solved real scenarios:

🔹 Enforcing naming conventions
🔹 Ensuring mandatory resource tagging
🔹 Restricting deployment regions
🔹 Enforcing backup policies on disks
🔹 Preventing creation of non-compliant resources

Best Practices

Start with baseline policies first
Use initiatives instead of individual assignments
Enable PR-based approvals
Always test policies in lower environments
Maintain clear documentation

Conclusion

Implementing EPAC transformed our governance model from manual and reactive → automated and proactive.

For teams managing complex Azure environments, EPAC provides:

Scalability
Consistency
Security

If you are still managing policies manually, this is the right time to: 👉 Move to Policy as Code

Mastering Task View in Windows 11: A Smarter Way to Work and Stay Organized

JohnNaguib — Sat, 02 May 2026 09:18:17 GMT

If you’ve ever found yourself juggling multiple windows, tabs, and apps on your computer, you’re not alone. Modern workflows demand multitasking, but without the right tools, things can quickly become chaotic. That’s where Task View in Windows 11 comes in a powerful yet often underused feature that can completely transform how you work.

https://dellenny.com/mastering-task-view-in-windows-11-a-smarter-way-to-work-and-stay-organized/

GitHub Copilot in the Classroom: Help or Hindrance?

JohnNaguib — Sat, 02 May 2026 09:13:37 GMT

The rise of AI in education has sparked a wave of excitement and concern. Among the most talked-about tools is GitHub Copilot, an AI-powered coding assistant that can generate code in real time. For students and educators alike, it raises an important question: is Copilot a helpful learning companion, or does it risk becoming a shortcut that undermines real understanding?

https://dellenny.com/github-copilot-in-the-classroom-help-or-hindrance/

TrustedMissingIdentityClaimSource - OIDC

hunk0227 — Sat, 02 May 2026 06:18:56 GMT

Hello,

Good day!
I'm setting up OIDC connection thru SharePoint subscription edition referring to the below link
https://learn.microsoft.com/en-us/SharePoint/security-for-sharepoint-server/set-up-oidc-auth-in-sharepoint-server-with-msaad

I was able to get in thru Entra (that means OIDC connection works) but sharepoint return me this exception when getting in to site collection.

_layouts/15/_login/default.aspx?errorCode=TrustedMissingIdentityClaimSource=https%3A%2F%2FSPdemo.local%2F_layouts%2F15%2FAuthenticate.aspx%3FSource%3D%252F

Workaround suggested done like reconfigured SPTrust, certificates and SharePoint Web application multiple times but yet still no avail. Unfortunately, this have been greyed out for me as i cannot find a concrete resolution.

Is anyone have experience the same exception, or perhaps share thoughts what are missing here.

Thank you!

Talk to Your Data: Fabric Data Agents for D365, Power Platform & Beyond

MehrdadAbdollahi — Sat, 02 May 2026 06:05:36 GMT

📢 Upcoming Session at The Fabric Café

🗓 May 13th at 6:00 PM PT
🎙 Speakers: Gaston Cruz & Alex Rostan
🎤 Host: Mehrdad Abdollahi
📌 Title: Talk to Your Data: Fabric Data Agents for D365, Power Platform & Beyond

What if your business users could simply ask questions and get answers directly from enterprise data?

In this session, you’ll learn how to connect Dynamics 365 and Power Platform data into Microsoft Fabric, and build Fabric Data Agents that enable natural language interaction with your data.

🚀 What you’ll learn:
✔ Ingest and model D365 & Power Platform data into a Lakehouse & Semantic Model
✔ Build a Fabric Data Agent that understands business context
✔ Expose your agent via Copilot Studio and embed it in Microsoft Teams

👥 Who should attend?
Data architects, Power Platform makers, D365 owners, and business leaders driving AI-powered transformation.

💡 Tip: Bring a key KPI or report your team uses — and learn how to turn it into a conversation-ready data experience!

#TheFabricCafe #MicrosoftFabric #Microsoft #MicrosoftLearn #MicrosoftFabricCommunity #AI #PowerPlatform #Dynamics365 #DataEngineering #DataAnalytics #Copilot #DataCommunity

AI Unleashed: The Foundry and studio shaping tomorrow's creativity

Pouya — Sat, 02 May 2026 03:40:35 GMT

🚀 AI Unleashed: The Foundry & Studios Shaping Tomorrow’s Creativity

What if creativity wasn’t limited by time, resources, or technical complexity?

Join this upcoming session where we explore how AI Foundries and modern AI studios are transforming the way businesses innovate, build, and deliver exceptional customer experiences.

From virtual try-on experiences in retail to intelligent chatbots that enhance customer support, AI-powered platforms are unlocking new possibilities:
✨ Accelerate innovation without heavy development overhead
✨ Create immersive, personalized customer journeys
✨ Automate processes while improving efficiency
✨ Turn ideas into scalable solutions—faster than ever

Whether you're a developer, business leader or tech enthusiast, this session will give you practical insights into how AI platforms are reshaping creativity and digital experiences.

🎯 Don’t miss this opportunity to see how AI is redefining what’s possible.

📺 Register and join live → StreamYard Session

🗓️ Date: 8 May 2026
⏰ Time: 18:00 (AEST) / 16:00 (SGT) / 10:00 (CEST)
🎙️ Speaker: Senthamil Selvan
📌 Topic: AI Unleashed: The Foundry and studio shaping tomorrow's creativity

Enabling AI‑Driven Workforce Transformation with People Skills and Workforce Insights Agent: April 2026 M365 Champions Community call

TiffanyLee — Fri, 01 May 2026 22:16:33 GMT

Hello Champions!

Here’s a recap and top Q+A from our April M365 Champions monthly call of 2026, featuring: Enabling AI‑Driven Workforce Transformation with People Skills and Workforce Insights Agent with Anirudh Bajaj ,Microsoft Senior Product Manager and Jenny Huang, Microsoft Senior Customer Experience Program manager.

We kicked off the call with the annoucement of our digital premiere of More Than Code: SharePoint community film. More Than Code is a documentary-style short film featuring voices from MVPs, customers, and Microsoft leaders. Discover how connection, creativity, and collaboration helped turn a product into a global movement. Click here to watch the digital premiere today!

Anirudh Bajaj and Jenny Huang led the main topic this month with a deep dive into People Skills and Workforce Insights Agent. First, Anirudh highlighted a shift in how organizations must approach AI—moving beyond simply using AI tools to actively redesigning workflows around them. He introduced People Skills as a foundational capability to enable this transformation by providing visibility into workforce skills through AI-powered inferencing. By analyzing Microsoft 365 activity and combining it with a large pre-built skills taxonomy, the solution generates dynamic skill profiles for employees that can be validated or customized by users. These skills power scenarios such as finding experts, improving collaboration, and enabling personalized learning. Anirudh emphasized privacy and user control, noting that employees can manage their profiles, opt out of inferencing, and control sharing, while admins can configure governance policies.

Next, Anirudh introduced his colleague Jenny Huang who introduced us to the Workforce Insights Agent. The Workforce Insights Agent is designed to help leaders translate skills and organizational data into actionable decisions. She framed the agent within the growing need for strategic workforce planning in an AI-driven environment, where leaders must quickly understand team structure, skills distribution, and emerging gaps. The agent integrates data from Microsoft 365 profiles and external HR systems, enabling leaders to query organizational data conversationally and receive real-time insights, visualizations, and recommendations. Jenny outlined key use cases such as analyzing team composition, identifying hiring or skill gaps, and guiding talent movement and upskilling strategies. Currently available in the Microsoft Frontier program, the agent emphasizes governed data access and encourages customer feedback to shape its evolution toward general availability, positioning it as a critical decision-support tool for AI-driven organizational transformation.

Q+A from this month's session:

1. When will the Workforce Insights Agent be generally available (GA)?

Answer: There is no confirmed GA timeline yet. The team is actively relying on customer feedback from the Frontier program to determine readiness and scope before moving to GA.

2. Can we add a two layer skill approval? so we are getting confirmation person is actually skilled in "skill"? and then add levels (1 - beginner, 5 - expert (teaching level professional)?

Answer: We have not added expertise levels to the People Skills as an "expert" is very subjective and defined differently by organizations, and organizations have different level requirements (1-to-5 vs 1-to-10 vs categories (expert, beginner etc.) so it is challenging to standardize this. We are exploring offering skill depth/context in certain contexts e.g. while matching a person to an open role- to understand their skill depth. More to come on this.

3. People Skill AI - What is the distinction between this and Workforce Insights?

Answer: People Skills AI inferencing uses intelligence to make skill recommendation (AI inferred skills) . This is a data layer that is used by Copilot and Workforce Insights Agent to help you with decision making.

4. Can you help me understand what Agents explicitly we are talking about? There used to be a 'Skills' Agent, which I can no longer see in my environment (Frontier). I also cant see a 'People' Agent. I do however see a 'workforce insights' agent.

Answer: Skills agent and People agent are both retired in March and the capabilities are now folded into M365 Copilot chat and Workforce Insights agent.

5. Can these People skills be fed into a program like Workday, so they can be added to the Skills Cloud and be used for promotions, reviews, etc.?

Answer: Confirmed skills in People Skills can be exported for you to use it in other platforms/systems. Due to privacy reason, we are not allowed to provide export on inferred skills.

6. Is Viva Engage included in E5 licensing?

Answer: Yes! The core/standard experience is included the E5 license, which does include communities and integration in Teams.

7. Are these calls recorded and/or archived so we can review later?

Answer: Yes, the recording will be published on the Driving Adoption community and our YouTube Community Learning channel. However, only Champion program members have access to the presentation (see the link in your welcome email or the latest newsletter). If you're not a member yet, join here: https://aka.ms/BecomeAChampion.

8. Is frontier a paid programme?

Answer: It's not an additional to your M365 Copilot license, but you do need to opt into the program. More info here: Explore AI Early Access in Microsoft 365 | Microsoft Frontier

9. Is the People Skills Agent and Learning Agent available to Copilot Chat Basic users or only the paid version of Copilot?

Answer: People Skills agent was deprecated in March and the capability are now available through M365 Copilot chat and Workforce Insights agent for M365 Copilot licensed users.

10. Can you see aggregated information from People Skills? Like see what level people are at with AI across the enterprise?

Answer: There are 2 aspects here: one is skills aggregation report and another one is skills level/proficiency. The first part can be done via skills landscape report (via Viva Insights, shown as a Power BI report), or through Workforce Insights agent which I will share more later Ani's presentation. On skills level/proficiency piece, it's not available today in People Skills.

11. Can you also add linkedin profile to the sources for People Skills inference?

Answer: Not directly the profile but if you have an inventory of what skills are attached/assigned to an user, you can import the list into People Skills as 3rd party skills.

12. On an enterprise level can an organisation navigate a recommended career path for an employee based on their people skills? How does this data show up?

Answer: Today Copilot can make general career and learning recommendations for users based on their skills. However, People Skills does not yet support career path suggestions based on job architecture/open job roles.

13. Are these features only managed by the Admins? We have a single tenant so our Agency doesn't have access to the Admin console. Many of these Admin features are not usable for us without a more granular way to manage them.

Answer: The admin controls can be set up by someone with the right permissions, these roles can do it today:

AI administrator
Knowledge administrator
Global admin

14. We are a multi-cloud environment (commercial and GCCH). Will these agents eventually be available on GCCH?

Answer: People Skills is currently only available in WW environment, we don't have an estimate on when we will be in GCC.

15. Am I understanding correctly that the PAX API allows access to our tenant's data by a third party? What data controls guide that interaction?

Answer: You can bring in 3P data via connectors or push APIs into MODIS and apply access control there.

16. Who can access the Workforce Insights Agent?

Answer: Any user with a Microsoft 365 Copilot license can access the agent. Currently, only through the Frontier program.

The tenant must also meet requirements (e.g., sufficient Copilot licenses in the environment).

17. Where does Workforce Insights Agent get its data from?

Answer: Default data comes from Microsoft 365 profiles (Entra). Additional data can be brought in via:

HR systems (Workday, SuccessFactors)
APIs or CSV uploads

Please note: Sensitive data is governed using strict data policies and access controls.

18. Do users have control over their skills and data?

Answer: Users are able to:

Confirm, edit, or remove inferred skills
Add their own skills manually
Opt out of AI inferencing entirely
Control whether their skills are shared across the organization

19. Can admins control how People Skills is deployed?

Answer:

Enable/disable the feature by user, group, or tenant
Control inferencing and sharing settings
Apply privacy and compliance policies
Pilot the feature in a limited scope before broad rollout

20. What are the main use cases of Workforce Insights Agent?

Answer: The agent helps with:

Understanding team structure and composition
Identifying skill gaps and distribution
Supporting workforce planning decisions
Driving talent movement and upskilling strategies

Join us next month on May 26th for our next community call featuring Copilot Hub & Agents 365.

Haven't joined the program yet? It's free! Join here: https://aka.ms/M365Champion.
Bring your questions to our discussion forum: https://aka.ms/DriveAdoption.

Update to disabling Teams meeting recording expiration notification emails

Eddie_Harmon — Fri, 01 May 2026 21:12:58 GMT

Hello,

Thank you to everyone who shared their thoughts and feedback regarding the planned Upcoming change: disabling Teams meeting recording expiration notification emails. After carefully reviewing the feedback from this discussion, survey responses, and support channels, we have decided to pause the rollout of this change. The updates originally planned for June 1st will not take effect on that date.

What this means for you:

- Email notifications for expired Teams meeting recordings will continue as they do today.

- No action is required on your part.

- Recording expiration and deletion policies remain unchanged.

Your input along with ongoing, internal engineering discussions helped shaped this decision. We want to make sure that any changes we make to the notification experience truly work for your organizations, and the feedback we received made it clear that we need more time to get this right.

We're still committed to improving the notification experience for Teams meeting recordings and will provide updates here and through the Message Center when we have more to share. In the meantime, please continue to share your thoughts in this discussion.

Thank you for your patience and for being part of our community.

Lock down AI, web, and private apps: what’s new in Internet Access and Private Access

Sinead_ODonovan — Fri, 01 May 2026 20:56:02 GMT

One theme is crystal clear across the security industry: AI is transforming security, and security must transform with it. Organizations everywhere are embracing generative AI to boost productivity and accelerate innovation. But with this rapid adoption comes new challenges that security teams can’t ignore:

Which AI tools are your employees using?
Is sensitive data being uploaded to unsanctioned services?
How do you prevent AI-specific attacks like prompt injection?
How do you secure private apps without slowing down users?

These aren’t hypothetical questions. They’re the reality for every organization today. And the answer starts with identity.

Identity: The foundation for AI and app security

Traditional network security was built for a time when users, devices, and applications were mostly on-premises and predictable. Today, employees work from anywhere on any device, and generative AI and SaaS apps often sit outside the corporate perimeter. Static controls struggle to keep pace, creating gaps that increase risk.

That’s why we built Microsoft Entra Internet Access and Microsoft Entra Private Access within the Global Secure Access platform. These solutions extend Zero Trust protections to web, SaaS, AI, and private-app traffic. They provide the visibility and control organizations need to embrace AI and hybrid work with confidence—without slowing innovation.

A key capability of Microsoft Entra Internet Access is the Secure Web and AI Gateway, which applies identity-centric network controls to web and AI traffic. Identity-based network security ties access decisions to the user’s sign-in risk, device posture, and data sensitivity—not just an IP address or network location. This approach delivers consistent protection everywhere users work, reduces risk, and helps organizations scale AI adoption securely across the enterprise.

Late last year, we introduced most of the capabilities in public preview at Microsoft Ignite. Today, we’re excited to share the latest features now generally available in Microsoft Entra Internet Access and Private Access and to announce brand-new capabilities in public preview.

Figure 1: Microsoft’s identity-centric Secure Access Service Edge (SASE) solution.

Public preview: More flexibility for diverse environments

Microsoft Entra Internet Access & Microsoft Entra Private Access

We’re introducing new capabilities in public preview, giving you more options to secure every scenario:

BYOD with client in Microsoft Entra Private Access lets you enforce Zero Trust for unmanaged devices, so employees and contractors can securely access private apps without compromising security or user experience.
Explicit Forward Proxy for Microsoft Entra Internet Access extends secure web access to agentless and legacy devices using PAC file-based proxy configuration.
Secure Browser Integration enables Intune-managed Microsoft Edge to route internet traffic through Microsoft Entra Internet Access using Explicit Forward Proxy with TLS termination and inspection, delivering deep visibility and policy enforcement for secure browsing.
Shadow MCP visibility identifies unauthorized or high‑risk MCP servers on the network traffic and surfaces MCP data paths, logs, and observability to help monitor and manage AI‑related risk.

These new features help you reduce risk across every device type, simplify deployment, and deliver consistent protection everywhere.

Now generally available: New AI security capabilities

Microsoft Entra Internet Access

AI adoption is accelerating, but so are the risks. Employees often experiment with AI tools without IT approval, creating compliance and data security gaps. With Microsoft Entra Internet Access, you can see what is happening, protect what matters, and simplify how you manage it all.

Shadow AI discovery gives you visibility into unsanctioned AI tools and SaaS apps so you can uncover unknown risks and make informed decisions before enforcing policy.
Prompt Injection Protection helps block malicious prompts that could trick AI models into exposing sensitive data, reducing AI-specific attack risk without slowing innovation.
Network content filtering prevents sensitive files from being uploaded to unsanctioned AI services, reducing compliance risk and data loss.
URL filtering and threat intelligence block access to risky or malicious sites, enforce acceptable use policies, and reduce data leakage.
Cloud firewall for remote networks provides advanced network-layer protection for traffic from remote sites, enabling granular policy enforcement and reducing exposure to threats.
iOS support and remote network connectivity extend protection everywhere your users work.

The result is simple. Your teams can use AI tools to work smarter while you maintain control and reduce risk without introducing friction.

Figure2: Demo of prompt injection protection.

Now generally available: New capabilities for modernizing app connectivity

Microsoft Entra Private Access

While Internet Access secures your web and AI traffic, Microsoft Entra Private Access helps you replace legacy VPNs with Zero Trust Network Access for private apps:

External User Access enforces Zero Trust for partners and contractors, simplifying onboarding while maintaining strong security.
Intelligent Local Access improves user experience by routing traffic efficiently, reducing latency, and delivering consistent security without unnecessary backhauling.
The result is a better experience for users and simpler operations for your IT teams.

Ready to secure AI and modernize identity?

Watch the Microsoft Entra Showcase webinar
Watch the Microsoft Entra Mechanics video for a deep dive into AI-aware protections
Start your journey today: Entra Suite Trial

-Sinead O’Donovan | VP of Product Management, Identity and Network Access

Sinead O'Donovan | LinkedIn

Additional resources

Learn more about Microsoft Entra

Prevent identity attacks, ensure least privilege access, unify access controls, and improve the experience for users with comprehensive identity and network access solutions across on-premises and clouds.

High-Fidelity Network Observability at Scale— ACNS Metrics Filtering and Log Aggregation Now GA

chandanAggarwal — Fri, 01 May 2026 20:29:20 GMT

We are thrilled to announce that Advanced Container Networking Services (ACNS) for Azure Kubernetes Service (AKS) now delivers two powerful observability features in General Availability: container network metrics filtering and container network log filtering and aggregation. Together, these capabilities set a new standard for Kubernetes network observability, giving you high-fidelity visibility at dramatically lower cost and noise. These capabilities fundamentally redefine how network observability works at scale while delivering up to 97% cost reduction.

Why this is a Milestone?

Most Kubernetes observability solutions face a fundamental tension: collect everything and drown in noise and cost, or sample and miss the signals that matter. ACNS breaks that tradeoff.

With this release, Azure becomes the first cloud provider to deliver on-node metrics filtering and flow log aggregation for Kubernetes networking, capabilities now also contributed to the upstream Hubble project, making them available to the broader open-source community.

For AKS customers running Cilium-based clusters, this means:

Every flow you care about is captured. Everything else is dropped at the source.
Log volume is compressed by up to 45% through aggregation, without losing security verdicts or error context.
Costs scale with what you monitor, not with cluster size.

What’s been improved in ACNS observability?

This release introduces two capabilities that work together: container network metrics filtering and container network log filtering and aggregation. Both are available on AKS clusters with the Cilium data plane and give you precise controls to keep observability costs predictable while maintaining the visibility you need.

Container Network Metrics Filtering

Container network metrics are generated for all pods by default whenever ACNS is enabled. With metrics filtering, you now control what gets collected at the point of ingestion, on the node, before anything is scraped or transmitted.

A single ContainerNetworkMetric CRD per cluster defines which metric types (dns, flow, tcp, drop), namespaces, pod labels, and protocols to ingest. It supports both include and exclude filters, so you can maintain broad collection while carving out specific workloads or namespaces. Anything that doesn't match is dropped on the node. Changes reconcile in a few seconds, with no Cilium agent or Prometheus restarts required.

Container Network Log Filtering and Aggregation

Unlike metrics, container network logs are not generated automatically. You start capturing network flows only after applying a ContainerNetworkLog CRD that defines exactly which traffic to capture-by namespace, pod, service, protocol, or verdict. Only matching flows are logged, giving you a precise, targeted view rather than a fire hose.

This is where Azure's first-to-market innovation comes in. Flow log aggregation, now built into ACNS and contributed upstream to Hubble for the open-source community, groups similar flows into summarized records every 30 seconds. The result is dramatically reduced data volume while preserving security verdicts, service identity, and error context. What previously required custom post-processing pipelines is now built directly into the platform before storage costs are incurred.

Every matched flow log captures: source and destination pods, namespaces, ports, protocols, traffic direction, and policy verdicts.

Logs are stored in a Log Analytics workspace (ContainerNetworkLogs table) with a choice of using the Analytics or Basic tier. Built-in Azure portal dashboards are available for both tiers. Logs can also be exported to external log collectors such as Splunk or Datadog.

First to Market: Azure and the upstream Hubble Contribution

ACNS's filtering and aggregation capabilities were engineered from the ground up to solve real production observability challenges at scale. Rather than keeping this innovation proprietary, Azure contributed the log aggregation and filtering capabilities to the upstream Hubble project, the observability layer of the Cilium ecosystem.

This means:

AKS customers get a fully managed, Azure-native experience with portal dashboards, Log Analytics integration, and Grafana visualization, out of the box.
The broader open-source community gains access to the same filtering and aggregation primitives through upstream Hubble.

Azure is the first to ship this capability in a managed Kubernetes service, and the first to give it back to the community.

Key Benefits

💰 Lower observability cost. Metrics filtering drops unwanted data on the node before Prometheus ever scrapes it. Flow log aggregation compresses log data by up to 97% in lab testing. Your cost scales with what you choose to monitor, not with cluster size.

📉 Less noise, more signal. Metrics filtering carves out the namespaces and workloads that matter, so dashboards show only relevant signals. Log filters scope collection to specific pods and verdicts. Engineers start every investigation with data that's already relevant.

⚡ Faster root-cause isolation. Every metric carries source and destination pod context. Targeted flow logs add the forensic detail, which policy, destination, or port is involved. Together, they cut mean time to resolution from hours of guesswork to minutes of structured investigation.

🔒 Full signal, zero gaps. ACNS doesn't sample. Within the scope you define, every flow is captured and every pattern is preserved. Aggregation compresses volume without losing security verdicts or error context.

Who Benefits

Platform engineers managing multi-tenant clusters can scope data collection per namespace, so each team gets visibility into their own traffic without contributing to a shared cost pool.

SREs can isolate packet drops, TCP resets, or DNS failures to a specific workload in minutes, starting with data that's already scoped to what matters.

Decision-makers evaluating observability spend get predictable, controllable ingestion costs that scale with intent, not infrastructure size.

How to optimize ACNS metrics and logs with filtering?

Enable ACNS on your AKS cluster with the Cilium data plane:

az aks create --enable-acns

Or on an existing cluster:

az aks update --resource-group $RESOURCE_GROUP --name $CLUSTER --enable-acns

Apply a ContainerNetworkMetric CRD to filter which metrics are collected on each node. Start by excluding noisy system namespaces, then scope to business-critical workloads.
Apply a ContainerNetworkLog CRD to define which flows to capture.
Enable Azure Monitor integration with --enable-container-network-logs to send logs to a Log Analytics workspace, or export logs from the node to an external logging system such as Splunk or Datadog.
Check your dashboards. Open your cluster in the Azure portal and go to Monitor > Insights > Networking for bytes, drops, DNS errors, and flows. For flow logs, use the built-in Azure portal dashboards available for both Basic and Analytics tiers.

Conclusion

Kubernetes network observability has long meant choosing between visibility and cost. With container network metrics filtering and log filtering and aggregation now GA in ACNS and contributed to upstream Hubble for the open-source community, that tradeoff is gone.

Azure is first to market with this capability. AKS customers get it fully managed, out of the box, with built-in dashboards with Log Analytics integration. And the broader Cilium ecosystem gets it through upstream Hubble.

High-fidelity visibility. Lower cost. No compromise.

Learn more:

Container network metrics overview: Container network metrics overview - Azure Kubernetes Service | Microsoft Learn
Container network logs overview: Container Network Logs Overview - Azure Kubernetes Service | Microsoft Learn
Configure container network metrics filtering: Configure Container network metrics filtering for Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn
Set up container network logs: Set up container network logs - Azure Kubernetes Service | Microsoft Learn

Join us May 6: Learn how Microsoft Teams is helping SMBs thrive

MiikkaOksanen — Fri, 01 May 2026 20:15:09 GMT

Running a small or medium-sized business means wearing every hat at once — sales lead, customer support, IT admin, and marketing team, often before lunch. The tools you rely on need to keep up without getting in the way. That's exactly the conversation we're hosting next week.

On Wednesday, May 6, the Microsoft Teams SMB product team is going live for a focused 40-minute session designed specifically for small and medium businesses.

What you'll learn

Angela Chin, Principal PM Manager on the Teams SMB team, will walk through the latest updates in Microsoft Teams built with smaller businesses in mind:

Get going faster — onboarding improvements that take SMBs from sign-up to collaboration in minutes.
Deeper customer connections — features that turn Teams into a front door for your customers.
Effortless cross-business collaboration — work with vendors, contractors, and partner businesses as naturally as your own team.

Bring your questions

Connect directly with the Teams SMB product team, ask questions, and share feedback that shapes the roadmap.

Who should attend

SMB owners and operators, IT pros supporting smaller businesses, partners and consultants, and anyone evaluating Teams.

Save your seat

Wednesday, May 6, 2026, 8:05–8:45 AM PT. Free to attend.

👉 Register here

Microsoft ODBC Driver 17.11.1 for SQL Server Released

DavidLevy — Fri, 01 May 2026 20:00:00 GMT

We are pleased to announce the general availability of Microsoft ODBC Driver 17.11.1 for SQL Server, released on April 30, 2025. This servicing update delivers important bug fixes and expands Linux platform support.

Key Highlights

Stability and correctness fixes for parameter array processing, including accurate updates to SQL_ATTR_PARAMS_PROCESSED_PTR and improved row counting when SQL_PARAM_IGNORE is used in parameter arrays.
Fixed a connection error that could occur when processing Data Classification metadata in ODBC asynchronous mode.
Updated RPM packaging rules to allow installation of multiple driver versions side by side.
Corrected XA recovery to ensure proper computation of transaction IDs and recovery of missing transactions.
Debian package installation now honors license acceptance for successful completion.

New Platform Support

Platform	Versions
macOS	14, 15, 26
Debian	13
Red Hat Enterprise Linux	10
Oracle Linux	9, 10
SUSE Linux Enterprise Server	16
Ubuntu	24.04, 25.10
Alpine Linux	3.21, 3.22, 3.23

Download

The driver is available for download from the Microsoft ODBC Driver for SQL Server documentation page.

Linux Installation

Install or update using your distribution's package manager:

Debian/Ubuntu:

sudo apt-get update sudo apt-get install msodbcsql17

Red Hat/Oracle Linux:

sudo yum install msodbcsql17

SUSE:

sudo zypper install msodbcsql17

Alpine:

sudo apk add msodbcsql17

Feedback

We welcome your feedback. Please report issues on the SQL Server feedback site or open an issue on the ODBC Driver GitHub repository.

Migrating frontline mobile devices: Aligning stakeholders before real-world testing

Intune_Support_Team — Fri, 01 May 2026 19:37:13 GMT

By: Carol Burns - Principal Product Manager | Microsoft Intune and Sucheta Gawade, Microsoft MVP (Azure & Security / Intune)

Practitioner perspective from Sucheta Gawade, Microsoft MVP (Azure & Security / Intune), with deep experience in secure frontline mobility, including regulated healthcare environments.

In the previous article, we focused on understanding the reality of your frontline device estate - what devices you have, how they’re used, and which tasks they must support. Now that discovery is complete, the next step is to assess what you’ve found and align your people and processes before beginning real‑world testing with Microsoft Intune and representative users and devices. This is where you turn discovery into an actionable plan your team can execute in real operational conditions.

Many organizations refer to this stage as a Proof of Concept (POC) or pilot. In this article, we use these terms to describe limited real‑world validation of frontline workflows with representative users and devices, rather than internal IT feasibility testing. Use the pilot to confirm that users can reliably complete critical tasks in live operational environments before wider rollout.

Translate discovery into decisions

Discovery produces facts, but readiness requires decisions. Before beginning real‑world testing with representative users and devices, your team should be able to answer questions like:

Are we migrating “as‑is,” or do we plan on correcting identity and usage anti‑patterns such as shared credentials or personal use on corporate devices?
Which workflows are non‑negotiable and must work on day one, and which can be improved later?
Do we need to refresh hardware now, or can we migrate current devices and plan standardization at refresh time?
What are our top constraints (OS support, connectivity, etc.)?

A useful way to structure discovery output is to categorize findings and determine whether they support limited real‑world testing or require further alignment before proceeding.

Pre‑requisites for real‑world testing typically include clear ownership of devices and apps, supported OS versions, and a manageable device or OEM mix.

Items that often require alignment before real‑world testing include shared devices without a defined shared‑device model, shared credentials or unclear authentication approaches, personal use on corporate devices (which affects wipe/re‑enroll decisions), certified app or peripheral constraints, and network or certificate dependencies that could impact enrollment and compliance.

Identify the stakeholders you must align (and why)

Real‑world testing of frontline workflows depends on more than technical readiness. A clear stakeholder map helps surface operational dependencies early and ensures that limited validation activities can be conducted safely without disrupting day‑to‑day work.

Not every environment requires all of the roles listed below at this stage, but these are the most common stakeholders needed to support limited real‑world testing of frontline workflows.

Operational stakeholders

Stakeholder	Why they matter	What to align
Operations / business leadership	Define frontline outcomes and approve change windows.	Critical workflows, downtime tolerance, shift patterns, pilot locations, operational sign‑off criteria.
Funding owners / procurement	Discovery often uncovers refresh or licensing gaps.	Device and accessory funding, carrier plans, spares, and standardization strategy.
Change management	Testing may introduce new sign‑in flows or device behaviors.	Communications plan, support readiness, rollback and escalation processes, exception management.

In addition to operational alignment, technical readiness across supporting IT teams is required to ensure testing reflects production like conditions.

Technical and support stakeholders

Stakeholder	Why they matter	What to align
Endpoint or Microsoft Intune owners	Build policy, enrollment, apps, and compliance.	Device categories, management models, policy approach, rollout waves.
Architecture team	Ensure alignment with enterprise standards.	Reference architecture, lifecycle approach, dependency mapping.
Microsoft Identity / Microsoft Entra team	Underpins Conditional Access and shared‑device patterns.	Authentication model, shared device sign‑in patterns, break‑glass scenarios.
Network team	Enrollment depends on connectivity and certificate flows.	Wi‑Fi (EAP‑TLS), proxies, segmentation, roaming, known dead zones.
Security / risk / compliance	Define guardrails and exceptions.	Wipe policies, logging, least privilege, auditability.
App owners / vendors	Critical frontline workflows depend on app behavior.	Compatibility, offline behavior, deployment approach.
Support / service Desk	Manage user impact during testing.	Runbooks, escalation paths, enrollment troubleshooting, shift‑based support.
Project management (large environments)	Coordinate testing across teams.	Timeline, risk tracking, cross‑team communications.

Readiness checklist before real‑world testing

Real‑world testing often produces limited value when it focuses primarily on Microsoft Intune enrollment rather than operational use. Enrollment is a starting point, but the goal of this stage is to confirm that critical frontline workflows function reliably end‑to‑end in production‑like conditions.

Readiness area	Questions to consider before real‑world testing
Licensing	Do you have the correct Intune licenses for the devices or users in scope? Are any add-ons needed?
Identity	Is Microsoft Entra configured for your enrollment approach? Are Conditional Access policies ready for real‑world testing? For shared devices, what sign‑in model will you use?
Stakeholder alignment	Who owns the success criteria? Who approves the testing scope and change window? Who funds required accessories or device refresh?
Operational readiness	Who provides day‑to‑day support for test devices? What is the escalation path for a broken critical workflow? What is the rollback or recovery plan?
Device lifecycle decisions	Will you test by migrating existing devices as‑is, replacing end‑of‑life devices first, or using testing to define the future standard?
OEM and ecosystem readiness	Are the devices still supported by the OEM? Are required peripherals supported? Do rugged or certified requirements limit device options?

Lack of clear ownership for testing success criteria is a common cause of inconclusive pilots, particularly where operational workflows span multiple teams.

Decide what your real‑world testing must validate

Real‑world testing often produces limited value when it focuses primarily on enrollment rather than operational use. Enrollment is a starting point, but the goal of this stage is to confirm that critical frontline workflows function reliably end‑to‑end in production‑like conditions.

Real‑world testing should validate high‑value operational outcomes, ensuring:

Critical workflows function end‑to‑end
- Scanning, inventory, delivery confirmation, POS, etc.
- Session transitions match shift patterns
- Offline or degraded‑mode behavior works as expected (where relevant)
Security works without disrupting operations
- Compliance and Conditional Access do not block legitimate frontline activity
- Wipe and recovery processes are realistic for shared devices
- App protection controls align with user experience
Supportability is operationally viable
- Device reset and re‑enroll processes are documented
- Troubleshooting steps are known and repeatable
- Escalation paths exist for frontline‑impacting incidents
Representative device scenarios are included
Include the different frontline scenarios identified during discovery, such as:
- Shared vs assigned devices
- Different OEM models or OS versions
- Sites with known connectivity constraintso
- Common peripherals that may introduce migration risk (for example, scanners or printers)

Plan for future standardization (without delaying testing)

You may need to begin real‑world testing using the environment you have today. However, this stage can also be used to identify patterns that may shape future procurement and standardization decisions without delaying validation activities.

Practical prompts to add to your planning:

If you could reset procurement going forward, would you reduce OEM or device model sprawl?
What might your target “approved device set” look like for the next refresh cycle?
Which procurement models could support consistent enrollment, warranty coverage, and access to spares across shifts?

Standardization doesn’t need to be a prerequisite for real‑world testing, but it can become a valuable outcome of the migration effort over time.

Moving from assessment to real‑world testing

After you’ve aligned stakeholders, clarified dependencies, and defined what your real‑world testing must validate, you’re ready to move from assessment to limited operational testing with representative users and devices.

The key takeaway is this: discovery tells you what’s real, but readiness determines whether you can safely test it in live operational conditions.

As always, we welcome your feedback and experience. If you’ve already tested frontline workflows in operational conditions, what advice would you give organizations preparing for this stage? Share your thoughts in the comments below or reach out to us on X @IntuneSuppTeam.

Explore the From the frontlines: Frontline worker management with Microsoft Intune series for additional guidance on managing frontline workers and devices.

Title Plan Update - May 1, 2026

anbordianu — Fri, 01 May 2026 18:27:44 GMT

📁May 1, 2026 - Title Plan Now Available

Access the latest Instructor-Led Training (ILT) updates anytime at http://aka.ms/Courseware_Title_Plan to ensure you're always working from the most current version.

📌 Reminder: To help you stay informed more quickly and consistently, we’ve moved to a weekly publishing cadence for the title plan. This means each update may include fewer changes but ensures you’re always up to date.

Databricks Lakebase: The operational database for AI agents and apps

anishekkamal — Fri, 01 May 2026 18:23:22 GMT

Understanding the Evolution: From Lakehouse to Lakebase

The modern data landscape has long been characterized by a fundamental schism: Online Transaction Processing (OLTP) systems, designed for high-frequency, low-latency transactions in applications, and Online Analytical Processing (OLAP) systems, optimized for complex queries, reporting, and machine learning on vast datasets. This division historically necessitated intricate and often fragile Extract, Transform, Load (ETL) processes to move and synchronize data between these disparate environments, leading to increased complexity, data duplication, and governance challenges.

Databricks Lakehouse architecture emerged to unify data warehousing and data lake f

unctionalities for analytical workloads, offering the flexibility of data lakes with the performance and governance of data warehouses. However, a critical piece remained: native, high-performance OLTP capabilities directly within this unified environment. This is where Databricks Lakebase enters the picture, representing a significant evolution by bringing fully managed PostgreSQL OLTP capabilities directly into the Databricks Data Intelligence Platform.

Lakebase addresses the need for a single, governed platform that can seamlessly handle both transactional and analytical workloads, thereby simplifying data architectures, reducing operational overhead, and accelerating the development of real-time applications and AI agents. By integrating OLTP at the core of the lakehouse, Databricks aims to create a truly unified data and AI platform.

1.Visualizing the architectural shift: Lakebase integrates seamlessly within the Databricks Lakehouse ecosystem.

The Architectural Innovation: Separation of Compute and Storage

At the heart of Databricks Lakebase's efficiency and scalability lies its innovative architecture, which fundamentally separates compute from storage. Unlike traditional monolithic databases where these components are tightly coupled, Lakebase decouples them, offering distinct advantages:

Elastic Scaling and Cost Efficiency

The transactional compute layer in Lakebase is serverless and ephemeral, meaning it can scale up or down dynamically based on demand. This includes the ability to scale to zero during periods of inactivity, significantly optimizing cost by ensuring you only pay for the compute resources actively used. Data, on the other hand, is persisted directly into low-cost, durable cloud object storage (e.g., Azure Blob Storage) using open formats like Delta Lake. This design not only reduces storage costs but also prevents vendor lock-in and allows other engines within the Databricks platform to access the data directly.

Open Data Formats and Interoperability

By storing data in open formats, Lakebase ensures high interoperability within the Databricks ecosystem and beyond. This approach eliminates the need for complex and time-consuming ETL processes to move transactional data to the analytical layer, as the data is inherently accessible to both. This foundational integration streamlines data pipelines and provides a unified view of data across all workloads.

Key Technical Capabilities and Features

Databricks Lakebase offers a rich set of features that make it a compelling solution for modern data architectures:

PostgreSQL Compatibility: Lakebase provides full PostgreSQL semantics, including ACID transactions, indexing capabilities, and support for standard JDBC/psql clients. This familiarity allows developers to leverage existing skills and tools, minimizing the learning curve.
Fully Managed Service: Databricks handles the complexities of provisioning, scaling, patching, backups, and ensuring high availability, freeing up development teams to focus on application logic rather than database administration.
Managed Change Data Capture (CDC): A crucial feature, managed CDC ensures that operational data in Lakebase remains synchronized with Delta Lake tables for analytical consumption. This continuous synchronization is vital for keeping BI models and AI applications updated with the freshest transactional data.
Autoscaling (Lakebase Autoscaling): The latest iteration of Lakebase features intelligent autoscaling of compute resources. It dynamically adjusts Compute Units (CU) based on various metrics like CPU load, memory usage, and working set size, preventing performance bottlenecks and out-of-memory (OOM) issues. It also supports branching and instant restore, enhancing developer agility and operational resilience.
Databricks Apps Synergy: Lakebase is designed to serve as the transactional backend for Databricks Apps, enabling the creation and deployment of interactive applications directly on the platform, leveraging governed data and powerful analytics.

Governance, Security, and Cost Efficiency with Lakebase

Adopting Databricks Lakebase brings significant benefits in terms of data governance, security, and overall cost management, aligning with the principles of a modern data intelligence platform.

2.Reverse ETL with Lakebase simplifies data activation for operational analytics.

Unified Governance through Unity Catalog

One of Lakebase's most powerful integrations is with Unity Catalog, Databricks' unified governance solution. This integration provides a single pane of glass for managing data assets across the entire Databricks Data Intelligence Platform. Lakebase databases can be registered as catalogs within Unity Catalog, extending its robust governance framework to operational data. This means:

Consistent Access Control: Policies defined for your lakehouse data automatically apply to Lakebase, ensuring uniform security and access management across both operational and analytical workloads.
Centralized Auditing and Lineage: Unity Catalog provides comprehensive auditing capabilities and data lineage tracking for Lakebase assets, simplifying compliance and offering transparent insights into data flows.
Simplified Security Management: By unifying governance, organizations can reduce the complexity of managing security policies across disparate systems, enhancing overall data security posture.

Robust Security and Data Protection

Lakebase is designed with enterprise-grade security in mind, leveraging existing cloud infrastructure and Databricks' security features:

Network Integration: It integrates seamlessly with cloud networking services (e.g., Azure Private Link) for secure, private connectivity.
Identity Management: Integration with enterprise identity providers (e.g., Microsoft Entra ID) ensures secure authentication and authorization.
Data Encryption: Data is encrypted at rest and in transit, protecting sensitive information throughout its lifecycle.
High Availability and Disaster Recovery: As a fully managed service, Lakebase inherently provides features for high availability and point-in-time recovery, ensuring operational resilience.

Optimized Cost Efficiency

The architectural separation of compute and storage, coupled with advanced autoscaling capabilities, contributes to significant cost savings compared to traditional database architectures:

Pay-as-you-go Compute: With serverless and autoscaling compute, you only pay for the resources consumed during active processing, with the ability to scale down to zero when idle.
Low-Cost Storage: Leveraging economical cloud object storage for data persistence drastically reduces storage costs.
Reduced ETL Overhead: By eliminating the need for complex ETL pipelines between OLTP and OLAP, organizations save on infrastructure, development, and maintenance costs associated with data movement and transformation. This can lead to reported savings of 40-50% in many environments.

Lakebase in Action: Powering Real-Time Applications and AI Agents

Databricks Lakebase opens up new possibilities for building intelligent, data-driven applications that require both transactional capabilities and deep analytical insights. Its unified approach simplifies development and accelerates time-to-market for innovative solutions.

Real-World Use Cases

Personalized Recommendations: Build real-time recommendation engines that leverage fresh transactional data from Lakebase to provide immediate and highly relevant suggestions to users.
Customer Segmentation and Real-Time Updates: Maintain and update customer profiles and segments in real-time, enabling personalized experiences and targeted marketing campaigns.
Feature Stores for Machine Learning: Utilize Lakebase as a feature store to serve low-latency features to AI models, ensuring that predictions and decisions are based on the most current data.
Stateful AI Agents: Develop AI agents that can maintain conversational state and interact dynamically with users, using Lakebase as a reliable backend for transactional data.
Order Processing Systems: Implement operational applications that require high-frequency reads, writes, and updates, such as order management or inventory systems, directly on the Databricks platform.
Interactive Workflow Tools: Create interactive data applications and dashboards that allow users to both view analytical insights and perform transactional updates within the same environment.

A Practical Code Snippet

Developing with Lakebase feels familiar due to its PostgreSQL compatibility. Here’s a simple example demonstrating basic CRUD (Create, Read, Update, Delete) operations within a Lakebase table:

-- Create a schema for your application CREATE SCHEMA app AUTHORIZATION CURRENT_USER; -- Create a table to store session data for an AI agent CREATE TABLE app.sessions ( session_id UUID PRIMARY KEY, user_id TEXT NOT NULL, state JSONB NOT NULL, created_at TIMESTAMPTZ DEFAULT now(), updated_at TIMESTAMPTZ ); -- Create an index to optimize queries on agent status CREATE INDEX ON app.sessions ((state->>'agentStatus')); -- Insert a new session record INSERT INTO app.sessions(session_id, user_id, state) VALUES (gen_random_uuid(), 'u-123', '{"agentStatus":"active","score":0.82}'); -- Update an existing session's state UPDATE app.sessions SET state = jsonb_set(state, '{score}', '0.91'::jsonb), updated_at = now() WHERE user_id='u-123'; -- Query active sessions SELECT user_id, state->>'score' as current_score FROM app.sessions WHERE (state->>'agentStatus') = 'active';

This SQL snippet showcases how developers can interact with Lakebase using standard PostgreSQL syntax, enabling rapid application development within the Databricks environment.

The Lakebase Advantage: Performance and Reliability

Beyond its unified architecture, Lakebase is engineered for predictable performance and robust reliability, essential for mission-critical operational applications.

The radar chart above provides an opinionated comparison of Databricks Lakebase against traditional OLTP systems across several key attributes. Lakebase demonstrates superior performance predictability, dynamic scalability, cost efficiency, and ease of management, coupled with strong data governance due to its integration with Unity Catalog. Traditional OLTP systems, while effective for their specific purposes, often score lower in these cloud-native, unified data platform metrics.

Reliability Features for Business Continuity

Lakebase integrates several critical reliability features that ensure business continuity and data integrity:

Branching: This feature allows developers to create isolated, production-like environments for testing changes without affecting the main operational database. It promotes safer development practices and faster iteration cycles.
Instant Restore and Point-in-Time Recovery (PITR): In the event of data corruption or accidental deletion, Lakebase enables quick restoration to a previous state, minimizing downtime and ensuring data resilience.
High Availability: As a managed service, Lakebase is designed for high availability, with automated failover mechanisms and robust infrastructure ensuring continuous operation.

Validation and Troubleshooting: Ensuring a Smooth Lakebase Experience

Successful implementation and ongoing operation of Databricks Lakebase rely on proper validation and an understanding of common troubleshooting steps. This section provides a framework for ensuring your Lakebase deployment meets performance and reliability expectations.

An introductory video to Lakebase, explaining its core functionality and benefits for data apps and AI agents.

Key Validation Steps

After provisioning and configuring your Lakebase instance, it's crucial to perform a series of validation tests:

Connectivity Verification: Confirm successful connections from your applications or development tools (e.g., psql, JDBC clients) to the Lakebase instance. Ensure that Unity Catalog registration is visible and properly configured for governance.
Performance Baseline: Conduct baseline QPS (Queries Per Second) tests and monitor latency under expected load conditions. Validate that autoscaling events occur as anticipated and that performance targets are met.
Data Synchronization (CDC): Test the end-to-end data flow by inserting/updating records in Lakebase and verifying their timely appearance in Delta Lake tables via managed CDC. If reverse synchronization (Delta to Lakebase) is configured, validate that as well.
Governance and Security Checks: Confirm that Unity Catalog permissions are correctly enforced for Lakebase assets and that audit logs accurately reflect data access and modification events. Verify network security configurations (e.g., Private Link) are functioning as intended.

Common Troubleshooting Scenarios

While Lakebase is designed for stability, understanding potential issues and their resolutions is key to efficient operation:

Problem Area	Symptom	Potential Cause(s)	Troubleshooting Step(s)
Performance	High latency, slow queries, autoscaling not triggering as expected.	Inefficient queries, missing indexes, insufficient compute resources, working set exceeding memory.	Inspect query plans, add appropriate indexes, monitor CU utilization, review autoscaling logs, consider increasing initial compute capacity if persistently underperforming.
Data Sync (CDC)	Stale data in Delta Lake, sync job failures, data inconsistencies.	Incorrect Unity Catalog permissions, CDC configuration errors, network issues, regional feature limitations.	Verify Unity Catalog access for CDC process, check CDC job logs for errors, confirm network connectivity between Lakebase and Delta Lake, consult Databricks documentation for regional CDC availability.
Connectivity	Unable to connect from application, authentication failures.	Incorrect connection strings, firewall rules blocking access, misconfigured private endpoints, invalid credentials/tokens.	Double-check connection parameters, review network security group (NSG) and firewall rules, validate Private Link configuration, ensure correct user/service principal credentials.
Governance	Unauthorized access, unexpected data visibility, audit log discrepancies.	Incorrect Unity Catalog access policies, schema mismatches, misconfigured external locations.	Review and refine Unity Catalog grants on Lakebase catalogs and schemas, verify external location configurations, ensure consistent data object naming conventions.
Feature Limitations	Specific PostgreSQL features or extensions not working.	Managed environment restrictions, unsupported extensions.	Consult Databricks documentation for supported PostgreSQL versions and extensions in Lakebase. Adapt application logic to use supported alternatives if necessary.

By proactively monitoring and understanding these aspects, Cloud Solution Architects can ensure robust and efficient operation of Lakebase within their Databricks ecosystem.

Conclusion

Databricks Lakebase represents a pivotal advancement in data architecture, fundamentally reshaping how organizations approach operational and analytical workloads. By seamlessly integrating a fully managed PostgreSQL OLTP engine directly into the Databricks Data Intelligence Platform, Lakebase addresses the long-standing challenge of data fragmentation. This unification not only simplifies complex ETL processes and reduces operational overhead but also extends robust governance and security through Unity Catalog across the entire data estate. The innovative separation of compute and storage, coupled with intelligent autoscaling, delivers unparalleled cost efficiency and dynamic performance. For Cloud Solution Architects, Lakebase offers a compelling path to building scalable, real-time applications and sophisticated AI agents, leveraging fresh transactional data alongside comprehensive analytical insights—all within a single, consistent, and highly performant environment. This strategic evolution of the lakehouse architecture empowers enterprises to unlock new levels of agility, innovation, and data-driven decision-making.