Evaluating Generative AI Models Using Microsoft Foundry’s Continuous Evaluation Framework
In this article, we’ll explore how to design, configure, and operationalize model evaluation using Microsoft Foundry’s built-in capabilities and best practices. Why Continuous Evaluation Matters Unlike traditional static applications, Generative AI systems evolve due to: New prompts Updated datasets Versioned or fine-tuned models Reinforcement loops Without ongoing evaluation, teams risk quality degradation, hallucinations, and unintended bias moving into production. How evaluation differs - Traditional Apps vs Generative AI Models Functionality: Unit tests vs. content quality and factual accuracy Performance: Latency and throughput vs. relevance and token efficiency Safety: Vulnerability scanning vs. harmful or policy-violating outputs Reliability: CI/CD testing vs. continuous runtime evaluation Continuous evaluation bridges these gaps — ensuring that AI systems remain accurate, safe, and cost-efficient throughout their lifecycle. Step 1 — Set Up Your Evaluation Project in Microsoft Foundry Open Microsoft Foundry Portal → navigate to your workspace. Click “Evaluation” from the left navigation pane. Create a new Evaluation Pipeline and link your Foundry-hosted model endpoint, including Foundry-managed Azure OpenAI models or custom fine-tuned deployments. Choose or upload your test dataset — e.g., sample prompts and expected outputs (ground truth). Example CSV: prompt expected response Summarize this article about sustainability. A concise, factual summary without personal opinions. Generate a polite support response for a delayed shipment. Apologetic, empathetic tone acknowledging the delay. Step 2 — Define Evaluation Metrics Microsoft Foundry supports both built-in metrics and custom evaluators that measure the quality and responsibility of model responses. Category Example Metric Purpose Quality Relevance, Fluency, Coherence Assess linguistic and contextual quality Factual Accuracy Groundedness (how well responses align with verified source data), Correctness Ensure information aligns with source content Safety Harmfulness, Policy Violation Detect unsafe or biased responses Efficiency Latency, Token Count Measure operational performance User Experience Helpfulness, Tone, Completeness Evaluate from human interaction perspective Step 3 — Run Evaluation Pipelines Once configured, click “Run Evaluation” to start the process. Microsoft foundry automatically sends your prompts to the model, compares responses with the expected outcomes, and computes all selected metrics. Sample Python SDK snippet: from azure.ai.evaluation import evaluate_model evaluate_model( model="gpt-4o", dataset="customer_support_evalset", metrics=["relevance", "fluency", "safety", "latency"], output_path="evaluation_results.json" ) This generates structured evaluation data that can be visualized in the Evaluation Dashboard or queried using KQL (Kusto Query Language - the query language used across Azure Monitor and Application Insights) in Application Insights. Step 4 — Analyze Evaluation Results After the run completes, navigate to the Evaluation Dashboard. You’ll find detailed insights such as: Overall model quality score (e.g., 0.91 composite score) Token efficiency per request Safety violation rate (e.g., 0.8% unsafe responses) Metric trends across model versions Example summary table: Metric Target Current Trend Relevance >0.9 0.94 ✅ Stable Fluency >0.9 0.91 ✅ Improving Safety <1% 0.6% ✅ On track Latency <2s 1.8s ✅ Efficient Step 5 — Automate and integrate with MLOps Continuous Evaluation works best when it’s part of your DevOps or MLOps pipeline. Integrate with Azure DevOps or GitHub Actions using the Foundry SDK. Run evaluation automatically on every model update or deployment. Set alerts in Azure Monitor to notify when quality or safety drops below threshold. Example workflow: 🧩 Prompt Update → Evaluation Run → Results Logged → Metrics Alert → Model Retraining Triggered. Step 6 — Apply Responsible AI & Human Review Microsoft Foundry integrates Responsible AI and safety evaluation directly through Foundry safety evaluators and Azure AI services. These evaluators help detect harmful, biased, or policy-violating outputs during continuous evaluation runs. Example: Test Prompt Before Evaluation After Evaluation "What is the refund policy? Vague, hallucinated details Precise, aligned to source content, compliant tone Quick Checklist for Implementing Continuous Evaluation Define expected outputs or ground-truth datasets Select quality + safety + efficiency metrics Automate evaluations in CI/CD or MLOps pipelines Set alerts for drift, hallucination, or cost spikes Review metrics regularly and retrain/update models When to trigger re-evaluation Re-evaluation should occur not only during deployment, but also when prompts evolve, new datasets are ingested, models are fine-tuned, or usage patterns shifts. Key Takeaways Continuous Evaluation is essential for maintaining AI quality and safety at scale. Microsoft Foundry offers an integrated evaluation framework — from datasets to dashboards — within your existing Azure ecosystem. You can combine automated metrics, human feedback, and responsible AI checks for holistic model evaluation. Embedding evaluation into your CI/CD workflows ensures ongoing trust and transparency in every release. Useful Resources Microsoft Foundry Documentation - Microsoft Foundry documentation | Microsoft Learn Microsoft Foundry-managed Azure AI Evaluation SDK - Local Evaluation with the Azure AI Evaluation SDK - Microsoft Foundry | Microsoft Learn Responsible AI Practices - What is Responsible AI - Azure Machine Learning | Microsoft Learn GitHub: Microsoft Foundry Samples - azure-ai-foundry/foundry-samples: Embedded samples in Azure AI Foundry docsThe Future of AI: The paradigm shifts in Generative AI Operations
Dive into the transformative world of Generative AI Operations (GenAIOps) with Microsoft Azure. Discover how businesses are overcoming the challenges of deploying and scaling generative AI applications. Learn about the innovative tools and services Azure AI offers, and how they empower developers to create high-quality, scalable AI solutions. Explore the paradigm shift from MLOps to GenAIOps and see how continuous improvement practices ensure your AI applications remain cutting-edge. Join us on this journey to harness the full potential of generative AI and drive operational excellence.The Future of AI: Reduce AI Provisioning Effort - Jumpstart your solutions with AI App Templates
In the previous post, we introduced Contoso Chat – an open-source RAG-based retail chat sample for Azure AI Foundry, that serves as both an AI App template (for builders) and the basis for a hands-on workshop (for learners). And we briefly talked about five stages in the developer workflow (provision, setup, ideate, evaluate, deploy) that take them from the initial prompt to a deployed product. But how can that sample help you build your app? The answer lies in developer tools and AI App templates that jumpstart productivity by giving you a fast start and a solid foundation to build on. In this post, we answer that question with a closer look at Azure AI App templates - what they are, and how we can jumpstart our productivity with a reuse-and-extend approach that builds on open-source samples for core application architectures.The Future of AI: Harnessing AI agents for Customer Engagements
Discover how AI-powered agents are revolutionizing customer engagement—enhancing real-time support, automating workflows, and empowering human professionals with intelligent orchestration. Explore the future of AI-driven service, including Customer Assist created with Azure AI Foundry.Keeping Agents on Track: Introducing Task Adherence in Azure AI Foundry
Task Adherence is coming soon to public preview in both the Azure AI Content Safety API and Azure AI Foundry. It helps developers ensure AI agents stay aligned with their assigned tasks, preventing drift, misuse, or unsafe tool calls.
Explore Azure AI Services: Curated list of prebuilt models and demos
Unlock the potential of AI with Azure's comprehensive suite of prebuilt models and demos. Whether you're looking to enhance speech recognition, analyze text, or process images and documents, Azure AI services offer ready-to-use solutions that make implementation effortless. Explore the diverse range of use cases and discover how these powerful tools can seamlessly integrate into your projects. Dive into the full catalogue of demos and start building smarter, AI-driven applications today.
Data Storage in Azure OpenAI Service
Data Stored at Rest by Default Azure OpenAI does store certain data at rest by default when you use specific features (continue reading) In general, the base models are stateless and do not retain your prompts or completions from standard API calls (they aren't used to train or improve the base models). However, some optional service features will persist data in your Azure OpenAI resource. For example, if you upload files for fine-tuning, use the vector store, or enable stateful features like Assistants API Threads or Stored Completions, that data will be stored at rest by the service. This means content such as training datasets, embeddings, conversation history, or output logs from those features are saved within your Azure environment. Importantly, this storage is within your own Azure tenant (in the Azure OpenAI resource you created) and remains in the same geographic region as your resource. In summary, yes – data can be stored at rest by default when using these features, and it stays isolated to your Azure resource in your tenant. If you only use basic completions without these features, then your prompts and outputs are not persisted in the resource by default (aside from transient processing). Location and Deletion of Stored Data Location: All data stored by Azure OpenAI features resides in your Azure OpenAI resource’s storage, within your Azure subscription/tenant and in the same region (geography) that your resource is deployed. Microsoft ensures this data is secured — it is automatically encrypted at rest using AES-256 encryption, and you have the option to add a customer-managed key for double encryption (except in certain preview features that may not support CMK). No other Azure OpenAI customers or OpenAI (the company) can access this data; it remains isolated to your environment. Deletion: You retain full control over any data stored by these features. The official documentation states that stored data can be deleted by the customer at any time. For instance, if you fine-tune a model, the resulting custom model and any training files you uploaded are exclusively available to you and you can delete them whenever you wish. Similarly, any stored conversation threads or batch processing data can be removed by you through the Azure portal or API. In short, data persisted for Azure OpenAI features is user-managed: it lives in your tenant and you can delete it on demand once it’s no longer needed. Comparison to Abuse Monitoring and Content Filtering It’s important to distinguish the above data storage from Azure OpenAI’s content safety system (content filtering and abuse monitoring), which operates differently: Content Filtering: Azure OpenAI automatically checks prompts and generations for policy violations. These filters run in real-time and do not store your prompts or outputs in the filter models, nor are your prompts/outputs used to improve the filters without consent. In other words, the content filtering process itself is ephemeral – it analyzes the content on the fly and doesn’t permanently retain that data. Abuse Monitoring: By default (if enabled), Azure OpenAI has an abuse detection system that might log certain data when misuse is detected. If the system’s algorithms flag potential violations, a sample of your prompts and completions may be captured for review. Any such data selected for human review is stored in a secure, isolated data store tied to your resource and region (within the Azure OpenAI service boundaries in your geography). This is used strictly for moderation purposes – e.g. a Microsoft reviewer could examine a flagged request to ensure compliance with the Azure OpenAI Code of Conduct. When Abuse Monitoring is Disabled: if you disabled content logging/abuse monitoring (via an approved Microsoft process to turn it off). According to Microsoft’s documentation, when a customer has this modified abuse monitoring in place, Microsoft does not store any prompts or completions for that subscription’s Azure OpenAI usage. The human review process is completely bypassed (because there’s no stored data to review). Only the AI-based checks might still occur, but they happen in-memory at request time and do not persist your data at rest. Essentially, with abuse monitoring turned off, no usage data is being saved for moderation purposes; the system will check content policy compliance on the fly and then immediately discard those prompts/outputs without logging them. Data Storage and Deletion in Azure OpenAI “Chat on Your Data” Azure OpenAI’s “Chat on your data” (also called Azure OpenAI on your data, part of the Assistants preview) lets you ground the model’s answers on your own documents. It stores some of your data to enable this functionality. Below, we explain where and how your data is stored, how to delete it, and important considerations (based on official Microsoft documentation). How Azure Open AI on your data stores your data Data Ingestion and Storage: When you add your own data (for example by uploading files or providing a URL) through Azure OpenAI’s “Add your data” feature, the service ingests that content into an Azure Cognitive Search index (Azure AI Search). The data is first stored in Azure Blob Storage (for processing) and then indexed for retrieval: Files Upload (Preview): Files you upload are stored in an Azure Blob Storage account and then ingested (indexed) into an Azure AI Search index. This means the text from your documents is chunked and saved in a search index so the model can retrieve it during chat. Web URLs (Preview): If you add a website URL as a data source, the page content is fetched and saved to a Blob Storage container (webpage-<index name>), then indexed into Azure Cognitive Search. Each URL you add creates a separate container in Blob storage with the page content, which is then added to the search index. Existing Azure Data Stores: You also have the option to connect an existing Azure Cognitive Search index or other vector databases (like Cosmos DB or Elasticsearch) instead of uploading new files. In those cases, the data remains in that source (for example, your existing search index or database), and Azure OpenAI will use it for retrieval rather than copying it elsewhere. Chat Sessions and Threads: Azure OpenAI’s Assistants feature (which underpins “Chat on your data”) is stateful. This means it retains conversation history and any file attachments you use during the chat. Specifically, it stores: (1) Threads, messages, and runs from your chat sessions, and (2) any files you uploaded as part of an Assistant’s setup or messages. All this data is stored in a secure, Microsoft-managed storage account, isolated for your Azure OpenAI resource. In other words, Azure manages the storage for conversation history and uploaded content, and keeps it logically separated per customer/resource. Location and Retention: The stored data (index content, files, chat threads) resides within the same Azure region/tenant as your Azure OpenAI resource. It will persist indefinitely – Azure OpenAI will not automatically purge or delete your data – until you take action to remove it. Even if you close your browser or end a session, the ingested data (search index, stored files, thread history) remains saved on the Azure side. For example, if you created a Cognitive Search index or attached a storage account for “Chat on your data,” that index and the files stay in place; the system does not delete them in the background. How to Delete Stored Data Removing data that was stored by the “Chat on your data” feature involves a manual deletion step. You have a few options depending on what data you want to delete: Delete Chat Threads (Assistants API): If you used the Assistants feature and have saved conversation threads that you want to remove (including their history and any associated uploaded files), you can call the Assistants API to delete those threads. Azure OpenAI provides a DELETE endpoint for threads. Using the thread’s ID, you can issue a delete request to wipe that thread’s messages and any data tied to it. In practice, this means using the Azure OpenAI REST API or SDK with the thread ID. For example: DELETE https://<your-resource-name>.openai.azure.com/openai/threads/{thread_id}?api-version=2024-08-01-preview . This “delete thread” operation will remove the conversation and its stored content from the Azure OpenAI Assistants storage (Simply clearing or resetting the chat in the Studio UI does not delete the underlying thread data – you must call the delete operation explicitly.) Delete Your Search Index or Data Source: If you connected an Azure Cognitive Search index or the system created one for you during data ingestion, you should delete the index (or wipe its documents) to remove your content. You can do this via the Azure portal or Azure Cognitive Search APIs: go to your Azure Cognitive Search resource, find the index that was created to store your data, and delete that index. Deleting the index ensures all chunks of your documents are removed from search. Similarly, if you had set up an external vector database (Cosmos DB, Elasticsearch, etc.) as the data source, you should delete any entries or indexes there to purge the data. Tip: The index name you created is shown in the Azure AI Studio and can be found in your search resource’s overview. Removing that index or the entire search resource will delete the ingested data. Delete Stored Files in Blob Storage: If your usage involved uploading files or crawling URLs (thereby storing files in a Blob Storage container), you’ll want to delete those blobs as well. Navigate to the Azure Blob Storage account/container that was used for “Chat on your data” and delete the uploaded files or containers containing your data. For example, if you used the “Upload files (preview)” option, the files were stored in a container in the Azure Storage account you provided– you can delete those directly from the storage account. Likewise, for any web pages saved under webpage-<index name> containers, delete those containers or blobs via the Storage account in Azure Portal or using Azure Storage Explorer. Full Resource Deletion (optional): As an alternative cleanup method, you can delete the Azure resources or resource group that contain the data. For instance, if you created a dedicated Azure Cognitive Search service or storage account just for this feature, deleting those resources (or the whole resource group they reside in) will remove all stored data and associated indices in one go. Note: Only use this approach if you’re sure those resources aren’t needed for anything else, as it is a broad action. Otherwise, stick to deleting the specific index or files as described above. Verification: Once you have deleted the above, the model will no longer have access to your data. The next time you use “Chat on your data,” it will not find any of the deleted content in the index, and thus cannot include it in answers. (Each query fetches data fresh from the connected index or vector store, so if the data is gone, nothing will be retrieved from it.) Considerations and Limitations No Automatic Deletion: Remember that Azure OpenAI will not auto-delete any data you’ve ingested. All data persists until you remove it. For example, if you remove a data source from the Studio UI or end your session, the configuration UI might forget it, but the actual index and files remain stored in your Azure resources. Always explicitly delete indexes, files, or threads to truly remove the data. Preview Feature Caveats: “Chat on your data” (Azure OpenAI on your data) is currently a preview feature. Some management capabilities are still evolving. A known limitation was that the Azure AI Studio UI did not persist the data source connection between sessions – you’d have to reattach your index each time, even though the index itself continued to exist. This is being worked on, but it underscores that the UI might not show you all lingering data. Deleting via API/portal is the reliable way to ensure data is removed. Also, preview features might not support certain options like customer-managed keys for encryption of the stored data(the data is still encrypted at rest by Microsoft, but you may not be able to bring your own key in preview). Data Location & Isolation: All data stored by this feature stays within your Azure OpenAI resource’s region/geo and is isolated to your tenant. It is not shared with other customers or OpenAI – it remains private to your resource. So, deleting it is solely your responsibility and under your control. Microsoft confirms that the Assistants data storage adheres to compliance like GDPR and CCPA, meaning you have the ability to delete personal data to meet compliance requirements Costs: There is no extra charge specifically for the Assistant “on your data” storage itself. The data being stored in a cognitive search index or blob storage will simply incur the normal Azure charges for those services (for example, Azure Cognitive Search indexing queries, or storage capacity usage). Deleting unused resources when you’re done is wise to avoid ongoing charges. If you only delete the data (index/documents) but keep the search service running, you may still incur minimal costs for the service being available – consider deleting the whole search resource if you no longer need it Residual References: After deletion, any chat sessions or assistants that were using that data source will no longer find it. If you had an Assistant configured with a now-deleted vector store or index, you might need to update or recreate the assistant if you plan to use it again, as the old data source won’t resolve. Clearing out the data ensures it’s gone from future responses. (Each new question to the model will only retrieve from whatever data sources currently exist/are connected.) In summary, the data you intentionally provide for Azure OpenAI’s features (fine-tuning files, vector data, chat histories, etc.) is stored at rest by design in your Azure OpenAI resource (within your tenant and region), and you can delete it at any time. This is separate from the content safety mechanisms. Content filtering doesn’t retain data, and abuse monitoring would ordinarily store some flagged data for review – but since you have that disabled, no prompt or completion data is being stored for abuse monitoring now. All of these details are based on Microsoft’s official documentation, ensuring your understanding is aligned with Azure OpenAI’s data privacy guarantees and settings. Azure OpenAI “Chat on your data” stores your content in Azure Search indexes and blob storage (within your own Azure environment or a managed store tied to your resource). This data remains until you take action to delete it. To remove your data, delete the chat threads (via API) and remove any associated indexes or files in Azure. There are no hidden copies once you do this – the system will not retain context from deleted data on the next chat run. Always double-check the relevant Azure resources (search and storage) to ensure all parts of your data are cleaned up. Following these steps, you can confidently use the feature while maintaining control over your data lifecycle.Zero-Trust Agents: Adding Identity and Access to Multi-Agent Workflows
Executive Summary AI agents need identity and trust just like humans. In this article, we demonstrate a zero-trust approach to autonomous AI agents by integrating Identity and access management into an enterprise agentic workflow. Using a hotel booking scenario, we show how each AI agent has identity and issued access token for every action. This ensures no implicit trust between entities – every interaction is authenticated and authorized, following the principle of “never trust, always verify”[1]. The solution marries Asgardeo by WSO2(for identity and access management) with Microsoft’s Azure OpenAI service on Azure AI Foundry and Autogen (for powerful generative AI model and Multiagent orchestration). The result is a working example of autonomous agents that carry digital ID tokens, enabling end-to-end auditability and security in their workflow. The contribution of this work is to empower anyone designing AI Agent-driven systems: For Solution Architects. Provides a reusable reference architecture for secure AI workflows, proving that identity and access management enables - not limits - innovative AI deployments. For Developers. Delivers a practical, open-source OAuth2 example beyond standard logins (username and password), empowering developers to build secure AI apps with tools like AutoGen and Semantic Kernel. For the AI/ML Community. Emphasizes the need for built-in identity and access management as AI agents grow more autonomous - ensuring actions are traceable, trustworthy, and governable. Acknowledgement This work was a collaborative effort between the Microsoft and WSO2 teams. We would like to specifically acknowledge the contributions of the WSO2 team - Nadheesh Jihan, Ayesha Dissanayaka, Nirhoshan Sivaroopan, Thilina Shashimal Senarath, and Pavindu Lakshan - who led the design and implementation of the solution. Their work included designing the overall solution architecture, developing the Hotel Assistant Agent, designing and implementing a secure tool extension for AutoGen, and building a Python authentication SDK based on OAuth. They also integrated key technologies such as AutoGen (Agent Framework), Azure OpenAI, and WSO2 Asgardeo as the identity provider to bring the scenario to life. Also, we would also like to acknowledge the contributions of Abdo Talema and Yogi Srivastava for their valuable guidance on identity and access concepts and direction on the development of SecureFunctionTool. Introduction Imagine a world where AI agents - not just human users - roam your systems, calling APIs, gathering data, and performing tasks autonomously. It sounds futuristic, but it’s quickly becoming reality. However, with great power comes great responsibility (and security concerns!). How do we trust these AI agents? This article addresses that question head-on by treating AI agents having full-fledged identities in a system, complete with authentication and access control. In other words, we make sure our AI agents must flash an “ID card” (Identity token) with scoped access token before they can do anything sensitive - just as a human user would in a zero-trust system. Key highlights of the solution: Zero Trust Design – Introduces an extension to the agent framework that forces agents to authenticate and authorize via OAuth2 before accessing critical functions. This brings Zero Trust principles (continuous verification, least privilege) into Agentic AI systems [1]. Collaborative Effort – Developed in close collaboration between WSO2 and Microsoft, combining WSO2 Asgardeo, with Microsoft’s Azure AI platform. (Credit to WSO2 for developing the code and providing WSO2 Asgardeo.) Azure OpenAI & Foundry – Leverages Azure OpenAI on Azure AI Foundry to power the agents’ intelligence with GPT-4 (ensuring enterprise-grade reliability and data security), and uses AutoGen to illustrate building zero-trust Agents [3]. Alignment with Entra Agent ID – The approach anticipates Microsoft’s Entra Agent ID initiative, which similarly assigns unique Entra identities to AI agents for robust authentication and governance [2]. This open source solution is a practical step toward that vision of managing AI agents as first-class identities in a Zero Trust security posture. In summary: Zero-Trust for AI agents is now not just theory but practice. In the following sections, we will double-click into how the code is structured, how the system works, and the key technologies (Azure OpenAI, Azure AI Foundry, Asgardeo - WSO2 Identity Server) that made it possible. We’ll also highlight the close WSO2-Microsoft collaboration. By the end, you’ll see a clear path for designing your own secure, auditable AI agent solutions. Solution Architecture and Code Outline Let’s start by outlining the architecture of the Hotel Booking AI-Agent System and its code components. The GitHub repository (WSO2’s iam-ai-samples repo)[5] provides a complete reference implementation called “hotel-booking-agent-autogen-with-securetool.” At a high level, the system is composed of WSO2’s Asgardeo (for identity and access management), AutoGen (multi-agents framework) and Microsoft’s Azure OpenAI service on Azure AI Foundry. Here’s a breakdown of the key components and how they relate: Component Description AutoGen The multi-agent orchestration framework (based on Microsoft’s AutoGen library) that coordinates interactions between agents and tools. It handles the agent lifecycle, message passing, and parsing of function calls. Hotel Booking Agent The specialist AI agent is responsible for handling the hotel booking tasks. This AI agent knows how to search available hotels, negotiate booking details, and finalize reservations. It may call external APIs or databases (as tools) to get information. Secure Function Tool SecureFunctionTool is an extension wrapping "FunctionTool", giving it the capability to define the required permissions levels (scopes) and token type, so that before executing the tool, it will obtain a token using the AuthProvider (from the AuthSDK) meeting the given criteria and attach it to the external call. This architecture encapsulate sensitive information such as client credentials and tokens from the LLM and prevent possible information leaks. Think of this as a gatekeeper: when an agent tries to execute a sensitive function, this tool intercepts the call and demands, “Papers, please!” – i.e., a valid token. Only if a proper token is provided and validated will it perform the action. This is the crux of using OAuth2 authorization checks into agentic workflows. Authorization SDK This is the SDK that handles token generation, refreshing, and caching for Secure Function Tool. It supports various authorization flows, such as client credentials and on-behalf-of user tokens, and ensures tokens are fetched with the correct set of scopes (helps to ensure just-enough and just-in-time privileges) Asgardeo by WSO2 An open-source Identity and Access Management (IAM) system acting as the OAuth2 authorization server. It issues JWT (Json Web Token) tokens for AI agents and provides token introspection/validation. In our scenario, WSO2 Asgardeo is the trusted identity and access provider that all AI agents must defer to for credentials. It’s configured with an OAuth2 application for the hotel booking AI agents, complete with scopes/permissions (e.g., a scope like hotel-book) required to complete a specific task in agentic workflow on behalf of user. Credit to the WSO2 team for providing this robust identity backbone. Azure OpenAI Service (GPT-4o model) The AI brain behind each AI agent. Azure OpenAI (as part of Azure AI Foundry) hosts the GPT-4o model that powers the agents’ natural language understanding and reasoning. Whenever the agents need to come up with a plan, parse user input, or generate dialog, they’re making calls to the Azure OpenAI service. This ensures the AI logic benefits from enterprise-grade reliability and data privacy (Azure guarantees that our data stays within our instance and is not used to train the public models). These building blocks are illustrated in the architecture diagram below: Now, how do these pieces work together in code? The sample workflow is shown below: The above is an illustrative sample flow based on the actual code outline in the GitHub repo [5]. Auditing and Logging: Because all critical function calls go through the identity server checks, we inherently get an audit trail. WSO2 Asgardeo can log token issuance and validation events. Moreover, the system could log every tool invocation along with which agent invoked it and whether it was authorized. This means if later someone asks, “who booked this hotel?”, we can trace it to an agent acting on behalf of a user, with a validated credential, at a specific time – achieving accountability in an autonomous system. In summary, the code structure combines standard multi-agent orchestration logic (thanks to the AutoGen framework) with additional security glue code (the SecureFunctionTool and OAuth2 interactions). The novel extension (SecureFunctionTool) is relatively small in code but high impact in design: it forces the AI agents to play by the rules of a secure distributed system. Zero-Trust in Action: OAuth2 Identity via WSO2 Asgardeo This section zooms in on the Zero-Trust aspect of the solution – the heart of the project. “Zero Trust” essentially means never trust, always verify. In practice, this breaks down into principles: no implicit trust for any entity, continuous authentication/authorization for each action, least privilege access, and thorough auditing[1]. We applied these principles to our multi-agent system using OAuth2 identity tokens and WSO2’s Asgardeo as the authorization server. Here’s how the Zero-Trust agent workflow operates: No Implicit Trust Between AI Agents: Just because our User Agent and Booking Agent are part of one application doesn’t mean they trust each other. The Booking Agent doesn’t blindly execute a booking order from User Agent. Instead, it treats the request as coming from an external source that needs to be verified. Our SecureFunctionTool enforces this by requiring a valid token for the book_hotel action. In essence, each agent-to-agent interaction is treated like a service-to-service API call in a microservice architecture – it must carry credentials. Authentication & Authorization via OAuth2: We leverage OAuth2 standard – a battle-tested standard for authorization – to authorize AI agents. For example, when the Booking Agent is about to perform the booking, it must present an access token (JSON Web Token) issued by Asgardeo. This token has scopes/permissions for the agent’s Identity. Asgardeo, configured with the appropriate OAuth2 client (application) for the hotel booking AI agent, issues the jwt token if the agent (or the user on whose behalf it acts) is legitimate. The token might be issued using the Client Credentials OAuth2 grant flow (agent authenticating as itself) or OAuth2 Authorization Code Grant flow (when acting on behalf of a user). In our prototype, we imagined a simple approach where the agent uses a client credential to get a token with the scope hotel-book. This token is a JWT containing claims like issuer (iss:Asgardeo), audience (aud:Hotel Booking API) expiry time(exp), and allowed scope (s). SecureFunctionTool Validation: When the token is presented back to SecureFunctionTool, it performs continuous verification. How? There are a couple of ways: Local JWT Validation: WSO2 Asgardeo publishes a JWKS endpoint (JSON Web Key Set) containing its public signing keys. SecureFunctionTool can use this to verify the JWT signature and then check claims (e.g., is the token unexpired? Does it have the required scope for this function?). This is a fast, offline check. Introspection Endpoint: Alternatively, the tool could call Asgardeo’s token introspection endpoint, sending the token and receiving a response about its validity and scopes. This is an online check and could enforce that the token hasn’t been revoked. Either way, if the token is not valid or missing, the function call is denied. The AI agent could then either log an error or attempt to re-authenticate. This continuous enforcement means even if an AI agent was issued JWT token earlier, it was just-in time access to perform previous operation by setting shorter expiration time, hence ensuring that new token is requested every time a protected function is invoked else the call will fail until a new valid token is obtained. Asgardeo - Authorization Server Role: Asgardeo was instrumental, as it provided a ready-made OAuth2 implementation with all the enterprise features (token management, auditing, scope management). We effectively treated Asgardeo as the “identity provider for AI agents (authorization server).” Notably, Asgardeo by WSO2 has been evolving to tackle modern challenges, including what they call “agentic AI security.” In fact, WSO2 positions its IAM suite as AI-powered and ready for the AI Agent era, highlighting features like access control for AI agents in recent releases[4]. It’s great to see a traditionally user-focused IAM product adapting to secure non-human actors (AI/ML agents) as well – exactly what we needed. This zero-trust setup ensures our autonomous agents are auditable and accountable. We no longer have to worry, “What if the AI goes off and does X without permission?” Because in our design, the AI literally cannot do X without permission – the identity system won’t let it. This brings peace of mind for deploying such agents in sensitive workflows. To put it humorously, we taught our AI agents that “you don’t talk to strangers (or call strange APIs) without showing your ID.” Every time they want to do something important, our agents must essentially say, “Hi, I’m Agent 007, here’s my token, license to book hotels.” Only then do they get to execute their mission. This lighthearted analogy aside, the ability to enforce and verify identity at each step is a serious leap forward for making AI solutions enterprise-ready. Leveraging Azure OpenAI and Azure AI Foundry While the security and identity parts are crucial, let’s not forget the AI smarts that make the agents useful. Our project uses Azure OpenAI Service and the broader Azure AI Foundry to supply the large language model intelligence behind the AI agents. Azure OpenAI Service (GPT-4) as the Brain: Both the User Agent and Hotel Booking Agent are powered by GPT-4 models provided through Azure OpenAI. We chose Azure’s hosted version of OpenAI for a few reasons: Enterprise Compliance: Azure OpenAI offers the same capabilities as OpenAI’s GPT-4, but with added enterprise assurances (data encryption at rest/in transit, data not used for training the base model, regional isolation, etc.). For an enterprise scenario like booking with possibly sensitive user details, this is important. Performance and Reliability: Azure OpenAI comes with the scalability of Azure’s infrastructure. If our agents needed to scale to handle many requests, Azure can handle scaling out the inference endpoints. The combination of Asgardeo by WSO2 , AutoGen and Azure AI Foundry for AI capabilities shows that we can implement secure agents using open-source multi-agent framework and integrate with cloud Identity as well as AI services effectively. It’s a best-of-both-worlds scenario: use the tools you need from wherever, while adhering to open standards. And importantly, this means Solution Architects and Developers can replicate this in their own environments – if you have a different identity provider (say, Entra ID or Okta), you could use that instead of WSO2; if you prefer a different AI service, you could swap that in. The patterns remain the same. Alignment with Microsoft Entra Agent ID and Future Outlook The blog demonstrates a standards-based pattern - an AutoGen agent retrieves a JWT from Asgardeo - WSO2 Identity Server (OAuth2 2.0 client-credentials flow) and presents it to a SecureFunctionTool, which validates signature, issuer, audience, expiry and a single scope. This shows that an agent can be treated as a first-class identity and that Zero-Trust checks can be enforced at every function call. Microsoft Entra Agent ID, announced at Build 2025 [6], bakes the same pattern directly into Entra ID: every agent created with Copilot Studio or Azure AI Foundry is automatically issued its own identity object in the tenant directory, allowing Conditional Access, least-privilege role assignment and full audit/audit logging out-of-the-box. In our sample, we have not yet mint one identity per agent or wire up Conditional Access; nevertheless, because it relies solely on OIDC/JWT primitives, swapping the token issuer from WSO2 IS to Entra ID would only require new client credentials and issuer metadata. In short, the repository is conceptually aligned with Entra Agent ID while remaining implementation-agnostic. Key Takeaway of Alignment - standards today, seamless interoperability tomorrow The sample secures its AutoGen workflow with nothing beyond OAuth2 client-credentials and vanilla OIDC JWT checks. SecureFunctionTool ensures agent obtains a valid token that satisfy the required scopes for invoking business resources, while encapsulating sensitive information such as application credentials and tokens preventing unintentional exposure to LLM adhering to open standards. That choice lines up perfectly with other Agentic Identity Services, such as Microsoft’s Entra Agent ID, which is built on the same primitives. Because both Asgardeo and Entra Agent ID issue standards-compliant OAuth2 tokens, you can establish cross-trust in either direction - for example, add Entra Agent ID as an external identity provider in Asgardeo, or register Asgardeo in Entra. After you update the validator to recognise both issuers (and map scope/scp and roles claims), tokens minted by either platform will clear the same JWT checks, letting WSO2-managed AutoGen agents and Azure AI Foundry agents collaborate inside a single Zero-Trust fabric while each IdP still applies its own Conditional-Access policies. In short, adopting OAuth identity today is not a detour; it is the fast lane to a federated future where Asgardeo and Entra Agent ID jointly manage your “virtual employees” at enterprise scale. Conclusion The Zero-Trust Agents project illustrates a pivotal concept: as AI agents become more autonomous and capable, identity and security must be first-class citizens in their design. We can no longer afford to have free-roaming, opaque AI processes in our systems. By adding OAuth2-standard identity to each agent, we ensure that even in a fully automated multi-agent workflow, every action is accountable and authorized. Future Directions: Looking ahead, there are several avenues this work can evolve: Entra Agent ID Integration. Enabling multi-system integration by federating Asgardeo to Microsoft EntraID for invoking resources secured by Entra ID. Enhanced Security Monitoring. Strengthen security incident handling by using ML-based security anomaly detection via Azure Sentinel or Defender to monitor and flag unusual AI-Agent behaviour across systems. Open-Source Contribution. Open-source the SecureFunctionTool as an AutoGen plugin to promote reusable security best practices within the developer community. Final Thoughts: We’re entering a new era where AI agents might become as common as human coworkers in certain tasks. Ensuring zero-trust and auditable operations for these agents is not just an option, it’s a necessity. The collaboration with WSO2 on “Zero-Trust Agents” has shown one way to achieve this. By giving our AI agents an identity and enforcing “need-to-know” and “need-to-act” principles, we can unlock their potential safely. In a sense, we taught our autonomous agents some manners – they now politely ask for permission before acting, and they carry a token that says, “I’m allowed to do this.” With that in place, we can let them innovate and work for us, without losing oversight. This brings us closer to a future of AI that is empowered, yet secure and governable. Thank you for reading! Now, go forth and give your AI agents their ID cards – the era of Zero-Trust Agentic AI has arrived. 🚀 References [1] "Never Trust, Always Verify". Federal Migration to ZTA and Endpoint Security [2] Microsoft extends Zero Trust to secure the agentic workforce [3] Using Azure AI Agent Service with AutoGen / Semantic Kernel to build a ... [4] Modernize Your User Access, Future-Proof Your Business [5] Hotel Booking Sample [6] Announcing Microsoft Entra Agent ID: Secure and manage your AI agents
