Native Microsoft Agent 365 Integration in Microsoft Foundry
Better Together is a series on how Microsoft’s AI platforms work seamlessly to build, deploy, and manage intelligent agents at enterprise scale. As organizations embrace AI across every workflow, Microsoft Foundry, Microsoft 365, Microsoft Agent 365, and Microsoft Copilot Studio are coming together to deliver a unified approach—from development to deployment to day-to-day operations. This three-part series explores how these technologies connect to help enterprises build AI agents that are secure, governed, and deeply integrated with Microsoft’s product ecosystem. This blog focuses on Part 2: Microsoft Foundry + A365Microsoft Agent 365 native Integration, showing how organizations can build, deploy, and customize Microsoft Agent 365 agents directly from Foundry. What Is Microsoft Agent 365? Microsoft Agent 365 is the control plane for enterprise AI agents, allowing IT to register, secure, and scale agents across Microsoft 365 and third-party environments. AI agents act more like people than code—they bring skills, learn from context, and leverage enterprise data to complete tasks. Like with people in the enterprise, they need to be protected from digital threats, governed with the right IT controls, and managed following enterprise policies. Our philosophy is simple: treat agents like users. Extend your existing identity, security, compliance, and productivity infrastructure to agents using familiar tools adapted for their unique needs. Each agent receives its own identity, policies, and access controls, ensuring it operates effectively while staying compliant. With Agent 365, organizations can: Manage AI agents at scale with unified identity and lifecycle controls Enforce least-privilege access and compliance with Defender, Entra, and Purview Boost productivity through native integration with Microsoft 365 apps and Work IQ Monitor activity and apply policies from a single, secure registry Learn more about Microsoft Agent 365 Foundry: The Ideal Place for Developers to Build AI Agents Microsoft Foundry is the ideal platform for building, testing, and deploying Agent 365 agents. It provides a unified environment where developers can create enterprise-ready AI agents that are secure, governed, and fully integrated with Microsoft 365. At Ignite, Foundry introduces support for Agent 365 hosted (containerized) agents, giving developers a consistent, scalable runtime managed entirely within the Microsoft cloud. This initial release focuses on hosted agents to provide a fully managed and secure environment from development to deployment. With Foundry, developers can: Author agents quickly using low-code or pro-code workflows Test and iterate in a secure, hosted environment Integrate frontier AI models from Microsoft, OpenAI, Meta, DeepSeek, and xAI Package and deploy agents with Microsoft identity, security, and governance built in Through its native integration with Microsoft Agent 365, Foundry also provides: Foundry-hosted runtime for seamless agent execution Azure Bot Service and Microsoft 365 app integration (Teams, Outlook, M365 Copilot) MCP-connected tools from Microsoft Agent 365 Simplified preparation flow for publishing to M365 Copilot, Teams and BizChat Apps Together, Foundry and Microsoft Agent 365 let organizations build, host, and manage AI agents natively, making them enterprise-ready from day one. What Can Employees Do with Agent 365? With Agent 365, employees can: Automate email triage and meeting preparation Summarize and generate content Locate organizational knowledge instantly Orchestrate cross-system workflows and approvals Advanced teams can also: Integrate internal knowledge bases Create business-specific workflows Extend actions using Foundry APIs and connectors Why It Matters This integration makes Agent 365 agents enterprise-ready out of the box—combining the authoring power of Microsoft Foundry with the security and manageability of the Microsoft 365 ecosystem. IT retains control over policy, compliance, and lifecycle management, while business users gain intelligent agents that work across the tools they already use. Get Started Early access to Microsoft Agent 365 is available through the Frontier preview program, offering hands-on experience with Microsoft’s latest AI innovations. 🔗 [Quickstart — Publish an Agent to A365 GitHub Sample]Publishing Agents from Microsoft Foundry to Microsoft 365 Copilot & Teams
Better Together is a series on how Microsoft’s AI platforms work seamlessly to build, deploy, and manage intelligent agents at enterprise scale. As organizations embrace AI across every workflow, Microsoft Foundry, Microsoft 365, Agent 365, and Microsoft Copilot Studio are coming together to deliver a unified approach—from development to deployment to day-to-day operations. This three-part series explores how these technologies connect to help enterprises build AI agents that are secure, governed, and deeply integrated with Microsoft’s product ecosystem. Series Overview Part 1: Publishing from Foundry to Microsoft 365 Copilot and Microsoft Teams Part 2: Foundry + Agent 365 — Native Integration for Enterprise AI Part 3: Microsoft Copilot Studio Integration with Foundry Agents This blog focuses on Part 1: Publishing from Foundry to Microsoft 365 Copilot—how developers can now publish agents built in Foundry directly to Microsoft 365 Copilot and Teams in just a few clicks. Build once. Publish everywhere. Developers can now take an AI agent built in Microsoft Foundry and publish it directly to Microsoft 365 Copilot and Microsoft Teams in just a few clicks. The new streamlined publishing flow eliminates manual setup across Entra ID, Azure Bot Service, and manifest files, turning hours of configuration into a seamless, guided flow in the Foundry Playground. Simplifying Agent Publishing for Microsoft 365 Copilot & Microsoft Teams Previously, deploying a Foundry AI agent into Microsoft 365 Copilot and Microsoft Teams required multiple steps: app registration, bot provisioning, manifest editing, and admin approval. With the new Foundry → M365 integration, the process is straightforward and intuitive. Key capabilities No-code publishing — Prepare, package, and publish agents directly from Foundry Playground. Unified build — A single agent package powers multiple Microsoft 365 channels, including Teams Chat, Microsoft 365 Copilot Chat, and BizChat. Agent-type agnostic — Works seamlessly whether you have a prompt agent, hosted agent, or workflow agent. Built-in Governance — Every agent published to your organization is automatically routed through Microsoft 365 Admin Center (MAC) for review, approval, and monitoring. Downloadable package — Developers can download a .zip for local testing or submission to the Microsoft Marketplace. For pro-code developers, the experience is also simplified. A C# code-first sample in the Agent Toolkit for Visual Studio is searchable, featured, and ready to use. Why It Matters This integration isn’t just about convenience; it’s about scale, control, and trust. Faster time to value — Deliver intelligent agents where people already work, without infrastructure overhead. Enterprise control — Admins retain full oversight via Microsoft 365 Admin Center, with built-in approval, review and governance flows. Developer flexibility — Both low-code creators and pro-code developers benefit from the unified publishing experience. Better Together — This capability lays the groundwork for Agent 365 publishing and deeper M365 integrations. Real-world scenarios YoungWilliams built Priya, an AI agent that helps handle government service inquiries faster and more efficiently. Using the one-click publishing flow, Priya was quickly deployed to Microsoft Teams and M365 Copilot without manual setup. This allowed Young Williams’ customers to provide faster, more accurate responses while keeping governance and compliance intact. “Integrating Microsoft Foundry with Microsoft 365 Copilot fundamentally changed how we deliver AI solutions to our government partners,” said John Tidwell, CTO of YoungWilliams. “With Foundry’s one-click publishing to Teams and Copilot, we can take an idea from prototype to production in days instead of weeks—while maintaining the enterprise-grade security and governance our clients expect. It’s a game changer for how public services can adopt AI responsibly and at scale.” Availability Publishing from Foundry to M365 is in Public Preview within the Foundry Playground. Developers can explore the preview in Microsoft Foundry and test the Teams / M365 publishing flow today. SDK and CLI extensions for code-first publishing are generally available. What’s Next in the Better Together Series This blog is part of the broader Better Together series connecting Microsoft Foundry, Microsoft 365, Agent 365, and Microsoft Copilot Studio. Continue the journey: Foundry + Agent 365 — Native Integration for Enterprise AI (Link) Start building today [Quickstart — Publish an Agent to Microsoft 365 ] Try it now in the new Foundry PlaygroundImplementing MCP Remote Servers with Azure Function App and GitHub Copilot Integration
Introduction In the evolving landscape of AI-driven applications, the ability to seamlessly connect large language models (LLMs) with external tools and data sources is becoming a cornerstone of intelligent system design. Model Context Protocol (MCP) — a specification that enables AI agents to discover and invoke tools dynamically, based on context. While MCP is powerful, implementing it from scratch can be daunting !!! That’s where Azure Functions comes in handy. With its event-driven, serverless architecture, Azure Functions now supports a preview extension for MCP, allowing developers to build remote MCP servers that are scalable, secure, and cloud-native. Further, In VS Code, GitHub Copilot Chat in Agent Mode can connect to your deployed Azure Function App acting as an MCP server. This connection allows Copilot to leverage the tools and services exposed by your function app. Why Use Azure Functions for MCP? Serverless Simplicity: Deploy MCP endpoints without managing infrastructure. Secure by Design: Leverage HTTPS, system keys, and OAuth via EasyAuth or API Management. Language Flexibility: Build in .NET, Python, or Node.js using QuickStart templates. AI Integration: Enable GitHub Copilot, VS Code, or other AI agents to invoke your tools via SSE endpoints. Prerequisites Python version 3.11 or higher Azure Functions Core Tools >= 4.0.7030 Azure Developer CLI To use Visual Studio Code to run and debug locally: Visual Studio Code Azure Functions extension An storage emulator is needed when developing azure function app in VScode. you can deploy Azurite extension in VScode to meet this requirement. Press enter or click to view image in full size You can run the Azurite in VS Code as shown below. C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator> .\azurite.exe Press enter or click to view image in full size alternatively, you can also run Azurite in docker container as shown below. docker run -p 10000:10000 -p 10001:10001 -p 10002:10002 \ mcr.microsoft.com/azure-storage/azurite For more information about setting up Azurite, visit Use Azurite emulator for local Azure Storage development | Microsoft Learn Github Repositories Following Github repos are needed to setup this PoC. Repository for MCP server using Azure Function App https://github.com/mafzal786/mcp-azure-functions-python.git Repository for AI Foundry agent as MCP Client https://github.com/mafzal786/ai-foundry-agent-with-remote-mcp-using-azure-functionapp.git Clone the repository Run the following command to clone the repository to start building your MCP server using Azure function app. git clone https://github.com/mafzal786/mcp-azure-functions-python.git Run the MCP server in VS Code Once cloned. Open the folder in VS Code. Create a virtual environment in VS Code. Change directory to “src” in a new terminal window, install the python dependencies and start the function host locally as shown below. cd src pip install -r requirements.txt func start Note: by default this will use the webhooks route: /runtime/webhooks/mcp/sse. Later we will use this in Azure to set the key on client/host calls: /runtime/webhooks/mcp/sse?code=<system_key> Press enter or click to view image in full size MCP Inspector In a new terminal window, install and run MCP Inspector. npx @modelcontextprotocol/inspector Click to load the MCP inspector. Also provide the generated proxy session token. http://127.0.0.1:6274/#resources In the URL type and click “Connect”: http://localhost:7071/runtime/webhooks/mcp/sse Once connected, click List Tools under Tools and select “hello_mcp” tool and click “Run Tool” for testing as shown below. Press enter or click to view image in full size Select another tool such as get_stockprice and run it as shown below. Press enter or click to view image in full size Deploy Function App to Azure from VS Code For deploying function app to azure from vs code, make sure you have Azure Tools extension enabled in VS Code. To learn more about Azure Tools extension, visit the following Azure Extensions if your VS code environment is not setup for Azure development, follow Configure Visual Studio Code for Azure development with .NET — .NET | Microsoft Learn Once Azure Tools are setup, sign in to Azure account with Azure Tools Press enter or click to view image in full size Once Sign-in is completed, you should be able to see all of your existing resources in the Resources view. These resources can be managed directly in VS Code. Look for Function App in Resource, right click and click “Deploy to Function App”. Press enter or click to view image in full size If you already have it deployed, you will get the following pop-up. Click “Deploy” Press enter or click to view image in full size This will start deploying your function app to Azure. In VS Code, Azure tab will display the following. Press enter or click to view image in full size Once the deployment is completed, you can view the function app and all the tools in Azure portal under function app as shown below. Press enter or click to view image in full size Get the mcp_extension key from Functions → App Keys in Function App. Press enter or click to view image in full size This mcp_extension key would be needed in mcp.json file in VS code, if you would like to test the MCP server using Github Copilot in VS Code. Your entries in mcp.json file will look like as below for example. { "inputs": [ { "type": "promptString", "id": "functions-mcp-extension-system-key", "description": "Azure Functions MCP Extension System Key", "password": true }, { "type": "promptString", "id": "functionapp-name", "description": "Azure Functions App Name" } ], "servers": { "remote-mcp-function": { "type": "sse", "url": "https://${input:functionapp-name}.azurewebsites.net/runtime/webhooks/mcp/sse", "headers": { "x-functions-key": "${input:functions-mcp-extension-system-key}" } }, "local-mcp-function": { "type": "sse", "url": "http://0.0.0.0:7071/runtime/webhooks/mcp/sse" } } } Test Azure Function MCP Server in MCP Inspector Launch MCP Inspector and provide the Azure Function in MCP inspector URL. Provide authentication as shown below. Bearer token is mcp_extension key. Testing an MCP server with GitHub Copilot Testing an MCP server with GitHub Copilot involves configuring and utilizing the server within your development environment to provide enhanced context and capabilities to Copilot Chat. Steps to Test an MCP Server with GitHub Copilot: Ensure Agent Mode is Enabled: Open Copilot Chat in Visual Studio Code and select “Agent” mode. This mode allows Copilot to interact with external tools and services, including MCP servers. Add the MCP Server: Open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) and run the command MCP: Add Server. Press enter or click to view image in full size Follow the prompts to configure the server. You can choose to add it to your workspace settings (creating a .vscode/mcp.json file) . Select HTTP or Server-Sent events Press enter or click to view image in full size Specify the URL and click Enter Press enter or click to view image in full size Provide a name of your choice Press enter or click to view image in full size Select scope as Global or workspace. I selected Workspace Press enter or click to view image in full size This will generate mcp.json file in .vscode or create a new entry if mcp.json already exists as shown below. Click Start to “start” the server. Also make sure your Azure function app is locally running with func start command. Press enter or click to view image in full size Now Type the prompt as shown below. Press enter or click to view image in full size Try another tool as below. Press enter or click to view image in full size VS code terminal output for reference. Press enter or click to view image in full size Testing an MCP server with Claude Desktop Claude Desktop is a standalone AI application that allows users to interact with Claude AI models directly from their desktop, providing a seamless and efficient experience. you can download Claude desktop at Download Claude In this article, I have added another tool to utilize to test your MCP server running in Azure Function app. Modify claude_desktop_config.json with the following. you can find this file in window environment at C:\Users\<username>\AppData\Roaming\Claude { "mcpServers": { "my mcp": { "command": "npx", "args": [ "mcp-remote", "http://localhost:7071/runtime/webhooks/mcp/sse" ] } } } Note: If claude_desktop_config.json does not exists, click on setting in Claude desktop under user and visit developer tab. You will see you MCP server in Claude Desktop as shown below. Press enter or click to view image in full size Type the prompt such as “What is the stock price of Tesla” . After submitting, you will notice that it is invoking the tool “get_stockprice” from the MCP server running locally and configured in the .json earlier. Click Allow once or Allow always as shown below. Following output will be displayed. Press enter or click to view image in full size Now lets try weather related prompt. As you can see, it has invoked “get_weatheralerts” tool from MCP server. Press enter or click to view image in full size Azure AI Foundry agent as MCP Client Use the following Github repo to set up Azure AI Foundry agent as MCP client. git clone https://github.com/mafzal786/ai-foundry-agent-with-remote-mcp-using-azure-functionapp.git Open the code in VS code and follow the instructions mentioned in README.md file at Github repo. Once you execute the code, following output will show up in VS code. Press enter or click to view image in full size In this code, message is hard coded. Change the content to “what is weather advisory for Florida” and rerun the program. It will call get_weatheralerts tool and output will look like as below. Press enter or click to view image in full size Conclusion The integration of Model Context Protocol (MCP) with Azure Functions marks a pivotal step in democratizing AI agent development. By leveraging Azure’s serverless architecture, developers can now build remote MCP servers that scale automatically, integrate seamlessly with other Azure services, and expose modular tools to intelligent agents like GitHub Copilot. This setup not only simplifies the deployment and management of MCP servers but also enhances the developer experience — allowing tools to be invoked contextually by AI agents in environments like VS Code, GitHub Codespaces, or Copilot Studio[2]. Whether you’re building a tool to query logs, calculate metrics, or manage data, Azure Functions provides the flexibility, security, and scalability needed to bring your AI-powered workflows to life. As the MCP spec continues to evolve, and GitHub Copilot expands its agentic capabilities, this architecture positions you to stay ahead — offering a robust foundation for cloud-native AI tooling that’s both powerful and future-proof.
Interactive AI Avatars: Building Voice Agents with Azure Voice Live API
Azure Voice Live API recently reached General Availability, marking a significant milestone in conversational AI technology. This unified API surface doesn't just enable speech-to-speech capabilities for AI agents—it revolutionizes the entire experience by streaming interactions through lifelike avatars. Built on the powerful speech-to-speech capabilities of the GPT-4 Realtime model, Azure Voice Live API offers developers unprecedented flexibility: - Out-of-the-box or custom avatars from Azure AI Services - Wide range of neural voices, including Indic languages like the one featured in this demo - Single API interface that handles both audio processing and avatar streaming - Real-time responsiveness with sub-second latency In this post, I'll walk you through building a retail e-commerce voice agent that demonstrates this technology. While this implementation focuses on retail apparel, the architecture is entirely generic and can be adapted to any domain—healthcare, banking, education, or customer support—by simply changing the system prompt and implementing domain-specific tools integration. The Challenge: Navigating Uncharted Territory At the time of writing, documentation for implementing avatar features with Azure Voice Live API is minimal. The protocol-specific intricacies around avatar video streaming and the complex sequence of steps required to establish a live avatar connection were quite overwhelming. This is where Agent mode in GitHub Copilot in Visual Studio Code proved extremely useful. Through iterative conversations with the AI agent, I successfully discovered the approach to implement avatar streaming without getting lost in low-level protocol details. Here's how different AI models contributed to this solution: - Claude Sonnet 4.5: Rapidly architected the application structure, designing the hybrid WebSocket + WebRTC architecture with TypeScript/Vite frontend and FastAPI backend - GPT-5-Codex (Preview): Instrumental in implementing the complex avatar streaming components, handling WebRTC peer connections, and managing the bidirectional audio flow Architecture Overview: A Hybrid Approach The architecture comprises of these components 🐳 Container Application Architecture Vite Server: Node.js-based development server that serves the React application. In development, it provides hot module replacement and proxies API calls to `FastAPI`. In production, the React app is built into static files served by FastAPI. FastAPI with ASGI: Python web framework running on `uvicorn ASGI server`. ASGI (Asynchronous Server Gateway Interface) enables handling multiple concurrent connections efficiently, crucial for WebSocket connections and real-time audio processing. 🤖 AI & Voice Services Integration Azure Voice Live API: Primary service that manages the connection to GPT-4 Realtime Model, provides avatar video generation, neural text-to-speech, and WebSocket gateway functionality GPT-4 Realtime Model: Accessed through Azure Voice Live API for real-time audio processing, function calling, and intelligent conversation management 🔄 Communication Flows Audio Flow: Browser → WebSocket → FastAPI → WebSocket → Azure Voice Live API → GPT-4 Realtime Model Video Flow: Browser ↔ WebRTC Direct Connection ↔ Azure Voice Live API (bypasses backend for performance) Function Calls: GPT-4 Realtime (via Voice Live) → FastAPI Tools → Business APIs → Response → GPT-4 Realtime (via Voice Live) 🤖 Business process automation Workflows / RAG Shipment Logic App Agent: Analyzes orders, validates data, creates shipping labels, and updates tracking information Conversation Analysis Agent: Azure Logic App Reviews complete conversations, performs sentiment analysis, generates quality scores with justification, and stores insights for continuous improvement Knowledge Retrieval: Azure AI Search is used to reason over manuals and help respond to Customer queries on policies, products The solution implements a hybrid architecture that leverages both WebSocket proxying and direct WebRTC connections for optimal performance. This design ensures the conversational audio flow remains manageable and secure through the backend, while the bandwidth-intensive avatar video streams directly to the browser for optimal performance. The flow used in the Avatar communication: ``` Frontend FastAPI Backend Azure Voice Live API │ │ │ │ 1. Request Session │ │ │─────────────────────────►│ │ │ │ 2. Create Session │ │ │─────────────────────────►│ │ │ │ │ │ 3. Session Config │ │ │ (with avatar settings)│ │ │─────────────────────────►│ │ │ │ │ │ 4. session.updated │ │ │ (ICE servers) │ │ 5. ICE servers │◄─────────────────────────│ │◄─────────────────────────│ │ │ │ │ │ 6. Click "Start Avatar" │ │ │ │ │ │ 7. Create RTCPeerConn │ │ │ with ICE servers │ │ │ │ │ │ 8. Generate SDP Offer │ │ │ │ │ │ 9. POST /avatar-offer │ │ │─────────────────────────►│ │ │ │ 10. Encode & Send SDP │ │ │─────────────────────────►│ │ │ │ │ │ 11. session.avatar. │ │ │ connecting │ │ │ (SDP answer) │ │ 12. SDP Answer │◄─────────────────────────│ │◄─────────────────────────│ │ │ │ │ │ 13. setRemoteDescription │ │ │ │ │ │ 14. WebRTC Handshake │ │ │◄─────────────────────────┼─────────────────────────►│ │ (Direct Connection) │ │ │ │ │ │ 15. Video/Audio Stream │ │ │◄────────────────────────────────────────────────────│ │ (Bypasses Backend) │ │ ``` For more technical details, refer to the technical details behind the implementation, refer to the GitHub Repo shared in this post. Here is a video of the demo of the application in action.The Future of AI: The paradigm shifts in Generative AI Operations
Dive into the transformative world of Generative AI Operations (GenAIOps) with Microsoft Azure. Discover how businesses are overcoming the challenges of deploying and scaling generative AI applications. Learn about the innovative tools and services Azure AI offers, and how they empower developers to create high-quality, scalable AI solutions. Explore the paradigm shift from MLOps to GenAIOps and see how continuous improvement practices ensure your AI applications remain cutting-edge. Join us on this journey to harness the full potential of generative AI and drive operational excellence.The Future of AI: Reduce AI Provisioning Effort - Jumpstart your solutions with AI App Templates
In the previous post, we introduced Contoso Chat – an open-source RAG-based retail chat sample for Azure AI Foundry, that serves as both an AI App template (for builders) and the basis for a hands-on workshop (for learners). And we briefly talked about five stages in the developer workflow (provision, setup, ideate, evaluate, deploy) that take them from the initial prompt to a deployed product. But how can that sample help you build your app? The answer lies in developer tools and AI App templates that jumpstart productivity by giving you a fast start and a solid foundation to build on. In this post, we answer that question with a closer look at Azure AI App templates - what they are, and how we can jumpstart our productivity with a reuse-and-extend approach that builds on open-source samples for core application architectures.The Future of AI: Unleashing the Potential of AI Translation
The Co-op Translator automates the translation of markdown files and text within images using Azure AI Foundry. This open-source tool leverages advanced Large Language Model (LLM) technology through Azure OpenAI Services and Azure AI Vision to provide high-quality translations. Designed to break language barriers, the Co-op Translator features an easy-to-use command line interface and Python package, making technical content globally accessible with minimal manual effort.Transforming Customer Support with Azure OpenAI, Azure AI Services, and Voice AI Agents
Customer support today is under immense pressure to meet the rising expectations of speed, personalization, and always-on availability. Yet, businesses still struggle with 1. Long wait times and call center 2. queues 3. Disconnected support channels 4. Limited availability of agents outside business hours 5. Repetitive issues consuming valuable human time 6. Frustrated users due to lack of immediate and contextual answers These inefficiencies are costing businesses over $3.7 trillion annually in poor service delivery, while over 70% of agents (based on the research) spend excessive time searching for the right answers instead of resolving problems directly How Voice AI Agents Are Transforming the Support Experience Enter the era of voice-enabled AI agents—powered by Azure OpenAI, Azure AI Services, and ServiceNow—designed to completely transform the way customers engage with support systems. These agents can now: Handle complex user queries in natural language Access enterprise systems (like CRM, ITSM, HR) in real-time Automate repetitive tasks such as password resets, ticket status updates, or return tracking Escalate only when human assistance is truly needed Create connected, seamless, and intelligent support experiences across departments Let’s take a closer look at four architecture patterns that showcase how enterprises can deploy these agents effectively. 🔷 Architecture Pattern 1: Unified Voice Agent with Azure AI + ServiceNow + CRM Integration In this architecture, the customer support journey begins when a user initiates a voice-based conversation through a front-end interface such as a web application, mobile app, or smart device. The captured audio is streamed directly to Azure OpenAI GPT-4o's real-time API, which performs immediate speech-to-text transcription, interprets the intent behind the request, and prepares the initial system response—all in a single seamless stream. Once the user’s intent is understood (e.g., "create a ticket", "check incident status", or "list recent issues"), GPT-4o passes control to Semantic Kernel, which orchestrates the next steps through function calling. Semantic Kernel hosts pre-defined tools (functions) that map to ServiceNow API actions, such as createIncident, getIncidentStatus, listIncidents, or searchKnowledgeBase. These function calls are then securely routed to ServiceNow via REST APIs. ServiceNow executes the appropriate actions—whether it's creating a new support ticket, retrieving the status of an open incident, or searching its Knowledge Base. CRM data is also seamlessly accessed, if needed, to enrich responses with personalized context such as customer history or case metadata. The result from ServiceNow (e.g., an incident ID or KB article summary) is then sent back to Azure GPT-4o, which converts the structured data into a natural spoken response. This final audio output is delivered to the user in real time, completing the end-to-end conversational loop. Additionally, tools like Azure Monitor or Application Insights can be integrated to log telemetry, track usage trends, monitor latency, and analyze user satisfaction over time. This architecture enables organizations to streamline customer support operations, reduce wait times, and deliver natural, intelligent assistance across any channel—voice-first. 🔷 Architecture Pattern 2: Scalable Customer Support with Multi-Agent Voice Architecture This architecture introduces a modular and distributed agent-based design to deliver intelligent, scalable customer support through a voice interface. The process starts with the User Proxy Agent, which acts as the entry point for all user conversations. It captures voice input and forwards the request to the Master Agent, which serves as the brain of the architecture. The Master Agent, empowered with a large language model (LLM) and memory, interprets the intent behind the user's input and dynamically routes the request to the most appropriate domain-specific agent. These include specialized agents such as the Activation Agent, Root Agent, Sales Agent, or Technical Agent, each designed to handle specific workflows or business tasks. The Activation Agent connects to web services and handles provisioning or onboarding scenarios. The Root Agent taps into document search systems (like Azure Cognitive Search) to answer questions grounded in internal documentation. The Sales Agent is equipped with structured logic models (SLMs) and CRM access to retrieve sales-related data from backend databases. The Technical Agent is containerized via Docker and built to manage backend diagnostics, code-level issues, or infrastructure status—often connecting to systems like ServiceNow for real-time ITSM execution. Once the task is executed by the respective agent, results are passed back through the Master Agent and ultimately to the User Proxy Agent, which synthesizes the output into a voice response and delivers it to the user. The presence of shared memory between agents allows for maintaining context across multi-turn conversations, enabling complex, multi-step interactions (e.g., “Create a ticket, check the latest order status, and escalate it if unresolved.”) without breaking continuity. This architecture is ideal for enterprises looking to scale customer support horizontally, adding new agents without disrupting existing workflows. It enables parallelism, specialization, and real-time orchestration, providing faster resolutions while reducing the burden on human agents. Best suited for distributed support operations across IT, HR, sales, and field support—where task-specific intelligence and modular scale are critical. 🔷 Architecture Pattern 3: Customer Support Reinvented with Voice RAG + Azure AI + ServiceNow This architecture brings a cutting-edge twist to Retrieval-Augmented Generation (RAG) by enabling it through a Voice AI agent—creating a truly conversational experience grounded in enterprise knowledge. By combining Azure OpenAI models with the ServiceNow Knowledge Base, this pattern ensures accurate, voice-driven support for employees or customers in real time. The process begins when a user interacts with a voice-enabled interface—via phone, web, or embedded assistant. The Voice AI agent streams the audio to Azure OpenAI GPT-4o, which transcribes the voice input, understands the intent, and then triggers a RAG pipeline. Instead of relying solely on the model’s internal memory, the system performs a real-time query against the ServiceNow Product Knowledge Base, retrieving relevant knowledge articles, troubleshooting guides, or support workflows. These results are embedded directly into the prompt, creating an enriched context that is passed to the language model via Azure AI Foundry. The model then generates a natural, contextually accurate spoken response, which is converted back into audio and voiced to the user—creating a seamless end-to-end Voice RAG experience. This approach ensures that responses are not only conversational but also deeply grounded in trusted enterprise knowledge. Ideal for helpdesk automation, HR support, and IT troubleshooting—where users prefer speaking naturally and need verified, document-backed responses in real time. 🔷 Architecture Pattern 4: Conversational Customer Support with AI Avatars and Azure AI This architecture delivers rich, conversational experiences by integrating AI avatars, Azure AI, and ServiceNow to offer human-like, intelligent customer support across channels. It merges natural speech, facial expression, and enterprise data to create a highly engaging support assistant. The interaction begins when a user speaks with an AI avatar application, whether embedded in a web portal, mobile device, or kiosk. The voice is captured and processed through a speech-to-text pipeline, which feeds the Avatar Module and Live Discussions Engine to manage lip-sync, emotional tone, and turn-taking. Behind the scenes, the avatar is connected to Azure AI services, including Custom Neural Voice (CNV) and Azure OpenAI, which enable the avatar to understand intent and generate responses in natural, conversational language. Most critically, the system integrates directly with the ServiceNow platform. Through secure APIs, the avatar queries ServiceNow to: Retrieve case status updates Provide summaries of incident history Look up Knowledge Base articles Trigger incident creation if needed These ServiceNow results are then passed through the text-to-speech module, with support for multilingual voice synthesis, and rendered by the avatar using expressive animation. Responses are visually delivered as live or pre-rendered avatar videos, creating a truly interactive and personalized experience. This pattern not only answers basic questions but also surfaces dynamic enterprise data—turning the AI avatar into a frontline voice agent capable of real-time, connected support across IT, HR, or customer service domains. Best for branded digital experiences, frontline support stations, or HR/IT helpdesk automation where facial presence, empathy, and backend integration are essential. ✨ Closing Thoughts: The Future of Customer Support Is Here Customer expectations have evolved—and so must the way we deliver support. By combining the power of Azure OpenAI, Azure AI Services, and ServiceNow, we’re not just automating tasks—we’re reinventing how organizations connect with their users. Whether it's: A unified voice agent handling IT tickets and CRM queries, A multi-agent architecture scaling across departments, A voice-enabled RAG system delivering knowledge-grounded answers in real time, or A human-like AI avatar offering face-to-face support— These architectures are driving a new era of intelligent, conversational, and scalable customer service. 👉 Join us at the Microsoft Booth during ServiceNow Knowledge 2025 (starting May 6th) to experience these solutions live, explore the tech behind them, and imagine how they can transform your business. Let’s build the future of support—together.Project Maria: Bringing Speech and Avatars Together for Next-Generation Customer Experiences
In an age where digital transformation influences nearly every aspect of business, companies are actively seeking innovative ways to differentiate their customer interactions. Traditional text-based chatbots, while helpful, often leave users wanting a more natural, personalized, and efficient experience. Imagine hosting a virtual brand ambassador—a digital twin of yourself or your organization’s spokesperson—capable of answering customer queries in real time with a lifelike voice and expressive 2D or 3D face. This is where Project Maria comes in. Project Maria is an internal Microsoft initiative that integrates cutting-edge speech-to-text (STT), text-to-speech (TTS), large language model and avatar technologies. Using Azure AI speech and custom neural voice models, it seeks to create immersive, personalized interactions for customers—reducing friction, increasing brand loyalty, and opening new business opportunities in areas such as customer support, product briefings, digital twins, live marketing events, safety briefings, and beyond. In this blog post, we will dive into: The Problem and Rationale for evolving beyond basic text-based solutions. Speech-to-Text (STT), Text-to-Speech (TTS) Pipelines, Azure OpenAI GPT-4o Real-Time API that power natural conversations. Avatar Models in Azure, including off-the-shelf 2D avatars and fully customized custom avatar Neural Voice Model Creation, from data gathering to training and deployment on Azure. Security and Compliance considerations for handling sensitive voice assets and data. Use Cases from customer support to digital brand ambassadors and safety briefings. Real-World Debut of Project Maria, showcased at the AI Leaders’ Summit in Seattle. Future Outlook on how custom avatar will reshape business interactions, scale presence, and streamline time-consuming tasks. If you’re developing or considering a neural (custom) voice + avatar models for your product or enterprise, this post will guide you through both conceptual and technical details to help you get started—and highlight where the field is heading next. 1. The Problem: Limitations of Text-Based Chatbots 1.1 Boredom and Fatigue in Text Interactions Text-based chatbots have come a long way, especially with the advent of powerful Large Language Models (LLMs) and Small Large Models (SLMs). Despite these innovations, interactions can still become tedious—often requiring users to spend significant personal time crafting the right questions. Many of us have experienced chatbots that respond with excessively verbose or repetitive messages, leading to boredom or even frustration. In industries that demand immediacy—like healthcare, finance, or real-time consumer support—purely text-based exchanges can feel slow and cumbersome. Moreover, text chat requires a user’s full attention to read and type, whether in a busy contact center environment or an internal knowledge base where employees juggle multiple tasks. 1.2 Desire for More Engaging and Efficient Modalities Today’s users expect something closer to human conversation. Devices ranging from smartphones to smart speakers and in-car infotainment systems have normalized voice-based interfaces. Adding an avatar—whether a 2D or 3D representation—deepens engagement by combining speech with a friendly visual persona. This can elevate brand identity: an avatar that looks, talks, and gestures like your company’s brand ambassador or a well-known subject-matter expert. 1.3 The Need for Scalability In a busy customer support environment, human representatives simply can’t handle an infinite volume of conversations or offer 24/7 coverage across multiple channels. Automation is essential, yet providing high-quality automated interactions remains challenging. While a text-based chatbot might handle routine queries, a voice-based, avatar-enabled agent can manage more complex requests with greater dynamism and personality. By giving your digital support assistant both a “face” and a voice aligned with your brand, you can foster deeper emotional connections and provide a more genuine, empathetic experience. This blend of automation and personalization scales your support operations, ensuring higher customer satisfaction while freeing human agents to focus on critical or specialized tasks. 2. The Vision: Project Maria’s Approach Project Maria addresses these challenges by creating a unified pipeline that supports: Speech-to-Text (STT) for recognizing user queries quickly and accurately. Natural Language Understanding (NLU) layers (potentially leveraging Azure OpenAI or other large language models) for comprehensive query interpretation. Text-to-Speech (TTS) that returns highly natural-sounding responses, possibly in multiple languages, with customized prosody and style. Avatar Rendering, which can be a 2D animated avatar or a more advanced 3D digital twin, bringing personality and facial expressions to the conversation. By using Azure AI Services—particularly the Speech and Custom Neural Voice offerings—can deliver brand-specific voices. This ensures that each brand or individual user’s avatar can match (or approximate) a signature voice, turning a run-of-the-mill voice assistant into a truly personal digital replicas 3. Technical Foundations 3.1 Speech-to-Text (STT) At the heart of the system is Azure AI Services for Speech, which provides: Real-time transcription capabilities with a variety of languages and dialects. Noise suppression, ensuring robust performance in busy environments. Streaming APIs, critical for real-time or near-real-time interactions. When a user speaks, audio data is captured (for example, via a web microphone feed or a phone line) and streamed to the Azure service. The recognized text is returned in segments, which the NLU or conversation manager can interpret. 3.1.1 Audio Pipeline Capture: The user’s microphone audio is captured by a front-end (e.g., a web app, mobile app, or IoT device). Pre-processing: Noise reduction or volume normalization might be applied locally or in the cloud, ensuring consistent input. Azure STT Ingestion: Data is sent to the Speech service endpoint, authenticated via subscription keys or tokens (more on security later). Result Handling: The recognized text arrives in partial hypotheses (partial transcripts) and final recognized segments. Project Maria (Custom Avatar) processes these results to understand user intent 3.2 Text-to-Speech (TTS) Once an intent is identified and a response is formulated, the system needs to deliver speech output. Standard Neural Voices: Microsoft provides a wide range of prebuilt voices in multiple languages. Custom Neural Voice: For an even more personalized experience, you can train a voice model that matches a brand spokesperson or a distinct voice identity. This is done using your custom datasets, ensuring the final system speaks exactly like the recorded persona. 3.2.1 Voice Font Selection and Configuration In a typical architecture: The conversation manager (which could be an orchestrator or a custom microservice) provides the text output to the TTS service. The TTS service uses a configured voice font—like en-US-JennyNeural or a custom neural voice ID (like Maria Neural Voice) if you have a specialized voice model. The synthesized audio is returned as an audio stream (e.g., PCM or MP3). You can play this in a webpage directly or in a native app environment. Azure OpenAI GPT-4o Real-Time API integrates with Azure's Speech Services to enable seamless interactions. First, your speech is transcribed in near real time. GPT-4o then processes this text to generate context-aware responses, which are converted to natural-sounding audio via Azure TTS. This audio is synchronized with avatar models to create a lifelike, engaging interface 3.3 Real-Time Conversational Loop Maria is designed for real-time or text to speech conversations. The user’s speech is continuously streamed to Azure STT. The recognized text triggers a real-time inference step for the next best action or response. The response is generated by Azure OpenAI model (like GPT-4o) or other LLM/SLM The text is then synthesized to speech, which the user hears with minimal latency. 3.4 Avatars: 2D and Beyond 3.4.1 Prebuilt Azure 2D Avatars Azure AI Speech Services includes an Avatar capability that can be activated to display a talking head or a 2D animated character. Developers can: Choose from prebuilt characters or import basic custom animations. Synchronize lip movements to the TTS output. Overlay brand-specific backgrounds or adopt transparency for embedding in various UIs. 3.4.2 Fully Custom Avatars (Customer Support Agent Like Maria) For organizations wanting a customer support agent, subject-matter expert, or brand ambassador: Capture: Record high-fidelity audio and video of the person you want to replicate. The more data, the better the outcome (though privacy and licensing must be considered). Modeling: Use advanced 3D or specialized 2D animation software (or partner with Microsoft’s custom avatar creation solutions) to generate a rigged model that matches the real person’s facial geometry and expressions. Integration: Once the model is rigged, it can be integrated with the TTS engine. As text is converted to speech, the avatar automatically animates lip shapes and facial expressions in near real time. 3.5 Latency and Bandwidth Considerations When building an interactive system, keep an eye on: Network latency: Real-time STT and TTS require stable, fast connections. Compute resources: If hosting advanced ML or high concurrency, scaling containers (e.g., via Docker and Kubernetes) is critical. Avatars: Real-time animation might require sending frames or instructions to a client’s browser or device. 4. Building the Model: Neural Voice Model Creation 4.1 Data Gathering To train a custom neural voice, you typically need: High-quality audio clips: Ideally recorded in a professional studio to minimize background noise, with the same microphone setup throughout. Matching transcripts for each clip. Minimum data duration: Microsoft recommends a certain threshold (e.g., 300+ utterances, typically around 30 minutes to a few hours of recorded speech, depending on the complexity of the final voice needed). 4.2 Training Process Data Upload: Use the Azure Speech portal or APIs to upload your curated dataset. Model Training: Azure runs training jobs that often require a few hours (or more). This step includes: Acoustic feature extraction (spectrogram analysis). Language or phoneme modeling for the relevant language and accent. Prosody tuning, ensuring the voice can handle various styles (cheerful, empathetic, urgent, etc.). Quality Checks: After training, you receive an initial voice model. You can generate test phrases to assess clarity, intonation, and overall quality. Iteration: If the voice quality is not satisfactory, you gather more data or refine the existing data (removing noisy segments or inaccurate transcripts). 4.3 Deployment Once satisfied with the custom neural voice: Deploy the model to an Azure endpoint within your subscription. Configure your TTS engine to use the custom endpoint ID instead of a standard voice. 5. Securing Avatar and Voice Models Security is paramount when personal data, brand identity, or intellectual property is on the line. 5.1 API Keys and Endpoints Azure AI Services requires an API key or an OAuth token to access STT/TTS features. Store keys in Azure Key Vault or as secure environment variables. Avoid hard-coding them in the front-end or source control. 5.2 Access Control Role-Based Access Control (RBAC) at both Azure subscription level and container (e.g., Docker or Kubernetes) level ensures only authorized personnel can deploy or manage the containers running these services. Network Security: Use private endpoints if you want to limit exposure to the public internet. 5.3 Intellectual Property Concerns Avatar and Voice Imitation: A avatar model and custom neural voice that mimics a specific individual must be authorized by that individual. Azure has a verification process in place to ensure consent. Data Storage: The training audio data and transcripts must be securely stored, often with encryption at rest and in transit. 6. Use Cases: Bringing It All Together 6.1 Customer Support A digital avatar that greets users on a website or mobile app can handle first-level queries: “Where can I find my billing information?” “What is your return policy?” By speaking these answers aloud with a friendly face and voice, the experience is more memorable and can reduce queue times for human agents. If the question is too complex, the avatar can seamlessly hand off to a live agent. Meanwhile, transcripts of the entire conversation are stored (e.g., in Azure Cosmos DB), enabling data analytics and further improvements to the system. 6.2 Safety Briefings and Public Announcements Industries like manufacturing, aviation, or construction must repeatedly deliver consistent safety messages. A personal avatar can recite crucial safety protocols in multiple languages, ensuring nothing is lost in translation. Because the TTS voice is consistent, workers become accustomed to the avatar’s instructions. Over time, you could even create a brand or site-specific “Safety Officer” avatar that fosters familiarity. 6.3 Digital Twins at Live Events Suppose you want your company’s spokesperson to simultaneously appear at multiple events across the globe. With a digital twin: The spokesperson’s avatar and voice “present” in real time, responding to local audience questions. This can be done in multiple languages, bridging communication barriers instantaneously. Attendees get a sense of personal interaction, while the real spokesperson can focus on core tasks, or appear physically at another event entirely. 6.4 AI Training and Education In e-learning platforms, a digital tutor can guide students through lessons, answer questions in real time, and adapt the tone of voice based on the difficulty of the topic or the student’s performance. By offering a face and voice, the tutor becomes more engaging than a text-only system. 7. Debut: Maria at the AI Leaders Summit in Seattle Project Maria had its first major showcase at the AI Leaders Summit in Seattle last week. We set up a live demonstration: Live Conversations: Attendees approached a large screen that displayed Maria’s 2D avatar. On-the-Fly: Maria recognized queries with STT, generated text responses from an internal knowledge base (powered by GPT-4o or domain-specific models), then spoke them back with a custom Azure neural voice. Interactive: The avatar lip-synced to the output speech, included animated gestures for emphasis, and even displayed text-based subtitles for clarity. The response was overwhelmingly positive. Customers praised the fluid voice quality and the lifelike nature of Maria’s avatar. Many commented that they felt they were interacting with a real brand ambassador, especially because the chosen custom neural voice had just the right inflections and emotional range. 8. Technical Implementation Details Below is a high-level architecture of how Project Maria might be deployed using containers and Azure resources. Front-End Web App: Built with a modern JavaScript framework (React, Vue, Angular, etc.). Captures user audio through the browser’s WebRTC or MediaStream APIs. Connects via WebSockets or RESTful endpoints for STT requests. Renders the avatar in a <canvas> element or using a specialized avatar library. Backend: Containerized with Docker. Exposes endpoints for STT streaming (optionally passing data directly to Azure for transcription). Integrates with the TTS service, retrieving synthesized audio buffers. Returns the audio back to the front-end in a continuous stream for immediate playback. Avatar Integration: The back-end or a specialized service handles lip-sync generation (e.g., via phoneme mapping from the TTS output). The front-end renders the 2D or 3D avatar in sync with the audio playback. This can be done by streaming timing markers that indicate which phoneme is currently active. Data and Conversation Storage: Use an Azure Cosmos DB or a similar NoSQL solution to store transcripts, user IDs, timestamps, and optional metadata (e.g., conversation sentiment). This data can later be used to improve the conversation model, evaluate performance, or train advanced analytics solutions. Security: All sensitive environment variables (like Azure API keys) are loaded securely, either through Azure Key Vault or container orchestration secrets. The system enforces user authentication if needed. For instance, an internal HR system might restrict the avatar-based service to employees only. Scaling: Deploy containers in Azure Kubernetes Service (AKS), setting up auto-scaling to handle peak loads. Monitor CPU/memory usage, as well as TTS quota usage. For STT, ensure the service tier can handle simultaneous requests from multiple users. 9. Securing Avatar Models and Voice Data 9.1 Identity Management Each avatar or custom neural voice is tied to a specific subscription. Using Azure Active Directory (Azure AD), you can give fine-grained permissions so that only authorized DevOps or AI specialists can alter or redeploy the voice. 9.2 API Gateways and Firewalls For enterprise contexts, you might place an API Gateway in front of your containerized services. This central gateway can: Inspect requests for anomalies, Enforce rate-limits, Log traffic to meet compliance or auditing requirements. 9.3 Key Rotation and Secrets Management Frequently rotates keys to minimize the risk of compromised credentials. Tools like Azure Key Vault or GitHub’s secret storage features can automate the rotation process, ensuring minimal downtime. 10. The Path Forward: Scaling Custom Avatar 10.1 Extended Personalization While Project Maria currently focuses on voice and basic facial expressions, future expansions include: Emotion Synthesis: Beyond standard TTS expressions (friendly, sad, excited), we can integrate emotional AI to dynamically adjust the avatar’s tone based on user sentiment. Gesture Libraries: 2D or 3D avatars can incorporate hand gestures, posture changes, or background movements to mimic a real person in conversation. This reduces the “uncanny valley” effect. 10.2 Multilingual, Multimodal As businesses operate globally, multilingual interactions become paramount. We have seen many use cases to: Auto-detect language from a user’s speech and respond in kind. Offer real-time translation, bridging non-English speakers to brand content. 10.3 Agent Autonomy Systems like Maria won’t just respond to direct questions; they can initiate proactivity: Send voice-based notifications or warnings when critical events happen. Manage long-running tasks such as scheduling or triaging user requests, akin to an “executive assistant” for multiple users simultaneously. 10.4 Ethical and Social Considerations With near-perfect replicas of voices, there is a growing concern about identity theft, misinformation, and deepfakes. Companies implementing digital twins must: Secure explicit consent from individuals. Implement watermarking or authentication for voice data. Educate customers and employees on usage boundaries and disclaimers 11. Conclusion Project Maria represents a significant leap in how businesses and organizations can scale their presence, offering a humanized, voice-enabled digital experience. By merging speech-to-text, text-to-speech, and avatar technologies, you can: Boost Engagement: A friendly face and familiar voice can reduce user fatigue and build emotional resonance. Extend Brand Reach: Appear in many locations at once via digital twins, creating personalized interactions at scale. Streamline Operations: Automate repetitive queries while maintaining a human touch, freeing up valuable employee time. Ensure Security and Compliance: By using Azure’s robust ecosystem of services and best practices for voice data. As demonstrated at the AI Leaders Summit in Seattle, Maria is already reshaping how businesses think about communication. The synergy of avatars, neural voices, and secure, cloud-based AI is paving the way for the next frontier in customer interaction. Looking ahead, we anticipate that digital twins—like Maria—will become ubiquitous, automating not just chat responses but a wide range of tasks that once demanded human presence. From personalized marketing to advanced training scenarios, the possibilities are vast. In short, the fusion of STT, TTS, and avatar technologies is more than a novel gimmick; it is an evolution in human-computer interaction. By investing in robust pipelines, custom neural voice training, and carefully orchestrated containerized deployments, businesses can unlock extraordinary potential. Project Maria is our blueprint for how to do it right—secure, customizable, and scalable—helping organizations around the world transform user experiences in ways that are both convenient and captivating. If you’re looking to scale your brand, innovate in human-machine dialogues, or harness the power of digital twins, we encourage you to explore Azure AI Services’ STT, TTS, and Avatar solutions. Together, these advancements promise a future where your digital self (or brand persona) can meaningfully interact with users anytime, anywhere. Detailed Technical Implementation:- https://learn.microsoft.com/en-us/azure/ai-services/speech-service/text-to-speech-avatar/what-is-custom-text-to-speech-avatar Text to Speech with Multi-Agent Orchestration Framework:- https://github.com/ganachan/Project_Maria_Accelerator_tts Contoso_Maria_Greetings.mp4
