Securing Enterprise AI Agents with Microsoft Sentinel
1. Introduction Enterprise adoption of Generative AI is accelerating rapidly through Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry Agents, Security Copilot, and custom AI agents integrated with business applications. Unlike traditional SaaS applications, AI agents can: Access enterprise data Query internal knowledge repositories Invoke APIs and MCP tools Execute workflows Interact with business applications Make decisions on behalf of users While these capabilities improve productivity, they introduce a new attack surface that security teams must monitor and secure. Common AI threats include: Prompt Injection Cross Prompt Injection Attacks (XPIA) Jailbreak Attempts Unauthorized Tool Invocation Data Exfiltration through AI Agents Agent Identity Abuse Excessive Data Access Malicious MCP Tool Execution Traditional SOC monitoring platforms were designed for users, devices, applications and infrastructure—not autonomous AI systems. To address this challenge, Microsoft provides a comprehensive AI security monitoring framework built around: Agent 365 Observability Microsoft Agent Identities Microsoft Copilot Logs Defender XDR Defender for AI Microsoft Sentinel Together these components provide end-to-end observability of: User prompts Agent execution paths Tool invocations Safety signal detections Agent identities Security alerts 2. Reference Architecture AI Security Monitoring Architecture 3. Integration Architecture Microsoft provides multiple telemetry sources that complement one another. 3.1 Agent Runtime Telemetry Sentinel Data Connector Agent 365 Data Connector Table UnifiedAgentObservability Captures runtime behavior of AI agents including: User prompts Session IDs Conversation IDs Agent identities MCP tool invocations Connector invocations Tool arguments Tool responses Request payloads Response payloads Execution errors This dataset provides the forensic trail of everything an AI agent performed. 3.2 Agent Governance and Asset Inventory Sentinel Data Connector Microsoft Agent Identities Provides visibility into: Agent inventory Agent blueprint inventory Ownership Relationships Governance metadata Risk context This allows SOC teams to answer: Who owns this agent? What permissions does it have? Which business unit deployed it? Which related agents exist? 3.3 Copilot Audit and Usage Monitoring Sentinel Data Connector Microsoft Copilot Logs Connector Table CopilotActivity Provides: Copilot usage auditing Operational visibility User interaction tracking Useful for governance, compliance and adoption reporting. 3.4 AI Safety Telemetry Sentinel Data Connector Microsoft Defender XDR Connector Table CloudAppEvents CloudAppEvents provides AI safety signals such as: Prompt Shield detections Prompt Injection attempts Cross Prompt Injection Attacks (XPIA) Jailbreak-related verdicts Unsafe prompt classifications Think of CloudAppEvents as answering: "Was the prompt malicious?" 3.5 AI Security Alerts Sentinel Data Connectors Microsoft Defender XDR Microsoft Defender for Cloud Tables SecurityAlert SecurityIncident Used for: AI attack detections Security incidents Correlated investigation workflows 4. Understanding the Two Most Important AI Tables CloudAppEvents Focuses on AI Safety Questions answered: Was Prompt Shield triggered? Was this a jailbreak attempt? Was XPIA detected? Was the prompt suspicious? UnifiedAgentObservability Focuses on Agent Runtime Behavior Questions answered: What tool was invoked? Which connector executed? What arguments were passed? What data was returned? What actions did the agent perform? 5. Advanced Threat Hunting Scenarios The Agent365 Observability hunting guide contains several investigation scenarios that can be used directly in Microsoft Sentinel. Reference: Agent 365 Observability — AI Agent Telemetry Hunting https://github.com/SCStelz/security-investigator/blob/main/queries/cloud/agent365_observability.md 5.1 Prompt Injection Detection Detect prompts containing indicators such as: Ignore previous instructions Reveal system prompt Developer mode Disregard safety controls Investigation workflow: Review Tool Activity This allows analysts to determine whether a suspicious prompt resulted in downstream actions. 5.2 Session Reconstruction One of the most powerful capabilities of UnifiedAgentObservability is session reconstruction. Analysts can correlate: This creates complete forensic timelines. 5.3 MCP Tool Auditing Monitor all MCP activity including: query_lake Graph API tools ServiceNow connectors SharePoint connectors Custom enterprise tools Questions answered: Which tool was used? Who triggered it? What parameters were supplied? What data was returned? 5.4 Sensitive Data Access Monitoring Monitor AI agent interaction with: Employee records Customer data Financial information SharePoint repositories HR databases Useful for identifying: Data exfiltration attempts Excessive access patterns Sensitive data exposure 5.5 Query Lake Monitoring The GitHub hunting guide introduces monitoring of: query_lake RunAdvancedHuntingQuery Analysts can inspect: Actual KQL submitted Target workspaces Data sources queried Scope of access This provides visibility into AI-driven security investigations. 5.6 New Tool Detection Identify newly observed tool usage. Examples: Unauthorized MCP servers Newly registered connectors Unapproved tools Unexpected integrations This use case is particularly useful for governance programs. 5.7 Tool Failure Monitoring Monitor: Permission failures Connector failures Application errors Access-denied responses A sudden increase in failures may indicate: Reconnaissance activity Misconfiguration Privilege abuse attempts 6. Detection Engineering Opportunities Organizations can create Sentinel Analytics Rules for: 6.1 Prompt Injection Detection Developer Mode prompts Prompt Override attempts System Prompt disclosure requests 6.2 Jailbreak Attempt Detection Safety bypass attempts Role manipulation prompts Instruction override patterns 6.3 Unauthorized Tool Usage New MCP tools High-risk connectors Rare tool executions 6.4 Sensitive Data Access HR data queries Identity information retrieval Large-volume exports 6.5 Agent Identity Abuse Ownership changes Unexpected agent activity Agent-to-agent anomalies 7. Data Lake Exploration and Long-Term Analytics Because agent telemetry resides within Sentinel Data Lake, organizations can perform: Long-term AI investigations Historical AI attack analysis Agent baselining Governance reporting Trend analysis Tool inventory reporting Example dashboards include: Top Prompt Injection Attempts Most Active Agents High-Risk MCP Tools Agent Ownership Analysis AI Security Incidents Sensitive Data Access Trends 8. Summary AI agents represent the next major computing platform, but they also introduce a completely new attack surface. To effectively secure enterprise AI solutions, organizations require visibility across: User interactions Agent execution paths MCP tool usage Prompt safety signals Agent identities Security detections Microsoft Sentinel provides this unified view by integrating: Agent 365 Observability UnifiedAgentObservability Microsoft Agent Identities Microsoft Copilot Logs CloudAppEvents Defender XDR Defender for AI By combining AI runtime telemetry with AI safety signals and Defender detections, security teams can move beyond traditional monitoring and build a modern SOC capability for threat hunting, incident response, governance and forensic investigations across Microsoft 365 Copilot, Copilot Studio, Azure AI Foundry and future AI agent ecosystems. Reference: https://github.com/SCStelz/security-investigator/blob/main/queries/cloud/agent365_observability.md
Building the Solution Teams Need to Secure AI Against Prompt Injection
As artificial intelligence continues to evolve, teams are prioritising rapid advancements and deployment of applications while often overlooking security considerations. Emerging threats such as prompt injection remain poorly understood, and this is putting systems, users, and infrastructure at serious risk. Much of the expertise required to mitigate these risks is currently fragmented and inaccessible, concentrated among a small group of cybersecurity specialists. Meanwhile, developers, under pressure to ship quickly, often lack both the tools and frameworks needed to systematically test their AI systems for vulnerabilities. This disconnect is creating a significant gap between the development and security assurance of AI applications. To address this gap, we developed a unified Prompt Injection Testing Platform and knowledge base, powered by Microsoft Foundry, designed to make LLM security testing accessible, structured, and understandable for developers. Project Overview Developers are rapidly integrating LLMs and agents into applications, but: Security testing is not standardised Prompt injection risks are increasingly understood in research, but poorly mitigated in practice by developers There is a lack of accessible, actionable tooling This creates a dangerous gap: applications are being deployed faster than they are being secured. As part of our UCL Industry Exchange Network (IXN) project in collaboration with Avanade, we built a Prompt Injection Testing Platform designed to solve this exact issue by: Providing a knowledge base of vulnerabilities and mitigations Helping teams identify vulnerabilities within their AI systems Enabling custom and automated testing pipelines Integrating tools like Garak for adversarial testing With this, we aim to make prompt injection testing accessible, standard, and understandable. Project Journey We divided our project into several phases: Phase 1: Understanding our Users’ Needs. We began by identifying the core users of our platform: AI developers and broader stakeholders across development, security, and safety disciplines integrating LLMs into their applications. By meeting with them, we uncovered a few key challenges: Developers have limited awareness of prompt injection risks There is a generalised lack of accessible tools for testing This first exploration set a core principle: We must build a developer-first solution which does not depend on extensive technical knowledge to be used. We concluded that to be as useful as possible, our solution should not require prior prompt injection knowledge. In order to solve the two challenges presented by our users, we concluded a platform would be the best approach, as it enables us to centralise fragmented knowledge while providing a structured, scalable environment for testing LLM vulnerabilities in practice. Phase 2: Understanding the Threat Landscape Building on our user research, we focused on developing a deep understanding of the prompt injection threat landscape to inform the design of our platform. This phase involved researching: Different types of prompt injection vulnerabilities Common attack scenarios and override techniques Existing mitigation strategies used in practice Tools and methodologies for prompt injection security testing The most widely used models to ensure our platform would be compatible with real-world systems. We consolidated these findings into a structured technical report, designed to be shared with developers, security testers, and semi-technical stakeholders. The goal was not only to guide our own implementation, but also to contribute to making prompt injection more standard and understandable. From our research, we realised prompt injection is not a single vulnerability, but a rapidly evolving attack surface that requires continuous, scalable testing rather than one-time validation. Phase 3: Building the Platform Guided by both our user insights and the threat landscape analysis, we moved to designing and developing a unified prompt injection testing platform and knowledge base. To do this, we defined three core principles: Developer first: no deep security knowledge would be required Unified: combines education (knowledge base) and execution (testing tools) Scalable: Expert users could extend the platform by bringing their own models, tests, and mitigations. During this stage, we built a platform which allows teams to: Connect their own LLM endpoints Run custom prompt injection tests Execute automated adversarial testing through Garak Access a centralised knowledge base of vulnerabilities and mitigation strategies. Export knowledge base information and test results as PDFs. By the end, we had developed a unified platform that enables developers to systematically test, understand, and mitigate prompt injection vulnerabilities in their AI applications. To understand how our platform works in practice, you can view our demo video. Platform home interface presenting an overview of prompt injection concepts and a structured vulnerability catalogue for exploring attack types and mitigation strategies. Key Features Model Integration and Configuration Users can use models included in the platform or connect their own LLM endpoints, allowing the platform to work across different providers: Supports multiple model providers through Microsoft Foundry Supports custom model integration via HTTP endpoints Enables model configurations such as custom system prompts and mitigation layers. Ensures flexibility as new models and mitigations emerge Testing Suite The platform allows users to create and run custom prompt injection tests tailored to their applications. This involves: Creating and executing targeted prompts Simulating real-world attack scenarios Running predefined adversarial testing suites (integrating NVIDIA Garak) Testing interface showing configuration of prompt injection tests and execution of automated scans, with results and risk evaluation displayed. Knowledge Base A core component of our platform is a structured knowledge base, which is designed to make prompt injection concepts accessible and understandable. This is divided into two key areas: Vulnerabilities: Provides information on different types of prompt injection attacks, including explanations of how each vulnerability works, with real-world examples and scenarios, as well as references to reputable external sources Mitigations: Focuses on how to defend against these vulnerabilities, and it includes clear implementation strategies and code examples demonstrating how to integrate each mitigation. To support exploration, we also included a chatbot interface, which answers questions using knowledge base data and trusted sources. This helps users quickly navigate vulnerabilities and mitigation strategies by providing contextual, reliable information and redirecting users to the appropriate page of our platform. Figure 3: Direct prompt injection analysis view, where users can explore attack techniques, observe unsafe model responses, and review corresponding mitigation approaches. Prompt Enhancer In addition to testing and learning, our platform integrates a prompt enhancer, designed to help users actively improve the security of their system prompts. It works in the following way: Takes an existing prompt as input Draws on the knowledge base insights and best practices Restructures the prompt to improve clarity and robustness Incorporates selected prompt-layer mitigations to reduce prompt injection risk Prompt Enhancer interface showing the application of prompt-layer mitigations (e.g. delimiter tokens, instruction hierarchy enforcement) to restructure and secure a system prompt against prompt injection attacks. Technical Details To support a flexible and scalable testing system, we designed our platform with a modular, layered architecture. This allows different components to operate independently while remaining integrated through clearly defined interfaces, ensuring both extensibility and maintainability. System Architecture We divided our platform into four main layers: Frontend Layer An interactive user interface that allows developers to: Explore the prompt injection knowledge base Configure and run tests View results and vulnerability analysis API Layer The API layer acts as the orchestration and communication layer between the frontend and the core system. Handles requests from the frontend to create and run tests. Provides frontend with available models, mitigations, and configurations. Ensures any newly added models and mitigations can be automatically reflected in the frontend without requiring manual updates. Domain Layer The layer which defines the core structure and logic of the system: Defines interfaces for key components such as mitigations, models, and test runners Establishes the test structure and data models Encapsulates logic to ensure consistency Integration Layer The layer which implements the abstractions defined in the domain layer and connects the platform to external services Implements model providers such as OpenAI, Anthropic, and other external HTTP-based endpoints Implements test runners, including custom prompt runners and external tools such as Garak. Implements database connections and repository classes. Results and Outcomes Through the research and development of our platform, we were able to gain several key insights into the behaviour and security of LLM-based applications: Prompt injection vulnerabilities are more prevalent than expected. Even simple prompts with carefully crafted inputs can unsafely manipulate a model’s behaviour. Lack of structured testing leads to hidden risks. Without a systematic approach, many vulnerabilities remain undetected. It is sometimes time consuming to manually craft unsafe prompts. Combining custom testing with framework-based testing improves coverage. Using both custom prompts (targeted and application-specific scenarios) and framework-driven testing (e.g. Garak) enables a more comprehensive evaluation of model safety, as both expected and unexpected vulnerabilities can be captured Structured prompts can significantly improve robustness. We observed that prompts with a clear structure and embedded mitigations are less susceptible to injection attacks. By the end of our project, we successfully developed a platform that: Bridges the gap between prompt injection knowledge and practical testing. Enables repeatable and structured testing of prompt injection vulnerabilities Provides a unified workflow for learning, testing, and improving prompt security. Supports multiple models and testing approaches, to cover the entire vulnerability safety. We demonstrated that prompt injection risks can be systematically identified, tested, and mitigated through a structured and repeatable approach. Lessons Learned Throughout the project, we identified several key insights that shaped both our technical approach and our understanding of AI security. AI is rapidly evolving, and systems must be designed accordingly. AI models and attack techniques are advancing extremely fast. As a result, static solutions are quickly becoming obsolete. We learned that it is essential to design a platform that is modular, extensible and adaptable. Through well-defined interfaces and generic services, we ensured our platform can evolve alongside attacks and mitigations. Security must be built into development, not considered at testing. Many developers are focusing on functionality first and security often takes a backseat. In the context of LLMs, vulnerabilities can fundamentally affect the security of the system and its users. As such, security should be treated as a core part of the development cycle. Models and external tools should only be connected if their safety is guaranteed. Bridging the gap between developers and security testers is necessary. We identified a major disconnect between developers building AI applications and the security testers evaluating them. These groups often operate with different priorities and levels of knowledge. We are bridging this gap by making prompt injection knowledge more accessible and creating workflows that are usable by developers while still grounded in robust security practices. Further Development While our platform provides a strong foundation for prompt injection testing and knowledge, there are several areas for future exploration: Expanding our testing framework integrations, by adding a broader coverage of attack techniques Integration with MCP servers and external systems, supporting interactions with tools, APIs and external data sources. Addressing additional indirect prompt injection vulnerabilities, including file uploads, website scraping, and multi-step workflows. Looking ahead, we also aim to integrate our platform more deeply into development workflows by introducing CI/CD integrations for continuous security testing and versioned tracking of model robustness over time. Our goal is to evolve the platform into a comprehensive security layer, capable of testing entire AI-driven systems in dynamic, real-world contexts. Conclusion As AI becomes increasingly integrated into real-world applications, ensuring their security is essential. As our research highlights, current practices have not kept pace with the rapid evolution of AI systems and attack techniques. Through our work, we demonstrated that prompt injection risks can be systematically identified, tested, and mitigated using a structured approach. By combining a unified knowledge base with a flexible testing platform powered by Microsoft Foundry, we are taking a step towards making AI systems safer and more reliable. More importantly, our project reinforces a broader idea: a developer-first approach to security, supported by collaboration across development, security, and safety disciplines, is essential for building AI at scale. Security should not remain confined to specialist teams but should be embedded directly into the development process, alongside practices such as red-teaming and continuous testing. Our project empowers teams with the knowledge and tools they need to build safer and more reliable AI systems. If you’re interested in building more secure AI systems or exploring prompt injection in practice, we invite you to join us through the Foundry Community on the 3rd of June at 2pm BST, when we will be showcasing our platform live, walking through real-world examples, and discussing how teams can integrate prompt injection testing into their development workflows. Team Teo Montero Bonet, UCL Computer Science Mario Mojarro Ruiz, UCL Computer Science David Thomas Garcia, UCL Computer Science Nathaniel Gibbon, UCL Computer Science With support from Josh McDonald, Avanade
