microsoft perception
2 TopicsBenchmarking Microsoft Perception for Agentic Pen Testing
AI agents are expanding what security teams can automate, including penetration testing. But applying AI to offensive security raises an important challenge: an agent must be capable enough to uncover and validate real vulnerabilities, while operating within the boundaries of an authorized assessment. At Microsoft Ignite, we announced the Web App Pen Test Agent, a Microsoft Perception red team agent. It brings AI-powered penetration testing into defenders’ workflows, helping them find vulnerabilities before attackers do and, more importantly, determine which ones can actually be exploited. To understand how effectively it can do that, we evaluated the agent against Incalmo’s WebAppSec Index v1.0, a benchmark that tests an agent’s ability to discover and exploit vulnerabilities while staying within the scope of an authorized assessment. The results showed strength on both dimensions. The Web App Pen Test Agent achieved about twice the task solve rate of OpenAI Codex running the same model, 31.0% compared with 15.2%, despite operating with guardrails that Codex did not have. On safety-constrained tests, which check whether an agent respects specified testing boundaries, it passed 91.0%, compared with 50.0% for the Terminus baseline harness. Putting the Web App Pen Test Agent to work The ability to move from a potential weakness to evidence of real exploitability is what makes the Web App Pen Test Agent valuable. In Microsoft Perception, security teams can put it to work through the Run web pen test playbook, a ready-to-use workflow that tests authorized web applications for exploitable weaknesses across authentication, APIs, injection, and business logic. The Protect web app playbook then carries that work forward: it recommends targeted remediation, such as Web Application Firewall (WAF) rule changes in Azure Front Door, and applies them once a human approves. Together, the playbooks connect offensive testing with defensive action. But for that workflow to be meaningful, the underlying agent has to deliver. That is what we set out to measure with Incalmo, an independent research lab that creates evaluations and training environments across many domains in cybersecurity. A benchmark that covers many scenarios Incalmo’s WebAppSec Index v1.0 benchmark uses 30 complex open-source web applications as test environments for agentic red teaming, including content management systems, forums, and developer tools. Across those environments are 175 evaluation tasks, each defining a specific objective for the agent to accomplish under defined testing conditions. Multiple tasks can use the same application. The applications were modified to introduce new features and novel vulnerabilities. This makes it less likely that an agent can succeed simply by remembering its training data or searching online. Instead, it must investigate the application in front of it and determine how to exploit the vulnerability itself. For this evaluation, we looked at unauthenticated (black-box) and authenticated (grey-box) testing. In unauthenticated testing, the agent evaluates the application without signing in, while in authenticated testing, it can evaluate functionality available after sign in. We report their combined average task solve rate separately from the results of safety-constrained tests, which assess adherence to specified testing boundaries. weakness tag; a task can appear in more than one row. Source: Incalmo, WebAppSec Index v1.0. We used the benchmark to compare Microsoft Perception's Web App Pen Test Agent against two baselines: OpenAI Codex for quality and the Terminus baseline harness for safety. OpenAI Codex is a general-purpose coding agent; in this study, it ran in a benchmark harness with network access to the test applications. Terminus is a model-agnostic agent harness used as a baseline for safety-constrained tasks. All three agents used the same underlying model, GPT-5.5, so differences in results reflect the agent design rather than the model. The study did not evaluate OpenAI's Codex Security or Daybreak Red Team. Quality: Solving more challenges The Web App Pen Test Agent solved benchmark tasks at about twice the rate of OpenAI Codex: 31.0% compared with 15.2%, a difference of 15.8 percentage points. This result is particularly notable given that the Codex harness operated without the guardrails imposed on our production agent and had several additional capabilities, including the ability to install arbitrary packages, authenticate repeatedly, and take actions without scope restrictions. A higher solve rate means more benchmark objectives completed with verifiable exploitation evidence. For security teams, that can help shift penetration testing from “this might be vulnerable” toward “here is evidence of what can actually be exploited.” Safety: Knowing when not to act Discovery performance is only part of the evaluation. An agent that completes a task by violating its rules of engagement has not met the requirements of an authorized assessment. WebAppSec Index includes deliberately constrained tasks in which successful exploitation requires an action outside the permitted boundaries. In those cases, success means respecting the constraint, even when doing so prevents the agent from completing the attack objective. Monitors detect observable violations, such as traffic to an out-of-scope service. The Web App Pen Test Agent passed 91.0% of these safety-constrained tests, compared with 50.0% for the Terminus baseline harness, which has no built-in guardrails for enforcing testing scope. Figure 4. Staying in bounds more often. On Incalmo’s WebAppSec Index v1.0, the Web App Pen Test Agent passed 91.0% of safety-constrained tests, compared with 50.0% for the Terminus baseline harness. These tests measure adherence to specified testing boundaries, not general safety or harm reduction. Both harnesses used GPT 5.5. What this means and what’s next Taken together, the two evaluations tell a more complete story: the agent demonstrated a greater ability to complete exploitation objectives with evidence, while also more consistently recognizing when executing an objective would require stepping outside the authorized scope. It shows that Microsoft Perception’s AI-powered pen testing is capable enough to meaningfully test defenses, while remaining governed by the boundaries defenders establish. The Web App Pen Test Agent is available in invite-only public preview. If your organization is interested in testing the agent on your authorized web applications, contact your Microsoft account team to discuss participation in the preview. Get started with Microsoft Perception today, and read what’s new at Ignite 2026 to learn more. Acknowledgements Special thanks to the Perception Red Pre-breach Team, led by Abe Starosta, for incubating and driving offensive security for web applications.800Views2likes0CommentsMore signals, stronger defense: Microsoft Perception expands to third-party data
In this article Microsoft Perception expands to third-party data across AWS, Okta, and Salesforce data A real-world attack: From MFA fatigue to data theft How Perception agents respond Demo: The Triage Agent in action Beyond triage: Investigation and detection authoring across third-party data What's next: More signals, stronger defense Mitigation and protection guidance Learn more Most security teams don't defend a single-vendor environment. Employees might sign in through Okta, workloads may run in AWS, and customer data may live in Salesforce, often alongside Microsoft Entra ID, Microsoft 365 and Azure. Attackers know this. They exploit gaps across systems and move between them, while defenders are left piecing together signals across different consoles, logs, and response tools. Customers have consistently asked for security that works across their entire environment, beyond just Microsoft, connecting signals and helping them act on threats regardless of where they originate. This is the vision behind extending Microsoft Perception to third-party security data. Microsoft Perception is a multi-agent security system within Microsoft Defender that orchestrates specialized red, blue, and green team agents to find and validate security weaknesses, triage and investigate threats, and prioritize and remediate risks at machine speed, while keeping humans in control. Perception's blue team agents bring that intelligence into everyday security operations, helping security teams triage alerts and investigate threats with the depth and reasoning of their most experienced responders. Microsoft Perception expands to third-party data At Ignite, Perception expands its reach across third-party security data, starting with AWS, Okta and Salesforce. Through connectors in Microsoft Defender, Perception agents can reason over these signals to help teams detect, triage, and investigate threats across their environments. Triage Agent: Now generally available, it reasons over specific Okta and Salesforce alerts generated by Microsoft security products, delivering verdicts supporting reasoning so analysts can focus on the alerts that matter most. Investigation Agent: Reasons across Microsoft and supported third-party data to reconstruct the attack story, with tailored expertise for AWS and Okta. This helps defenders trace threats across identity providers into Microsoft 365 and cloud environments. Detection Authoring Agent: Helps security teams build custom detections using third-party data, including AWS, Okta and Salesforce, to extend detection coverage and keep pace with the apps the business uses. A real-world attack spanning identity and productivity systems To show what this means in practice, let's follow a real attack from start to finish, where the organization did not have Perception in place, and see how Perception agents' reasoning over Okta signals could have helped defenders detect and investigate the threat. The attacker didn't need a zero-day. They needed one employee to tap 'approve'. In a recent incident analyzed by Microsoft Security Research, an attacker targeted an Okta-federated organization with sign-in attempts from dozens of hosting, VPN and anonymizer IP addresses across more than 20 countries. The attacker also sent 157 push-verification prompts to multiple employees. When a prompt was finally approved, the attacker went a step further. Within the first hour, they enrolled a phone factor of their own on the victim's Okta account: a second key to the front door. We know Defender customers use a mix of identity providers, including Microsoft Entra and third-party solutions like Okta. That's why we're extending Perception's capabilities to work across these environments. When an attack spans multiple identity systems, analysts are often left piecing together different signals, logs, and response actions. Perception agents help connect those signals and accelerate triage and investigation, including across Okta alerts. Below, we walk through a real incident step by step, show how Perception agents can help, and see the Triage Agent in action. The attack: a factor of their own We reconstructed the incident below from Okta System Log and Microsoft Defender telemetry. Step 1: The push storm The attacker hit the Okta tenant with thousands of sign-in events and sent 157 push prompts from about two dozen IP addresses. This was not a blind password spray. Successes outnumbered failures by roughly eight to one, which tells us the passwords were already known (most likely from an earlier leak). The attacker only needed someone to approve a push, and someone did. Step 2: Enrolling their own factor Within the first hour, the attacker authenticated as the first victim and enrolled a new Okta MFA factor from an anonymizer IP address that they went on to use successfully multiple times, establishing another way to maintain access to the compromised account. Step 3: Data theft Next, the attacker set out to turn access into data theft. Because Okta was federated with Microsoft 365 through single sign-on (SSO), the compromised identity also opened a path into Microsoft 365, including the first victim's mailbox and SharePoint sites. The newly enrolled MFA factor enabled continued authentication, allowing the attacker to sustain access beyond the initial compromise. How Perception agents respond Each Perception agent plays a distinct role in helping defenders understand and respond to threats. The Triage Agent assesses whether an alert is a True Positive or False Positive, the Investigation Agent reconstructs what happened and how far it spread, and the Detection Authoring Agent helps teams build custom detections to identify similar threats. The Triage Agent reasons over Okta and Salesforce alerts The Triage Agent now extends its capabilities to Okta and Salesforce alerts generated by Microsoft security products, evaluating each alert and delivering a verdict with supporting reasoning. This helps analysts quickly understand which alerts need attention and focus on the threats that matter. Demo: The Triage Agent in action To see the Triage Agent end to end, we simulated a multi-stage attack against a user in our research tenant, including a password spray, repeated MFA failures, an MFA reset, a new session from a new IP address, and finally enumeration of SSO applications. The resulting alert, Okta SSO app enumeration from risky sign-in, was triaged by the Triage Agent in the Microsoft Defender portal. As the agent runs, the activity map shows each step of its analysis, including status and duration, giving analysts visibility into how it reaches its conclusion. The agent also produces a detailed report with its findings, verdict, and supporting reasoning for analyst review. The agent first investigates the Okta user, with all its subtasks returning a malicious verdict. It then connects the activity into a clear sequence: a password spray, successful compromise, repeated MFA failures and denials, an MFA reset, eventual MFA success, and a high-risk session used for SSO app enumeration. It also looked for benign explanations, such as a user recovering their own account, and found none. The final step brings the evidence together into a verdict. The agent classified the alert as a true positive and explained why: the sequence of compromise signals, the use of the compromised session for SSO enumeration as a common post-compromise behavior, and the absence of any benign recovery context. Beyond triage: investigation and detection authoring The Investigation Agent: Investigate across sources, with specialized capabilities for AWS and Okta The Investigation Agent correlates security signals across Microsoft and third-party data to reconstruct attacks, assess their scope and impact, and recommend prioritized response actions. Rather than following a predefined workflow, it adapts its investigation to the incident, reasoning across data available in Microsoft Defender through connected sources. With specialized capabilities for AWS and Okta, the agent knows where to look for evidence, how to interpret relevant signals, and what to investigate next, including tracing an attack from Okta into Microsoft 365 or AWS. Figure 5 shows part of the agent's report from a simulated incident in our research environment, where an attacker signed in through Okta, used single sign-on to reach an AWS application and assumed an AWS role through SAML. The report opens with a clear verdict and confidence level, followed by a plain-language summary of what happened and why. The agent correctly identified the activity as authorized security testing rather than a real compromise, and confirmed what did not happen: no persistence, no destructive actions, and no data exfiltration. An attack graph traces the full path from the source IP addresses through Okta into AWS, so an analyst can see the cross-cloud story at a glance and download the report to share with the team. Beyond what Figure 5 shows, the full report includes a comprehensive attack timeline, prioritized response actions to mitigate the attack, and the tactics, techniques and procedures (TTPs), indicators of compromise (IOCs) and compromised entities identified across available data sources. It also captures response actions already taken, so analysts can see whether those actions stopped the attack. An investigation typically completes in 20 to 40 minutes, giving analysts a verdict, a full report and a clear picture of what happened and what to do next. Detection Authoring Agent: Extend detection coverage across AWS, Okta and Salesforce Detection authoring starts when a security team launches a playbook in Microsoft Defender and selects a supported threat intelligence report The Threat Intelligence Agent extracts relevant findings, which the Detection Authoring Agent uses to explore available telemetry, identify detection opportunities and develop a coverage plan. It then authors the detections, measures and tunes their noise, and validates the result before an analyst reviews the proposed detections. Like the Investigation Agent, it is source-agnostic: it helps build custom detections over Microsoft and third-party data from any vendor that flows into Microsoft Defender. For AWS, Okta and Salesforce, it is specially tailored, with built-in knowledge for each data source and the attack patterns that matter there, so detection coverage keeps pace with the apps the business actually uses. Figure 6 shows a completed Agent session in the Microsoft Defender portal, started from an open-source intelligence (OSINT) profile of a campaign targeting Salesforce environments. The Threat Intelligence Agent summarized the campaign's tradecraft, including vishing, Okta-themed credential harvesting, malicious connected-app consent, and high-volume data access. The Detection Authoring Agent then delivered four validated detections covering suspicious OAuth posture, bursts of SSO- or SaaS-themed phishing URL clicks, Salesforce API bursts from uncommon ISPs, and high-volume Salesforce exports or report collection. Together, they cover five of the seven MITRE ATT&CK techniques the Threat Intelligence Agent identified. Two detections were ready to deploy and two were flagged for tuning, and all four are marked for analyst review before they go live. Back to the real incident How does this map to the rogue MFA incident? When we replayed our Okta MFA fatigue detection logic against the incident's telemetry, it surfaced the primary victim. This is one of the alert types the Triage Agent can evaluate, helping defenders quickly recognize the signs of a successful MFA fatigue attack. From there, the Investigation Agent, with its Okta expertise, is designed to expand the story: from the push storm to the factor enrollment, to the SSO hop into Microsoft 365 and the data within the attacker's reach. And once this technique is captured in threat intelligence, the Detection Authoring Agent can turn it into a custom detection for the rogue factor enrollment itself, so defenders can catch the attack at one of its earliest steps. What's next: Extending Perception across more security data Okta, AWS, and Salesforce are the first step, not the finish line. Because the Investigation Agent and the Detection Authoring Agent already work with data from any vendor that flows into Microsoft Defender, the next step is deeper, tailored expertise for more identity providers, SaaS applications and clouds. We also plan to bring the Triage Agent to more sources and alerts over time. Our goal is simple: wherever your users sign in and wherever your data lives, Perception agents should be able to follow the attacker there. Attackers don't stop at product boundaries, and neither should defenders. By extending Perception across third-party security data, we're helping security teams connect the dots, investigate threats, and strengthen defenses across their environments, bringing us closer to a future where specialized agents work together to defend at the scale and speed of modern threats. Learn more To explore Microsoft Perception's capabilities and see agentic defense in action, visit the Microsoft Perception product page Project Perception | Microsoft Security.306Views1like0Comments