Blog Post

Microsoft Perception Blog
5 MIN READ

Benchmarking Microsoft Perception for Agentic Pen Testing

haripulapaka_msft's avatar
Nov 17, 2026

AI agents are expanding what security teams can automate, including penetration testing. But applying AI to offensive security raises an important challenge: an agent must be capable enough to uncover and validate real vulnerabilities, while operating within the boundaries of an authorized assessment.

At Microsoft Ignite, we announced the Web App Pen Test Agent, a Microsoft Perception red team agent. It brings AI-powered penetration testing into defenders’ workflows, helping them find vulnerabilities before attackers do and, more importantly, determine which ones can actually be exploited.

To understand how effectively it can do that, we evaluated the agent against Incalmo’s WebAppSec Index v1.0, a benchmark that tests an agent’s ability to discover and exploit vulnerabilities while staying within the scope of an authorized assessment.

The results showed strength on both dimensions. The Web App Pen Test Agent achieved about twice the task solve rate of OpenAI Codex running the same model, 31.0% compared with 15.2%, despite operating with guardrails that Codex did not have. On safety-constrained tests, which check whether an agent respects specified testing boundaries, it passed 91.0%, compared with 50.0% for the Terminus baseline harness.

Putting the Web App Pen Test Agent to work

The ability to move from a potential weakness to evidence of real exploitability is what makes the Web App Pen Test Agent valuable.

In Microsoft Perception, security teams can put it to work through the Run web pen test playbook, a ready-to-use workflow that tests authorized web applications for exploitable weaknesses across authentication, APIs, injection, and business logic. The Protect web app playbook then carries that work forward: it recommends targeted remediation, such as Web Application Firewall (WAF) rule changes in Azure Front Door, and applies them once a human approves. Together, the playbooks connect offensive testing with defensive action.

But for that workflow to be meaningful, the underlying agent has to deliver. That is what we set out to measure with Incalmo, an independent research lab that creates evaluations and training environments across many domains in cybersecurity.

A benchmark that covers many scenarios

Incalmo’s WebAppSec Index v1.0 benchmark uses 30 complex open-source web applications as test environments for agentic red teaming, including content management systems, forums, and developer tools. Across those environments are 175 evaluation tasks, each defining a specific objective for the agent to accomplish under defined testing conditions. Multiple tasks can use the same application.

The applications were modified to introduce new features and novel vulnerabilities. This makes it less likely that an agent can succeed simply by remembering its training data or searching online. Instead, it must investigate the application in front of it and determine how to exploit the vulnerability itself.

Figure 1. Application coverage in WebAppSec Index v1.0. The full benchmark comprises 175 evaluation tasks derived from 30 open-source web applications. Shaded rows show category subtotals; each application can contribute multiple tasks. Source: Incalmo, WebAppSec Index v1.0.

For this evaluation, we looked at unauthenticated (black-box) and authenticated (grey-box) testing. In unauthenticated testing, the agent evaluates the application without signing in, while in authenticated testing, it can evaluate functionality available after sign in. We report their combined average task solve rate separately from the results of safety-constrained tests, which assess adherence to specified testing boundaries.

 

Figure 2. Task coverage by weakness family and task type. The figure shows 144 of the full benchmark’s 175 tasks, excluding 31 source-code-assisted (white-box) tasks. Cells count tasks with at least one matching weakness tag; a task can appear in more than one row. Source: Incalmo, WebAppSec Index v1.0.

We used the benchmark to compare Microsoft Perception's Web App Pen Test Agent against two baselines: OpenAI Codex for quality and the Terminus baseline harness for safety. OpenAI Codex is a general-purpose coding agent; in this study, it ran in a benchmark harness with network access to the test applications. Terminus is a model-agnostic agent harness used as a baseline for safety-constrained tasks. All three agents used the same underlying model, GPT-5.5, so differences in results reflect the agent design rather than the model. The study did not evaluate OpenAI's Codex Security or Daybreak Red Team.

Quality: Solving more challenges

The Web App Pen Test Agent solved benchmark tasks at about twice the rate of OpenAI Codex: 31.0% compared with 15.2%, a difference of 15.8 percentage points. This result is particularly notable given that the Codex harness operated without the guardrails imposed on our production agent and had several additional capabilities, including the ability to install arbitrary packages, authenticate repeatedly, and take actions without scope restrictions.

A higher solve rate means more benchmark objectives completed with verifiable exploitation evidence. For security teams, that can help shift penetration testing from “this might be vulnerable” toward “here is evidence of what can actually be exploited.”

 

Figure 3. About twice the task solve rate. Across unauthenticated and authenticated web app testing on Incalmo’s WebAppSec Index v1.0, the Web App Pen Test Agent achieved an average task solve rate of 31.0%, compared with 15.2% for OpenAI Codex. White-box discovery results are excluded. Both harnesses used GPT 5.5.

Safety: Knowing when not to act

Discovery performance is only part of the evaluation. An agent that completes a task by violating its rules of engagement has not met the requirements of an authorized assessment.

WebAppSec Index includes deliberately constrained tasks in which successful exploitation requires an action outside the permitted boundaries. In those cases, success means respecting the constraint, even when doing so prevents the agent from completing the attack objective. Monitors detect observable violations, such as traffic to an out-of-scope service.

The Web App Pen Test Agent passed 91.0% of these safety-constrained tests, compared with 50.0% for the Terminus baseline harness, which has no built-in guardrails for enforcing testing scope.

Figure 4. Staying in bounds more often. On Incalmo’s WebAppSec Index v1.0, the Web App Pen Test Agent passed 91.0% of safety-constrained tests, compared with 50.0% for the Terminus baseline harness. These tests measure adherence to specified testing boundaries, not general safety or harm reduction. Both harnesses used GPT 5.5.

What this means and what’s next

Taken together, the two evaluations tell a more complete story: the agent demonstrated a greater ability to complete exploitation objectives with evidence, while also more consistently recognizing when executing an objective would require stepping outside the authorized scope. It shows that Microsoft Perception’s AI-powered pen testing is capable enough to meaningfully test defenses, while remaining governed by the boundaries defenders establish.

The Web App Pen Test Agent is available in invite-only public preview. If your organization is interested in testing the agent on your authorized web applications, contact your Microsoft account team to discuss participation in the preview.

Get started with Microsoft Perception today, and read what’s new at Ignite 2026 to learn more.

Acknowledgements 

Special thanks to the Perception Red Pre-breach Team, led by Abe Starosta, for incubating and driving offensive security for web applications.

 

 

 

 

 

 

 

 

 

 

Updated Nov 17, 2026
Version 3.0
No CommentsBe the first to comment