From Idea to Production — Building Microsoft Security Store Advisor with an Agentic SDLC

Janaki_ramachandran

Microsoft

May 14, 2026

Microsoft Security Store Advisor is a new AI assistant inside Microsoft Security Store that helps customers find security solutions and agents in natural language. The team behind it builds it with an Agentic SDLC.

This is how that idea became a product — the engineering choices that gave AI an accountable role at every stage, and the lessons that shaped the system. We'll cover what an Agentic SDLC is, the five stages we ran Advisor through, what broke at each stage, and what we changed to ship.

The concept: what we mean by Agentic SDLC

Most engineering teams use AI in one of two modes today.

The first is scattered helpful moments — a code suggestion, a test draft, a PR summary — useful in isolation but never compounding across features.
The second is a fully autonomous agent expected to ship a feature on its own — fast, but faster than a security-sensitive product can responsibly review.

We took a third path: Agentic SDLC.

In an Agentic SDLC, work flows through a pipeline of focused agents — each owning one job, built on the GitHub Copilot CLI, coordinated by a stateful orchestrator, and gated by a human at every transition.

An orchestrator agent owns the run end-to-end, holds context between handoffs, enforces iteration caps, and pauses at every human gate.
A spec agent turns intent into a structured requirement.
An architecture agent proposes a design, and a separate design review agent on a different model setup independently critiques it.
An implementation agent writes the code, and a separate code review agent challenges it.
A test agent generates coverage.
A continuous improvement agent watches every feature run and proposes updates to the team’s instruction files, so the next feature run starts from a smarter system.
After the change ships, a bug hunter, an incident response agent, and a telemetry & feedback agent keep working on the team’s behalf — every output passes through a human triage gate before becoming action.

Two design choices keep this real instead of theatrical:

Generators and reviewers run on different model setups. The same model is rarely the best critic of its own output.
Instruction files travel with every agent. Plain-language, version-controlled descriptions of how this team builds software — coding patterns, security and privacy expectations, dependency rules, review standards — written once, applied everywhere.

Figure 1: The Agentic SDLC pipeline. Each phase produces a reviewable artifact; the orchestrator carries context forward; every critical decision stays with a person.

The product: Microsoft Security Store Advisor

Microsoft Security Store Advisor is an AI assistant inside Microsoft Security Store — the marketplace for security solutions and Copilot agents that integrate with Microsoft Security. Customers describe what they need in natural language; Advisor surfaces relevant solutions and Security Copilot agents from Microsoft and partners.

Every recommendation has to be defensible. Every input — including prompts that may try to inject through partner-supplied catalog content — has to be handled responsibly. Every release has to clear a high security bar. That made it the right product to build with an Agentic SDLC.

Idea to production, stage by stage

1. Spec — turning intent into scope. On the first runs, the spec agent treated every adjacent mention as a commitment. A request that said "and we may eventually want X" came back with X listed as a goal. Designs ballooned. Reviewers spent meetings saying "not now." We changed two things: the instruction file gained an explicit non-goals section per feature, and the spec agent was required to produce both goals and non-goals — and a list of open questions for the human — before requesting scope approval. After that, scope decisions happened up front in one or two turns instead of being relitigated through design and code review.

2. Architecture — making the reviewer actually review. Our first architecture pipeline ran the design agent and the design reviewer on the same model with near-identical prompts. The reviewer agreed with itself ~95% of the time. We were getting one opinion in two tabs. We moved the reviewer to a different model family, gave it a skeptical system prompt focused specifically on security, scale, privacy, and operability gaps, and added an explicit checklist in the instruction file. The reviewer started catching real issues — including misuse cases we would otherwise have shipped — before any human opened the design.

3. Implementation — stop rewriting the world. The implementation agent kept proposing whole-file rewrites — a 30-line bug fix would come back as a 600-line replacement — because whole files fit cleanly in its context window. Pull requests got huge, hard to review, and occasionally regressed unrelated code. Two disciplines fixed it. First, the orchestrator only loads files in the spec's blast radius (a dependency-graph cut from the changed symbols). Second, the implementation agent must produce diff-only edits with per-hunk justification — no rewrites without an explicit reason. PR sizes dropped ~70%, and human reviewers spent their time on judgment calls instead of diffing whole files.

Sidebar: How many iterations is enough? Three. Across both the architecture and implementation stages, the generator and reviewer agents iterate — propose, critique, revise — before any human is asked to approve. We tested how many rounds to allow. One round was too few; the reviewer kept catching fixable issues that the generator could have addressed cleanly. Five or more rarely converged further; quality plateaued while cost and latency kept climbing. The sweet spot was three. After three iterations, if the loop has not converged, we break and escalate to a human rather than retry indefinitely. That cap keeps cost, latency, and review fatigue bounded — and it surfaces the question “does this need a person?” quickly when the answer is yes.

4. Test — green tests are not enough. The test agent generated tests that hit the spec but missed the messy real-world inputs Advisor would actually see — malformed queries, partial authentication states, unusual throttling patterns, prompt-injection-style strings. We fed the test agent synthetic inputs shaped from sanitized production telemetry, plus an explicit security-cases checklist from the instruction file (authentication and authorization boundaries, injection-style inputs, rate-limit behavior). Defects per feature reaching pre-production fell by 38%.

5. Post-production — agents that help instead of paging. The first bug hunter cried wolf. Every anomaly became a candidate ticket; on-call started filtering the channel. An alerting agent that gets ignored is worse than no alerting agent — the noise eroded trust in the signal. Candidates now get a severity × novelty score; only above-threshold items page on-call. The rest land in a weekly triage review where patterns become backlog items. The incident response agent was scoped narrowly: assemble runbooks and recent context for the human on-call, never act on the system itself. The telemetry & feedback agent proposes backlog items and instruction-file updates from production patterns — but a person decides whether they land.

Figure 2: After release, three post-production agents — bug hunter, incident response, and telemetry & feedback — keep working on the team’s behalf and feed back into the development pipeline.

**Governance and Observability:** Three rails run the length of the pipeline. **Every stage requires human approval** before the next agent can proceed — no agent, no matter how confident, bypasses scope, design, or pull-request review. Every agent **logs its actions** to the team's work-tracking system (Azure DevOps, in our case): which agent ran, what artifacts it produced, which approvals were granted, and against which version of the instruction files. And every session's agent outputs, reviewer decisions, and linked work items are captured in a **standardized review template**, so any reviewer or auditor can reconstruct exactly how a change was produced months after it shipped. The audit trail is automatic, the gates are non-negotiable, and the same record that makes a release reviewable is what feeds the continuous-improvement agent.

**Closing the loop: the continuous improvement agent.** The most important agent in the system is also the quietest. After every feature run, a continuous improvement agent reads the full record — the generator/reviewer back-and-forth, what the human reviewers actually changed, the questions that took multiple turns to settle, the design notes that got rewritten, the production patterns the post-production agents surfaced — and proposes updates to the team’s instruction files. A review point that came up three times becomes a new line in the code-review checklist. A security concern that surfaced late becomes a non-goal in the spec template. A misread API contract becomes a coding-pattern note. A person reviews and approves the proposed updates before they land — but once they do, the next feature run starts from a smarter system. That is how lessons stop being one-off and start compounding across features. Mechanically, the agent diffs proposed instruction-file updates against the current set, attaches the evidence (which sessions/PRs/incidents prompted the change), and opens a PR against the instruction repo.

The pattern, every time: The first version was almost always a generator without a real reviewer. The fix, every time, was a different reviewer setup, a tighter instruction file, or a clearer human gate. Agents got more useful as the system around them got more disciplined.

Figure 3: Post-production loop

What it bought us

Across the four features shipped in March:

Cycle time from approved spec to production dropped ~60% vs. the team's prior 6-month baseline for comparable scope.
Pre-production defects per feature fell by 38% vs. the team's prior 6-month baseline for comparable scope.

The security wins are harder to express as a single number, but they show up everywhere in the pipeline: every spec, design, and pull request is independently challenged on a different model setup before any human reviews it; the team’s security expectations live in version-controlled instruction files that every agent applies on every task; and no agent — no matter how confident — can bypass scope, design, or pull request approval.

AI prepares the review. People own the decision.

What is still hard

A few things we haven't solved:

Evals for the continuous-improvement agent. Measuring whether an instruction-file change actually makes the next feature better is slow — we need more time and features under our belts, before signal stabilizes.
Cross-repo/ Cross-team orchestration. The orchestrator handles single-repo features cleanly; multi-repo/ multi-team conversations and changes still need a human to lead.