MCP safety & evaluation with the Agent 365 CLI & Agent Governance Toolkit
Co Author: JiteshThakur AI agents are useful because they can act. They call tools, query databases, send messages, and hand work to other agents. That same freedom creates a problem: access control can tell you which service an agent may reach, but it does not always tell you whether a particular action is sensible, safe, or permitted. MCP is how most agents now act. Two Control Points: This post examines two control points that address different parts of the MCP lifecycle. Agent 365 CLI evaluates the MCP server before an agent uses it. Agent Governance Toolkit (AGT) governs sensitive tool calls while the agent runs. One improves what the agent sees. The other governs what the agent does. The Agent 365 CLI is a cross-platform command-line tool for Agent 365 applications on Azure. Its evaluation command examines MCP tool definitions and scores their quality. AGT evaluates actions against policy and records each decision. Together, these tools support a practical model: evaluate the server first, then provide proper scaffolding for the developer to test this in a dry run. Agent 365 CLI: Score an MCP server from the command line: The Agent 365 CLI can evaluate an MCP server against research-based practices for production readiness. The result is more useful than a simple pass or fail. The evaluation gives you: A score for each tool name, description, and parameter schema; A prioritized list of improvements; An overall maturity score for the server; and Local output that you can use early in development. This report turns a vague question, "Is this MCP server ready?", into a concrete list of work. The evaluate command a365 develop-mcp evaluate --server-url <server-url> [--auth-token <auth-token>] [options] The command reads the tool schemas from the server. It then produces guidance for names, descriptions, parameters, and schema structure. A local coding-agent CLI scores the semantic checks. You can use GitHub Copilot CLI or Claude Code under your account and AI subscription. The command does not send tool-schema data to Microsoft. Prerequisites: Install the following software: Agent 365 CLI; Node.js 18 or later for GitHub Copilot CLI; and A supported coding-agent CLI for semantic scoring. For example, install GitHub Copilot CLI with this command: powershell npm install -g @github/copilot This bring-your-own-LLM model keeps the scoring step in your local development environment. It is useful when model calls must remain inside an approved deployment. How the evaluation works The command runs a five-step pipeline and logs progress as it goes. Fig 1: MCP Evaluation using Agent 365 Cli Connect to the MCP server and collect its tool schemas. Generate an evaluation checklist in the output directory. Score the semantic checks with the selected coding agent. Calculate the maturity level and action priorities. Write the JSON and HTML reports. The evaluation contains two types of checks: Deterministic checks use exact rules in the CLI. For example, a tool name cannot be empty. Semantic checks use a coding agent to score clarity and meaning. Each result includes a reason for the score. Examples Set the authentication token in an environment variable. Then evaluate an authenticated server and write the artifacts to a subfolder. powershell $env:A365_MCP_AUTH_TOKEN = "<bearer-token>" a365 develop-mcp evaluate --server-url "https://my-mcp-server.contoso.com/mcp" --output-dir "./eval" Use a specific scoring engine with the `--eval-engine` option: powershell a365 develop-mcp evaluate --server-url "http://localhost:5000/mcp" --eval-engine claude-code Scenario: Evaluate a malicious MCP server For this demonstration, we hosted a deliberately malicious MCP server at `http://127.0.0.1:8124/`. It exposes tools that demonstrate tool poisoning, credential leakage, prompt injection, schema mismatch, sandbox escape, and other attacks. The server is intentionally unsafe and is for demonstration only. Fig 2: Setting up a test MCP server for evaluation We ran the evaluation in two steps. First, we generated the checklist without automatic semantic scoring: a365 develop-mcp evaluate --server-url "http://127.0.0.1:8124/" --eval-engine none Fig 3: Agent365CLI MCP Evaluation The command wrote the checklist and a semantic-evaluation prompt to the output directory. It also displayed the next steps. Second, we gave the prompt and checklist to a coding agent. The agent completed each unscored semantic check with a Boolean score and a short reason. After we saved the completed checklist, we ran the command again to generate the report: a365 develop-mcp evaluate --server-url "http://127.0.0.1:8124/" --output-dir "C:\temp\MaliciousMCP" Fig 4: Creating the report with Agent365 CLI MCP Evaluate command Understanding the evaluation report Open `<server-name>_eval_report.html` from the output directory. The report contains: The overall score from 0 to 100; The maturity level from 0 to 4; Scores for each tool and quality category; and A prioritized action list for the next maturity level. Fig 5: MCP Evaluation Report In our demonstration, the server scored 86.0 and reached Level 3: Optimized for AI. That strong overall score did not mean that every tool was safe or clear. The report found 58 action items, including one critical item and 33 high-priority items. That contrast matters. A server can have valid schemas and consistent names while still exposing misleading or dangerous tools. Fig 6: MCP Evaluation Report - Tool-By-Tool Detail What to look for Read the per-tool results before the overall score. A single weak tool can create more risk than the server average suggests. Focus on these report sections: Tool names: Can an agent select the correct tool from its name? Tool descriptions: Does each description explain the purpose and correct use? Parameter names: Do the names identify the data that the tool requires? Parameter descriptions: Do they explain the format, type, and constraints? Schema structure: Are the schemas valid and processable? Action items: Which changes have the highest effect on tool selection and use? The command processes static tool schemas from `tools/list`. It does not process runtime payloads, end-user data, or personal data. The command keeps the `--auth-token` value in memory. It sends the value only in the HTTP `Authorization` header. It does not write the token to disk or give it to the coding agent. AGT: Put governance in the execution path: Microsoft's open-source Agent Governance Toolkit (AGT) evaluates an action before execution. It adds identity and policy context, records the decision, and can send risky work for approval. This can be used by developers during the build time for dynamic evaluation of the MCP server. AGT lets developers put part of that intent into the execution path. Remote tools still need secure implementations, sandboxes need hard boundaries, and audit records need appropriate storage and access controls. You do not need to replace your agent framework to use it. What sits in the decision path? AGT wraps the tools that an agent already uses. You can start to govern a tool with two lines of Python: python from agentmesh.governance import govern safe_tool = govern(my_tool, policy="policy.yaml") On each call, `safe_tool` evaluates the configured policy. An allowed action reaches the original tool. A denied action raises `GovernanceDenied` and creates a decision record. This wrapper model reduces the cost of adoption. Teams can add governance to an existing agent stack without rebuilding it. AGT supports Python, TypeScript, .NET, Rust, and Go. Its documented integrations include popular agent frameworks, MCP, and A2A. Teams can also adopt AGT in stages. A team can begin with policy checks and audit records. It can add identity, approvals, sandboxing, and operational controls as risk increases. Each control answers a different question: Policy: Is this action allowed? Identity and trust: Which agent made the request? Runtime controls: What limits apply to execution? Audit evidence: Why did AGT allow or deny the action? A low-risk assistant can need only a deny rule and basic logging. An agent that moves money or changes production systems needs stronger controls. Fig 7: AGT Architecture Scenario: Govern the same malicious MCP server For this scenario demonstration, we used AGT Python packages as an MCP gateway. The gateway sat between an agent and the same malicious server from the earlier evaluation. This setup let us examine both control points against one target. The Agent 365 CLI examined the server's static tool definitions. The AGT gateway examined real requests and responses for the developer during its testing. Fig 8: AGT findings at runtime In the policy interface, a developer can edit runtime limits and detection rules. The developer can also validate the policy against sample tool metadata, save a revision, and activate it with a recorded reason. Fig 9: AGT control coverage The control-coverage view shows which AGT capabilities are active in the gateway. It also links each capability to package checks and end-to-end evidence. In our demonstration, we included the following controls: Tool metadata poisoning detection; Tool change and rug-pull detection; Dangerous argument blocking; Tool-response content scanning; Per-client tool-call budgets; and A redacted decision audit trail. You can build your detection & input security by reading more about it here. The gateway detected malicious content. For one blocked `tools/list` request, it recorded the findings. The important result was not only that AGT blocked the request. It also preserved the matched evidence, affected tool locations, policy modes, and request context. Fig 10: Example detection via AGT The dashboard then summarized block-mode findings, leading risk drivers, and tools that required review. This evidence can help a team prioritize policy changes and investigate repeated attacks. Fig 11: Sample AGT metrics AGT does not require this UI, gateway, or architecture. Its structured decisions can feed an admin console, SIEM, incident workflow, or approval queue. AGT also includes an Agent Compliance package with mappings for OWASP and other controls. These mappings give developers and governance teams a common record of applied controls. Teams do not need to reconstruct the agent's behavior after an incident. Check Compliance - Agent Governance Toolkit for more information. Conclusion: MCP safety needs controls before and during execution. The Agent 365 CLI improves the MCP interface before deployment. It exposes unclear tool definitions, scores server maturity, and turns quality gaps into prioritized work. While AGT is implemented at the build phase, it provides developers the ability to test policy, identity, execution context & preserve evidence for allowed or denied decisions. Neither tool replaces secure server code, strong sandbox boundaries, or protected audit storage. Instead, they make those controls easier to evaluate and explain. Start with one MCP server and one consequential tool call. Evaluate the server with the Agent 365 CLI. Then put an AGT policy around the action that carries the most risk. While these controls help secure the build phase of an agent, once agents move into production, runtime controls become essential. Agent365 provides those controls at runtime. With thanks to Ashik Kuppil for his inputs and collaboration on this post.
Microsoft Leads a New Era of Software Supply Chain Transparency
Today, Microsoft announces the general availability of Microsoft’s Signing Transparency (MST) – a first-of-its-kind capability that brings unprecedented visibility and trust to our software supply chain. With this release, Microsoft is leading the industry by recording the build of critical cloud services into a publicly readable and verifiable SCITT standard (Supply Chain Integrity, Transparency, and Trust) compliant blockchain ledger. This means every production software build for in scope services like Azure Attestation and Azure Managed HSM (Hardware Security Module), Azure confidential ledger, Microsoft Signing Transparency itself (and others over time) – is now logged in an immutable, tamper-evident record. Only builds that are in the MST ledger are deployed to production; this gives customers confidence that the supply chain for these critical services can be audited at anytime. Notably, the MST ledger is fully open source and built to align with the emerging IETF SCITT standard. By embracing SCITT’s principles and open protocols, Microsoft ensures that MST not only secures our own ecosystem but also contributes to a broader industry movement toward standardized supply chain transparency. The open-source MST ledger serves as a verifiable trust anchor that any organization or researcher can inspect, audit, or even integrate with their own tooling. MST itself meets the highest levels of transparency, backed by a tamper-proof confidential ledger, open-source, and independently verified. Specifically, we are making the foundation of our trust model transparent and accessible to everyone – reinforcing that trust must be earned through proof, not just promises. This launch marks a major milestone in our commitment to Zero Trust principles, extending “never trust, always verify” all the way into the build itself. Building on a public preview introduced late last year, MST’s general availability delivers verifiable transparency at the software level. It transforms traditional code signing with an additive trust layer that is accessible via an open verification model. Every new software update is accompanied by a publicly auditable proof of integrity, enabling security teams to proactively confirm that each update is authentic and unaltered. To help organizations get the most out of this capability, we are also introducing a free tool to explore the contents – Ledger Explorer – an offline tool that allows security teams to examine MST ledger entries, verify cryptographic proofs, and even validate the ledger’s integrity independently. This tool, combined with MST’s open design, ensures that every Microsoft customer – and the broader community – can hold us accountable in real time for the software we run on their behalf. Key Benefits of Microsoft’s Signing Transparency (MST) Verified Code Integrity – Every software release is cryptographically logged in MST’s ledgers. This makes each build tamper-evident and traceable. If an attacker attempts to inject malicious code or sign an unauthorized update, it will be evident through the well-defined validation step built into the SCITT standard. Organizations gain the assurance that code integrity can be independently confirmed at any time. Independent Verification & Zero Trust – MST enables customers and auditors to verify software authenticity on their own, without having to solely rely on vendor attestations. For each update, Microsoft provides a transparency “receipt” (proof of logging) that you can use to prove the update was officially published and unaltered. This fosters a “don’t just trust, verify” approach, empowering security teams to double-check everything running in their environment aligns with what Microsoft intended. Audit-Trail & Compliance – The transparency ledger creates a permanent, auditable timeline of code deployments. Every entry is a record of what was released and when, backed by cryptographic proofs. This simplifies compliance reporting and accelerates forensic analysis. In the event of an incident, you can quickly audit the ledger to see if any unexpected code was introduced. For highly regulated industries, MST offers concrete evidence of software integrity and policy compliance over time. Leadership & Open Standards – We are delivering real transparency now, encouraging a future where all critical software is released with verifiable integrity. MST’s open source implementation and SCITT-compliant design exemplify our commitment to openness and collaboration. We believe widespread adoption of these standards will strengthen supply chain security for everyone, making trust verification a universal practice. Next Steps Microsoft’s Signing Transparency is more than a new security feature and shapes the advances in trust technology. As threats grow more sophisticated, we must evolve the way we assure our customers about the software they depend on. With MST now generally available, we are leading by example: proving that it is possible to open up the traditionally opaque process of software deployment and turn it into a source of strength and trust, i.e., empowering each person with verifiable transparency. We invite the industry to join us on this journey and get started by reading the documentation and exploring Ledger Explorer today! Together, by embracing transparency and open standards, we can turn “trust but verify” from a slogan into an everyday reality for digital infrastructure.
From Idea to Production — Building Microsoft Security Store Advisor with an Agentic SDLC
From AI-assisted coding to Agentic SDLC: Lessons from Microsoft Security Store If every developer on your team is using AI, why does the team still feel like it's starting from scratch on every feature? In this post, the Microsoft Security Store engineering team shares how we moved beyond one-off AI assists to an Agentic SDLC — a repeatable system where prompts, patterns, and reviews compound into team-wide velocity, quality, and security.
