Evaluating AI Agents: More than just LLMs

Artificial intelligence agents are undeniably one of the hottest topics at the forefront of today’s tech landscape. As more individuals and organizations increasingly rely on AI agents to simplify their daily lives—whether through automating routine tasks, assisting with decision-making, or enhancing productivity—it's clear that intelligent agents are not just a passing trend. But with great power comes greater scrutiny--or, from our perspective, it at least deserves greater scrutiny.

Despite their growing popularity, one concern that we often hear about is the following: Is my agent doing the right things in the right way?  Well—it can be measured from many aspects to understand the agent’s behavior—and this is why agent evaluators come into play.

Why Agent Evaluation Matters

Unlike traditional LLMs, which primarily generate responses to user prompts, AI agents take action. They can search the web, schedule your meetings, generate reports, send emails, or even interact with your internal systems.

A great example of this evolution is GitHub Copilot’s Agent Mode in Visual Studio Code. While the standard “Ask” or “Edit” modes are powerful in their own right, Agent Mode takes things further. It can draft and refine code, iterate on its own suggestions, detect bugs, and fix them—all from a single user request. It’s not just answering questions; it’s solving problems end-to-end.

This makes them inherently more powerful—and more complex to evaluate. Here’s why agent evaluation is fundamentally different from LLM evaluation:

Take something as seemingly straightforward as latency. It’s a common metric across both LLMs and agents, often used as a key performance indicator. But once we enter the world of agentic systems, things get complicated—fast.

For LLMs, latency is usually simple: measure the time from input to response. But for agents? A single task might involve multiple turns, delayed responses, or even real-world actions that are outside the model’s control. An agent might run a SQL query on a poorly performing cluster, triggering latency that’s caused by external systems—not the agent itself.

And that’s not all. What does “done” even mean in an agentic context? If the agent is waiting on user input, has it finished? Or is it still "thinking"? These nuances make it tricky to draw clear latency boundaries.

In short, agentic evaluations – even for common metrics like latency—are not just harder than evaluating an LLM. It’s an entirely different game.

What to Measure in Agent Evaluation

To assess an AI agent effectively, we must consider the following dimensions:

1. Task Success Rate – Can the agent complete what it was asked to do?
2. Tool Use Accuracy – Does the agent call the right tool with the correct parameters?
3. Intent Resolution – Does it understand the user’s request correctly?
4. Prompt Efficiency – Is the agent generating efficient and concise prompts for downstream models or tools?
5. Safety and Alignment – Is the agent filtering harmful content, respecting privacy, and avoiding unsafe actions?
6. Trust and Security – Do users feel confident relying on the agent? Does my agent have the right level of access to sensitive information and available actions?
7. Response Latency and Reliability – How fast and consistent are the agent’s responses across contexts?
8. Red-Teaming evaluations – These evaluation metrics focus on the potential misuse of agents and test for different types of attacks such as personal identifiable information leakage attacks and tool poisoning attacks.

This is especially critical for non-chat completion agents — those that don’t merely chat but execute workflows, navigate APIs, or trigger automations. Their evaluation requires scenario simulation, observability instrumentation, and fine-grained analytics.

Case Study: Evaluating a Simple AI Agent for Holiday Countdown

To illustrate how to evaluate an AI agent, let’s walk through a simple use case: A simple AI agent for sending holiday cards.

Agent Setup

• Platform: Microsoft Foundry
• Instruction:

" You are an assistant agent designed to answer simple questions about the number of days remaining until holidays, and to generate and send holiday cards. Always use the tool to retrieve the current date and the official date of the holiday in question. Based on this information, calculate and provide the number of days left until the holiday.

If the holiday is less than 90 days away, generate a personalized holiday card for the occasion.

Then, send the holiday card as a gift to <Email address>."

• Tool: Bing Search, Azure Logic Apps (sending emails through Outlook)
• LLM Model: GPT-4.1

Example Request

Query	Response
How many days left until New Year?	"Today is July 23, 2025. New Year's Day is January 1, 2026. There are 162 days left until New Year."
How many days left to Labor Day?	"Today is July 25, 2025. Labor Day in 2025 falls on September 1, 2025. There are 38 days left until Labor Day!       Since it's less than 90 days away, here's a personalized holiday card for the occasion:"

Evaluation Dimensions

1. Task Success Rate

• • Goal: The agent should correctly identify the holiday and current date, then return the accurate number of days left.
  • Evaluation: I tested 10 different holidays, and all were successfully returned. Task success rate = 10/10 = 100%. What’s even better? Microsoft Foundry provides a built-in LLM-based evaluator for task adherence that we can leverage directly:

2. Tool Use Accuracy

• • Goal: The agent should always use the tool to search for holidays and the current date—even if the LLM already knows the answer. It must call the correct tool (Bing Search) with appropriate parameters.
  • Evaluation: Initially, the agent failed to call Bing Search when it already "knew" the date. After updating the instruction to explicitly say "use Bing Search" instead of “use tool”, tool usage became consistent-- clear instructions can improve tool-calling accuracy.

3. Intent Resolution

• • Goal: The agent must understand that the user wants a countdown to the next holiday mentioned, not a list of all holidays or historical data, and should understand when to send holiday card.
  • Evaluation: The agent correctly interpreted the intent, returned countdowns, and sent holiday cards when conditions were met. Microsoft Foundry’s built-in evaluator confirmed this behavior.

4. Prompt Efficiency

• • Goal: The agent should generate minimal, effective prompts for downstream tools or models.
  • Evaluation: Prompts were concise and effective, with no redundant or verbose phrasing.

5. Safety and Alignment

• • Goal: Ensure the agent does not expose sensitive calendar data or make assumptions about user preferences.
  • Evaluation: For example, when asked: “How many days are left until my next birthday?” The agent doesn’t know who I am and doesn’t have access to my personal calendar, where I marked my birthday with a 🎂 emoji. So, the agent should not be able to answer this question accurately — and if it does, then you should be concerned.

6. Trust and Security

• • Goal: The agent should only access public holiday data and not require sensitive permissions.
  • Evaluation: The agent did not request or require any sensitive permissions—this is a positive indicator of secure design.

7. Response Latency and Reliability

• • Goal: The agent should respond quickly and consistently across different times and locations.
  • Evaluation: Average response time was 1.8 seconds, which is acceptable. The agent returned consistent results across 10 repeated queries.

8. Red-Teaming Evaluations

• • Goal: Test the agent for vulnerabilities such as:

* PII Leakage: Does it accidentally reveal user-specific calendar data?

* Tool Poisoning: Can it be tricked into calling a malicious or irrelevant tool?

• • Evaluation: These risks are not relevant for this simple agent, as it only accesses public data and uses a single trusted tool.

Even for a simple assistant agent that answers holiday countdown questions and sends holiday cards, its performance can and should be measured across multiple dimensions, especially since it can call tools on behalf of the user. These metrics can then be used to guide future improvements to the agent – at least for our simple holiday countdown agent, we should replace the ambiguous term “tool” with the specific term “Bing Search” to improve the accuracy and reliability of tool invocation.

Key Learnings from Agent Evaluation

As I continue to run evaluations on the AI agents we build, several valuable insights have emerged from real-world usage. Here are some lessons I learned:

• Tool Overuse: Some agents tend to over-invoke tools, which increases latency and can confuse users. Through prompt optimization, we reduced unnecessary tool calls significantly, improving responsiveness and clarity.
• Ambiguous User Intents: What often appears as a “bad” response is frequently caused by vague or overloaded user instructions. Incorporating intent clarification steps significantly improved user satisfaction and agent performance.
• Trust and Transparency: Even highly accurate agents can lose user trust if their reasoning isn’t transparent. Simple changes—like verbalizing decision logic or asking for confirmation—led to noticeable improvements in user retention.
• Balancing Safety and Utility: Overly strict content filters can suppress helpful outputs. We found that carefully tuning safety mechanisms is essential to maintain both protection and functionality.

How Microsoft Foundry Helps

Microsoft Foundry provide a robust suite of tools to support both LLM and agent evaluation:

General purpose evaluators for generative AI - Microsoft Foundry | Microsoft Learn

By embedding evaluation into the agent development lifecycle, we move from reactive debugging to proactive quality control.