Building an agent has become surprisingly easy. Building an agentic application that is reliable, secure, scalable, and doesn't cost a fortune is still hard.
Over the last couple of years, agents have moved very quickly from prototypes to real applications. At the same time, the definition of an agent has expanded considerably. We now have agents that use tools, skills, maintain memory, communicate with other agents, execute code, interact with enterprise systems, and take actions on behalf of users.
All these capabilities are useful, but they also make it easier to over-engineer an application.
This is the first post in a series about building agentic applications with Microsoft Foundry. Foundry Agent Service brings together the tools to build agents, run them in production, and deliver them to users through channels such as Microsoft Teams and Microsoft 365 Copilot. We will cover topics such as evaluation and optimization, security, context management, and scale in much more detail in future posts. For this first post, however, we step back and look at some of the basic engineering principles that we believe are important regardless of the use case.
Some of them sound obvious. In practice, they are also very easy to get wrong.
Start with the business problem
Agentic AI is very popular at the moment, but an agent is only as useful as the challenge it is solving. Before creating an agent, define the problem you are trying to solve.
Agentic AI allows applications to deal with situations where the steps required to accomplish a task cannot necessarily be defined in advance. An agent can reason about the information available to it, decide which tools it needs, use their results to determine what to do next, and adapt its behavior along the way.
However, flexibility comes at a cost.
Every time you introduce reasoning where deterministic application logic would have been enough, you are adding latency, cost, and nondeterminism. Sometimes that is exactly the tradeoff you want to make. Sometimes it isn't.
Math is a simple example. LLMs have become much better at math, but if your application already has all the inputs necessary to calculate a value, there is very little reason to spend tokens asking a model to calculate it. Code is faster, cheaper, and deterministic.
The same is true for many validation rules, transformations, authorization checks, and predefined business logic.
This doesn't mean those operations can't be part of an agentic workflow. They absolutely can. It means the agent can orchestrate them instead of replacing them.
A principle we like to use is: if no reasoning is required, an agent is probably not required either.
Microsoft Foundry gives us a lot of flexibility in how to build agents. We can define prompt agents through instructions and tools, use the Microsoft Copilot harness (currently in preview) for more complex, multi-step work, or bring our own code and orchestration with hosted agents, while leveraging any agent framework (LangGraph, Microsoft Agent Framework, and more). If you need to bring voice interactions to your application, voice agents for prompt and hosted agents make it simple. Toolbox lets us share tools across agents, and the choice of models means we can pick one that fits the task rather than design everything around a single model. This flexibility is valuable precisely because not every workload should have the same architecture.
The objective shouldn't be to build the most agentic solution possible. It should be to build the simplest solution that solves the problem well.
Define success early
This is another step that is surprisingly easy to skip, especially during a proof of concept.
Before spending too much time optimizing prompts, tools, models, or architecture, define what a successful agent means for your use case.
For a customer support scenario, success could mean reducing the average time it takes to resolve a case. It could mean increasing case deflection without negatively affecting customer satisfaction. For an insurance claim resolution agent, it might mean reducing the amount of manual work required to complete a process.
Those are very different goals, and they result in very different evaluation strategies.
Technical metrics are important as well. We want to understand latency, token consumption, tool failures, answer quality, groundedness, and many other characteristics of the system. But a technically impressive agent is not necessarily a useful one.
Those same success criteria should guide model selection. Compare candidate models on representative tasks from your application: can they produce grounded answers, choose the right tools, and complete the task within your latency and cost constraints? Choose the model that meets those requirements, rather than defaulting to the most capable model for every task. Re-run those evaluations before switching models.
This becomes especially important when we try to calculate ROI. If we didn't establish the baseline and what we expected the agent to improve, it becomes very difficult to demonstrate later that the application created value.
Define these measures early and keep them close to the application throughout its lifecycle. They will eventually become the foundation for your evaluations and production monitoring.
Don't build it alone
There is another tool available to us that we think developers sometimes forget about when building agents: other agents.
Coding agents have become extremely capable. Whether you use Github Copilot, OpenAI Codex, Anthropic Claude Code, or one of the many other tools available today, they can help with a significant part of the work involved in building an agentic application. And you should use them.
We don't mean asking a coding agent to blindly build your entire application from a one-line description and hoping for the best. The engineering principles in this post still apply. You need to understand the problem you are solving, make architectural decisions, define your success criteria, and validate what gets built.
But there is very little value in manually writing every piece of boilerplate when a coding agent can help you scaffold the project, integrate APIs, create tests, troubleshoot an authentication issue, or iterate on your agent implementation.
This gets particularly interesting when the coding agent understands your target platform.
Bring Foundry into the workflow you already use. Foundry Toolkit for VS Code, Azure Developer CLI, and GitHub Copilot integrations support local development, scaffolding, debugging, and deployment. These tools give coding agents platform-specific guidance and give developers ways to work with Foundry from their terminal and development environment, carrying the same agent implementation toward production.
The important part for us is that coding agents should not be treated as fancy autocomplete anymore. Give them the context they need and access to the appropriate skills and developer tools. Let them help you inspect your project, implement changes, run tests, evaluate results, and iterate.
Interestingly, many of the recommendations in this post apply to your coding agent too. Give it vague instructions and little context, and you will probably spend your afternoon correcting it. Give it clear requirements, the right tools, platform-specific guidance, and a way to validate its work, and the experience is very different.
You are building agents. You might as well use agents to help you do it.
If you want to get started, Microsoft's guide for setting up the Microsoft Foundry development environment covers the Foundry developer tooling and how to configure your environment.
Be very clear about what you expect from an agent
Ever seen an agent get things completely wrong? It happens often. Not because the underlying models aren't capable, but because we tend to expect them to understand information that we never actually supplied, like you would do with your co-worker. The difference is your co-worker has months, if not years of context working with you. An agent only has the context available to it.
If the instructions are ambiguous, the agent has to resolve that ambiguity. If two instructions conflict, it must decide which one to follow. If a tool has a vague description, the LLM model has to guess when that tool should be used. The more guessing we introduce, the less predictable the application becomes. Take a tool described simply as:
def get_customer_data():
# Gets customer information.
...There is technically nothing wrong with that description, but there is a lot the model needs to figure out. What customer information does it return? When should it be called? What identifier does it expect? Should it be used for transactions as well? What about support cases?
We can remove most of that uncertainty ourselves:
def get_customer_profile(customer_id):
"""Returns the customer's account profile, including:
- name,
- account tier,
- region and
- account creation date.
Use this tool when account profile information is required.
Do not use it to retrieve transactions or support cases. """
...This is a small example, but the same idea applies across a complete agentic application.
Instructions, tool definitions, knowledge sources, memory, skills, and even descriptions of other agents are ultimately information we are providing to a model so that it can make a decision. The clearer that information is, the less work the model has to do figuring out what we intended.
Longer isn't automatically better either. Adding another three pages to the system prompt every time the agent makes a mistake will eventually create other problems.
That leads us directly into context management.
Pay attention to what goes into the context
When an agent invokes an LLM, a surprising amount of information can end up contributing to that request.
There are the agent instructions and the user's input, but potentially also conversation history, tool definitions, retrieved documents, memory, skills, previous tool results, and information coming from other agents. All of that competes for context.
Too little context and the model doesn't have enough information to do the job. Too much context and you are paying to process information that might not even be relevant to the current task. Worse, useful information can become harder to identify among everything else you provided.
This is also why simply adding more tools isn't always an improvement. Each tool needs a description and input schema that the model can understand, so a growing toolset gives the agent more information to navigate. In Foundry, tool search retrieves relevant tools instead of loading every definition into each request, while skills supply reusable procedures when needed. A toolbox provides a shared, governed endpoint; tool search is what addresses the context cost of presenting the whole catalog.
The same applies to conversation history and memory. Persisting something doesn't automatically mean it should be provided to the model on every subsequent request. This is why we think context management needs to be treated as an architectural concern, not just a prompt engineering problem.
You need to understand what information is reaching the model, why it is there, how often it is being sent, and whether it is actually improving the outcome. And once applications become more complex, understanding this by looking at the final response becomes almost impossible. Doing it by hand gets tedious quickly, which is where insights and optimization come in later in this post.
Which brings us to one of our favorite topics.
Make your agent observable from day one
Agentic applications can fail in many creative ways.
Maybe the model picked the wrong tool. Maybe it picked the correct tool but passed the wrong parameters. Perhaps retrieval returned poor information. Perhaps retrieval returned too much information. Maybe the system prompt contains two instructions that made perfect sense independently but contradict each other when combined.
From the user's perspective, all of those problems often look exactly the same: The agent gave me a bad answer. And good luck debugging that from the output alone.
This is why we strongly recommend investing in tracing and observability early, including during development.
When you can inspect the actual execution of the agent, the conversation changes considerably. Instead of guessing why the agent failed, you can see which tools were called, what was retrieved, where time was spent, and what happened throughout the execution.
OpenTelemetry (OTEL) traces are particularly useful here because agentic applications are still distributed applications. The model invocation is important, but so are the tool calls, retrieval operations, downstream services, and other agents involved in completing the request.
A trace can show that an answer was bad because retrieval introduced irrelevant context, that latency increased because the agent made five unnecessary calls, or that a tool description caused the model to choose the wrong operation. These are all things we can fix, but only when we can see what the agent is doing.
Foundry's tracing helps developers inspect this behavior and understand what happens inside their agentic applications.
Figure 1: Foundry tracing showing a successful tool invocation with input tool parameters and output responseFigure 2: Foundry tracing being used to debug a high latency invocation. The tool invocation reaches a timeout exception but the agent retries the invocation leading to a successful turn but with high latency
Insights in Foundry (currently in preview) goes a step further: it analyzes production traces and groups recurring behavior into insights, each with representative traces and a likely cause, so you aren't reading traces one at a time. Once you can observe the behavior, you can start systematically measuring whether it is getting better.
Build evaluation into the development lifecycle
Evaluations shouldn't be something you run once before going to production.
Every meaningful change to an agent has the potential to change its behavior. Change the system instructions and something else might regress. Add a tool and the agent now has another possible choice. Change the model and assumptions you made during development might no longer hold. Change retrieval and the model receives different information.
This means you want an evaluation dataset early in development.
It doesn't have to contain thousands of examples. Start with representative scenarios that are important to the application. Include the happy paths, but also include ambiguity, unusual phrasing, expected tool calls, cases where the agent should ask for clarification, and cases where it should not take an action at all.
You don't have to build that first dataset by hand. If your agent already emits traces, Foundry can convert them into an evaluation dataset using intelligent sampling, which picks a representative set instead of a random slice. The same traces can also seed a rubric evaluator. If you don't have production traces yet, Foundry can also generate a synthetic dataset from your agent's instructions, a prompt, or a reference document such as a policy or spec, which covers prelaunch scenarios and edge cases. The two complement each other: traces show real user behavior, synthetic data covers what users haven't asked yet.
The success criteria we defined earlier become useful here. Some evaluations will be technical, such as task completion, tool selection, groundedness, latency, or token consumption. Others can be much closer to the business outcome.
Foundry gives us a few ways to turn those criteria into evaluations. The built-in evaluators cover common measures such as relevance, groundedness, task completion, tool call accuracy, and safety risks, so you get a first signal without writing anything. For criteria specific to your application, rubric evaluators use an LLM as the judge to score responses against weighted criteria you define, and you can auto-generate the first version of the rubric from your agent's instructions. Custom evaluators cover the rest: code-based (a Python function for deterministic checks such as format or length), prompt-based (your own judge prompt), and endpoint-based (an HTTP endpoint you host, useful when scoring logic lives in your own infrastructure). Rubric and custom evaluators are currently in preview.
As the application evolves, the dataset should evolve with it. Production failures that matter are particularly good candidates for future regression tests.
Evaluation also has a practical limitation: it costs money.
Running every evaluator against every interaction in production can quickly become expensive. This is why an evaluation strategy matters just as much as having evaluations.
Match the evaluation scope to the stage of development. During development, run smaller benchmarks frequently. Before important releases, run larger test sets through your CI/CD pipelines. In production, evaluate representative samples. The aim is enough data to understand behavior and identify regressions, not every evaluator on every request.
Foundry brings evaluation into the same development lifecycle as the agent. Combined with agent insights, that gives us a way to move from diagnosing behavior to testing improvements. The agent optimizer uses that evaluation evidence to generate and rank candidate instructions, skills, tool descriptions, and model selections for prompt and hosted agents, which takes over much of the manual tuning that context management otherwise needs. We still need to assess quality, latency, and token consumption together: a version that scores better does not automatically cost less to run.
There is enough here for an entire post, so we will explore evaluation and agent optimization separately in this series.
Be deliberate about identity and isolation
Personalization is important, but it comes with an equally important requirement: my experience with an agent can be different from yours, but my data should never accidentally become part of yours.
That sounds obvious. Implementing it correctly requires thinking about several different concepts.
Conversation state, session isolation, memory, user identity, agent identity, and authorization are related, but they aren't interchangeable.
Memory, for example, can make an agent much more useful. The agent might remember preferences or information that allows it to provide a better experience over time. Foundry's managed memory, currently in preview, provides persistence without requiring developers to provision an external memory database. But managing the infrastructure doesn't answer the governance questions: whose information is it, where is it stored, how is it scoped, when should it be retrieved, and how long should it be retained? Those decisions still belong in the application design.
Sessions introduce another boundary. In hosted agent architectures in Foundry, session isolation provides an execution boundary for a workload. Prompt-based experiences have their own mechanisms for maintaining conversation state. In either case, developers need to be deliberate about where state lives and what is shared (and can't be shared) between interactions.
Identity becomes especially important when agents start using tools. Suppose we ask an agent to retrieve a document from a system we use at work. The easiest implementation might be to give the agent a highly privileged application identity capable of reading everything. It is probably not the implementation we want.
Microsoft Entra ID gives us the capabilities to do this differently. One pattern you will encounter is On-Behalf-Of, usually shortened to OBO.
At a very high level, OBO allows an application to call another service using the user's delegated identity. If you are using an agent to access information on your behalf, the downstream system can still make an authorization decision based on who you are and what you are allowed to access. Foundry supports this through user delegation, and the configuration guide shows how to set it up.
You don't need to understand every OAuth detail to appreciate why that matters. If you cannot access a document, adding an LLM between you and the document shouldn't suddenly give you access to it.
Foundry provides security building blocks, but they still need to be configured for the application. Microsoft Entra Agent ID gives agents an enterprise identity, and the Foundry runtime honors Agent 365 governance actions. Private connectivity, tool authentication and runtime guardrails help control access and behavior, with support varying by component and configuration. Security, identity, memory, and isolation get complicated quickly, so we will examine those boundaries and tradeoffs in a follow-on post. Until then, the identity and access overview is a good map of where each decision sits, and agent identity concepts explains how Foundry uses Entra Agent ID.
Keep application logic in your application
Earlier we mentioned math as an example of something that doesn't necessarily belong inside an LLM invocation. The same principle extends much further. Validation, transformations, deterministic routing, permission checks, sorting, filtering, and established business rules are operations that traditional software is very good at. Use it.
An agent is part of an application. It doesn't replace the application. If a user asks an ambiguous question that requires interpreting intent and deciding which system can answer it, that's a good place for reasoning. Once the system knows exactly which API to invoke and has received structured data back, many subsequent operations are better handled by code.
This distinction is useful for reliability, but it can also have a significant impact on latency and cost. A few unnecessary LLM calls probably won't matter during a demo. At production volume, they matter a lot.
There is one nuance here. Telling an LLM not to calculate something directly doesn't mean an LLM can't help solve the problem. LLMs are also very good at generating code. For some workloads, generating code and executing it in an appropriate controlled environment can be a much better architecture than asking the model to reason through every intermediate calculation in tokens. This is especially interesting for data analysis and tasks where the requested computation itself is dynamic. Foundry gives us an off-the-shelf Code Interpreter tool that acts as a secure sandbox for this approach.
The decision is not whether the model is capable of doing something. In many cases, it is. The more useful engineering question is whether a model invocation is the right component of the application to do it.
Plan for scale: within an agent, and across your organization
A successful proof of concept eventually faces two different scale problems: one agent serving thousands of users, and an organization creating thousands of agents. The first is about production capacity and reliability. The second is about reuse, ownership, and governance.
An agent that works very well for ten people needs to continue working when thousands of users invoke it concurrently. At that point you start thinking about throughput, latency, quotas, model capacity, downstream dependencies, rate limits, connection management, resiliency, and cost.
Agent runtimes can help with part of that problem, but your agent is only one component in the architecture. Its models, tools, data sources, APIs, and other dependencies all need to support the level of traffic you expect.
This is also where capabilities such as AI Gateway become interesting. As AI usage grows, organizations need mechanisms for managing access to models, controlling and observing traffic, enforcing policies, and understanding consumption across applications.
Cost governance needs explicit ownership and limits: set budgets for each application or team, track consumption against them, and define who responds when spending approaches those limits. Azure Cost Management remains the system of record for spend. AI Gateway provides project-scoped tokens-per-minute limits and total token quotas; those are token controls, not native dollar-budget enforcement in Agent Service. Decide in advance how budget alerts and request controls will work together, including when to alert, throttle, or stop requests, rather than waiting for the bill to reveal a problem.
And there is another kind of scale that we find equally important: what happens when the problem isn't one agent serving thousands of users, but an organization creating thousands of agents?
The requirements change. Different teams may independently create similar tools. Every agent might implement authentication differently. Tool integrations get duplicated. Agents need owners and lifecycle management. Developers need ways to discover capabilities that already exist. Organizations need governance without preventing teams from moving quickly.
Tools should become reusable building blocks rather than implementations hidden inside individual agents. This is one of the reasons we find approaches such as reusable Foundry tooling and MCP particularly interesting. A well-designed capability can be exposed once and consumed by multiple agents instead of every team independently rebuilding the same integration.
The agents themselves can also become reusable. A specialized agent built by one team might provide useful capabilities to another agent or application. Multi-agent communication makes it possible to compose those capabilities instead of creating one enormous agent that knows how to do everything.
Distribution is another part of this problem. Users shouldn't necessarily need to visit a separate application for every agent an organization creates. Depending on the use case, agents may need to appear in applications, websites, Microsoft Teams, Microsoft 365 Copilot, or other channels users already work in. Foundry Agent Service provides a publishing path into Microsoft Teams and Microsoft 365 Copilot, with Agent 365 integration for tenant-wide governance. That connects the engineering decision about where an agent runs with the adoption question of where people will actually use it.
Once organizations reach this point, the problem is no longer how to build an agent. The problem is how to provide an agentic platform that allows many teams to build agents without every team independently solving identity, tooling, observability, evaluation, governance, integration, and deployment.
We are going to spend much more time on that problem later in the series.
And this is just the start
The goal isn’t to add more AI at every step. It’s to build applications that solve real problems and keep improving them as we learn. These principles are a starting point; their value comes from putting them into practice. In subsequent posts, we’ll get hands-on with evaluation and optimization, security and identity, context management, and scaling, exploring what Microsoft Foundry provides, the architectural tradeoffs, and how to turn what we learn into better applications, and more business value.
Microsoft