Authors: Han Che, Microsoft; Ahmed AbdulJalil Ebrahim, Gulf Air
On November 9, Gulf Air reached a milestone: two AI agents in its Falcon Eye application—a voice agent and a chat agent—went live for users.
The milestone did not come from finding one perfect prompt. Gulf Air had spent weeks building a demanding pool of evaluation questions, studying traces, finding gaps, changing instructions and code, and testing again. Ahmed reported that the team generated approximately 10,000 candidate questions across different business perspectives, then ran selected questions in smaller batches to exercise real aviation and management logic.
Ahmed AbdulJalil Ebrahim described the impact plainly:
|
" What evaluation, insights, and optimize did for us was really amazing. We were able to fix business logic the agent had not understood." |
For us, the important part of this story is not that one tool produced one answer. It is that Gulf Air assembled a repeatable improvement practice around the hard questions its business would actually ask.
The application had to do more than retrieve an answer
Falcon Eye is an important Gulf Air application used for operational analysis and management decision support. Ahmed described a five-agent vision spanning UI summaries, smart search, dynamic reporting, real-time voice, and a reasoning chat experience.
These agents connect to operational data through APIs and a Postgres MCP. They can analyze information, perform calculations, and help users create reports or dashboards. That makes them useful, but it also raises the quality bar.
A simple factual test cannot establish whether an agent:
- selected the right operational field for the user's question;
- followed an aviation-specific rule;
- generated an appropriate query;
- performed a calculation correctly;
- preserved the user's data permissions; or
- remained reliable when the same intent appeared in a different form.
For senior leaders asking operational or financial questions, a fluent answer is not enough. The reasoning path and the business result both matter.
Gulf Air built an evaluation pool around the hardest business questions
Ahmed described this generated question pool as covering multiple business perspectives. He emphasized that these were not only straightforward questions such as total departures or delays. They were intended to test the business logic that had been difficult to express through a system prompt alone.
The team selected questions from that pool and generally ran them in batches of 100. The evaluation results separated successful answers, failures, no-answers, and errors. Gulf Air then examined the resulting traces and insights, brought the evidence into VS Code with its engineering context, implemented changes, and evaluated again.
The loop looked like this:
|
Run difficult evaluation questions |
Human review was a deliberate part of the process. Other team members approached the agent differently and documented issues that the original test design had missed. Their findings became new inputs to the next improvement cycle.
Immediately before our November 9 conversation, Ahmed had completed another 400-question run and applied additional fixes. This was not a one-time benchmark. It was an operating rhythm.
Evidence became useful when it entered the engineering workflow
The workflow did not stop at an evaluation score or a list of traces. Gulf Air used the evidence to decide what to change.
Insights from traces helped group recurring behaviors and connect them with representative executions. VS Code supplied the surrounding engineering context: code, APIs, MCP access, reports, and prior findings. The team could then investigate whether an issue came from instructions, tool behavior, application logic, or the test itself.
This distinction matters. An insight can identify a pattern and point to a likely direction, but it is not automatically a validated fix. Gulf Air still reviewed the evidence, implemented the change in the appropriate system, and ran the agent again.
That developer-led pattern is also central to the latest Microsoft Foundry experience:
- Scheduled and on-demand analysis can process new traces incrementally after the initial lookback.
- Linked and highlighted traces let developers inspect the evidence behind an insight.
- Custom categories and additional instructions help focus future analysis on business-specific concerns.
- A proposed change can be copied into a coding workflow when the issue belongs in code or application configuration.
- For eligible instruction or tool-description issues, developers can use fix insights and create an agent optimizer run.
The handoff is important, but so is its boundary: a proposed fix is a starting point for validation, not an automatic production change.
From a selected insight to an evaluated candidate
From the Insights tab, a developer can choose Select insights, select the relevant insights, and choose Fix insights to create an optimization run. Foundry uses the selected evidence to prepare supported replay scenarios, proposes candidate changes, and compares them with the current agent version as the baseline.
The goal is not to accept whichever candidate has the highest single score. Developers can inspect:
- which selected issues improved;
- the candidate's actual instruction or tool-description changes;
- per-scenario and overall evaluation results;
- quality comparisons with the baseline;
- token usage and estimated cost;
- latency; and
- whether independent conversations were available to check for broader regressions.
Sometimes the responsible result is still the baseline. A candidate might fix one failure while regressing elsewhere, the evaluator might find the result inconclusive, or the available evidence might be too narrow. The developer reviews those trade-offs and explicitly decides whether to promote a candidate.
Nothing in this workflow automatically changes the active agent. Human review, promotion, testing, and production activation remain separate decisions.
Gulf Air assembled this handoff manually. The new select-insights-to-fix experience is designed to shorten part of that journey by carrying verified insight evidence into a controlled optimization run.
Confidence was the business outcome
The value of this work was not a single score or a one-time test. Each evaluation cycle produced reusable evidence - failure cases, traces, human-reviewed findings, and proposed changes that the team could carry into the next iteration. Validation became part of the engineering workflow rather than a final checkpoint before launch.
The business outcome was confidence. Ahmed described the workflow as giving Gulf Air a level of confidence in its agents that the team had not achieved before. He also reported that Gulf Air management was impressed by the voice and chat-agent demonstration on launch day.
From Gulf Air: what changed for our team
“The agent refinement experience changed the way we approach agent AI development at Gulf Air. Instead of relying on individual tests or continually adjusting prompts, we can systematically challenge our agents with thousands of real business questions, understand where they struggle, and use evaluation, traces and insights to identify the underlying issues. This approach helps us refine how agents understand complex aviation business logic and gives our teams greater confidence in the agents we move into production. More importantly, it establishes a continuous development cycle where real-world usage and evaluation drive the next improvement.”
Five lessons for teams moving agents into production
Gulf Air's experience suggests five practical lessons.
1. Build the evaluation set around business difficulty
Start with the questions your business struggles to encode, not only the questions your agent already answers well. Include ambiguity, calculations, tool use, permissions, and domain rules.
2. Use insights to find patterns beyond predefined evaluations
Evaluations test the requirements a team already knows how to define. Insights can look across production traces for recurring behavior the team did not anticipate, bring related executions together, and show representative evidence. Custom categories and additional instructions can focus that analysis on business-specific concerns. Developers still review the cited traces and compare the finding with business expectations before deciding that an issue is real.
3. Turn recurring guidance into a reusable skill
When a verified issue reflects guidance the agent should apply across many future interactions, repeatedly tuning prompts might not be enough. A versioned skill can capture that learning in a form the team can review, evaluate, and deliberately adopt.
Not every issue belongs in a skill. Code defects, tool-description problems, data-quality issues, and platform failures still need to go to the workflow that owns the responsible layer.
4. Compare trade-offs, not only quality scores
A candidate can improve one behavior while using more tokens, adding latency, or regressing on normal conversations. Compare candidates with the baseline and inspect the underlying evaluation cases.
5. Keep production changes explicit
Continuous analysis can discover new patterns, and optimization can test candidates. Neither should remove the developer's responsibility to review evidence, approve changes, and test the promoted version before sending production traffic to it.
The next chapter: keep learning after launch
Production is not the end of the evaluation set. It is where new evidence begins.
Real users ask questions in ways a development team did not anticipate. Data changes, tools evolve, and a response that works for one scenario can fail at the edge of another. The opportunity is to turn those new production behaviors into the next round of learning without losing the controls that made the launch possible.
For Gulf Air and other enterprises building complex agents, that means establishing a repeatable rhythm:
- Observe new behavior in production traces.
- Verify the pattern against representative evidence and business expectations.
- Turn confirmed failures into evaluation coverage.
- Test candidate changes against the current baseline.
- Compare quality, latency, and cost rather than optimizing one measure in isolation.
- Promote a change only after human review, then watch the new version under real traffic.
This is the larger promise of continuous improvement in Microsoft Foundry. Insights from production traces can help teams discover what deserves attention. Evaluation makes expected behavior testable. Agent optimizer can compare possible improvements. Developers remain responsible for deciding what is correct, what is safe, and what should reach production.
The next milestone is not an agent that changes itself without oversight. It is a controlled learning system in which production evidence, business context, evaluation, and human decisions reinforce one another.
For Gulf Air, the first milestone is already real: difficult questions became evidence, evidence became engineering work, and that work helped two agents reach production with greater confidence.