| Welcome to “Raising the Bar,” a new series exploring how rigorous evaluation helps us continuously improve Microsoft Copilot. We’re kicking off with a deep dive into FinanceBench and the loop behind it: learning from real finance work, building representative evaluations, measuring quality reliably, and using the results to improve Copilot in Excel. |
Our earlier inside look at Copilot in Excel Evals blog explained how we evaluate Copilot in Excel and use our evaluation system to improve. Here, we go deeper on FinanceBench, one of the more than 20 benchmarks we use to measure Copilot in Excel, and the continuous improvement loop behind it. We learn from customers using Copilot for finance work, build evals cases that reflect it, make the grading trustworthy, capture the judgement of experts, and use the results to improve both the benchmark and product itself.
FinanceBench: a deep vertical benchmark
FinanceBench is a benchmark curated from over 1,000 finance eval cases, each written and reviewed with finance professionals. These cases mirror the multi-step, real-world work for an analyst, FP&A lead, or banker: building or extending a three-statement model, a discounted cash flow, a comparable market analysis, a leveraged buy out, a driver-based forecast, or a management reporting pack. This type of work sits at the top of the complexity ladder we mentioned in our June blog, where several Excel capabilities and domain judgment need to come together for a correct result.
Building and curating FinanceBench
The mix of cases in FinanceBench come from these key sources: direct from finance practitioners via expert data partners, customer engagements and forward deployed engineer engagements, user research, and analysis of production signals. Together, they show which workflows customers care about most and how well Copilot handles them today. Since these sources keep producing new signal, we can keep evolving the benchmark too.
Each case in FinanceBench contains four key parts:
- Starting artifacts: a real-world model, with its complexity and data quirks, plus any additional grounding data
- Prompt(s): the task or workflow the user wants to accomplish, including multi-turn
- Expected output(s): a workbook or other artifacts that show the ideal result
- Rubrics: the itemized criteria used to grade the output
Improving Copilot in Excel requires evals that accurately reflect how finance professionals work. To ensure this, every test case undergoes additional review by finance professionals and agents to validate its realism and accuracy.
Beyond finance workflows, we curate cases by workbook shape, source data, complexity, and features used. This coverage enables us to tag and slice results for targeted insights. For example, our system can measure quality for a specific scenario such as "summarizing a revenue forecasting model in large workbooks with pivot tables.”
Rubrics and Grading
Rubrics define the set of specific requirements that an AI-generated response must meet. For example, in a three-statement financial modeling task, the criteria might require that interest expense be calculated using average debt balances or that an unbalanced balance sheet be clearly flagged for follow-up. A single FinanceBench test case typically includes dozens of these criteria to evaluate response quality.
Criteria look at different dimensions of quality such as correctness, task completion, aesthetics, and adherence to Excel best practices. FinanceBench goes deeper and measures whether the result meets the requirements of the finance task, including its formula relationships, model conventions, links between assumptions and drivers, and auditability.
Writing good rubrics often turned out to be harder than building the rest of the case. Finance experts know what good work looks like, but turning that into criteria automated graders can judge consistently is difficult. We've seen it fail in both directions:
- Too specific, so correct work fails: One early rubric required a lookup be built with XLOOKUP. The model used INDEX and MATCH, which works equally well, but is marked wrong.
- Too generic, so wrong work passes: A check to verify a final figure without validating it’s sourced by linked formula vs a value typed into the cell. We needed to ensure accuracy criteria checked both the method and the resulting values.
How we grade depends on the type of rubric criteria. Deterministic checks, such as whether a cell holds the expected value or a figure is formula-driven rather than hardcoded, use a codified grader that checks the workbook in code and returns the verdict. Broader checks, such as whether the financial model is easy to follow or the formatting suits the task, use an agentic grader. An agentic grader is an AI model that inspects the workbook and decides whether each criterion is met. We can further adjust the weight of criteria based on importance, such as greater weights on accuracy and formatting, before returning a final score.
To ensure high quality rubrics, we start with people. Finance experts review every case in Excel and answer two questions: does the expected output pass each rubric, and does each rubric fairly measure good work? We then rewrite criteria so they can be graded consistently. To find what's left, we run each task multiple times to detect variation and re-grade the same outputs to ensure the grader agrees with itself. Criteria that return different verdicts for the same work get fixed before they affect a result. This runs continuously, not once.
Improving the benchmark to improve the product
From June to July, Copilot in Excel pass rate on FinanceBench v2 continued to climb. When a benchmark gets too easy to pass, it stops telling us what to fix. When Copilot in Excel topped an 80% pass rate, to 85.1%, we knew it was time to revved FinanceBench to be even more challenging, trustworthy, and realistic.
Based on production customer signals and failure analysis, we added more extensive rubrics for aesthetics and formatting to FinanceBench v3. We also deepened the expert review of multiple outputs which surfaced subtle quality issues that we added to rubrics as well.
Copilot in Excel has since improved its pass rate on the harder FinanceBench v3 by 31pts, using a mix of product changes, model upgrades, and skill optimization. Pass rate is only one dimension of improvement. We also measure response quality, latency, and token consumption to understand the tradeoffs between different models, skills, and product configurations. A change that improves task success but significantly increases latency isn't necessarily the best outcome. Our evaluations help us make these tradeoffs deliberately, balancing quality, performance, and efficiency.
We don't think the benchmark is ever finished (we’re already working on v4). A benchmark only helps if it continues to highlight area for improvement and the cases reward the right things. If the expected output or rubric is wrong, scores can climb toward the wrong answer. As we make updates, we continue to leverage expert reviews with automated checks to maintain quality across the full set. Each round of improvements has helped us acheive better results for customers.
A hill-climbing example: the three-statement model skill
Once we have a challenging benchmark, we hill-climb against it. Here is an example from improving the three-statement model skill, a common financial model that other finance work builds on. By analyzing eval failures and trajectories, we uncovered the issues in the skill:
- Building a model in stages: some customers asked for a model in parts, such as the model now and a dependent schedule later, and we included cases where auditability rubrics expect unfinished items to stay visible. The initial skilled pushed toward forcing the balance sheet to balance, so the agent over-completed the task beyond the requested stage. We fixed the skill to recognize staged requests, leave dependent cells blank and highlighted, and say what remains.
- Interest calculations: The skill suggested using beginning balances to avoid circular references, even when the task called for average balances. Our rubrics to follow workbook conventions highlighted this issue, and we changed the skill to match the convention set by the workbook or the user request.
Across three-statement tasks, the updated skill improved pass rates by 12pts for GPT 5.6 and 7pts for Opus 5.5, as measured on 114 FinanceBench cases.
Skills are just one lever to improve the Copilot. We also tune prompts, build tools, update and train models. We run and monitor evals after every change and keep the improvement loop going.
Running the improvement loop with agents
More of our improvement work now runs as automated loops. Agents help with analysis and experimentation, while human experts still decide what high-quality output looks like. Three loops carry most of the work:
- Learning from real usage: Agents regularly analyze privacy-compliant production signals to find where Copilot struggles, such as recoverable errors, abandoned tasks, or patterns of dissatisfaction. They trace those failures through Copilot's steps to find likely causes and confirm eval coverage. When the production signals and the evals point to the same problem, we fix it first.
- Improving the rubrics: Agents attempt each case several times, grade the results, and flag criteria that look unfair or incomplete. They then propose revisions, and an expert reviews the changes. We can regrade existing outputs as many times as needed until the grader agrees with human judgment.
- Experimenting with changes automatically: Agents analyze failures, propose changes to prompts or skills, evaluate their impact on quality, latency, and efficiency, and keep only what improves the results. This works best with guardrails and we review every change before it ships. Not every experiment improves results. In fact, many don't. But, we still learn from them because each one rules something out.
These loops reinforce each other. Production signals show us what to investigate and include in evals; better rubrics make measurements more accurate; and accurate measurement shows whether fixes helped. We're increasingly using agents throughout this process, and these automated loops complement expert judgment and help us iterate faster.
Evolving FinanceBench
We continue to update FinanceBench not only when the product and models improve, but also as we learn more about the way Finance teams work. FinanceBench started with finance work in Excel, but finance workflows can go beyond Excel. For example, a financial analyst might build a model in Excel and use its results to create a pitch deck in PowerPoint or produce an investment memo in Word. These experiences are now embedded into Copilot too and teams can use Cowork to delegate long-running work across these apps.
Our goal for FinanceBench is a realistic, trustworthy, and challenging benchmark for Finance work. As workflows span more applications, our evals need to follow the work, not just the application. We're expanding FinanceBench to evaluate more cross-app and cross-host scenarios, including workflows that combine financial analysis in Excel with documents and presentations, and leverage context, memory, and personalization from Microsoft 365. We now run FinanceBench across Copilot in Excel, Copilot chat and Cowork, to ensure finance and spreadsheet work is high-quality everywhere. Through it all, we continue to leverage our finance experts, review loops, and quality controls throughout.
Closing
FinanceBench is one part of how we're improving finance work in Copilot. It helps us get there through a continuous loop: learn from customer feedback and production signals, test what matters, and use the results to improve both Copilot and the tests themselves.
Want to help raise the bar? The next time you use Copilot in Excel, give the response a thumbs up or thumbs down and tell us why. Your feedback helps show us where Copilot falls short on real work, and what we should measure and improve next.