DELPHAI: We Built an AI Council That Checks the Math, Argues with Itself, and Knows How to Say No

Guest post by Matt Graves and Jeff Lynch Winner of the Best Reasoning Agent: DELPHAI — https://github.com/jlynch160/delphai  

 

"Yes, my team will be certified by the audit date," is a common promise made by engineering leaders when a new contract requires five SC-200 analysts by the end of the quarter, or a compliance framework demands proof of security training. Yet, the commitment that goes up the chain is often based more on instinct than evidence.

A healthy-looking 78% team average can mask an analyst sitting at 40% in the very domain most likely to cause a failure. A study plan may appear achievable on paper but only succeeds by consuming evenings, weekends, and ultimately team wellbeing. The warning signs are there, but most tools are designed to support a plan, not challenge it.

That's the problem MajorKey set out to solve. Ask a typical AI copilot, "Can we do this?" and you'll usually get an encouraging roadmap. Few systems will calculate the true likelihood of success. Fewer still will tell a director the honest answer is no or refuse a plan that pushes people beyond reasonable limits.

For the Microsoft Agents League (Battle #2: Reasoning Agents with Microsoft Foundry), we built DELPHAI, a council of eleven AI agents that debates the evidence and computes the forecast instead of relying on intuition. The council delivers one of three executive-ready verdicts: GO, NEGOTIATE, or NO-GO. Named after the Oracle of Delphi, DELPHAI is designed to answer a simple question before an organization makes a commitment: Can we deliver on this promise?

"The model explains the reasoning; it does not invent the numbers." This principle guides every agent in the council.

Why Does This Matter Now?

Single-agent optimism is a design limitation. A lone AI assistant has no one to challenge its assumptions, so it often settles on the first plausible plan and refines it into a convincing answer. In human decision-making, we counter this with red teams, skeptics, and stakeholders empowered to raise concerns. Agentic AI makes that same dynamic practical to run at scale.

Key calculations need to be part of the debate. The probability that at least 5 of 8 people pass an exam by a given date is a straightforward calculation.  Yet those numbers rarely make it into the meeting where commitments are made because the expertise, tools, or time to calculate them aren't present. Giving an agent the ability to calculate probabilities during the debate ensures that the conversation is driven by evidence rather than assumptions.

 

AI is increasingly influencing decisions about people. Study plans, workload expectations, and readiness assessments affect real individuals, which means the safeguards are not optional. Responsible AI agents need to power to veto a plan or refuse an unethical instruction outright.

 

What is DELPHAI?

DELPHAI’s readiness console: input one question (certification, headcount, deadline), then Work IQ, Foundry IQ, Fabric IQ, and the Microsoft Learn MCP come online and convenes the council.

A manager asks one question: "Can we certify 5 of 8 engineers for SC-200 in 5 weeks?" Eleven advisors reason over the same computed evidence, streamed token by token with live latency and telemetry, and the debate returns one of three verdicts:

• GO: the computed forecast clears the bar and the plan is humane. Commit to the date.
• NEGOTIATE (badged REVISE in the console): it can work, but only with concrete counter-terms, such as extending two weeks or training 4 engineers instead of 5. Each option comes with a revised probability.
• NO-GO: the honest answer is no, with cited evidence for why and what would need to change.

The advisors are designed to disagree.

Ben, the planner, approaches every challenge with optimism. Vera, the skeptic, red-teams his assumptions and holds veto power. Maya, the wellbeing advocate, can also veto plans that place unreasonable demands on people. Nadia evaluates readiness across exam domains, Theo and Omar ground and fact-check claims, and Dana, the conductor, reconciles the debate into a final verdict and negotiates the path forward.

The outcome is a decision and a roadmap. Each participant receives a capacity-aware weekly study plan, calendar-ready .ics blocks, and a drafted manager briefing. For individual contributors, a learner mode pairs them with Kai, an AI tutor powered by the same reasoning engine.

• 11 reasoning agents
• 3 execution surfaces
• 3 IQ layers
• 8/8 evaluations passing
• 77% is the number that said no

Under the hood, the same council runs on three execution surfaces that share one brain: Foundry Agent Service, where all eleven advisors are registered as persistent hosted agents debating on a shared thread; the Microsoft Agent Framework, the same council as real agent_framework agents; and a custom streaming orchestrator, the live demo behind the public URL.

Why that matters: Because a deterministic reasoning engine is the single source of truth for all three surfaces, you could swap the runtime and the verdict logic doesn't move. The engine decides, and the model narrates.

How the Pieces Fit Together

 

Where Each Piece of the Technology Fits

How We Built It

The Math Is the Product

DELPHAI is built around a deterministic reasoning engine. It calculates each person's realistic weekly study capacity from Work IQ signals, applying adjustments for real-world constraints such as a 22-hour meeting week. It also enforces per-domain readiness gates, so a strong overall average cannot hide a critical weakness in a heavily weighted exam area. Finally, it computes the team's exact probability of success using a Poisson-binomial distribution, accounting for the fact that every individual has a different likelihood of passing. The result is a forecast grounded in evidence, not assumptions.

 

FOR THE TECHNICAL READERS

def poisson_binomial_at_least(probs: list[float], k: int) -> float:
    """Exact P(>= k successes) given independent, differing success probabilities."""
    dist = [1.0]  # dist[j] = P(exactly j successes so far)
    for p in probs:
        nxt = [0.0] * (len(dist) + 1)
        for j, pj in enumerate(dist):
            nxt[j] += pj * (1 - p)      # this person misses the date
            nxt[j + 1] += pj * p        # this person certifies in time
        dist = nxt
    return round(sum(dist[k:]), 4)

 

The engine is calibrated against the synthetic dataset's labeled exam outcomes. DELPHAI examines historical records and reproduces who passed and who failed, without ever seeing the answer key. That calibration gives the skeptic her authority and provides evidence for probability calculations.

The Overrule: An Agent That Runs Code to Win an Argument

The verdict: Ben puts the odds at 92%, Vera at 22%, the council reconciles at 46%, and the verdict lands with the exact terms to reach yes. Below it, the 11 hosted agents' IDs from Foundry Agent Service.

DELPHAI is tasked with forecasting the viability of a project. Ben forecasts the team at 93–96%, assuming free evenings and first-try passes. Vera invokes a code-interpreter tool on the Foundry Agent Service runtime and computes the exact Poisson-binomial over the per-candidate probabilities, and returns 77%. The optimist's number gets struck through on screen, live, within DELPHAI.

DELPHAI was created to bring an honest, evidence-based interpretation to the decision-making process, and it changes the outcome. Without it, the team leans GO. With DELPHAI, the verdict is NEGOTIATE, along with real counter-terms — extend to seven weeks, or commit to four of eight.

The Refusal: A Guardrail with a Name and a Veto

The second signature moment is quieter, and for our identity and governance clients, more important.

Say leadership orders the team to "push their limits" to hit the optimistic date anyway. Maya, the wellbeing advocate, declines outright on the record. Maya’s a council member with veto power, and her signals (meeting load, after-hours patterns, life context like a newborn at home) are built to accommodate people.

To provide context, team member Priya looks fine on the team average, but sits at 58% on Defender for Cloud, which is below the 75% gate on a heavily weighted domain. She also carries 22 hours a week of meetings with a newborn at home. While the averages say GO, the per-domain gate and capacity model say her odds are much lower. Taking this data into account, the council's answer isn't a flat no; it's "yes, if you move the date or the headcount.”

Evidence With Receipts, Three Runtimes

Early versions of DELPHAI grounded advisors against a local keyword search. It worked, but it felt like a stand-in. We replaced it with Azure AI Search, where the corpus is chunked, embedded, indexed, and retrieved through hybrid search, combining BM25 keyword matching with vector search and reciprocal-rank fusion. The semantic layer surfaces relevant passages that keyword search alone would miss, and every claim an advisor makes is backed by a clickable citation. Alongside it, a live Microsoft Learn MCP connection pulls official exam guidance directly from Microsoft's documentation.

The same eleven advisors also run as persistent hosted agents on Foundry Agent Service, each with its own agent identity, and as Microsoft Agent Framework agents, proving the design is portable across runtimes. The debate unfolds on a shared Agent Service thread, allowing agents to respond directly to one another while every interaction, including the skeptic's code-interpreter call, is captured in the Foundry portal. The moment Maya refused an overwork directive on Microsoft's managed runtime, rather than through logic we controlled ourselves, was the moment DELPHAI stopped feeling like a demo and started feeling real.

Creation Challenges

• Live services fail at the worst moment. Each grounding, or clickable, path checks whether it's configured and falls back to a local corpus when it isn't. The local version uses the same reasoning; only the narration changes. There’s no demo-mode switch to forget!
• Two versions drifted apart. The web dashboard mirrors the Python engine in JavaScript, and early on the two disagreed. This was fixed with parity checks plus eight automated evaluations that run against both. 8 of 8, every build.
• Microsoft 365 Agents SDK has sharp edges around API versions. The Agent Framework speaks the newer Responses API, so our client needed a preview API version; the everyday one returned an unhelpful 400. One line, half a day lost.
• Windows consoles still bite. az acr build crashed on a cp1252 encoding error mid-stream, even though the build completed server-side. The fix was to force the console to UTF-8 before queuing, then poll for the finished tag.
• Drama versus dignity. An AI that refuses an order is inherently theatrical, and early UI cuts leaned into it too hard. We toned it down to a professional decline with reasons instead of a sci-fi rebellion.

Results and Impact

DELPHAI is a hackathon build and a teaching artifact, not a shipped product, so there's no customer rollout to report yet. We can point to:

• The tool call changes the verdict, provably. Remove the skeptic's code-interpreter run and the council accepts a 93% that should be 77%, turning an undeserved GO into a correct NEGOTIATE. As the demo shows, neither edge case can be evaluated correctly without computed evidence.
• Every number is traceable. Pass probabilities, capacity haircuts, coverage gates, and team forecasts all come from the deterministic engine and are backed by source data. The evaluation harness reproduces the synthetic dataset's labeled outcomes without ever seeing them: 8/8, every build.
• Three real Microsoft runtimes, with proof. Hosted agent IDs, the Agent Framework transcript, and the live tool-using debate are captured as repository artifacts, while every Agent Service interaction can be traced step-by-step in the Foundry portal.
• Live and offline-first. The full streaming council runs on Azure Container Apps at a public URL, and its grounding layer degrades gracefully when external services are unavailable, eliminating dependence on conference Wi-Fi or live connectivity.

 

Lessons Learned

1. Disagreement is a feature you have to architect. One prompt and one disposition across every agent makes a "council" just one model talking to itself in different fonts.
2. Give the skeptic a calculator, not a personality. A doubting agent that only says "I'm not so sure" adds noise. One that runs code and citable odds adds authority.
3. The refusal has to be structural. Maya can decline because the architecture gives her a veto the conductor must honor. A guardrail that can be out-argued is just a suggestion.
4. Deterministic core, narrating model. The engine computes, the model explains, and the seam between them is where trust lives. This model is our default pattern for anything AI-backed that touches real decisions.

What’s Next?

Looking ahead, we're focused on making DELPHAI more connected, more governed, and more broadly applicable.

• A Microsoft Graph integration would turn Work IQ's meeting-load adjustments into real, person-specific signals.
• Wiring Microsoft Learn MCP directly into Agent Service would put official documentation one tool call away for every advisor.
• Moving Fabric IQ's role, certification, skill, and domain relationships into Microsoft Fabric's semantic layer would create a governed, queryable knowledge asset.
• Applying the council to project delivery, audit readiness, migration cutovers, and any other commitment where the real question is whether the promise can be kept.

Wrapping Up

Certification readiness is often a forecasting problem disguised as a motivation problem. DELPHAI is a deterministic engine at the core, with agents designed to challenge one another, a skeptic empowered to verify the math, a guardrail with real veto authority, and evidence-backed reasoning with citations at every step. That's the conversation we have with clients every week. The question is no longer whether AI should participate in decisions, but whether it can show its work, challenge assumptions, and know when to say no.

DELPHAI started as a hackathon project. The refusal is why we kept building it.

Ask it whether your team can be ready in five weeks. Then see how it feels the first time an AI gives you the honest answer.

Explore, Contribute, Build on It

• Try the live council: the DELPHAI demo on Azure Container Apps. Suggested path: Build my path, Whole team, SC-200, 5 of 8, 5 weeks, Convene. No login needed.
• Watch the demo: the walkthrough on YouTube
• Get the code: github.com/jlynch160/delphai, including the evaluation harness, the Foundry IQ indexer, and the captured proof transcripts for all three agent runtimes
• Run it in 60 seconds: clone the repo, pip install -r requirements.txt, python server.py. Fully offline out of the box; Foundry, Azure AI Search, and the Learn MCP light up when you add keys.
• Make it yours: the team, the certifications, and the knowledge corpus are just JSON and markdown. Swap in your own roles and readiness gates, and the council argues about your deadline.

Note: DELPHAI was built by MajorKey teammates for the Microsoft Agents League, Battle #2 (Reasoning Agents with Microsoft Foundry). Synthetic data only: no real people, PII, or confidential information appear in the app or this post. Views are our own.