Forum Discussion

Wiliam_Rosa's avatar
Wiliam_Rosa
MCT
Sep 18, 2026

An Azure Databricks data agent that never gets re-evaluated is a production model with no monitoring

Every ML team has learned, usually the hard way, that a model left unevaluated after deployment will silently degrade: the data distribution shifts, the original assumption stops holding, and nobody notices until the wrong output has already done damage. What's strange is that the same discipline rarely shows up when the "model" in production is a conversational data agent. The team validates the Genie Agent with a handful of demo questions, approves it, ships it, and from that point on its accuracy becomes an article of faith instead of a tracked metric.

Genie Ontology, the layered architecture Azure Databricks documented to give business context to data agents, has a final layer that exists precisely to close this gap: continuous evaluation and improvement, backed by a concrete mechanism called Genie Agent Benchmarks. It's worth understanding how it works, because it's the piece most likely to get skipped when a deadline is tight, and the most expensive one to be missing once the agent is already in production, quietly getting things wrong.

The mechanism: two ways to measure "correct," depending on the question type

A Genie Agent can hold up to 500 benchmark questions, each one running as an isolated conversation, with no thread context carried over, exactly as if a new user were asking for the first time. There are two evaluation modes, and the choice between them depends on the type of question:

Chat mode: compares the SQL the agent generated (or its result) against a "ground truth" answer supplied by whoever wrote the benchmark. The rule is objective: identical SQL is "Good," an identical result set in a different sort order is also "Good," a number that matches to 4 significant digits also counts. An empty result, an extra column, or a diverging single-cell value is "Bad." The comparison covers up to 5,000 rows, so any question whose plausible result exceeds that needs an explicit ORDER BY on both sides to avoid a false negative caused by truncation.

Agent mode: used when the response isn't a simple tabular result to compare, but a multi-step, text-based reasoning report instead. Here an LLM judge grades the response against an optional "evaluation note" that whoever wrote the benchmark provided, describing what the correct answer needs to contain.

The mode is chosen at run time, not when the question is registered, so the same question set can be re-evaluated either way depending on the kind of response the agent is currently producing.

My take: the detail I find most sensible about this design is that Chat mode rewards legitimate variation (different sort order, numeric precision) without rewarding structural error (an extra column, an empty result). A naive benchmark that demands byte-for-byte equality generates too much noise, flags a correct answer as wrong just because the agent sorted differently, and teams end up distrusting and ignoring their own benchmark over time. Calibrating the bar to separate "cosmetic difference" from "actual error" is what makes a team trust the number enough to act when it drops.

Hands-on: registering benchmarks with phrasing variation

A best practice the documentation calls out explicitly, and one that's easy to skip when benchmarks get registered in a hurry: the same business question rarely reaches the agent phrased the same way twice. A real user asks "what was last quarter's revenue" and "how much did we bill in Q3" about the same underlying data, and an agent that gets one phrasing right and the other wrong isn't actually reliable, even if a naive benchmark with only one phrasing reports 100% accuracy.

The registration flow, done directly from the Benchmarks tab of the Genie Agent:

Question 1: "What was total revenue for the last closed quarter?"
Question 2: "How much did we bill in Q3?"
Question 3: "Last quarter's revenue, what was it?"

SQL Answer (same for all 3):
SELECT SUM(order_amount) AS total_revenue
FROM sales.orders
WHERE fiscal_quarter = (
  SELECT MAX(fiscal_quarter) FROM sales.orders
  WHERE closed_date IS NOT NULL
)

 

The official recommendation is two to four phrasing variations per real business question. Running this set after any change to a Metric View or to an underlying term definition quickly shows whether the change broke one specific phrasing without breaking the others, a signal that would normally go unnoticed until a real user runs into it.

The loop that closes itself: from benchmark back to agent context

The real payoff isn't the isolated accuracy number, it's what happens after a benchmark run. Reviewing question by question doesn't scale, so the recommended flow uses Genie Code itself to analyze the entire run at once: it reviews what was expected, what the agent actually generated, and the agent's current context, then proposes instruction or context adjustments for each gap found, for whoever manages the agent to accept or reject individually.

That same "batch review via Genie Code" pattern also applies to real usage, not just formal benchmarking: the agent's Monitor tab brings a weekly digest of message volume, active users, and positive/negative feedback rate, and the "Analyze Agent Usage" button launches Genie Code to comb through six months of real messages looking for recurring topics and repeated problems, with citations linking straight back to the original conversation. In practice, this means the signal for "where the agent is getting it wrong" comes from two complementary sources: the controlled benchmark, designed to cover what the team already knows matters, and real usage, which reveals what nobody thought to test.

What this doesn't solve

User feedback alone doesn't change agent behavior automatically, the documentation is explicit about this: someone with manage permission needs to review the feedback and decide whether it becomes a context adjustment or not, so a benchmark without periodic human review is still a mechanism gathering dust. Chat mode also depends entirely on the quality of the registered ground-truth SQL, a question with no SQL Answer falls into mandatory manual review, and a wrong ground-truth SQL silently teaches the agent to be wrong the "right" way. And the detailed result of an individual evaluation run stays visible for only one week, so anyone wanting to track an accuracy trend over months needs to export or log the number somewhere external, the tool doesn't keep long-term history on its own.

Is it worth investing in this from day one?

Registering benchmarks before the agent even reaches production can feel like redundant work when the initial demo already convinced everyone, but it's actually the opposite: the benchmark only earns its value later, when the Metric View changes, when a business term gets redefined, when the agent gains a new data source. Without an already-registered set of questions with known-correct answers, each of those changes demands manual validation from scratch. With it, the same question asked last time reruns in seconds, and the answer to "did this break anything" becomes a number instead of a guess.

References

azure databricks
No RepliesBe the first to reply