Many eval questions have a short, fixed answer. Did the agent follow the policy? Did it complete the task? Does the response meet the rubric criterion? A generative LLM judge writes a paragraph to answer questions like these. Often you only need the answer and how sure the judge is. You can now use Microsoft-Decision-1, Microsoft's decision model, as the judge in Foundry Evaluations, in public preview. It returns a probability for every possible answer, and the evaluator turns it into the score and verdict you already use. In our tests it was much faster and cheaper than all 12 chat judges we compared: a median of 0.59 seconds and $0.12 per 1,000 evaluator calls, against 2.1 to 9.0 seconds and $0.70 to $34.39.
Authors: Charles Kim and Ahmad Qardahji
Why this matters
Many eval questions have a short, fixed answer. Did the agent follow the policy? Did it complete the task? Does the response meet the rubric criterion? A generative LLM judge writes a paragraph to answer questions like these. Often you only need the answer and how sure the judge is.
You can now use Microsoft-Decision-1, Microsoft's decision model, as the judge in Foundry Evaluations, in public preview. It returns a probability for every possible answer, and the evaluator turns it into the score and verdict you already use. In our tests it was much faster and cheaper than all 12 chat judges we compared: a median of 0.59 seconds and $0.12 per 1,000 evaluator calls, against 2.1 to 9.0 seconds and $0.70 to $34.39. Results vary a lot by evaluator, so this post shows the numbers for each one, what the probability threshold changes, and how to try it on your own data.
What a decision model is
Microsoft-Decision-1 is Microsoft's decision model in the Foundry model catalog, in public preview since 9 October 2026. It's built on Qwen3.5-9B and post-trained by Microsoft. Given an input and a question with a fixed set of answers, it returns a probability for each answer instead of generated text. It doesn't write explanations.
Generative models write text. A decision model picks from a fixed list and tells you how probable each answer is. Microsoft teams have used it for this kind of work: Xbox Research sorted more than 10,000 pieces of open-ended feedback into themes that researchers defined, and the Copilot team tested it for judging the quality of chat and agent responses.
That's why it fits evals. Most eval checks are a decision: pass or fail, or a level on a scale.
What a decision model gives you
We ran Decision-1 and 12 chat judges behind the same evaluators on the same labelled checks. Here's what we measured, and what to keep in mind:
| Benefit | What we measured | Keep in mind |
|
Much faster |
A median of 0.59 seconds per call, against 2.14 to 8.97 seconds for the chat judges, which took 3.6 to 15 times as long. It returns about 2 output tokens per call; the chat judges wrote 170 to 440. |
We timed the same 60 coherence and task completion checks for every judge, sent one at a time to separate deployments, so treat the ratios as rough. Other Microsoft posts report larger speedups on other tasks. |
|
Much cheaper |
$0.12 per 1,000 evaluator calls, against $0.70 to $34.39 for the chat judges, which cost 5.7 to 282 times as much. |
Estimated from the tokens each call used, at the list price of the deployment type we used for each judge, with no cached-input discount. Your cost depends on your data and deployments. |
|
Fewer changed verdicts on a rerun |
When we ran the same call twice on 942 internal checks, Decision-1's verdict changed on 17 (1.8%), against 33 for GPT-5.5, 43 for GPT-4.1 and 47 for GPT-4.1 mini. |
We didn't rerun the newer chat judges. Decision-1 isn't deterministic, so repeat borderline checks when the result matters. |
|
A probability with every verdict |
Each result includes the model's probability for every possible answer. Set how strict a pass is with |
The probability isn't the chance that the verdict is right. Pass or fail verdicts near the cutoff were right about half the time: 51% of 241 checks within 0.15 of 0.75, against 74% for the rest. |
|
Strictness you can set |
At the 0.75 default it caught 78% of failing responses and passed 70% of good ones. It caught more failing responses than 10 of the 12 chat judges. At 0.5 it caught 69% and passed 79%. |
Strict isn't the same as accurate. At 0.5, Decision-1 passed about as many good responses as GPT-6.1 Sol, GPT-5.5, GPT-5.6 Luna and GPT-5.6 Sol (79% against 78% to 80%, on the checks each shared with it) but caught fewer failing ones (69% against 75% to 78%). |
|
Short, typed answers |
Decision-1 answers with typed values instead of text, so there's nothing to parse. Our evaluators read every answer it returned. |
Most chat judges also returned almost no answers our evaluators couldn't read. The exceptions were Grok 4.1 Fast (52 answers) and gpt-oss-120b (165 of 166 in a first test on fluency). |
The trade-offs: Decision-1 doesn't write a reason for its verdict, and it was less accurate than most of the chat judges we compared (see How we validated it).
How a Decision-1 evaluation works
When you choose a Decision-1 deployment as the judge, the evaluator doesn't ask for a written answer. For an individual metric such as coherence, the evaluator sends your data with two typed questions in one call: whether the metric applies, and its grade. Composite evaluators in the evaluator catalog, such as output quality, and rubric evaluators combine the questions for several metrics or criteria into the same request. For each metric:
- Does the metric apply? If the probability of yes is under 0.5, the result is reported as not applicable and isn't scored. In our tests this skipped 2% of all calls (385 of 19,733), but it varied by evaluator: 29% of single-turn groundedness checks (12 of 41).
- The grade: a yes or no question for pass or fail evaluators, or a 1 to 5 scale for scored evaluators.
Decision-1 answers each question with probabilities, and the evaluator turns them into the result:
| Question type | What Decision-1 returns | How the evaluation uses it |
|
Yes or no (noul), for example "Did the agent follow the instructions?" |
The probability that the answer is yes. |
For applicable inputs, pass or fail evaluators assign 1 (pass) when the probability meets the configured threshold, 0.75 by default, and 0 (fail) otherwise. |
|
Ordered score (score), for example rating coherence on a five-level scale |
A probability for each level and a native score calculated as the probability-weighted average of the zero-based level indexes. |
Scale evaluators select the most probable level, rather than using the native weighted-average score. They map it to the evaluator's scale, such as 1 to 5, and apply its pass threshold. Ties select the lower level. |
|
Category choice (choice), for example "Should this ticket go to billing, technical support, or account support?" |
The selected category and a probability for every category. |
The built-in evaluators covered here don't use this question type. You can use it directly through the model API for classification, routing, or custom evaluations. |
Rubric evaluators combine individual criterion scores into an overall result according to the rubric's scoring configuration.
The probabilities help in three ways:
- You see how sure the model was, so a pass at 0.95 and a pass at 0.76 look different.
- You control how strict the evaluation is. Raise
probability_thresholdto require more confidence before a result passes, or lower it to be less strict. - You can find results to review. Verdicts near the cutoff are the least reliable (see the table above), so review those first. On 1 to 5 scales, look for results whose probability is split across the pass mark: when the chance of a passing level (3 or higher) was between 0.4 and 0.6, 43% of the verdicts were right, against 84% when the split was clear.
There is one trade-off. Decision-1 doesn't explain its verdict. The reason field names the rule that produced the result, for example "Selected coherence level 4 from the highest-probability level. No model rationale is available." If you need a written reason for each result, use an LLM judge for that evaluator. You can use both types of judge in the same project.
Where Decision-1 fits in the improvement loop
Foundry observability works as a loop: trace your agent, evaluate it, monitor it in production, and optimize it. Decision-1 is a judge, so it fits where your evaluators run.
- Evaluate. Use Decision-1 for pass or fail and scale checks during development and in your CI/CD pipeline.
- Monitor. Continuous evaluation scores a sample of your production traffic, and Decision-1's low cost per call lets you score more of it.
- Optimize. We recommend an LLM judge for the evaluators that drive Agent Optimizer. The optimizer passes each judge's reasons to its reflection model. Decision-1 doesn't write reasons, so the optimizer would get only its scores.
What you can do in the preview
- Choose a Microsoft-Decision-1 deployment as the judge in cloud evaluations, continuous evaluation and the Foundry portal, and in the Azure AI Evaluation SDK (azure-ai-evaluation 1.18.12 or later) on your own computer or in your CI/CD pipeline.
- Run 19 built-in evaluators with Decision-1: coherence, fluency, groundedness, relevance, retrieval, response completeness, similarity, intent resolution, task adherence, task completion, tool call accuracy, tool call success, tool input accuracy, tool output utilization, tool selection, customer satisfaction and deflection rate, plus the output quality and tool use quality composites. We report accuracy for 11 of them (see How we validated it).
- In the SDK (release notes), 10 of them are public evaluator classes: coherence, fluency, groundedness, relevance, retrieval, response completeness, similarity, intent resolution, task adherence and tool call accuracy. QAEvaluator, which runs several of them together, accepts a Decision-1 judge too. Run the others in a cloud evaluation.
- Run your rubric evaluators with Decision-1. The task-level rubric evaluator doesn't use it.
- See the score or verdict, and the probabilities behind it, for each result.
- Set
probability_thresholdon pass or fail evaluators. Scale evaluators keep the thresholds you use today. - See which model and deployment judged each result, plus token usage.
- Keep your current judges. LLM judges work the same as before.
How we validated it
We ran Decision-1 and 12 chat judges behind the same built-in evaluators from the Foundry evaluator catalog, the code that the Foundry evaluation service runs, on 2,621 labelled checks from 11 benchmark sets, at default settings and with no threshold tuning. We measured 11 evaluators. Six of them have public SDK classes: coherence, fluency, groundedness, intent resolution, task adherence and tool call accuracy. The other five, task completion, tool call success, tool input accuracy, tool output utilization and tool selection, run in cloud evaluations. Accuracy in this post is how often a judge's pass or fail verdict matched the reference label. For scale evaluators, that's pass or fail at the evaluator's threshold, not the exact 1 to 5 score. The 95% intervals resample whole conversations. Each covers one comparison, and we made many, so treat results close to zero as uncertain. How we measured, at the end of this post, has the details.
Figure 3 shows three chat judges next to Decision-1. Foundry's docs recommend the GPT-5.6 family as the judge for composite evaluators and name GPT-5.6 Luna as the low-cost option. GPT-6 Luna is the newest low-cost model we tested, and GPT-4.1 is an older judge.
- Against GPT-4.1, the result depends on how you count. On the 2,509 checks both scored, Decision-1 was right 74% of the time and GPT-4.1 72%. The difference, -2.2 points for GPT-4.1 in the table below, has an interval that includes zero (-4.4 to +0.3). Decision-1's lead comes from task completion, 31% of the checks, where GPT-4.1 was the least accurate of the 12 chat judges (62% against 77% for Decision-1). Without task completion, or counting each evaluator equally, GPT-4.1 was 3.6 points ahead, and it was ahead on 8 of the 11 evaluators. Decision-1 is the stricter judge: it caught 78% of the failing responses, against 54% for GPT-4.1, and passed 70% of the good ones, against 86%.
- By evaluator. On the 2,470 checks that Decision-1, GPT-5.6 Luna, GPT-6 Luna and GPT-4.1 all scored, Decision-1 had the highest accuracy of the four on coherence and fluency, and the lowest on 8 of the other 9 evaluators. On task completion it was ahead of GPT-4.1 and behind GPT-5.6 Luna and GPT-6 Luna. The largest gaps were on tool selection, tool input accuracy and tool output utilization, for example tool selection at 71% against 90% to 93%. Every judge scored between 46% and 57% on task adherence, so its results on these sets say little about any judge. Several evaluators have fewer than 100 checks, so small differences there aren't meaningful (Figure 3).
- Rubric evaluators. These run in cloud evaluations. With a 7-criterion quality rubric and a 6-criterion tool rubric, Decision-1 gave a verdict on 79% of labelled criteria and each chat judge on 99% or more. The 544 criteria Decision-1 left without a verdict, most of them marked not applicable, weren't a random sample: 395 were task completion criteria (50% of the 795 it was asked to grade), and 420 were labelled as failing. On the criteria both judges scored, Decision-1's verdicts matched the labels 73% of the time, against 75% for GPT-6 Luna, 74% for GPT-5.5, 73% for gpt-oss-120b, 72% for Grok 4.1 Fast, 71% for GPT-4.1 and 68% for GPT-4.1 mini. Counting every criterion without a verdict as wrong, Decision-1 drops to 58% and GPT-4.1 to 65%. If a criterion must always be graded, such as whether the task was completed, set
always_applicableto true in your rubric so the judge can't skip it (Figure 4). - Single-turn checks. On the internal single-turn quality and tool-use sets alone, counting each evaluator equally, Decision-1 was right 75.7% of the time, against 85.9% for GPT-4.1, 83.6% for GPT-5.6 Luna and 83.1% for GPT-6 Luna, on the 469 checks all four scored. A separate Microsoft evaluation of those sets found a similar gap. Which sets you weigh, and how, changes the headline.
These are early results on our benchmark sets. Treat the numbers as directional, and confirm them on your own data before you switch judges.
How Decision-1 compares with other models
The table compares each chat judge with Decision-1 on the labelled checks both scored. The difference columns are the chat judge's accuracy minus Decision-1's, in percentage points, with 95% intervals from resampling conversations. The first counts every check, and the second counts each evaluator equally. A positive number means the chat judge was more accurate.
| Judge | Checks | Accuracy (Decision-1 on the same checks) | Difference, all checks | Difference, each evaluator equal | Failing caught | Good passed | Cost per 1,000 calls | Seconds per call |
|
Microsoft-Decision-1 |
2,513 |
74% |
|
|
78% |
70% |
$0.12 |
0.59 |
|
GPT-6.1 Sol |
2,497 |
78% (74%) |
+4.2 (+2.8 to +5.8) |
+4.6 (+2.4 to +7.0) |
78% |
78% |
$11.89 |
5.59 |
|
GPT-5.5 |
2,513 |
78% (74%) |
+4.1 (+2.6 to +5.7) |
+5.3 (+2.9 to +7.7) |
76% |
79% |
$34.39 |
4.98 |
|
GPT-5.6 Sol |
2,510 |
78% (74%) |
+3.9 (+2.3 to +5.5) |
+4.8 (+2.5 to +7.5) |
75% |
80% |
$25.45 |
5.84 |
|
GPT-5.6 Luna |
2,489 |
78% (74%) |
+3.9 (+2.3 to +5.6) |
+5.5 (+2.9 to +8.3) |
75% |
80% |
$1.36 |
2.86 |
|
Grok 4.1 Fast |
2,453 |
77% (74%) |
+3.3 (+1.4 to +5.4) |
+5.3 (+1.7 to +8.7) |
63% |
89% |
$0.96 |
8.97 |
|
GPT-6 Luna |
2,479 |
77% (74%) |
+3.2 (+1.6 to +4.8) |
+5.0 (+2.4 to +7.5) |
75% |
78% |
$0.70 |
3.51 |
|
DeepSeek V3.2 |
2,500 |
76% (74%) |
+2.4 (+0.6 to +4.1) |
+4.2 (+1.8 to +6.7) |
74% |
78% |
$2.87 |
2.32 |
|
Mistral Large 3 |
2,098 |
76% (78%) |
-2.2 (-4.5 to -0.1) |
+1.3 (-1.5 to +4.2) |
63% |
87% |
$2.65 |
4.92 |
|
GPT-4o |
2,476 |
74% (74%) |
+0.1 (-1.9 to +2.2) |
+1.6 (-1.0 to +4.5) |
63% |
84% |
$13.27 |
2.14 |
|
Llama 4 Maverick |
2,504 |
73% (74%) |
-0.2 (-2.4 to +2.0) |
+2.0 (-1.1 to +5.1) |
57% |
88% |
$1.23 |
2.56 |
|
GPT-4.1 |
2,509 |
72% (74%) |
-2.2 (-4.4 to +0.3) |
+3.6 (+1.0 to +6.4) |
54% |
86% |
$11.14 |
6.18 |
|
GPT-4.1 mini |
2,512 |
69% (74%) |
-5.0 (-7.6 to -2.4) |
+0.5 (-2.6 to +3.6) |
39% |
94% |
$2.20 |
2.15 |
Failing caught is the share of failing responses the judge failed, and good passed is the share of good responses it passed. Mistral Large 3 scored fewer checks (2,098) because its deployment rejects a parameter that our task adherence request sends, so compare it with Decision-1's 78% on those checks, not with the other judges. Its evaluator average leaves task adherence out.
GPT-6.1 Sol, GPT-5.5, GPT-5.6 Luna and GPT-5.6 Sol were the most accurate, 3.9 to 4.2 points above Decision-1 on the checks each shared with it, and the differences among them are small. With each evaluator counted equally, GPT-5.6 Luna had the largest gap (+5.5 points), and it costs $1.36 per 1,000 calls, against $11.89 to $34.39 for the others.
Results at other probability thresholds
Pass or fail evaluators turn Decision-1's probability into a verdict with a threshold, 0.75 by default. We rescored the same answers at thresholds from 0.3 to 0.9. Scale evaluators don't change, because they use the most probable level.
| Threshold | Accuracy | Failing responses caught | Good responses passed |
|
0.3 |
74.4% |
61% |
86% |
|
0.5 |
74.2% |
69% |
79% |
|
0.6 |
74.3% |
73% |
75% |
|
0.75 (default) |
73.8% |
78% |
70% |
|
0.9 |
73.2% |
83% |
65% |
- Overall accuracy hardly moves: it stays between 73.2% and 74.4% at every threshold from 0.3 to 0.9, on 2,513 checks. Only the 1,514 checks from pass or fail evaluators change; the 999 from scale evaluators don't. At every threshold, its accuracy over all checks stayed within 3 points of GPT-4.1's and 3.7 to 4.8 points below GPT-6.1 Sol's, on the checks each shared with Decision-1.
- The threshold changes which mistakes you get. At 0.75 Decision-1 caught 78% of failing responses and passed 70% of good ones. At 0.5 it caught 69% and passed 79%. At 0.9 it caught 83% and passed 65%. On the pass or fail checks alone, failing responses caught rises from 57% at 0.3 to 89% at 0.9, and good responses passed falls from 87% to 48%.
- Some evaluators do better at another threshold. Tool selection went from 71% at 0.75 to 88% at 0.3, and tool output utilization from 78% to 87%. Task completion hardly changes, and tool call success stays between 96% and 99% up to 0.8, then drops to 93% at 0.9. These tool evaluators have only 76 to 82 labelled checks each, and we picked the best threshold on the same checks.
- A separate threshold per evaluator gained under a point when we chose it on four fifths of the conversations and tested it on the rest (74.3%, against 73.8% at the default). It lost about 2 points when we chose it on the other benchmark sets and tested it on a new one (71.8%).
Our advice: keep 0.75 if a missed failure costs you more than a false alarm. Lower it, for example to 0.5, if you'd rather pass borderline good responses. If you set a threshold per evaluator, choose it on your own labelled data and check it on conversations you didn't use to choose it. You can set it for each pass or fail evaluator in a cloud evaluation, and on task adherence in the SDK (see Set up a Decision-1 judge).
What it costs
Decision-1 is billed per token. Its list price is $0.042 per 1 million input tokens, the same in all 37 regions in the Azure retail price list (meter "Microsoft Decision 1 1M Tokens", effective 1 October 2026), and Microsoft's launch post says output tokens are free. Decision-1 returns typed answers rather than text, so output is only a few tokens per call anyway. Our estimates price them at the input rate, which doesn't change any figure in this post.
A worked example: in our tests, an evaluator call on the labelled checks used about 2,900 Decision-1 tokens on average. Scoring 10,000 rows with one evaluator is about 29 million tokens, or about $1.22 at list price.
Chat judges cost more because their tokens cost more, they read a longer prompt (about 4,200 to 4,600 tokens per call, against 2,900), and they write more: a reason with every score, plus hidden reasoning for the reasoning models (about 170 to 440 output tokens per call, against 2). Figure 7 shows the estimated cost per 1,000 evaluator calls and the median time per call for every judge.
Cost is an estimate from the tokens each call reported, at list prices, and time is the median of 60 calls sent one at a time. How we measured has the details.
Set up a Decision-1 judge
1. Deploy Microsoft-Decision-1
- Open the Foundry portal and go to your project.
- Select Model catalog, then search for and select Microsoft-Decision-1.
- Select Deploy, configure the deployment, and select Deploy.
- On the deployment details page, copy the resource endpoint and the deployment name.
You need the Cognitive Services Contributor role to deploy models. You can also deploy with the Azure CLI:
az cognitiveservices account deployment create \
--name <ACCOUNT_NAME> \
--resource-group <RESOURCE_GROUP> \
--deployment-name <DEPLOYMENT_NAME> \
--model-name "Microsoft-Decision-1" \
--model-format Microsoft \
--model-version "1" \
--sku-name GlobalStandard \
--sku-capacity 50
Each unit of capacity allows 1 request and 1,000 tokens per minute. An evaluator call used about 2,900 tokens in our tests, so 50 units handle about 17 calls a minute. Set the capacity for the number of calls you plan to send, within your quota. The example uses GlobalStandard. To keep processing in the data zone closest to your application, use --sku-name DataZoneStandard where it's available.
2. Run an evaluation with the SDK
Install azure-ai-evaluation 1.18.12 or later:
pip install "azure-ai-evaluation>=1.18.12" azure-identity
Set environment variables for the resource endpoint (not the project endpoint) and the deployment name that you copied, and sign in with the Azure CLI so that DefaultAzureCredential can use your identity. Your identity needs the Foundry User role on the Foundry resource. A role on the project alone isn't enough.
export AZURE_ENDPOINT="https://<resource>.services.ai.azure.com"
export DECISION_DEPLOYMENT_NAME="<your-deployment-name>"
az login
Save these two lines as data.jsonl in the folder you run the code from. Each line is a JSON object with query and response fields:
{"query": "How do I reset my password?", "response": "Open Settings, select Security, then select Reset password and follow the email link."}
{"query": "Can I change my plan?", "response": "Yes. Go to Billing, select Change plan, and choose a new plan."}
Then point an evaluator at your deployment:
import os
from azure.identity import DefaultAzureCredential
from azure.ai.evaluation import CoherenceEvaluator, evaluate
credential = DefaultAzureCredential()
# Use your Microsoft-Decision-1 deployment as the judge.
decision_judge = {
"type": "decision",
"azure_endpoint": os.environ["AZURE_ENDPOINT"], # https://<resource>.services.ai.azure.com
"azure_deployment": os.environ["DECISION_DEPLOYMENT_NAME"], # your deployment name
}
coherence = CoherenceEvaluator(model_config=decision_judge, credential=credential)
# Score one row.
print(coherence(
query="How do I reset my password?",
response="Open Settings, select Security, then select Reset password and follow the email link.",
))
# Score a dataset: a JSONL file with query and response fields.
results = evaluate(data="data.jsonl", evaluators={"coherence": coherence})
Here's part of what the one-row call printed in our run, with the probabilities rounded:
{'coherence': 4.0,
'coherence_result': 'pass',
'coherence_reason': 'Selected coherence level 4 from the highest-probability level. No model rationale is available.',
'coherence_threshold': 3,
'coherence_properties': {'decision_model': {
'probabilities': {'0': 0.002, '1': 0.018, '2': 0.056, '3': 0.874, '4': 0.049},
...},
...},
'coherence_prompt_tokens': 269,
'coherence_completion_tokens': 2,
'coherence_model': 'microsoft-decision-1',
...}
The probabilities are keyed 0 to 4 for levels 1 to 5, so '3' is level 4, the most probable level and the score. The result passes because 4 is at or above coherence's threshold of 3.
Task adherence is a pass or fail evaluator, so you can also set its probability threshold:
from azure.ai.evaluation import TaskAdherenceEvaluator
# Require a probability of at least 0.6 to pass (default 0.75).
adherence = TaskAdherenceEvaluator(
model_config=decision_judge, credential=credential, probability_threshold=0.6
)
3. Run it in Foundry
Cloud evaluations use the Microsoft Foundry SDK (azure-ai-projects 2.2.0 or later). To create a project client and submit a run, see cloud evaluation. In the testing criteria, pass your Decision-1 deployment name as the evaluator's model, the same way you choose any judge:
testing_criteria = [
{
"type": "azure_ai_evaluator",
"name": "coherence",
"evaluator_name": "builtin.coherence",
"initialization_parameters": {"model": "<your-decision-1-deployment>"},
"data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
}
]
This fragment sets the judge for one evaluator. The rest of the request is the same as for any cloud evaluation of a dataset. For a pass or fail evaluator, such as task completion, you can also add probability_threshold to initialization_parameters.
In the Foundry portal, select your Decision-1 deployment as the judge model when you create an evaluation.
4. Review and tune
Run Decision-1 next to your current judge on a sample of your data before you switch, and look at the results where they disagree. For each pass or fail evaluator, choose probability_threshold based on which mistake costs you more (see Results at other probability thresholds), and check it on labelled examples you didn't use to choose it.
What to know in the preview
- Only the evaluators listed above can use a Decision-1 judge. Risk and safety evaluators, text-overlap metrics such as F1, BLEU and ROUGE, audio evaluators and the task-level rubric evaluator don't use it.
- Decision-1 reads text and JSON, not images, audio or video. Your input and any one question must fit in the model's 32,768-token context window, and your input and all the questions in a request must fit in the model's request limit. A request that's too long fails with an error whose properties name the reason, for example
context_window_exceeded. In our tests, 47 of 14,491 first-run calls, labelled or not (0.3%), were too long. - Use azure-ai-evaluation 1.18.12 or later. Version 1.18.11 is the first release whose evaluator classes accept a Decision judge, and 1.18.12 adds a readable reason when a result isn't applicable.
- In the SDK, evaluators that accept a conversation score each turn separately and aggregate the results. Cloud evaluations use the evaluator catalog's whole-conversation rubrics, which is what we measured, so SDK results on conversations can differ from ours.
- Decision-1 isn't deterministic: about 2 in 100 verdicts changed when we ran the same call twice.
- There are no written reasons. Features that learn from the judge's explanations, such as Agent Optimizer, get only scores from Decision-1, so we recommend an LLM judge for them.
- This preview supports Microsoft-Decision-1. The Azure OpenAI Decisions API isn't supported as a judge yet (see What's next).
- Evaluation runs in network-isolated (VNet) projects don't support a Decision-1 judge in this preview.
- Test on your own data first. Run Decision-1 next to your current judge on a sample, and compare the results before you switch.
What's next
These are the areas we're working on or exploring. They aren't commitments, and we'll share results as we measure them.
- The Azure OpenAI Decisions API as a judge. Azure OpenAI has a Decisions API,
/openai/v1/decisions, that returns typed answers (yes or no, a choice, or a score) from gpt-6-luna, its only supported model today, and accepts images as well as text. It's rolling out to regions where gpt-6-luna is available, except East US 2. We're working on adding it as a second decision judge, so the same evaluators can run on either model. We haven't measured it yet. As an ordinary chat judge, GPT-6 Luna was right 77% of the time on the checks it shared with Decision-1 (74% for Decision-1), at $0.70 per 1,000 calls. - Decision models for safety evaluators. Foundry's risk and safety evaluators run on hosted evaluation models today. Decision-1's model card lists content-safety flagging, and Microsoft tested it on 5,250 requests across 11 benchmarks covering harmful content, jailbreak attempts and prompt injection. We plan to measure decision models against the current safety evaluators on labelled safety data before we offer them there. The model docs note the risk: used as a safety filter, the model might miss subtle harmful content or flag benign content.
- Decision models in cluster analysis. Evaluation cluster analysis groups your results and uses a model you choose to name the clusters and suggest fixes. A decision model could assign each result to a failure category quickly and at low cost. Cluster analysis also uses evaluator feedback, which Decision-1 doesn't write, so we'll check how it works on results that Decision-1 judged.
- Review queues. We're exploring a way to send verdicts near the cutoff to an LLM judge or a person. The model docs recommend this pattern, and our results support it: pass or fail verdicts near the cutoff were right about half the time.
- Default thresholds for each evaluator. A cutoff chosen for each evaluator helped a little on new conversations from the same datasets and hurt on new datasets, so we'll look for cutoffs that hold up on new data before we change any default.
Tell us what you want to evaluate with a decision model. Share your feedback in the community link below.
How we measured
Checks and labels. The 2,621 labelled checks come from 11 benchmark sets: Foundry's internal quality and tool-use sets, each at turn and conversation level, and seven sets from six public benchmarks (BFCL v4, used for both task completion and tool call accuracy, plus AgentIF, FaithDial, FED, Tau Voice and Tau3). The internal sets were labelled by Foundry's science team, and the public sets use the labels published with each benchmark, turned into pass or fail (on 1 to 5 scales, 3 or higher passes). We report accuracy for 11 evaluators. Decision-1 also ran relevance, customer satisfaction, deflection rate and the output quality and tool use quality composites on these sets, but the sets have no reference labels for those results, so we don't report their accuracy. We didn't run retrieval, response completeness or similarity.
Intervals. The checks come from 1,464 distinct conversations, because the internal sets hold the same conversations at both levels, some benchmarks reuse a conversation, and some inputs were labelled for two evaluators. The 95% intervals resample whole conversations (2,000 draws), and the threshold tests keep copies of the same conversation on one side of each split.
Versions. Decision-1's results come from azure-ai-evaluation 1.18.12, built from source on 8 November 2026 (the same code as the 1.18.12 release on PyPI), and the evaluator definitions as of 8 November 2026. Nine chat judges ran on 27 and 28 October 2026 on azure-ai-evaluation 1.18.10, GPT-6 Luna on 10 November, and GPT-5.6 Sol and GPT-6.1 Sol on 12 November, all with the same code. Every judge ran the evaluator catalog's code, not the SDK's evaluator classes, so fixes to the SDK classes, such as the tool call success fix in 1.18.11, don't affect these runs. We checked the catalog changes between those dates, and none of them changes how a chat judge scores the 11 evaluators compared here. One of them affects Decision-1: its conversation-level task completion rubric now marks a conversation as not applicable, instead of failing it, when no user turn states a request. Decision-1's results include this change.
Judges and sets left out. Phi-4 ran only on a 59-check sample. gpt-oss-120b ran first on fluency, where our evaluators couldn't read 165 of its 166 answers, so it appears only in the rubric results. One internal multi-turn tool set is left out because its conversations are stored as a single string.
Missing verdicts. No judge scored every check. 60 checks got no verdict from any judge, mostly BFCL conversations that the evaluators reject or skip. Beyond those, Decision-1 marked 44 checks not applicable and 4 were too long. The chat judges marked 0 to 42 not applicable, our evaluators couldn't read 52 answers from Grok 4.1 Fast and 1 answer from GPT-4.1, rate limits and connection errors left a few checks without a verdict (GPT-4o 3, GPT-5.6 Luna 3, Grok 4.1 Fast 1), and Mistral Large 3's deployment rejected a parameter that our task adherence request sends (424 checks). Counting every missing verdict as wrong, except those 60, gives the same picture: Decision-1 72.4%, GPT-4.1 71.7%, GPT-6 Luna 75.2%, GPT-5.6 Luna 76.3%, GPT-5.5 77.6%, GPT-5.6 Sol 77.4% and GPT-6.1 Sol 77.2%.
Cost and time. Cost is estimated from the tokens each call reported on the labelled checks each judge ran (failed and retried calls left out), at the East US 2 list price of the deployment type we used for each judge, with no cached-input discount: for example GPT-4.1 on a regional Standard deployment at $2.20 per million input tokens and $8.80 per million output tokens, GPT-6 Luna on Data Zone Standard at $0.11 and $0.55, and GPT-6.1 Sol on Data Zone Standard at $2.20 and $11. Global Standard prices are about 10% lower, and cached input costs less, so chat-judge costs can be lower than shown. Decision-1 stays the cheapest. Time per call is the median of the same 60 checks for every judge (coherence and task completion on the internal quality sets), sent one at a time. The deployments differed in type, size and region: for example, Decision-1 was timed on a dedicated test deployment and GPT-6 Luna in Sweden Central. Timings also vary with load: GPT-4.1's median was 6.18 seconds in this test and 3.14 seconds across its full run. It isn't a production-load benchmark. The cost and time multiples come from unrounded values ($0.1218 per 1,000 calls and 0.5945 seconds for Decision-1).
Related reading
- Deploy and use Microsoft-Decision-1 in Microsoft Foundry
- Azure AI Evaluation client library for Python
- Introducing Microsoft-Decision-1 in Microsoft Foundry for decision and classification workloads
- Introducing Microsoft-Decision-1, our model for fast decision-making
- Use the Azure OpenAI Decisions API
- Evaluate More, Spend Less: Batching Microsoft Foundry Evaluators for Efficient Evaluation
- Which Reflection Model Should Power your Agent Optimizer?
- Improve production agents with closed-loop evaluations in Microsoft Foundry
Get started today
- Start building in Microsoft Foundry: https://ai.azure.com
- Find Microsoft-Decision-1 in the model catalog: https://aka.ms/decision-1
- Read the Microsoft-Decision-1 announcement: https://aka.ms/decision1
- Read the docs: Deploy and use Microsoft-Decision-1, the Azure AI Evaluation SDK, cloud evaluation and Foundry observability.
- Join the community: https://aka.ms/ai/discord