Forum Discussion
Microsoft-Decision-1 scores changed on the same deployment / version 1 – anyone else?
We use Microsoft-Decision-1 in Foundry (DataZoneStandard, model version 1, Sweden Central) as a yes/no gate with `noul` questions and a fixed probability threshold.
Between Oct 23 (~15:00 UTC) and Oct 24 (~22:00 UTC) the scores changed for our calibration requests:
- Same deployment, same API key, no changes in the activity log.
- We had this running for pretty much since the announcement and probabilities for the same requests where near identical.
- Now: Repeated calls of the same query can vary a lot (0.915 vs 0.932 seconds apart).
- Re-scoring 10,000 stored requests: Results are all over the place.
- Most low scores barely move, but clear positives dropped (0.99 -> 0.78) and some negatives rose (0.00 -> 0.90).
- Pinning the deployment (versionUpgradeOption = NoAutoUpgrade) changed nothing.
The model now seems to read the instructions much more literally: removing one qualifying word from our context note moved a clear positive from 0.92 to 0.996. Did anyone else see Decision-1 probabilities shift around Oct 23/24? Has Microsoft said whether updates (e.g. the announced rebase onto MAI/OpenAI base models) can happen behind "version 1"?
2 Replies
Hi ChrisOezbek
One thing you can check yourself before the support case: the response has a model field that names the underlying model, separate from your deployment name. If you stored full responses, compare that field for calls before Oct 23 and after Oct 24. A difference is solid evidence to attach for support. I don't know whether it carries a build or version detail, so treat a match as inconclusive. If you only kept the scores, start logging that field with a UTC timestamp now.
Since one removed word moved a clear positive from 0.92 to 0.996, the question is leaning hard on its wording. If you haven't yet, add a criteria block with explicit true and false conditions to the noul question. Then rebuild the threshold from a labeled sample instead of the old calibration. The docs say to validate thresholds on labeled examples, and a middle band that goes to review instead of one hard cut-off would be my own suggestion for a gate like yours.
The reversals across your 10,000 stored requests deserve investigation separately from the smaller variation between repeated calls. A fixed threshold that worked before should not be assumed valid for the changed score distribution.
Microsoft's launch announcement discusses future rebasing, but it does not confirm that your deployment changed behind version 1 on October 23–24. I cannot establish that cause from the public information. Azure's activity log records resource-management operations, so an unchanged log does not by itself explain inference behavior.
Preserve representative before-and-after examples with byte-identical state, instructions and criteria, the API version, returned model name, UTC timestamps and request identifiers. Submit those through Azure Support with the Sweden Central deployment details. Keep the unchanged examples distinct from tests where you removed a qualifying word. Microsoft's Decision-1 guidance recommends explicit true/false criteria and validating thresholds against labeled workload examples; it does not establish that NoAutoUpgrade restores your previous calibration.
