Forum Discussion
Weird problem when comparing the answers from chat playground and answer from api
I'm running into a weird issue with Azure AI Foundry (gpt-4o-mini) and need help.
I'm building a chatbot that classifies each user message into:
- follow-up to previous message
- repeat of an earlier message
- brand-new query
The classification logic works perfectly in the Azure AI Foundry Chat Playground.
But when I use the exact same prompt in Python via:
- AzureChatOpenAI() (LangChain)
- or the official Azure OpenAI code from "View Code" (client.chat.completions.create())
…I get totally different and often wrong results.
I’ve already verified:
- same deployment name (gpt-4o-mini)
- same temperature / top_p / max_tokens
- same system and user messages
- even tried copy-pasting the full system prompt from the Playground
But the API version still behaves very differently.
It feels like Azure AI Foundry’s Chat Playground is using some kind of hidden system prompt, invisible scaffolding, or extra formatting that is NOT shown in the UI and NOT included in the “View Code” snippet. The Playground output is consistently more accurate than the raw API call.
Question:
Does the Chat Playground apply hidden instructions or pre-processing that we can’t see?
And is there any way to:
- view those hidden prompts, or
- replicate Playground behavior exactly through the API or LangChain?
If anyone has run into this or knows how to get identical behavior outside the Playground, I’d really appreciate the help.
2 Replies
I would compare the exact API version, full message array, deployment, temperature, top_p, max_tokens, and any chat history included by the playground. LangChain can also wrap or reorder messages, so test once with the raw Azure OpenAI client only. For classification, use temperature 0, a fixed output schema, and possibly seed for best-effort reproducibility.
The Azure AI Foundry Chat Playground incorporates additional hidden system messages and underlying scaffolding that are not visible in the user interface or included in the “View Code” snippet. These implicit instructions influence and refine the model’s behavior, which accounts for the greater accuracy and consistency often observed in Playground outputs compared to direct API calls. While these hidden prompts cannot be accessed directly, similar results can be achieved by explicitly defining system messages and carefully structuring prompt formatting when working with the API or through LangChain.
Safety system messages - Azure OpenAI in Microsoft Foundry Models | Microsoft Learn
