GPT Realtime in Production: Which Context Strategy Should You Actually Use?

Are You Running Azure GPT Realtime in Production?

The question that started this:

Are you running Azure gpt-realtime in production? If so, you've already run into the question I want to talk about: how should you manage conversation context across turns?

It sounds like an implementation detail. It isn't. The way you answer it determines whether your per-call cost is 30 cents or 90 cents, whether your contact-center latency stays under two seconds, and whether your customer has to repeat their account number five times on the same call.

I deployed gpt-realtime through Microsoft Foundry and tested seven strategies against the same workload — a 10-turn patient-onboarding conversation for a healthcare support scenario. Same model, same region, same system prompt. Only the strategy changed.

Why context strategy is the lever:

Every turn of a realtime voice call sends audio tokens to the model and gets audio tokens back. What you send on turn N depends on what you've decided to remember from turns 1 through N-1. That decision sets your token volume, and token volume sets your bill.

Azure gpt-realtime pricing on the realtime API: $32/M for audio input, $0.40/M for cached audio input (a ~99% discount), $64/M for audio output. The cache discount is enormous — which means prefix stability matters. The strategies below are essentially seven different answers to what belongs in the prompt prefix on turn N?

The seven strategies

A) Full history. Single persistent WebSocket; the server retains every conversation item. Turn 10 sees turns 1–9 in full. Total: 18,123 tokens. Cache hit rate: 50.5% — a stable growing prefix caches beautifully. Perfect memory, simplest code. Worst tail latency though — turn 6 took 6.3 seconds as the prefix grew. And one dropped WebSocket loses the entire call's context.

B) Stateless. Fresh WebSocket per turn. Model sees only the system prompt and the current question. Total: 6,557 tokens — the floor. Cache hit rate was low. Zero conversational memory — usable only when each turn stands alone.

C) Sliding window. Fresh connection per turn, but inject the last 2 Q+A pairs. Turn 5 sees turns 3+4. Total: 8,655 tokens. Short-term coherence works ("yes, that one") but turn-1 details are gone by turn 5. Cache rate drops to 20.4% — the moving window keeps breaking the prefix.

D) Compression every 5 turns. gpt-realtime-mini produces a ~70-word summary every 5 turns and injects it into the system prompt; prior individual turns are dropped. Total: 8,660 tokens. Cache rate: 7.1% — each summary refresh invalidates the prefix. Semantic long-term memory survives (account number, original complaint). Our 10-turn test only fires one compression event; the architecture is designed for 30-turn calls it doesn't get to prove itself on here.

E) Sliding + Compression. The hybrid. Compressed summary of old turns in the system prompt, plus the last 2 turns verbatim. Total: 7,975 tokens — the tightest of any memory-preserving strategy. Both timescales covered: "earlier you mentioned..." (summary) and "wait, what did you just say?" (window). Cache rate 10.4%. My production pick for contact centers that tolerate reconnects.

F) Server-side truncation. The "let the API handle it" strategy. Single persistent WebSocket like Mode A, but you pass a truncation config asking the server to drop oldest items when post-instructions context exceeds 8,000 tokens, retaining 80% of the window. No client-side history code at all. it finished last. Total: 18,247 tokens. The 8K threshold never fired on a 10-turn workload — context stayed under it — so Mode F behaved identically to Full History but with slightly more overhead. When it does fire on a longer call, the cache breaks at the trigger turn and savings are partial. You also cannot tune what gets dropped; the server picks oldest indiscriminately. Useful as a safety net layered on another strategy, not as a primary pattern.

G) In-session delete. This is the production-voice strategy. Single persistent WebSocket stays alive the entire call — that is the critical property. After each turn, the client checks response.done → usage.total_tokens. When it exceeds a trigger (we used 600), the client (1) summarizes the oldest turns into a bullet list, (2) calls conversation.item.delete on each superseded item server-side, and (3) inserts the summary as a system message — all without dropping the WebSocket. The last two turns are always kept verbatim. This mirrors the OpenAI cookbook pattern. Total: 14,721 tokens. Cache rate: 23.9%. 

Volume is the lever, but production fit isn't volume:

Ranked by total tokens: B (6,557) → E (7,975) → C (8,655) → D (8,660) → G (14,721) → A (18,123) → F (18,247).

Look at A and F — the two highest cache hit rates (50.5%, 46.3%) — and they're the two most expensive. Cache rate is a vanity metric. A 50% discount on 18K tokens still costs more than a lower discount on 7K tokens. The number that pays the bill is total tokens.

But Stateless wins on volume because it isn't doing the hard thing. It has no memory. It's the right pattern when each turn stands alone — a pharmacist's voice-driven drug lookup, a medication-adherence check-in, a pharmacy IVR replacement, anywhere the workflow holds the state and the LLM is just the voice interface.

Three takeaways from this benchmarks are:  1) optimize for total tokens, not cache hit rate. 2)  cache rate is a vanity metric; a high rate on a big context is worse than a low rate on a small one. 3) pick the cheapest strategy that meets your constraints, not the cheapest overall. Live voice needs G. Lookups want B. Contact centers want E. The leaderboard tells you what's cheapest; your use case tells you what's acceptable.

A few principles hold across most enterprise voice and conversational AI deployments on gpt-realtime.

1. Start with your call shape, not the leaderboard. A 3-turn lookup, a 10-turn support call, and a 30-turn escalation each have different right answers pick the strategy that fits your typical session, not the one that wins a benchmark.
2. Optimize for total tokens. Cache hit rate looks attractive but only discounts the cached portion of the input; total token volume drives both your bill and your latency. A high cache rate on a bloated context is worse than a modest cache rate on a tight one.
3. Treat the system prompt as sacred prefix space. Anything that changes turn-to-turn breaks the cache for everything after it, so put stable content (instructions, persona, long-term summary) first and volatile content (the current turn, sliding window) last.
4. Decide your latency posture early. If you have a strict sub-2-second SLA, reconnect-based strategies are off the table,  you need a pattern that keeps a single WebSocket alive for the whole call. If reconnects are acceptable, you have more options and can usually save tokens.
5. Use server-side truncation as a safety net, not a primary strategy. It guarantees context can't grow unbounded, but with default thresholds it rarely fires on the calls where you'd benefit.
6. Measure before you optimize. Instrument token counts per turn, cache hit rate, and end-to-end latency in production for at least a week before picking a strategy — synthetic benchmarks (including this one) only approximate what real customer conversations look like.

Github:- 

https://github.com/ganachan/gpt-realtime-context-strategies/tree/main