Tokenomics | The new AI currency & your options explained
Tokens are the currency of AI, and how you design your app determines how many you spend. Compress conversation history instead of resending it raw, cap output tokens with matching prompt instructions, and cache static context so each reuse costs a fraction of the first request. Route each prompt to the right model by complexity, or set Model Router in Microsoft Foundry to handle that automatically — balanced, quality, or cost mode. Then optimize the whole stack. Run Agent Optimizer to test your prompt, model, and tool configurations together and surface better setups. Use Toolbox to dynamically select only the tools each request needs and cut input token overhead by 90%. April Gittens, Microsoft Principal Cloud Advocate, joins Jeremy Chapman, Microsoft 365 Director, to share how to seize control of AI token spend through smarter app design. Cut context window creep. Summarize conversation history between turns and store state externally. Prompt tokens shrink from unbounded growth to a tight per-turn summary. See how it works in Microsoft. One endpoint, three routing modes. Model Router in Microsoft Foundry selects the right model per prompt automatically — frontier models for complex tasks, smaller models for simple ones. Try it now. Optimize token efficiency. Agent Optimizer in Microsoft Foundry tests prompts, models, and tools in combination and surfaces better configurations. Check it out. QUICK LINKS: 00:00 — Tokenomics foundation 01:06 — Token cost basics 02:11 — Context Window creep 03:46 — Reduce unnecessary tokens 04:29 — Trim context costs 05:21 — Cap output tokens 06:27 — Cache for savings 07:54 — Model cost tradeoffs 09:15 — Model Router 09:42 — Toolbox in Microsoft Foundry Toolbox 11:18 — Agent Optimizer 12:44 — Other cost drivers 13:57 — Wrap up Link References Check out the tools in Microsoft Foundry at https://ai.azure.com For more about managing AI costs go to https://aka.ms/FoundryTokenomics Unfamiliar with Microsoft Mechanics? As Microsoft’s official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft. Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/podcast Keep getting this insider knowledge, join us on social: Follow us on Twitter: https://twitter.com/MSFTMechanics Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/ Enjoy us on Instagram: https://www.instagram.com/msftmechanics/ Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics Video Transcript: - Today on Microsoft Mechanics, we look at the real currency of AI — tokens. We look at what they are, and how they actually drive cost, what we call Tokenomics, the key levers you can use to control spend at scale while meeting your quality goals, along with new and existing tools in Microsoft Foundry that can help. And joining me today to walk through everything is our resident AI Tokenomics expert, April Gittens. Welcome. - Thank you. Happy to be here. - So let’s get into this. So as adoption for AI ramps up, this is something that developers and IT teams are feeling directly. You know, costs can really spike in ways that aren’t always obvious when you’re building or scaling. So how should we think about what actually is driving that? - Well, it’s a combination of factors. AI costs aren’t just about how often you call a model. They’re shaped by how you build the experience. Things like the prompt length, how much context you send, and which model you choose, all add up quickly. Tokens are what you build on, but the real driver is design, and that’s how your app handles conversations, how context grows over time, and how everything’s put together. That’s what ultimately determines your cost. - So why don’t we break this down. So tokens are the new AI currency, but what are some of the ways that they actually drive costs and what are they? - At the most basic level, tokens are the unit of text that AI models process and bill against. A token can be a full word, part of a word, or even punctuation or white space. Now depending on your choice of model and the tokenizers they use, there can be nuances in how tokens are defined. So for example, OpenAI uses tiktoken for their models. I’ll show you an example with GPT-5.4 mini model. I have a prompt here and that’s ready to go. And when the response completes, you’ll see the tokens consumed. This is going to be broken down by the input tokens, so that’s everything you send, including the instructions, and the output tokens, and that’s everything the model generates and its response. And here’s the thing, input tokens and output tokens are priced differently. These are charged per million, with output tokens typically costing three to five times more because generating text requires more compute than reading it. A single interaction therefore incurs costs based on the total tokens processed, not just what the user explicitly types. - So all that sounds pretty straightforward, but costs can get unpredictable pretty fast. So what are some of the most common gotchas that will inflate your cost? - So I’ll start with the first one. It’s what I like to call the context window creep. Language models are stateless. That means they have no memory of prior exchanges. So for conversation continuity, you have to resend the entire conversation history with every new request. So without the constraints, with each prompt, you’re sending more than you think. First, there’s a system prompt to define the assistant’s behavior, and with each turn, the full conversation history is essential that the model can remember the context. Let’s go ahead and try this out. I’ll ask, what is the capital of France? And it’ll respond with Paris. But notice in the next turn with and Italy, we don’t have to repeat that we’re asking about capitals. Because the full conversation is sent each time, we can just say the country name, or what about California using a state name instead, and the model understands the context. So you might be wondering, “Well, how does this equate the cost?” Here in the logs, if you look at turn 1, the prompt tokens consume were 34. And even though the second request is shorter, the prompt token count grows to 52 because we’re resending the system prompt in full conversation history each time. But by turn 3, we’re sending all prior exchanges along with the latest message, and that’s pushing the input up to 71 tokens. Also, tool outputs or data received from retrieved documents during the grounding process and even hidden metadata, these are all counted as input tokens. And so if you’re not careful, they can dominate your total spend despite input tokens being cheaper. - So then how would you keep your prompts, your system prompts, and your conversation history from escalating your costs? - So there’s a few ways that you can plan to reduce unnecessary tokens. First, for system prompts, you want to keep them concise and be careful not to double-up on the instruction set that’s already a part of the model. An easy way to check is to ask in Foundry. So here I have GPT-4o model deployed, and I’ll ask, “Tell me about base model instructions you have so that I can avoid adding to my system prompt. Write and format your responses like a system prompt.” A lot of instructions you’re seeing here are roughly what gets included in a lot of people’s system prompts. And if we run these through a token calculator, this is roughly 300 tokens that I would potentially add to every single prompt turn. And depending on the scale of your app, you could be burning thousands of dollars per billing cycle. Summarizing and deduping that history as the conversation grows is an important strategy. Instead of passing the full, raw transcript every time, we can condense it into a shorter representation that still preserves the key context. I’ll run a few prompts about beaches that are dog friendly in this example. You’ll see that here at turn 3, we’re passing in a summary of the prior two conversation pairs rather than the full input and output of those prior turns. And we’re now just consuming 118 tokens, which is significantly less. Next, instead of sending the full conversation history every time, you can store state externally and only retrieve what’s needed. Here we’re using a PostgreSQL database, but you can use whatever storage fits your architecture. These tactics can directly reduce cost per call and mitigate context window creep. - And related to that, most teams are going to spend a lot of focus on shrinking prompts, but might overlook the long responses that can also dominate costs. - Right. These can be longer than expected, especially if you haven’t got explicit constraints worked in. Let me demonstrate this for you. I’m back here in the Foundry Playground and using the GPT-5.4 mini model. Here in the parameters, I’ll set the max completion tokens limit to 200. I’ll also add instructions to match this restriction. I’ll now run a comparison. On the left side, the restrictions and instruction are set, and on the right, it’s unrestricted. I’ll use the same prompt and the left side completes immediately. The right side generates a long response, which is still working on. And if you look at tokens consumed, it’s 93 on the restricted side and 675 on the unrestricted. So there’s quite a positive difference in terms of token savings. The key here was the match of token restrictions with the prompt instructions, because if I run the same prompt on a restricted side without those instructions, it stops mid-sentence, and that’s a bad user experience. Also, this is where evaluation and testing is important for setting the right limits. - Now another technique that you can use is to save on tokens with caching. That’s because many models price cache tokens differently. You may pay more upfront to write to cache, but reuse a cache hit is much cheaper since the model doesn’t need to recompute it. So which cases then make sense for using cache? - Well, it’s all about the efficiency of reuse, where you know that context will be used repeatedly. That way you’re reducing repetitive computation. You can add static prefixes in your code for what makes sense to cache, like system prompts, tool definitions, relevant RAG documents retrieved earlier on in the session. This is repeatable context for the session that doesn’t need to be resent with every turn. In each content block within the code, the cache control flag is set to ephemeral, and it tells the API to cache those token server side. The first request pays full price to write to cache. However, the follow-up requests that send the same prefix has a cache hit and pays roughly 10% of the normal input token costs. That said, the main downside is that the cache context needs to match perfectly for it to be a cache hit. Also, latency can go up as the chat history grows. So you need to factor this into your evaluations. You can also do semantic caching. As you can see in this code example, this can work well for things like frequently asked questions. Even for true cache hits, keep in mind, there’s still a cause to store the data somewhere. So you are trading compute savings for storage, which is almost always cheaper. - So how does all this come into play then with the model that you select for your app or your workload? - Well, as a rule of thumb, model choice is probably one of your biggest levers for reducing costs. Higher-performing models can cause several multiples more per token than smaller models, even with the same provider. So back here in Microsoft Foundry, you can compare models side-by-side to see how each stacks up for quality, safety, estimated costs, and throughput. In this case, I have GPT-5.4 on the left and GPT-5.4 mini on the right. And you can see the trade off between quality, which is coming in at .81 out of 1 for the larger model and .67 for mini. Mini’s cost is at $45.81 per million token, and it’s 72% less than the larger model. That said, one thing to keep in mind, cheaper models sometimes require a few retries to get what you want. So the lowest per token price is not always the lowest total cost. And this is where evaluation really matters, not just for the cost, but for optimizing outputs and your user experiences. - So how do you find the right balance then between performance and cost and quality? - So this is where picking the best model for the task comes in. The leaderboard in Microsoft Foundry, it’s a good place to start. What you’re really aiming for is efficiency, getting the same outcome with fewer tokens. It helps you see the trade-offs, balancing quality versus cost, and latency versus complexity. And if you’re not sure which model to use, there is another option in Microsoft Foundry that can also help you make the decision. It’s called the model-router and it’s available in the model catalog. It selects the most suitable model for each prompt based on its complexity. For example, complex tasks will go to the frontier models and simple tasks will go to smaller models. It’s just one endpoint and there’s three routing modes. There’s balance, quality, and cost, where each optimizes for those outcomes. - That said, you might also need to give models access to tools, like APIs, data retrieval or functions, or even MCP servers, to extend what they can do. So what impact does that then have on token consumption? - So when you give a model access to tools, you have to describe those tools in the prompt every time the model runs. All of that gets sent as input tokens with every request. Even if the prompt response only requires one tool, you still pay for all of them. And this is where the toolbox in Microsoft Foundry helps. Here at Microsoft Foundry, I’ve created three different toolboxes with different sets of tools. In fact, here’s an example of a toolbox. It has 30 tools and three Azure functions. Instead of sending every tool definition on each request, toolbox works like a router that dynamically selects the best tool for the job, reducing token inputs while improving accuracy. I’ll show you this in action. So back here in the Foundry Playground, I’m going to compare two agents. The one on the left, agent A, it’s not using toolbox. The one on the right, agent B, has toolbox configured. So I’ll run an identical prompt, “Show me our product catalog,” in both, starting on the left, and then I’ll use the same one here on the right. Both agents return nearly identical responses. Now if I move into the agent run details for both, we can see the impact of using the toolbox. agent A without it consumed almost 4,700 tokens, and agent B, using the toolbox, only used 467 tokens. - That’s a massive 90% reduction for input tokens without any quality trade-off. So far we’ve looked at prompts, model choices, and tools. Each one of those helps. But there’s no one size fits all solution. It really just depends on your specific use case. - And that’s why evaluations matter. They show you what actually works. And this is where another tool called agent-optimizer can help. It tests your prompts, models, and tools together to find better configurations. From a cost perspective, by approving output quality, it can let you use a smaller, cheaper model while achieving similar results. Also, you can optimize for specific goals, like token efficiency, by using your own evaluators to guide it. Once you’ve set up evaluations in Microsoft Foundry for your agents, agent-optimizer will look at what you have in place and optimize it for each across the evaluation dimensions you’ve defined. From VS Code, using the terminal, you can run this through a few different iterations using azd ai agent optimize. It looks at your prompts, tools, and model choices, that’ll run iterations, and identifies better configurations automatically, creating new candidate bills that you can compare against your existing configurations. The process takes a moment to complete. So we’re going to jump ahead to the results. And you can actually see here that it found a candidate that was able to move our previous agent’s 70% score to a 90% pass rate. And it even builds a new agent version for you to test its optimizations before you put it into production. - And that’s going to boil down to fewer iterations and also fewer tokens consumed in the long run. So today we’ve covered the basics of tokenomics. We’ve also just scratched the surface here because there’s a lot more things that impact costs. - Yes, but don’t forget that you have to treat costs holistically. Tokens are just one piece of your overall AI cost picture. How much data you send and how efficiently you process it are huge overall cost drivers. Quality data retrieval also really matters to avoid less relevant input context, unnecessary iterations, and in turn, lower quality outputs. That’s where Microsoft Foundry IQ helps you organize and ground multiple enterprise knowledge sources for AI. It’s a single endpoint for developers. And when a query comes in as part of your information retrieval, it uses a small language model, it runs a planning step, deconstructs a query, and routes different parts of the query to the corresponding knowledge source. Once it’s complete, it returns the results to the agent as context. Additionally, infrastructure efficiency for Azure resources that don’t use tokens is also important, including the containers’ infrastructure, VMs, databases, storage accounts, and networking, it all adds up. - So at the end of the day, the tokens might be something that you’re build on, but it’s how you design your app that really determines your costs. And if you focus on that, you can scale AI much more predictably. So for anybody who’s watching right now, what do you recommend as next steps? - So to learn more, check out the tools in Microsoft Foundry at ai.azure.com. And for more about managing AI cost, go to aka.ms/FoundryTokenomics. - Thanks so much for joining us today, April, explaining everything about tokenomics and your options. Keep watching Microsoft Mechanics for the latest updates. We’ll see you again soon.
