Event banner
Event details
When using Azure OpenAI on your data, do the search terms retrieved from Azure Cognitive Search count toward the number of tokens and thus to the charge for Azure OpenAI? If so, is there any way for customers to predict or estimate the costs of using Azure OpenAI on your data?
- Super helpful, thanks!
robertlee-msftMicrosoft
Adding to Liam's answer, if you are only using vector search to retrieve documents, and not passing those documents to an Azure OpenAI prompt, then the usage of vector search by itself will not result in any calls to Azure OpenAI. Meaning, once you've generated the vector embeddings for your content (which will consume token counts in the embedding model/endpoint), there are no further costs beyond the search service itself. It's only when you pass the retrieved documents where additional costs can come into play.- liamca-msft
Hi Barry, yes, whatever is retrieved from Azure Cognitive Search and injected into the Azure OpenAI Prompt will count towards the token count (and thus the cost to execute that Azure OpenAI request. Pablo created a really good blog post that goes into more details on this. This is ultimately why the relevance of Azure Cognitive Search is so important and why we spend a lot of our time in making sure we can get the best results in the top 3-5 results to help minimize the number of tokens that are required. As for the cost, you will typically be "chunking" your data which should give you a good idea on the cost of any Azure OpenAI request as you can look at the typical number of tokens per chunk x the number of results you add to the prompt.
- the link to Pablo's post seems to not be right?
- Hello - looks like there was a misprint in the URL which I've since fixed. Please try it again and let me know if it works for you.
