Cut Costs and Speed Up AI API Responses with Semantic Caching in Azure API Management

This article is part of a series of articles on API Management and Generative AI. We believe that adding Azure API Management to your AI projects can help you scale your AI models, make them more secure and easier to manage.

We previously covered the hidden risks of AI APIs in today's AI-driven technological landscape. In this article, we dive deeper into one of the supported Gen AI policies in API Management, which allows you to minimize Azure OpenAI costs and make your applications more performant by reducing the number of calls sent to your LLM service.

How does it currently work without the semantic caching policy?

For simplicity, let's look at a scenario where we only have a single client app, a single user, and a single model deployment. This of course does not represent most real-world use-cases, as you often have multiple users talking to different services.

Take the following cases into consideration: -

• A user lands on your application and sends in a query (query 1),
• They then send the exact same query again, with similar verbiage, in the same session (query 2),
• The user changes the wording of the query, but it is still relevant and related to the original query (query 3)
• The last query, (query 4), is completely different and unrelated to the previous queries.

In a normal implementation, all these queries will cost you tokens (TPM), resulting in higher cuts in your billing. Your users are also likely to experience some latency as they wait for the LLM to build a response with each call. As the user base grows, you anticipate that the expenses will grow exponentially, making it more expensive to run your system eventually.

How does Semantic caching in Azure API Management fix this?

Let's look at the same scenario as described above (at a high level first), with a flow diagram representing how you can cut costs and boost your app's performance with the semantic cache policy.

• When the user sends in the first query, the LLM will be used to generate a response, which will then be stored in the cache.
• Queries 2 and 3 are somewhat related to query 1, which could be a semantic similarity, or exact match, or could contain a specified keyword, i.e.. price. In all these cases, a lookup will be performed, and the appropriate response will be retrieved from the cache, without waiting on the LLM to regenerate a response.
• Query 4, which is different from the previous prompts, will require the call to be passed through to the LLM, then grabs the generated response and stores it in the cache for future searches.

Okay. Tell me more - How does this work and how do I set it up?

Think about this - What would be the likelihood of your users asking related questions or exactly comparable questions in your app? I'd argue that the odds are quite high.

Semantic caching for Azure OpenAI API requests

To start, you will need to add Azure OpenAI Service APIs to your Azure API Management instance with semantic caching enabled. Luckily, this step has been reduced to just a one-click step. I'll link a tutorial on this in the 'Resources' section.

Before you get to configure the policies, you first need to set up a backend for the embeddings API. Oh yes, as part of your deployments, you will need an embedding model to convert your input to the corresponding vector representation, allowing Azure Redis cache to perform the vector similarity search. This step also allows you to set a score_threshold, a parameter used to determine how similar user queries need to be to retrieve responses from the cache.

Next, is to add the two policies that you need: azure-openai-semantic-cache-store/ llm-semantic-cache-store and azure-openai-semantic-cache-lookup/ llm-semantic-cache-lookup

The azure-openai-semantic-cache-store policy will cache the completions and requests to the configured cache service. You can use the internal Azure Redis enterprise or any another external cache as long as it's a Redis-compatible cache in Azure API Management.

The second policy, azure-openai-semantic-cache-lookup, based on the proximity result of the similarity search and the score_threshold, will perform a cache lookup through the compilation of cached requests and completions. In addition to the score_threshold attribute, you will also specify the id of the embeddings backend created in an earlier step and can choose to omit the system messages from the prompt at this step.

These two policies enhance your system's efficiency and performance by reusing completions, increasing response speed, and making your API calls much cheaper.

Alright, so what should be my next steps?

This article just introduced you to one of the many Generative AI supported capabilities in Azure API Management. We have more policies that you can use to better manage your AI APIs, covered in other articles in this series. Do check them out.

Do you have any resources I can look at in the meantime to learn more?

Absolutely! Check out: -

1. Using external Redis-compatible cache in Azure API Management documentation
2. Use Azure Cache for Redis as a semantic cache tutorial
3. Enable semantic caching for Azure OpenAI APIs in Azure API Management article
4. Improve the performance of an API by adding a caching policy in Azure API Management Learn module