Azure Cognitive Search AMA: Vector search, Azure OpenAI Service, generative apps, plugins & more

One good example of how Azure OpenAI integrates with other products is vector search. You can Azure OpenAI to create embeddings from your content and load them into Azure Cognitive Search. We also created Azure OpenAI on your data to help you chat over data that lives in your search service: Introducing Azure OpenAI Service On Your Data in Public Preview - Microsoft Community Hub

Azure OpenAI service also supports function calling which gives you another way to integrate Azure OpenAI with othe Azure services.

Hey Carew - this AMA is targeted for questions around Azure Cognitive Search so I don't think we'll be able to answer this question for you. You could try posting your question in the Power Platform Community.

Hi Barry, yes, whatever is retrieved from Azure Cognitive Search and injected into the Azure OpenAI Prompt will count towards the token count (and thus the cost to execute that Azure OpenAI request. Pablo created a really good blog post that goes into more details on this. This is ultimately why the relevance of Azure Cognitive Search is so important and why we spend a lot of our time in making sure we can get the best results in the top 3-5 results to help minimize the number of tokens that are required. As for the cost, you will typically be "chunking" your data which should give you a good idea on the cost of any Azure OpenAI request as you can look at the typical number of tokens per chunk x the number of results you add to the prompt.

Hi Sanket,

For the RAG as a service, I would highly recommend you take a look at Azure OpenAI on your Data as well as the related blog post,  which I think is likely very close to what you might be looking for which is a really easy way to get up and running with RAG and enterprise ChatGPT. As for futures for vector search, our main goals are to simplify a lot of the complexities that come with vectorization of content which include looking at how to simplify ingestion and queries when vectorization is required as well as topics such as effective chunking of data.

Hi Adam,

One good option is to use Azure Data Factory, which has a Snowflake connector. to ingest data. Using this, you can also take the data copied and send it to Azure Cognitive Search. It is however important to note that you can also do this programmatically if you prefer using our PUSH api which allows you to format your data in JSON format and send it directly to Azure Cognitive Search

There is new ranking for vector search: vector search queries can be ranked using cosine similarity, euclidean distance, or dot products. Azure Cognitive Search also supports hyrbid search which combines keyword based search and vector search results using reciprocal rank fusion.

This document has a great overview of how ranking works with vector search: Query vector data in a search index - Azure Cognitive Search | Microsoft Learn

Vector search performance is a very interesting topic. The Approximate Nearest Neighbors methods, being approximate, use different approaches to influence the speed/recall tradeoff. In the limit, you can achieve perfect recall by comparing the query vector to all vectors in the database but that’s obviously too expensive. To improve speed, ANN algorithms leverage compression techniques lower precision, or data structures to partition the indexed data to reduce the number of vector comparisons that need to be made. All of it to say that statements about performance should only be made at a specific recall target. Another dimension to this problem is price since you can choose to spend more on hardware to improve search latency and throughput. In an ideal benchmark, you’d pick configurations comparable on price and recall before you look at search performance since they are all related.

Both Cognitive Search and Redis use HNSW as the Approximate Nearest Neighbors algorithm which is one of the leading methods for applications optimizing for low latency and high recall and data that’s indexed incrementally and can change over time. The main differences in price/performance could stem from differences in implementation and other functionalities the service offers in addition to Vector search which are included in the price i.e., scaling, security, compliance, integration with other services, etc.

Hope this helps you reason about Vector search performance in Azure Cognitive Search and how it compares. We’re planning to publish results of our benchmarks which compares Vector search performance between different Cognitive Search SKUs and service topologies – how adding replicas and partitions changes Vector search performance profile. The basic intuition is that adding replicas improves throughput and adding partitions reduces latency.

This appears to be a question about the 1000 field limit in Azure Cognitive Search. There isn't an easy workaround to the field limit, but you do have some options to get the experience you are looking for.

These are 3 options I would suggest considering:

1. Field reuse within an index.  If each product catalog entry would use less than 1000 fields, you could reuse the same fields for multiple product types with your application managing the mapping of the meaning of each attribute to the underlying field in the index.  This option does have drawbacks since each field would have the same settings across multiple product types.
2. Extend catalog to multiple indexes.  You could effectively store your catalog in multiple indexes where the full set of fields will extend across the indexes.  This would work best if an individual catalog entry doesn't have more than 1000 fields again and would run into some cross-index query issues.  But you should be able to get faceting to work in this option.

3. Attribute metadata fields.  Define a few general fields in your index to store attribute metadata. These fields could include "attribute_name," "attribute_value," and "product_id" or any other relevant identifiers.  For each product, store its ETIM classification attributes as separate records in the attribute metadata fields. This means you'll have multiple attribute records for each product, one for each ETIM attribute associated with that product.  To implement this solution, you'll need to update your data indexing and search query pipelines to work with the attribute metadata fields and apply filters accordingly. It may require some adjustments to your existing code, but it should provide a way to handle the complex attribute requirements.

Good news: Function calling is now available in Azure OpenAI Service - Microsoft Community Hub