Azure Cognitive Search is now Azure AI Search, and semantic search is now semantic ranker. See below for more details.
We are thrilled to announce the public preview of integrated vectorization, a ground-breaking capability of vector search in Azure AI Search (previously Azure Cognitive Search). This feature is designed to streamline the process of chunking, generating, storing, and querying vectors for vector search in Azure AI Search. This feature significantly speeds up the development of the vectorization pipeline and minimizes maintenance tasks during data ingestion and query time. It paves the way for seamless integration of vectors into your applications with traditional search demands and Retrieval-Augmented Generation (RAG) applications.
What is Integrated Vectorization?
Integrated vectorization is a new feature of Azure AI Search that allows chunking and vectorization of data during ingestion through built-in pull-indexers, and vectorization of text queries through vectorizers. With a deployed Azure OpenAI Service embedding model or a custom embedding model, integrated vectorization facilitates automatic chunking and vectorization during data ingestion from various Azure sources such as Blob Storage, SQL, Cosmos DB, Data Lake Gen2, and more. Furthermore, Azure AI Search now incorporates vectorizers referencing your own embedding models that automatically vectorize text queries, effectively eliminating the need for client application coding logic.
Figure 1 - Integrated vectorization diagram
Key Concepts in Integrated Vectorization
Vector search: In Azure AI Search, this is a capability for indexing, storing, and retrieving vector embeddings from a search index. By representing text as vectors, vector search can identify the most similar documents based on their proximity in a vector space. In vector search, vectorization refers to the conversion of text data into vector embeddings.
Chunking: Process of dividing data into smaller manageable parts (chunks) that can be processed independently. Chunking is required if source documents are too large for the maximum input size of embedding and/or large language models.
Retrieval Augmented Generation (RAG): Architecture that augments the capabilities of a Large Language Model (LLM) like ChatGPT by adding an information retrieval system (i.e., Azure AI Search) that provides the data.
Choosing Between Integrated Vectorization and Other Options in Azure
In the rapidly evolving field of RAG applications, there are various systems offering chunking and vectorization capabilities. Consequently, you might find yourself pondering the best choice for different scenarios.
For instance, Microsoft's Azure platform now provides a solution that facilitates the creation of end-to-end applications using the RAG pattern across multiple data sources, including Azure AI Search, all from the convenience of Azure AI Studio. In such a case, it's advisable to utilize the feature for chunking and vectorization, as this functionality is optimized to work seamlessly with your chosen options.
However, if your aim is to construct your RAG application or a traditional application using vector search through alternative methods, or if the built-in functionality of does not align with your specific - business requirements, integrated vectorization is an excellent alternative.
Benefits of Integrated Vectorization
Here are some of the key benefits of integrated vectorization:
Streamlined Maintenance: Users no longer need to maintain a separate data chunking and vectorization pipeline, reducing overhead and simplifying data maintenance.
Up-to-date Results: Use Azure AI Search to pull indexers for incremental tasks, to pull recent results from your search. The feature works seamlessly with Azure AI Search pull indexers to handle incremental tasks. This allows your search service to deliver recent results.
Reduced Complexity: Automatically vectorize your data, to reduce complexity and increase accuracy.
Increased Relevancy: Generate index projections that map one source document to all corresponding vectorized chunks, to enhance the relevance of results. Moreover, this will simplify application development workflows for those developing Retrieval-Augmented Generation (RAG) applications where data chunking is required for retrieval.
Getting Started with Integrated Vectorization
Getting started with integrated vectorization is easy. With just a few clicks in the Azure portal, you can automatically parse your data, divide it into chunks, vectorize them, and project all these vectorized chunks into a and start taking advantage of the many benefits of Azure AI Search.
Data Ingestion Chunking and Vectorization
Follow these quick steps to import, chunk, vectorize and index your data:
1. In the Azure portal, navigate to the Overview section in your Azure AI search service and choose Import your data with vectors from the menu.
Figure 2- Import and vectorize data wizard
2. Select and configure your data source.
Figure 3 - Connect to your data
3. Go to the Vectorize and Enrich data section, add your Azure OpenAI Service enrichment model, and choose a schedule.
Figure 4 - Vectorize and enrich data
4. Add a prefix for the child objects that will be created as part of the deployment.
5. Review and create.
Figure 5 - Review and create child resources
The Import and vectorize data wizard creates the following AI Search child resources:
- A data source.
- A chunked index with its respective vector fields and vectorizers.
- A skillset. The skillset definition has all the logic for the chunking, vectorization, and index projections to map the chunks to the index.
- A built-in pull indexer.
In case you need more customization for chunking or vectorizing operations, you can take advantage of the 2023-10-01 Preview REST API, Azure AI Search latest preview SDKs (.NET,Python,Java and JavaScript) and modify any of the child resources listed above for integrated vectorization.
Vectorization at query time
When using the Import and vectorize data wizard, the selected embedding model is automatically included as a vectorizer in the index vector field. The vectorizer assigned to the index and linked to a vector field will automatically vectorize any text query submitted.
Utilize the Search explorer within the Azure portal to execute text queries against vector fields with vectorizers. The system will automatically vectorize the query. There are multiple ways to access the Search Explorer within the portal:
1. Once the Import and vectorize data wizard has finished and the indexing operation is complete, wait a few minutes and then click on Start Searching.
Figure 6 - Start Searching
2. Alternatively, in the Azure portal, under your search service Overview tab, select Search explorer.
Figure 7 – Overview - Search Explorer
3. You can also use the embedded Search Explorer within an index by selecting the Indexes tab and clicking on the index name.
Figure 8 - Index hyperlink
Upon accessing the Search explorer, you can simply enter a text query. If you have a vector field with an associated vectorizer, the query will be automatically vectorized when you click on Search, and the matching results will be displayed.
Figure 9 - Text query that will be automatically vectorized
To hide the vector fields and make it easier to view the matching results, click on Query options. Then, turn on the toggle button for Hide vector values in search results. Close the Query options and resubmit the search query.
Figure 10 - Hide vector values in search results
Figure 11 - Results with the vector fields hidden
This is a code sample of how a vectorizer looks like in the index JSON definition.
{
"name": "vectorized-index,
"vectorSearch": {
"algorithms": [
{
"name": "myalgorithm",
"kind": "hnsw"
}
],
"vectorizers": [
{
"name": "openai",
"kind": "azureOpenAI",
"azureOpenAIParameters":
{
"resourceUri": "<AzureOpenAIURI>”,
"apiKey": "<AzureOpenAIKey>",
"deploymentId": "<modelDeploymentID>"
}
}
],
"profiles": [
{
"name": "myprofile",
"algorithm": "myalgorythm",
"vectorizer":"openai"
}
]
},
"fields": [
{
"name": "chunkKey",
"type": "Edm.String",
"key": true,
"analyzer": "keyword"
},
{
"name": "parentKey",
"type": "Edm.String"
},
{
"name": "page",
"type": "Edm.String"
},
{
"name": "vector",
"type": "Collection(Edm.Single)",
"dimensions": 1536,
"vectorSearchProfile": "myprofile",
"searchable": true,
"retrievable": true,
"filterable": false,
"sortable": false,
"facetable": false
}
]
}
And here is a code sample of how a text query call submitted against that vector field looks like.
POST <AISearchEndpoint>/indexes/<indexName>/docs/search?api-version=2023-10-01-Preview
{
"vectorQueries": [
{
"kind": "text",
"text":"<add your query>",
"fields": "vector"
}
],
"select": "chunkKey, parentKey, page"
}
Skills that Make Integrated Vectorization Possible During Data Ingestion
Integrated vectorization during data ingestion is made possible by a combination of skills and configurations in Azure AI Search pull indexers. An Azure AI Search skill refers to a singular operation that modifies content in a particular manner. This often involves operations such as text recognition or extraction. However, it can also include a utility skill that refines or alters previously established enrichments. The output of this operation is typically text-based, thereby facilitating its use in comprehensive text queries. Skill operations are orchestrated in skillsets.
A skillset in Azure AI Search is a reusable resource linked to an indexer. It includes one or more skills which invoke either built-in AI functions or external custom processing on documents fetched from an outside data source.
The key skills and configuration that make Integrated vectorization possible include:
- Text split cognitive skill: This skill is designed to break down your data into smaller, manageable chunks. This step is essential for adhering to the input constraints of the embedding model and ensuring the data can be efficiently processed. The smaller data pieces not only facilitate the vectorization process but also enhance the query process, especially for RAG applications. Additionally, the skill enables overlapping of data, which is instrumental in preserving the semantic meaning in multiple scenarios, thereby enhancing the accuracy and quality of search results.
- Azure OpenAI Embedding skill: This skill offers a reliable method to call upon your Azure OpenAI Service embedding model. It empowers generation of highly accurate and precise vectors, thereby improving the overall effectiveness for semantic queries.
- Custom Web API skill: This skill allows you to use your own custom embedding model for vectorization, giving you even more control over the vectorization process.
- Index Projections: This configuration allows you to map one source document associated with multiple chunks. This functionality allows you to have either:
- An index with vectorized chunks only
- An index with the parent documents and vectorized chunks altogether
- Two separate indexes: An index with the full indexed documents and an index with the vectorized chunks.
Figure 12 - Diagrammatic View of Data Ingestion Skills in Azure AI Search's Integrated Vectorization Process
In conclusion, Integrated Vectorization allows you to improve the accuracy and precision of your searches while reducing complexity and overhead. With just a few clicks, you can import your data into Azure AI Search index and start taking advantage of the many benefits Azure AI Search, including easy integration as retriever in RAG applications. Try it out today and see the difference it can make for your search results.
What is next?
Please refer to the integrated vectorization documentation for a more comprehensive understanding of this capability.
We are thrilled to announce the general availability of features vector search and semantic ranker (previously semantic search). Vector search offers a comprehensive vector database solution to store, index, query, filter and retrieve your AI data in a secure, enterprise-grade environment while semantic ranker enhances the relevance of search results by contextually ranking them. Using integrated vectorization with vector search and semantic ranker offers a robust suite of capabilities designed to deliver results tailored to your specific data and needs. To further enhance your search results with these features, we recommend reviewing Outperforming vector search with hybrid retrieval and ranking capabilities and the best retrieval strategies for generative AI.
Azure AI Search's integrated vectorization is compatible with various Azure data sources, including Azure SQL. Here is a compelling user story detailing the process of importing data from Azure SQL databases. This story showcases how integrated vectorization can be leveraged to construct your RAG applications effectively.
Azure Cognitive Search is now Azure AI Search
As of November 15, 2023, Azure Cognitive Search has a new name: Azure AI Search. All existing deployments, configurations, and integrations will continue to function as they currently do. There are no changes to pricing and no breaking changes to APIs or SDKs.
Semantic search, a feature of Azure AI Search (f.k.a Azure Cognitive Search) has been renamed to ‘semantic ranker’ as of November 15, 2023. This new name will not impact functionality, customer experience, or operations, and arrives with a new pay-as-you-go business model that reduces operational costs for customers.
Microsoft
