Introduction

Keeping pace with AI representation learning requires continuous integration and adaptation to new advancements. In line with this, we're excited to announce new updates to Azure AI Search's integrated vectorization (preview) feature. It now supports native multimodal search capabilities, that seamlessly manage both text and images during indexing and querying. Moreover, we've incorporated support for new embedding models from both Azure AI Studio model catalog and Azure OpenAI, further broadening the spectrum of our services. These enhancements provide developers with intuitive solutions that simplify the integration process for vector or hybrid search, reduce maintenance efforts, and enrich retrieval augmented generation (RAG) applications with more relevant responses. These updates are part of REST API 2024-05-01 Preview version.

Integrated Vectorization: A Snapshot of Core Concepts

Integrated vectorization, a key low-code/no-code (via Azure portal) feature in Azure AI Search, is designed to streamline data transformation to better support vector or hybrid search. It comprises of two key phases with specific functionality:

1. Data ingestion phase: Simplified data chunking and embedding creation.         

Figure 1 – Indexing pipeline steps, including embedding creation as part of integrated vectorization and new functionality

1. Query time phase: User incoming query embedding generation.

      Figure 2 – RAG architecture that showcases orchestrator application contacting AI Search index configured with vectorizers to automatically convert data or images (as applicable) to embedding representations.

Using the Import and vectorize data wizard in the Azure portal automates these stages, efficiently preparing the index with the appropriate configuration for vector or hybrid search.

Embedding generation is provided during both phases. Integrated vectorization supports natively the embedding models described in this article and the functionality is designed to simplify the process of accessing the models from the indexing pipeline and index configuration. These improvements aim to facilitate task integration, expedite processes and ensure seamless maintenance. In essence, this upgraded functionality extends our commitment to making your experience more efficient and less labor-intensive.

Adding Multimodal Embedding Support

We have expanded integrated vectorization in Azure AI Search to support multimodal scenarios, transparently handling text and images during both indexing and search! This is made possible by integrating the Azure AI Vision service that includes vector multimodal embedding capabilities. This development greatly streamlines the process of creating image embeddings from the indexing pipeline, facilitates the storage of image vector representations in the AI Search index and image search at query time.

This feature offers a significant boost to developer productivity in RAG scenarios integrating the GPT-4 Turbo model with Vision. It not only simplifies the task of creating multimodal embeddings from images or text and landing automatically to the AI Search index, but also querying specific images or text representing those images. This enhanced functionality now enables developers to accelerate their RAG application integration.

This feature operates through the utilization of the AI Vision embedding skill during the data ingestion phase of the pull-indexing pipeline and it is configured as part of a skillset and the AI Vision vectorizer within the index configuration for query embeddings.

Please note that to utilize the AI Vision embedding skill, you must create a multi-service resource for Azure AI Services that includes a multimodal embeddings model deployment. This functionality is only available in Azure regions that support both the Azure AI Vision multimodal embeddings model and Azure AI Search, also both services must be in the same region.

Towards the end of this tutorial, you will find a 'How-To' section. This section will guide you on how to configure this functionality using the Azure portal in a few straightforward steps and other available options.

Using AI Studio Model Catalog Embedding Models as part of Integrated Vectorization

Azure AI Studio recently introduced its model catalog, the hub to discover and use a wide range of models that enable you to build Generative AI applications. This model catalog allows you to deploy models easily, including embedding models, to an online endpoint. We've streamlined our integrated vectorization process to support natively those models.

Now, you can utilize the Azure AI Studio model catalog available embedding models, such

as Cohere models, CLIP, among others, natively during data ingestion for creating embeddings by configuring an AML skill. To complement this feature, an AML vectorizer has been introduced for use at query time, ensuring the chosen model is used to create the associated embeddings for user queries and search vectors in the index. This eliminates the need for your orchestrator in your RAG applications to vectorize data, offering a low-code option.

For this new feature, explore the end-to-end experience from adding the embedding model in Azure AI Studio until the vectorized data is available in your AI Search index.

As you approach the end of this tutorial, a detailed 'How-To' section awaits. This portion provides a short step-by-step guide on setting up this functionality via the Azure portal, as well as other available options.

Latest Azure OpenAI Embedding Model Versions

The integrated vectorization feature in Azure AI Search now supports the latest Azure OpenAI embedding models. These models support choosing the vector length you need, and we support it now by adding a new dimensions parameter as part of the Azure OpenAI embedding skill while indexing and through Azure OpenAI Vectorizer at query time, specifically:

text-embedding-3-small: A highly efficient model that outperforms its predecessor, text-embedding-ada-002, in terms of cost and performance on MIRACL and MTEB benchmarks.
text-embedding-3-large: An advanced model that delivers superior performance on the same benchmarks.

For details about Azure OpenAI embedding models above refer to the Azure OpenAI Service announcement.

Guidelines for Configuring the Latest Features via the Azure Portal and Other Available Options

Use the "Import and vectorize data" wizard

To try out the newly released functionalities, navigate to your AI Search service on the Azure portal and utilize the Import and vectorize data wizard. This tool is designed to guide you through the creation of a fully operational search index, ready for vector and hybrid queries suitable for RAG applications, all within a few straightforward steps.

1. Go to the Azure portal and in your AI Search service, click on Import and vectorize data wizard.

         Figure 3 - Import and vectorize data wizard

2. Configure your data source and click on Next.

Figure 4 - Data source configuration

3. Under Vectorize your text step, choose your preferred text embedding model and choose the required options and when completed click on Next. Note that the model endpoints must be deployed in advance.

a) Azure OpenAI. This now includes support for the latest Azure OpenAI embedding models.

Figure 5 - Choose Azure OpenAI embedding model

 b) Azure AI Studio model catalog.

Figure 6 - Azure AI Studio catalog embedding model

 c) Azure AI Vision.

Figure 7 - Text vectorization with AI Vision embeddings model

4. Under Vectorize your images step, choose optionally an image or multimodal (image/text) embeddings model. You can choose any deployment in an AI multi-service account or in AI Studio model catalog. Click on Next.

Figure 8 - Image embeddings model

5. In Advanced settings, it is recommended that you enable semantic ranker for improved results and put the indexer on a schedule. Click on Next.

Figure 9 - Advanced settings

6. Review and click on Create.

Other options to configure integrated vectorization

There are a number of other ways to use the integrated vectorization functionality depending on your scenario and whether you prefer to start in the Azure portal or with code:

1. Via the Azure Portal

Build individual AI Search components of the indexing pipeline independently for added customization. This includes the dataset, index, skillset, and finally the indexer.

Figure 10 - Individual object creation for customized indexing pipeline

2. Through the supported SDKs (Python, .NET, Java, and JavaScript)

A great A Python Jupyter notebook is available that demonstrates how to configure your end-to-end indexing pipeline with the functionality addressed in this article, plus several other Azure AI Search new released features.

3. Using REST API:

• For multimodal embedding support
  Our AI Vision embedding skill and vectorizer documentation provide a comprehensive guide.
• For Azure AI Studio model catalog embedding model support
  Access AML skill and new AML vectorizer documentation to learn how to use the REST API for this functionality.
• For support of Azure OpenAI newest embedding models
  Review Azure OpenAI embedding skill and vectorizer.

More news

We have announced more features and improvements in AI Search. Learn more about these announcements in our blog posts:

• Automatically index Fabric OneLake files in Azure AI Search, now in preview.
• Announcing cost-effective RAG at scale with Azure AI Search: more storage, vector capacity and performance in new Azure AI S and L tiers.
• Azure AI Search’s new hybrid and vector search updates.

What’s next?

Keep an eye out for more news on the latest features of Azure AI Search and their role in simplifying integration for RAG applications!

Getting started with Azure AI Search

• Learn more about Azure AI Search and about all the latest features. 
• Start creating a search service in the Azure Portal, Azure CLI, the Management REST API, ARM template, or a Bicep file. 
• Learn more about integrated vectorization and why chunking and vectors are important in your RAG solutions.
• Learn about Retrieval Augmented Generation in Azure AI Search.
• Explore our preview client libraries in Python, .NET, Java, and JavaScript, offering diverse integration methods to cater to varying user needs. Read the blog: Outperforming vector search with hybrid retrieval and ranking capabilities and watch a video on Microsoft Mechanics: How vector search and semantic ranking improve your AI prompts. 
• Explore how to create end-to-end RAG applications with Azure AI Studio.