Introducing DiskANN Vector Index in Azure Database for PostgreSQL

This post is co-authored by Harsha Simhadri, Principal Researcher (DiskANN co-author) and Maxim Lukiyanov, Principal PM

 

We're thrilled to announce the preview of DiskANN, a leading vector indexing algorithm, on Azure Database for PostgreSQL - Flexible Server! Developed by Microsoft Research and used extensively at Microsoft in global services such as Bing and Microsoft 365, DiskANN enables developers to build highly accurate, performant and scalable Generative AI applications surpassing pgvector’s HNSW and IVFFlat in both latency and accuracy. DiskANN also overcomes a long-standing limitation of pgvector in filtered vector search, where it occasionally returns incorrect results.

 

The DiskANN algorithm we’re releasing today is built on a new high-performance implementation started by the original Microsoft Research (MSR) authors. Our team has worked closely with MSR to fine-tune it specifically for PostgreSQL.

What is DiskANN?

 

DiskANN is an approximate nearest neighbor search algorithm designed for efficient vector search at any scale. It offers a hard to achieve combination of high recall, high throughput, and low query latency required by modern production-grade semantic search and Retrieval Augmented Generation (RAG) applications. 

 

DiskANN optimized layout on Azure PostgreSQL- Flexible Server

 

 

DiskANN implements the following powerful techniques:

• Optimized storage – allows algorithm to scale beyond limits of RAM without sacrificing search speed.
• Vector quantization – keeps quantized vectors in memory. Our implementation of DiskANN on PostgreSQL balances the interactions between quantized and unquantized vectors, delivering both low latency and high accuracy.
• Iterative post filtering - enhances the accuracy of filtered vector search results without compromising on speed or precision.
• New Vamana graph structure robust to index updates – Vamana is more robust to changes than existing graph indices by maintaining accuracy despite many insertions, modifications, and deletions, without the need for expensive index rebuilds.
• Learn more about DiskANN from Microsoft.
  
  

The Filtered Vector Search Problem

 

Vector similarity search involves creating vector representations of data (like images or text), storing them in an index, and searching through that index with a query vector to find relevant results. While running vector search queries users often apply filters on non-vectorized data to improved relevance, i.e. filtering by a date or price on the vector query. Despite this common use case pgvector support for filtered vector search has been limited. Imagine you want to search for the top 3 rental listings to stay at for under $30 on a specific date, using the following query:

 

WITH all_filtered_listings AS
(
SELECT l.listing_id, l.name, c.price, c.date, l.embedding
FROM listings_search l
INNER JOIN calendar c ON l.listing_id = c.listing_id
WHERE c.date = '2025-01-02' AND c.price < 30   
)
SELECT listing_id, name, price, date, summary
FROM all_filtered_listings
ORDER BY embedding <=> @query_vector
LIMIT 3;

 

 

When using HNSW on pgvector, a filtered vector similarity search to get top 3 most relevant rows across a significant corpus of data in the database will be translated by a query planner into a 2-step query plan:

1) The first step is to order by the most relevant rows in the dataset (40 rows, by default). The number of similar rows to return is defined by the ef_search parameter which controls the size of the dynamic list used during the search process.

2) apply post-filter (price under $30 and on Jan 2^nd, 2025) to remove rows that don’t satisfy predicate filter. Then return 3 rows matching the filters (LIMIT 3 defined in the query).

 

However, pgvector implementation does not guarantee that 3 rows will be returned to the user after post-filtering. On a low selectivity filters the post-filter step may not find any matching in the 40 most relevant rows. When this happens pgvector simply returns no results at all even if there are rows in the table that satisfy the conditions. 

 

Using HNSW. Imagine a scenario where a user searched a property listing database for listing like a given query and wanted only listings less than $30 on a specific date

 

Increasing number of relevant rows HNSW returns might mitigate this issue, however in pgvector ‘s HNSW the max value of relevant rows that can be returned is 1000 rows, so if the query selectivity requires scanning more than the 1000 results returned by the index there is no way to get expected results using HNSW. This situation frequently occurs while searching large datasets resulting in low recall when using HNSW on filtered queries.

Introducing high accuracy filtered vector search with iterative post filtering

 

With DiskANN on Postgres, we are introducing iterative post-filtering which returns highly relevant results with exceptional speed, especially on filtered search queries. Through iterative post-filtering, DiskANN enhances the accuracy of search results by iteratively fetching the next set of closest items until the desired number of results is met, all without compromising on speed or precision. This innovative approach is beneficial for filtered search queries, where maintaining high relevance and low latency is crucial.

 

When using DiskANN, a filtered vector similarity search to get top 3 most relevant rows across a significant corpus of data will be translated by a query planner into a 3-step query plan:

1) The first step is to order by the most relevant rows in the dataset. The number of similar rows to return is defined by the l_value_is parameter in DiskANN.

2) apply post-filter (price under $30 and on Jan 2^nd, 2025) to remove rows that don’t satisfy predicate filter.

3) Keep iterating through all the rows available in order of similarity until 3 rows matching the filters are found (LIMIT 3 defined in the query).

 

Using DiskANN. Imagine a scenario where a user searched a property listing database for listing similar to a given query and wanted only listings less than $30 on a specific date

 

Scalability for the Future

 

Not only does DiskANN improve recall, it can also store and search high volumes of vector data with ease, as DiskANN leverages SSDs to use less memory, providing unparalleled scalability and efficiency. DiskANN leverages quantization to store the compressed vectors in a graph in memory and then looks up full vectors from SSD for final comparison retaining precision. This advancement ensures that even with expansive datasets, the system remains responsive and cost-effective.

 

How storage is optimized on DiskANN

 

Product Quantization Support (Coming Soon)

 

DiskANN uses product quantization (PQ) to dramatically reduce the memory footprint of the vectors. Unlike other quantization techniques, the PQ algorithm can compress vectors more effectively, significantly improving performance.  DiskANN using PQ can keep more data in memory, reducing the need to access slower storage, as well as using less compute when comparing compressed vectors. Additionally, Product Quantization (PQ) maintains high recall rates by ranking the top candidates using the full-precision vectors loaded from storage. 

 

Vector Quantization on DiskANN

 

 

Robust to Insertions, Deletions, and Modifications

 

The DiskANN graph index does not degrade over time with high volumes of inserts, updates, or deletes. This is unlike typical vector databases in the market today, which are built using HNSW and other less robust methods require computationally expensive full index rebuilds to maintain accuracy. DiskANN's robust design aims to minimize maintenance and ensure consistent accuracy, making it ideal for production environments where any dip in search precision is unacceptable.

 

Stability of DiskANN compared to HNSW

 

Embrace the Future of AI with Azure Database for PostgreSQL

 

The integration of state-of-the-art vector search capabilities into Azure Database for PostgreSQL is a game-changer for AI applications. Moving beyond standard vector search, DiskANN vector indexing options empower Azure Database for PostgreSQL to offer unparalleled precision and efficiency. With DiskANN’s advanced features, Azure Database for PostgreSQL enables you to:

• Reduce Operational complexity & Costs: Benefit from lower costs thanks to DiskANN's efficient algorithm and efficiently making use of Postgres storage via well designed index-access-method implementation
• Unlock Scalability: Scale your AI applications seamlessly with confidence knowing Azure Database for PostgreSQL can leverage SSDs and uses less memory for unparalleled scalability.
  
  

Using DiskANN on Azure Database for PostgreSQL

 

Using DiskANN on Azure Database for PostgreSQL is easy.

1. Enable the pgvector & diskann Extension: Allowlist the pgvector and diskann extension within your server configuration.  

 

Activating DiskANN in Azure Database for PostgreSQL

 

 

2. Create a Vector Column: Define a table to store your vector data, including a column of type vector for the vector embeddings.
3. Index the Vector Column: Create an index on the vector column to optimize search performance. The diskann PostgreSQL extension is compatible with pgvector, it uses the same types, distance functions and syntactic style.

 

CREATE EXTENSION IF NOT EXISTS pg_diskann CASCADE;
CREATE INDEX my_table_embedding_diskann_idx ON my_table USING diskann (embedding vector_cosine_ops);

 

 

4. Perform Vector Searches: Use SQL queries to search for similar vectors based on various distance metrics (cosine similarity in the example below).

 

SELECT * FROM my_vectors ORDER BY vector_column <=> @query_vector LIMIT 10;

 

Ready to Dive In?

 

Use the DiskANN preview today and explore the future of AI-driven applications with the power of Azure Database for PostgreSQL!

1. Enroll in the DiskANN early preview 
2. Once you have been approved.  Run our end-to-end sample to test on DiskANN vs HNSW on a sample database here.

Learn More

 

The integration of DiskANN with Azure Database for PostgreSQL opens up a new frontier for building robust, scalable, and efficient AI-driven applications. By leveraging advanced vector search capabilities, you can enhance the performance of your AI applications and deliver more accurate results faster than ever before.

• Learn more about DiskANN in Azure Database for PostgreSQL 
• Azure Database for PostgreSQL in Semantic Kernel
• Azure Database for PostgreSQL | LangChain
• Vector Database | Microsoft Learn
• DiskANN – Microsoft Research

 

Special thanks to the team behind the DiskANN open source project and Suryansh Gupta / Naren Datha for their contributions to the DiskANN paginated search code.