Vector search finds candidates. Reranking decides what your RAG app reads

You ask a retrieval-augmented generation (RAG) application a question. Vector search returns ten passages that are clearly related to the topic. The passage that actually contains the answer, however, is ranked seventh, while the language model receives only the first five.

Retrieval did not completely fail. It found the evidence, but ordered it below less useful context. Reranking addresses that gap between a passage that is semantically similar and a passage that is relevant to the user's specific question.

This article demonstrates that pattern in four Azure services using the Stanford Question Answering Dataset (SQuAD). The goal is not to declare a winning service or publish a quality benchmark. It is to show where retrieval, rank fusion, and model-based reranking run in each architecture, and to illustrate how the position of a known source passage can change.

What this demonstration establishes

The examples show rank movement for three selected questions. They do not establish that one reranker or service is universally more accurate. A production decision requires a larger, representative query set and aggregate relevance, latency, and cost measurements.

Retrieval and reranking are different stages

A production search pipeline commonly uses two stages:

1. Retrieve for recall. Fast retrieval narrows a large corpus to a bounded candidate set. It can use vector search, keyword search, or both.
2. Rerank for precision. A more expensive model evaluates only those candidates against the original query and produces the final order.

Reciprocal Rank Fusion (RRF) belongs between those two ideas. RRF is a model-free rank aggregation method that merges independent result lists, usually vector and keyword results. For a document d, a typical score is:

This gives a clearer three-part vocabulary:

RRF often improves hybrid retrieval when exact names, dates, identifiers, or terms matter. A learned reranker can then read the query and each candidate together, capturing interactions that separately generated embeddings can miss. The learned stage costs more, so it should operate on tens of candidates rather than the whole corpus.

The following image describes the general process:

Retrieval process in 3 stages

Why use SQuAD for this demonstration?

SQuAD 1.1 contains crowd-written questions over more than 500 Wikipedia articles. Its packaged splits contain 87,599 training rows and 10,570 validation rows. Each row includes a question, a context passage, and one or more answer spans inside that passage.

That source-context mapping gives this demonstration a useful label: the context associated with a question is treated as its gold passage. We can then inspect whether each search stage moves that passage up or down.

This is convenient, but it is not a perfect passage-ranking benchmark. SQuAD was designed for extractive question answering, and another passage in the corpus might also answer a question. The gold context is therefore a reproducible reference, not proof that every other passage is irrelevant.

The results shown here use the 2,067 unique contexts in the SQuAD validation split and 1,536-dimensional embeddings. The repository default should be set to the same corpus size before treating the screenshots or rank transitions as directly reproducible.

Three illustrative questions

Each notebook selects a seeded demonstration question when it runs. The three saved examples were collected across separate runs; the current notebooks do not execute all three questions in one pass. A benchmark harness should iterate over a fixed question list and save all stage results in one structured output.

Capability boundaries at a glance

Azure AI Search: native hybrid retrieval and semantic ranking

How it works: Azure AI Search provides the most integrated pipeline in this demonstration. A hybrid query runs keyword and vector retrieval, combines the lists with RRF, and passes up to the top 50 results to the built-in semantic ranker.
The semantic ranker assigns @search.rerankerScore values from 0 to 4 and can return extractive captions and answers.
The semantic configuration identifies the fields that carry the meaning of each document:

semantic_search = SemanticSearch(
    configurations=[
        SemanticConfiguration(
            name=SEMANTIC_CONFIG,
            prioritized_fields=SemanticPrioritizedFields(
                title_field=SemanticField(field_name="title"),
                content_fields=[SemanticField(field_name="content")],
            ),
        )
    ]
)

This tells the semantic ranker which text fields to evaluate.

The query then enables semantic ranking after hybrid retrieval:

results = search_client.search(
    search_text=question,
    vector_queries=[vector_query],
    query_type="semantic",
    semantic_configuration_name=SEMANTIC_CONFIG,
    top=10,
)

The important constraint is candidate recall. Semantic ranking does not search the corpus again. If the correct passage is absent from the hybrid top 50, the semantic stage cannot recover it.
See 01_azure_ai_search_reranking.ipynb for the complete setup and query path.

Test results for Azure AI Search

Question 1: semantic reranker moved down the gold passage

Question 2: semantic reranker moved again doen the gold passage one position.

Question 3: semantic reranker moved the gold passage to the first postition.

These examples show that semantic reranking improves relevance selectively, not universally. It strongly helps the construction query, moving the correct passage from rank 4 to rank 1, but slightly degrades the Super Bowl and Tesla queries by one position. This reinforces that semantic ranking should be evaluated across a representative query set using aggregate metrics such as MRR or NDCG, rather than judged from a single result.

Azure SQL Database: vector retrieval plus external Cohere reranking

How it works. The Azure SQL notebook retrieves 20 candidates with exact cosine distance and sends their text to Cohere Rerank v4.0 Fast through sys.sp_invoke_external_rest_endpoint.
The vector column and query vector must have the same dimensions. This repository uses 1,536-dimensional embeddings:

SELECT TOP (@ candidate_count) context_id,
  title,
  content,
  1 - VECTOR_DISTANCE(
    'cosine',
    CAST(@ query_vector AS VECTOR(1536)),
    embedding
  ) AS similarity
FROM dbo.documents
ORDER BY similarity DESC;

For reranking, we selected Cohere Rerank v4.0 Fast (Cohere-rerank-v4.0-fast), a fast version of Cohere’s fourth-generation relevance-ranking model. The model is deployed in Microsoft Foundry, where its Azure Direct inference endpoint is available in the deployment details within the Foundry portal.

Azure SQL can call REST APIs directly using sp_invoke_external_rest_endpoint. Because Azure SQL allowlists Azure AI’s *.cognitiveservices.azure.com domain, we translate the equivalent Foundry endpoint from *.services.ai.azure.com while preserving the Cohere reranking route.

from urllib.parse import urlsplit, urlunsplit 

def sql_compatible_endpoint(endpoint: str) -> str: 
    """Convert an Azure Direct endpoint to Azure SQL's allowed hostname.""" 
    parts = urlsplit(endpoint)
    if parts.hostname.endswith(".services.ai.azure.com"):
        resource = parts.hostname.removesuffix(".services.ai.azure.com")
        hostname = f"{resource}.cognitiveservices.azure.com"
    elif parts.hostname.endswith(".cognitiveservices.azure.com"):
        hostname = parts.hostname
    else:
        raise ValueError("Expected an Azure AI Services endpoint.")
    return urlunsplit(
        (parts.scheme, hostname, parts.path, parts.query, "")
    )

Then I defined a re-rank with cohere function, starting by loading the endpoint and setting the authentication:

def rerank_with_cohere( cursor, question: str, candidates: list[dict], top_n: int = 10, ) -> list[dict]:
    """ Rerank candidate documents by calling Cohere through Azure SQL. Each candidate must contain a 'content' field. """ 
    if not candidates: 
        return [] 
    sql_endpoint = sql_compatible_endpoint( os.environ["COHERE_RERANK_ENDPOINT"] ) 
    model = os.environ["COHERE_RERANK_MODEL"] 
    access_token = credential.get_token(
            "https://cognitiveservices.azure.com/.default"
        ).token
    headers = json.dumps({"Authorization": f"Bearer {access_token}"})
    payload = json.dumps(
        {
            "model": model,
            "query": question,
            "documents": [row["content"] for row in candidates],
            "top_n": min(k, len(candidates)),
        },
        ensure_ascii=False,
    )

    cursor.execute(
        """
        DECLARE @url NVARCHAR(4000) = CAST(? AS NVARCHAR(4000));
        DECLARE @headers NVARCHAR(4000) = CAST(? AS NVARCHAR(4000));
        DECLARE Payload NVARCHAR(MAX) = CAST(? AS NVARCHAR(MAX));
        DECLARE Response NVARCHAR(MAX);
        DECLARE @status INT;

        EXEC @status = sys.sp_invoke_external_rest_endpoint
            @url = @url,
            @method = 'POST',
            @headers = @headers,
            Payload = Payload,
            @timeout = 60,
            @retry_count = 2,
            Response = Response OUTPUT;

        SELECT @status, Response;
        """,
        sql_endpoint,
        headers,
        payload,
    )
    status, response_text = cursor.fetchone()
    if status != 0:
        raise RuntimeError(f"Reranker endpoint returned HTTP status {status}.")

    response = json.loads(response_text)["result"]

You can see the complete implementation in the  02_azure_sql_reranking.ipynb notebook.

Test results for Azure SQL Db

Question 1: the re-ranking model moved the gold passage to position 1.

Question 2: the re-ranking model moved the gold passage from position 2 to position 1.

Question 3: the re-ranking model moved the gold passage to position 1.

Across the three sample questions, Cohere reranking consistently moved the correct SQuAD passage closer to the top: from rank 5 to 1 for the Super Bowl question, 3 to 2 for the Tesla question, and 8 to 1 for the construction question. These examples show how vector search provides a strong candidate set, while reranking applies deeper query-document relevance scoring to improve the final ordering. The results are illustrative rather than a complete quality benchmark, so broader evaluation across many queries is still recommended.

Azure Database for PostgreSQL flexible server: pgvector, SQL RRF, and an optional model

How it works: PostgreSQL makes the pipeline components explicit. The notebook uses pgvector for vector similarity, PostgreSQL full-text search for keyword retrieval, and SQL to implement RRF.
Vector retrieval uses cosine distance:

SELECT context_id,
  title,
  content,
  1 - (embedding <= > % (query_vector) s:: vector) AS similarity
FROM squad_docs
ORDER BY  embedding <= > % (query_vector) s:: vector
LIMIT % (candidate_count) s;

The hybrid query independently ranks vector and keyword hits, then combines positions rather than raw scores:

SELECT d.context_id,
  COALESCE(1.0 / (60 + v.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score
FROM squad_docs AS d
  LEFT JOIN vector_hits AS v USING (context_id)
  LEFT JOIN keyword_hits AS k USING (context_id)
WHERE
  v.context_id IS NOT NULL OR k.context_id IS NOT NULL
ORDER BY rrf_score DESC;

This is not a built-in PostgreSQL RRF operator. It is transparent, hand-written SQL over native retrieval primitives, which makes weighting and debugging flexible but leaves implementation and tuning with the application team.
The notebook's optional learned stage sends the vector candidates from Python to a Foundry deployment of Cohere Rerank v4.0 Fast. This path was chosen because the tested Flexible Server azure_ai extension version expected the older serverless reranking endpoint contract. Microsoft documentation still describes azure_ai.rank() as a preview function whose default model is Cohere Rerank v3.5, even though that model retired on May 14, 2026. Treat  this as a version-specific compatibility issue and verify current extension behavior before selecting an architecture.
Azure HorizonDB is a different product path. Its AI Model Management feature can provision Cohere Rerank v4.0 Fast as default-reranker, but that management feature is currently a limited preview. It should not be described as a generally available Flexible Server capability.

See 03_azure_postgres_reranking.ipynb for the full SQL and optional external model path.

Test results for Azure SQL for PostgreSQL Flexible Server

Question 1: The re-ranking model clearly made the difference.

Question 2: The re-reanking model improved the results by one position.

Question 3: The re-reanking model and the RRF achived the same result moving the gold passage to the top.

The tests show that PostgreSQL vector search provides a useful candidate set, SQL RRF can substantially improve results when keyword evidence is strong, and the Cohere semantic reranker is the most consistent overall: it moved the correct passage to rank 1 in two tests and from rank 3 to rank 2 in the Tesla test. RRF produced the biggest gain for the construction question, moving the correct passage from outside the vector top five to rank 1, but did not improve every query. The scores across stages are not directly comparable because cosine similarity, RRF score, and Cohere relevance use different scales.

Azure Cosmos DB for NoSQL: hybrid search with built-in RRF 

How it works: Azure Cosmos DB for NoSQL supports native hybrid ranking with VectorDistance, FullTextScore, and RRF inside ORDER BY RANK:

SELECT TOP K C.context_id, 
c.title, 
c.text 
FROM c
ORDER BY RANK RRF( VectorDistance(c.vector, @query_vector), FullTextScore(c.text, @term1, @term2, @term3) )

The notebook extracts distinct terms from the question before building the full-text part of the query. That token selection is application logic and can materially affect the hybrid ranking, so production evaluation should test analyzers, languages, term extraction, and optional RRF weights.
Cosmos DB Semantic Reranker is an SDK-integrated preview feature. The application first runs a query, serializes the resulting documents, and submits those documents with the user's context string:

result = container.semantic_rerank(
    context=question,
    documents=documents,
    options={
        "return_documents": False,
        "top_k": min(k, len(documents)),
        "sort": True,
        "document_type": "json",
        "target_paths": "title,text",
    },
)

The service accepts at most 50 documents per rerank call and returns relevance scores from 0 to 1, plus inference latency and token usage. It uses the Microsoft semantic ranking model also used by Azure AI Search. The reranking call requires Microsoft Entra authentication, the appropriate Semantic Reranker role, and an account-linked inference endpoint.

The 04_azure_cosmosdb_reranking.ipynb in the shared repo contains and end-to-end implementation.

Test results for Azure Cosmos Db

Question 1: The re-ranking model improved results while RRF made the gold passage dissapeared from the top 5.

Question 2: The re-reanking model improved the results.

Question 3: The re-reanking model and the RRF improved the results.

The results show that vector search provides a strong baseline, while hybrid RRF and semantic reranking improve different queries in different ways. Hybrid RRF helps when exact keywords matter, moving the construction answer into the top results, while the semantic reranker delivers the strongest overall ordering, promoting the correct construction passage from hybrid rank 3 to rank 1 and improving the Super Bowl answer from rank 5 to rank 2. However, it does not always place the gold passage first, as seen in the Tesla example, confirming that reranking improves relevance but is query-dependent and should be evaluated across a larger test set.

What the examples do and do not show

The four services expose different ownership boundaries:
•  Azure AI Search owns hybrid fusion and learned semantic ranking inside the search service.
•  Azure SQL owns vector retrieval and outbound REST invocation in this example, while Foundry owns model inference.
•  PostgreSQL supplies vector and full-text primitives; the application owns the RRF SQL and optional Cohere call.
•  Cosmos DB provides native hybrid RRF and integrates a separate preview inference call through its SDK.
Across three selected questions, the known source passage often moved substantially. That supports the practical value of testing a second-stage ranker. It does not prove that semantic reranking always improves top-1 accuracy, that RRF is universally beneficial, or that scores from different stages can be compared directly.
Cosine similarity, RRF score, Azure AI Search reranker score, Cohere relevance, and Cosmos DB semantic relevance all have different definitions and scales. Compare rank positions and task-level metrics, not raw values across systems.

Turn the demonstration into an evaluation

For a production RAG system, convert the notebook pattern into a repeatable evaluation harness:

1. Build a representative labeled query set from real user tasks.
2. Freeze corpus, chunking, embedding model, dimensions, and candidate counts for each run.
3. Record ranks after retrieval, fusion, and learned reranking.
4. Measure Recall@k or Hit@k to verify that retrieval finds relevant evidence.
5. Measure Mean Reciprocal Rank (MRR) when the position of the first relevant result matters.
6. Use NDCG when judgments include multiple passages or graded relevance.
7. Record latency percentiles, inference usage, request cost, and failure rates.
8. Evaluate the generated answer separately for correctness, citation support, and refusal behavior.

Also test the operational cases that a three-question demonstration cannot cover: empty keyword results, missing gold passages, long documents, multilingual text, filters, partial outages, token expiration, throttling, model retirement, and low-confidence scores.

Practical guidance

• Retrieve broadly enough that the correct evidence can reach the learned stage.
• Use RRF when vector and keyword retrieval provide complementary signals.
• Rerank a bounded candidate set, commonly 20 to 50 passages, and measure the latency cost.
• Keep citations and source identifiers through every rank transformation.
• Version the corpus, embedding model, dimensions, query set, and reranker deployment.
• Do not hard-code assumptions about model endpoints or lifecycle dates. Verify current service documentation and the deployed extension or SDK version.
• Add thresholds or fallback behavior only after calibrating scores on your own data.
• Judge the full RAG chain. Better passage order is valuable only when it improves grounded answers for users.

Vector search is built to find plausible candidates quickly. Rank fusion can reconcile retrieval signals, and a learned reranker can decide which candidates best address the question. The right architecture depends on where your data lives, which service boundaries you want to operate, and what your evaluation says about quality, latency, and cost.

Resources

• Companion repository
• Azure AI Search semantic ranker
• Azure SQL VECTOR_DISTANCE
• Azure SQL sp_invoke_external_rest_endpoint
• Azure Database for PostgreSQL AI functions
• Microsoft Foundry model retirement schedule
• Azure Cosmos DB hybrid search
• Azure Cosmos DB Semantic Reranker
• SQuAD dataset card

Dataset attribution

Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. SQuAD 1.1 is distributed under CC BY-SA 4.0.