When the Answer Isn’t in the Document: Metadata-Aware Agentic RAG with Foundry IQ

The problem

SharePoint libraries often contain years of carefully curated metadata, including root cause, severity, business unit, document type, owner and classification. If that metadata never reaches the retrieval pipeline, however, a RAG system cannot use it.

This creates a common blind spot: the information that explains what a document is about may exist in its metadata rather than its text.

Consider an incident review containing this statement:

“Investigation found that a DNS record was repointed during a migration while the old target was still serving writes.”

The associated Root Cause Category is Configuration change, but those words do not appear in the document. Another review might mention that a configuration change was suspected and ruled out. Text-only retrieval can therefore favour the wrong document.

In the sample dataset used for this project, none of the 19 reviews classified as configuration-change incidents contained the word “configuration”, while six reviews with other root causes did.

Why tuning alone cannot fix it

With a native SharePoint knowledge source, document text is extracted, chunked and embedded, while standard properties are retained. Custom SharePoint columns are not part of that ingestion path.

Fields such as Root Cause Category, Contributing Factors, Detection Method and Severity therefore never become structured fields available to retrieval. If the metadata is absent from the index, search tuning cannot make it count.

Flattening all metadata into a single text string is also problematic:

Notification Service | Sev2 | Configuration change | Missing runbook

This preserves the words but loses their meaning. A root cause, severity and contributing factor represent different concepts and should remain separate fields.

Bring a custom Azure AI Search index

The approach is to create and control a custom Azure AI Search index, then expose it to Foundry IQ as a searchIndex knowledge source. This combines metadata control with agentic query planning and a governed retrieval surface.

Architecture

The key design principle is simple: every chunk carries its parent document's metadata.

Figure 1. Metadata-aware retrieval architecture with Azure AI Search and Foundry IQ.

The ingestion pipeline extracts a document, creates chunks, attaches document-level metadata to every chunk, generates embeddings and writes the records to Azure AI Search. Foundry IQ then uses the index as a knowledge source.

Azure AI Search holds the metadata. Foundry IQ decides when to use it.

Figure 2. Detailed implementation view, including the baseline and metadata-aware retrieval paths.

At query time, the search service calls the configured models for vectorisation and query planning. Where key authentication is disabled, the search service's managed identity needs the required access. In the tested implementation, the Cognitive Services User role resolved HTTP 401 responses from retrieve requests.

Put metadata on every chunk

A chunk can only be retrieved and ranked using information available on that chunk. Each metadata property should therefore remain in its own field:

{
  "id": "doc1035-000",
  "parentId": "doc1035",
  "title": "Notification Service: failed sign-ins",
  "content": "<extracted review text>",
  "rootCauseCategory": "Configuration change",
  "contributingFactors": "Missing runbook; Change outside window",
  "detectionMethod": "Customer report",
  "severity": "Sev2"
}

A separate metadata index does not solve the same problem because a match against metadata in one index cannot raise the ranking of a content chunk in another.

What influences retrieval

Four areas matter in this design:

• Search fields: searchFields determine which fields generated subqueries search.
• Semantic configuration: prioritised fields give the semantic reranker access to important metadata.
• Query hints: hints help the planner connect natural-language requests to known metadata values.
• Filters: a baseFilter or request-level filterAddOn can enforce constraints.

Boost hints influence ranking without excluding documents. Filter hints can narrow the result set when the planner determines that a filter is appropriate.

"queryHints": {
  "boosts": [{
    "kind": "fieldValue",
    "field": "rootCauseCategory",
    "fieldValues": ["Configuration change", "Capacity",
      "Software defect", "Dependency failure", "Certificate expiry"],
    "boost": 3.0,
    "boostInstructions": "Boost when the user describes what caused an incident."
  }]
}

Hints are best effort. Hard requirements should use a base filter or document-level access control rather than relying on the planner to select a hint.

The tested configuration used the 2026-08-01-preview API. Current documentation should be checked for supported models, limits and behaviour before implementation.

Why can't vector search or semantic ranking solve this?

This is probably the most obvious question: if embeddings understand meaning, and semantic ranking understands context, why do we need metadata at all?

The answer is that vector search and semantic ranking solve a different problem. This approach complements them rather than replacing them.

Vector search finds similarity, not recorded facts

In a typical RAG pipeline, document chunks are converted into embeddings, numerical representations of their meaning. The user's question is embedded as well, and vector search finds chunks that are close to it in vector space.

For example, a passage saying:

“The gateway routing rule was updated shortly before the incident.”

might rank highly for:

“Find incidents caused by configuration changes.”

That's useful. But it doesn't necessarily mean Configuration change was the confirmed root cause. The passage could describe an investigation path, a contributing factor, or even something that was later ruled out.

Vector similarity is essentially asking:

“Does this content mean something similar to what the user is asking about?”

Metadata lets us ask a different question:

“What was this incident actually classified as?”

If the incident review already contains a curated rootCauseCategory = Configuration change field, we should not require an embedding model to rediscover that fact from the prose every time somebody searches.

What about semantic ranking?

Semantic ranking helps, but at a different stage.

In Azure AI Search, semantic ranker applies secondary ranking to an initial result set, using language understanding to promote results that are more semantically relevant to the query.

That can help distinguish between:

• a configuration change that caused an incident
• a configuration change that was merely investigated
• a configuration change that was explicitly ruled out

But semantic ranking still works with the information supplied to the retrieval pipeline. It cannot make use of a classification stored in a separate metadata field if that field was never supplied to retrieval or ranking in the first place.

The difference becomes clearer with a real query

Consider:

“Show me configuration change incidents that customers reported.”

A content-first approach primarily asks:

“Does this passage sound relevant to configuration changes and customer reports?”

A metadata-aware approach can additionally ask:

“Is the recorded root cause Configuration change, and is the recorded detection method Customer report?”

Those signals can influence retrieval in different ways:

• Boost: favour incidents with those metadata values while still allowing other relevant results.
• Filter: restrict the candidate set when those classifications represent hard constraints.

There is an important nuance here: traditional RAG can absolutely use metadata filters and hybrid search. So the contribution here isn't a new embedding algorithm, nor is metadata an alternative to vector search.

It is about making structured knowledge a first-class retrieval signal, and wiring that information into candidate retrieval, ranking and Foundry IQ's agentic query planning rather than asking the system to infer every classification from document text.

And importantly, in the comparison below, both approaches use the same Azure AI Search index.

So the comparison isn't:

vector search vs. metadata.

It's:

content-first retrieval vs. retrieval that can also use the structured information the organisation already knows.

Measuring the impact

Two knowledge bases were created over the same Azure AI Search index. The baseline searched title and content. The metadata-aware configuration used the same content plus structured metadata, semantic configuration and query hints.

Ten questions were run through both configurations. Correctness was determined directly from the metadata attached to returned reviews, rather than by asking an LLM whether a result appeared relevant.

For the question “What can be learned from outages caused by configuration changes?”, the planner generated:

rootCauseCategory eq 'Configuration change'

Across ten questions and four runs using gpt-5.4-mini for query planning, the baseline returned 26 matching reviews in the top five. The metadata-aware configuration returned 39 to 41, against a best possible score of 50.

The gains appeared where metadata carried the missing meaning:

• Configuration changes: 0 → 3
• Configuration changes reported by customers: 0 → 3
• Capacity in production: 1 → 5
• Alert fatigue: 3 → 5

Queries whose important terms already appeared in the document text scored 5/5 in both configurations. The result is therefore more specific than “metadata improves search”: metadata closes a retrieval gap when document text does not fully describe what the document represents.

A failure mode worth handling

Because query planning uses an LLM, the planner can occasionally generate a filter that the search index rejects. Two invalid examples observed during testing were:

severity eq Sev1
changeRelated eq true

The first omitted quotes around a string. The second treated an Edm.String field containing Yes and No as a Boolean. The latter caused the retrieve request to fail with HTTP 502.

The fix was to make the schema and allowed values explicit:

This is a text field, not a Boolean. The only valid values are
'Yes' and 'No'. Never compare the field with true or false.

Application code should also handle retrieval failures rather than assuming every generated filter will be valid.

Production considerations

A custom index provides control, but the solution now owns ingestion. Options include the SharePoint indexer with custom columns, an Azure Function using Microsoft Graph, or an existing pipeline built with Logic Apps, Azure Data Factory or Microsoft Fabric.

A custom Azure AI Search index does not automatically inherit the SharePoint permissions model. Where users have different document permissions, ACLs must be carried into ingestion and retrieval. Query hints should also be regenerated when taxonomy values change.

Key takeaways

1. Confirm that the required metadata reaches the index.
2. Keep metadata structured rather than flattening it into one text field.
3. Attach parent-document metadata to every chunk.
4. Use search fields, semantic configuration, query hints and filters deliberately.
5. Describe field types and valid values clearly in filter instructions.
6. Evaluate against a baseline using objective correctness rules.

Try the proof of concept

The complete proof of concept, synthetic dataset, index scripts, knowledge-base configurations and A/B evaluation harness are available in the GitHub - shikha-msft/foundryiq-metadata-agentic-rag.

References

• Create a search index knowledge source
• Create a knowledge base
• Query a knowledge base via API or MCP
• What is Foundry IQ?