Building a Self-Updating RAG System: Handling Document Uploads, Updates and Deletions

The Hidden Challenge of Production RAG

Imagine your organization has deployed an AI assistant to answer questions about employee policies.

An employee asks, “How many days of annual leave am I entitled to?”

The assistant responds confidently: “15 days.”

There is just one problem. The company updated its policy last week. Employees are now entitled to 20 days.

The updated PDF was uploaded successfully. The storage service is healthy. The search index is responding. The language model is generating answers without errors.

Yet the answer is wrong.

This is a different kind of production failure. Nothing appears to be broken at the infrastructure level. The failure lies in the relationship between the source document and the information the AI retrieves.

Retrieval-Augmented Generation (RAG) systems depend on external knowledge to ground their responses. But that knowledge is not static. Policies change, contracts get revised, product documentation evolves, and outdated documents are removed.

When the underlying documents change, the associated chunks, embeddings, metadata, and search-index entries must be managed accordingly. Otherwise, the system can continue generating plausible answers from information that is no longer valid.

Building a RAG pipeline is a retrieval problem. Keeping it correct as data changes is a data lifecycle engineering problem.

In this article, we will design a self-updating RAG system that handles document creation, updates, and deletion while addressing asynchronous processing, version consistency, failed jobs, and stale retrieval results.

The goal is not simply to index documents automatically. It is to make sure that the right version of the right document is available to the right user when a query is executed.

Understand the Problem Before Designing the Architecture

A basic RAG pipeline is relatively straightforward:

1. Load a document.
2. Extract its text.
3. Split the text into smaller chunks.
4. Generate an embedding for each chunk.
5. Store the chunks and embeddings in a searchable index.
6. Retrieve relevant chunks when a user asks a question.
7. Pass the retrieved context to a language model to generate an answer.

This works well when the document collection is relatively static.

The difficulty begins when the source data changes after indexing.

Consider three common scenarios.

Scenario A: A document is uploaded

A new policy is added to the knowledge base. The system must process it and make its content searchable.

Until indexing finishes, the document may not be available to the chatbot.

Scenario B: A document is updated

The original policy allows 15 days of annual leave. A revised policy allows 20 days.

If the new version is indexed without correctly deactivating the old version, both versions may appear in search results. The language model may then select the wrong passage or combine conflicting information.

Scenario C: A document is deleted

An administrator removes a document from the application. However, deleting the original PDF from storage does not automatically delete the corresponding records from Azure AI Search.

The content may remain searchable until its indexed chunks are removed. Azure AI Search document deletion itself can also take time to be reflected in index statistics. Microsoft Learn: Delete documents in Azure AI Search

These scenarios reveal an important design principle:

The source document, its metadata, the background processing jobs, and the search index must be treated as parts of one logical lifecycle—even though they are separate systems.

We need a design that can handle failures between these systems without exposing outdated or unauthorized content.

Design the System Around Clear Responsibilities

Rather than making the upload API responsible for every operation, we can separate the system into a document management path, an asynchronous processing path, and a query path.

The Proposed Architecture

The design uses the following components:

The background worker can be implemented as a separate Python service or another suitable compute component. It does not have to be part of the FastAPI process.

 

 

Why separate these responsibilities?

Suppose a user uploads a 100-page contract.

The application may need to extract text, split thousands of characters into chunks, generate embeddings, and send multiple indexing requests. Performing all of that synchronously inside the upload request increases latency and makes the API vulnerable to timeouts.

Instead, FastAPI accepts the request, persists the document and its metadata, and arranges for the work to be processed asynchronously.

The worker performs the expensive operations independently. This allows ingestion to scale separately from query traffic and makes retries easier to manage.

The query path remains separate: it retrieves eligible, relevant chunks and uses them to ground the model's response.

 

Establish a source of truth

Azure Cosmos DB should hold the authoritative lifecycle metadata in this design. Azure AI Search is the retrieval index, not the authority on whether a document is currently active.

For example, a document record might look like this:

 

{
  "id": "policy-123",
  "documentId": "policy-123",
  "currentVersion": 2,
  "indexedVersion": 1,
  "status": "PROCESSING",
  "blobPath": "policies/leave-policy.pdf"
}

 

Here, version 2 is the desired version, but version 1 is still the last successfully indexed version.

The application must not assume that the latest upload has already become searchable simply because the metadata record has been updated.

This distinction between the desired version and the indexed version is central to reliable lifecycle management.

 

Implementing Document Uploads

A reliable upload operation involves more than storing a file and sending a message to a queue.

It must establish a durable record of the document, identify the version being processed, and ensure that background processing can be retried safely.

Step 1: Accept and persist the document

When a client uploads a document, FastAPI validates the request, checks the user's permissions, and stores the original file in Azure Blob Storage.

The application then creates or updates the document metadata in Cosmos DB.

At this stage, the document can be marked as PROCESSING until the ingestion pipeline completes.

Step 2: Schedule background processing

The system creates an ingestion job containing enough information for the worker to locate the source and determine exactly which version to process.

For example:

{
  "operationId": "op-789",
  "operation": "INGEST",
  "documentId": "policy-123",
  "documentVersion": 2,
  "blobPath": "policies/leave-policy-v2.pdf"
}

The worker should process the specified version rather than blindly reading whatever file happens to exist at a mutable storage path.

For systems that overwrite the same blob path, use blob versioning, an immutable version-specific path, or another mechanism that ensures the worker reads the intended content.

Step 3: Extract, chunk, and embed

The worker downloads the appropriate document and extracts its text.

The extracted text is divided into chunks. Each chunk should carry identifiers that connect it to its parent document and version.

For example:

{
  "id": "policy-123-v2-chunk-001",
  "documentId": "policy-123",
  "documentVersion": 2,
  "chunkNumber": 1,
  "content": "Employees are entitled to 20 days of annual leave."
}

The worker generates an embedding for each chunk and writes the chunk, its metadata, and its vector to Azure AI Search.

The exact index schema depends on the retrieval strategy, embedding dimensions, filtering requirements, and whether hybrid search is used.

Step 4: Mark the version as indexed

After the worker confirms that all required indexing operations succeeded, it updates the document metadata to record the indexed version and processing status.

For example:

{
  "id": "policy-123",
  "currentVersion": 2,
  "indexedVersion": 2,
  "status": "ACTIVE"
}

This update must be conditional on the document still being at version 2. If a newer version has already been uploaded, the older job must not mark itself as the current version.

Why idempotency matters

Distributed systems can deliver the same job more than once. A worker may finish indexing but fail before acknowledging the queue message. The message can then be delivered again.

If every attempt creates new chunk IDs, duplicate content may accumulate in the index.

A practical solution is to derive chunk IDs deterministically from the document ID, version, and chunk identifier. Reprocessing the same version then targets the same index records instead of creating a new set each time.

This does not eliminate every failure mode, but it makes repeated indexing much safer.

Azure Service Bus supports duplicate detection for eligible tiers, but queue-level duplicate detection is not a substitute for idempotent worker logic. Microsoft Learn: Duplicate detection

Updating Documents Without Serving Stale Content

Document updates are the most interesting part of the lifecycle because they introduce a consistency problem.

Imagine the current leave policy is version 1, allowing 15 days. An administrator uploads version 2, allowing 20 days.

The worker starts processing version 2, but embedding generation takes time. Meanwhile, users continue querying the application.

What should those users see?

There are two common strategies. The right choice depends on whether the system prioritizes availability or strict freshness.

Strategy A: Make the new version active after indexing succeeds

In this approach, the existing active version remains available while the new version is processed.

The workflow is:

1. Create version 2 and mark it as processing.
2. Extract text and generate embeddings for version 2.
3. Index all required version 2 chunks.
4. Verify indexing success.
5. Atomically update the authoritative metadata to make version 2 active.
6. Clean up version 1's indexed chunks asynchronously.

This minimizes the time during which the document is unavailable.

However, it means that version 1 can remain available while version 2 is being processed. That may be acceptable for some knowledge bases but inappropriate for policies that must become effective immediately.

Strategy B: Disable the old version as soon as an update is accepted

For high-sensitivity documents, the application can immediately mark the old version as ineligible for retrieval.

It then processes the new version and activates it only after successful indexing.

The trade-off is that the document may temporarily have no eligible version. The application should return a clear message or use an approved fallback rather than silently serving obsolete information.

Prevent old jobs from overwriting new versions

Consider this sequence:

• Version 2 starts processing.
• Version 3 is uploaded before version 2 finishes.
• Version 3 finishes indexing first.
• Version 2 finishes later and attempts to update the metadata.

Without version checks, the late version 2 job could incorrectly reactivate stale content.

The worker must therefore use conditional updates or optimistic concurrency checks against the authoritative document version. A job may complete its indexing work, but it must not activate its version unless it is still eligible to do so.

Handle the lack of a cross-service transaction

Cosmos DB and Azure AI Search do not participate in a single distributed transaction in this architecture.

A metadata update can succeed while an indexing request fails. An indexing request can succeed while the subsequent metadata update fails.

The system should account for these partial failures using:

• Explicit lifecycle states.
• Conditional version updates.
• Retryable and idempotent operations.
• Periodic reconciliation between metadata and indexed records.

The important engineering principle is to make incomplete operations detectable and recoverable, rather than assuming every multi-step workflow succeeds atomically.

Deleting Documents Safely

Deletion requires special treatment because removing a document from the source does not automatically remove its indexed content.

More importantly, waiting for physical index cleanup before preventing retrieval creates a window in which content that should no longer be available may still be returned.

Step 1: Make the document ineligible for retrieval

When an authorized deletion request arrives, the application updates the authoritative metadata to indicate that the document is being deleted or has been logically deleted.

The retrieval path must check that status before using retrieved content.

For sensitive documents, this check must happen before the content is added to the LLM context. Merely hiding a document in the user interface is insufficient.

Step 2: Schedule physical cleanup

The application enqueues a deletion job that identifies the document and all indexed records associated with it.

The worker queries or otherwise identifies the corresponding chunk keys and deletes those records from Azure AI Search.

The deletion operation should be idempotent. Retrying a deletion for an already-removed chunk should not cause the workflow to fail unnecessarily.

Step 3: Verify cleanup

The worker checks whether the targeted records have been removed and records the cleanup result.

If an operation fails, the job should be retried according to a defined policy. Jobs that continue failing should be sent to a dead-letter queue for investigation and controlled recovery.

Azure AI Search supports deleting indexed documents by their document keys. Its documentation also describes how to verify deletion and why index statistics may not update immediately. Microsoft Learn: Delete documents

A simplified FastAPI deletion endpoint

The following example illustrates the API boundary. The helper functions represent application-specific implementations.

from fastapi import FastAPI, HTTPException

app = FastAPI()

@app.delete("/documents/{document_id}")
async def delete_document(document_id: str):
    document = await get_document_metadata(document_id)

    if document is None:
        raise HTTPException(
            status_code=404,
            detail="Document not found"
        )

    # Must enforce authorization and use a
    # concurrency-safe lifecycle transition.
    await mark_document_deleting(document_id)

    # The job must be published reliably.
    await enqueue_deletion_job(
        document_id=document_id,
        document_version=document["currentVersion"]
    )

    return {
        "documentId": document_id,
        "status": "DELETING"
    }

This is an illustrative example, not a complete production implementation. In particular, the metadata transition and message publication must be coordinated reliably.

One approach is the transactional outbox pattern: write the lifecycle change and an outbox event in the same Cosmos DB transactional batch when the data model and partitioning allow it, then publish the event asynchronously. The publisher can retry without losing the intent to schedule cleanup.

Deletion must also account for in-flight queries

Suppose a query retrieves a document's chunks just before the deletion request is processed.

The query might still hold those chunks in memory even after the document is marked deleted.

For stricter deletion guarantees, the application needs a defined policy for in-flight requests, cache invalidation, and authorization checks immediately before content is sent to the model. If the requirement is that no response may contain content after a deletion takes effect, that guarantee must explicitly cover requests already in progress.

Physical cleanup and retrieval eligibility are separate responsibilities. Both matter.

Coordinate Lifecycle Changes Reliably

Once ingestion and deletion run asynchronously, the queue becomes part of the system's correctness model.

A production worker should not assume that each message is delivered exactly once or that messages for a particular document always finish in order.

Design for at-least-once processing

A practical worker lifecycle is:

1. Receive a message using a lock-based delivery mode.
2. Validate the document ID, version, and operation.
3. Check whether the operation is still eligible to run.
4. Perform idempotent indexing or deletion.
5. Update lifecycle metadata conditionally.
6. Acknowledge the message only after successful processing.

If the worker fails before acknowledging the message, the message may be delivered again. That is expected behavior and should be handled by the design.

Add retry and dead-letter policies

Not all failures should be treated equally.

Transient network errors or throttling may succeed after a delay. Invalid documents, unsupported file formats, and persistent schema mismatches may require manual investigation.

The system should use bounded retries with appropriate backoff, record meaningful failure details, and route repeatedly failing messages to a dead-letter queue.

Use reconciliation as a safety net

Even with retries, the system needs a way to detect inconsistencies that escaped the normal workflow.

A reconciliation job can compare authoritative metadata against indexed document versions and lifecycle states.

It can identify cases such as:

• A document marked active whose current version is not fully indexed.
• Chunks belonging to an obsolete version.
• Search records for a logically deleted document.
• An operation that remained in a processing state beyond its expected duration.

Reconciliation should repair only states it can safely determine. It should not reactivate a version simply because its chunks exist in the index.

For applications using Azure Cosmos DB change feed, document metadata changes can also drive incremental background processing. The change feed captures inserts and updates in its standard latest-version mode; capturing deletions requires selecting and configuring an appropriate supported mode or using explicit lifecycle events. Microsoft Learn: Cosmos DB change feed

Where Microsoft Agent Framework Fits

A common mistake is to assume that every part of an agentic RAG application should be implemented as an AI agent.

Document uploads, version transitions, embedding generation, and index deletion are predictable operations with explicit success and failure conditions. These are generally best implemented with conventional application logic and background workers.

Microsoft Agent Framework becomes useful when the query itself requires multi-step reasoning, tool selection, or orchestration.

For example, a user might ask:

“Compare the current leave policy with the previous version and explain the changes that affect employees.”

An agent could use approved tools to retrieve the current policy and the permitted historical version, compare the relevant passages, and generate an evidence-based explanation.

The application must still control which documents and versions are eligible for retrieval. The agent should not independently decide whether a deleted document is authorized for use.

This distinction is important: deterministic workflows protect data integrity; agentic workflows help solve tasks that benefit from reasoning and tool use.

Microsoft Agent Framework provides both agent and workflow capabilities, allowing developers to choose an appropriate orchestration model for the task. Microsoft Agent Framework documentation

Production Readiness: What Should We Measure?

A pipeline that successfully indexes a document once is not necessarily production-ready.

The system needs observable lifecycle states and measurable recovery behavior.

Useful operational metrics include ingestion success rate, queue age, indexing latency, retry counts, dead-letter queue depth, time spent in each lifecycle state, and deletion cleanup latency.

Also test race conditions deliberately. Upload a new version while an earlier job is still running. Delete a document while a query is in flight. Simulate an indexing failure after metadata has been updated. Restart a worker after it has indexed content but before it acknowledges the message.

These tests expose the problems that a happy-path demonstration will not reveal.

Bringing It All Together

Keeping a RAG system accurate is not simply a matter of re-running embeddings whenever a file changes.

It requires a deliberate lifecycle design that connects source documents, versioned metadata, asynchronous processing, search-index operations, and retrieval-time authorization.

Azure Blob Storage can preserve the source documents, Cosmos DB can track authoritative lifecycle state, Service Bus can distribute background work, and Azure AI Search can provide retrieval. Microsoft Foundry supplies the models, while Microsoft Agent Framework can support the parts of the application that genuinely benefit from agentic orchestration.

The architecture becomes reliable when it assumes that failures will happen: messages may be repeated, jobs may finish out of order, index operations may partially succeed, and cleanup may take time.

Version checks, idempotent operations, explicit lifecycle states, reliable event publication, and reconciliation turn those failure modes into manageable engineering problems.

The objective is not merely to keep the index populated. It is to ensure that every answer is grounded in information that is current, eligible, and authorized for the user who asked the question.

Further Reading

• Azure AI Search: Delete documents
• Azure Cosmos DB: Change feed
• Azure Service Bus: Duplicate detection
• Microsoft Agent Framework

I'd be interested to hear from engineers building production RAG systems: how do you coordinate document updates and deletions across your source of truth and search index?