Introduction — The Monday morning bundle

Imagine a case manager opening a new request on Monday morning. The case is not represented by one neat digital form. It arrives as a bundle: a letter generated by a system, a report printed and scanned, a photograph embedded in a PDF, a multi-page invoice, and several pages of history that are useful to a human but irrelevant to the decision at hand.

The first instinct in an agentic solution is understandable: attach the bundle to an AI agent and ask for an answer. In a demonstration, that may work. In a regulated production workflow, it creates a fragile contract. The agent is asked to be a file reader, an OCR engine, a classifier, a data quality analyst, a policy researcher, and a decision assistant at the same time.

The architecture recommendation in this paper is to separate understanding from reasoning. Documents cross an evidence gateway first. The gateway turns mixed media into a compact, validated and traceable representation. The agent sees only the evidence needed for its task, with explicit uncertainty and links back to the source. Microsoft does not prohibit direct file attachments to agents; this default is a risk-based design choice for regulated workflows.

Fig-1 - Direct file-to-agent reasoning hides too many responsibilities inside one prompt.

Key idea

A PDF extension tells us how a file is packaged. It does not tell us whether the content is digital text, a scanned image, several embedded images, or a mixed bundle.

The problems hiding inside the file

1. The container is not the content

Two files with the same extension can behave very differently. One may expose clean text and headings. Another may contain only page-sized images. A third may alternate native text, photographs, signatures, tables and handwriting. Treating them as equivalent moves uncertainty into the agent runtime, where it becomes difficult to observe and control.

2. Context is finite and attention is selective

A long bundle can contain repeated headers, disclaimers, correspondence trails and prior history. Sending everything increases tokens and latency, but the more important cost is signal dilution. The agent must distinguish evidence from noise while also applying business rules. Relevant facts can be overlooked or outweighed by repeated but immaterial text.

3. Scans create silent quality failures

Low resolution, skew, shadows, handwritten annotations and multi-column layouts can alter extracted meaning. A date can lose a digit; an amount can lose a decimal point; a checkbox can be missed. Without field-level confidence and source location, these defects may look like ordinary, certain facts.

4. A summary alone is not evidence

Summaries are useful for navigation, but they compress nuance and can omit exceptions. A decision workflow needs typed facts, source references and a record of what was not confidently understood. The design goal is not merely shorter text. It is a stronger reasoning contract.

5. Direct attachments weaken governance

If every agent receives the complete document, data minimisation becomes difficult. Prompt logs and traces may carry unnecessary sensitive data. It is harder to apply role-specific access, retention and redaction when raw content is copied into multiple agent conversations.

• Variable input quality becomes variable decision quality.
• The same document may be interpreted differently by different agents.
• Failures are difficult to replay because the transformation steps are implicit.
• Citations are weak when a decision cannot point to a page, region or excerpt.
• Costs grow with pages rather than with the small set of facts required for the task

The pattern — Build an Evidence Gateway

The Evidence Gateway is a boundary between untrusted document media and trusted reasoning input. It is not one product and it is not only OCR. It is a sequence of controls that makes evidence fit for purpose.

 

Fig 2- Five responsibilities keep binary document handling outside the agent.

 

The evidence packet — A contract, not a transcript

The packet should be deliberately smaller than the document. It carries the facts required for a specific decision and enough provenance to prove where each fact came from. It can include short source excerpts, but it should not become a full text dump in a different envelope.

Fig 3 - A versioned contract exposes facts, uncertainty and origin.

Step by step — From arrival to recommendation

Step 1: Accept and quarantine

The workflow begins at a controlled ingestion endpoint. Validate the declared and detected media type, size and page count. Scan for malicious content. Calculate a cryptographic hash and assign stable document and case correlation identifiers. Store the original in an immutable or versioned landing zone with private network access.

• Reject or isolate unsupported, encrypted, corrupt or oversized content.
• Never rely on the filename extension as the sole media-type check.
• Keep raw files out of agent prompts, traces and application logs.

Step 2: Classify before extracting

Classify the bundle at page or logical-segment level. A single upload may contain several document types. Classification selects the right schema and extraction strategy. It also avoids forcing one large generic prompt to infer both document type and business meaning.

Step 3: Read all modalities

Use Azure Content Understanding in Foundry Tools for schema-aligned extraction, classification, confidence and grounding, and select Azure Document Intelligence in Foundry Tools where its OCR and layout models fit the document type. Preserve tables, selection marks and page structure when returned by the selected model. For image-only pages, assess image quality explicitly. The output at this stage is rich, but still requires the architecture’s task and governance controls before agent use.

Step 4: Map to a domain schema

Transform extracted content into versioned domain fields. Use data types, normalised dates and currencies, controlled enumerations and arrays for repeated items. Retain the extractor's raw value alongside the normalised value when reconciliation matters.

Step 5: Attach provenance and confidence

Every material field should answer two questions: “How sure are we?” and “Where did this come from?” Confidence must be attached at the smallest practical unit. Provenance should resolve to a document, page and region or minimal excerpt. This enables both automated quality gates and rapid human confirmation.

Step 6: Apply quality gates

Evaluate completeness, confidence, contradictions and business criticality. A missing low-impact note may be acceptable; an uncertain amount or date may not be. Thresholds should be field- and decision-specific rather than a single global confidence score.

• Auto-pass: required fields present, no critical conflicts, thresholds met.
• Review: recoverable ambiguity, low confidence on a material field, or contradictory sources.
• Reject/re-request: unreadable, incomplete, unsupported or integrity checks failed.

Step 7: Minimise for the task

Create a task-specific view of the evidence. A clinical relevance check, a reasonableness assessment and a payment assessment do not need identical fields. Remove unrelated identifiers and history. Tokenise or redact sensitive values unless the task explicitly requires them.

Step 8: Invoke specialist reasoning

Send the evidence packet to one narrowly instructed agent at a time. Where the selected Foundry Models inference API supports structured outputs, configure the JSON Schema explicitly; structured output is not automatic merely because an agent is used. Otherwise validate the agent response at the application boundary. Require a decision, rationale, cited evidence identifiers, uncertainties and next action. A final synthesis step may combine specialist outputs, but it should not invent new evidence.

Step 9: Persist and expose the audit trail

Persist extraction output separately from agent recommendations. Record schema version, model version, prompt or agent version, timestamps, outcome, cited evidence and human overrides. The operational status store should expose progress without duplicating sensitive source content.

Step 10: Learn from exceptions

Human corrections are high-value labelled data. Capture the original extraction, corrected value, reason and source anchor. Use this feedback to improve classification, schemas, thresholds and prompts. Do not silently overwrite the evidence history.

The human is a control plane, not a fallback

Human review should be intentionally designed. The reviewer should see the uncertain field beside the relevant crop or page, not search through the entire bundle. After correction, the workflow can resume from the failed gate rather than restart every stage.

 

Fig - 4 Confidence gates create a fast automation lane and a focused exception lane.

 

Fail closed on uncertainty

When material evidence is missing, contradictory or below threshold, the safe outcome is needs_review — not a confident guess.

Summary

The Evidence Gateway pattern converts uncontrolled files into controlled evidence. It recognises that document reading and decision reasoning are different workloads with different failure modes. The gateway handles media variability, OCR, classification, schemas, confidence, provenance and privacy. Agents handle bounded reasoning over approved facts.

• Land the original once and keep it immutable.
• Classify and understand mixed content before extraction.
• Shape only relevant facts into a versioned evidence contract.
• Carry confidence and source anchors with every material field.
• Route uncertainty to a focused human exception path.
• Give each agent the minimum evidence and policy context required.
• Persist extraction, reasoning and final action as separate auditable records.