[Bug][0.15.19][Windows CP932] GraphRAG Zero Ask returns an empty response after generating an answer

Summary

On a Japanese Windows installation, Bookshelf ask / ask_content with the GraphRAG Zero provider returns an empty answer even though retrieval succeeds and the language model generates a non-empty answer.

The worker logs that a non-empty answer was generated, then raises a CP932 UnicodeEncodeError for an EM DASH (U+2014). Bookshelf subsequently records provider success but returns no answer, citations, or source items.

The exact failing statement and output destination have not been identified. The character may occur in diagnostic logging rather than in the generated answer. This report distinguishes observed behavior from hypotheses; it does not establish that response serialization or standard output is the failing operation.

• Reproduced: October 16, 2026; report updated October 17, 2026.
• Current status: reproducible in the tested environment; no working fix verified.
• Workaround tested: a full restart with Python UTF-8 environment variables did not restore Ask.

Environment

• Product: Microsoft Discovery App Preview
• App version: 0.15.19
• VS Code shell version: 1.124.0
• Operating system: Windows 11 Enterprise, WindowsVersion 2009, OsBuildNumber 26300, 64-bit. 
• Observed console code page: 932
• Console output encoding: Shift-JIS
• Bookshelf provider: GraphRAG Zero
• GraphRAG Zero package: 0.9.0-ci.20261006.4211.1.abbf9992
• GraphRAG Zero source commit: abbf9992df0f5520898ce574ebec1ce4e2c74e16
• Worker packaging: PyInstaller, Windows x64, as recorded in the installed runtime manifest
• Embedding configuration: shelf metadata names Azure OpenAI; the active router binds grz-embed to a local ONNX embedding deployment.
• Answering route: Copilot through the local OpenAI-compatible shim
• Initial environment: PYTHONIOENCODING and PYTHONUTF8 unset; post-restart observations are documented below

Reproduction Steps

1. Start Microsoft Discovery App Preview on a Windows machine using code page 932.
2. Create a Bookshelf backed by GraphRAG Zero.
3. Ingest and index a PDF successfully.
4. Confirm that search returns relevant document chunks.
5. Run ask or ask_content, for example:
   • このBookshelfに登録されている資料の主要な内容を、出典に基づいて要約してください。
6. Observe that the call takes several seconds while the model generates an answer.
7. Observe that the returned result is empty:
   • answer: ""
   • no citations
   • no source items
   • totalCount: 0

The Japanese prompt means: "Please summarize the main contents of the materials registered in this Bookshelf, based on the sources."

The documented call used one shelf and maxResults=10. The caller requested strategy=hybrid and includeSources=true, while the SDK start log records strategy=Auto; this discrepancy is recorded without attributing the encoding failure to it. These steps describe the tested setup; reproduction with an arbitrary PDF or another machine has not been verified.

Expected Behavior

Bookshelf should return the generated answer with the applicable citations and source items. Worker text I/O should support Unicode independently of the Windows active code page, with matching encoding and decoding at both ends.

If a worker fails or produces an invalid response, Bookshelf should surface an actionable error rather than report successful execution with an empty answer.

Actual Behavior

The worker reports a non-empty generated answer, then logs a CP932 encoding exception. Bookshelf returns an empty result without surfacing that exception in the returned result.

list, get, status, provider health, and raw search succeeded. Ingestion/indexing also completed successfully before the failing Ask calls. A similar encoding exception was observed during a document-removal operation in earlier troubleshooting.

Error Details

The generic reproduction query from step 5 was executed after the restart. The SDK start log preserves the original Japanese query, but the captured worker stdout line displays mojibake. This does not by itself establish that the query passed to retrieval or the model was corrupted.

Selected messages from the same call, with log prefixes omitted:

bookshelf.ask.start query=このBookshelfに登録されている資料の主要な内容を、出典に基づいて要約してください。 
[GRZ] ask: query='����Bookshelf�ɓo�^����Ă��鎑���̎�v�ȓ��e���A�o�T�Ɋ�Â��ėv�񂵂Ă��������B' maxResults=10
[GRZ] ask: retrieving chunks from vector index
[GRZ] ask: falling back to direct OpenAI-compat answering endpoint
[GRZ] ask: OpenAI-compat answer generated (1883 chars)
[GRZ] Worker exception traceback: UnicodeEncodeError: 'cp932' codec can't encode character '\u2014' in position 20: illegal multibyte sequence bookshelf.ask.complete elapsedMs=11434 providerSuccessCount=1 providerErrorCount=0 hasAnswer=False answerLength=0 citationCount=0 itemCount=0

Attachable evidence containing only the generic query: sanitized log excerpt

Observations and Root Cause Hypotheses

The attached log establishes the following sequence for the generic reproduction:

1. GraphRAG Zero logs vector-index retrieval and fallback to the direct OpenAI-compatible answering endpoint.
2. It logs OpenAI-compat answer generated (1883 chars).
3. It logs a UnicodeEncodeError for U+2014 under CP932, at position 20.
4. Bookshelf completes after 11,434 ms with providerSuccessCount=1, providerErrorCount=0, and no answer, citations, or items.

Two areas warrant investigation:

1. Encoding at the worker/host boundary: a CP932 encoding operation fails after answer generation. Candidate locations include diagnostic logging, a standard stream, or a response-file write. The stdout mojibake also warrants checking the host's decoding configuration.
2. Failure propagation: the logged worker exception is followed by a successful provider count and an empty result. The handling of worker exceptions, exit status, and response-payload validation needs inspection.

Not yet established:

• The precise failing source line, stream/file, worker exit code, or response-file contents.
• Whether U+2014 belongs to the answer or a fixed diagnostic message.
• Whether the frozen worker inherited or honored the Python environment variables.
• Whether this reproduces on other Windows locales, operating systems, documents, or app versions.

Impact

High in the tested environment — Bookshelf Ask is unusable in the reproduced calls despite successful indexing, working search, and logged answer generation. Users receive a silent empty result and may incorrectly conclude that no relevant knowledge was found. The broader affected population is not yet known.

Raw search and direct source inspection remain possible, but they do not replace answer-first retrieval with citations.

Suggested Investigation and Fix Validation

• Capture the full traceback, failing output operation, worker exit code, effective stream encodings, and response-file status using non-sensitive test data.
• Ensure the packaged worker and host agree on UTF-8 for text streams and response files. If supported by the stream objects, explicit sys.stdout.reconfigure(encoding="utf-8") / sys.stderr.reconfigure(encoding="utf-8") may be appropriate; validate in the actual PyInstaller build rather than only in a standalone Python script.
• Prevent optional diagnostic logging failures from discarding a generated answer. ASCII-safe JSON alone would not address an encoding failure in a separate log message.
• Treat worker exceptions or invalid/missing response payloads as failures and surface an actionable error to the caller.
• Add Windows CP932 regression tests with Japanese text and U+2014 in queries, generated answers, and diagnostic messages. Exercise Ask, search, and removal separately.
• Verify that a successful Ask returns its answer and applicable sources, and that a deliberately failed worker cannot be counted as a silent success.

These are proposed investigation/fix directions, not an implemented or verified patch.

Attempted Workaround — Unsuccessful

Discovery was fully restarted using a launcher configured with:

• PYTHONIOENCODING=utf-8
• PYTHONUTF8=1

A newly opened integrated terminal confirmed:

PYTHONIOENCODING=utf-8 PYTHONUTF8=1

The terminal still reported Shift-JIS / code page 932. No global Windows locale change was performed. The worker's own effective environment and stream encodings were not directly inspected.

Raw search still returned a relevant result. The subsequent generic Ask produced the 1883-character-generation log and the same CP932 exception shown above, followed by an empty result.

Conclusion: this restart procedure did not resolve the issue. It does not prove why the worker continued using CP932; frozen-runtime isolation or explicit locale-based stream configuration remain hypotheses. These environment variables should not be presented as a verified workaround.

Additional Context

• Reindexing completed successfully with 111 chunks and a graph index before the failing Ask calls.
• Search returned relevant chunks for the same document and query terms.
• Multiple Japanese and English Ask prompts reproduced the empty-result behavior and CP932 exception. The generated answer contents were not inspected to establish the origin of the unencodable character.
• The public 0.15.19 preview manifest was the latest available build at the time of reproduction.

Source Availability and Requested Follow-up

On October 17, 2026, the public microsoft/discovery repository was inspected at commit 62cdbc671d0582d53ed2a6a89e0af99fc8d009eb (complete recursive tree: 2,186 entries). It contains community/catalog content and app distribution documentation. The Bookshelf Researcher entry is a prompt-based consumer of knowledge bases, not the local Bookshelf runtime implementation. The GraphRAG Zero worker source and the relevant Bookshelf SDK implementation were not found in that public tree.

The installed worker manifest identifies runtime/grz/grz_worker.py, source path vendor/resilience/graphrag-zero, and source commit abbf9992df0f5520898ce574ebec1ce4e2c74e16. These are provenance clues, not verified publicly accessible source locations.

Please route this report to the Discovery App Bookshelf / GraphRAG Zero runtime team to investigate encoding and failure propagation. Please advise whether an updated runtime build, a supported workaround, or an accessible source repository is available. No product code or installed binaries were modified during this investigation.