What production taught us about reviewing AI translations

When I first started thinking about translation review for Co-op Translator, I wanted a score. Something that could tell us whether a translated document was good enough, without asking a person to read every language after every update.

That seemed like a reasonable place to start. Co-op Translator translates documentation in repositories where the source keeps changing. A lesson gets a new section, a code example changes, or a contributor fixes a link. Each change can affect translations that have already been read, corrected, and used by learners. As a maintainer, I needed a practical way to decide whether the next update was safe to accept.

Lee Stott helped give those discussions a practical starting point when he shared Anthony Shaw's Using an LLM in GitHub Actions, which explored reviewing changed documentation and translations with a small model. Could a review attached to a repository change help maintainers who don't speak every target language decide what needed attention? Lee and I wanted to write about our approach, but it kept changing as we learned from actual failures. Those changes turned out to be the useful part of the story.

Our first idea was to use established text evaluation metrics. We considered BLEU, which compares machine translation with reference translations, and overlap-based measures such as ROUGE, originally developed for summarization. The attraction was obvious: run an evaluation and get a number we could track.

We didn't have a trusted translation for every document, in every language, at every source revision. Producing those references manually would require much of the work we were trying to make manageable. Asking another model to generate them would leave us needing to validate the references too. There can also be several perfectly good ways to translate the same passage, so matching one wording isn't the whole goal.

We set that approach aside for the live workflow. A carefully reviewed reference dataset could still help us compare models or catch regressions. It just wouldn't, by itself, tell us whether today's newly updated lesson was ready for readers.

The next idea was LLM-as-a-judge: ask a model to compare the source and translation and identify problems. We built an optional evaluation path that examines translated chunks and can recommend retranslation.

That raised two concerns. Another model pass adds API cost, depending on the models, token volumes, and scope of review. Did every file need it? And if I couldn't read the target language, how would I know whether the evaluator was right? Its findings could guide a reviewer, but a convincing explanation wasn't independent confirmation. I still saw a use for model-assisted evaluation; I didn't want its verdict to become the final approval.

Meanwhile, the issues arriving from real repositories were making the next steps more concrete. We saw tables that no longer rendered as tables, missing passages, text left untranslated, and output that had been shortened instead of fully translated. Meaning and terminology still mattered, but some of the failures we could act on immediately were more basic: the document hadn't survived the process intact.

One table bug was particularly instructive. The chunking code preserved the source's boundary whitespace, but the translation path added an extra newline when joining the translated chunks. If that boundary fell inside a Markdown table, the new blank line ended the table. The fix was to preserve the original boundaries during reassembly. A translation quality score would have been an indirect way to investigate what was ultimately a bug in our own code.

An earlier post with Hiroshi Yoshioka covered related lessons from Markdown and CJK formatting problems. That work reinforced something I now look for in every report: which part of this failure can we describe precisely enough for the software to detect or prevent it?

That has led us to put more effort into two parts of the workflow.

The first is preserving the work people have already done. A contributor may spend time fixing terminology or making a translated explanation easier to understand. If an unrelated source edit causes us to regenerate the entire translated document, we risk losing that improvement and asking someone to make it again.

There are different levels to this. The normal synchronization flow uses source metadata to identify translations that need updating. For finer-grained updates, the project also has an incremental Markdown path. When an integration supplies a previous source-and-translation baseline, it can match unchanged source blocks and reuse the corresponding blocks from the current translation, including edits a person has made. Changed source blocks are translated again.

That preservation is conditional. The source and translated structure must still align, and the previous baseline must be available. If the mapping is ambiguous, the implementation falls back to full translation. The ordinary CLI doesn't automatically have this block-level history. This is an area we're continuing to develop, and we need to be clear about when human edits can be preserved.

The second is checking the output while translation is happening, when the failed part is still small enough to retry.

We added these checks because, when translating long documents, we sometimes received shortened or incomplete responses instead of a full translation. The output could read fluently while leaving out parts of the source. We needed a way to catch these failures while recovery was still possible. Adding temporary markers gave us something concrete to check: did the response preserve the expected labels, in the expected order? If it didn't, we could reject it and retry.

In the Markdown translation path, we add temporary labels, called markers, to each line and to the start and end of a chunk. We ask the model to translate the text while keeping those labels unchanged and in order.

The response above includes the chunk's end label, but @@LINE_0003@@ is missing. We reject it and try again, without asking another model to judge the response. Recovery is bounded and can use smaller chunks where splitting is possible. When a response passes, we remove the labels and assemble the translated document using the original boundary whitespace. Readers never see these labels in the finished file.

We also have a separate, deterministic review workflow for repository-level checks such as missing or outdated translations, broken local links, and some Markdown and notebook integrity problems. These checks don't require an LLM call. They complement the checks during generation and leave room for people to review terminology, meaning, and readability.

Adding stricter checks brought its own lesson. Detecting a bad response is only part of the job. We also have to decide what happens when retrying doesn't fix it.

If a file keeps failing, repeatedly sending the same content through the same path consumes time and money without guaranteeing a better result. In the Markdown workflow, recognized unrecoverable translation failures are recorded in .co-op-translator.json under the language directory. The record includes the source hash, error details, translator version, and failure-policy version. An existing translated file is retained when generation fails before a replacement is written. It may be outdated, and it should still be treated that way.

On an ordinary later run, the recorded failure prevents the same unchanged attempt from being repeated indefinitely. A source change, a different translator version, or a changed failure policy can make the file eligible for another attempt. A forced update can also bypass that skip. This gives us a practical way to return to unresolved files after a relevant change, without pretending that the earlier attempt succeeded or that the next one is guaranteed to work.

At first, I was mostly worried about meaning. Had the translation misinterpreted an explanation? Had something that meant A in the source become B in the translated version? Those were the problems I expected a review system to help us find.

But the issues users reported kept drawing our attention to failures in the document itself: broken tables, missing chunks, and passages that hadn't been translated. When we investigated problems encountered while translating long documents, these were recurring issues we could reproduce and work on.

We decided to focus on those failures first. That led us to check responses during translation, attempt recovery when a check failed, and record unresolved failures so they could be revisited. These practical needs shaped the review system we have today.

Meaning, terminology, and readability still matter. Our priorities became clearer through the reports people sent us: preserve the document, catch the failures we can identify reliably, and give human reviewers a better starting point.