UmbraAudio/specs/27-translation-pipeline-multipass.md

Multi-pass semantic translation - design

Status: design, not implemented. Written 2026-10-12.

The problem

A word-for-word translation of this material produces sentences that are grammatical and wrong. In an audiobook nobody can skip a sentence, so a meaning error is a defect the listener hears and cannot escape. Register, tone and the relationship between a line and what precedes it all carry meaning, and all of it is lost by mapping words one to one.

The requirement is therefore: analyse the intent of a passage, then render that intent in the target language. One pass cannot do this, because the model doing the rendering cannot simultaneously hold the whole scene and attend to the sentence in front of it.

Proposed pipeline

Each stage has one job and produces an artefact that can be inspected. Nothing downstream may silently repair an upstream failure; each stage records why it was uncertain.

Pass 0 - Scene assembly (no LLM)

Group utterances into scenes and attach the surrounding narration. The sample chapter shows why: "One of my robots that Kainda decided to name Nanny is here. Say hello Nanny." is meaningless alone. The line before it says who is about to speak. A per-utterance translator gets this wrong every time.

Output: scene id, ordered utterances, speaker list, prior scene summary.

Pass 1 - Intent analysis (LLM, structured output)

For each utterance, produce a structured record, not prose:

• function - greeting, introduction, aside, punchline, instruction
• register - formal / neutral / casual, and whether it shifts mid-scene
• emotion - the value already used by the per-utterance variation controls
• addressee - who the line is aimed at, resolved
• depends_on - which earlier utterance must be understood for this to land
• literal_risk - would a direct rendering read as wrong, and why

depends_on is the important one: it tells Pass 2 which context to re-read, and it tells review what a human must check.

Pass 2 - Draft translation (LLM, context-aware)

Render Pass 1 into the target language, given the scene and the intent record - not the English sentence alone. Output carries depends_on forward so inconsistency can be detected.

Pass 3 - Register and voice consistency (LLM)

Check that the same character speaks consistently across a chapter: formality level, habitual phrasing, sentence rhythm. Characters are distinguished by voice as much as by content; a translation that makes everyone sound like the same polite translator has lost the book.

Pass 4 - Verification against source meaning (LLM)

The step a word-for-word pipeline has no equivalent for. Re-read the English source and the candidate translation and answer: did anything change in meaning? Specifically:

• did a pronoun's referent survive?
• did an implication become an assertion?
• did irony become sincerity, or the reverse?
• did a cultural or technical term lose its referent?

This is a critic pass over the whole chapter, not the same call that produced the draft. A model asked to check its own output in the same call is being asked to notice its own mistake while defending it.

Pass 5 - Human review queue

Only what Pass 4 flagged, plus everything Pass 1 marked literal_risk. The goal is a review queue a human can actually work through, not a wall of text. Target: same order as the speaker review, which landed at ~5%.

What must be preserved into the audio stage

The audio stage reads this and nothing else, so anything dropped here is gone:

Field	Why it matters
speaker	voice identity; must be the canonical primary_name, not an alias (see audit §6)
index	must line up 1:1 with the English utterance, or audio cannot be checked against source
emotion, pace, instruct	already drive Qwen3-TTS delivery; they must survive translation, and emotion names must be mapped, not translated literally
function	lets the narrator deliver a line correctly without re-deriving it

Hard problems specific to these languages

• Arabic is right-to-left and needs its own normalisation pass. Mixing Latin punctuation and numerals into RTL runs is a known corruption source.
• Chinese and Japanese need text normalisation the English pipeline has no reason to do: full-width forms, kana/kanji boundaries, punctuation that is not a space.
• Korean honorific level is lexical, not tonal. Speaker register in Pass 3 has to decide speech level explicitly, because a wrong level is a social error, not a stylistic one.
• Norwegian, Polish, Dutch have long compound words. The TTS engine will mispronounce what the translator joins or splits wrongly, so Pass 2 must not leave segmentation to chance.

The sample that exists

build_sample_translations.py produced the sample chapter in all 10 Qwen3-TTS languages. Those are single-pass LLM drafts, i.e. Pass 2 with Pass 1 skipped - which is exactly the failure mode described above, and they are labelled as such in every file. They exist to exercise the audio stage and agree the file format early, not to be used.

They are useful as a bad-case corpus: Pass 4 run over them should find real errors. If the critic cannot find errors in known-imperfect text, it is not doing its job.

Measured (2026-11-06): the sample files are not the book

The sample files under samples/translations/ were measured against the speaker-verified English in the project's database. 0 of 12 utterances match the database text exactly; the sample English is a hand-cleaned paraphrase of it, with the source's typos silently repaired ("the AI the runs the ship" -> "the AI that runs the ship", "Kianda's aunt" -> "Kainda's aunt", "I trade salvage" -> "I trade salvage"). The sample files also carry Kainda where the database carries the canonical Kainda Walters Foster Murphy, and Narrator where the database carries NULL.

So those files are wrong twice over: they are single-pass, and they were translated against a different English base than the one the audiobook speaks. They must not be treated as a translation of anything. The only other translation in the system is one placeholder row whose text is the English source under a zh-Hans code, which the resolver correctly ignores.

The real Deal Hunter Chinese edition

The production translation is not in this repository. It is built by UmbraTrans on oldbox (productions/deal-hunter-zh), and it does use the intent-based process:

Pass	What it does	Model
0	Paragraph/scene/chunk assembly, source indices kept	none
1	Scene intent: summary, register, tone, participants, risks	lfm2.5-2.6b
2	Context-aware draft, given glossary, character voices, scene analysis, prior chunk summary	qwen3.5-9b
3	Chapter voice/term consistency (optional)	critic model
4	Bilingual critic: meaning, pronoun, irony, term, omission	deepseek-r1-8b

Its Pass 1 output carries per-paragraph risks with guidance, which is intent analysis, not word mapping. Its critic runs on a different model from the drafter, which is the point of Pass 4.

The missing pass, and why it matters

That pipeline works on paragraphs, because epub and pdf are paragraph media. The audiobook needs utterances: this project has verified who speaks every one of them, and each clip is voiced separately by that speaker's voice. A paragraph-level translation cannot be spoken.

Measured on the real book: 3,047 paragraphs cover 4,317 source utterances. 2,322 paragraphs map to one utterance, 180 to two and 545 to three. All 725 multi-utterance paragraphs are narration; dialogue is already one paragraph per speech turn.

So the missing piece is an alignment pass that divides each translated paragraph between the utterances it was built from. It is implemented in UmbraTrans as tools/translation/align.py (Pass 5), and its output is a delivery payload in the shape this project's translations.import_translations already accepts.

Two rules the alignment must not break:

• a source utterance may only be absorbed into another utterance by the same speaker - a line must not be handed to a different character;
• the partition must reproduce the translated text exactly (modulo whitespace), so the alignment can never rewrite the translation.

A source utterance the target does not voice is delivered as voiced = false with the reason. The row is kept, so the gap stays visible; it is not deleted.

Measured on the real book (2026-11-06), 42 sampled multi-utterance paragraphs across seven chapters: 41 aligned by the model, 1 fell back to the mechanical split, 0 partition mismatches. The one fallback was a model answer that did not reproduce the translated text; the content check caught it and the paragraph was flagged rather than accepted.

The drafter re-flows the passage, and the alignment cannot see it

There is a defect in the production Chinese that the alignment pass is structurally unable to find, and it has to be stated plainly because it changes what the delivery means.

Pass 0 splits the original passage at sentence boundaries, so many paragraphs are sentence fragments - The chair in the middle has, "operations", on the console. Measured: 439 of 3,038 translated paragraphs (14.5%) are fragments. A fragment is exactly where an intent-based drafter is told to write natural target-language prose, and once it does that, the id-to-text correspondence drifts: the Chinese is good, but it is attached to the wrong paragraph.

In ch-15 the drift runs from p0094 to p0104 - eleven paragraphs. The Chinese for p0102 ("Yes, Captain.") is the translation of p0101, and the Chinese for p0104 opens with p0103's sentence. The audiobook would say the right words at the wrong place.

The alignment pass could not see it. A one-to-one paragraph is a pure copy, so there is nothing to divide and nothing to disagree; the drift is invisible to an alignment check by construction.

The verification stage (align.py --verify) finds it. It starts at a sentence fragment - the only place the drift is possible - and follows the run until the passage re-synchronises. The verdict is one of three:

verdict	meaning	action
aligned	the translation renders this paragraph	keep
incomplete	it renders this paragraph but leaves part out	keep, flag for review
drift	it renders another paragraph's content	drop, with a reason

Only drift deletes a line, and the paragraphs are judged as a window rather than one at a time. Both of those choices are measured, not assumed:

• The first version asked two yes/no questions per paragraph ("is anything missing", "is anything extra") and was wrong in both directions. It flagged five paragraphs that were fine - it reported the adverb then as missing from a translation that contains it, and Enthusiastically as missing from a translation that contains it - and it missed two paragraphs that are drifted, because those paragraphs are short and their Chinese looks plausible when seen alone. Five of fifteen flags were wrong.
• A re-flowed passage cannot be judged one paragraph at a time. The model calls the two paragraphs before a real drift "aligned" when their Chinese is demonstrably the neighbouring paragraph's, and the start of a drift is not knowable from any single paragraph. So inside a window that contains a confirmed drift, a fragment the model called aligned is dropped as well.

On the real ch-15 window this now finds exactly the drift: ten paragraphs dropped (p0095-p0104), and p0094 kept and flagged because its Chinese is a correct translation of its first two sentences. An earlier version dropped p0094 too - losing two good lines - and kept the two drifted paragraphs the model wrongly called aligned.

The cost of the choice is explicit. A fragment inside a broken window can be dropped on an unreliable verdict, so the audiobook has a gap there. The alternative is a line in the audiobook that belongs to a different sentence, and a gap is visible while a wrong line is silent.

A dropped paragraph still produces a row: voiced = false with the reason, so the gap says why it is there rather than looking like a paragraph nobody translated.

The root cause is upstream. The drafter should be told that each id's translation must cover that id's English only, even where that leaves the Chinese reading as a fragment. Fixing the drafter is a re-translation; until then the verification stage is what stands between the drifted Chinese and the audiobook.

The pipeline is Chinese-only, and one production says otherwise

Measured on oldbox (2026-11-06), the second production directory is productions/deal-hunter-es. It declares target_language: es-ES, its glossary says "script": "Spanish" with Spanish style rules and the Spanish title Cazador de Ofertas, and it has a complete 32-chapter run (39,083 s).

Its text is Chinese. All 3,038 translated paragraphs are Chinese, in the same proportion as the Chinese production (3,034 with Han characters, 4 Latin-only). It is a second Chinese draft filed under a Spanish name - a different Chinese wording from the deal-hunter-zh run (2,792 of 3,038 paragraphs differ), but Chinese.

The cause is in the pipeline, not in the run: the drafter's instructions are hard-coded to Simplified Chinese in three places.

• prompts.py SYSTEM_DRAFTER: "把英文太空冒险小说译成简体中文" - translate the English space adventure novel into Simplified Chinese.
• prompts.py draft_prompt: "请把下面英文译为简体中文" and a JSON key of "zh".
• prompts.py SYSTEM_CRITIC: "对照英文原文和中文译文".

The glossary is passed into the prompt, but the instruction says Chinese, so the model followed the instruction and ignored the glossary's target. The critic is hard-coded the same way, so it checked the Chinese text and found nothing wrong - there was nothing wrong with it, for the language it was actually asked about.

Two consequences, both measured rather than inferred:

1. No non-Chinese translation has ever been produced by this pipeline. The nine other Qwen3-TTS languages exist only as the single-pass sample files described above.
2. The English translation output would be wrong for any other language, because the analyst, the drafter and the critic are all told they are working on Chinese.

The pipeline needs a target-language parameter before it can be run for anything but Chinese. Until then, a production directory's name and its target_language field are a claim, not a measurement - which is why the delivery ingest now checks the script of the delivered text against the declared language and refuses a delivery whose text is written in another language (umbraaudio/languages.py:script_mismatch).

Two further consequences, both measured on oldbox:

3. The UmbraTrans dashboard reports the Spanish lane as complete. umbratrans_web/project_status.py derives each language's state from the presence and chapter count of deal-hunter-<tag>/runs/ch-N/translation.json. deal-hunter-es has 32 of 32, so the status module reports es-ES as complete - and it never looks at what the text says. The other nine target languages correctly report not_started, because their directories do not exist. A status derived from a directory name and a file count is the same "declared value disagrees with the measured value" defect, one layer up: the ingest now checks the script, and the status module does not.
4. A real Spanish run would silently reuse the Chinese translations. pipeline.py caches by file existence (_read returns the cached JSON unless --force is passed), and the run directory is chosen by the caller (--run-dir), not derived from target_language. So a Spanish run pointed at productions/deal-hunter-es without --force would find the cached ch-N-cXXX.draft.json files, skip the drafting, and publish the Chinese under a Spanish name. Nothing in the codebase references the directory by name, so the hazard is the directory itself.

Fixed (2026-11-07): the pipeline is parameterised by target language

The three hard-coded Chinese instructions are gone. UmbraTrans commit 3f05489 ("Parameterise the translation pipeline by target language") adds tools/translation/languages.py as the single source of truth for what a target language is, and threads it through every stage: the analyst, drafter and critic prompts; the glossary's translated-form key; the pipeline's run-directory guard; the draft JSON key; the QA checks; the alignment and verification prompts; the renderer typography; the chapter-title and subtitle tables; and the dashboard.

Ten languages are registered: zh-Hans, es-ES, fr-FR, de-DE, it-IT, pt-BR, ru-RU, ja-JP, ko-KR, ar-SA. Three rules make it safe:

1. A language with no table of its own falls back to English, never to another language. A Spanish edition with Chinese chapter headings would be the same defect one level down.
2. The prompt scaffolding stays in Chinese. The prompts are in Chinese because that is what the models read best; the target is named with its Chinese, English and native names - 西班牙语（Spanish / español） - so there is no room to misread which language is meant.
3. The script is measured at every boundary where a value is declared. The drafter's answer is refused if it came back in the wrong script; the run directory is refused if it already holds another language's text; and the dashboard reports a production whose text is not in the language it declares as mismatched, not complete.

Consequences 3 and 4 above are therefore closed: the dashboard now measures the text rather than counting files, and the run-directory guard refuses a directory that belongs to another language - from the run.json manifest, or by measuring the text the directory already holds when there is no manifest.

The Chinese production is unchanged by this. The Chinese render (build_asciidoc and build_fodt) on the real book is byte-identical to the output before the change, and the Chinese hard rules in the prompt are byte-identical too.

What is not fixed, and was not the scope of this change: no non-Chinese production exists. The pipeline can now produce one; it has not been run. Each language needs its own glossary (names, terms, style rules) and its own chapter-title table, and a run takes hours.

Measured (2026-11-07): what the Chinese edition actually contains

The full-book delivery imported into UmbraAudio is 4,305 rows: 4,215 voiced, 90 not voiced, 0 unmatched. The 90 not-voiced rows are two different things, and they should not be read as one:

• 44 collapse rows. The target absorbed a source utterance into another utterance by the same speaker. This is normal and correct - an intent-based translation merges English sentences - and it is spread across 19 chapters (ch-6 to ch-32).
• 46 drift rows in 9 chapters. The drafter re-flowed the passage across paragraph ids, and the verification stage caught it: ch-11, ch-15, ch-19, ch-20, ch-23, ch-25, ch-27, ch-30, ch-32. These cover 33 paragraphs, 356 source words - 1.08% of the book's paragraphs, 0.53% of its words, about 2.4 minutes of speech at 150 wpm.

Two of the dropped paragraphs are upstream translation defects, not alignment defects, and they are visible in the delivery rows:

• ch-15-p0100 - source "on the console. She looks around and sees a thin layer of dust on everything. She asks," - Chinese "是控制台上的". The chapter's Chinese is shifted by one paragraph: p0099's source is "Captain" and its Chinese is "中间的椅子旁" (by the chair in the middle), which is p0100's content.
• ch-25-p0089 - source about cuboid shapes with a control panel - Chinese "怎么才能接通电源？" (how do I power it up?). The Chinese belongs to a different sentence entirely.

Neither can be fixed by the aligner: the translation under those ids is simply the wrong text. Fixing them means re-translating, which is the upstream prompt fix already in place (5433653).

Open questions

1. Which model runs Passes 1-4? This is where the local LLM question returns, and it is a different question from the TTS model choice in spec 26.
2. Is Pass 3 (voice consistency) better done as a constraint passed into Pass 2 than as a separate pass?
3. Does the critic in Pass 4 use the same model as the drafter? Same model checking itself is weaker; a different model is better and slower.
4. Do translations get reviewed against the English by a bilingual reviewer, or only flagged by the critic?