UmbraTrans/tools/translation/README.md

Deal Hunter translation tools

This directory contains the translation and rendering pipeline used to produce the translated editions of Deal Hunter. It is deliberately separate from the web application: the book text and the translations are copyrighted and are not committed to this repository.

The pipeline is language-parameterised. The target language is a single setting that reaches every stage - the prompts, the pipeline plumbing, the chapter titles, the renderer typography and the alignment prompts - so a Spanish edition is a different configuration, not a different code path.

The target-language registry

languages.py is the single source of truth for what a target language is:

field	what it decides
name, native, chinese_name	how the language is named to the model and to the reader
scripts, required_script, forbidden_scripts	what the text must and must not be written in
glossary_key	which key a glossary uses for the translated form of a name or term
quote_open/close, ellipsis, dash, paren_open/close, title_open/close	the punctuation conventions
sentence_terminators, closing_marks, clause_marks	how the text is split into sentences and clauses
font_serif, font_sans, epub_font_stack	the typography
asciidoc_lang, odf_language, odf_country, rtl	the language tags and writing direction
chapter_prefix, chapter_suffix, chapter_numeral	the chapter-numbering convention

Ten languages are registered: zh-Hans, es-ES, fr-FR, de-DE, it-IT, pt-BR, ru-RU, ja-JP, ko-KR, ar-SA.

Two rules the registry enforces:

• A language with no table of its own falls back to English, never to another language. A Spanish edition with Chinese chapter headings is the same mislabelling defect as a Chinese draft under a Spanish name, one level down. A language that has a table but is missing a chapter from it gets a visible marker in its own language instead - an untranslated heading is a gap the reviewer has to see, not an English title that looks finished.
• The prompt scaffolding stays in Chinese. The prompts are written in Chinese because that is what the analyst, drafter and critic models read best; the target language is named with its Chinese name, its English name and its native name - 西班牙语（Spanish / español） - so there is no room to misread which language is meant.

Where the data lives

productions/deal-hunter-<lang>/     # git-ignored
  source/book.json                   # extracted English source model
  glossary/glossary.json             # names, places, terms, style rules
  runs/<chapter>/*.json              # analyses, drafts, critiques, final text
  runs/run.json                      # what this run directory is for
  runs/usage.jsonl                   # per-call token/latency log
  output/*.asciidoc, *.epub, *.odt, *.pdf, *.fodt   # review artefacts

The source bundle is the UmbraAudio controller bundle:

/home/umbra/audio-read/bundles/dh-bundle/manifest.jsonl

Its sha256 is recorded in Status.txt and in source/book.json.

Passes

Pass	What it does	Model
0	Paragraph/scene/chunk assembly; keeps source utterance indices	none
1	Scene intent: summary, register, tone, participants, risks	lfm2.5-2.6b
2	Context-aware draft translation with glossary and analysis	qwen3.5-9b
3	Chapter voice/term consistency (optional)	critic model
4	Bilingual critic: meaning, pronoun, irony, term, omission	deepseek-r1-8b, fallback qwen3.5-9b
5	Align the translation back onto the verified source utterances	drafter model

Every analysis, draft and critique is a file under runs/<chapter>/. A run can be stopped and resumed; existing artefacts are reused unless --force is given. LLM responses are cached by prompt hash, so editing a prompt invalidates the old answer.

The drafter is told to translate intent, not words, and is given a glossary, character voices, the scene analysis, and the previous chunk's summary. The critic uses a different model and returns a severity-sorted issue list; it does not rewrite the text.

The run-directory guard

The pipeline caches by file existence and the run directory is chosen by the caller, so pointing a Spanish run at a directory that already holds a Chinese run would skip drafting entirely and publish the Chinese under a Spanish name. The runner therefore refuses to reuse a directory that belongs to another language, on two checks - one from the claim and one from the measurement:

• runs/run.json records what the directory is for; a mismatch is refused.
• A directory with no manifest (written before the guard existed) is measured: the script of the text it already holds is compared against the target language.

The same check runs on every draft the drafter returns, so a model that answers in the wrong language is refused at the draft rather than at the ingest.

Pass 5 - aligning back to the utterances

Passes 0-4 work on paragraphs, because epub and pdf are paragraph media. The audiobook needs utterances: UmbraAudio has already verified who speaks every one of them, and every clip is voiced separately by that speaker's voice. So the last pass divides each translated paragraph between the source utterances it was built from.

The division is by meaning, not by position. An intent-based translation merges and splits sentences for the target language's sake, so the sentence counts do not line up on their own - that is the whole point of Passes 1-2. A model is given the source utterances and the translation and asked which part of the translation renders which source utterance.

Two rules the model cannot break:

• a source utterance may only be absorbed into another utterance by the same speaker - a line must not be handed to a different character;
• a scene break is silence, and silence has no translation.

The partition is checked against the translated text: the segments must reproduce it exactly once whitespace is removed. A model answer that drops, duplicates or invents text is rejected and the paragraph falls back to a mechanical split, which is reported as needing review. The alignment never rewrites the translation - it only decides where the boundaries fall.

When the target absorbed a source utterance, that utterance is recorded as not voiced with the reason. UmbraAudio's utterance_translations.voiced flag is what carries it: the row is kept so the gap is visible and reconstructable, and no clip is generated for it.

python3 tools/translation/align.py \
  --book productions/deal-hunter-zh/source/book.json \
  --run-dir productions/deal-hunter-zh/runs \
  --bundle /home/umbra/audio-read/bundles/dh-bundle/manifest.jsonl \
  --out productions/deal-hunter-zh/aligned/delivery.json \
  --chapters all

The output is a delivery payload in the shape UmbraAudio's translations.import_translations accepts, so the translation can be ingested without either side inventing a second contract:

python scripts/import_translations.py --project "DealHunterShip_4.0e_whole-to-audio" \
    --delivery productions/deal-hunter-zh/aligned/delivery.json --write

--no-llm runs the mechanical split only. It never loses text, but it cannot tell which clause renders which English sentence, so everything it touches is reported as needing review.

Verifying the paragraph boundaries

--verify catches the failure mode that the alignment pass cannot see by construction: the drafter re-flowing a passage across paragraph ids. A one-to-one paragraph is a pure copy, so there is nothing to divide and nothing to disagree - the drift is invisible to the aligner and visible only to a check that reads the paragraph against its neighbours.

The verdict is one of three: aligned, incomplete or drift. Only drift deletes a line, and a dropped paragraph still gets a row - voiced=false with the reason - so the gap says why rather than looking like a paragraph nobody translated. The paragraphs are judged as a window, because the start of a drift is not knowable from any single paragraph.

Commands

Every command takes --target-language. It defaults to the book's own target_language, and falls back to zh-Hans.

Extract the source:

python3 tools/translation/book.py \
  --target-language zh-Hans \
  --out productions/deal-hunter-zh/source/book.json

Translate a chapter:

python3 tools/translation/pipeline.py \
  --target-language zh-Hans \
  --book productions/deal-hunter-zh/source/book.json \
  --glossary productions/deal-hunter-zh/glossary/glossary.json \
  --run-dir productions/deal-hunter-zh/runs \
  --chapters ch-6

Selectors: all, front, body, back, or a comma-separated list of chapter indices / ids / titles (-4,-3,-2,-1 must be quoted or passed as --chapters=-4,-3).

Align:

python3 tools/translation/align.py \
  --target-language zh-Hans \
  --book productions/deal-hunter-zh/source/book.json \
  --run-dir productions/deal-hunter-zh/runs \
  --out productions/deal-hunter-zh/aligned/delivery.json \
  --chapters all

Render:

python3 tools/translation/render.py \
  --target-language zh-Hans \
  --book productions/deal-hunter-zh/source/book.json \
  --run-dir productions/deal-hunter-zh/runs \
  --out-dir productions/deal-hunter-zh/output \
  --sample

Mechanical QA:

python3 tools/translation/qa.py \
  --target-language zh-Hans \
  --book productions/deal-hunter-zh/source/book.json \
  --run-dir productions/deal-hunter-zh/runs \
  --json-out productions/deal-hunter-zh/runs/qa.json \
  --strict

Models

The default server is the LAN LM Studio instance at http://192.168.123.16:1234/v1. Override with:

UMBRA_TRANSLATION_BASE_URL
UMBRA_TRANSLATION_MODEL
UMBRA_TRANSLATION_ANALYST_MODEL
UMBRA_TRANSLATION_CRITIC_MODEL

Qwen models emit reasoning tokens by default; the client sends reasoning_effort: "none" so the budget is spent on the translation. The small analyst model rejects LM Studio's JSON-schema grammar, so its answer is parsed from text and repaired if necessary.

Rendering

The AsciiDoc file is the canonical text output. The ODT is produced from a Flat ODF document (.fodt) so that page layout, styles and the table of contents are under our control. LibreOffice converts it and updates the TOC through UNO; a plain --convert-to leaves the index empty. A PDF is exported for quick review.

Front matter, body and back matter use different styles. The typography is the target language's:

• Chapter headings follow the language's own convention - Chinese writes 第一章　打捞 with Chinese numerals, Spanish writes Capítulo 1 Salvage, Japanese writes 第1章　… with Arabic numerals, Korean writes 제1장　….
• Fonts and language tags are the target's: the stylesheet declares fo:language and fo:country for the target, so LibreOffice hyphenates and shapes the text with the right rules, and an RTL target gets fo:direction="rtl".
• The fixed editorial strings the renderer writes around the translation - the contents heading, the review-sample notice, the translator credit, the title-page lines, the marker for an untranslated heading - are in the target language too. They are part of the book the reader receives.
• Latin runs inside a CJK text are marked as English so Chinese line-breaking does not split an English name in the middle of a word. For a Latin-script target the whole text is Latin, so the marking is skipped - marking a Spanish paragraph as English would tell the renderer it is English.

The book's title stays in English for every language: Deal Hunter is also the ship's name and the AI's name, so translating it would rename a character. The subtitle is translated.

The script check

The registry can tell what script a text is written in, and that is enough to catch the mistake that actually happens: a pipeline configured for one language that silently produces another.

script_mismatch(code, text) returns a reason, or None when the text could be that language. It is deliberately about the script, not the language: it cannot tell Spanish from Portuguese, but it can tell Chinese from Spanish. Japanese without kana is Chinese; Korean without Hangul is Chinese; Chinese carrying kana is a Japanese translation in the wrong lane.

The check runs at three boundaries:

• the draft - the pipeline refuses a drafter answer that is in the wrong script, rather than caching it;
• the run directory - the guard above;
• the dashboard - umbratrans_web/project_status.py measures a sample of the production's translated text and reports a production whose text is not in the language it declares as mismatched, not complete.

The dashboard check exists because of a measured defect: the deal-hunter-es production on oldbox declared target_language: es-ES, had a Spanish glossary with Spanish style rules, and a complete 32-chapter run - and all 3,038 translated paragraphs were Chinese. The pipeline's drafter, critic and analyst instructions were hard-coded to Simplified Chinese, so the model followed the instruction rather than the glossary, and the critic checked Chinese text for Chinese correctness and found nothing wrong. A chapter count is a claim about the files; the script check is a measurement of the text.

What this is not

• It is not a native-speaker-reviewed translation. The critic and the QA script catch mechanical defects; a human editor is still required.
• It does not recover the original DOCX paragraphing. The bundle is sentence-level, so paragraphs are reconstructed and can be reviewed.
• It does not produce the audio. The alignment pass hands UmbraAudio one row per utterance; voicing, loudness and the audiobook masters are UmbraAudio's job.
• It does not translate a language it has no glossary for. A glossary supplies the names, the terms and the style rules; each language needs its own.