diff --git a/tools/translation/README.md b/tools/translation/README.md index 9020f56..14eb96f 100644 --- a/tools/translation/README.md +++ b/tools/translation/README.md @@ -1,19 +1,57 @@ -# Deal Hunter Chinese translation tools +# Deal Hunter translation tools This directory contains the translation and rendering pipeline used to produce -the Simplified Chinese edition of *Deal Hunter*. It is deliberately separate -from the web application: the book text and the translations are copyrighted -and are **not** committed to this repository. +the translated editions of *Deal Hunter*. It is deliberately separate from the +web application: the book text and the translations are copyrighted and are +**not** committed to this repository. + +The pipeline is language-parameterised. The target language is a single +setting that reaches every stage - the prompts, the pipeline plumbing, the +chapter titles, the renderer typography and the alignment prompts - so a +Spanish edition is a different configuration, not a different code path. + +## The target-language registry + +`languages.py` is the single source of truth for what a target language is: + +| field | what it decides | +|---|---| +| `name`, `native`, `chinese_name` | how the language is named to the model and to the reader | +| `scripts`, `required_script`, `forbidden_scripts` | what the text must and must not be written in | +| `glossary_key` | which key a glossary uses for the translated form of a name or term | +| `quote_open/close`, `ellipsis`, `dash`, `paren_open/close`, `title_open/close` | the punctuation conventions | +| `sentence_terminators`, `closing_marks`, `clause_marks` | how the text is split into sentences and clauses | +| `font_serif`, `font_sans`, `epub_font_stack` | the typography | +| `asciidoc_lang`, `odf_language`, `odf_country`, `rtl` | the language tags and writing direction | +| `chapter_prefix`, `chapter_suffix`, `chapter_numeral` | the chapter-numbering convention | + +Ten languages are registered: `zh-Hans`, `es-ES`, `fr-FR`, `de-DE`, `it-IT`, +`pt-BR`, `ru-RU`, `ja-JP`, `ko-KR`, `ar-SA`. + +Two rules the registry enforces: + +- **A language with no table of its own falls back to English, never to + another language.** A Spanish edition with Chinese chapter headings is the + same mislabelling defect as a Chinese draft under a Spanish name, one level + down. A language that *has* a table but is missing a chapter from it gets a + visible marker in its own language instead - an untranslated heading is a + gap the reviewer has to see, not an English title that looks finished. +- **The prompt scaffolding stays in Chinese.** The prompts are written in + Chinese because that is what the analyst, drafter and critic models read + best; the target language is named with its Chinese name, its English name + and its native name - `西班牙语（Spanish / español）` - so there is no room + to misread which language is meant. ## Where the data lives ``` -productions/deal-hunter-zh/ # git-ignored +productions/deal-hunter-/ # git-ignored source/book.json # extracted English source model glossary/glossary.json # names, places, terms, style rules runs//*.json # analyses, drafts, critiques, final text + runs/run.json # what this run directory is for runs/usage.jsonl # per-call token/latency log - output/*.asciidoc, *.odt, *.pdf # review artefacts + output/*.asciidoc, *.epub, *.odt, *.pdf, *.fodt # review artefacts ``` The source bundle is the UmbraAudio controller bundle: @@ -45,6 +83,23 @@ character voices, the scene analysis, and the previous chunk's summary. The critic uses a different model and returns a severity-sorted issue list; it does not rewrite the text. +### The run-directory guard + +The pipeline caches by file existence and the run directory is chosen by the +caller, so pointing a Spanish run at a directory that already holds a Chinese +run would skip drafting entirely and publish the Chinese under a Spanish name. +The runner therefore refuses to reuse a directory that belongs to another +language, on two checks - one from the claim and one from the measurement: + +- `runs/run.json` records what the directory is for; a mismatch is refused. +- A directory with no manifest (written before the guard existed) is + **measured**: the script of the text it already holds is compared against + the target language. + +The same check runs on every draft the drafter returns, so a model that +answers in the wrong language is refused at the draft rather than at the +ingest. + ## Pass 5 - aligning back to the utterances Passes 0-4 work on *paragraphs*, because epub and pdf are paragraph media. @@ -98,12 +153,30 @@ python scripts/import_translations.py --project "DealHunterShip_4.0e_whole-to-au tell which clause renders which English sentence, so everything it touches is reported as needing review. +### Verifying the paragraph boundaries + +`--verify` catches the failure mode that the alignment pass cannot see by +construction: the drafter re-flowing a passage across paragraph ids. A +one-to-one paragraph is a pure copy, so there is nothing to divide and nothing +to disagree - the drift is invisible to the aligner and visible only to a +check that reads the paragraph against its neighbours. + +The verdict is one of three: `aligned`, `incomplete` or `drift`. Only `drift` +deletes a line, and a dropped paragraph still gets a row - `voiced=false` with +the reason - so the gap says why rather than looking like a paragraph nobody +translated. The paragraphs are judged as a **window**, because the start of a +drift is not knowable from any single paragraph. + ## Commands +Every command takes `--target-language`. It defaults to the book's own +`target_language`, and falls back to `zh-Hans`. + Extract the source: ```bash python3 tools/translation/book.py \ + --target-language zh-Hans \ --out productions/deal-hunter-zh/source/book.json ``` @@ -111,6 +184,7 @@ Translate a chapter: ```bash python3 tools/translation/pipeline.py \ + --target-language zh-Hans \ --book productions/deal-hunter-zh/source/book.json \ --glossary productions/deal-hunter-zh/glossary/glossary.json \ --run-dir productions/deal-hunter-zh/runs \ @@ -121,10 +195,22 @@ Selectors: `all`, `front`, `body`, `back`, or a comma-separated list of chapter indices / ids / titles (`-4,-3,-2,-1` must be quoted or passed as `--chapters=-4,-3`). +Align: + +```bash +python3 tools/translation/align.py \ + --target-language zh-Hans \ + --book productions/deal-hunter-zh/source/book.json \ + --run-dir productions/deal-hunter-zh/runs \ + --out productions/deal-hunter-zh/aligned/delivery.json \ + --chapters all +``` + Render: ```bash python3 tools/translation/render.py \ + --target-language zh-Hans \ --book productions/deal-hunter-zh/source/book.json \ --run-dir productions/deal-hunter-zh/runs \ --out-dir productions/deal-hunter-zh/output \ @@ -135,6 +221,7 @@ Mechanical QA: ```bash python3 tools/translation/qa.py \ + --target-language zh-Hans \ --book productions/deal-hunter-zh/source/book.json \ --run-dir productions/deal-hunter-zh/runs \ --json-out productions/deal-hunter-zh/runs/qa.json \ @@ -165,9 +252,58 @@ ODF document (`.fodt`) so that page layout, styles and the table of contents are under our control. LibreOffice converts it and updates the TOC through UNO; a plain `--convert-to` leaves the index empty. A PDF is exported for quick review. -Front matter, body and back matter use different styles. Chapter headings use -Chinese numerals (`第一章　打捞`). Body text is 11 pt Noto Serif CJK SC with -1.5 line spacing and a two-character first-line indent. +Front matter, body and back matter use different styles. The typography is the +target language's: + +- **Chapter headings** follow the language's own convention - Chinese writes + `第一章　打捞` with Chinese numerals, Spanish writes `Capítulo 1 Salvage`, + Japanese writes `第1章　…` with Arabic numerals, Korean writes `제1장　…`. +- **Fonts and language tags** are the target's: the stylesheet declares + `fo:language` and `fo:country` for the target, so LibreOffice hyphenates and + shapes the text with the right rules, and an RTL target gets + `fo:direction="rtl"`. +- **The fixed editorial strings** the renderer writes around the translation - + the contents heading, the review-sample notice, the translator credit, the + title-page lines, the marker for an untranslated heading - are in the target + language too. They are part of the book the reader receives. +- **Latin runs inside a CJK text** are marked as English so Chinese + line-breaking does not split an English name in the middle of a word. For a + Latin-script target the whole text is Latin, so the marking is skipped - + marking a Spanish paragraph as English would tell the renderer it is English. + +The book's title stays in English for every language: *Deal Hunter* is also the +ship's name and the AI's name, so translating it would rename a character. The +subtitle is translated. + +## The script check + +The registry can tell what script a text is written in, and that is enough to +catch the mistake that actually happens: a pipeline configured for one +language that silently produces another. + +`script_mismatch(code, text)` returns a reason, or `None` when the text could +be that language. It is deliberately about the **script**, not the language: it +cannot tell Spanish from Portuguese, but it can tell Chinese from Spanish. +Japanese without kana is Chinese; Korean without Hangul is Chinese; Chinese +carrying kana is a Japanese translation in the wrong lane. + +The check runs at three boundaries: + +- **the draft** - the pipeline refuses a drafter answer that is in the wrong + script, rather than caching it; +- **the run directory** - the guard above; +- **the dashboard** - `umbratrans_web/project_status.py` measures a sample of + the production's translated text and reports a production whose text is not + in the language it declares as `mismatched`, not `complete`. + +The dashboard check exists because of a measured defect: the `deal-hunter-es` +production on oldbox declared `target_language: es-ES`, had a Spanish glossary +with Spanish style rules, and a complete 32-chapter run - and all 3,038 +translated paragraphs were Chinese. The pipeline's drafter, critic and analyst +instructions were hard-coded to Simplified Chinese, so the model followed the +instruction rather than the glossary, and the critic checked Chinese text for +Chinese correctness and found nothing wrong. A chapter count is a claim about +the *files*; the script check is a measurement of the *text*. ## What this is not @@ -178,3 +314,5 @@ Chinese numerals (`第一章　打捞`). Body text is 11 pt Noto Serif CJK SC wi - It does not produce the audio. The alignment pass hands UmbraAudio one row per utterance; voicing, loudness and the audiobook masters are UmbraAudio's job. +- It does not translate a language it has no glossary for. A glossary supplies + the names, the terms and the style rules; each language needs its own. diff --git a/tools/translation/align.py b/tools/translation/align.py index bc36b7e..d16cd71 100644 --- a/tools/translation/align.py +++ b/tools/translation/align.py @@ -72,6 +72,7 @@ from dataclasses import dataclass, field from pathlib import Path from typing import Any +from languages import TargetLanguage, by_code, resolve from llm import LLM, LLMError # The aligner compares English source against translated text, which is the @@ -80,15 +81,23 @@ from llm import LLM, LLMError # its sentence boundaries best, so it is the default. DEFAULT_ALIGN_MODEL = "qwen/qwen3.5-9b" -SYSTEM_ALIGNER = """你是一位双语对齐编辑，负责把已经译好的中文段落还原到英文原文的每一条话语上。 -你的任务不是翻译，也不是改写，而是判断：中文译文里的哪一部分对应英文原文里的哪一条话语。 -译文为了中文的自然可能合并或拆分了句子，所以不能按句子序号机械对应，要按意思对应。 +CHINESE = resolve("zh-Hans") + + +def system_aligner(target: TargetLanguage) -> str: + """The aligner's instructions, in the target language's terms. + + The task is the same in every language - map a translated paragraph back + onto the source utterances it was built from - but the sentence merging + the translation does is the *target's* merging, so the prompt has to name + the target. + """ + return f"""你是一位双语对齐编辑，负责把已经译好的{target.chinese_name}段落还原到英文原文的每一条话语上。 +你的任务不是翻译，也不是改写，而是判断：{target.chinese_name}译文里的哪一部分对应英文原文里的哪一条话语。 +译文为了{target.chinese_name}的自然可能合并或拆分了句子，所以不能按句子序号机械对应，要按意思对应。 每一条英文话语都必须恰好出现一次；被合并进别条话语的，用 merged_into 指明并把 target_text 留空。 只输出 JSON，不要解释，不要 Markdown 代码围栏。""" -_SENTENCE_TERMINATORS = "。！？!?；;…" -_CLOSING_MARKS = "”’」』）)】》\"'" - def normalise_target_language(value: str) -> str: """``zh-CN`` -> ``zh``. @@ -96,7 +105,14 @@ def normalise_target_language(value: str) -> str: UmbraAudio stores the dropdown code, not the locale, so the payload has to carry the base code or the delivery lands in a language the review UI does not offer. + + A registered language answers from the registry, which is the single + source of truth for what a language's base code is; an unregistered one + falls back to the string surgery. """ + language = by_code(value) + if language is not None: + return language.base return value.strip().split("-")[0].split("_")[0].lower() @@ -121,12 +137,19 @@ def _compact(text: str) -> tuple[str, list[int]]: return "".join(chars), positions -def split_target_sentences(text: str) -> list[str]: +def split_target_sentences( + text: str, + target: TargetLanguage | None = None, +) -> list[str]: """Split translated text into sentences, keeping the terminator. - Chinese terminates with a full stop, and a closing quote may follow it, - so the closing marks are pulled into the sentence they close. + The terminators and the closing marks are the target's own: Chinese + terminates with a full stop and a closing quote may follow it, while a + Spanish sentence ends with a question mark or a full stop. """ + target = target or CHINESE + terminators = target.sentence_terminators + closing = target.closing_marks sentences: list[str] = [] buffer: list[str] = [] index = 0 @@ -134,8 +157,8 @@ def split_target_sentences(text: str) -> list[str]: ch = text[index] buffer.append(ch) index += 1 - if ch in _SENTENCE_TERMINATORS: - while index < len(text) and text[index] in _CLOSING_MARKS: + if ch in terminators: + while index < len(text) and text[index] in closing: buffer.append(text[index]) index += 1 piece = "".join(buffer).strip() @@ -148,15 +171,20 @@ def split_target_sentences(text: str) -> list[str]: return sentences -def _split_at_comma(text: str) -> list[str] | None: +def _split_at_comma( + text: str, + target: TargetLanguage | None = None, +) -> list[str] | None: """Split one sentence in two at its last clause boundary. An intent-based translation merges English sentences with a comma, a colon or a dash rather than with a full stop, because that is what reads - naturally in Chinese. So when the target has fewer sentences than the - source has utterances, the boundary is a clause boundary. + naturally in the target language. So when the target has fewer sentences + than the source has utterances, the boundary is a clause boundary - and + which characters those are depends on the language. """ - for mark in ("——", "—", "：", "，", "、", "；", ",", ";"): + target = target or CHINESE + for mark in target.clause_marks: positions = [m.start() for m in re.finditer(re.escape(mark), text)] for position in reversed(positions): if position <= 0: @@ -172,7 +200,11 @@ def _split_at_comma(text: str) -> list[str] | None: return None -def mechanical_split(text: str, count: int) -> list[str] | None: +def mechanical_split( + text: str, + count: int, + target: TargetLanguage | None = None, +) -> list[str] | None: """Divide *text* into exactly *count* pieces, or return None. This is the fallback when the model cannot be trusted. It is not good @@ -185,7 +217,7 @@ def mechanical_split(text: str, count: int) -> list[str] | None: if count == 1: return [text.strip()] if text.strip() else None - pieces = split_target_sentences(text) + pieces = split_target_sentences(text, target) if not pieces: return None @@ -210,7 +242,7 @@ def mechanical_split(text: str, count: int) -> list[str] | None: target_index = None replacement = None for index in ordered: - candidate = _split_at_comma(pieces[index]) + candidate = _split_at_comma(pieces[index], target) if candidate is not None: target_index = index replacement = candidate @@ -351,8 +383,14 @@ def source_utterances(paragraph: dict, bundle: dict[str, dict]) -> list[SourceUt return out -def alignment_prompt(paragraph_id: str, sources: list[SourceUtterance], translation: str) -> str: +def alignment_prompt( + paragraph_id: str, + sources: list[SourceUtterance], + translation: str, + target: TargetLanguage | None = None, +) -> str: """The prompt that asks for the partition of a translated paragraph.""" + target = target or CHINESE lines = [] for source in sources: lines.append(f"[{source.source_id}] (说话者：{source.speaker}) {source.text}") @@ -361,20 +399,20 @@ def alignment_prompt(paragraph_id: str, sources: list[SourceUtterance], translat 英文原文（每一条都是一条已经确认说话者的话语）： {chr(10).join(lines)} -中文译文： +{target.chinese_name}译文： {translation} -请把中文译文分配回上面每一条英文话语。输出 JSON： +请把{target.chinese_name}译文分配回上面每一条英文话语。输出 JSON： {{ "segments": [ - {{"source_id": "utt-xxxxx", "target_text": "对应的中文片段"}}, + {{"source_id": "utt-xxxxx", "target_text": "对应的{target.chinese_name}片段"}}, {{"source_id": "utt-yyyyy", "target_text": "", "merged_into": "utt-xxxxx"}} ] }} 要求： - segments 必须为原文的每一条话语各给一项，source_id 一一对应，不多不少。 -- target_text 必须是中文译文里**原样出现**的连续片段，按原文顺序拼接起来必须等于整段译文，不得删改一个字。 +- target_text 必须是{target.chinese_name}译文里**原样出现**的连续片段，按原文顺序拼接起来必须等于整段译文，不得删改一个字。 - 如果译文把两条原文合并成了一句，被合并的那条 target_text 留空，用 merged_into 指明它并入了哪一条。 - merged_into 只能指向同一条段落里、同一说话者、且有 target_text 的话语。 - 不要翻译，不要润色，不要补充，只做对应。""" @@ -486,8 +524,10 @@ def align_paragraph( bundle: dict[str, dict], llm: LLM | None = None, model: str = DEFAULT_ALIGN_MODEL, + target: TargetLanguage | None = None, ) -> ParagraphAlignment: """Divide one translated paragraph between its source utterances.""" + target = target or CHINESE paragraph_id = str(paragraph.get("id") or "") sources = source_utterances(paragraph, bundle) @@ -511,8 +551,8 @@ def align_paragraph( if llm is not None: try: data, _usage = llm.chat_json( - SYSTEM_ALIGNER, - alignment_prompt(paragraph_id, sources, translation), + system_aligner(target), + alignment_prompt(paragraph_id, sources, translation, target), model=model, temperature=0.0, cache_key=f"align-{paragraph_id}", @@ -524,7 +564,7 @@ def align_paragraph( problem = f"alignment model failed: {exc}" else: problem = "; ".join(problems) - mechanical = mechanical_split(translation, len(sources)) + mechanical = mechanical_split(translation, len(sources), target) if mechanical is None: return ParagraphAlignment(paragraph_id=paragraph_id, method="failed", problems=[problem]) return ParagraphAlignment( @@ -536,7 +576,7 @@ def align_paragraph( problems=[problem], ) - mechanical = mechanical_split(translation, len(sources)) + mechanical = mechanical_split(translation, len(sources), target) if mechanical is None: return ParagraphAlignment( paragraph_id=paragraph_id, @@ -594,6 +634,7 @@ def verification_prompt( translation: str, previous_text: str = "", next_text: str = "", + target: TargetLanguage | None = None, ) -> str: """Ask which paragraph *translation* actually renders. @@ -602,15 +643,16 @@ def verification_prompt( a drifted paragraph's translation is a perfectly good translation - of the paragraph next door. """ + target = target or CHINESE previous_block = f"\n【前一段原文】\n{previous_text}\n" if previous_text else "" next_block = f"\n【后一段原文】\n{next_text}\n" if next_text else "" return f"""【本段原文】 {source_text} {previous_block}{next_block} -【本段中文译文】 +【本段{target.chinese_name}译文】 {translation} -请判断这段中文译文到底对应哪一段原文，三选一： +请判断这段{target.chinese_name}译文到底对应哪一段原文，三选一： - "aligned"：译文对应本段原文，哪怕不完美。措辞不同、近义词、漏掉一个副词、 数字或单复数的小差异，都算 aligned。 @@ -642,6 +684,7 @@ def verify_translation( model: str = DEFAULT_ALIGN_MODEL, previous_text: str = "", next_text: str = "", + target: TargetLanguage | None = None, ) -> tuple[str, str]: """Which paragraph does *translation* actually render? @@ -649,7 +692,7 @@ def verify_translation( A model failure is reported as ``drift``: the delivery must not carry a line nobody checked. """ - prompt = verification_prompt(source_text, translation, previous_text, next_text) + prompt = verification_prompt(source_text, translation, previous_text, next_text, target) try: data, _usage = llm.chat_json( SYSTEM_VERIFIER, @@ -679,6 +722,7 @@ def _apply_drift_verification( bundle: dict[str, dict], llm: LLM, model: str, + target: TargetLanguage | None = None, ) -> None: """Check the paragraphs whose translation may have drifted across ids. @@ -705,6 +749,7 @@ def _apply_drift_verification( gap there. The alternative - keeping it - puts a line in the audiobook that belongs to a different sentence, which is worse. """ + target = target or CHINESE by_id = {alignment.paragraph_id: alignment for alignment in alignments} paragraphs = list(chapter.get("paragraphs") or []) index_of = {str(p.get("id") or ""): i for i, p in enumerate(paragraphs)} @@ -774,6 +819,7 @@ def _apply_drift_verification( model, previous_text=previous_text, next_text=next_text, + target=target, ) if verdict == VERDICT_ALIGNED: @@ -819,6 +865,7 @@ def align_chapter( llm: LLM | None = None, model: str = DEFAULT_ALIGN_MODEL, verify: bool = False, + target: TargetLanguage | None = None, ) -> tuple[list[ParagraphAlignment], dict[str, Any]]: """Align every translated paragraph of one chapter. @@ -826,6 +873,7 @@ def align_chapter( passage across paragraph ids are checked as well - see :func:`verify_drift_run`. """ + target = target or CHINESE alignments: list[ParagraphAlignment] = [] for paragraph in chapter.get("paragraphs") or []: paragraph_id = str(paragraph.get("id") or "") @@ -841,10 +889,10 @@ def align_chapter( ) ) continue - alignments.append(align_paragraph(paragraph, translation, bundle, llm=llm, model=model)) + alignments.append(align_paragraph(paragraph, translation, bundle, llm=llm, model=model, target=target)) if verify and llm is not None: - _apply_drift_verification(alignments, chapter, translated, bundle, llm, model) + _apply_drift_verification(alignments, chapter, translated, bundle, llm, model, target) methods: dict[str, int] = {} problems: list[str] = [] @@ -869,9 +917,12 @@ def build_delivery( bundle: dict[str, dict], *, source_ref: str = "", + target: TargetLanguage | None = None, ) -> dict[str, Any]: """The payload UmbraAudio's ``import_translations`` accepts.""" - target = normalise_target_language(str(book.get("target_language") or "")) + if target is None: + target = resolve(book.get("target_language") or "zh-Hans") + target_code = target.base utterances: list[dict[str, Any]] = [] unmatched: list[str] = [] @@ -924,7 +975,7 @@ def build_delivery( ) return { - "target_language": target, + "target_language": target_code, "source_system": "umbratrans", "utterances": utterances, "unmatched": unmatched, @@ -977,11 +1028,17 @@ def main(argv: list[str] | None = None) -> int: ), ) parser.add_argument("--source-ref", default="") + parser.add_argument( + "--target-language", + default=None, + help="the language the translations are in. Defaults to the book's own.", + ) args = parser.parse_args(argv) from pipeline import select_chapters # noqa: PLC0415 - keeps the tools importable standalone book = json.loads(args.book.read_text(encoding="utf-8")) + target = resolve(args.target_language or book.get("target_language") or "zh-Hans") bundle_path = args.bundle or Path(str(book.get("bundle") or "")) if not bundle_path.exists(): print(f"bundle not found: {bundle_path}", file=sys.stderr) @@ -1012,6 +1069,7 @@ def main(argv: list[str] | None = None) -> int: llm=llm, model=args.model, verify=args.verify, + target=target, ) chapters.append((chapter, alignments)) reports.append(report) @@ -1021,7 +1079,7 @@ def main(argv: list[str] | None = None) -> int: flush=True, ) - delivery = build_delivery(book, chapters, bundle, source_ref=args.source_ref) + delivery = build_delivery(book, chapters, bundle, source_ref=args.source_ref, target=target) args.out.parent.mkdir(parents=True, exist_ok=True) args.out.write_text( json.dumps(delivery, ensure_ascii=False, indent=2) + "\n", diff --git a/tools/translation/book.py b/tools/translation/book.py index 3b234d6..8c3df5e 100644 --- a/tools/translation/book.py +++ b/tools/translation/book.py @@ -21,44 +21,73 @@ import re from dataclasses import asdict, dataclass, field from pathlib import Path +from languages import TargetLanguage, resolve + DEFAULT_BUNDLE = Path("/home/umbra/audio-read/bundles/dh-bundle/manifest.jsonl") -# Chinese chapter titles. Kept here, not inferred, so every render is stable. -# Reviewable in one place; the English title stays on the paragraph. -TITLE_ZH = { - "Sample Track": "音频版序言：声音介绍", - "Opening Track": "开篇说明", - "Copyright": "版权信息", - "Author's Note": "作者的话", - "Salvage": "打捞", - "Family Reunion": "家庭团聚", - "Selling Salvage": "出售打捞物", - "Frustration": "挫败", - "Next Bounty": "下一个赏金", - "Time for a Change": "改变的时刻", - "Trade Hub Vince-4": "贸易枢纽文斯-4", - "Can I Join the Crew": "我能加入船员吗", - "Salvage from Crashed Fighters": "坠毁战机的残骸", - "Where Is the Captain?": "船长在哪里？", - "Turn in Fugitives": "上交逃犯", - "Her New Ship": "她的新船", - "Introducing the new XO": "新任副舰长", - "A New Ship Salvage": "新船打捞", - "The Poker Game Distraction": "扑克牌局掩护", - "Deal Hunter Bounty": "Deal Hunter 的赏金", - "Selecting Bounties": "挑选赏金", - "Shuttle Scans": "穿梭机扫描", - "The Pirate Trap": "海盗陷阱", - "The Vid Interview": "视频采访", - "Hunting Pirates": "追猎海盗", - "The Friendly Pirate": "友好的海盗", - "Aunt Vicky": "维姬姑妈", - "Preparation": "准备", - "The Coronation": "加冕典礼", - "Long Live the King": "国王万岁", - "Let’s Make It Better": "让它变得更好", - "Let's Make It Better": "让它变得更好", - "Closing Track": "后记：感谢收听", +# Chapter titles per target language. Kept here, not inferred, so every render +# is stable and reviewable in one place; the English title stays on the +# paragraph. +# +# The Chinese table is the one the pipeline has always run with. A language +# with no table falls back to the English title rather than to another +# language's title: a Spanish edition with Chinese chapter headings is the +# same mislabelling defect as a Chinese draft under a Spanish name, one +# level down. +TITLES: dict[str, dict[str, str]] = { + "zh": { + "Sample Track": "音频版序言：声音介绍", + "Opening Track": "开篇说明", + "Copyright": "版权信息", + "Author's Note": "作者的话", + "Salvage": "打捞", + "Family Reunion": "家庭团聚", + "Selling Salvage": "出售打捞物", + "Frustration": "挫败", + "Next Bounty": "下一个赏金", + "Time for a Change": "改变的时刻", + "Trade Hub Vince-4": "贸易枢纽文斯-4", + "Can I Join the Crew": "我能加入船员吗", + "Salvage from Crashed Fighters": "坠毁战机的残骸", + "Where Is the Captain?": "船长在哪里？", + "Turn in Fugitives": "上交逃犯", + "Her New Ship": "她的新船", + "Introducing the new XO": "新任副舰长", + "A New Ship Salvage": "新船打捞", + "The Poker Game Distraction": "扑克牌局掩护", + "Deal Hunter Bounty": "Deal Hunter 的赏金", + "Selecting Bounties": "挑选赏金", + "Shuttle Scans": "穿梭机扫描", + "The Pirate Trap": "海盗陷阱", + "The Vid Interview": "视频采访", + "Hunting Pirates": "追猎海盗", + "The Friendly Pirate": "友好的海盗", + "Aunt Vicky": "维姬姑妈", + "Preparation": "准备", + "The Coronation": "加冕典礼", + "Long Live the King": "国王万岁", + "Let’s Make It Better": "让它变得更好", + "Let's Make It Better": "让它变得更好", + "Closing Track": "后记：感谢收听", + }, +} + +# The book's subtitle per target language. The title itself stays in English +# - "Deal Hunter" is also the ship's name and the AI's name, so translating it +# would rename a character - and the subtitle is the translated reading aid. +# A language with no subtitle falls back to none rather than to another +# language's. +SUBTITLES: dict[str, str] = { + "zh": "交易猎人", + "es": "Cazador de Ofertas", + "fr": "Chasseur de Primes", + "de": "Prämienjäger", + "it": "Cacciatore di Taglie", + "pt": "Caçador de Recompensas", + "ru": "Охотник за Головами", + "ja": "賞金ハンター", + "ko": "보상금 헌터", + "ar": "صياد المكافآت", } MAX_NARRATION_SENTENCES = 3 @@ -100,7 +129,7 @@ class Chapter: index: int row: int title: str - title_zh: str + title_target: str matter: str paragraphs: list[Paragraph] = field(default_factory=list) scenes: list[Scene] = field(default_factory=list) @@ -112,7 +141,8 @@ class Chapter: @dataclass class Book: title: str - title_zh: str + title_target: str + subtitle_target: str author: str source_language: str target_language: str @@ -129,7 +159,8 @@ class Book: def to_json(self) -> dict: return { "title": self.title, - "title_zh": self.title_zh, + "title_target": self.title_target, + "subtitle_target": self.subtitle_target, "author": self.author, "source_language": self.source_language, "target_language": self.target_language, @@ -170,12 +201,26 @@ def _clean_title(title: str) -> str: return title.replace("’", "'").strip() -def _title_zh(title: str) -> str: +def _title_for(title: str, target: TargetLanguage) -> str: + """The chapter title in the target language. + + Two different fallbacks, for two different situations: + + - A language with **no title table** falls back to the English title. It + does not fall back to another language's title: a Spanish edition + carrying Chinese chapter headings is the same defect as a Chinese draft + under a Spanish name, one level down. + - A language **with** a table, and a chapter missing from it, gets a + visible marker. An untranslated heading is a gap the reviewer has to + see, not an English title that looks finished. + """ + table = TITLES.get(target.base) + if table is None: + return title key = _clean_title(title) - if key in TITLE_ZH: - return TITLE_ZH[key] - # Unknown title: leave a visible marker rather than invent a translation. - return f"[待译] {title}" + if key in table: + return table[key] + return target.string("missing_title", "[Untranslated] {title}").format(title=title) def load_manifest(path: Path = DEFAULT_BUNDLE) -> tuple[list[dict], dict]: @@ -185,7 +230,8 @@ def load_manifest(path: Path = DEFAULT_BUNDLE) -> tuple[list[dict], dict]: return records, {"path": str(path), "sha256": digest} -def build_book(path: Path = DEFAULT_BUNDLE) -> Book: +def build_book(path: Path = DEFAULT_BUNDLE, target_language: str = "zh-Hans") -> Book: + target = resolve(target_language) records, meta = load_manifest(path) book_rec = next(r for r in records if r.get("kind") == "book") chapter_recs = sorted( @@ -201,10 +247,11 @@ def build_book(path: Path = DEFAULT_BUNDLE) -> Book: book = Book( title="Deal Hunter", - title_zh="交易猎人", + title_target=_title_for("Deal Hunter", target), + subtitle_target=SUBTITLES.get(target.base, ""), author="B. D. Murphy", source_language="en", - target_language="zh-CN", + target_language=target.code, bundle=meta["path"], bundle_sha256=meta["sha256"], ) @@ -220,7 +267,7 @@ def build_book(path: Path = DEFAULT_BUNDLE) -> Book: index=index, row=int(cr["database_row_id"]), title=title, - title_zh=_title_zh(title), + title_target=_title_for(title, target), matter=matter, paragraphs=paragraphs, ) @@ -241,11 +288,16 @@ def _paragraphs(chapter_id: str, title: str, matter: str, rows: list[dict]) -> l rows = rows[1:] # The author's-note heading is an utterance, not a paragraph. - if matter == "front_note" and rows and rows[0]["text"].strip().rstrip(".").lower() in { - "author's note", - "authors note", - "author’s note", - }: + if ( + matter == "front_note" + and rows + and rows[0]["text"].strip().rstrip(".").lower() + in { + "author's note", + "authors note", + "author’s note", + } + ): rows = rows[1:] paragraphs: list[Paragraph] = [] @@ -323,13 +375,12 @@ def _paragraphs(chapter_id: str, title: str, matter: str, rows: list[dict]) -> l continue # Narration: group consecutive sentences into a readable paragraph. - if buf and ( - buf[0].get("speaker_name_resolved") != speaker - or buf[0].get("utterance_type") != utype - ): + if buf and (buf[0].get("speaker_name_resolved") != speaker or buf[0].get("utterance_type") != utype): flush() sentences = _sentences(text) - if buf and (buf_sentences + sentences > MAX_NARRATION_SENTENCES or buf_words + len(text.split()) > MAX_NARRATION_WORDS): + if buf and ( + buf_sentences + sentences > MAX_NARRATION_SENTENCES or buf_words + len(text.split()) > MAX_NARRATION_WORDS + ): flush() buf.append(row) buf_words += len(text.split()) @@ -370,7 +421,7 @@ def load(path: Path) -> dict: def summary(book: Book) -> str: lines = [ - f"Book: {book.title} / {book.title_zh}", + f"Book: {book.title} / {book.title_target}", f"Author: {book.author}", f"Bundle: {book.bundle}", f"Bundle sha256: {book.bundle_sha256}", @@ -379,9 +430,7 @@ def summary(book: Book) -> str: ] for c in book.chapters: words = sum(p.words for p in c.paragraphs if p.type != "scene_break") - lines.append( - f"{c.index:>4} {c.matter:16} {len(c.paragraphs):>6} {words:>7} {c.title} / {c.title_zh}" - ) + lines.append(f"{c.index:>4} {c.matter:16} {len(c.paragraphs):>6} {words:>7} {c.title} / {c.title_target}") body = [c for c in book.chapters if c.matter == "body"] lines += [ "", @@ -398,8 +447,13 @@ def main() -> None: ap = argparse.ArgumentParser(description="Extract the Deal Hunter bundle into a book model.") ap.add_argument("--bundle", type=Path, default=DEFAULT_BUNDLE) ap.add_argument("--out", type=Path, required=True) + ap.add_argument( + "--target-language", + default="zh-Hans", + help="the language the book is being translated into", + ) args = ap.parse_args() - book = build_book(args.bundle) + book = build_book(args.bundle, args.target_language) save(book, args.out) print(summary(book)) print(f"\nwrote {args.out}") diff --git a/tools/translation/languages.py b/tools/translation/languages.py new file mode 100644 index 0000000..22890cb --- /dev/null +++ b/tools/translation/languages.py @@ -0,0 +1,904 @@ +# Copyright 2026 Umbra Tools (umbratools.com) +# SPDX-License-Identifier: Apache-2.0 + +"""The target languages this pipeline can translate into. + +The pipeline was written for one language and hard-coded to it: the drafter +was told to translate into Simplified Chinese, the critic was told to compare +English with Chinese, the glossary's translated forms were read from a ``zh`` +key, the chapter titles came from a ``TITLE_ZH`` table, and the renderer +typeset CJK. A production directory that declared ``target_language: es-ES`` +was therefore translated into Chinese and filed under a Spanish name. + +This module is the single place that says what a target language *is*: its +name, its script, its punctuation, its typography, and the key its +translated forms live under in a glossary. Everything else reads it, so a +second language is a data change rather than a code change. + +It also carries the script check. A production's ``target_language`` field is +a claim, and a claim has to be measured: the text in the production directory +is either written in the declared language or it is not, and the dashboard +must not report a Chinese draft under a Spanish name as a completed +Spanish translation. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path + +# --- scripts --------------------------------------------------------------- + +SCRIPT_LATIN = "latin" +SCRIPT_CYRILLIC = "cyrillic" +SCRIPT_HAN = "han" +SCRIPT_KANA = "kana" +SCRIPT_HANGUL = "hangul" +SCRIPT_ARABIC = "arabic" + +SCRIPT_NAMES: dict[str, str] = { + SCRIPT_LATIN: "Latin script", + SCRIPT_CYRILLIC: "Cyrillic script", + SCRIPT_HAN: "Han characters", + SCRIPT_KANA: "Japanese kana", + SCRIPT_HANGUL: "Hangul", + SCRIPT_ARABIC: "Arabic script", +} + +_SCRIPT_RANGES: tuple[tuple[str, int, int], ...] = ( + (SCRIPT_LATIN, 0x0041, 0x005A), + (SCRIPT_LATIN, 0x0061, 0x007A), + (SCRIPT_LATIN, 0x00C0, 0x024F), + (SCRIPT_CYRILLIC, 0x0400, 0x04FF), + (SCRIPT_HAN, 0x3400, 0x4DBF), + (SCRIPT_HAN, 0x4E00, 0x9FFF), + (SCRIPT_HAN, 0xF900, 0xFAFF), + (SCRIPT_KANA, 0x3040, 0x309F), + (SCRIPT_KANA, 0x30A0, 0x30FF), + (SCRIPT_KANA, 0x31F0, 0x31FF), + (SCRIPT_HANGUL, 0x1100, 0x11FF), + (SCRIPT_HANGUL, 0x3130, 0x318F), + (SCRIPT_HANGUL, 0xAC00, 0xD7AF), + (SCRIPT_ARABIC, 0x0600, 0x06FF), + (SCRIPT_ARABIC, 0x0750, 0x077F), +) + + +# --- the registry ---------------------------------------------------------- + + +@dataclass(frozen=True) +class TargetLanguage: + """One language the pipeline can translate into. + + ``code`` is the BCP-47 tag the production declares. ``base`` is the + two-letter code the delivery uses, because the downstream ingest folds + locale codes to their base (``zh-Hans`` becomes ``zh``). + """ + + code: str + base: str + name: str + native: str + # The language's name in Chinese. The prompts are written in Chinese, so + # "译成西班牙语" reads as a Chinese instruction while "译成Spanish" reads + # as a typo; the English and native names ride along so the model has no + # room to misread which language is meant. + chinese_name: str + scripts: tuple[str, ...] + # A script that has to be present for the text to be this language at all: + # Japanese without kana is Chinese, and Korean without Hangul is Chinese. + required_script: str | None = None + # Scripts that would mean the delivery is a different language. + forbidden_scripts: tuple[str, ...] = () + # The key a glossary uses for the translated form of a name or term. + glossary_key: str = "" + # Typography. + quote_open: str = '"' + quote_close: str = '"' + ellipsis: str = "..." + dash: str = "—" + paren_open: str = "(" + paren_close: str = ")" + title_open: str = '"' + title_close: str = '"' + font_serif: str = "Noto Serif" + font_sans: str = "Noto Sans" + # The font stack used for the EPUB stylesheet, which has to name real + # installed families rather than the two the FODT declares. + epub_font_stack: str = "Noto Serif, serif" + # The language tag for the AsciiDoc header (``:lang:``) and for the + # EPUB's ``lang`` attribute. + asciidoc_lang: str = "en" + # The language and country codes ODF wants (``fo:language``, + # ``fo:country``). They are separate from the BCP-47 tag: ODF wants + # ``zh`` and ``CN``, not ``zh-Hans``. + odf_language: str = "en" + odf_country: str = "US" + # Chapter numbering: "第 N 章" is a CJK convention, not a universal one. + chapter_prefix: str = "Chapter " + # The character that closes a chapter number in a CJK language: 章 in + # Chinese and Japanese, 장 in Korean. + chapter_suffix: str = "" + # "arabic" (1, 2, 3) or "chinese" (一, 二, 三). Japanese uses Arabic + # numerals with the 第 prefix; Chinese uses Chinese numerals. + chapter_numeral: str = "arabic" + rtl: bool = False + # The characters that end a sentence in this language. + sentence_terminators: str = ".!?…" + # The characters that can close a sentence after its terminator: the + # target's quotation marks and brackets. + closing_marks: str = "\"'”’)]}»" + # The clause boundaries the mechanical fallback splits at, strongest + # first. A translation merges English sentences with these rather than + # with a full stop, so they are where a paragraph can be divided. + clause_marks: tuple[str, ...] = ("—", "–", ":", ";", ",") + # True for the languages whose line-breaking and punctuation rules differ + # enough from Latin that the renderer needs to be told. + cjk: bool = False + + @property + def prompt_name(self) -> str: + """How to name this language inside a prompt.""" + return f"{self.chinese_name}（{self.name} / {self.native}）" + + @property + def label(self) -> str: + """A short label for a document header.""" + return f"{self.name} ({self.code})" + + def chapter_number(self, n: int) -> str: + """The chapter number as this language writes it.""" + if self.chapter_numeral == "chinese": + return _chinese_numeral(n) + return str(n) + + def chapter_heading(self, n: int, title: str) -> str: + """The full chapter heading, in this language's convention. + + Chinese writes 第一章　打捞; Spanish writes Capítulo 1 Salvage. The + convention is the language's, not the renderer's. + """ + if self.cjk: + return f"{self.chapter_prefix}{self.chapter_number(n)}{self.chapter_suffix}　{title}" + return f"{self.chapter_prefix}{self.chapter_number(n)} {title}" + + def string(self, key: str, default: str = "") -> str: + """A fixed editorial string in this language. + + The strings the renderer writes around the translated text - the + contents heading, the review-sample notice, the translator credit - + are part of the book, so they cannot be left in Chinese when the book + is not Chinese. A language without a table gets the English string, + never another language's. + """ + table = RENDER_STRINGS.get(self.base) or RENDER_STRINGS["en"] + return table.get(key) or default + + +_CN_DIGITS = "零一二三四五六七八九" + + +def _chinese_numeral(n: int) -> str: + """The Chinese numeral for *n*: 1 is 一, 10 is 十, 21 is 二十一.""" + if n <= 0: + return str(n) + if n <= 10: + return "十" if n == 10 else _CN_DIGITS[n] + if n < 20: + return "十" + _CN_DIGITS[n - 10] + if n < 100: + return _CN_DIGITS[n // 10] + "十" + (_CN_DIGITS[n % 10] if n % 10 else "") + return str(n) + + +# The ten target languages the dashboard offers. The codes match the tags the +# web dashboard already uses, so a production directory named for a tag and +# the language the pipeline was told to produce are the same thing. +TARGET_LANGUAGES: tuple[TargetLanguage, ...] = ( + TargetLanguage( + code="zh-Hans", + base="zh", + name="Simplified Chinese", + native="简体中文", + chinese_name="简体中文", + scripts=(SCRIPT_HAN,), + forbidden_scripts=(SCRIPT_KANA, SCRIPT_HANGUL), + glossary_key="zh", + quote_open="“", + quote_close="”", + ellipsis="……", + dash="——", + paren_open="（", + paren_close="）", + title_open="《", + title_close="》", + font_serif="Noto Serif CJK SC", + font_sans="Noto Sans CJK SC", + chapter_prefix="第", + chapter_suffix="章", + cjk=True, + sentence_terminators="。！？!?；;…", + closing_marks="”’」』）)】》\"'", + clause_marks=("——", "—", "：", "，", "、", "；", ",", ";"), + asciidoc_lang="zh-CN", + odf_language="zh", + odf_country="CN", + epub_font_stack="Noto Serif CJK SC, Source Han Serif SC, WenQuanYi Micro Hei, serif", + chapter_numeral="chinese", + ), + TargetLanguage( + code="es-ES", + base="es", + name="Spanish", + native="español", + chinese_name="西班牙语", + scripts=(SCRIPT_LATIN,), + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC), + glossary_key="es", + quote_open="“", + quote_close="”", + ellipsis="…", + title_open="«", + title_close="»", + asciidoc_lang="es-ES", + odf_language="es", + odf_country="ES", + chapter_prefix="Capítulo ", + ), + TargetLanguage( + code="fr-FR", + base="fr", + name="French", + native="français", + chinese_name="法语", + scripts=(SCRIPT_LATIN,), + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC), + glossary_key="fr", + quote_open="«", + quote_close="»", + ellipsis="…", + title_open="«", + title_close="»", + asciidoc_lang="fr-FR", + odf_language="fr", + odf_country="FR", + chapter_prefix="Chapitre ", + ), + TargetLanguage( + code="de-DE", + base="de", + name="German", + native="Deutsch", + chinese_name="德语", + scripts=(SCRIPT_LATIN,), + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC), + glossary_key="de", + quote_open="„", + quote_close="“", + ellipsis="…", + title_open="»", + title_close="«", + asciidoc_lang="de-DE", + odf_language="de", + odf_country="DE", + chapter_prefix="Kapitel ", + ), + TargetLanguage( + code="it-IT", + base="it", + name="Italian", + native="italiano", + chinese_name="意大利语", + scripts=(SCRIPT_LATIN,), + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC), + glossary_key="it", + quote_open="«", + quote_close="»", + ellipsis="…", + title_open="«", + title_close="»", + asciidoc_lang="it-IT", + odf_language="it", + odf_country="IT", + chapter_prefix="Capitolo ", + ), + TargetLanguage( + code="pt-BR", + base="pt", + name="Portuguese", + native="português", + chinese_name="葡萄牙语", + scripts=(SCRIPT_LATIN,), + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC), + glossary_key="pt", + quote_open="“", + quote_close="”", + ellipsis="…", + title_open="«", + title_close="»", + asciidoc_lang="pt-BR", + odf_language="pt", + odf_country="BR", + chapter_prefix="Capítulo ", + ), + TargetLanguage( + code="ru-RU", + base="ru", + name="Russian", + native="русский", + chinese_name="俄语", + scripts=(SCRIPT_CYRILLIC,), + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL), + glossary_key="ru", + quote_open="«", + quote_close="»", + ellipsis="…", + title_open="«", + title_close="»", + asciidoc_lang="ru-RU", + odf_language="ru", + odf_country="RU", + chapter_prefix="Глава ", + ), + TargetLanguage( + code="ja-JP", + base="ja", + name="Japanese", + native="日本語", + chinese_name="日语", + scripts=(SCRIPT_HAN, SCRIPT_KANA), + required_script=SCRIPT_KANA, + forbidden_scripts=(SCRIPT_HANGUL,), + glossary_key="ja", + quote_open="「", + quote_close="」", + ellipsis="……", + dash="——", + paren_open="（", + paren_close="）", + title_open="『", + title_close="』", + font_serif="Noto Serif CJK JP", + font_sans="Noto Sans CJK JP", + chapter_prefix="第", + chapter_suffix="章", + cjk=True, + sentence_terminators="。！？!?；;…", + closing_marks="”’」』）)】》\"'", + clause_marks=("——", "—", "：", "、", "，", "；", ",", ";"), + asciidoc_lang="ja-JP", + odf_language="ja", + odf_country="JP", + epub_font_stack="Noto Serif CJK JP, Source Han Serif JP, serif", + ), + TargetLanguage( + code="ko-KR", + base="ko", + name="Korean", + native="한국어", + chinese_name="韩语", + scripts=(SCRIPT_HANGUL,), + required_script=SCRIPT_HANGUL, + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA), + glossary_key="ko", + quote_open="“", + quote_close="”", + ellipsis="…", + dash="—", + paren_open="(", + paren_close=")", + title_open="『", + title_close="』", + font_serif="Noto Serif CJK KR", + font_sans="Noto Sans CJK KR", + cjk=True, + sentence_terminators="。！？!?…", + closing_marks="”’」』）)】》\"'", + clause_marks=("—", "–", ":", ";", ",", "、", "，"), + asciidoc_lang="ko-KR", + odf_language="ko", + odf_country="KR", + epub_font_stack="Noto Serif CJK KR, Source Han Serif KR, serif", + chapter_prefix="제", + chapter_suffix="장", + ), + TargetLanguage( + code="ar-SA", + base="ar", + name="Arabic", + native="العربية", + chinese_name="阿拉伯语", + scripts=(SCRIPT_ARABIC,), + required_script=SCRIPT_ARABIC, + forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC), + glossary_key="ar", + quote_open="”", + quote_close="“", + ellipsis="…", + dash="—", + title_open="«", + title_close="»", + font_serif="Noto Naskh Arabic", + font_sans="Noto Sans Arabic", + rtl=True, + asciidoc_lang="ar-SA", + odf_language="ar", + odf_country="SA", + epub_font_stack="Noto Naskh Arabic, Noto Sans Arabic, serif", + chapter_prefix="الفصل ", + # Arabic has its own question mark and semicolon, and its own comma. + sentence_terminators=".!?…؟؛", + closing_marks="”‘)]}»؟؛\"'", + clause_marks=("—", "–", ":", "؛", "،", ";", ","), + ), +) + +_BY_CODE: dict[str, TargetLanguage] = {lang.code: lang for lang in TARGET_LANGUAGES} +_BY_BASE: dict[str, TargetLanguage] = {lang.base: lang for lang in TARGET_LANGUAGES} +_BY_ALIAS: dict[str, TargetLanguage] = {} +for _lang in TARGET_LANGUAGES: + _BY_ALIAS[_lang.code.casefold()] = _lang + _BY_ALIAS[_lang.base.casefold()] = _lang + _BY_ALIAS[_lang.name.casefold()] = _lang + _BY_ALIAS[_lang.native.casefold()] = _lang + + +def by_code(value: str | None) -> TargetLanguage | None: + """Look up a language by tag, base code, English name or native name. + + ``'es'``, ``'es-ES'``, ``'es_ES'`` and ``'Spanish'`` all resolve to the + same entry, so a caller does not have to know which form a production + directory happens to use. + """ + if not value: + return None + raw = str(value).strip() + if not raw: + return None + language = _BY_ALIAS.get(raw.casefold()) + if language is not None: + return language + # 'es_ES' and 'zh_Hans' are common on disk; fold the separator. + folded = raw.replace("_", "-") + language = _BY_ALIAS.get(folded.casefold()) + if language is not None: + return language + # A tag we do not carry, but whose base we do ('pt-PT' -> Portuguese). + base = folded.split("-", 1)[0].casefold() + return _BY_BASE.get(base) + + +def resolve(value: str | None) -> TargetLanguage: + """The language for *value*, or the Chinese default with a warning. + + Falling back silently would recreate the defect this module exists to + prevent: a production that declares a language the pipeline does not + support would be translated into Chinese and filed under the wrong name. + So the fallback is loud. + """ + language = by_code(value) + if language is not None: + return language + print( + f"warning: target language {value!r} is not in the registry; falling back to Simplified Chinese", + flush=True, + ) + return _BY_BASE["zh"] + + +def script_counts(text: str) -> dict[str, int]: + """How many letters of *text* belong to each script. + + Only letters count. Punctuation, digits and whitespace say nothing about + which language a passage is written in, and a name kept in English inside + a Chinese paragraph must not pull the count towards Latin. + """ + counts: dict[str, int] = {} + for ch in text: + point = ord(ch) + for script, low, high in _SCRIPT_RANGES: + if low <= point <= high: + counts[script] = counts.get(script, 0) + 1 + break + return counts + + +def script_mismatch(code: str | None, text: str) -> str | None: + """Why *text* cannot be a translation into *code*, or None when it can. + + The check is deliberately about the *script*, not the language. It cannot + tell Spanish from Portuguese, but it can tell Chinese from Spanish, which + is the mistake that actually happens - a pipeline configured for one + language that silently produces another. + """ + language = by_code(code) + if language is None: + return None + + counts = script_counts(text) + total = sum(counts.values()) + if total == 0: + return None + + for forbidden in language.forbidden_scripts: + if counts.get(forbidden, 0) / total >= 0.10: + return f"the text is written in {SCRIPT_NAMES[forbidden]}, so it is not {language.name}" + + if language.required_script and not counts.get(language.required_script): + return f"the text has no {SCRIPT_NAMES[language.required_script]}, so it is not {language.name}" + + acceptable = sum(counts.get(script, 0) for script in language.scripts) + if acceptable / total < 0.5: + dominant = max(counts, key=lambda key: counts[key]) + return f"the text is mostly {SCRIPT_NAMES[dominant]}, so it is not {language.name}" + return None + + +# --- text helpers ---------------------------------------------------------- + +# How many chapters the dashboard samples when it measures a production's +# language. A handful is enough to know which script a book is written in and +# cheap enough to run on every page load. +SAMPLE_CHAPTERS = 8 + + +def sample_run_text(runs_dir: Path, max_chapters: int = SAMPLE_CHAPTERS) -> str: + """The translated text of a sample of chapters in a run directory. + + The sample is spread across the book rather than taken from the front: + the first chapter is often front matter, and a front-matter chapter can be + mostly numbers and names, which say nothing about which language the book + is in. + """ + if not runs_dir.is_dir(): + return "" + chapters = sorted(p for p in runs_dir.glob("ch-*") if (p / "translation.json").is_file()) + if not chapters: + return "" + if len(chapters) <= max_chapters: + chosen = chapters + else: + step = len(chapters) / max_chapters + chosen = [chapters[int(i * step)] for i in range(max_chapters)] + + parts: list[str] = [] + for chapter in chosen: + try: + data = json.loads((chapter / "translation.json").read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + paragraphs = data.get("paragraphs") or {} + if isinstance(paragraphs, dict): + parts.extend(str(value) for value in paragraphs.values() if isinstance(value, str)) + return " ".join(parts) + + +def check_runs_language(runs_dir: Path, code: str) -> str | None: + """Why the translations in *runs_dir* cannot be in *code*, or None. + + The dashboard reports a language's state from the presence and chapter + count of its run directory. That is a claim about the *files*, not about + the *text*: a directory full of Chinese can sit in a directory named for + Spanish and be reported as a complete Spanish translation. Measured on + oldbox, that is exactly what ``deal-hunter-es`` was - 3,038 paragraphs + of Chinese under a Spanish name, reported complete. + + So the text is measured, not the file names. + """ + language = by_code(code) + if language is None: + return None + sample = sample_run_text(runs_dir) + if not sample.strip(): + return None + return script_mismatch(language.code, sample) + + +# The clause boundary a language uses to join two sentences into one. Chinese +def punctuation_hint(language: TargetLanguage) -> str: + """How to describe the target's punctuation, for a prompt.""" + return ( + f"引号用{language.quote_open}{language.quote_close}成对；省略号用{language.ellipsis}；破折号用{language.dash}" + ) + + +# --- the strings the renderer writes --------------------------------------- + +# The renderer puts a small amount of fixed text around the translation: the +# contents heading, the review-sample notice, the translator credit, the +# chapter heading convention. That text is part of the book the reader +# receives, so it has to be in the book's language. Leaving it in Chinese is +# the same defect as a Chinese chapter title in a Spanish edition, one level +# down and much easier to miss. +# +# A language with no entry here gets the English string, never another +# language's. +RENDER_STRINGS: dict[str, dict[str, str]] = { + "en": { + "missing_title": "[Untranslated] {title}", + "toc_title": "Table of Contents", + "review_sample": "Review sample", + "translator_credit": "Translated by Umbra Tools", + "sample_note": ( + "This file is a review sample of the English translation. The translated " + "chapters are listed in the contents; the remaining chapters will be " + "completed in the full run." + ), + "audio_preface": "Audio Preface", + "audio_preface_intro": "The following is the opening of the Deal Hunter audiobook.", + "sample_heading": "About this review sample", + "sample_process": ( + "Translation process: paragraph reconstruction, intent analysis, first " + "draft, bilingual review. Terms are in glossary.json; the reviewer's " + "notes are in review.html." + ), + "sample_end": "End of sample", + "sample_end_note": ( + "That is the review sample. The full run will cover the remaining " + "chapters; the reviewer's issue list is provided with each chapter's " + "translation record." + ), + "author_line": "B. D. Murphy", + "translator_line": "Translated by Umbra Tools", + "edition_label": "English edition · Review sample", + }, + "zh": { + "missing_title": "[待译] {title}", + "toc_title": "目录", + "review_sample": "审阅样本", + "translator_credit": "译者：Umbra Tools", + "sample_note": "本文件是简体中文翻译的审阅样本。已译部分见目录；未列出的章节将在全文运行中补齐。", + "audio_preface": "音频版序言", + "audio_preface_intro": "以下是《Deal Hunter》有声版的开场白。", + "sample_heading": "审阅样本说明", + "sample_process": ( + "翻译流程：段落重建 → 意图分析 → 初译 → 双语审校。术语见 glossary.json，审校意见见 review.html。" + ), + "sample_end": "样本结束", + "sample_end_note": ("以上为审阅样本。全文运行将覆盖其余章节；审校问题清单会随各章翻译记录一并提供。"), + "author_line": "B·D·墨菲 著", + "translator_line": "Umbra Tools 译", + "edition_label": "简体中文译本 · 审阅样本", + }, + "es": { + "missing_title": "[Sin traducir] {title}", + "toc_title": "Índice", + "review_sample": "Muestra de revisión", + "translator_credit": "Traducido por Umbra Tools", + "sample_note": ( + "Este archivo es una muestra de revisión de la traducción al español. " + "Los capítulos traducidos aparecen en el índice; los capítulos " + "restantes se completarán en la ejecución completa." + ), + "audio_preface": "Prefacio del audiolibro", + "audio_preface_intro": "Lo que sigue es la introducción del audiolibro de Deal Hunter.", + "sample_heading": "Sobre esta muestra de revisión", + "sample_process": ( + "Proceso de traducción: reconstrucción de párrafos, análisis de intención, " + "primer borrador, revisión bilingüe. Los términos están en glossary.json; " + "las notas del revisor están en review.html." + ), + "sample_end": "Fin de la muestra", + "sample_end_note": ( + "Esta es la muestra de revisión. La ejecución completa cubrirá los " + "capítulos restantes; la lista de incidencias del revisor se entrega " + "junto con el registro de traducción de cada capítulo." + ), + "author_line": "B. D. Murphy", + "translator_line": "Traducido por Umbra Tools", + "edition_label": "Edición en español · Muestra de revisión", + }, + "fr": { + "missing_title": "[Non traduit] {title}", + "toc_title": "Table des matières", + "review_sample": "Échantillon de relecture", + "translator_credit": "Traduit par Umbra Tools", + "sample_note": ( + "Ce fichier est un échantillon de relecture de la traduction française. " + "Les chapitres traduits figurent dans la table des matières ; les " + "chapitres restants seront complétés lors de l'exécution complète." + ), + "audio_preface": "Préface de l'audiobook", + "audio_preface_intro": "Voici l'introduction de l'audiobook de Deal Hunter.", + "sample_heading": "À propos de cet échantillon", + "sample_process": ( + "Processus de traduction : reconstruction des paragraphes, analyse de " + "l'intention, premier jet, relecture bilingue. Les termes sont dans " + "glossary.json ; les notes du relecteur sont dans review.html." + ), + "sample_end": "Fin de l'échantillon", + "sample_end_note": ( + "Voici l'échantillon de relecture. L'exécution complète couvrira les " + "chapitres restants ; la liste des remarques du relecteur est fournie " + "avec le dossier de traduction de chaque chapitre." + ), + "author_line": "B. D. Murphy", + "translator_line": "Traduit par Umbra Tools", + "edition_label": "Édition française · Échantillon de relecture", + }, + "de": { + "missing_title": "[Nicht übersetzt] {title}", + "toc_title": "Inhaltsverzeichnis", + "review_sample": "Korrekturdurchschlag", + "translator_credit": "Übersetzt von Umbra Tools", + "sample_note": ( + "Diese Datei ist ein Korrekturdurchschlag der deutschen Übersetzung. " + "Die übersetzten Kapitel stehen im Inhaltsverzeichnis; die restlichen " + "Kapitel werden im vollständigen Lauf ergänzt." + ), + "audio_preface": "Hörbuch-Vorwort", + "audio_preface_intro": "Es folgt die Einleitung des Hörbuchs zu Deal Hunter.", + "sample_heading": "Über diesen Korrekturdurchschlag", + "sample_process": ( + "Übersetzungsablauf: Absatzrekonstruktion, Intentionsanalyse, Rohtext, " + "zweisprachige Korrektur. Begriffe stehen in glossary.json; die " + "Anmerkungen der Korrektorin stehen in review.html." + ), + "sample_end": "Ende des Durchschlags", + "sample_end_note": ( + "Das ist der Korrekturdurchschlag. Der vollständige Lauf deckt die " + "restlichen Kapitel ab; die Liste der Anmerkungen liegt dem " + "Übersetzungsprotokoll jedes Kapitels bei." + ), + "author_line": "B. D. Murphy", + "translator_line": "Übersetzt von Umbra Tools", + "edition_label": "Deutsche Ausgabe · Korrekturdurchschlag", + }, + "it": { + "missing_title": "[Non tradotto] {title}", + "toc_title": "Indice", + "review_sample": "Bozza di revisione", + "translator_credit": "Tradotto da Umbra Tools", + "sample_note": ( + "Questo file è una bozza di revisione della traduzione italiana. I " + "capitoli tradotti compaiono nell'indice; i restanti saranno completati " + "nella corsione completa." + ), + "audio_preface": "Prefazione dell'audiolibro", + "audio_preface_intro": "Quello che segue è l'introduzione dell'audiolibro di Deal Hunter.", + "sample_heading": "Informazioni su questa bozza", + "sample_process": ( + "Processo di traduzione: ricostruzione dei paragrafi, analisi " + "dell'intento, prima stesura, revisione bilingue. I termini sono in " + "glossary.json; le note del revisore sono in review.html." + ), + "sample_end": "Fine bozza", + "sample_end_note": ( + "Questa è la bozza di revisione. La corsione completa coprirà i " + "capitoli restanti; l'elenco delle osservazioni del revisore " + "accompagna il registro di traduzione di ogni capitolo." + ), + "author_line": "B. D. Murphy", + "translator_line": "Tradotto da Umbra Tools", + "edition_label": "Edizione italiana · Bozza di revisione", + }, + "pt": { + "missing_title": "[Não traduzido] {title}", + "toc_title": "Sumário", + "review_sample": "Amostra de revisão", + "translator_credit": "Traduzido por Umbra Tools", + "sample_note": ( + "Este arquivo é uma amostra de revisão da tradução para português. Os " + "capítulos traduzidos estão no sumário; os capítulos restantes serão " + "completados na execução completa." + ), + "audio_preface": "Prefácio do audiolivro", + "audio_preface_intro": "Segue-se a introdução do audiolivro de Deal Hunter.", + "sample_heading": "Sobre esta amostra de revisão", + "sample_process": ( + "Processo de tradução: reconstrução de parágrafos, análise de intenção, " + "primeira redação, revisão bilíngue. Os termos estão em glossary.json; " + "as notas do revisor estão em review.html." + ), + "sample_end": "Fim da amostra", + "sample_end_note": ( + "Esta é a amostra de revisão. A execução completa cobrirá os capítulos " + "restantes; a lista de observações do revisor acompanha o registro de " + "tradução de cada capítulo." + ), + "author_line": "B. D. Murphy", + "translator_line": "Traduzido por Umbra Tools", + "edition_label": "Edição em português · Amostra de revisão", + }, + "ru": { + "missing_title": "[Не переведено] {title}", + "toc_title": "Оглавление", + "review_sample": "Пробная вычитка", + "translator_credit": "Перевод: Umbra Tools", + "sample_note": ( + "Этот файл — пробная вычитка русского перевода. Переведённые главы " + "перечислены в оглавлении; остальные главы будут дополнены при полном " + "прогоне." + ), + "audio_preface": "Предисловие к аудиокниге", + "audio_preface_intro": "Ниже следует вступление к аудиокниге «Deal Hunter».", + "sample_heading": "Об этой пробной вычитке", + "sample_process": ( + "Порядок перевода: сборка абзацев, анализ замысла, черновик, двуязычная " + "вычитка. Термины — в glossary.json, замечания редактора — в review.html." + ), + "sample_end": "Конец пробы", + "sample_end_note": ( + "Это пробная вычитка. Полный прогон охватит остальные главы; список " + "замечаний редактора прилагается к протоколу перевода каждой главы." + ), + "author_line": "Б. Д. Морфи", + "translator_line": "Перевод: Umbra Tools", + "edition_label": "Русское издание · Пробная вычитка", + }, + "ja": { + "missing_title": "[未翻訳] {title}", + "toc_title": "目次", + "review_sample": "査読用サンプル", + "translator_credit": "翻訳：Umbra Tools", + "sample_note": ( + "このファイルは日本語訳の査読用サンプルです。翻訳済みの章は目次に記載されています。" + "残りの章は全文実行で補完されます。" + ), + "audio_preface": "オーディオブック前書き", + "audio_preface_intro": "以下は『Deal Hunter』オーディオブックの冒頭です。", + "sample_heading": "この査読用サンプルについて", + "sample_process": ( + "翻訳の流れ：段落の再構成、意図の分析、初稿、二言語査読。用語は " + "glossary.json、査読者の指摘は review.html にあります。" + ), + "sample_end": "サンプル終了", + "sample_end_note": ( + "以上が査読用サンプルです。全文実行が残りの章をすべて扱い、査読者の指摘一覧は各章の翻訳記録に添付されます。" + ), + "author_line": "B・D・マーフィ 著", + "translator_line": "Umbra Tools 訳", + "edition_label": "日本語版 · 査読用サンプル", + }, + "ko": { + "missing_title": "[미번역] {title}", + "toc_title": "목차", + "review_sample": "교정용 샘플", + "translator_credit": "번역: Umbra Tools", + "sample_note": ( + "이 파일은 한국어 번역의 교정용 샘플입니다. 번역된 장은 목차에 나열되어 있고, " + "남은 장은 전체 실행에서 완성됩니다." + ), + "audio_preface": "오디오북 머리말", + "audio_preface_intro": "다음은 『Deal Hunter』 오디오북의 도입부입니다.", + "sample_heading": "이 교정용 샘플에 대하여", + "sample_process": ( + "번역 절차: 문단 재구성, 의도 분석, 초안, 이중 언어 교정. 용어는 " + "glossary.json, 교정자 메모는 review.html에 있습니다." + ), + "sample_end": "샘플 끝", + "sample_end_note": ( + "이것이 교정용 샘플입니다. 전체 실행이 남은 장을 모두 다루며, 교정자의 지적 " + "목록은 각 장의 번역 기록과 함께 제공됩니다." + ), + "author_line": "B. D. 머피 지음", + "translator_line": "Umbra Tools 옮김", + "edition_label": "한국어판 · 교정용 샘플", + }, + "ar": { + "missing_title": "[لم يُترجم] {title}", + "toc_title": "المحتويات", + "review_sample": "عيّنة مراجعة", + "translator_credit": "ترجمة: Umbra Tools", + "sample_note": ( + "هذا الملف عيّنة مراجعة للترجمة العربية. الفصول المترجمة مدرجة في المحتويات، " + "وتُستكمل بقية الفصول في التشغيل الكامل." + ), + "audio_preface": "تمهيد الكتاب الصوتي", + "audio_preface_intro": "فيما يلي مقدمة الكتاب الصوتي لـ Deal Hunter.", + "sample_heading": "عن هذه عيّنة المراجعة", + "sample_process": ( + "مسار الترجمة: إعادة بناء الفقرات، تحليل القصد، المسودة الأولى، المراجعة " + "ثنائية اللغة. المصطلحات في glossary.json، وملاحظات المراجع في review.html." + ), + "sample_end": "نهاية العيّنة", + "sample_end_note": ( + "هذه هي عيّنة المراجعة. سيغطي التشغيل الكامل بقية الفصول، وتُرفق قائمة ملاحظات المراجع مع سجل ترجمة كل فصل." + ), + "author_line": "ب. د. مورفي", + "translator_line": "ترجمة: Umbra Tools", + "edition_label": "الطبعة العربية · عيّنة مراجعة", + }, +} diff --git a/tools/translation/pipeline.py b/tools/translation/pipeline.py index e74cbc1..996561b 100644 --- a/tools/translation/pipeline.py +++ b/tools/translation/pipeline.py @@ -24,16 +24,17 @@ from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) +from languages import SCRIPT_LATIN, TargetLanguage, resolve, script_mismatch # noqa: E402 from llm import DEFAULT_ANALYST, DEFAULT_CRITIC, DEFAULT_MODEL, LLM, LLMError, Usage # noqa: E402 from prompts import ( # noqa: E402 - SYSTEM_ANALYST, - SYSTEM_CRITIC, - SYSTEM_DRAFTER, analysis_prompt, consistency_prompt, critic_prompt, draft_prompt, load_glossary, + system_analyst, + system_critic, + system_drafter, ) CHARACTER_NOTES = """- 凯因达：聪明、直率、好奇，带一点叛逆；对权威不耐烦，对朋友温暖。不要把她写成娇滴滴的公主。 @@ -43,42 +44,101 @@ CHARACTER_NOTES = """- 凯因达：聪明、直率、好奇，带一点叛逆； - 罗曼诺：热情、商人腔、爱谈价格，但不油滑。 - 旁白：克制、清晰、有画面感；不替人物抒情。""" +# The default target, used where a caller has not supplied one. The pipeline +# was Chinese-only for its whole life, so the default keeps every existing +# call site behaving exactly as it did. +CHINESE = resolve("zh-Hans") + # Proper nouns that may legitimately stay in Latin script. Anything else that # survives in the Chinese text is reported, not silently accepted. LATIN_ALLOW = { - "ai", "audio", "authorbdmurphy", "b", "blazes", "brutus", "com", "d", - "deal", "dial", "eva", "felucca", "gisfrid", "goodreads", "hunter", - "http", "https", "isbn", "kabir", "kainda", "konrad", "llc", "l2", - "maul", "murphy", "nanny", "niñera", "romano", "sao", "umbra", "vince", - "wakam", "wake", "weston", "willamina", "witold", "worstauthors", - "worstauthor", "www", "zheng", "yi", "adonai", "adric", "albert", + "ai", + "audio", + "authorbdmurphy", + "b", + "blazes", + "brutus", + "com", + "d", + "deal", + "dial", + "eva", + "felucca", + "gisfrid", + "goodreads", + "hunter", + "http", + "https", + "isbn", + "kabir", + "kainda", + "konrad", + "llc", + "l2", + "maul", + "murphy", + "nanny", + "niñera", + "romano", + "sao", + "umbra", + "vince", + "wakam", + "wake", + "weston", + "willamina", + "witold", + "worstauthors", + "worstauthor", + "www", + "zheng", + "yi", + "adonai", + "adric", + "albert", } -def _normalise_quotes(text: str) -> str: - text = re.sub(r'"([^"\n]*)"', r"“\1”", text) +def _normalise_quotes(text: str, target: TargetLanguage | None = None) -> str: + """Balance the target language's quotation marks. + + The marks are the language's own: a German draft uses „ and “, a French + one « and ». The straight double quote the model sometimes emits is + converted to the target's pair. + """ + target = target or CHINESE + open_mark, close_mark = target.quote_open, target.quote_close + text = re.sub(r'"([^"\n]*)"', lambda m: open_mark + m.group(1) + close_mark, text) if '"' in text: parts = text.split('"') out = parts[0] for i, part in enumerate(parts[1:]): - out += ("“" if i % 2 == 0 else "”") + part + out += (open_mark if i % 2 == 0 else close_mark) + part text = out - if "“" not in text and text.endswith("”"): + if open_mark not in text and text.endswith(close_mark): text = text[:-1] - opens, closes = text.count("“"), text.count("”") + opens, closes = text.count(open_mark), text.count(close_mark) if opens > closes: - text += "”" * (opens - closes) + text += close_mark * (opens - closes) elif closes > opens: for _ in range(closes - opens): - idx = text.rfind("”") - if idx > text.rfind("“"): + idx = text.rfind(close_mark) + if idx > text.rfind(open_mark): text = text[:idx] + text[idx + 1 :] else: break return text -def sanitize(text: str) -> str: +def sanitize(text: str, target: TargetLanguage | None = None) -> str: + """Clean up one drafted paragraph. + + The Chinese replacements repair a set of mistakes the Chinese drafter + keeps making - a transliterated title, a transliterated ship name. They + cannot match in another language, so they are harmless there; the + punctuation rules at the end are the target's own. + """ + target = target or CHINESE text = text.strip() text = re.sub(r"^\[[^\]]{1,40}\]\s*", "", text) text = re.sub(r"^（(?:对话|旁白|叙述)[^）]{0,60}）\s*", "", text) @@ -98,8 +158,11 @@ def sanitize(text: str) -> str: text = text.replace("Umbra Audio星系系统", "Umbra Audio 系统") text = text.replace("音频星系系统", "音频系统") text = text.replace("在小时前", "一小时前") - text = text.replace("...", "……").replace("--", "——") - return _normalise_quotes(text).strip() + if target.ellipsis == "……": + text = text.replace("...", "……").replace("--", "——") + else: + text = text.replace("...", target.ellipsis).replace("--", target.dash) + return _normalise_quotes(text, target).strip() def latin_residue(text: str) -> tuple[list[str], list[str]]: @@ -135,6 +198,16 @@ def load_book(path: Path) -> dict: return json.loads(path.read_text(encoding="utf-8")) +def chapter_title_target(chapter: dict) -> str: + """The chapter's title in the target language. + + The field was called ``title_zh`` while the pipeline was Chinese-only and + is now called ``title_target``. The old name is still read, so a book file + written by an earlier version keeps working. + """ + return chapter.get("title_target") or chapter.get("title_zh") or chapter.get("title") or "" + + def select_chapters(book: dict, selector: str) -> list[dict]: chapters = book["chapters"] if selector in {"all", "*"}: @@ -178,7 +251,9 @@ def chunk_chapter(chapter: dict, max_words: int = 260, max_paras: int = 14) -> l current, current_words, current_scene = [], 0, None continue p_scene = scene_of.get(p["id"], 0) - if current and (p_scene != current_scene or current_words + len(p["text"].split()) > max_words or len(current) >= max_paras): + if current and ( + p_scene != current_scene or current_words + len(p["text"].split()) > max_words or len(current) >= max_paras + ): raw.append(current) current, current_words = [], 0 current.append(p) @@ -226,11 +301,15 @@ class Runner: max_words: int = 260, max_paras: int = 14, critic_limit: int = 0, + target_language: str = "zh-Hans", ) -> None: self.book = load_book(book_path) self.glossary = load_glossary(glossary_path) self.run_dir = run_dir self.run_dir.mkdir(parents=True, exist_ok=True) + self.target: TargetLanguage = resolve(target_language) + self.book_path = book_path + self.glossary_path = glossary_path self.model = model self.analyst_model = analyst_model self.critic_model = critic_model @@ -248,6 +327,7 @@ class Runner: usage_log=run_dir / "usage.jsonl", **kwargs, ) + self._guard_run_directory() # ----------------------------------------------------------------- files def chapter_dir(self, chapter: dict) -> Path: @@ -263,6 +343,72 @@ class Runner: def _write(self, path: Path, value: object) -> None: path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + # --------------------------------------------------------------- the guard + def _guard_run_directory(self) -> None: + """Refuse to reuse a run directory that belongs to another language. + + The pipeline caches by file existence, and the run directory is chosen + by the caller. So a Spanish run pointed at a directory that already + holds a Chinese run would skip drafting entirely and publish the + Chinese under a Spanish name - the exact defect the mislabelled + production had, and the reason it is worth a guard rather than a + convention. + + Two checks, one from the claim and one from the measurement: + + * The manifest records what the directory is for. A mismatch is + refused. + * A directory with no manifest (written before this guard existed) is + measured instead: the script of the text it already holds is + compared against the target language. + """ + manifest_path = self.run_dir / "run.json" + previous: dict = {} + if manifest_path.exists() and not self.force: + try: + previous = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + previous = {} + declared = previous.get("target_language") + if declared and declared != self.target.code: + raise SystemExit( + f"run directory {self.run_dir} holds a {declared} run, but this " + f"run is {self.target.code}. Use a different --run-dir, or pass " + "--force to overwrite it deliberately." + ) + self._measure_existing_language() + self._write( + manifest_path, + { + "target_language": self.target.code, + "target_name": self.target.name, + "glossary_key": self.target.glossary_key, + "book": str(self.book_path), + "glossary": str(self.glossary_path), + }, + ) + + def _measure_existing_language(self) -> None: + """What language does the run directory already hold?""" + for chapter_dir in sorted(self.run_dir.glob("ch-*")): + path = chapter_dir / "translation.json" + if not path.exists(): + continue + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return + paragraphs = data.get("paragraphs") or {} + sample = " ".join(str(value) for value in list(paragraphs.values())[:200]) + problem = script_mismatch(self.target.code, sample) + if problem: + raise SystemExit( + f"run directory {self.run_dir} already holds a translation that " + f"is not {self.target.name}: {problem}. Use a different " + "--run-dir, or pass --force to overwrite it deliberately." + ) + return + # ----------------------------------------------------------------- passes def analyse(self, chapter: dict, chunk: Chunk, previous_summary: str) -> dict: path = self.chapter_dir(chapter) / f"{chunk.id}.analysis.json" @@ -270,16 +416,17 @@ class Runner: if cached is not None: return cached prompt = analysis_prompt( - chapter_title=f"{chapter['title']} / {chapter['title_zh']}", + chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}", scene_index=chunk.scene_index, chunk_index=int(chunk.id.rsplit("c", 1)[-1]), chunks_in_scene=chunk.chunks_in_scene, paragraphs=chunk.paragraphs, previous_summary=previous_summary, glossary=self.glossary, + target=self.target, ) value, usage = self.llm.chat_json( - SYSTEM_ANALYST, + system_analyst(self.target), prompt, model=self.analyst_model, temperature=0.2, @@ -305,8 +452,7 @@ class Runner: # translated. Record that the analysis is degraded and continue with a # minimal one; the drafter falls back to the source and glossary. print( - f" {chunk.id}: analyst returned {type(value).__name__}, " - "using a minimal analysis", + f" {chunk.id}: analyst returned {type(value).__name__}, using a minimal analysis", flush=True, ) value = {"degraded": True, "note": "analyst returned non-object JSON"} @@ -321,7 +467,7 @@ class Runner: return cached expected = [p["id"] for p in chunk.paragraphs if p.get("type") != "scene_break"] prompt = draft_prompt( - chapter_title=f"{chapter['title']} / {chapter['title_zh']}", + chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}", scene_index=chunk.scene_index, chunk_index=int(chunk.id.rsplit("c", 1)[-1]), chunks_in_scene=chunk.chunks_in_scene, @@ -329,9 +475,10 @@ class Runner: analysis={k: v for k, v in analysis.items() if k != "_usage"}, previous_summary=previous_summary, glossary=self.glossary, + target=self.target, ) value, usage = self.llm.chat_json( - SYSTEM_DRAFTER, + system_drafter(self.target), prompt, model=self.model, temperature=0.45, @@ -358,26 +505,58 @@ class Runner: self._write(path, result) return result + def _draft_text(self, item: dict) -> str: + """The translated text of one drafted paragraph. + + The key is the target language's own, but the model is not always + disciplined about it, so a neutral key is accepted too. + """ + for key in (self.target.glossary_key, "text", "translation", "zh"): + value = item.get(key) + if isinstance(value, str) and value.strip(): + return value + return "" + def _normalise_draft(self, value: object, expected: list[str]) -> dict[str, str]: out: dict[str, str] = {} if isinstance(value, dict) and isinstance(value.get("paragraphs"), list): items = value["paragraphs"] elif isinstance(value, list): items = value - elif isinstance(value, dict) and value.get("id") and value.get("zh"): + elif isinstance(value, dict) and value.get("id") and self._draft_text(value): items = [value] elif isinstance(value, dict): - # maybe mapping id->zh + # maybe mapping id->text for k, v in value.items(): if isinstance(v, str) and k.startswith("ch-"): - out[k] = sanitize(v) + out[k] = sanitize(v, self.target) return {k: v for k, v in out.items() if k in expected and v} else: raise LLMError(f"drafter returned unexpected JSON: {str(value)[:200]}") for item in items: - if isinstance(item, dict) and item.get("id") and item.get("zh"): - out[str(item["id"])] = sanitize(str(item["zh"])) - return {k: v for k, v in out.items() if k in expected and v} + if isinstance(item, dict) and item.get("id"): + text = self._draft_text(item) + if text: + out[str(item["id"])] = sanitize(text, self.target) + out = {k: v for k, v in out.items() if k in expected and v} + self._check_draft_language(out) + return out + + def _check_draft_language(self, out: dict[str, str]) -> None: + """Is the draft actually written in the target language? + + The drafter is told which language to translate into, and it has been + known to ignore that and write Chinese anyway. A draft in the wrong + language is not a translation, so it is refused rather than stored: a + book that is 32 chapters in the wrong language is a failure that is + expensive to discover late. + """ + if not out: + return + sample = " ".join(out.values()) + problem = script_mismatch(self.target.code, sample) + if problem: + raise LLMError(f"the drafter answered in the wrong language: {problem}. Expected {self.target.name}.") def _fix_paragraph_quotes(self, chunk: Chunk, drafts: dict[str, str]) -> dict[str, str]: """Remove quotes the model wrapped around unquoted narration.""" @@ -397,16 +576,24 @@ class Runner: return drafts def _repair_residue(self, chapter: dict, chunk: Chunk, drafts: dict[str, str]) -> dict[str, str]: - """Translate stray lowercase English words in place, without touching names.""" + """Translate stray lowercase English words in place, without touching names. + + Only meaningful when the target is not written in Latin script: in a + Spanish or French draft a Latin word is the target language, not + residue. Running the check there would flag the whole book and + "repair" it into nonsense. + """ + if SCRIPT_LATIN in self.target.scripts: + return drafts by_id = {p["id"]: p for p in chunk.paragraphs} for pid, zh in list(drafts.items()): common, _proper = latin_residue(zh) if not common: continue source = by_id.get(pid, {}).get("text", "") - prompt = f"""下面这段中文译文里混入了英文单词：{", ".join(common)}。 -请只把这些英文单词译成中文，其余内容尽量保持不变；不要解释，不要改写剧情。 -输出 JSON：{{"zh": "修正后的中文译文"}} + prompt = f"""下面这段{self.target.chinese_name}译文里混入了英文单词：{", ".join(common)}。 +请只把这些英文单词译成{self.target.chinese_name}，其余内容尽量保持不变；不要解释，不要改写剧情。 +输出 JSON：{{"{self.target.glossary_key}": "修正后的{self.target.chinese_name}译文"}} 英文原文： {source} @@ -415,16 +602,16 @@ class Runner: {zh}""" try: value, _ = self.llm.chat_json( - SYSTEM_DRAFTER, + system_drafter(self.target), prompt, model=self.model, temperature=0.2, max_tokens=1500, cache_key=f"{chunk.id}.{pid}.residue", ) - fixed = value.get("zh") if isinstance(value, dict) else None + fixed = self._draft_text(value) if isinstance(value, dict) else None if fixed: - drafts[pid] = sanitize(str(fixed)) + drafts[pid] = sanitize(str(fixed), self.target) except LLMError as exc: print(f" {chunk.id}/{pid}: residue repair failed: {exc}", flush=True) return drafts @@ -437,7 +624,7 @@ class Runner: p = by_id[pid] try: prompt = draft_prompt( - chapter_title=f"{chapter['title']} / {chapter['title_zh']}", + chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}", scene_index=chunk.scene_index, chunk_index=0, chunks_in_scene=1, @@ -445,9 +632,10 @@ class Runner: analysis=analysis, previous_summary="", glossary=self.glossary, + target=self.target, ) value, _ = self.llm.chat_json( - SYSTEM_DRAFTER, + system_drafter(self.target), prompt, model=self.model, temperature=0.4, @@ -461,7 +649,7 @@ class Runner: except Exception: pass # Fallback to source text or empty to avoid blocking - result[pid] = sanitize(str(p.get("text", ""))) + result[pid] = sanitize(str(p.get("text", "")), self.target) return result def criticise(self, chapter: dict, chunk: Chunk, draft: dict[str, str]) -> dict: @@ -470,14 +658,15 @@ class Runner: if cached is not None: return cached prompt = critic_prompt( - chapter_title=f"{chapter['title']} / {chapter['title_zh']}", + chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}", paragraphs=chunk.paragraphs, draft=draft, glossary=self.glossary, + target=self.target, ) try: value, usage = self.llm.chat_json( - SYSTEM_CRITIC, + system_critic(self.target), prompt, model=self.critic_model, temperature=0.15, @@ -490,7 +679,7 @@ class Runner: print(f" critic {self.critic_model} failed ({exc}); falling back to {self.model}", flush=True) try: value, usage = self.llm.chat_json( - SYSTEM_CRITIC, + system_critic(self.target), prompt, model=self.model, temperature=0.15, @@ -562,14 +751,15 @@ class Runner: if cached is not None: return cached prompt = consistency_prompt( - chapter_title=f"{chapter['title']} / {chapter['title_zh']}", + chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}", paragraphs=paragraphs, draft=drafts, glossary=self.glossary, character_notes=CHARACTER_NOTES, + target=self.target, ) value, usage = self.llm.chat_json( - SYSTEM_CRITIC, + system_critic(self.target), prompt, model=self.critic_model, temperature=0.15, @@ -590,7 +780,7 @@ class Runner: for ci, chapter in enumerate(chapters, 1): chunks = chunk_chapter(chapter, self.max_words, self.max_paras) print( - f"[{ci}/{len(chapters)}] {chapter['id']} {chapter['title']} / {chapter['title_zh']}: " + f"[{ci}/{len(chapters)}] {chapter['id']} {chapter['title']} / {chapter_title_target(chapter)}: " f"{len(chapter['paragraphs'])} paragraphs, {len(chunks)} chunks", flush=True, ) @@ -599,12 +789,10 @@ class Runner: issues: list[dict] = [] for chunk in chunks: analysis = self.analyse(chapter, chunk, previous_summary) - previous_summary = str(analysis.get("summary_zh") or previous_summary) + previous_summary = str(analysis.get("summary") or analysis.get("summary_zh") or previous_summary) drafted = self.draft(chapter, chunk, analysis, previous_summary) drafts.update(drafted["paragraphs"]) - if self.do_critic and ( - self.critic_limit <= 0 or int(chunk.id.rsplit("c", 1)[-1]) < self.critic_limit - ): + if self.do_critic and (self.critic_limit <= 0 or int(chunk.id.rsplit("c", 1)[-1]) < self.critic_limit): crit = self.criticise(chapter, chunk, drafted["paragraphs"]) for issue in crit.get("issues", []) or []: issue = dict(issue) @@ -625,7 +813,8 @@ class Runner: "chapter_id": chapter["id"], "chapter_index": chapter["index"], "title": chapter["title"], - "title_zh": chapter["title_zh"], + "title_target": chapter_title_target(chapter), + "target_language": self.target.code, "paragraphs": drafts, "issues": issues, "chunks": [c.id for c in chunks], @@ -658,10 +847,19 @@ def report(run_dir: Path) -> str: def main() -> None: - ap = argparse.ArgumentParser(description="Run the Deal Hunter Chinese translation pipeline.") + ap = argparse.ArgumentParser(description="Run the Deal Hunter translation pipeline.") ap.add_argument("--book", type=Path, required=True) ap.add_argument("--glossary", type=Path, required=True) ap.add_argument("--run-dir", type=Path, required=True) + ap.add_argument( + "--target-language", + default=None, + help=( + "the language to translate into (zh-Hans, es-ES, fr-FR, de-DE, " + "it-IT, pt-BR, ru-RU, ja-JP, ko-KR, ar-SA). Defaults to the " + "book's own target_language." + ), + ) ap.add_argument("--chapters", default="all", help="all, front, body, back, or comma-separated indices/ids") ap.add_argument("--model", default=DEFAULT_MODEL) ap.add_argument("--analyst-model", default=DEFAULT_ANALYST) @@ -672,7 +870,9 @@ def main() -> None: ap.add_argument("--consistency", action="store_true") ap.add_argument("--max-words", type=int, default=260) ap.add_argument("--max-paras", type=int, default=14) - ap.add_argument("--critic-limit", type=int, default=0, help="criticise only the first N chunks per chapter (0 = all)") + ap.add_argument( + "--critic-limit", type=int, default=0, help="criticise only the first N chunks per chapter (0 = all)" + ) ap.add_argument("--report", action="store_true") args = ap.parse_args() @@ -681,6 +881,7 @@ def main() -> None: return book = load_book(args.book) + target_language = args.target_language or book.get("target_language") or "zh-Hans" chapters = select_chapters(book, args.chapters) runner = Runner( book_path=args.book, @@ -696,6 +897,7 @@ def main() -> None: max_words=args.max_words, max_paras=args.max_paras, critic_limit=args.critic_limit, + target_language=target_language, ) runner.run(chapters) diff --git a/tools/translation/prompts.py b/tools/translation/prompts.py index 4356304..f0c2ad1 100644 --- a/tools/translation/prompts.py +++ b/tools/translation/prompts.py @@ -6,6 +6,20 @@ The prompts follow specs/27: analyse the scene before rendering it, give the drafter the intent and the glossary, then check meaning with a different model. They deliberately do not ask for word-for-word equivalence. + +Every prompt is parameterised by the target language. The pipeline used to +be hard-coded to Simplified Chinese - the drafter was told to translate into +Chinese, the critic was told to compare English with Chinese, the glossary's +translated forms were read from a ``zh`` key - so a production that declared +``target_language: es-ES`` was translated into Chinese and filed under a +Spanish name. The target language now comes from +:class:`languages.TargetLanguage`, and the language's own script, punctuation +and glossary key follow from it. + +The prompt scaffolding stays in Chinese. It is the language the pipeline was +developed in, the analyst model handles it well, and what the drafter has to +be told is *which* language to translate into - not a translation of the +instructions themselves. """ from __future__ import annotations @@ -13,6 +27,8 @@ from __future__ import annotations import json from pathlib import Path +from languages import TargetLanguage + CORE_CHARACTERS = [ "Kainda Walters Foster Murphy", "Princess Kainda", @@ -49,55 +65,14 @@ CORE_TERMS = [ "coronation", ] - -def load_glossary(path: Path) -> dict: - return json.loads(path.read_text(encoding="utf-8")) - - -def _pick(mapping: dict, keys: list[str]) -> list[tuple[str, str]]: - out: list[tuple[str, str]] = [] - for key in keys: - if key in mapping: - value = mapping[key] - if isinstance(value, dict): - zh = value.get("zh") or value.get("short") or "" - note = value.get("note") - if note: - zh = f"{zh}（{note}）" - else: - zh = str(value) - if zh: - out.append((key, zh)) - return out - - -def glossary_block(glossary: dict, speakers: set[str], text: str) -> str: - chars = dict(glossary.get("characters", {})) - roles = dict(glossary.get("roles", {})) - places = dict(glossary.get("places", {})) - terms = dict(glossary.get("terms", {})) - - keys = list(CORE_CHARACTERS) - for s in sorted(speakers): - if s and s not in keys: - keys.append(s) - char_pairs = _pick({**chars, **roles}, keys) - - term_keys = list(CORE_TERMS) - low = text.lower() - for key in terms: - if key.lower() in low and key not in term_keys: - term_keys.append(key) - term_pairs = _pick(terms, term_keys) - place_pairs = _pick(places, list(places)) - - title = glossary.get("title", {}) - lines = ["【书名与专名规则】"] - if title: - en = title.get("en", "Deal Hunter") - sub = title.get("subtitle_zh", "") - lines.append(f"- 书名：{en}；正文书名写作《{en}》" + (f"，副标题可写“{sub}”" if sub else "")) - lines += [ +# The hard rules a language needs, as data rather than as code. The Chinese +# table is the one the pipeline has always run with, kept verbatim so the +# existing production is unaffected; a glossary may override it with its own +# "rules" list. A language with no built-in table falls back to rules derived +# from the glossary's own terms, and says so - a silently empty rule set is +# how the mislabelled Spanish production happened. +BUILTIN_RULES: dict[str, list[str]] = { + "zh": [ "- Deal Hunter 作为船名、AI 名或书名时保留英文，不译成中文；不要写成《交易猎人》", "- Hunter 作为 AI 的简称保留英文，不译成“猎人”", "- Umbra Audio 保留英文，不音译", @@ -111,16 +86,129 @@ def glossary_block(glossary: dict, speakers: set[str], text: str) -> str: "- salvage = 打捞物 / 打捞（不译作“救援”）", "- bounty = 赏金；bounty hunter = 赏金猎人", "- XO = 副舰长；captain = 舰长", - "", - "【人物与称谓】", - ] - lines += [f"- {en} = {zh}" for en, zh in char_pairs] + ], +} + +# Rules that hold for every language of this book. They are about the book, +# not about a language: *Deal Hunter* is the ship's name, the AI's name and +# the book's title, so it stays in English in every edition, and a name the +# glossary does not carry is never transliterated into existence. +# +# The Chinese table above carries its own wording of these, so the Chinese +# prompt is byte-for-byte what it always was and the existing production +# stays reproducible. +GLOBAL_RULES = [ + "- Deal Hunter 作为船名、AI 名或书名时保留英文，不译成{target_name}", + "- Hunter 作为 AI 的简称保留英文", + "- Umbra Audio 保留英文，不音译", + "- 只翻译原文出现的称谓；不要补全人物全名、头衔或身份", + "- 专名严格按术语表；术语表没有的名字保留原文，不要自行音译", +] + + +def load_glossary(path: Path) -> dict: + return json.loads(path.read_text(encoding="utf-8")) + + +def _translated(value: dict, target: TargetLanguage) -> str: + """The target-language form of a glossary entry. + + The key is the target's own: ``es`` for a Spanish glossary, ``zh`` for a + Chinese one. A glossary written for a different language is not silently + borrowed from - that is how Chinese text ends up under a Spanish name. + """ + for key in (target.glossary_key, "short", "target", "text"): + candidate = value.get(key) + if isinstance(candidate, str) and candidate.strip(): + return candidate + return "" + + +def _pick(mapping: dict, keys: list[str], target: TargetLanguage) -> list[tuple[str, str]]: + out: list[tuple[str, str]] = [] + for key in keys: + if key not in mapping: + continue + value = mapping[key] + if isinstance(value, dict): + rendered = _translated(value, target) + note = value.get("note") + if note: + rendered = f"{rendered}{target.paren_open}{note}{target.paren_close}" + else: + rendered = str(value) + if rendered: + out.append((key, rendered)) + return out + + +def _rules(glossary: dict, target: TargetLanguage, term_pairs: list[tuple[str, str]]) -> list[str]: + """The hard rules for the target language. + + Three sources, in order: the glossary's own ``rules`` list, the + language's built-in table, and - for a language with no table - the + global rules of this book plus the glossary's own terms. A language never + gets an empty rule set, and never gets another language's. + """ + own = glossary.get("rules") + if isinstance(own, list) and own: + return [str(line) for line in own] + built_in = BUILTIN_RULES.get(target.base) + if built_in is not None: + return built_in + # No table for this language: the book's own rules, then the glossary's + # terms, so the model is still told what the hard vocabulary is. + derived = [line.format(target_name=target.chinese_name) for line in GLOBAL_RULES] + if term_pairs: + derived.append("") + derived.append("【硬性术语：违反即为错译】") + derived.extend(f"- {en} = {rendered}" for en, rendered in term_pairs) + return derived + + +def glossary_block( + glossary: dict, + speakers: set[str], + text: str, + target: TargetLanguage, +) -> str: + chars = dict(glossary.get("characters", {})) + roles = dict(glossary.get("roles", {})) + places = dict(glossary.get("places", {})) + terms = dict(glossary.get("terms", {})) + + keys = list(CORE_CHARACTERS) + for s in sorted(speakers): + if s and s not in keys: + keys.append(s) + char_pairs = _pick({**chars, **roles}, keys, target) + + term_keys = list(CORE_TERMS) + low = text.lower() + for key in terms: + if key.lower() in low and key not in term_keys: + term_keys.append(key) + term_pairs = _pick(terms, term_keys, target) + place_pairs = _pick(places, list(places), target) + + title = glossary.get("title", {}) + lines = ["【书名与专名规则】"] + if title: + en = title.get("en", "Deal Hunter") + sub = title.get(f"subtitle_{target.glossary_key}", "") + lines.append( + f"- 书名：{en}；正文书名写作{target.title_open}{en}{target.title_close}" + + (f"，副标题可写{target.quote_open}{sub}{target.quote_close}" if sub else "") + ) + lines += _rules(glossary, target, term_pairs) + lines += ["", "【人物与称谓】"] + lines += [f"- {en} = {rendered}" for en, rendered in char_pairs] if place_pairs: lines.append("【地名与舰名】") - lines += [f"- {en} = {zh}" for en, zh in place_pairs] + lines += [f"- {en} = {rendered}" for en, rendered in place_pairs] if term_pairs: lines.append("【术语】") - lines += [f"- {en} = {zh}" for en, zh in term_pairs] + lines += [f"- {en} = {rendered}" for en, rendered in term_pairs] return "\n".join(lines) @@ -129,20 +217,29 @@ def style_block(glossary: dict) -> str: return "\n".join(f"- {v}" for v in style.values() if isinstance(v, str)) -SYSTEM_ANALYST = """你是一位双语文学编辑，负责在翻译前分析英文小说的场景。 +def system_analyst(target: TargetLanguage) -> str: + return ( + """你是一位双语文学编辑，负责在翻译前分析英文小说的场景。 你的任务不是翻译，而是读出这一段文字的意图、语气、人物关系和可能被直译毁掉的地方。 +摘要和术语建议请用""" + + target.prompt_name + + """写。 只输出 JSON，不要解释，不要 Markdown 代码围栏。""" + ) -SYSTEM_DRAFTER = """你是一位资深文学译者，把英文太空冒险小说译成简体中文。 -你的目标不是逐字对应，而是让中文读者读到和英文读者相同的场景、语气、人物关系和潜台词。 + +def system_drafter(target: TargetLanguage) -> str: + return f"""你是一位资深文学译者，把英文太空冒险小说译成{target.prompt_name}。 +你的目标不是逐字对应，而是让{target.chinese_name}读者读到和英文读者相同的场景、语气、人物关系和潜台词。 可以重组句子、合并或拆分分句，但不得增删事实、不得替作者解释、不得加译注。 -对话要像中文人物在说话；叙述要流畅、克制、有画面感。 -严格使用给出的术语表。未列入术语表的人名、地名、舰名保留英文原文，不要自行音译。 -不要补全原文没有出现的全名、头衔或身份；原文说“Princess Kainda”，不要写成“凯因达·沃尔特斯·福斯特·墨菲公主”。 -Deal Hunter 作为船名、AI 名或书名保留英文；Hunter 作为简称保留英文；Umbra Audio 保留英文。 +对话要像{target.chinese_name}人物在说话；叙述要流畅、克制、有画面感。 +严格使用给出的术语表。未列入术语表的人名、地名、舰名保留原文，不要自行音译。 +不要补全原文没有出现的全名、头衔或身份。 只输出 JSON，不要解释，不要 Markdown 代码围栏。""" -SYSTEM_CRITIC = """你是一位严格的双语审校。对照英文原文和中文译文，只找意义、指代、语气、术语和遗漏方面的问题。 + +def system_critic(target: TargetLanguage) -> str: + return f"""你是一位严格的双语审校。对照英文原文和{target.prompt_name}译文，只找意义、指代、语气、术语和遗漏方面的问题。 不要因为“风格不同”而报错；只有当读者会因此理解错、或人物声音被抹平、或事实被改变时才报。 不要改写全文，只报告问题并给出最小修正建议。 只输出 JSON，不要解释，不要 Markdown 代码围栏。""" @@ -152,11 +249,11 @@ def _paragraph_lines(paragraphs: list[dict]) -> str: lines = [] for p in paragraphs: if p.get("type") == "scene_break": - lines.append(f'[{p["id"]}] (场景分隔)') + lines.append(f"[{p['id']}] (场景分隔)") continue speaker = p.get("speaker") or "narrator" ptype = p.get("type") or "narration" - lines.append(f'[{p["id"]}] ({ptype}; {speaker}) {p["text"]}') + lines.append(f"[{p['id']}] ({ptype}; {speaker}) {p['text']}") return "\n".join(lines) @@ -171,7 +268,7 @@ def _draft_paragraph_lines(paragraphs: list[dict]) -> str: for p in paragraphs: if p.get("type") == "scene_break": continue - lines.append(f'[{p["id"]}] {p["text"]}') + lines.append(f"[{p['id']}] {p['text']}") return "\n".join(lines) @@ -184,6 +281,7 @@ def analysis_prompt( paragraphs: list[dict], previous_summary: str, glossary: dict, + target: TargetLanguage, ) -> str: speakers = {p.get("speaker") or "narrator" for p in paragraphs} text = " ".join(p.get("text", "") for p in paragraphs) @@ -192,17 +290,17 @@ def analysis_prompt( 场景 {scene_index + 1}，第 {chunk_index + 1}/{chunks_in_scene} 段 上一段场景摘要：{previous_summary or "（无，本章开头）"} -{glossary_block(glossary, speakers, text)} +{glossary_block(glossary, speakers, text, target)} {style_block(glossary)} 请分析下面这段英文。输出 JSON： {{ - "summary_zh": "1-2句中文（80字以内），概括本段发生了什么、人物想要什么、语气如何；供下一段翻译参考", + "summary": "1-2句{target.chinese_name}（80字以内），概括本段发生了什么、人物想要什么、语气如何；供下一段翻译参考", "register": "正式/中性/口语（15字以内）", "tone": "例如：紧张、戏谑、疲惫、克制（15字以内）", "participants": ["本段实际说话或被点名的人物"], - "proposed_terms": [{{"en": "术语表里没有的专名", "zh": "建议译名", "why": "简短理由"}}], + "proposed_terms": [{{"en": "术语表里没有的专名", "{target.glossary_key}": "建议译名", "why": "简短理由"}}], "risks": [{{"id": "段落id", "issue": "直译会错在哪里", "guidance": "译者应注意什么"}}] }} @@ -223,6 +321,7 @@ def draft_prompt( analysis: dict, previous_summary: str, glossary: dict, + target: TargetLanguage, ) -> str: speakers = {p.get("speaker") or "narrator" for p in paragraphs} text = " ".join(p.get("text", "") for p in paragraphs) @@ -232,31 +331,29 @@ def draft_prompt( 场景 {scene_index + 1}，第 {chunk_index + 1}/{chunks_in_scene} 段 上一段场景摘要：{previous_summary or "（无，本章开头）"} -{glossary_block(glossary, speakers, text)} +{glossary_block(glossary, speakers, text, target)} {style_block(glossary)} 翻译前的场景分析（必须遵守其中的语气、人物关系和风险提示）： {analysis_text} -请把下面英文译为简体中文。输出 JSON： -{{"paragraphs": [{{"id": "段落id", "zh": "中文译文"}}]}} +请把下面英文译为{target.prompt_name}。输出 JSON： +{{"paragraphs": [{{"id": "段落id", "{target.glossary_key}": "{target.chinese_name}译文"}}]}} 规则： - 段落 id 必须与原文一一对应，数量和顺序不变。 -- 每个 id 只输出一段中文；不要把两段合并成一个 id。 +- 每个 id 只输出一段译文；不要把两段合并成一个 id。 - **每个 id 的译文只能覆盖该 id 自己的英文。** 原文常在句子中间被切成多段 （例如一段是"The chair in the middle has"，下一段是'"Captain"'），译文也必须 - 照这个切分来，哪怕某一段的中文读起来不完整。绝对不要把相邻段落的内容提前 + 照这个切分来，哪怕某一段的译文读起来不完整。绝对不要把相邻段落的内容提前 写进前一个 id，也不要把本段的内容挪到下一个 id。宁可让某段译文是半句。 - 只输出译文。绝对不要把段落 id、说话人、类型、括号注释写进译文。 - 不要输出英文原文，不要输出解释。 -- 不要把英文单词留在译文里（术语表允许保留的专名和书名除外）；每个英文实词都必须译成中文。 -- 对话用中文“”，并保证每个引号成对；不要把英文直引号 " 原样搬进译文。叙述中不要滥用引号。 -- 如果原文有省略号、破折号、打断，保留其节奏；省略号用……，破折号用——。 -- 专名严格按术语表；术语表没有的名字保留英文。 -- 指恒星系时，system / star system 译作“星系”（如 Albert system = 阿尔伯特星系；236 systems = 236 个星系）。 -- 指船载、音频、计算机等设备时，system 译作“系统”（如 ship systems = 飞船系统；Umbra Audio system = Umbra Audio 系统）。不要一律译作“星系”。 +- 不要把英文单词留在译文里（术语表允许保留的专名和书名除外）；每个英文实词都必须译成{target.chinese_name}。 +- 对话用{target.quote_open}{target.quote_close}，并保证每个引号成对；不要把英文直引号 " 原样搬进译文。叙述中不要滥用引号。 +- 如果原文有省略号、破折号、打断，保留其节奏；省略号用{target.ellipsis}，破折号用{target.dash}。 +- 专名严格按术语表；术语表没有的名字保留原文。 英文原文： {_draft_paragraph_lines(paragraphs)}""" @@ -268,6 +365,7 @@ def critic_prompt( paragraphs: list[dict], draft: dict[str, str], glossary: dict, + target: TargetLanguage, ) -> str: speakers = {p.get("speaker") or "narrator" for p in paragraphs} text = " ".join(p.get("text", "") for p in paragraphs) @@ -281,17 +379,15 @@ def critic_prompt( "speaker": p.get("speaker"), "type": p.get("type"), "en": p["text"], - "zh": draft.get(p["id"], ""), + target.glossary_key: draft.get(p["id"], ""), } ) return f"""书名：Deal Hunter（交易猎人） 本章：{chapter_title} -{glossary_block(glossary, speakers, text)} +{glossary_block(glossary, speakers, text, target)} -请逐段对照英文与中文，只报告会让读者理解错、或改变人物声音、或丢失/增加事实的问题。 - -注意：原文 “236 systems in the Dial space sector” 中的 systems 指恒星系，译作“星系”是正确的，不要改成“系统”。ship systems / audio system 才译作“系统”。 +请逐段对照英文与{target.chinese_name}译文，只报告会让读者理解错、或改变人物声音、或丢失/增加事实的问题。 输出 JSON： {{ @@ -301,9 +397,9 @@ def critic_prompt( "id": "段落id", "severity": "block|major|minor", "type": "meaning|pronoun|irony|term|omission|addition|register|misalignment", - "problem": "中文说明问题", + "problem": "{target.chinese_name}说明问题", "source_excerpt": "英文片段（不超过30词）", - "target_excerpt": "中文片段（不超过40字）", + "target_excerpt": "{target.chinese_name}片段（不超过40字）", "fix": "最小修正建议，不要重译整段" }} ] @@ -336,18 +432,19 @@ def consistency_prompt( draft: dict[str, str], glossary: dict, character_notes: str, + target: TargetLanguage, ) -> str: speakers = {p.get("speaker") or "narrator" for p in paragraphs} text = " ".join(p.get("text", "") for p in paragraphs) pairs = [ - {"id": p["id"], "speaker": p.get("speaker"), "zh": draft.get(p["id"], "")} + {"id": p["id"], "speaker": p.get("speaker"), target.glossary_key: draft.get(p["id"], "")} for p in paragraphs if p.get("type") != "scene_break" ] return f"""书名：Deal Hunter（交易猎人） 本章：{chapter_title} -{glossary_block(glossary, speakers, text)} +{glossary_block(glossary, speakers, text, target)} 人物声音约束： {character_notes} diff --git a/tools/translation/qa.py b/tools/translation/qa.py index fa3b5a7..3f7f90d 100644 --- a/tools/translation/qa.py +++ b/tools/translation/qa.py @@ -1,11 +1,16 @@ # Copyright 2026 Umbra Tools (umbratools.com) # SPDX-License-Identifier: Apache-2.0 -"""Mechanical QA for the Deal Hunter Chinese translation. +"""Mechanical QA for a translated chapter. This does not judge literary quality. It catches the defects that can be detected without a human: missing paragraphs, leaked prompt metadata, stray English, unbalanced quotes, duplicated punctuation and forbidden title forms. + +The checks are the target language's own. A Spanish draft is full of Latin +words, so the "stray English" check cannot run there - it would flag the +whole book. The quote characters, the punctuation and the script count come +from the same registry the prompts and the renderer use. """ from __future__ import annotations @@ -18,14 +23,40 @@ from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parent)) +from book import SUBTITLES +from languages import SCRIPT_LATIN, TargetLanguage, resolve, script_counts # noqa: E402 from pipeline import latin_residue # noqa: E402 +CHINESE = resolve("zh-Hans") + + +def forbidden_titles(target: TargetLanguage) -> list[str]: + """The translated book title, in the target's own book-title marks. + + The rule is "the book title stays in English", so the form that must not + appear in the text is the *translated* title. The Chinese table is the + one the pipeline has always used; every other language derives its own + from the subtitle table, and a language with no subtitle has nothing to + forbid. + """ + if target.base == "zh": + return ["《交易猎人》"] + subtitle = SUBTITLES.get(target.base, "") + if not subtitle: + return [] + return [f"{target.title_open}{subtitle}{target.title_close}"] + def _load(path: Path) -> dict: return json.loads(path.read_text(encoding="utf-8")) -def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]: +def check_chapter( + chapter: dict, + data: dict, + target: TargetLanguage | None = None, +) -> tuple[list[dict], dict]: + target = target or CHINESE drafts: dict[str, str] = data.get("paragraphs", {}) by_id = {p["id"]: p for p in chapter["paragraphs"] if p.get("type") != "scene_break"} expected = list(by_id) @@ -38,40 +69,46 @@ def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]: if pid not in by_id: issues.append({"id": pid, "level": "major", "kind": "unknown_id", "detail": "id not in source"}) + # A target written in Latin script has Latin words by definition, so the + # residue check only applies where Latin script is foreign. + check_latin = SCRIPT_LATIN not in target.scripts + forbidden = forbidden_titles(target) + double_punctuation = r"[。，！？；：]{2,}" if target.base == "zh" else r"[.!?;]{2,}" + chars = 0 for pid, zh in drafts.items(): zh = str(zh) - chars += len(re.findall(r"[\u4e00-\u9fff]", zh)) + counts = script_counts(zh) + chars += sum(counts.get(script, 0) for script in target.scripts) if re.search(r"^\[ch-[^\]]+\]", zh) or re.search(r"（(?:对话|旁白|叙述)(?:；[^）]*)?）", zh): issues.append({"id": pid, "level": "block", "kind": "metadata_leak", "detail": zh[:40]}) if '"' in zh: issues.append({"id": pid, "level": "major", "kind": "straight_quote", "detail": zh[:60]}) - if zh.count("“") != zh.count("”"): + opens = zh.count(target.quote_open) + closes = zh.count(target.quote_close) + if opens != closes: issues.append( { "id": pid, "level": "major", "kind": "quote_balance", - "detail": f"{zh.count('“')} open / {zh.count('”')} close", + "detail": f"{opens} open / {closes} close", } ) - if re.search(r"[。，！？；：]{2,}", zh): + if re.search(double_punctuation, zh): issues.append({"id": pid, "level": "minor", "kind": "double_punctuation", "detail": zh[:60]}) - if "《交易猎人》" in zh: - issues.append({"id": pid, "level": "major", "kind": "title_translated", "detail": zh[:60]}) - common, proper = latin_residue(zh) - if common: - issues.append( - {"id": pid, "level": "major", "kind": "english_residue", "detail": ", ".join(common)} - ) - if proper: - issues.append( - {"id": pid, "level": "info", "kind": "proper_name_review", "detail": ", ".join(proper)} - ) + for form in forbidden: + if form in zh: + issues.append({"id": pid, "level": "major", "kind": "title_translated", "detail": zh[:60]}) + break + if check_latin: + common, proper = latin_residue(zh) + if common: + issues.append({"id": pid, "level": "major", "kind": "english_residue", "detail": ", ".join(common)}) + if proper: + issues.append({"id": pid, "level": "info", "kind": "proper_name_review", "detail": ", ".join(proper)}) source = by_id.get(pid, {}).get("text", "") - star_system = re.search( - r"\bstar system\b|\b(?!The\b)[A-Z][a-z]+\s+system\b|\b\d+\s+systems\b", source - ) + star_system = re.search(r"\bstar system\b|\b(?!The\b)[A-Z][a-z]+\s+system\b|\b\d+\s+systems\b", source) if star_system: phrase = star_system.group(0).lower() if any( @@ -89,7 +126,11 @@ def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]: ) ): star_system = None - if star_system and "系统" in zh: + # The mistranslation this catches is a Chinese one: a star system + # rendered with the word for an installed system. In another language + # the wrong word is a different word and the critic, not this check, + # is what finds it. + if star_system and target.base == "zh" and "系统" in zh: issues.append( {"id": pid, "level": "major", "kind": "system_mistranslated", "detail": "star system -> 系统"} ) @@ -98,23 +139,26 @@ def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]: "chapter_id": chapter["id"], "chapter_index": chapter["index"], "title": chapter["title"], + "target_language": target.code, "expected": len(expected), "translated": len(drafts), - "chinese_chars": chars, + "target_chars": chars, "issues": len(issues), } return issues, metrics def main() -> None: - ap = argparse.ArgumentParser(description="Mechanical QA for the Chinese translation.") + ap = argparse.ArgumentParser(description="Mechanical QA for a translated chapter.") ap.add_argument("--book", type=Path, required=True) ap.add_argument("--run-dir", type=Path, required=True) + ap.add_argument("--target-language", default=None) ap.add_argument("--json-out", type=Path) ap.add_argument("--strict", action="store_true", help="exit 1 on any block/major issue") args = ap.parse_args() book = _load(args.book) + target = resolve(args.target_language or book.get("target_language") or "zh-Hans") all_issues: list[dict] = [] metrics: list[dict] = [] @@ -123,7 +167,7 @@ def main() -> None: if not path.exists(): continue data = _load(path) - issues, m = check_chapter(chapter, data) + issues, m = check_chapter(chapter, data, target) for issue in issues: issue["chapter_id"] = chapter["id"] issue["chapter_title"] = chapter["title"] @@ -136,11 +180,12 @@ def main() -> None: by_level[issue["level"]] = by_level.get(issue["level"], 0) + 1 by_kind[issue["kind"]] = by_kind.get(issue["kind"], 0) + 1 + print(f"TARGET: {target.label}") print("CHAPTERS") for m in metrics: print( f" {m['chapter_id']:6} {m['title'][:28]:28} " - f"{m['translated']:>4}/{m['expected']:<4} paras {m['chinese_chars']:>6} chars {m['issues']:>3} issues" + f"{m['translated']:>4}/{m['expected']:<4} paras {m['target_chars']:>6} chars {m['issues']:>3} issues" ) print() print("ISSUES BY LEVEL:", dict(sorted(by_level.items()))) @@ -148,9 +193,7 @@ def main() -> None: print() for issue in all_issues: if issue["level"] in {"block", "major"}: - print( - f" [{issue['level']:5}] {issue['kind']:20} {issue['chapter_id']}/{issue['id']}: {issue['detail']}" - ) + print(f" [{issue['level']:5}] {issue['kind']:20} {issue['chapter_id']}/{issue['id']}: {issue['detail']}") elif issue["level"] == "minor": print(f" [minor] {issue['kind']:20} {issue['chapter_id']}/{issue['id']}: {issue['detail']}") else: diff --git a/tools/translation/render.py b/tools/translation/render.py index 2805f43..3917375 100644 --- a/tools/translation/render.py +++ b/tools/translation/render.py @@ -7,6 +7,12 @@ The AsciiDoc file is the canonical text output. The ODT is produced from a Flat ODF (.fodt) document so that styles, page breaks and the table of contents are under our control, then converted with LibreOffice. The TOC is updated through UNO, because a plain ``--convert-to`` leaves an empty index. + +The typography is the target language's: its fonts, its language and country +codes, its chapter-numbering convention, its quotation marks, and the fixed +editorial strings the renderer writes around the translation. A Spanish +edition typeset in Chinese fonts with Chinese chapter headings is the same +defect as a Chinese draft under a Spanish name, one level down. """ from __future__ import annotations @@ -23,10 +29,11 @@ import time from pathlib import Path from xml.sax.saxutils import escape +from book import SUBTITLES +from languages import SCRIPT_LATIN, TargetLanguage, resolve from llm import DEFAULT_ANALYST, DEFAULT_CRITIC, DEFAULT_MODEL -FONT_SERIF = "Noto Serif CJK SC" -FONT_SANS = "Noto Sans CJK SC" +CHINESE = resolve("zh-Hans") def _esc(text: str) -> str: @@ -36,14 +43,21 @@ def _esc(text: str) -> str: LATIN_RUN = re.compile(r"[A-Za-z][A-Za-z0-9'’.\-·]*(?:\s+[A-Za-z][A-Za-z0-9'’.\-·]*)*") -def _rich_esc(text: str) -> str: +def _rich_esc(text: str, target: TargetLanguage | None = None) -> str: """Escape text and mark Latin runs as English. A Chinese paragraph tells LibreOffice to use Chinese line-breaking for the whole run, which can split an English name in the middle of a word ("Deal Hu / nter"). A character style with fo:language="en" keeps the Latin run on Western line-breaking rules without changing its font. + + For a target written in Latin script this is not applied at all: the whole + text is Latin, and marking it as English would tell the renderer that a + Spanish paragraph is English. """ + target = target or CHINESE + if SCRIPT_LATIN in target.scripts: + return _esc(text) out: list[str] = [] last = 0 for m in LATIN_RUN.finditer(text): @@ -82,18 +96,38 @@ def _ordered_chapters(book: dict, run_dir: Path | None, include_missing: bool = return [c for c in out if chapter_drafts(run_dir, c["id"])] +def _title_of(chapter: dict) -> str: + """The chapter's title in the target language. -def _cn_num(n: int) -> str: - digits = "零一二三四五六七八九" - if n <= 0: - return str(n) - if n <= 10: - return "十" if n == 10 else digits[n] - if n < 20: - return "十" + digits[n - 10] - if n < 100: - return digits[n // 10] + "十" + (digits[n % 10] if n % 10 else "") - return str(n) + The field was called ``title_zh`` while the pipeline was Chinese-only + and is now called ``title_target``. The old name is still read, so a book + file written by an earlier version keeps working. + """ + return chapter.get("title_target") or chapter.get("title_zh") or chapter.get("title") or "" + + +def _subtitle_of(book: dict, target: TargetLanguage | None = None) -> str: + """The book's subtitle in the target language. + + The old book format had no subtitle field: it stored the subtitle in the + book-level ``title_zh``, so that is read as a fallback. The table is the + last resort, and a language with no entry in it gets no subtitle rather + than another language's. + """ + target = target or CHINESE + return ( + book.get("subtitle_target") or book.get("subtitle_zh") or book.get("title_zh") or SUBTITLES.get(target.base, "") + ) + + +def _book_title(book: dict) -> str: + """The book's title. + + The title stays in English for every language: "Deal Hunter" is also the + ship's name and the AI's name, so translating it would rename a + character. + """ + return book.get("title_target") or book.get("title") or "Deal Hunter" def _body_chapter_number(book: dict, chapter: dict) -> int: @@ -124,24 +158,34 @@ def _is_author_note_code(text: str) -> bool: # --------------------------------------------------------------------- AsciiDoc -def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str = "") -> str: +def build_asciidoc( + book: dict, + run_dir: Path, + *, + target: TargetLanguage | None = None, + sample: bool = False, + date: str = "", +) -> str: + target = target or CHINESE chapters = _ordered_chapters(book, run_dir) + title = _book_title(book) + subtitle = _subtitle_of(book, target) lines: list[str] = [] - lines.append("= Deal Hunter: 交易猎人") + lines.append(f"= {title}: {subtitle}" if subtitle else f"= {title}") lines.append("B. D. Murphy") lines.append(":doctype: book") - lines.append(":lang: zh-CN") + lines.append(f":lang: {target.asciidoc_lang}") lines.append(":toc: left") lines.append(":toclevels: 2") lines.append(f":revdate: {date or time.strftime('%Y-%m-%d')}") if sample: - lines.append(":revremark: 审阅样本") + lines.append(f":revremark: {target.string('review_sample')}") lines.append(":icons: font") lines.append("") - lines.append("// 译者：Umbra Tools") + lines.append(f"// {target.string('translator_credit')}") if sample: lines.append("") - lines.append("NOTE: 本文件是简体中文翻译的审阅样本。已译部分见目录；未列出的章节将在全文运行中补齐。") + lines.append(f"NOTE: {target.string('sample_note')}") lines.append("") seen_audio = False @@ -154,15 +198,15 @@ def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str continue seen_audio = True lines.append(f"// source: {c['id']} (audio front matter)") - lines.append("== 音频版序言") + lines.append(f"== {target.string('audio_preface')}") lines.append("") - lines.append("以下是《Deal Hunter》有声版的开场白。") + lines.append(target.string("audio_preface_intro")) lines.append("") for ac in [x for x in book["chapters"] if x["matter"] == "front_audio"]: adrafts = chapter_drafts(run_dir, ac["id"]) if not adrafts: continue - lines.append(f"=== {ac['title_zh']}") + lines.append(f"=== {_title_of(ac)}") lines.append("") for p in ac["paragraphs"]: zh = adrafts.get(p["id"]) @@ -178,9 +222,9 @@ def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str if c["matter"] == "body": n = _body_chapter_number(book, c) - heading = f"第{_cn_num(n)}章　{c['title_zh']}" + heading = target.chapter_heading(n, _title_of(c)) else: - heading = c["title_zh"] + heading = _title_of(c) lines.append(f"// source: {c['id']} (database row {c['row']}), {len(c['paragraphs'])} paragraphs") lines.append(f"== {heading}") lines.append("") @@ -206,15 +250,25 @@ def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str # --------------------------------------------------------------------- Flat ODF -STYLES = f""" +def build_styles(target: TargetLanguage | None = None) -> str: + """The document stylesheet for the target language. + + The fonts, the language and country codes and the writing direction are + the target's. A document that declares ``fo:language="zh"`` while its + text is Spanish tells LibreOffice to hyphenate and shape it as Chinese, + which is how a Latin-script edition ends up with the wrong line breaks. + """ + target = target or CHINESE + direction = ' fo:direction="rtl"' if target.rtl else "" + return f""" - - + + - + @@ -294,31 +348,48 @@ STYLES = f""" """ -def _odt_paragraph(style: str, text: str, outline: int | None = None) -> str: +def _odt_paragraph( + style: str, + text: str, + outline: int | None = None, + target: TargetLanguage | None = None, +) -> str: + body = _rich_esc(text, target) if outline: - return f'{_rich_esc(text)}' - return f'{_rich_esc(text)}' + return f'{body}' + return f'{body}' -def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = "") -> str: +def build_fodt( + book: dict, + run_dir: Path, + *, + target: TargetLanguage | None = None, + sample: bool = False, + date: str = "", +) -> str: + target = target or CHINESE chapters = _ordered_chapters(book, run_dir) body: list[str] = [] + title = _book_title(book) + subtitle = _subtitle_of(book, target) # Title page - body.append(_odt_paragraph("Title", "Deal Hunter")) - body.append(_odt_paragraph("Subtitle", "交易猎人")) - body.append(_odt_paragraph("Author", "B·D·墨菲 著")) - body.append(_odt_paragraph("Small", "Umbra Tools 译")) + body.append(_odt_paragraph("Title", title, target=target)) + if subtitle: + body.append(_odt_paragraph("Subtitle", subtitle, target=target)) + body.append(_odt_paragraph("Author", target.string("author_line"), target=target)) + body.append(_odt_paragraph("Small", target.string("translator_line"), target=target)) if sample: - body.append(_odt_paragraph("Small", "简体中文译本 · 审阅样本")) - body.append(_odt_paragraph("Small", date or time.strftime("%Y-%m-%d"))) + body.append(_odt_paragraph("Small", target.string("edition_label"), target=target)) + body.append(_odt_paragraph("Small", date or time.strftime("%Y-%m-%d"), target=target)) body.append('') # Table of contents body.append( '' '' - '目录' + f'{target.string("toc_title")}' '' "" '' @@ -329,19 +400,9 @@ def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = " ) if sample: - body.append(_odt_paragraph("ChapterHeading", "审阅样本说明", 1)) - body.append( - _odt_paragraph( - "FrontText", - "本文件是《Deal Hunter》简体中文翻译的审阅样本。已译部分见目录；未列出的章节将在全文运行中补齐。", - ) - ) - body.append( - _odt_paragraph( - "FrontText", - "翻译流程：段落重建 → 意图分析 → 初译 → 双语审校。术语见 glossary.json，审校意见见 review.html。", - ) - ) + body.append(_odt_paragraph("ChapterHeading", target.string("sample_heading"), 1, target=target)) + body.append(_odt_paragraph("FrontText", target.string("sample_note"), target=target)) + body.append(_odt_paragraph("FrontText", target.string("sample_process"), target=target)) seen_audio = False for c in chapters: @@ -352,46 +413,41 @@ def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = " if seen_audio: continue seen_audio = True - body.append(_odt_paragraph("ChapterHeading", "音频版序言", 1)) - body.append(_odt_paragraph("FrontText", "以下是《Deal Hunter》有声版的开场白。")) + body.append(_odt_paragraph("ChapterHeading", target.string("audio_preface"), 1, target=target)) + body.append(_odt_paragraph("FrontText", target.string("audio_preface_intro"), target=target)) for ac in [x for x in book["chapters"] if x["matter"] == "front_audio"]: adrafts = chapter_drafts(run_dir, ac["id"]) if not adrafts: continue - body.append(_odt_paragraph("SectionHeading", ac["title_zh"], 2)) + body.append(_odt_paragraph("SectionHeading", _title_of(ac), 2, target=target)) for p in ac["paragraphs"]: zh = adrafts.get(p["id"]) if not zh: continue if p.get("type") == "scene_break": - body.append(_odt_paragraph("SceneBreak", "* * *")) + body.append(_odt_paragraph("SceneBreak", "* * *", target=target)) else: - body.append(_odt_paragraph("FrontText", zh)) + body.append(_odt_paragraph("FrontText", zh, target=target)) continue if c["matter"] == "body": n = _body_chapter_number(book, c) - heading = f"第{_cn_num(n)}章　{c['title_zh']}" + heading = target.chapter_heading(n, _title_of(c)) else: - heading = c["title_zh"] - body.append(_odt_paragraph("ChapterHeading", heading, 1)) + heading = _title_of(c) + body.append(_odt_paragraph("ChapterHeading", heading, 1, target=target)) for p in c["paragraphs"]: zh = drafts.get(p["id"]) if not zh: continue if p.get("type") == "scene_break": - body.append(_odt_paragraph("SceneBreak", "* * *")) + body.append(_odt_paragraph("SceneBreak", "* * *", target=target)) continue - body.append(_odt_paragraph(_paragraph_style(p, c["matter"]), zh)) + body.append(_odt_paragraph(_paragraph_style(p, c["matter"]), zh, target=target)) if sample: - body.append(_odt_paragraph("ChapterHeading", "样本结束", 1)) - body.append( - _odt_paragraph( - "FrontText", - "以上为审阅样本。全文运行将覆盖其余章节；审校问题清单会随各章翻译记录一并提供。", - ) - ) + body.append(_odt_paragraph("ChapterHeading", target.string("sample_end"), 1, target=target)) + body.append(_odt_paragraph("FrontText", target.string("sample_end_note"), target=target)) return f""" -{STYLES} +{build_styles(target)} - Deal Hunter · 交易猎人 + {_esc(title + (f" · {subtitle}" if subtitle else ""))} 1 @@ -526,17 +582,21 @@ def render( sample: bool = False, date: str = "", pdf: bool = True, + target_language: str = "zh-Hans", ) -> dict[str, Path]: + target = resolve(target_language) book = json.loads(book_path.read_text(encoding="utf-8")) out_dir.mkdir(parents=True, exist_ok=True) - stem = "deal-hunter-zh-sample" if sample else "deal-hunter-zh" + stem = f"deal-hunter-{target.base}-sample" if sample else f"deal-hunter-{target.base}" asciidoc = out_dir / f"{stem}.asciidoc" fodt = out_dir / f"{stem}.fodt" odt = out_dir / f"{stem}.odt" pdf_path = out_dir / f"{stem}.pdf" if pdf else None - asciidoc.write_text(build_asciidoc(book, run_dir, sample=sample, date=date), encoding="utf-8") - fodt.write_text(build_fodt(book, run_dir, sample=sample, date=date), encoding="utf-8") + asciidoc.write_text( + build_asciidoc(book, run_dir, target=target, sample=sample, date=date), encoding="utf-8" + ) + fodt.write_text(build_fodt(book, run_dir, target=target, sample=sample, date=date), encoding="utf-8") convert_with_uno(fodt, odt, pdf_path) out = {"asciidoc": asciidoc, "fodt": fodt, "odt": odt} if pdf_path: @@ -545,17 +605,23 @@ def render( # Enhance EPUB with cover image and metadata epub_path = out_dir / f"{stem}.epub" cover_path = book_path.parent / "covers" / "cover.png" - enhance_epub(epub_path, book, cover_path if cover_path.exists() else None) + enhance_epub(epub_path, book, cover_path if cover_path.exists() else None, target) out["epub"] = epub_path return out -def enhance_epub(epub_path: Path, book: dict, cover_path: Path | None = None) -> None: +def enhance_epub( + epub_path: Path, + book: dict, + cover_path: Path | None = None, + target: TargetLanguage | None = None, +) -> None: """Add cover image, ISBN, publisher, and translator metadata to EPUB. Pandoc's ODT->EPUB conversion loses some metadata. This function post-processes the EPUB to add cover image and complete metadata. """ + target = target or CHINESE import zipfile import tempfile import shutil @@ -625,26 +691,41 @@ def enhance_epub(epub_path: Path, book: dict, cover_path: Path | None = None) -> cover_meta.set("name", "cover") cover_meta.text = cover_id - # Add CJK-optimized CSS + # Add the target language's typography to the stylesheet css_path = tmpdir / "EPUB" / "styles" / "stylesheet1.css" if css_path.exists(): css_content = css_path.read_text(encoding="utf-8") - cjk_css = ''' - -/* CJK typography for Chinese body text */ -html { - font-family: "Noto Serif CJK SC", "Source Han Serif SC", "WenQuanYi Micro Hei", serif; -} -html[lang="zh-CN"], body[lang="zh-CN"] { - font-family: "Noto Serif CJK SC", "Source Han Serif SC", "WenQuanYi Micro Hei", serif; + lang = target.asciidoc_lang + font_stack = target.epub_font_stack + if target.cjk: + css_extra = f""" +/* CJK typography for {lang} body text */ +html {{ + font-family: {font_stack}; +}} +html[lang="{lang}"], body[lang="{lang}"] {{ + font-family: {font_stack}; line-break: strict; word-break: normal; -} -p[lang="zh-CN"] { - font-family: "Noto Serif CJK SC", "Source Han Serif SC", "WenQuanYi Micro Hei", serif; +}} +p[lang="{lang}"] {{ + font-family: {font_stack}; line-break: strict; word-break: normal; -} +}}""" + else: + css_extra = f""" +/* Typography for {lang} body text */ +html {{ + font-family: {font_stack}; +}} +html[lang="{lang}"], body[lang="{lang}"] {{ + font-family: {font_stack}; +}} +p[lang="{lang}"] {{ + font-family: {font_stack}; +}}""" + target_css = css_extra + ''' /* Dialogue styling */ .utterance__text { font-size: 1.1em; @@ -666,7 +747,7 @@ h1, h2 { font-size: 0.8em; } ''' - css_path.write_text(css_content + chr(10) + cjk_css, encoding="utf-8") + css_path.write_text(css_content + chr(10) + target_css, encoding="utf-8") # Add publisher metadata metadata = root.find("opf:metadata", ns) @@ -677,6 +758,8 @@ h1, h2 { translator = etree.SubElement(metadata, "{http://purl.org/dc/elements/1.1/}contributor") translator.set("{http://www.idpf.org/2007/opf}role", "trl") translator.text = "Umbra Tools Translation Team" + language = etree.SubElement(metadata, "{http://purl.org/dc/elements/1.1/}language") + language.text = target.asciidoc_lang tree.write(str(opf_path), xml_declaration=True, encoding="UTF-8", standalone=True) @@ -721,11 +804,13 @@ def write_manifest( out_dir: Path, sample: bool, files: dict[str, Path], + target: TargetLanguage | None = None, ) -> Path: """Write the reproducibility manifest next to the rendered files. It contains hashes, counts and model names, never the book text. """ + target = target or CHINESE book = json.loads(book_path.read_text(encoding="utf-8")) chapters: list[dict] = [] total = translated = issues = scene_breaks = 0 @@ -746,7 +831,7 @@ def write_manifest( "index": c["index"], "matter": c["matter"], "title": c["title"], - "title_zh": c["title_zh"], + "title_target": _title_of(c), "paragraphs_expected": expected, "paragraphs_translated": n, "critic_issues": chapter_issues, @@ -755,7 +840,9 @@ def write_manifest( glossary = run_dir.parent / "glossary" / "glossary.json" manifest = { "title": book.get("title"), - "title_zh": book.get("title_zh"), + "title_target": _book_title(book), + "subtitle_target": _subtitle_of(book, target), + "target_language": target.code, "author": book.get("author"), "generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "sample": sample, @@ -788,14 +875,21 @@ def write_manifest( def main() -> None: - ap = argparse.ArgumentParser(description="Render Deal Hunter Chinese AsciiDoc + ODT.") + ap = argparse.ArgumentParser(description="Render Deal Hunter AsciiDoc + ODT in the target language.") ap.add_argument("--book", type=Path, required=True) ap.add_argument("--run-dir", type=Path, required=True) ap.add_argument("--out-dir", type=Path, required=True) + ap.add_argument( + "--target-language", + default=None, + help="the language the translations are in. Defaults to the book's own.", + ) ap.add_argument("--sample", action="store_true") ap.add_argument("--date", default="") ap.add_argument("--no-pdf", action="store_true") args = ap.parse_args() + book = json.loads(args.book.read_text(encoding="utf-8")) + target_language = args.target_language or book.get("target_language") or "zh-Hans" out = render( book_path=args.book, run_dir=args.run_dir, @@ -803,6 +897,7 @@ def main() -> None: sample=args.sample, date=args.date, pdf=not args.no_pdf, + target_language=target_language, ) out["manifest"] = write_manifest( book_path=args.book, @@ -810,6 +905,7 @@ def main() -> None: out_dir=args.out_dir, sample=args.sample, files=out, + target=resolve(target_language), ) for k, v in out.items(): print(f"{k:9} {v}") diff --git a/tools/translation/tests/test_align.py b/tools/translation/tests/test_align.py index cd59784..a486774 100644 --- a/tools/translation/tests/test_align.py +++ b/tools/translation/tests/test_align.py @@ -31,8 +31,13 @@ from align import ( # noqa: E402 repartition, source_utterances, split_target_sentences, + system_aligner, verify_translation, ) +from languages import resolve # noqa: E402 + +CHINESE = resolve("zh-Hans") +SPANISH = resolve("es-ES") def bundle_fixture() -> dict[str, dict]: @@ -840,5 +845,56 @@ class DriftVerificationTests(unittest.TestCase): self.assertEqual(report["needs_review"], 2) +class TargetLanguageTests(unittest.TestCase): + """The alignment pass must work in the language it is aligning.""" + + def test_the_aligner_is_told_the_target_language(self) -> None: + self.assertIn("中文", system_aligner(CHINESE)) + self.assertIn("西班牙语", system_aligner(SPANISH)) + self.assertNotIn("中文", system_aligner(SPANISH)) + + def test_a_spanish_sentence_splits_on_the_spanish_terminators(self) -> None: + text = "La silla está sucia. ¿Who está ahí? ¡Ven!" + sentences = split_target_sentences(text, SPANISH) + self.assertEqual(len(sentences), 3) + self.assertTrue(sentences[0].endswith("sucia.")) + self.assertTrue(sentences[1].endswith("ahí?")) + + def test_a_chinese_sentence_splits_on_the_chinese_terminators(self) -> None: + text = "椅子很脏。谁在那里？快来！" + self.assertEqual(len(split_target_sentences(text, CHINESE)), 3) + + def test_the_mechanical_split_uses_the_target_clause_marks(self) -> None: + """A Spanish translation merges sentences with a comma, not a full stop.""" + text = "La silla estaba sucia, y el suelo tenía polvo" + pieces = mechanical_split(text, 2, SPANISH) + self.assertIsNotNone(pieces) + assert pieces is not None + self.assertEqual(len(pieces), 2) + # The contract is whitespace-insensitive: the split never rewrites + # the translation, it only divides it. + self.assertEqual("".join(pieces).replace(" ", ""), text.replace(" ", "")) + + def test_the_delivery_carries_the_base_language_code(self) -> None: + alignment = align_paragraph( + { + "id": "ch-1-p0000", + "type": "narration", + "text": "The ship was large.", + "source_indices": [0], + "source_ids": ["utt-100"], + }, + "El barco era grande.", + bundle_fixture(), + ) + delivery = build_delivery( + {"target_language": "es-ES"}, + [({"id": "ch-1", "paragraphs": []}, [alignment])], + bundle_fixture(), + target=SPANISH, + ) + self.assertEqual(delivery["target_language"], "es") + + if __name__ == "__main__": unittest.main() diff --git a/tools/translation/tests/test_render.py b/tools/translation/tests/test_render.py new file mode 100644 index 0000000..1cc5cae --- /dev/null +++ b/tools/translation/tests/test_render.py @@ -0,0 +1,158 @@ +# Copyright 2026 Umbra Tools (umbratools.com) +# SPDX-License-Identifier: Apache-2.0 + +"""Tests for the renderer. + +The renderer writes the book the reader receives, so its typography is part +of the translation: the fonts, the language codes, the chapter-numbering +convention and the fixed editorial strings all have to be the target +language's. A Spanish edition typeset with Chinese fonts and Chinese chapter +headings is the same defect as a Chinese draft filed under a Spanish name, +one level down and much easier to miss. +""" + +from __future__ import annotations + +import json +import sys +import tempfile +import unittest +from pathlib import Path + +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE.parent)) + +from languages import resolve # noqa: E402 +from render import build_asciidoc, build_fodt, build_styles # noqa: E402 + +CHINESE = resolve("zh-Hans") +SPANISH = resolve("es-ES") +ARABIC = resolve("ar-SA") + + +def _book(target: str = "zh-Hans", subtitle: str = "交易猎人") -> dict: + return { + "title": "Deal Hunter", + "title_target": "Deal Hunter", + "subtitle_target": subtitle, + "target_language": target, + "chapters": [ + { + "id": "ch-6", + "index": 6, + "row": 6, + "matter": "body", + "title": "Salvage", + "title_target": "Salvage", + "paragraphs": [ + {"id": "ch-6-p0000", "type": "narration", "text": "The chair has"}, + {"id": "ch-6-p0001", "type": "dialogue", "text": "Hello."}, + ], + }, + ], + } + + +def _run_dir(root: Path, text: str) -> Path: + chapter = root / "ch-6" + chapter.mkdir(parents=True, exist_ok=True) + (chapter / "translation.json").write_text( + json.dumps({"paragraphs": {"ch-6-p0000": text, "ch-6-p0001": text}}), + encoding="utf-8", + ) + return root + + +class RenderTypographyTests(unittest.TestCase): + """The renderer's typography is the target language's.""" + + def test_the_asciidoc_language_is_the_target_language(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "La silla estaba sucia.") + asciidoc = build_asciidoc(_book("es-ES", "Cazador de Ofertas"), run, target=SPANISH) + self.assertIn(":lang: es-ES", asciidoc) + self.assertNotIn("zh-CN", asciidoc) + + def test_the_stylesheet_carries_the_target_language_codes(self) -> None: + styles = build_styles(SPANISH) + self.assertIn('fo:language="es"', styles) + self.assertIn('fo:country="ES"', styles) + self.assertIn('style:font-name="Noto Serif"', styles) + self.assertNotIn('fo:language="zh"', styles) + + def test_an_rtl_target_gets_a_right_to_left_direction(self) -> None: + self.assertIn('fo:direction="rtl"', build_styles(ARABIC)) + self.assertNotIn('fo:direction="rtl"', build_styles(SPANISH)) + + def test_the_chapter_heading_follows_the_target_convention(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "La silla estaba sucia.") + spanish = build_asciidoc(_book("es-ES"), run, target=SPANISH) + chinese = build_asciidoc(_book(), run, target=CHINESE) + self.assertIn("== Capítulo 1 Salvage", spanish) + self.assertIn("== 第一章　Salvage", chinese) + + def test_a_latin_target_does_not_mark_its_text_as_english(self) -> None: + """The Latin-run span exists so Chinese line-breaking leaves names alone. + + For a Spanish paragraph the whole text is Latin, and marking it as + English would tell the renderer that the paragraph is English. + """ + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "La silla estaba sucia.") + spanish = build_fodt(_book("es-ES"), run, target=SPANISH) + chinese = build_fodt(_book(), run, target=CHINESE) + self.assertNotIn('text:style-name="Latin"', spanish) + self.assertIn('text:style-name="Latin"', chinese) + + def test_the_fixed_strings_are_the_target_language_s(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "La silla estaba sucia.") + spanish = build_asciidoc(_book("es-ES"), run, target=SPANISH, sample=True) + chinese = build_asciidoc(_book(), run, target=CHINESE, sample=True) + self.assertIn("Muestra de revisión", spanish) + self.assertIn("Traducido por Umbra Tools", spanish) + self.assertNotIn("审阅样本", spanish) + self.assertIn("审阅样本", chinese) + + def test_the_contents_heading_is_the_target_language_s(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "La silla estaba sucia.") + spanish = build_fodt(_book("es-ES"), run, target=SPANISH) + self.assertIn(">Índice", spanish) + self.assertNotIn(">目录", spanish) + + def test_the_running_header_carries_the_target_subtitle(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "La silla estaba sucia.") + spanish = build_fodt(_book("es-ES", "Cazador de Ofertas"), run, target=SPANISH) + self.assertIn("Deal Hunter · Cazador de Ofertas", spanish) + + def test_the_old_book_format_still_renders(self) -> None: + """A book file written before the subtitle field existed must still work. + + The old format stored the subtitle in the book-level ``title_zh``. + """ + book = _book() + del book["subtitle_target"] + book["title_zh"] = "交易猎人" + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "中间的椅子旁。") + asciidoc = build_asciidoc(book, run, target=CHINESE) + fodt = build_fodt(book, run, target=CHINESE) + self.assertIn("= Deal Hunter: 交易猎人", asciidoc) + self.assertIn("交易猎人", fodt) + + def test_a_chapter_with_no_target_title_falls_back_to_the_english_one(self) -> None: + """A missing translation is visible; borrowing another language's is not.""" + book = _book("es-ES") + del book["chapters"][0]["title_target"] + with tempfile.TemporaryDirectory() as tmp: + run = _run_dir(Path(tmp), "La silla estaba sucia.") + asciidoc = build_asciidoc(book, run, target=SPANISH) + self.assertIn("Salvage", asciidoc) + self.assertNotIn("打捞", asciidoc) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/translation/tests/test_translation.py b/tools/translation/tests/test_translation.py index 6c0d765..13cf014 100644 --- a/tools/translation/tests/test_translation.py +++ b/tools/translation/tests/test_translation.py @@ -10,17 +10,24 @@ residue detection, JSON extraction and the mechanical QA checks. from __future__ import annotations +import json import sys +import tempfile import unittest from pathlib import Path HERE = Path(__file__).resolve().parent sys.path.insert(0, str(HERE.parent)) -from llm import parse_json # noqa: E402 -from pipeline import latin_residue, sanitize # noqa: E402 -from prompts import critic_prompt, draft_prompt # noqa: E402 -from qa import check_chapter # noqa: E402 +from book import SUBTITLES, _title_for # noqa: E402 +from languages import SCRIPT_LATIN, TargetLanguage, by_code, resolve # noqa: E402 +from llm import LLMError, parse_json # noqa: E402 +from pipeline import Runner, latin_residue, sanitize # noqa: E402 +from prompts import critic_prompt, draft_prompt, glossary_block, system_critic, system_drafter # noqa: E402 +from qa import check_chapter, forbidden_titles # noqa: E402 + +CHINESE = resolve("zh-Hans") +SPANISH = resolve("es-ES") class PromptInstructionTests(unittest.TestCase): @@ -34,24 +41,26 @@ class PromptInstructionTests(unittest.TestCase): ``--verify`` stage. """ - def _draft(self) -> str: + def _draft(self, target: TargetLanguage | None = None) -> str: return draft_prompt( chapter_title="Salvage", scene_index=0, chunk_index=0, chunks_in_scene=1, paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}], - analysis={"summary_zh": "摘要"}, + analysis={"summary": "摘要"}, previous_summary="", glossary={}, + target=target or CHINESE, ) - def _critic(self) -> str: + def _critic(self, target: TargetLanguage | None = None) -> str: return critic_prompt( chapter_title="Salvage", paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}], draft={"ch-6-p0000": "中间的椅子旁"}, glossary={}, + target=target or CHINESE, ) def test_the_drafter_is_told_to_keep_each_id_to_its_own_english(self) -> None: @@ -73,6 +82,249 @@ class PromptInstructionTests(unittest.TestCase): self.assertIn("段落错位", prompt) +class TargetLanguageTests(unittest.TestCase): + """The pipeline must produce the language it was told to produce. + + Measured on oldbox: the production directory ``deal-hunter-es`` declared + ``target_language: es-ES`` and held 3,038 paragraphs of Chinese, because + the drafter, the critic and the glossary lookup were all hard-coded to + Simplified Chinese. These tests pin the parameterisation that fixes it. + """ + + def test_a_language_is_found_by_tag_base_or_name(self) -> None: + self.assertEqual(by_code("es-ES").base, "es") + self.assertEqual(by_code("es").base, "es") + self.assertEqual(by_code("es_ES").base, "es") + self.assertEqual(by_code("Spanish").base, "es") + self.assertEqual(by_code("简体中文").base, "zh") + + def test_an_unknown_language_falls_back_loudly(self) -> None: + # The fallback prints a warning; the point is that it does not + # silently pretend the language was recognised. + self.assertEqual(resolve("xx-YY").base, "zh") + + def test_the_drafter_is_told_the_target_language(self) -> None: + prompt = draft_prompt( + chapter_title="Salvage", + scene_index=0, + chunk_index=0, + chunks_in_scene=1, + paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}], + analysis={"summary": "摘要"}, + previous_summary="", + glossary={}, + target=SPANISH, + ) + self.assertIn("西班牙语", prompt) + self.assertIn("español", prompt) + self.assertNotIn("简体中文", prompt) + + def test_the_drafter_asks_for_the_target_language_key(self) -> None: + prompt = draft_prompt( + chapter_title="Salvage", + scene_index=0, + chunk_index=0, + chunks_in_scene=1, + paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}], + analysis={"summary": "摘要"}, + previous_summary="", + glossary={}, + target=SPANISH, + ) + self.assertIn('"es":', prompt) + self.assertNotIn('"zh":', prompt) + + def test_the_critic_is_told_the_target_language(self) -> None: + prompt = critic_prompt( + chapter_title="Salvage", + paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}], + draft={"ch-6-p0000": "la silla"}, + glossary={}, + target=SPANISH, + ) + self.assertIn("西班牙语", prompt) + self.assertNotIn("中文译文", prompt) + + def test_the_glossary_reads_the_target_language_key(self) -> None: + glossary = { + "title": {"en": "Deal Hunter"}, + "characters": {"Kainda": {"zh": "凯因达", "es": "Kainda"}}, + "terms": {"bounty": {"zh": "赏金", "es": "recompensa"}}, + } + spanish = glossary_block(glossary, set(), "bounty", SPANISH) + self.assertIn("recompensa", spanish) + self.assertNotIn("赏金", spanish) + + def test_a_glossary_value_for_another_language_is_not_borrowed(self) -> None: + """A Chinese-only glossary must not leak Chinese into a Spanish run.""" + glossary = { + "characters": {"Kainda": {"zh": "凯因达"}}, + "terms": {}, + } + spanish = glossary_block(glossary, set(), "", SPANISH) + self.assertNotIn("凯因达", spanish) + + def test_the_chinese_rules_are_kept_for_chinese(self) -> None: + """The existing production must not lose its hard rules.""" + glossary = {"characters": {}, "terms": {}, "places": {}} + chinese = glossary_block(glossary, set(), "jump", CHINESE) + self.assertIn("恒星系 system / star system = 星系", chinese) + self.assertIn("不译成中文", chinese) + + def test_every_target_gets_the_books_own_rules(self) -> None: + """The rules about this book are not Chinese's and not Spanish's. + + *Deal Hunter* is the ship's name, the AI's name and the book's title, + so it stays in English in every edition. The Chinese table carries + its own wording of this rule, so a language with no table must not + lose it. + """ + glossary = {"characters": {}, "terms": {}, "places": {}} + for target in (CHINESE, SPANISH): + block = glossary_block(glossary, set(), "", target) + self.assertIn("Deal Hunter", block) + self.assertIn("保留英文", block) + self.assertIn("Hunter 作为 AI 的简称保留英文", block) + self.assertIn("Umbra Audio 保留英文", block) + + def test_a_language_without_a_table_gets_no_other_languages_terms(self) -> None: + """A Spanish run must not inherit the Chinese hard vocabulary.""" + glossary = {"characters": {}, "terms": {}, "places": {}} + spanish = glossary_block(glossary, set(), "jump", SPANISH) + self.assertNotIn("星系", spanish) + self.assertNotIn("跃迁", spanish) + + def test_a_glossary_can_supply_its_own_rules(self) -> None: + glossary = { + "characters": {}, + "terms": {}, + "places": {}, + "rules": ["- Deal Hunter se conserva en inglés"], + } + spanish = glossary_block(glossary, set(), "", SPANISH) + self.assertIn("Deal Hunter se conserva en inglés", spanish) + + def test_the_system_prompts_name_the_target_language(self) -> None: + self.assertIn("简体中文", system_drafter(CHINESE)) + self.assertIn("西班牙语", system_drafter(SPANISH)) + self.assertIn("Spanish", system_critic(SPANISH)) + self.assertNotIn("中文译文", system_critic(SPANISH)) + + def test_quotes_are_normalised_to_the_target_language(self) -> None: + self.assertEqual(sanitize('"hola"', SPANISH), "“hola”") + self.assertEqual(sanitize('"hallo"', resolve("de-DE")), "„hallo“") + self.assertEqual(sanitize('"bonjour"', resolve("fr-FR")), "«bonjour»") + + def test_the_ellipsis_is_the_target_language_s(self) -> None: + self.assertEqual(sanitize("...", CHINESE), "……") + self.assertEqual(sanitize("...", SPANISH), "…") + + def test_the_residue_check_is_only_meaningful_for_a_non_latin_target(self) -> None: + """A Spanish draft is full of Latin words; they are not residue.""" + self.assertIn("silla", latin_residue("la silla del capitán")[0]) + self.assertEqual(latin_residue("中间那把椅子")[0], []) + # So the repair is skipped for a Latin-script target: the check would + # otherwise flag the whole Spanish draft and "repair" it into + # nonsense. + self.assertIn(SCRIPT_LATIN, SPANISH.scripts) + self.assertNotIn(SCRIPT_LATIN, CHINESE.scripts) + + +class RunDirectoryGuardTests(unittest.TestCase): + """A run directory belongs to one language. + + Measured on oldbox: the production directory ``deal-hunter-es`` declared + ``target_language: es-ES`` and held 3,038 paragraphs of Chinese, because + the pipeline caches by file existence and the run directory is chosen by + the caller. A Spanish run pointed at that directory would have skipped + drafting entirely and published the Chinese under a Spanish name. + + The guard refuses that. It checks the manifest first (a claim) and then + measures the text that is actually there (the evidence). + """ + + def _runner(self, run_dir: Path, target: str, **kwargs: object) -> Runner: + book = run_dir / "book.json" + glossary = run_dir / "glossary.json" + book.write_text( + json.dumps({"target_language": target, "chapters": []}), + encoding="utf-8", + ) + glossary.write_text(json.dumps({}), encoding="utf-8") + return Runner( + book_path=book, + glossary_path=glossary, + run_dir=run_dir / "runs", + target_language=target, + **kwargs, + ) + + def test_a_run_directory_records_its_language(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self._runner(root, "es-ES") + manifest = json.loads((root / "runs" / "run.json").read_text(encoding="utf-8")) + self.assertEqual(manifest["target_language"], "es-ES") + self.assertEqual(manifest["glossary_key"], "es") + + def test_a_spanish_run_refuses_a_chinese_run_directory(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self._runner(root, "zh-Hans") + with self.assertRaises(SystemExit) as caught: + self._runner(root, "es-ES") + self.assertIn("zh-Hans", str(caught.exception)) + self.assertIn("es-ES", str(caught.exception)) + + def test_a_spanish_run_refuses_a_directory_holding_chinese_text(self) -> None: + """No manifest - an older directory - so the text itself is measured.""" + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self._runner(root, "zh-Hans") + chapter = root / "runs" / "ch-1" + chapter.mkdir(parents=True, exist_ok=True) + (chapter / "translation.json").write_text( + json.dumps({"paragraphs": {"ch-1-p0000": "凯因达走到门前。"}}), + encoding="utf-8", + ) + (root / "runs" / "run.json").unlink() + with self.assertRaises(SystemExit) as caught: + self._runner(root, "es-ES") + self.assertIn("not Spanish", str(caught.exception)) + + def test_a_spanish_run_accepts_a_spanish_run_directory(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self._runner(root, "es-ES") + chapter = root / "runs" / "ch-1" + chapter.mkdir(parents=True, exist_ok=True) + (chapter / "translation.json").write_text( + json.dumps({"paragraphs": {"ch-1-p0000": "Kainda se acercó a la puerta."}}), + encoding="utf-8", + ) + runner = self._runner(root, "es-ES") + self.assertEqual(runner.target.code, "es-ES") + + def test_force_overrides_the_manifest(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self._runner(root, "zh-Hans") + runner = self._runner(root, "es-ES", force=True) + self.assertEqual(runner.target.code, "es-ES") + + def test_a_draft_in_the_wrong_language_is_refused(self) -> None: + """The drafter has been known to ignore the target language.""" + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + runner = self._runner(root, "es-ES") + with self.assertRaises(LLMError) as caught: + runner._normalise_draft( + {"paragraphs": [{"id": "ch-1-p0000", "es": "凯因达走到门前。"}]}, + ["ch-1-p0000"], + ) + self.assertIn("wrong language", str(caught.exception)) + + class SanitizeTests(unittest.TestCase): def test_strips_dialogue_label(self) -> None: self.assertEqual(sanitize("（对话；凯因达）“我在游艇上。”"), "“我在游艇上。”") @@ -184,5 +436,51 @@ class QaTests(unittest.TestCase): self.assertNotIn("system_mistranslated", {i["kind"] for i in issues}) +class ChapterTitleTests(unittest.TestCase): + """The chapter title is the target language's, or nothing else's.""" + + def test_a_language_with_no_table_falls_back_to_the_english_title(self) -> None: + """Never to another language's title.""" + self.assertEqual(_title_for("Salvage", SPANISH), "Salvage") + self.assertEqual(_title_for("Salvage", resolve("ja-JP")), "Salvage") + + def test_a_language_with_a_table_uses_it(self) -> None: + self.assertEqual(_title_for("Salvage", CHINESE), "打捞") + + def test_a_chapter_missing_from_the_table_is_marked_in_the_target_language(self) -> None: + """A missing heading is a visible gap, not an English title that looks done.""" + chinese = _title_for("Brand New Chapter", CHINESE) + self.assertIn("待译", chinese) + self.assertIn("Brand New Chapter", chinese) + + def test_the_subtitle_is_the_target_language_s(self) -> None: + self.assertEqual(SUBTITLES["zh"], "交易猎人") + self.assertEqual(SUBTITLES["es"], "Cazador de Ofertas") + # A language with no subtitle gets none, not another language's. + self.assertNotIn("en", SUBTITLES) + + +class ForbiddenTitleTests(unittest.TestCase): + """The translated title must not appear in the text, in any language.""" + + def test_the_chinese_form_is_forbidden(self) -> None: + self.assertIn("《交易猎人》", forbidden_titles(CHINESE)) + + def test_the_spanish_form_is_forbidden(self) -> None: + self.assertIn("«Cazador de Ofertas»", forbidden_titles(SPANISH)) + + def test_a_language_without_a_subtitle_forbids_nothing(self) -> None: + """A language with no translated title has no forbidden form.""" + language = TargetLanguage( + code="xx-XX", + base="xx", + name="Example", + native="example", + chinese_name="示例语", + scripts=(SCRIPT_LATIN,), + ) + self.assertEqual(forbidden_titles(language), []) + + if __name__ == "__main__": unittest.main() diff --git a/umbratrans_web/project_status.py b/umbratrans_web/project_status.py index e58766c..3934e8d 100644 --- a/umbratrans_web/project_status.py +++ b/umbratrans_web/project_status.py @@ -11,11 +11,20 @@ For each program, the adapter returns every configured target language plus the English source row. Languages without a production directory are reported as ``not_started``; languages with a production directory report real chapter progress from the runs directory. + +The chapter count is a claim about the *files*, not about the *text*. A +directory full of Chinese can sit in a directory named for Spanish and be +reported as a complete Spanish translation - measured on oldbox, that is +exactly what ``deal-hunter-es`` was. So the text is measured too, and a +production whose text is not in the language it claims is reported as +``mismatched`` rather than ``complete``. """ from __future__ import annotations +import importlib.util import json +import sys from dataclasses import dataclass from datetime import datetime from pathlib import Path @@ -75,6 +84,60 @@ def _max_mtime(paths: list[Path]) -> str | None: return datetime.fromtimestamp(max(mtimes)).isoformat() +# The translation language registry, loaded once. It lives in +# ``tools/translation/languages.py``, which is not a distributed package, so +# it is loaded from the repository by path. It is the same file the pipeline +# uses, so the dashboard cannot disagree with it about which script a +# language is written in. +_LANGUAGE_REGISTRY: Any | None = None +_LANGUAGE_REGISTRY_TRIED = False + + +def _language_registry() -> Any | None: + """The translation language registry, or None if it cannot be loaded.""" + global _LANGUAGE_REGISTRY, _LANGUAGE_REGISTRY_TRIED + if _LANGUAGE_REGISTRY_TRIED: + return _LANGUAGE_REGISTRY + _LANGUAGE_REGISTRY_TRIED = True + path = Path(__file__).resolve().parents[1] / "tools" / "translation" / "languages.py" + try: + if path.is_file(): + spec = importlib.util.spec_from_file_location("umbratrans_languages", path) + if spec and spec.loader: + module = importlib.util.module_from_spec(spec) + # The module has to be in sys.modules before it is executed: + # @dataclass resolves its own module through sys.modules. + sys.modules[spec.name] = module + spec.loader.exec_module(module) + _LANGUAGE_REGISTRY = module + return module + from tools.translation import languages # type: ignore[import-not-found] + + _LANGUAGE_REGISTRY = languages + return languages + except Exception: + return None + + +def _language_problem(runs_dir: Path | None, tag: str) -> str | None: + """Why the text in a production is not in the language it claims. + + Returns None when the text is in the declared language. When the check + cannot be performed at all it says so, rather than passing silently: a + production that cannot be verified is not one that can be reported + complete. + """ + if runs_dir is None or not runs_dir.is_dir(): + return None + registry = _language_registry() + if registry is None: + return "the language registry could not be loaded, so the text could not be checked" + try: + return registry.check_runs_language(runs_dir, tag) + except Exception as exc: # pragma: no cover - defensive + return f"the language could not be checked: {exc}" + + def _count_translated_chapters(runs_dir: Path) -> tuple[int, int]: """Return (done, total) chapter counts from the runs directory.""" done = 0 @@ -119,8 +182,6 @@ class ProjectStatusAdapter: prod_name = _SLUG_TO_PROD.get(getattr(project, "slug", "")) prod_dir = self.productions_root / prod_name if prod_name else None runs_dir = prod_dir / "runs" if prod_dir else None - out_progress = prod_dir / "output-progress" if prod_dir else None - out_final = prod_dir / "output" if prod_dir else None # Chapter totals come from the source book when available; fall back # to the number of chapter directories found in runs. @@ -135,7 +196,12 @@ class ProjectStatusAdapter: "language_label": "English (source)", "role": "source", "source_present": True, - "translation": {"state": "complete", "chapters_done": total, "chapters_total": total}, + "translation": { + "state": "complete", + "chapters_done": total, + "chapters_total": total, + "language_problem": None, + }, "audio": {"state": "unknown"}, "outputs": {}, "review": {"state": "ok"}, @@ -149,41 +215,41 @@ class ProjectStatusAdapter: continue # source row already added above label, role = target_langs[tag] - if prod_dir and prod_dir.is_dir(): - # A language-specific production directory would be named - # deal-hunter- (e.g. deal-hunter-es for Spanish). - lang_prod_name = f"deal-hunter-{tag.split('-')[0]}" - lang_prod_dir = self.productions_root / lang_prod_name - lang_runs = lang_prod_dir / "runs" if lang_prod_dir.is_dir() else None - lang_out_progress = lang_prod_dir / "output-progress" if lang_prod_dir.is_dir() else None - lang_out_final = lang_prod_dir / "output" if lang_prod_dir.is_dir() else None + # A language-specific production directory would be named + # deal-hunter- (e.g. deal-hunter-es for Spanish). It is + # looked up on its own: a Spanish production exists or it does + # not, regardless of whether the Chinese one is there. + lang_prod_name = f"deal-hunter-{tag.split('-')[0]}" + lang_prod_dir = self.productions_root / lang_prod_name + lang_runs = lang_prod_dir / "runs" if lang_prod_dir.is_dir() else None + lang_out_progress = lang_prod_dir / "output-progress" if lang_prod_dir.is_dir() else None + lang_out_final = lang_prod_dir / "output" if lang_prod_dir.is_dir() else None - if lang_runs and lang_runs.is_dir(): - lang_done, lang_total = _count_translated_chapters(lang_runs) - state = ( - "complete" - if lang_total and lang_done >= lang_total - else ("in_progress" if lang_done else "not_started") - ) - outputs = _read_outputs( - lang_out_progress or Path(), lang_out_final or Path() - ) - updated_at = _max_mtime( - [lang_runs / f"ch-{i}" / "translation.json" for i in range(1, 40)] - if lang_runs.is_dir() - else [] - ) - else: - state = "not_started" - lang_done, lang_total = 0, 0 - outputs = {} - updated_at = None + if lang_runs and lang_runs.is_dir(): + lang_done, lang_total = _count_translated_chapters(lang_runs) + state = ( + "complete" + if lang_total and lang_done >= lang_total + else ("in_progress" if lang_done else "not_started") + ) + outputs = _read_outputs(lang_out_progress or Path(), lang_out_final or Path()) + updated_at = _max_mtime( + [lang_runs / f"ch-{i}" / "translation.json" for i in range(1, 40)] if lang_runs.is_dir() else [] + ) else: state = "not_started" lang_done, lang_total = 0, 0 outputs = {} updated_at = None + # The chapter count says the files are there. The script says + # which language they are in. A production that declares one + # language and holds another is not complete in the language + # it declares. + language_problem = _language_problem(lang_runs, tag) + if language_problem: + state = "mismatched" + languages.append( { "language_tag": tag, @@ -194,12 +260,11 @@ class ProjectStatusAdapter: "state": state, "chapters_done": lang_done, "chapters_total": lang_total, + "language_problem": language_problem, }, "audio": {"state": "not_started"}, "outputs": outputs, - "review": { - "state": "needs_review" if state == "complete" else "pending" - }, + "review": {"state": "needs_review" if state in {"complete", "mismatched"} else "pending"}, "updated_at": updated_at, } ) diff --git a/umbratrans_web/tests/test_project_status.py b/umbratrans_web/tests/test_project_status.py new file mode 100644 index 0000000..ab84e96 --- /dev/null +++ b/umbratrans_web/tests/test_project_status.py @@ -0,0 +1,96 @@ +# Copyright 2026 Umbra Tools (umbratools.com) +# SPDX-License-Identifier: Apache-2.0 + +"""Tests for the project-status adapter. + +The adapter reports a language's state from the productions directory. The +chapter count is a claim about the *files*; the script check is a measurement +of the *text*. Measured on oldbox, ``deal-hunter-es`` held 3,038 paragraphs +of Chinese and was reported as a complete Spanish translation, so the +measurement is the part that matters. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from types import SimpleNamespace + +from umbratrans_web.project_status import ProjectStatusAdapter + +PROJECT = SimpleNamespace(slug="dealhunter-trans", id="11111111-1111-1111-1111-111111111111") + + +def _production(root: Path, tag: str, chapters: dict[str, str]) -> Path: + """Write a production directory with the given chapters of text.""" + prod = root / f"deal-hunter-{tag}" + for chapter_id, text in chapters.items(): + directory = prod / "runs" / chapter_id + directory.mkdir(parents=True, exist_ok=True) + (directory / "translation.json").write_text( + json.dumps({"paragraphs": {"ch-1-p0000": text}}, ensure_ascii=False), + encoding="utf-8", + ) + return prod + + +def _row(adapter: ProjectStatusAdapter, tag: str) -> dict: + status = adapter.status_for(PROJECT) + row = next(r for r in status["languages"] if r["language_tag"] == tag) + return row["translation"] + + +def test_a_production_in_its_own_language_is_complete(tmp_path: Path) -> None: + _production(tmp_path, "zh", {"ch-1": "凯因达走到门前。"}) + _production(tmp_path, "fr", {"ch-1": "Kainda s'approcha de la porte."}) + adapter = ProjectStatusAdapter(tmp_path) + + assert _row(adapter, "zh-Hans")["state"] == "complete" + assert _row(adapter, "fr-FR")["state"] == "complete" + assert _row(adapter, "fr-FR")["language_problem"] is None + + +def test_a_mislabelled_production_is_not_reported_complete(tmp_path: Path) -> None: + """The defect this check exists for: Chinese filed under a Spanish name.""" + _production(tmp_path, "es", {"ch-1": "凯因达走到门前。", "ch-2": "门是关着的。"}) + adapter = ProjectStatusAdapter(tmp_path) + + row = _row(adapter, "es-ES") + assert row["state"] == "mismatched" + assert row["chapters_done"] == 2 + assert "not Spanish" in row["language_problem"] + + +def test_a_production_with_no_runs_is_not_started(tmp_path: Path) -> None: + adapter = ProjectStatusAdapter(tmp_path) + row = _row(adapter, "de-DE") + assert row["state"] == "not_started" + assert row["language_problem"] is None + + +def test_a_mislabelled_production_needs_review(tmp_path: Path) -> None: + _production(tmp_path, "es", {"ch-1": "凯因达走到门前。"}) + adapter = ProjectStatusAdapter(tmp_path) + status = adapter.status_for(PROJECT) + row = next(r for r in status["languages"] if r["language_tag"] == "es-ES") + assert row["review"]["state"] == "needs_review" + + +def test_a_language_with_no_table_of_its_own_is_not_borrowed(tmp_path: Path) -> None: + """A Chinese production must not be accepted as Japanese. + + Japanese is written in Han characters *and* kana. A text with no kana at + all is Chinese, however many Han characters it has. + """ + _production(tmp_path, "ja", {"ch-1": "凯因达走到门前。"}) + adapter = ProjectStatusAdapter(tmp_path) + row = _row(adapter, "ja-JP") + assert row["state"] == "mismatched" + assert "kana" in row["language_problem"] + + +def test_every_configured_language_is_reported(tmp_path: Path) -> None: + adapter = ProjectStatusAdapter(tmp_path) + status = adapter.status_for(PROJECT) + tags = {r["language_tag"] for r in status["languages"]} + assert {"en", "zh-Hans", "es-ES", "fr-FR"} <= tags diff --git a/web/assets/styles.css b/web/assets/styles.css index b5c0c83..833e913 100644 --- a/web/assets/styles.css +++ b/web/assets/styles.css @@ -51,6 +51,7 @@ button:focus-visible, a:focus-visible, input:focus-visible, select:focus-visible .status-pill--info { color: #bfdbfe; background: #1e3a5f; } .status-pill--success { color: #a7f3d0; background: #065f46; } .status-pill--warning { color: #fde68a; background: #78350f; } +.status-pill--danger { color: #fecaca; background: #7f1d1d; } .status-pill--muted { color: #cbd5e1; background: #334155; } .status-dot { display: inline-block; width: .55rem; height: .55rem; border-radius: 50%; } .status-dot--good { background: #34d399; box-shadow: 0 0 0 .25rem rgba(52, 211, 153, .12); } diff --git a/web/assets/umbra-app.js b/web/assets/umbra-app.js index b956291..c8b8f86 100644 --- a/web/assets/umbra-app.js +++ b/web/assets/umbra-app.js @@ -292,6 +292,7 @@ if (state === 'complete') return { text: 'Text translated', tone: 'success' }; if (state === 'in_progress') return { text: 'In progress', tone: 'info' }; if (state === 'needs_review') return { text: 'Needs review', tone: 'warning' }; + if (state === 'mismatched') return { text: 'Wrong language', tone: 'danger' }; return { text: 'Not started', tone: 'muted' }; } @@ -458,14 +459,20 @@ chapters = 'Complete'; } + // A production whose text is not in the language it declares says why, + // rather than just looking unfinished. + var problem = tr.language_problem || ''; + var badgeNode = el('span', { class: 'status-pill status-pill--' + badge.tone, text: badge.text }); + if (problem) { + badgeNode.setAttribute('title', problem); + } + return el('div', { class: 'language-row' + (code === state.languageCode ? ' language-row--selected' : ''), role: 'button', tabIndex: '0' }, [ el('div', { class: 'language-row__label' }, [ el('strong', { text: label }), el('span', { class: 'language-row__role', text: isSource ? 'Source' : chapters }) ]), - el('div', { class: 'language-row__status' }, [ - el('span', { class: 'status-pill status-pill--' + badge.tone, text: badge.text }) - ]), + el('div', { class: 'language-row__status' }, [ badgeNode ]), el('div', { class: 'language-row__actions' }, languageActions(program, code, status)) ]); }