umbragit/UmbraTrans
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Parameterise the translation pipeline by target language

Browse Source 
The pipeline was hard-coded to Simplified Chinese in three places: the
drafter was told to translate into Chinese, the critic was told to compare
English with Chinese, and the glossary's translated forms were read from a
"zh" key. A production that declared target_language: es-ES was therefore
translated into Chinese and filed under a Spanish name - measured on oldbox,
deal-hunter-es held 3,038 paragraphs of Chinese and was reported complete.

A new registry, tools/translation/languages.py, is the single source of
truth for what a target language is: its names, its script, its punctuation,
its glossary key, its typography and its chapter-numbering convention. Ten
languages are registered. The prompt scaffolding stays in Chinese - that is
what the models read best - and the target is named with its Chinese, English
and native names so there is no room to misread it.

The registry now reaches every stage:

- prompts.py: the analyst, drafter, critic and glossary blocks take the
  target. The hard rules about this book - Deal Hunter is the ship's name,
  the AI's name and the book's title, so it stays in English - are the
  language's own table for Chinese and the global rules for everything else,
  so a language with no table does not lose them. The Chinese rules are
  byte-for-byte what they were.
- pipeline.py: --target-language, a run-directory guard that refuses to
  reuse a directory belonging to another language (from the manifest, or by
  measuring the text it already holds), a draft-keyed-by-target reader, and
  a refusal of a draft that came back in the wrong script.
- book.py: per-language chapter-title and subtitle tables. A language with no
  table falls back to the English title, never to another language's; a
  language with a table and a missing chapter gets a visible marker in its
  own language.
- qa.py: the target's quote marks, punctuation, script count and forbidden
  title forms. The Latin-residue check is skipped for a Latin target, where
  it would flag the whole book.
- align.py: the aligner and verifier prompts name the target, and the
  sentence and clause splitting uses the target's own terminators.
- render.py: the fonts, the language and country codes, the writing
  direction, the chapter headings and the fixed editorial strings around the
  translation are the target's. The Latin-run span is skipped for a Latin
  target, where marking the text as English would tell the renderer it is
  English.
- project_status.py: the dashboard measures a sample of a production's text
  and reports a production whose text is not in the language it declares as
  "mismatched", not "complete". A chapter count is a claim about the files;
  the script check is a measurement of the text.

Also fixes: the dashboard gated every language's production on the source
production existing, so a Spanish production could never be reported even
when it was there.

The Chinese render is byte-identical to before on the real book, and the
Chinese rules are unchanged. Suite: 620 passed, the same 68 pre-existing
environment failures, +49 new tests.
This commit is contained in:
B.D. Murphy
2026-11-07 05:53:17 -08:00
parent 54336535e3
commit 3f05489e25
15 changed files with 2691 additions and 418 deletions
Download Patch File Download Diff File Expand all files Collapse all files

156
tools/translation/README.md
View File

@@ -1,19 +1,57 @@
# Deal Hunter Chinese translation tools
# Deal Hunter translation tools
This directory contains the translation and rendering pipeline used to produce
the Simplified Chinese edition of *Deal Hunter*. It is deliberately separate
from the web application: the book text and the translations are copyrighted
and are **not** committed to this repository.
the translated editions of *Deal Hunter*. It is deliberately separate from the
web application: the book text and the translations are copyrighted and are
**not** committed to this repository.
The pipeline is language-parameterised. The target language is a single
setting that reaches every stage - the prompts, the pipeline plumbing, the
chapter titles, the renderer typography and the alignment prompts - so a
Spanish edition is a different configuration, not a different code path.
## The target-language registry
`languages.py` is the single source of truth for what a target language is:
| field | what it decides |
|---|---|
| `name`, `native`, `chinese_name` | how the language is named to the model and to the reader |
| `scripts`, `required_script`, `forbidden_scripts` | what the text must and must not be written in |
| `glossary_key` | which key a glossary uses for the translated form of a name or term |
| `quote_open/close`, `ellipsis`, `dash`, `paren_open/close`, `title_open/close` | the punctuation conventions |
| `sentence_terminators`, `closing_marks`, `clause_marks` | how the text is split into sentences and clauses |
| `font_serif`, `font_sans`, `epub_font_stack` | the typography |
| `asciidoc_lang`, `odf_language`, `odf_country`, `rtl` | the language tags and writing direction |
| `chapter_prefix`, `chapter_suffix`, `chapter_numeral` | the chapter-numbering convention |
Ten languages are registered: `zh-Hans`, `es-ES`, `fr-FR`, `de-DE`, `it-IT`,
`pt-BR`, `ru-RU`, `ja-JP`, `ko-KR`, `ar-SA`.
Two rules the registry enforces:
- **A language with no table of its own falls back to English, never to
another language.** A Spanish edition with Chinese chapter headings is the
same mislabelling defect as a Chinese draft under a Spanish name, one level
down. A language that *has* a table but is missing a chapter from it gets a
visible marker in its own language instead - an untranslated heading is a
gap the reviewer has to see, not an English title that looks finished.
- **The prompt scaffolding stays in Chinese.** The prompts are written in
Chinese because that is what the analyst, drafter and critic models read
best; the target language is named with its Chinese name, its English name
and its native name - `西班牙语（Spanish / español）` - so there is no room
to misread which language is meant.
## Where the data lives
```
productions/deal-hunter-zh/ # git-ignored
productions/deal-hunter-<lang>/ # git-ignored
source/book.json # extracted English source model
glossary/glossary.json # names, places, terms, style rules
runs/<chapter>/*.json # analyses, drafts, critiques, final text
runs/run.json # what this run directory is for
runs/usage.jsonl # per-call token/latency log
output/*.asciidoc, *.odt, *.pdf # review artefacts
output/*.asciidoc, *.epub, *.odt, *.pdf, *.fodt # review artefacts
```
The source bundle is the UmbraAudio controller bundle:
@@ -45,6 +83,23 @@ character voices, the scene analysis, and the previous chunk's summary. The
critic uses a different model and returns a severity-sorted issue list; it does
not rewrite the text.
### The run-directory guard
The pipeline caches by file existence and the run directory is chosen by the
caller, so pointing a Spanish run at a directory that already holds a Chinese
run would skip drafting entirely and publish the Chinese under a Spanish name.
The runner therefore refuses to reuse a directory that belongs to another
language, on two checks - one from the claim and one from the measurement:
- `runs/run.json` records what the directory is for; a mismatch is refused.
- A directory with no manifest (written before the guard existed) is
**measured**: the script of the text it already holds is compared against
the target language.
The same check runs on every draft the drafter returns, so a model that
answers in the wrong language is refused at the draft rather than at the
ingest.
## Pass 5 - aligning back to the utterances
Passes 0-4 work on *paragraphs*, because epub and pdf are paragraph media.
@@ -98,12 +153,30 @@ python scripts/import_translations.py --project "DealHunterShip_4.0e_whole-to-au
tell which clause renders which English sentence, so everything it touches is
reported as needing review.
### Verifying the paragraph boundaries
`--verify` catches the failure mode that the alignment pass cannot see by
construction: the drafter re-flowing a passage across paragraph ids. A
one-to-one paragraph is a pure copy, so there is nothing to divide and nothing
to disagree - the drift is invisible to the aligner and visible only to a
check that reads the paragraph against its neighbours.
The verdict is one of three: `aligned`, `incomplete` or `drift`. Only `drift`
deletes a line, and a dropped paragraph still gets a row - `voiced=false` with
the reason - so the gap says why rather than looking like a paragraph nobody
translated. The paragraphs are judged as a **window**, because the start of a
drift is not knowable from any single paragraph.
## Commands
Every command takes `--target-language`. It defaults to the book's own
`target_language`, and falls back to `zh-Hans`.
Extract the source:
```bash
python3 tools/translation/book.py \
--target-language zh-Hans \
--out productions/deal-hunter-zh/source/book.json
```
@@ -111,6 +184,7 @@ Translate a chapter:
```bash
python3 tools/translation/pipeline.py \
--target-language zh-Hans \
--book productions/deal-hunter-zh/source/book.json \
--glossary productions/deal-hunter-zh/glossary/glossary.json \
--run-dir productions/deal-hunter-zh/runs \
@@ -121,10 +195,22 @@ Selectors: `all`, `front`, `body`, `back`, or a comma-separated list of
chapter indices / ids / titles (`-4,-3,-2,-1` must be quoted or passed as
`--chapters=-4,-3`).
Align:
```bash
python3 tools/translation/align.py \
--target-language zh-Hans \
--book productions/deal-hunter-zh/source/book.json \
--run-dir productions/deal-hunter-zh/runs \
--out productions/deal-hunter-zh/aligned/delivery.json \
--chapters all
```
Render:
```bash
python3 tools/translation/render.py \
--target-language zh-Hans \
--book productions/deal-hunter-zh/source/book.json \
--run-dir productions/deal-hunter-zh/runs \
--out-dir productions/deal-hunter-zh/output \
@@ -135,6 +221,7 @@ Mechanical QA:
```bash
python3 tools/translation/qa.py \
--target-language zh-Hans \
--book productions/deal-hunter-zh/source/book.json \
--run-dir productions/deal-hunter-zh/runs \
--json-out productions/deal-hunter-zh/runs/qa.json \
@@ -165,9 +252,58 @@ ODF document (`.fodt`) so that page layout, styles and the table of contents are
under our control. LibreOffice converts it and updates the TOC through UNO; a
plain `--convert-to` leaves the index empty. A PDF is exported for quick review.
Front matter, body and back matter use different styles. Chapter headings use
Chinese numerals (`第一章　打捞`). Body text is 11 pt Noto Serif CJK SC with
1.5 line spacing and a two-character first-line indent.
Front matter, body and back matter use different styles. The typography is the
target language's:
- **Chapter headings** follow the language's own convention - Chinese writes
`第一章　打捞` with Chinese numerals, Spanish writes `Capítulo 1 Salvage`,
Japanese writes `第1章　…` with Arabic numerals, Korean writes `제1장　…`.
- **Fonts and language tags** are the target's: the stylesheet declares
`fo:language` and `fo:country` for the target, so LibreOffice hyphenates and
shapes the text with the right rules, and an RTL target gets
`fo:direction="rtl"`.
- **The fixed editorial strings** the renderer writes around the translation -
the contents heading, the review-sample notice, the translator credit, the
title-page lines, the marker for an untranslated heading - are in the target
language too. They are part of the book the reader receives.
- **Latin runs inside a CJK text** are marked as English so Chinese
line-breaking does not split an English name in the middle of a word. For a
Latin-script target the whole text is Latin, so the marking is skipped -
marking a Spanish paragraph as English would tell the renderer it is English.
The book's title stays in English for every language: *Deal Hunter* is also the
ship's name and the AI's name, so translating it would rename a character. The
subtitle is translated.
## The script check
The registry can tell what script a text is written in, and that is enough to
catch the mistake that actually happens: a pipeline configured for one
language that silently produces another.
`script_mismatch(code, text)` returns a reason, or `None` when the text could
be that language. It is deliberately about the **script**, not the language: it
cannot tell Spanish from Portuguese, but it can tell Chinese from Spanish.
Japanese without kana is Chinese; Korean without Hangul is Chinese; Chinese
carrying kana is a Japanese translation in the wrong lane.
The check runs at three boundaries:
- **the draft** - the pipeline refuses a drafter answer that is in the wrong
script, rather than caching it;
- **the run directory** - the guard above;
- **the dashboard** - `umbratrans_web/project_status.py` measures a sample of
the production's translated text and reports a production whose text is not
in the language it declares as `mismatched`, not `complete`.
The dashboard check exists because of a measured defect: the `deal-hunter-es`
production on oldbox declared `target_language: es-ES`, had a Spanish glossary
with Spanish style rules, and a complete 32-chapter run - and all 3,038
translated paragraphs were Chinese. The pipeline's drafter, critic and analyst
instructions were hard-coded to Simplified Chinese, so the model followed the
instruction rather than the glossary, and the critic checked Chinese text for
Chinese correctness and found nothing wrong. A chapter count is a claim about
the *files*; the script check is a measurement of the *text*.
## What this is not
@@ -178,3 +314,5 @@ Chinese numerals (`第一章　打捞`). Body text is 11 pt Noto Serif CJK SC wi
- It does not produce the audio. The alignment pass hands UmbraAudio one row
per utterance; voicing, loudness and the audiobook masters are UmbraAudio's
job.
- It does not translate a language it has no glossary for. A glossary supplies
the names, the terms and the style rules; each language needs its own.

128
tools/translation/align.py
View File

@@ -72,6 +72,7 @@ from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from languages import TargetLanguage, by_code, resolve
from llm import LLM, LLMError
# The aligner compares English source against translated text, which is the
@@ -80,15 +81,23 @@ from llm import LLM, LLMError
# its sentence boundaries best, so it is the default.
DEFAULT_ALIGN_MODEL = "qwen/qwen3.5-9b"
SYSTEM_ALIGNER = """你是一位双语对齐编辑，负责把已经译好的中文段落还原到英文原文的每一条话语上。
你的任务不是翻译，也不是改写，而是判断：中文译文里的哪一部分对应英文原文里的哪一条话语。
译文为了中文的自然可能合并或拆分了句子，所以不能按句子序号机械对应，要按意思对应。
CHINESE = resolve("zh-Hans")
def system_aligner(target: TargetLanguage) -> str:
"""The aligner's instructions, in the target language's terms.
The task is the same in every language - map a translated paragraph back
onto the source utterances it was built from - but the sentence merging
the translation does is the *target's* merging, so the prompt has to name
the target.
"""
return f"""你是一位双语对齐编辑，负责把已经译好的{target.chinese_name}段落还原到英文原文的每一条话语上。
你的任务不是翻译，也不是改写，而是判断：{target.chinese_name}译文里的哪一部分对应英文原文里的哪一条话语。
译文为了{target.chinese_name}的自然可能合并或拆分了句子，所以不能按句子序号机械对应，要按意思对应。
每一条英文话语都必须恰好出现一次；被合并进别条话语的，用 merged_into 指明并把 target_text 留空。
只输出 JSON，不要解释，不要 Markdown 代码围栏。"""
_SENTENCE_TERMINATORS = "。！？!?；;…"
_CLOSING_MARKS = "”’」』）)】》\"'"
def normalise_target_language(value: str) -> str:
"""``zh-CN`` -> ``zh``.
@@ -96,7 +105,14 @@ def normalise_target_language(value: str) -> str:
UmbraAudio stores the dropdown code, not the locale, so the payload has
to carry the base code or the delivery lands in a language the review UI
does not offer.
A registered language answers from the registry, which is the single
source of truth for what a language's base code is; an unregistered one
falls back to the string surgery.
"""
language = by_code(value)
if language is not None:
return language.base
return value.strip().split("-")[0].split("_")[0].lower()
@@ -121,12 +137,19 @@ def _compact(text: str) -> tuple[str, list[int]]:
return "".join(chars), positions
def split_target_sentences(text: str) -> list[str]:
def split_target_sentences(
text: str,
target: TargetLanguage | None = None,
) -> list[str]:
"""Split translated text into sentences, keeping the terminator.
Chinese terminates with a full stop, and a closing quote may follow it,
so the closing marks are pulled into the sentence they close.
The terminators and the closing marks are the target's own: Chinese
terminates with a full stop and a closing quote may follow it, while a
Spanish sentence ends with a question mark or a full stop.
"""
target = target or CHINESE
terminators = target.sentence_terminators
closing = target.closing_marks
sentences: list[str] = []
buffer: list[str] = []
index = 0
@@ -134,8 +157,8 @@ def split_target_sentences(text: str) -> list[str]:
ch = text[index]
buffer.append(ch)
index += 1
if ch in _SENTENCE_TERMINATORS:
while index < len(text) and text[index] in _CLOSING_MARKS:
if ch in terminators:
while index < len(text) and text[index] in closing:
buffer.append(text[index])
index += 1
piece = "".join(buffer).strip()
@@ -148,15 +171,20 @@ def split_target_sentences(text: str) -> list[str]:
return sentences
def _split_at_comma(text: str) -> list[str] | None:
def _split_at_comma(
text: str,
target: TargetLanguage | None = None,
) -> list[str] | None:
"""Split one sentence in two at its last clause boundary.
An intent-based translation merges English sentences with a comma, a
colon or a dash rather than with a full stop, because that is what reads
naturally in Chinese. So when the target has fewer sentences than the
source has utterances, the boundary is a clause boundary.
naturally in the target language. So when the target has fewer sentences
than the source has utterances, the boundary is a clause boundary - and
which characters those are depends on the language.
"""
for mark in ("——", "—", "：", "，", "、", "；", ",", ";"):
target = target or CHINESE
for mark in target.clause_marks:
positions = [m.start() for m in re.finditer(re.escape(mark), text)]
for position in reversed(positions):
if position <= 0:
@@ -172,7 +200,11 @@ def _split_at_comma(text: str) -> list[str] | None:
return None
def mechanical_split(text: str, count: int) -> list[str] | None:
def mechanical_split(
text: str,
count: int,
target: TargetLanguage | None = None,
) -> list[str] | None:
"""Divide *text* into exactly *count* pieces, or return None.
This is the fallback when the model cannot be trusted. It is not good
@@ -185,7 +217,7 @@ def mechanical_split(text: str, count: int) -> list[str] | None:
if count == 1:
return [text.strip()] if text.strip() else None
pieces = split_target_sentences(text)
pieces = split_target_sentences(text, target)
if not pieces:
return None
@@ -210,7 +242,7 @@ def mechanical_split(text: str, count: int) -> list[str] | None:
target_index = None
replacement = None
for index in ordered:
candidate = _split_at_comma(pieces[index])
candidate = _split_at_comma(pieces[index], target)
if candidate is not None:
target_index = index
replacement = candidate
@@ -351,8 +383,14 @@ def source_utterances(paragraph: dict, bundle: dict[str, dict]) -> list[SourceUt
return out
def alignment_prompt(paragraph_id: str, sources: list[SourceUtterance], translation: str) -> str:
def alignment_prompt(
paragraph_id: str,
sources: list[SourceUtterance],
translation: str,
target: TargetLanguage | None = None,
) -> str:
"""The prompt that asks for the partition of a translated paragraph."""
target = target or CHINESE
lines = []
for source in sources:
lines.append(f"[{source.source_id}] (说话者：{source.speaker}) {source.text}")
@@ -361,20 +399,20 @@ def alignment_prompt(paragraph_id: str, sources: list[SourceUtterance], translat
英文原文（每一条都是一条已经确认说话者的话语）：
{chr(10).join(lines)}
中文译文：
{target.chinese_name}译文：
{translation}
请把中文译文分配回上面每一条英文话语。输出 JSON：
请把{target.chinese_name}译文分配回上面每一条英文话语。输出 JSON：
{{
"segments": [
{{"source_id": "utt-xxxxx", "target_text": "对应的中文片段"}},
{{"source_id": "utt-xxxxx", "target_text": "对应的{target.chinese_name}片段"}},
{{"source_id": "utt-yyyyy", "target_text": "", "merged_into": "utt-xxxxx"}}
]
}}
要求：
- segments 必须为原文的每一条话语各给一项，source_id 一一对应，不多不少。
- target_text 必须是中文译文里**原样出现**的连续片段，按原文顺序拼接起来必须等于整段译文，不得删改一个字。
- target_text 必须是{target.chinese_name}译文里**原样出现**的连续片段，按原文顺序拼接起来必须等于整段译文，不得删改一个字。
- 如果译文把两条原文合并成了一句，被合并的那条 target_text 留空，用 merged_into 指明它并入了哪一条。
- merged_into 只能指向同一条段落里、同一说话者、且有 target_text 的话语。
- 不要翻译，不要润色，不要补充，只做对应。"""
@@ -486,8 +524,10 @@ def align_paragraph(
bundle: dict[str, dict],
llm: LLM | None = None,
model: str = DEFAULT_ALIGN_MODEL,
target: TargetLanguage | None = None,
) -> ParagraphAlignment:
"""Divide one translated paragraph between its source utterances."""
target = target or CHINESE
paragraph_id = str(paragraph.get("id") or "")
sources = source_utterances(paragraph, bundle)
@@ -511,8 +551,8 @@ def align_paragraph(
if llm is not None:
try:
data, _usage = llm.chat_json(
SYSTEM_ALIGNER,
alignment_prompt(paragraph_id, sources, translation),
system_aligner(target),
alignment_prompt(paragraph_id, sources, translation, target),
model=model,
temperature=0.0,
cache_key=f"align-{paragraph_id}",
@@ -524,7 +564,7 @@ def align_paragraph(
problem = f"alignment model failed: {exc}"
else:
problem = "; ".join(problems)
mechanical = mechanical_split(translation, len(sources))
mechanical = mechanical_split(translation, len(sources), target)
if mechanical is None:
return ParagraphAlignment(paragraph_id=paragraph_id, method="failed", problems=[problem])
return ParagraphAlignment(
@@ -536,7 +576,7 @@ def align_paragraph(
problems=[problem],
)
mechanical = mechanical_split(translation, len(sources))
mechanical = mechanical_split(translation, len(sources), target)
if mechanical is None:
return ParagraphAlignment(
paragraph_id=paragraph_id,
@@ -594,6 +634,7 @@ def verification_prompt(
translation: str,
previous_text: str = "",
next_text: str = "",
target: TargetLanguage | None = None,
) -> str:
"""Ask which paragraph *translation* actually renders.
@@ -602,15 +643,16 @@ def verification_prompt(
a drifted paragraph's translation is a perfectly good translation - of
the paragraph next door.
"""
target = target or CHINESE
previous_block = f"\n【前一段原文】\n{previous_text}\n" if previous_text else ""
next_block = f"\n【后一段原文】\n{next_text}\n" if next_text else ""
return f"""【本段原文】
{source_text}
{previous_block}{next_block}
【本段中文译文】
【本段{target.chinese_name}译文】
{translation}
请判断这段中文译文到底对应哪一段原文，三选一：
请判断这段{target.chinese_name}译文到底对应哪一段原文，三选一：
- "aligned"：译文对应本段原文，哪怕不完美。措辞不同、近义词、漏掉一个副词、
数字或单复数的小差异，都算 aligned。
@@ -642,6 +684,7 @@ def verify_translation(
model: str = DEFAULT_ALIGN_MODEL,
previous_text: str = "",
next_text: str = "",
target: TargetLanguage | None = None,
) -> tuple[str, str]:
"""Which paragraph does *translation* actually render?
@@ -649,7 +692,7 @@ def verify_translation(
A model failure is reported as ``drift``: the delivery must not carry a
line nobody checked.
"""
prompt = verification_prompt(source_text, translation, previous_text, next_text)
prompt = verification_prompt(source_text, translation, previous_text, next_text, target)
try:
data, _usage = llm.chat_json(
SYSTEM_VERIFIER,
@@ -679,6 +722,7 @@ def _apply_drift_verification(
bundle: dict[str, dict],
llm: LLM,
model: str,
target: TargetLanguage | None = None,
) -> None:
"""Check the paragraphs whose translation may have drifted across ids.
@@ -705,6 +749,7 @@ def _apply_drift_verification(
gap there. The alternative - keeping it - puts a line in the audiobook
that belongs to a different sentence, which is worse.
"""
target = target or CHINESE
by_id = {alignment.paragraph_id: alignment for alignment in alignments}
paragraphs = list(chapter.get("paragraphs") or [])
index_of = {str(p.get("id") or ""): i for i, p in enumerate(paragraphs)}
@@ -774,6 +819,7 @@ def _apply_drift_verification(
model,
previous_text=previous_text,
next_text=next_text,
target=target,
)
if verdict == VERDICT_ALIGNED:
@@ -819,6 +865,7 @@ def align_chapter(
llm: LLM | None = None,
model: str = DEFAULT_ALIGN_MODEL,
verify: bool = False,
target: TargetLanguage | None = None,
) -> tuple[list[ParagraphAlignment], dict[str, Any]]:
"""Align every translated paragraph of one chapter.
@@ -826,6 +873,7 @@ def align_chapter(
passage across paragraph ids are checked as well - see
:func:`verify_drift_run`.
"""
target = target or CHINESE
alignments: list[ParagraphAlignment] = []
for paragraph in chapter.get("paragraphs") or []:
paragraph_id = str(paragraph.get("id") or "")
@@ -841,10 +889,10 @@ def align_chapter(
)
)
continue
alignments.append(align_paragraph(paragraph, translation, bundle, llm=llm, model=model))
alignments.append(align_paragraph(paragraph, translation, bundle, llm=llm, model=model, target=target))
if verify and llm is not None:
_apply_drift_verification(alignments, chapter, translated, bundle, llm, model)
_apply_drift_verification(alignments, chapter, translated, bundle, llm, model, target)
methods: dict[str, int] = {}
problems: list[str] = []
@@ -869,9 +917,12 @@ def build_delivery(
bundle: dict[str, dict],
*,
source_ref: str = "",
target: TargetLanguage | None = None,
) -> dict[str, Any]:
"""The payload UmbraAudio's ``import_translations`` accepts."""
target = normalise_target_language(str(book.get("target_language") or ""))
if target is None:
target = resolve(book.get("target_language") or "zh-Hans")
target_code = target.base
utterances: list[dict[str, Any]] = []
unmatched: list[str] = []
@@ -924,7 +975,7 @@ def build_delivery(
)
return {
"target_language": target,
"target_language": target_code,
"source_system": "umbratrans",
"utterances": utterances,
"unmatched": unmatched,
@@ -977,11 +1028,17 @@ def main(argv: list[str] | None = None) -> int:
),
)
parser.add_argument("--source-ref", default="")
parser.add_argument(
"--target-language",
default=None,
help="the language the translations are in. Defaults to the book's own.",
)
args = parser.parse_args(argv)
from pipeline import select_chapters # noqa: PLC0415 - keeps the tools importable standalone
book = json.loads(args.book.read_text(encoding="utf-8"))
target = resolve(args.target_language or book.get("target_language") or "zh-Hans")
bundle_path = args.bundle or Path(str(book.get("bundle") or ""))
if not bundle_path.exists():
print(f"bundle not found: {bundle_path}", file=sys.stderr)
@@ -1012,6 +1069,7 @@ def main(argv: list[str] | None = None) -> int:
llm=llm,
model=args.model,
verify=args.verify,
target=target,
)
chapters.append((chapter, alignments))
reports.append(report)
@@ -1021,7 +1079,7 @@ def main(argv: list[str] | None = None) -> int:
flush=True,
)
delivery = build_delivery(book, chapters, bundle, source_ref=args.source_ref)
delivery = build_delivery(book, chapters, bundle, source_ref=args.source_ref, target=target)
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(
json.dumps(delivery, ensure_ascii=False, indent=2) + "\n",

108
tools/translation/book.py
View File

@@ -21,11 +21,21 @@ import re
from dataclasses import asdict, dataclass, field
from pathlib import Path
from languages import TargetLanguage, resolve
DEFAULT_BUNDLE = Path("/home/umbra/audio-read/bundles/dh-bundle/manifest.jsonl")
# Chinese chapter titles. Kept here, not inferred, so every render is stable.
# Reviewable in one place; the English title stays on the paragraph.
TITLE_ZH = {
# Chapter titles per target language. Kept here, not inferred, so every render
# is stable and reviewable in one place; the English title stays on the
# paragraph.
#
# The Chinese table is the one the pipeline has always run with. A language
# with no table falls back to the English title rather than to another
# language's title: a Spanish edition with Chinese chapter headings is the
# same mislabelling defect as a Chinese draft under a Spanish name, one
# level down.
TITLES: dict[str, dict[str, str]] = {
"zh": {
"Sample Track": "音频版序言：声音介绍",
"Opening Track": "开篇说明",
"Copyright": "版权信息",
@@ -59,6 +69,25 @@ TITLE_ZH = {
"Let’s Make It Better": "让它变得更好",
"Let's Make It Better": "让它变得更好",
"Closing Track": "后记：感谢收听",
},
}
# The book's subtitle per target language. The title itself stays in English
# - "Deal Hunter" is also the ship's name and the AI's name, so translating it
# would rename a character - and the subtitle is the translated reading aid.
# A language with no subtitle falls back to none rather than to another
# language's.
SUBTITLES: dict[str, str] = {
"zh": "交易猎人",
"es": "Cazador de Ofertas",
"fr": "Chasseur de Primes",
"de": "Prämienjäger",
"it": "Cacciatore di Taglie",
"pt": "Caçador de Recompensas",
"ru": "Охотник за Головами",
"ja": "賞金ハンター",
"ko": "보상금 헌터",
"ar": "صياد المكافآت",
}
MAX_NARRATION_SENTENCES = 3
@@ -100,7 +129,7 @@ class Chapter:
index: int
row: int
title: str
title_zh: str
title_target: str
matter: str
paragraphs: list[Paragraph] = field(default_factory=list)
scenes: list[Scene] = field(default_factory=list)
@@ -112,7 +141,8 @@ class Chapter:
@dataclass
class Book:
title: str
title_zh: str
title_target: str
subtitle_target: str
author: str
source_language: str
target_language: str
@@ -129,7 +159,8 @@ class Book:
def to_json(self) -> dict:
return {
"title": self.title,
"title_zh": self.title_zh,
"title_target": self.title_target,
"subtitle_target": self.subtitle_target,
"author": self.author,
"source_language": self.source_language,
"target_language": self.target_language,
@@ -170,12 +201,26 @@ def _clean_title(title: str) -> str:
return title.replace("’", "'").strip()
def _title_zh(title: str) -> str:
def _title_for(title: str, target: TargetLanguage) -> str:
"""The chapter title in the target language.
Two different fallbacks, for two different situations:
- A language with **no title table** falls back to the English title. It
does not fall back to another language's title: a Spanish edition
carrying Chinese chapter headings is the same defect as a Chinese draft
under a Spanish name, one level down.
- A language **with** a table, and a chapter missing from it, gets a
visible marker. An untranslated heading is a gap the reviewer has to
see, not an English title that looks finished.
"""
table = TITLES.get(target.base)
if table is None:
return title
key = _clean_title(title)
if key in TITLE_ZH:
return TITLE_ZH[key]
# Unknown title: leave a visible marker rather than invent a translation.
return f"[待译] {title}"
if key in table:
return table[key]
return target.string("missing_title", "[Untranslated] {title}").format(title=title)
def load_manifest(path: Path = DEFAULT_BUNDLE) -> tuple[list[dict], dict]:
@@ -185,7 +230,8 @@ def load_manifest(path: Path = DEFAULT_BUNDLE) -> tuple[list[dict], dict]:
return records, {"path": str(path), "sha256": digest}
def build_book(path: Path = DEFAULT_BUNDLE) -> Book:
def build_book(path: Path = DEFAULT_BUNDLE, target_language: str = "zh-Hans") -> Book:
target = resolve(target_language)
records, meta = load_manifest(path)
book_rec = next(r for r in records if r.get("kind") == "book")
chapter_recs = sorted(
@@ -201,10 +247,11 @@ def build_book(path: Path = DEFAULT_BUNDLE) -> Book:
book = Book(
title="Deal Hunter",
title_zh="交易猎人",
title_target=_title_for("Deal Hunter", target),
subtitle_target=SUBTITLES.get(target.base, ""),
author="B. D. Murphy",
source_language="en",
target_language="zh-CN",
target_language=target.code,
bundle=meta["path"],
bundle_sha256=meta["sha256"],
)
@@ -220,7 +267,7 @@ def build_book(path: Path = DEFAULT_BUNDLE) -> Book:
index=index,
row=int(cr["database_row_id"]),
title=title,
title_zh=_title_zh(title),
title_target=_title_for(title, target),
matter=matter,
paragraphs=paragraphs,
)
@@ -241,11 +288,16 @@ def _paragraphs(chapter_id: str, title: str, matter: str, rows: list[dict]) -> l
rows = rows[1:]
# The author's-note heading is an utterance, not a paragraph.
if matter == "front_note" and rows and rows[0]["text"].strip().rstrip(".").lower() in {
if (
matter == "front_note"
and rows
and rows[0]["text"].strip().rstrip(".").lower()
in {
"author's note",
"authors note",
"author’s note",
}:
}
):
rows = rows[1:]
paragraphs: list[Paragraph] = []
@@ -323,13 +375,12 @@ def _paragraphs(chapter_id: str, title: str, matter: str, rows: list[dict]) -> l
continue
# Narration: group consecutive sentences into a readable paragraph.
if buf and (
buf[0].get("speaker_name_resolved") != speaker
or buf[0].get("utterance_type") != utype
):
if buf and (buf[0].get("speaker_name_resolved") != speaker or buf[0].get("utterance_type") != utype):
flush()
sentences = _sentences(text)
if buf and (buf_sentences + sentences > MAX_NARRATION_SENTENCES or buf_words + len(text.split()) > MAX_NARRATION_WORDS):
if buf and (
buf_sentences + sentences > MAX_NARRATION_SENTENCES or buf_words + len(text.split()) > MAX_NARRATION_WORDS
):
flush()
buf.append(row)
buf_words += len(text.split())
@@ -370,7 +421,7 @@ def load(path: Path) -> dict:
def summary(book: Book) -> str:
lines = [
f"Book: {book.title} / {book.title_zh}",
f"Book: {book.title} / {book.title_target}",
f"Author: {book.author}",
f"Bundle: {book.bundle}",
f"Bundle sha256: {book.bundle_sha256}",
@@ -379,9 +430,7 @@ def summary(book: Book) -> str:
]
for c in book.chapters:
words = sum(p.words for p in c.paragraphs if p.type != "scene_break")
lines.append(
f"{c.index:>4} {c.matter:16} {len(c.paragraphs):>6} {words:>7} {c.title} / {c.title_zh}"
)
lines.append(f"{c.index:>4} {c.matter:16} {len(c.paragraphs):>6} {words:>7} {c.title} / {c.title_target}")
body = [c for c in book.chapters if c.matter == "body"]
lines += [
"",
@@ -398,8 +447,13 @@ def main() -> None:
ap = argparse.ArgumentParser(description="Extract the Deal Hunter bundle into a book model.")
ap.add_argument("--bundle", type=Path, default=DEFAULT_BUNDLE)
ap.add_argument("--out", type=Path, required=True)
ap.add_argument(
"--target-language",
default="zh-Hans",
help="the language the book is being translated into",
)
args = ap.parse_args()
book = build_book(args.bundle)
book = build_book(args.bundle, args.target_language)
save(book, args.out)
print(summary(book))
print(f"\nwrote {args.out}")

904
tools/translation/languages.py Normal file
View File

@@ -0,0 +1,904 @@
# Copyright 2026 Umbra Tools (umbratools.com)
# SPDX-License-Identifier: Apache-2.0
"""The target languages this pipeline can translate into.
The pipeline was written for one language and hard-coded to it: the drafter
was told to translate into Simplified Chinese, the critic was told to compare
English with Chinese, the glossary's translated forms were read from a ``zh``
key, the chapter titles came from a ``TITLE_ZH`` table, and the renderer
typeset CJK. A production directory that declared ``target_language: es-ES``
was therefore translated into Chinese and filed under a Spanish name.
This module is the single place that says what a target language *is*: its
name, its script, its punctuation, its typography, and the key its
translated forms live under in a glossary. Everything else reads it, so a
second language is a data change rather than a code change.
It also carries the script check. A production's ``target_language`` field is
a claim, and a claim has to be measured: the text in the production directory
is either written in the declared language or it is not, and the dashboard
must not report a Chinese draft under a Spanish name as a completed
Spanish translation.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
# --- scripts ---------------------------------------------------------------
SCRIPT_LATIN = "latin"
SCRIPT_CYRILLIC = "cyrillic"
SCRIPT_HAN = "han"
SCRIPT_KANA = "kana"
SCRIPT_HANGUL = "hangul"
SCRIPT_ARABIC = "arabic"
SCRIPT_NAMES: dict[str, str] = {
SCRIPT_LATIN: "Latin script",
SCRIPT_CYRILLIC: "Cyrillic script",
SCRIPT_HAN: "Han characters",
SCRIPT_KANA: "Japanese kana",
SCRIPT_HANGUL: "Hangul",
SCRIPT_ARABIC: "Arabic script",
}
_SCRIPT_RANGES: tuple[tuple[str, int, int], ...] = (
(SCRIPT_LATIN, 0x0041, 0x005A),
(SCRIPT_LATIN, 0x0061, 0x007A),
(SCRIPT_LATIN, 0x00C0, 0x024F),
(SCRIPT_CYRILLIC, 0x0400, 0x04FF),
(SCRIPT_HAN, 0x3400, 0x4DBF),
(SCRIPT_HAN, 0x4E00, 0x9FFF),
(SCRIPT_HAN, 0xF900, 0xFAFF),
(SCRIPT_KANA, 0x3040, 0x309F),
(SCRIPT_KANA, 0x30A0, 0x30FF),
(SCRIPT_KANA, 0x31F0, 0x31FF),
(SCRIPT_HANGUL, 0x1100, 0x11FF),
(SCRIPT_HANGUL, 0x3130, 0x318F),
(SCRIPT_HANGUL, 0xAC00, 0xD7AF),
(SCRIPT_ARABIC, 0x0600, 0x06FF),
(SCRIPT_ARABIC, 0x0750, 0x077F),
)
# --- the registry ----------------------------------------------------------
@dataclass(frozen=True)
class TargetLanguage:
"""One language the pipeline can translate into.
``code`` is the BCP-47 tag the production declares. ``base`` is the
two-letter code the delivery uses, because the downstream ingest folds
locale codes to their base (``zh-Hans`` becomes ``zh``).
"""
code: str
base: str
name: str
native: str
# The language's name in Chinese. The prompts are written in Chinese, so
# "译成西班牙语" reads as a Chinese instruction while "译成Spanish" reads
# as a typo; the English and native names ride along so the model has no
# room to misread which language is meant.
chinese_name: str
scripts: tuple[str, ...]
# A script that has to be present for the text to be this language at all:
# Japanese without kana is Chinese, and Korean without Hangul is Chinese.
required_script: str | None = None
# Scripts that would mean the delivery is a different language.
forbidden_scripts: tuple[str, ...] = ()
# The key a glossary uses for the translated form of a name or term.
glossary_key: str = ""
# Typography.
quote_open: str = '"'
quote_close: str = '"'
ellipsis: str = "..."
dash: str = "—"
paren_open: str = "("
paren_close: str = ")"
title_open: str = '"'
title_close: str = '"'
font_serif: str = "Noto Serif"
font_sans: str = "Noto Sans"
# The font stack used for the EPUB stylesheet, which has to name real
# installed families rather than the two the FODT declares.
epub_font_stack: str = "Noto Serif, serif"
# The language tag for the AsciiDoc header (``:lang:``) and for the
# EPUB's ``lang`` attribute.
asciidoc_lang: str = "en"
# The language and country codes ODF wants (``fo:language``,
# ``fo:country``). They are separate from the BCP-47 tag: ODF wants
# ``zh`` and ``CN``, not ``zh-Hans``.
odf_language: str = "en"
odf_country: str = "US"
# Chapter numbering: "第 N 章" is a CJK convention, not a universal one.
chapter_prefix: str = "Chapter "
# The character that closes a chapter number in a CJK language: 章 in
# Chinese and Japanese, 장 in Korean.
chapter_suffix: str = ""
# "arabic" (1, 2, 3) or "chinese" (一, 二, 三). Japanese uses Arabic
# numerals with the 第 prefix; Chinese uses Chinese numerals.
chapter_numeral: str = "arabic"
rtl: bool = False
# The characters that end a sentence in this language.
sentence_terminators: str = ".!?…"
# The characters that can close a sentence after its terminator: the
# target's quotation marks and brackets.
closing_marks: str = "\"'”’)]}»"
# The clause boundaries the mechanical fallback splits at, strongest
# first. A translation merges English sentences with these rather than
# with a full stop, so they are where a paragraph can be divided.
clause_marks: tuple[str, ...] = ("—", "–", ":", ";", ",")
# True for the languages whose line-breaking and punctuation rules differ
# enough from Latin that the renderer needs to be told.
cjk: bool = False
@property
def prompt_name(self) -> str:
"""How to name this language inside a prompt."""
return f"{self.chinese_name}（{self.name} / {self.native}）"
@property
def label(self) -> str:
"""A short label for a document header."""
return f"{self.name} ({self.code})"
def chapter_number(self, n: int) -> str:
"""The chapter number as this language writes it."""
if self.chapter_numeral == "chinese":
return _chinese_numeral(n)
return str(n)
def chapter_heading(self, n: int, title: str) -> str:
"""The full chapter heading, in this language's convention.
Chinese writes 第一章　打捞; Spanish writes Capítulo 1 Salvage. The
convention is the language's, not the renderer's.
"""
if self.cjk:
return f"{self.chapter_prefix}{self.chapter_number(n)}{self.chapter_suffix}　{title}"
return f"{self.chapter_prefix}{self.chapter_number(n)} {title}"
def string(self, key: str, default: str = "") -> str:
"""A fixed editorial string in this language.
The strings the renderer writes around the translated text - the
contents heading, the review-sample notice, the translator credit -
are part of the book, so they cannot be left in Chinese when the book
is not Chinese. A language without a table gets the English string,
never another language's.
"""
table = RENDER_STRINGS.get(self.base) or RENDER_STRINGS["en"]
return table.get(key) or default
_CN_DIGITS = "零一二三四五六七八九"
def _chinese_numeral(n: int) -> str:
"""The Chinese numeral for *n*: 1 is 一, 10 is 十, 21 is 二十一."""
if n <= 0:
return str(n)
if n <= 10:
return "十" if n == 10 else _CN_DIGITS[n]
if n < 20:
return "十" + _CN_DIGITS[n - 10]
if n < 100:
return _CN_DIGITS[n // 10] + "十" + (_CN_DIGITS[n % 10] if n % 10 else "")
return str(n)
# The ten target languages the dashboard offers. The codes match the tags the
# web dashboard already uses, so a production directory named for a tag and
# the language the pipeline was told to produce are the same thing.
TARGET_LANGUAGES: tuple[TargetLanguage, ...] = (
TargetLanguage(
code="zh-Hans",
base="zh",
name="Simplified Chinese",
native="简体中文",
chinese_name="简体中文",
scripts=(SCRIPT_HAN,),
forbidden_scripts=(SCRIPT_KANA, SCRIPT_HANGUL),
glossary_key="zh",
quote_open="“",
quote_close="”",
ellipsis="……",
dash="——",
paren_open="（",
paren_close="）",
title_open="《",
title_close="》",
font_serif="Noto Serif CJK SC",
font_sans="Noto Sans CJK SC",
chapter_prefix="第",
chapter_suffix="章",
cjk=True,
sentence_terminators="。！？!?；;…",
closing_marks="”’」』）)】》\"'",
clause_marks=("——", "—", "：", "，", "、", "；", ",", ";"),
asciidoc_lang="zh-CN",
odf_language="zh",
odf_country="CN",
epub_font_stack="Noto Serif CJK SC, Source Han Serif SC, WenQuanYi Micro Hei, serif",
chapter_numeral="chinese",
),
TargetLanguage(
code="es-ES",
base="es",
name="Spanish",
native="español",
chinese_name="西班牙语",
scripts=(SCRIPT_LATIN,),
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
glossary_key="es",
quote_open="“",
quote_close="”",
ellipsis="…",
title_open="«",
title_close="»",
asciidoc_lang="es-ES",
odf_language="es",
odf_country="ES",
chapter_prefix="Capítulo ",
),
TargetLanguage(
code="fr-FR",
base="fr",
name="French",
native="français",
chinese_name="法语",
scripts=(SCRIPT_LATIN,),
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
glossary_key="fr",
quote_open="«",
quote_close="»",
ellipsis="…",
title_open="«",
title_close="»",
asciidoc_lang="fr-FR",
odf_language="fr",
odf_country="FR",
chapter_prefix="Chapitre ",
),
TargetLanguage(
code="de-DE",
base="de",
name="German",
native="Deutsch",
chinese_name="德语",
scripts=(SCRIPT_LATIN,),
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
glossary_key="de",
quote_open="„",
quote_close="“",
ellipsis="…",
title_open="»",
title_close="«",
asciidoc_lang="de-DE",
odf_language="de",
odf_country="DE",
chapter_prefix="Kapitel ",
),
TargetLanguage(
code="it-IT",
base="it",
name="Italian",
native="italiano",
chinese_name="意大利语",
scripts=(SCRIPT_LATIN,),
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
glossary_key="it",
quote_open="«",
quote_close="»",
ellipsis="…",
title_open="«",
title_close="»",
asciidoc_lang="it-IT",
odf_language="it",
odf_country="IT",
chapter_prefix="Capitolo ",
),
TargetLanguage(
code="pt-BR",
base="pt",
name="Portuguese",
native="português",
chinese_name="葡萄牙语",
scripts=(SCRIPT_LATIN,),
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
glossary_key="pt",
quote_open="“",
quote_close="”",
ellipsis="…",
title_open="«",
title_close="»",
asciidoc_lang="pt-BR",
odf_language="pt",
odf_country="BR",
chapter_prefix="Capítulo ",
),
TargetLanguage(
code="ru-RU",
base="ru",
name="Russian",
native="русский",
chinese_name="俄语",
scripts=(SCRIPT_CYRILLIC,),
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL),
glossary_key="ru",
quote_open="«",
quote_close="»",
ellipsis="…",
title_open="«",
title_close="»",
asciidoc_lang="ru-RU",
odf_language="ru",
odf_country="RU",
chapter_prefix="Глава ",
),
TargetLanguage(
code="ja-JP",
base="ja",
name="Japanese",
native="日本語",
chinese_name="日语",
scripts=(SCRIPT_HAN, SCRIPT_KANA),
required_script=SCRIPT_KANA,
forbidden_scripts=(SCRIPT_HANGUL,),
glossary_key="ja",
quote_open="「",
quote_close="」",
ellipsis="……",
dash="——",
paren_open="（",
paren_close="）",
title_open="『",
title_close="』",
font_serif="Noto Serif CJK JP",
font_sans="Noto Sans CJK JP",
chapter_prefix="第",
chapter_suffix="章",
cjk=True,
sentence_terminators="。！？!?；;…",
closing_marks="”’」』）)】》\"'",
clause_marks=("——", "—", "：", "、", "，", "；", ",", ";"),
asciidoc_lang="ja-JP",
odf_language="ja",
odf_country="JP",
epub_font_stack="Noto Serif CJK JP, Source Han Serif JP, serif",
),
TargetLanguage(
code="ko-KR",
base="ko",
name="Korean",
native="한국어",
chinese_name="韩语",
scripts=(SCRIPT_HANGUL,),
required_script=SCRIPT_HANGUL,
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA),
glossary_key="ko",
quote_open="“",
quote_close="”",
ellipsis="…",
dash="—",
paren_open="(",
paren_close=")",
title_open="『",
title_close="』",
font_serif="Noto Serif CJK KR",
font_sans="Noto Sans CJK KR",
cjk=True,
sentence_terminators="。！？!?…",
closing_marks="”’」』）)】》\"'",
clause_marks=("—", "–", ":", ";", ",", "、", "，"),
asciidoc_lang="ko-KR",
odf_language="ko",
odf_country="KR",
epub_font_stack="Noto Serif CJK KR, Source Han Serif KR, serif",
chapter_prefix="제",
chapter_suffix="장",
),
TargetLanguage(
code="ar-SA",
base="ar",
name="Arabic",
native="العربية",
chinese_name="阿拉伯语",
scripts=(SCRIPT_ARABIC,),
required_script=SCRIPT_ARABIC,
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
glossary_key="ar",
quote_open="”",
quote_close="“",
ellipsis="…",
dash="—",
title_open="«",
title_close="»",
font_serif="Noto Naskh Arabic",
font_sans="Noto Sans Arabic",
rtl=True,
asciidoc_lang="ar-SA",
odf_language="ar",
odf_country="SA",
epub_font_stack="Noto Naskh Arabic, Noto Sans Arabic, serif",
chapter_prefix="الفصل ",
# Arabic has its own question mark and semicolon, and its own comma.
sentence_terminators=".!?…؟؛",
closing_marks="”‘)]}»؟؛\"'",
clause_marks=("—", "–", ":", "؛", "،", ";", ","),
),
)
_BY_CODE: dict[str, TargetLanguage] = {lang.code: lang for lang in TARGET_LANGUAGES}
_BY_BASE: dict[str, TargetLanguage] = {lang.base: lang for lang in TARGET_LANGUAGES}
_BY_ALIAS: dict[str, TargetLanguage] = {}
for _lang in TARGET_LANGUAGES:
_BY_ALIAS[_lang.code.casefold()] = _lang
_BY_ALIAS[_lang.base.casefold()] = _lang
_BY_ALIAS[_lang.name.casefold()] = _lang
_BY_ALIAS[_lang.native.casefold()] = _lang
def by_code(value: str | None) -> TargetLanguage | None:
"""Look up a language by tag, base code, English name or native name.
``'es'``, ``'es-ES'``, ``'es_ES'`` and ``'Spanish'`` all resolve to the
same entry, so a caller does not have to know which form a production
directory happens to use.
"""
if not value:
return None
raw = str(value).strip()
if not raw:
return None
language = _BY_ALIAS.get(raw.casefold())
if language is not None:
return language
# 'es_ES' and 'zh_Hans' are common on disk; fold the separator.
folded = raw.replace("_", "-")
language = _BY_ALIAS.get(folded.casefold())
if language is not None:
return language
# A tag we do not carry, but whose base we do ('pt-PT' -> Portuguese).
base = folded.split("-", 1)[0].casefold()
return _BY_BASE.get(base)
def resolve(value: str | None) -> TargetLanguage:
"""The language for *value*, or the Chinese default with a warning.
Falling back silently would recreate the defect this module exists to
prevent: a production that declares a language the pipeline does not
support would be translated into Chinese and filed under the wrong name.
So the fallback is loud.
"""
language = by_code(value)
if language is not None:
return language
print(
f"warning: target language {value!r} is not in the registry; falling back to Simplified Chinese",
flush=True,
)
return _BY_BASE["zh"]
def script_counts(text: str) -> dict[str, int]:
"""How many letters of *text* belong to each script.
Only letters count. Punctuation, digits and whitespace say nothing about
which language a passage is written in, and a name kept in English inside
a Chinese paragraph must not pull the count towards Latin.
"""
counts: dict[str, int] = {}
for ch in text:
point = ord(ch)
for script, low, high in _SCRIPT_RANGES:
if low <= point <= high:
counts[script] = counts.get(script, 0) + 1
break
return counts
def script_mismatch(code: str | None, text: str) -> str | None:
"""Why *text* cannot be a translation into *code*, or None when it can.
The check is deliberately about the *script*, not the language. It cannot
tell Spanish from Portuguese, but it can tell Chinese from Spanish, which
is the mistake that actually happens - a pipeline configured for one
language that silently produces another.
"""
language = by_code(code)
if language is None:
return None
counts = script_counts(text)
total = sum(counts.values())
if total == 0:
return None
for forbidden in language.forbidden_scripts:
if counts.get(forbidden, 0) / total >= 0.10:
return f"the text is written in {SCRIPT_NAMES[forbidden]}, so it is not {language.name}"
if language.required_script and not counts.get(language.required_script):
return f"the text has no {SCRIPT_NAMES[language.required_script]}, so it is not {language.name}"
acceptable = sum(counts.get(script, 0) for script in language.scripts)
if acceptable / total < 0.5:
dominant = max(counts, key=lambda key: counts[key])
return f"the text is mostly {SCRIPT_NAMES[dominant]}, so it is not {language.name}"
return None
# --- text helpers ----------------------------------------------------------
# How many chapters the dashboard samples when it measures a production's
# language. A handful is enough to know which script a book is written in and
# cheap enough to run on every page load.
SAMPLE_CHAPTERS = 8
def sample_run_text(runs_dir: Path, max_chapters: int = SAMPLE_CHAPTERS) -> str:
"""The translated text of a sample of chapters in a run directory.
The sample is spread across the book rather than taken from the front:
the first chapter is often front matter, and a front-matter chapter can be
mostly numbers and names, which say nothing about which language the book
is in.
"""
if not runs_dir.is_dir():
return ""
chapters = sorted(p for p in runs_dir.glob("ch-*") if (p / "translation.json").is_file())
if not chapters:
return ""
if len(chapters) <= max_chapters:
chosen = chapters
else:
step = len(chapters) / max_chapters
chosen = [chapters[int(i * step)] for i in range(max_chapters)]
parts: list[str] = []
for chapter in chosen:
try:
data = json.loads((chapter / "translation.json").read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
continue
paragraphs = data.get("paragraphs") or {}
if isinstance(paragraphs, dict):
parts.extend(str(value) for value in paragraphs.values() if isinstance(value, str))
return " ".join(parts)
def check_runs_language(runs_dir: Path, code: str) -> str | None:
"""Why the translations in *runs_dir* cannot be in *code*, or None.
The dashboard reports a language's state from the presence and chapter
count of its run directory. That is a claim about the *files*, not about
the *text*: a directory full of Chinese can sit in a directory named for
Spanish and be reported as a complete Spanish translation. Measured on
oldbox, that is exactly what ``deal-hunter-es`` was - 3,038 paragraphs
of Chinese under a Spanish name, reported complete.
So the text is measured, not the file names.
"""
language = by_code(code)
if language is None:
return None
sample = sample_run_text(runs_dir)
if not sample.strip():
return None
return script_mismatch(language.code, sample)
# The clause boundary a language uses to join two sentences into one. Chinese
def punctuation_hint(language: TargetLanguage) -> str:
"""How to describe the target's punctuation, for a prompt."""
return (
f"引号用{language.quote_open}{language.quote_close}成对；省略号用{language.ellipsis}；破折号用{language.dash}"
)
# --- the strings the renderer writes ---------------------------------------
# The renderer puts a small amount of fixed text around the translation: the
# contents heading, the review-sample notice, the translator credit, the
# chapter heading convention. That text is part of the book the reader
# receives, so it has to be in the book's language. Leaving it in Chinese is
# the same defect as a Chinese chapter title in a Spanish edition, one level
# down and much easier to miss.
#
# A language with no entry here gets the English string, never another
# language's.
RENDER_STRINGS: dict[str, dict[str, str]] = {
"en": {
"missing_title": "[Untranslated] {title}",
"toc_title": "Table of Contents",
"review_sample": "Review sample",
"translator_credit": "Translated by Umbra Tools",
"sample_note": (
"This file is a review sample of the English translation. The translated "
"chapters are listed in the contents; the remaining chapters will be "
"completed in the full run."
),
"audio_preface": "Audio Preface",
"audio_preface_intro": "The following is the opening of the Deal Hunter audiobook.",
"sample_heading": "About this review sample",
"sample_process": (
"Translation process: paragraph reconstruction, intent analysis, first "
"draft, bilingual review. Terms are in glossary.json; the reviewer's "
"notes are in review.html."
),
"sample_end": "End of sample",
"sample_end_note": (
"That is the review sample. The full run will cover the remaining "
"chapters; the reviewer's issue list is provided with each chapter's "
"translation record."
),
"author_line": "B. D. Murphy",
"translator_line": "Translated by Umbra Tools",
"edition_label": "English edition · Review sample",
},
"zh": {
"missing_title": "[待译] {title}",
"toc_title": "目录",
"review_sample": "审阅样本",
"translator_credit": "译者：Umbra Tools",
"sample_note": "本文件是简体中文翻译的审阅样本。已译部分见目录；未列出的章节将在全文运行中补齐。",
"audio_preface": "音频版序言",
"audio_preface_intro": "以下是《Deal Hunter》有声版的开场白。",
"sample_heading": "审阅样本说明",
"sample_process": (
"翻译流程：段落重建 → 意图分析 → 初译 → 双语审校。术语见 glossary.json，审校意见见 review.html。"
),
"sample_end": "样本结束",
"sample_end_note": ("以上为审阅样本。全文运行将覆盖其余章节；审校问题清单会随各章翻译记录一并提供。"),
"author_line": "B·D·墨菲 著",
"translator_line": "Umbra Tools 译",
"edition_label": "简体中文译本 · 审阅样本",
},
"es": {
"missing_title": "[Sin traducir] {title}",
"toc_title": "Índice",
"review_sample": "Muestra de revisión",
"translator_credit": "Traducido por Umbra Tools",
"sample_note": (
"Este archivo es una muestra de revisión de la traducción al español. "
"Los capítulos traducidos aparecen en el índice; los capítulos "
"restantes se completarán en la ejecución completa."
),
"audio_preface": "Prefacio del audiolibro",
"audio_preface_intro": "Lo que sigue es la introducción del audiolibro de Deal Hunter.",
"sample_heading": "Sobre esta muestra de revisión",
"sample_process": (
"Proceso de traducción: reconstrucción de párrafos, análisis de intención, "
"primer borrador, revisión bilingüe. Los términos están en glossary.json; "
"las notas del revisor están en review.html."
),
"sample_end": "Fin de la muestra",
"sample_end_note": (
"Esta es la muestra de revisión. La ejecución completa cubrirá los "
"capítulos restantes; la lista de incidencias del revisor se entrega "
"junto con el registro de traducción de cada capítulo."
),
"author_line": "B. D. Murphy",
"translator_line": "Traducido por Umbra Tools",
"edition_label": "Edición en español · Muestra de revisión",
},
"fr": {
"missing_title": "[Non traduit] {title}",
"toc_title": "Table des matières",
"review_sample": "Échantillon de relecture",
"translator_credit": "Traduit par Umbra Tools",
"sample_note": (
"Ce fichier est un échantillon de relecture de la traduction française. "
"Les chapitres traduits figurent dans la table des matières ; les "
"chapitres restants seront complétés lors de l'exécution complète."
),
"audio_preface": "Préface de l'audiobook",
"audio_preface_intro": "Voici l'introduction de l'audiobook de Deal Hunter.",
"sample_heading": "À propos de cet échantillon",
"sample_process": (
"Processus de traduction : reconstruction des paragraphes, analyse de "
"l'intention, premier jet, relecture bilingue. Les termes sont dans "
"glossary.json ; les notes du relecteur sont dans review.html."
),
"sample_end": "Fin de l'échantillon",
"sample_end_note": (
"Voici l'échantillon de relecture. L'exécution complète couvrira les "
"chapitres restants ; la liste des remarques du relecteur est fournie "
"avec le dossier de traduction de chaque chapitre."
),
"author_line": "B. D. Murphy",
"translator_line": "Traduit par Umbra Tools",
"edition_label": "Édition française · Échantillon de relecture",
},
"de": {
"missing_title": "[Nicht übersetzt] {title}",
"toc_title": "Inhaltsverzeichnis",
"review_sample": "Korrekturdurchschlag",
"translator_credit": "Übersetzt von Umbra Tools",
"sample_note": (
"Diese Datei ist ein Korrekturdurchschlag der deutschen Übersetzung. "
"Die übersetzten Kapitel stehen im Inhaltsverzeichnis; die restlichen "
"Kapitel werden im vollständigen Lauf ergänzt."
),
"audio_preface": "Hörbuch-Vorwort",
"audio_preface_intro": "Es folgt die Einleitung des Hörbuchs zu Deal Hunter.",
"sample_heading": "Über diesen Korrekturdurchschlag",
"sample_process": (
"Übersetzungsablauf: Absatzrekonstruktion, Intentionsanalyse, Rohtext, "
"zweisprachige Korrektur. Begriffe stehen in glossary.json; die "
"Anmerkungen der Korrektorin stehen in review.html."
),
"sample_end": "Ende des Durchschlags",
"sample_end_note": (
"Das ist der Korrekturdurchschlag. Der vollständige Lauf deckt die "
"restlichen Kapitel ab; die Liste der Anmerkungen liegt dem "
"Übersetzungsprotokoll jedes Kapitels bei."
),
"author_line": "B. D. Murphy",
"translator_line": "Übersetzt von Umbra Tools",
"edition_label": "Deutsche Ausgabe · Korrekturdurchschlag",
},
"it": {
"missing_title": "[Non tradotto] {title}",
"toc_title": "Indice",
"review_sample": "Bozza di revisione",
"translator_credit": "Tradotto da Umbra Tools",
"sample_note": (
"Questo file è una bozza di revisione della traduzione italiana. I "
"capitoli tradotti compaiono nell'indice; i restanti saranno completati "
"nella corsione completa."
),
"audio_preface": "Prefazione dell'audiolibro",
"audio_preface_intro": "Quello che segue è l'introduzione dell'audiolibro di Deal Hunter.",
"sample_heading": "Informazioni su questa bozza",
"sample_process": (
"Processo di traduzione: ricostruzione dei paragrafi, analisi "
"dell'intento, prima stesura, revisione bilingue. I termini sono in "
"glossary.json; le note del revisore sono in review.html."
),
"sample_end": "Fine bozza",
"sample_end_note": (
"Questa è la bozza di revisione. La corsione completa coprirà i "
"capitoli restanti; l'elenco delle osservazioni del revisore "
"accompagna il registro di traduzione di ogni capitolo."
),
"author_line": "B. D. Murphy",
"translator_line": "Tradotto da Umbra Tools",
"edition_label": "Edizione italiana · Bozza di revisione",
},
"pt": {
"missing_title": "[Não traduzido] {title}",
"toc_title": "Sumário",
"review_sample": "Amostra de revisão",
"translator_credit": "Traduzido por Umbra Tools",
"sample_note": (
"Este arquivo é uma amostra de revisão da tradução para português. Os "
"capítulos traduzidos estão no sumário; os capítulos restantes serão "
"completados na execução completa."
),
"audio_preface": "Prefácio do audiolivro",
"audio_preface_intro": "Segue-se a introdução do audiolivro de Deal Hunter.",
"sample_heading": "Sobre esta amostra de revisão",
"sample_process": (
"Processo de tradução: reconstrução de parágrafos, análise de intenção, "
"primeira redação, revisão bilíngue. Os termos estão em glossary.json; "
"as notas do revisor estão em review.html."
),
"sample_end": "Fim da amostra",
"sample_end_note": (
"Esta é a amostra de revisão. A execução completa cobrirá os capítulos "
"restantes; a lista de observações do revisor acompanha o registro de "
"tradução de cada capítulo."
),
"author_line": "B. D. Murphy",
"translator_line": "Traduzido por Umbra Tools",
"edition_label": "Edição em português · Amostra de revisão",
},
"ru": {
"missing_title": "[Не переведено] {title}",
"toc_title": "Оглавление",
"review_sample": "Пробная вычитка",
"translator_credit": "Перевод: Umbra Tools",
"sample_note": (
"Этот файл — пробная вычитка русского перевода. Переведённые главы "
"перечислены в оглавлении; остальные главы будут дополнены при полном "
"прогоне."
),
"audio_preface": "Предисловие к аудиокниге",
"audio_preface_intro": "Ниже следует вступление к аудиокниге «Deal Hunter».",
"sample_heading": "Об этой пробной вычитке",
"sample_process": (
"Порядок перевода: сборка абзацев, анализ замысла, черновик, двуязычная "
"вычитка. Термины — в glossary.json, замечания редактора — в review.html."
),
"sample_end": "Конец пробы",
"sample_end_note": (
"Это пробная вычитка. Полный прогон охватит остальные главы; список "
"замечаний редактора прилагается к протоколу перевода каждой главы."
),
"author_line": "Б. Д. Морфи",
"translator_line": "Перевод: Umbra Tools",
"edition_label": "Русское издание · Пробная вычитка",
},
"ja": {
"missing_title": "[未翻訳] {title}",
"toc_title": "目次",
"review_sample": "査読用サンプル",
"translator_credit": "翻訳：Umbra Tools",
"sample_note": (
"このファイルは日本語訳の査読用サンプルです。翻訳済みの章は目次に記載されています。"
"残りの章は全文実行で補完されます。"
),
"audio_preface": "オーディオブック前書き",
"audio_preface_intro": "以下は『Deal Hunter』オーディオブックの冒頭です。",
"sample_heading": "この査読用サンプルについて",
"sample_process": (
"翻訳の流れ：段落の再構成、意図の分析、初稿、二言語査読。用語は "
"glossary.json、査読者の指摘は review.html にあります。"
),
"sample_end": "サンプル終了",
"sample_end_note": (
"以上が査読用サンプルです。全文実行が残りの章をすべて扱い、査読者の指摘一覧は各章の翻訳記録に添付されます。"
),
"author_line": "B・D・マーフィ 著",
"translator_line": "Umbra Tools 訳",
"edition_label": "日本語版 · 査読用サンプル",
},
"ko": {
"missing_title": "[미번역] {title}",
"toc_title": "목차",
"review_sample": "교정용 샘플",
"translator_credit": "번역: Umbra Tools",
"sample_note": (
"이 파일은 한국어 번역의 교정용 샘플입니다. 번역된 장은 목차에 나열되어 있고, "
"남은 장은 전체 실행에서 완성됩니다."
),
"audio_preface": "오디오북 머리말",
"audio_preface_intro": "다음은 『Deal Hunter』 오디오북의 도입부입니다.",
"sample_heading": "이 교정용 샘플에 대하여",
"sample_process": (
"번역 절차: 문단 재구성, 의도 분석, 초안, 이중 언어 교정. 용어는 "
"glossary.json, 교정자 메모는 review.html에 있습니다."
),
"sample_end": "샘플 끝",
"sample_end_note": (
"이것이 교정용 샘플입니다. 전체 실행이 남은 장을 모두 다루며, 교정자의 지적 "
"목록은 각 장의 번역 기록과 함께 제공됩니다."
),
"author_line": "B. D. 머피 지음",
"translator_line": "Umbra Tools 옮김",
"edition_label": "한국어판 · 교정용 샘플",
},
"ar": {
"missing_title": "[لم يُترجم] {title}",
"toc_title": "المحتويات",
"review_sample": "عيّنة مراجعة",
"translator_credit": "ترجمة: Umbra Tools",
"sample_note": (
"هذا الملف عيّنة مراجعة للترجمة العربية. الفصول المترجمة مدرجة في المحتويات، "
"وتُستكمل بقية الفصول في التشغيل الكامل."
),
"audio_preface": "تمهيد الكتاب الصوتي",
"audio_preface_intro": "فيما يلي مقدمة الكتاب الصوتي لـ Deal Hunter.",
"sample_heading": "عن هذه عيّنة المراجعة",
"sample_process": (
"مسار الترجمة: إعادة بناء الفقرات، تحليل القصد، المسودة الأولى، المراجعة "
"ثنائية اللغة. المصطلحات في glossary.json، وملاحظات المراجع في review.html."
),
"sample_end": "نهاية العيّنة",
"sample_end_note": (
"هذه هي عيّنة المراجعة. سيغطي التشغيل الكامل بقية الفصول، وتُرفق قائمة ملاحظات المراجع مع سجل ترجمة كل فصل."
),
"author_line": "ب. د. مورفي",
"translator_line": "ترجمة: Umbra Tools",
"edition_label": "الطبعة العربية · عيّنة مراجعة",
},
}

312
tools/translation/pipeline.py
View File

@@ -24,16 +24,17 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from languages import SCRIPT_LATIN, TargetLanguage, resolve, script_mismatch # noqa: E402
from llm import DEFAULT_ANALYST, DEFAULT_CRITIC, DEFAULT_MODEL, LLM, LLMError, Usage # noqa: E402
from prompts import ( # noqa: E402
SYSTEM_ANALYST,
SYSTEM_CRITIC,
SYSTEM_DRAFTER,
analysis_prompt,
consistency_prompt,
critic_prompt,
draft_prompt,
load_glossary,
system_analyst,
system_critic,
system_drafter,
)
CHARACTER_NOTES = """- 凯因达：聪明、直率、好奇，带一点叛逆；对权威不耐烦，对朋友温暖。不要把她写成娇滴滴的公主。
@@ -43,42 +44,101 @@ CHARACTER_NOTES = """- 凯因达：聪明、直率、好奇，带一点叛逆；
- 罗曼诺：热情、商人腔、爱谈价格，但不油滑。
- 旁白：克制、清晰、有画面感；不替人物抒情。"""
# The default target, used where a caller has not supplied one. The pipeline
# was Chinese-only for its whole life, so the default keeps every existing
# call site behaving exactly as it did.
CHINESE = resolve("zh-Hans")
# Proper nouns that may legitimately stay in Latin script. Anything else that
# survives in the Chinese text is reported, not silently accepted.
LATIN_ALLOW = {
"ai", "audio", "authorbdmurphy", "b", "blazes", "brutus", "com", "d",
"deal", "dial", "eva", "felucca", "gisfrid", "goodreads", "hunter",
"http", "https", "isbn", "kabir", "kainda", "konrad", "llc", "l2",
"maul", "murphy", "nanny", "niñera", "romano", "sao", "umbra", "vince",
"wakam", "wake", "weston", "willamina", "witold", "worstauthors",
"worstauthor", "www", "zheng", "yi", "adonai", "adric", "albert",
"ai",
"audio",
"authorbdmurphy",
"b",
"blazes",
"brutus",
"com",
"d",
"deal",
"dial",
"eva",
"felucca",
"gisfrid",
"goodreads",
"hunter",
"http",
"https",
"isbn",
"kabir",
"kainda",
"konrad",
"llc",
"l2",
"maul",
"murphy",
"nanny",
"niñera",
"romano",
"sao",
"umbra",
"vince",
"wakam",
"wake",
"weston",
"willamina",
"witold",
"worstauthors",
"worstauthor",
"www",
"zheng",
"yi",
"adonai",
"adric",
"albert",
}
def _normalise_quotes(text: str) -> str:
text = re.sub(r'"([^"\n]*)"', r"“\1”", text)
def _normalise_quotes(text: str, target: TargetLanguage | None = None) -> str:
"""Balance the target language's quotation marks.
The marks are the language's own: a German draft uses „ and “, a French
one « and ». The straight double quote the model sometimes emits is
converted to the target's pair.
"""
target = target or CHINESE
open_mark, close_mark = target.quote_open, target.quote_close
text = re.sub(r'"([^"\n]*)"', lambda m: open_mark + m.group(1) + close_mark, text)
if '"' in text:
parts = text.split('"')
out = parts[0]
for i, part in enumerate(parts[1:]):
out += ("“" if i % 2 == 0 else "”") + part
out += (open_mark if i % 2 == 0 else close_mark) + part
text = out
if "“" not in text and text.endswith("”"):
if open_mark not in text and text.endswith(close_mark):
text = text[:-1]
opens, closes = text.count("“"), text.count("”")
opens, closes = text.count(open_mark), text.count(close_mark)
if opens > closes:
text += "”" * (opens - closes)
text += close_mark * (opens - closes)
elif closes > opens:
for _ in range(closes - opens):
idx = text.rfind("”")
if idx > text.rfind("“"):
idx = text.rfind(close_mark)
if idx > text.rfind(open_mark):
text = text[:idx] + text[idx + 1 :]
else:
break
return text
def sanitize(text: str) -> str:
def sanitize(text: str, target: TargetLanguage | None = None) -> str:
"""Clean up one drafted paragraph.
The Chinese replacements repair a set of mistakes the Chinese drafter
keeps making - a transliterated title, a transliterated ship name. They
cannot match in another language, so they are harmless there; the
punctuation rules at the end are the target's own.
"""
target = target or CHINESE
text = text.strip()
text = re.sub(r"^\[[^\]]{1,40}\]\s*", "", text)
text = re.sub(r"^（(?:对话|旁白|叙述)[^）]{0,60}）\s*", "", text)
@@ -98,8 +158,11 @@ def sanitize(text: str) -> str:
text = text.replace("Umbra Audio星系系统", "Umbra Audio 系统")
text = text.replace("音频星系系统", "音频系统")
text = text.replace("在小时前", "一小时前")
if target.ellipsis == "……":
text = text.replace("...", "……").replace("--", "——")
return _normalise_quotes(text).strip()
else:
text = text.replace("...", target.ellipsis).replace("--", target.dash)
return _normalise_quotes(text, target).strip()
def latin_residue(text: str) -> tuple[list[str], list[str]]:
@@ -135,6 +198,16 @@ def load_book(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8"))
def chapter_title_target(chapter: dict) -> str:
"""The chapter's title in the target language.
The field was called ``title_zh`` while the pipeline was Chinese-only and
is now called ``title_target``. The old name is still read, so a book file
written by an earlier version keeps working.
"""
return chapter.get("title_target") or chapter.get("title_zh") or chapter.get("title") or ""
def select_chapters(book: dict, selector: str) -> list[dict]:
chapters = book["chapters"]
if selector in {"all", "*"}:
@@ -178,7 +251,9 @@ def chunk_chapter(chapter: dict, max_words: int = 260, max_paras: int = 14) -> l
current, current_words, current_scene = [], 0, None
continue
p_scene = scene_of.get(p["id"], 0)
if current and (p_scene != current_scene or current_words + len(p["text"].split()) > max_words or len(current) >= max_paras):
if current and (
p_scene != current_scene or current_words + len(p["text"].split()) > max_words or len(current) >= max_paras
):
raw.append(current)
current, current_words = [], 0
current.append(p)
@@ -226,11 +301,15 @@ class Runner:
max_words: int = 260,
max_paras: int = 14,
critic_limit: int = 0,
target_language: str = "zh-Hans",
) -> None:
self.book = load_book(book_path)
self.glossary = load_glossary(glossary_path)
self.run_dir = run_dir
self.run_dir.mkdir(parents=True, exist_ok=True)
self.target: TargetLanguage = resolve(target_language)
self.book_path = book_path
self.glossary_path = glossary_path
self.model = model
self.analyst_model = analyst_model
self.critic_model = critic_model
@@ -248,6 +327,7 @@ class Runner:
usage_log=run_dir / "usage.jsonl",
**kwargs,
)
self._guard_run_directory()
# ----------------------------------------------------------------- files
def chapter_dir(self, chapter: dict) -> Path:
@@ -263,6 +343,72 @@ class Runner:
def _write(self, path: Path, value: object) -> None:
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
# --------------------------------------------------------------- the guard
def _guard_run_directory(self) -> None:
"""Refuse to reuse a run directory that belongs to another language.
The pipeline caches by file existence, and the run directory is chosen
by the caller. So a Spanish run pointed at a directory that already
holds a Chinese run would skip drafting entirely and publish the
Chinese under a Spanish name - the exact defect the mislabelled
production had, and the reason it is worth a guard rather than a
convention.
Two checks, one from the claim and one from the measurement:
* The manifest records what the directory is for. A mismatch is
refused.
* A directory with no manifest (written before this guard existed) is
measured instead: the script of the text it already holds is
compared against the target language.
"""
manifest_path = self.run_dir / "run.json"
previous: dict = {}
if manifest_path.exists() and not self.force:
try:
previous = json.loads(manifest_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
previous = {}
declared = previous.get("target_language")
if declared and declared != self.target.code:
raise SystemExit(
f"run directory {self.run_dir} holds a {declared} run, but this "
f"run is {self.target.code}. Use a different --run-dir, or pass "
"--force to overwrite it deliberately."
)
self._measure_existing_language()
self._write(
manifest_path,
{
"target_language": self.target.code,
"target_name": self.target.name,
"glossary_key": self.target.glossary_key,
"book": str(self.book_path),
"glossary": str(self.glossary_path),
},
)
def _measure_existing_language(self) -> None:
"""What language does the run directory already hold?"""
for chapter_dir in sorted(self.run_dir.glob("ch-*")):
path = chapter_dir / "translation.json"
if not path.exists():
continue
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return
paragraphs = data.get("paragraphs") or {}
sample = " ".join(str(value) for value in list(paragraphs.values())[:200])
problem = script_mismatch(self.target.code, sample)
if problem:
raise SystemExit(
f"run directory {self.run_dir} already holds a translation that "
f"is not {self.target.name}: {problem}. Use a different "
"--run-dir, or pass --force to overwrite it deliberately."
)
return
# ----------------------------------------------------------------- passes
def analyse(self, chapter: dict, chunk: Chunk, previous_summary: str) -> dict:
path = self.chapter_dir(chapter) / f"{chunk.id}.analysis.json"
@@ -270,16 +416,17 @@ class Runner:
if cached is not None:
return cached
prompt = analysis_prompt(
chapter_title=f"{chapter['title']} / {chapter['title_zh']}",
chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}",
scene_index=chunk.scene_index,
chunk_index=int(chunk.id.rsplit("c", 1)[-1]),
chunks_in_scene=chunk.chunks_in_scene,
paragraphs=chunk.paragraphs,
previous_summary=previous_summary,
glossary=self.glossary,
target=self.target,
)
value, usage = self.llm.chat_json(
SYSTEM_ANALYST,
system_analyst(self.target),
prompt,
model=self.analyst_model,
temperature=0.2,
@@ -305,8 +452,7 @@ class Runner:
# translated. Record that the analysis is degraded and continue with a
# minimal one; the drafter falls back to the source and glossary.
print(
f" {chunk.id}: analyst returned {type(value).__name__}, "
"using a minimal analysis",
f" {chunk.id}: analyst returned {type(value).__name__}, using a minimal analysis",
flush=True,
)
value = {"degraded": True, "note": "analyst returned non-object JSON"}
@@ -321,7 +467,7 @@ class Runner:
return cached
expected = [p["id"] for p in chunk.paragraphs if p.get("type") != "scene_break"]
prompt = draft_prompt(
chapter_title=f"{chapter['title']} / {chapter['title_zh']}",
chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}",
scene_index=chunk.scene_index,
chunk_index=int(chunk.id.rsplit("c", 1)[-1]),
chunks_in_scene=chunk.chunks_in_scene,
@@ -329,9 +475,10 @@ class Runner:
analysis={k: v for k, v in analysis.items() if k != "_usage"},
previous_summary=previous_summary,
glossary=self.glossary,
target=self.target,
)
value, usage = self.llm.chat_json(
SYSTEM_DRAFTER,
system_drafter(self.target),
prompt,
model=self.model,
temperature=0.45,
@@ -358,26 +505,58 @@ class Runner:
self._write(path, result)
return result
def _draft_text(self, item: dict) -> str:
"""The translated text of one drafted paragraph.
The key is the target language's own, but the model is not always
disciplined about it, so a neutral key is accepted too.
"""
for key in (self.target.glossary_key, "text", "translation", "zh"):
value = item.get(key)
if isinstance(value, str) and value.strip():
return value
return ""
def _normalise_draft(self, value: object, expected: list[str]) -> dict[str, str]:
out: dict[str, str] = {}
if isinstance(value, dict) and isinstance(value.get("paragraphs"), list):
items = value["paragraphs"]
elif isinstance(value, list):
items = value
elif isinstance(value, dict) and value.get("id") and value.get("zh"):
elif isinstance(value, dict) and value.get("id") and self._draft_text(value):
items = [value]
elif isinstance(value, dict):
# maybe mapping id->zh
# maybe mapping id->text
for k, v in value.items():
if isinstance(v, str) and k.startswith("ch-"):
out[k] = sanitize(v)
out[k] = sanitize(v, self.target)
return {k: v for k, v in out.items() if k in expected and v}
else:
raise LLMError(f"drafter returned unexpected JSON: {str(value)[:200]}")
for item in items:
if isinstance(item, dict) and item.get("id") and item.get("zh"):
out[str(item["id"])] = sanitize(str(item["zh"]))
return {k: v for k, v in out.items() if k in expected and v}
if isinstance(item, dict) and item.get("id"):
text = self._draft_text(item)
if text:
out[str(item["id"])] = sanitize(text, self.target)
out = {k: v for k, v in out.items() if k in expected and v}
self._check_draft_language(out)
return out
def _check_draft_language(self, out: dict[str, str]) -> None:
"""Is the draft actually written in the target language?
The drafter is told which language to translate into, and it has been
known to ignore that and write Chinese anyway. A draft in the wrong
language is not a translation, so it is refused rather than stored: a
book that is 32 chapters in the wrong language is a failure that is
expensive to discover late.
"""
if not out:
return
sample = " ".join(out.values())
problem = script_mismatch(self.target.code, sample)
if problem:
raise LLMError(f"the drafter answered in the wrong language: {problem}. Expected {self.target.name}.")
def _fix_paragraph_quotes(self, chunk: Chunk, drafts: dict[str, str]) -> dict[str, str]:
"""Remove quotes the model wrapped around unquoted narration."""
@@ -397,16 +576,24 @@ class Runner:
return drafts
def _repair_residue(self, chapter: dict, chunk: Chunk, drafts: dict[str, str]) -> dict[str, str]:
"""Translate stray lowercase English words in place, without touching names."""
"""Translate stray lowercase English words in place, without touching names.
Only meaningful when the target is not written in Latin script: in a
Spanish or French draft a Latin word is the target language, not
residue. Running the check there would flag the whole book and
"repair" it into nonsense.
"""
if SCRIPT_LATIN in self.target.scripts:
return drafts
by_id = {p["id"]: p for p in chunk.paragraphs}
for pid, zh in list(drafts.items()):
common, _proper = latin_residue(zh)
if not common:
continue
source = by_id.get(pid, {}).get("text", "")
prompt = f"""下面这段中文译文里混入了英文单词：{", ".join(common)}。
请只把这些英文单词译成中文，其余内容尽量保持不变；不要解释，不要改写剧情。
输出 JSON：{{"zh": "修正后的中文译文"}}
prompt = f"""下面这段{self.target.chinese_name}译文里混入了英文单词：{", ".join(common)}。
请只把这些英文单词译成{self.target.chinese_name}，其余内容尽量保持不变；不要解释，不要改写剧情。
输出 JSON：{{"{self.target.glossary_key}": "修正后的{self.target.chinese_name}译文"}}
英文原文：
{source}
@@ -415,16 +602,16 @@ class Runner:
{zh}"""
try:
value, _ = self.llm.chat_json(
SYSTEM_DRAFTER,
system_drafter(self.target),
prompt,
model=self.model,
temperature=0.2,
max_tokens=1500,
cache_key=f"{chunk.id}.{pid}.residue",
)
fixed = value.get("zh") if isinstance(value, dict) else None
fixed = self._draft_text(value) if isinstance(value, dict) else None
if fixed:
drafts[pid] = sanitize(str(fixed))
drafts[pid] = sanitize(str(fixed), self.target)
except LLMError as exc:
print(f" {chunk.id}/{pid}: residue repair failed: {exc}", flush=True)
return drafts
@@ -437,7 +624,7 @@ class Runner:
p = by_id[pid]
try:
prompt = draft_prompt(
chapter_title=f"{chapter['title']} / {chapter['title_zh']}",
chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}",
scene_index=chunk.scene_index,
chunk_index=0,
chunks_in_scene=1,
@@ -445,9 +632,10 @@ class Runner:
analysis=analysis,
previous_summary="",
glossary=self.glossary,
target=self.target,
)
value, _ = self.llm.chat_json(
SYSTEM_DRAFTER,
system_drafter(self.target),
prompt,
model=self.model,
temperature=0.4,
@@ -461,7 +649,7 @@ class Runner:
except Exception:
pass
# Fallback to source text or empty to avoid blocking
result[pid] = sanitize(str(p.get("text", "")))
result[pid] = sanitize(str(p.get("text", "")), self.target)
return result
def criticise(self, chapter: dict, chunk: Chunk, draft: dict[str, str]) -> dict:
@@ -470,14 +658,15 @@ class Runner:
if cached is not None:
return cached
prompt = critic_prompt(
chapter_title=f"{chapter['title']} / {chapter['title_zh']}",
chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}",
paragraphs=chunk.paragraphs,
draft=draft,
glossary=self.glossary,
target=self.target,
)
try:
value, usage = self.llm.chat_json(
SYSTEM_CRITIC,
system_critic(self.target),
prompt,
model=self.critic_model,
temperature=0.15,
@@ -490,7 +679,7 @@ class Runner:
print(f" critic {self.critic_model} failed ({exc}); falling back to {self.model}", flush=True)
try:
value, usage = self.llm.chat_json(
SYSTEM_CRITIC,
system_critic(self.target),
prompt,
model=self.model,
temperature=0.15,
@@ -562,14 +751,15 @@ class Runner:
if cached is not None:
return cached
prompt = consistency_prompt(
chapter_title=f"{chapter['title']} / {chapter['title_zh']}",
chapter_title=f"{chapter['title']} / {chapter_title_target(chapter)}",
paragraphs=paragraphs,
draft=drafts,
glossary=self.glossary,
character_notes=CHARACTER_NOTES,
target=self.target,
)
value, usage = self.llm.chat_json(
SYSTEM_CRITIC,
system_critic(self.target),
prompt,
model=self.critic_model,
temperature=0.15,
@@ -590,7 +780,7 @@ class Runner:
for ci, chapter in enumerate(chapters, 1):
chunks = chunk_chapter(chapter, self.max_words, self.max_paras)
print(
f"[{ci}/{len(chapters)}] {chapter['id']} {chapter['title']} / {chapter['title_zh']}: "
f"[{ci}/{len(chapters)}] {chapter['id']} {chapter['title']} / {chapter_title_target(chapter)}: "
f"{len(chapter['paragraphs'])} paragraphs, {len(chunks)} chunks",
flush=True,
)
@@ -599,12 +789,10 @@ class Runner:
issues: list[dict] = []
for chunk in chunks:
analysis = self.analyse(chapter, chunk, previous_summary)
previous_summary = str(analysis.get("summary_zh") or previous_summary)
previous_summary = str(analysis.get("summary") or analysis.get("summary_zh") or previous_summary)
drafted = self.draft(chapter, chunk, analysis, previous_summary)
drafts.update(drafted["paragraphs"])
if self.do_critic and (
self.critic_limit <= 0 or int(chunk.id.rsplit("c", 1)[-1]) < self.critic_limit
):
if self.do_critic and (self.critic_limit <= 0 or int(chunk.id.rsplit("c", 1)[-1]) < self.critic_limit):
crit = self.criticise(chapter, chunk, drafted["paragraphs"])
for issue in crit.get("issues", []) or []:
issue = dict(issue)
@@ -625,7 +813,8 @@ class Runner:
"chapter_id": chapter["id"],
"chapter_index": chapter["index"],
"title": chapter["title"],
"title_zh": chapter["title_zh"],
"title_target": chapter_title_target(chapter),
"target_language": self.target.code,
"paragraphs": drafts,
"issues": issues,
"chunks": [c.id for c in chunks],
@@ -658,10 +847,19 @@ def report(run_dir: Path) -> str:
def main() -> None:
ap = argparse.ArgumentParser(description="Run the Deal Hunter Chinese translation pipeline.")
ap = argparse.ArgumentParser(description="Run the Deal Hunter translation pipeline.")
ap.add_argument("--book", type=Path, required=True)
ap.add_argument("--glossary", type=Path, required=True)
ap.add_argument("--run-dir", type=Path, required=True)
ap.add_argument(
"--target-language",
default=None,
help=(
"the language to translate into (zh-Hans, es-ES, fr-FR, de-DE, "
"it-IT, pt-BR, ru-RU, ja-JP, ko-KR, ar-SA). Defaults to the "
"book's own target_language."
),
)
ap.add_argument("--chapters", default="all", help="all, front, body, back, or comma-separated indices/ids")
ap.add_argument("--model", default=DEFAULT_MODEL)
ap.add_argument("--analyst-model", default=DEFAULT_ANALYST)
@@ -672,7 +870,9 @@ def main() -> None:
ap.add_argument("--consistency", action="store_true")
ap.add_argument("--max-words", type=int, default=260)
ap.add_argument("--max-paras", type=int, default=14)
ap.add_argument("--critic-limit", type=int, default=0, help="criticise only the first N chunks per chapter (0 = all)")
ap.add_argument(
"--critic-limit", type=int, default=0, help="criticise only the first N chunks per chapter (0 = all)"
)
ap.add_argument("--report", action="store_true")
args = ap.parse_args()
@@ -681,6 +881,7 @@ def main() -> None:
return
book = load_book(args.book)
target_language = args.target_language or book.get("target_language") or "zh-Hans"
chapters = select_chapters(book, args.chapters)
runner = Runner(
book_path=args.book,
@@ -696,6 +897,7 @@ def main() -> None:
max_words=args.max_words,
max_paras=args.max_paras,
critic_limit=args.critic_limit,
target_language=target_language,
)
runner.run(chapters)

275
tools/translation/prompts.py
View File

@@ -6,6 +6,20 @@
The prompts follow specs/27: analyse the scene before rendering it, give the
drafter the intent and the glossary, then check meaning with a different
model. They deliberately do not ask for word-for-word equivalence.
Every prompt is parameterised by the target language. The pipeline used to
be hard-coded to Simplified Chinese - the drafter was told to translate into
Chinese, the critic was told to compare English with Chinese, the glossary's
translated forms were read from a ``zh`` key - so a production that declared
``target_language: es-ES`` was translated into Chinese and filed under a
Spanish name. The target language now comes from
:class:`languages.TargetLanguage`, and the language's own script, punctuation
and glossary key follow from it.
The prompt scaffolding stays in Chinese. It is the language the pipeline was
developed in, the analyst model handles it well, and what the drafter has to
be told is *which* language to translate into - not a translation of the
instructions themselves.
"""
from __future__ import annotations
@@ -13,6 +27,8 @@ from __future__ import annotations
import json
from pathlib import Path
from languages import TargetLanguage
CORE_CHARACTERS = [
"Kainda Walters Foster Murphy",
"Princess Kainda",
@@ -49,55 +65,14 @@ CORE_TERMS = [
"coronation",
]
def load_glossary(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8"))
def _pick(mapping: dict, keys: list[str]) -> list[tuple[str, str]]:
out: list[tuple[str, str]] = []
for key in keys:
if key in mapping:
value = mapping[key]
if isinstance(value, dict):
zh = value.get("zh") or value.get("short") or ""
note = value.get("note")
if note:
zh = f"{zh}（{note}）"
else:
zh = str(value)
if zh:
out.append((key, zh))
return out
def glossary_block(glossary: dict, speakers: set[str], text: str) -> str:
chars = dict(glossary.get("characters", {}))
roles = dict(glossary.get("roles", {}))
places = dict(glossary.get("places", {}))
terms = dict(glossary.get("terms", {}))
keys = list(CORE_CHARACTERS)
for s in sorted(speakers):
if s and s not in keys:
keys.append(s)
char_pairs = _pick({**chars, **roles}, keys)
term_keys = list(CORE_TERMS)
low = text.lower()
for key in terms:
if key.lower() in low and key not in term_keys:
term_keys.append(key)
term_pairs = _pick(terms, term_keys)
place_pairs = _pick(places, list(places))
title = glossary.get("title", {})
lines = ["【书名与专名规则】"]
if title:
en = title.get("en", "Deal Hunter")
sub = title.get("subtitle_zh", "")
lines.append(f"- 书名：{en}；正文书名写作《{en}》" + (f"，副标题可写“{sub}”" if sub else ""))
lines += [
# The hard rules a language needs, as data rather than as code. The Chinese
# table is the one the pipeline has always run with, kept verbatim so the
# existing production is unaffected; a glossary may override it with its own
# "rules" list. A language with no built-in table falls back to rules derived
# from the glossary's own terms, and says so - a silently empty rule set is
# how the mislabelled Spanish production happened.
BUILTIN_RULES: dict[str, list[str]] = {
"zh": [
"- Deal Hunter 作为船名、AI 名或书名时保留英文，不译成中文；不要写成《交易猎人》",
"- Hunter 作为 AI 的简称保留英文，不译成“猎人”",
"- Umbra Audio 保留英文，不音译",
@@ -111,16 +86,129 @@ def glossary_block(glossary: dict, speakers: set[str], text: str) -> str:
"- salvage = 打捞物 / 打捞（不译作“救援”）",
"- bounty = 赏金；bounty hunter = 赏金猎人",
"- XO = 副舰长；captain = 舰长",
"",
"【人物与称谓】",
]
lines += [f"- {en} = {zh}" for en, zh in char_pairs]
],
}
# Rules that hold for every language of this book. They are about the book,
# not about a language: *Deal Hunter* is the ship's name, the AI's name and
# the book's title, so it stays in English in every edition, and a name the
# glossary does not carry is never transliterated into existence.
#
# The Chinese table above carries its own wording of these, so the Chinese
# prompt is byte-for-byte what it always was and the existing production
# stays reproducible.
GLOBAL_RULES = [
"- Deal Hunter 作为船名、AI 名或书名时保留英文，不译成{target_name}",
"- Hunter 作为 AI 的简称保留英文",
"- Umbra Audio 保留英文，不音译",
"- 只翻译原文出现的称谓；不要补全人物全名、头衔或身份",
"- 专名严格按术语表；术语表没有的名字保留原文，不要自行音译",
]
def load_glossary(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8"))
def _translated(value: dict, target: TargetLanguage) -> str:
"""The target-language form of a glossary entry.
The key is the target's own: ``es`` for a Spanish glossary, ``zh`` for a
Chinese one. A glossary written for a different language is not silently
borrowed from - that is how Chinese text ends up under a Spanish name.
"""
for key in (target.glossary_key, "short", "target", "text"):
candidate = value.get(key)
if isinstance(candidate, str) and candidate.strip():
return candidate
return ""
def _pick(mapping: dict, keys: list[str], target: TargetLanguage) -> list[tuple[str, str]]:
out: list[tuple[str, str]] = []
for key in keys:
if key not in mapping:
continue
value = mapping[key]
if isinstance(value, dict):
rendered = _translated(value, target)
note = value.get("note")
if note:
rendered = f"{rendered}{target.paren_open}{note}{target.paren_close}"
else:
rendered = str(value)
if rendered:
out.append((key, rendered))
return out
def _rules(glossary: dict, target: TargetLanguage, term_pairs: list[tuple[str, str]]) -> list[str]:
"""The hard rules for the target language.
Three sources, in order: the glossary's own ``rules`` list, the
language's built-in table, and - for a language with no table - the
global rules of this book plus the glossary's own terms. A language never
gets an empty rule set, and never gets another language's.
"""
own = glossary.get("rules")
if isinstance(own, list) and own:
return [str(line) for line in own]
built_in = BUILTIN_RULES.get(target.base)
if built_in is not None:
return built_in
# No table for this language: the book's own rules, then the glossary's
# terms, so the model is still told what the hard vocabulary is.
derived = [line.format(target_name=target.chinese_name) for line in GLOBAL_RULES]
if term_pairs:
derived.append("")
derived.append("【硬性术语：违反即为错译】")
derived.extend(f"- {en} = {rendered}" for en, rendered in term_pairs)
return derived
def glossary_block(
glossary: dict,
speakers: set[str],
text: str,
target: TargetLanguage,
) -> str:
chars = dict(glossary.get("characters", {}))
roles = dict(glossary.get("roles", {}))
places = dict(glossary.get("places", {}))
terms = dict(glossary.get("terms", {}))
keys = list(CORE_CHARACTERS)
for s in sorted(speakers):
if s and s not in keys:
keys.append(s)
char_pairs = _pick({**chars, **roles}, keys, target)
term_keys = list(CORE_TERMS)
low = text.lower()
for key in terms:
if key.lower() in low and key not in term_keys:
term_keys.append(key)
term_pairs = _pick(terms, term_keys, target)
place_pairs = _pick(places, list(places), target)
title = glossary.get("title", {})
lines = ["【书名与专名规则】"]
if title:
en = title.get("en", "Deal Hunter")
sub = title.get(f"subtitle_{target.glossary_key}", "")
lines.append(
f"- 书名：{en}；正文书名写作{target.title_open}{en}{target.title_close}"
+ (f"，副标题可写{target.quote_open}{sub}{target.quote_close}" if sub else "")
)
lines += _rules(glossary, target, term_pairs)
lines += ["", "【人物与称谓】"]
lines += [f"- {en} = {rendered}" for en, rendered in char_pairs]
if place_pairs:
lines.append("【地名与舰名】")
lines += [f"- {en} = {zh}" for en, zh in place_pairs]
lines += [f"- {en} = {rendered}" for en, rendered in place_pairs]
if term_pairs:
lines.append("【术语】")
lines += [f"- {en} = {zh}" for en, zh in term_pairs]
lines += [f"- {en} = {rendered}" for en, rendered in term_pairs]
return "\n".join(lines)
@@ -129,20 +217,29 @@ def style_block(glossary: dict) -> str:
return "\n".join(f"- {v}" for v in style.values() if isinstance(v, str))
SYSTEM_ANALYST = """你是一位双语文学编辑，负责在翻译前分析英文小说的场景。
def system_analyst(target: TargetLanguage) -> str:
return (
"""你是一位双语文学编辑，负责在翻译前分析英文小说的场景。
你的任务不是翻译，而是读出这一段文字的意图、语气、人物关系和可能被直译毁掉的地方。
摘要和术语建议请用"""
+ target.prompt_name
+ """写。
只输出 JSON，不要解释，不要 Markdown 代码围栏。"""
)
SYSTEM_DRAFTER = """你是一位资深文学译者，把英文太空冒险小说译成简体中文。
你的目标不是逐字对应，而是让中文读者读到和英文读者相同的场景、语气、人物关系和潜台词。
def system_drafter(target: TargetLanguage) -> str:
return f"""你是一位资深文学译者，把英文太空冒险小说译成{target.prompt_name}。
你的目标不是逐字对应，而是让{target.chinese_name}读者读到和英文读者相同的场景、语气、人物关系和潜台词。
可以重组句子、合并或拆分分句，但不得增删事实、不得替作者解释、不得加译注。
对话要像中文人物在说话；叙述要流畅、克制、有画面感。
严格使用给出的术语表。未列入术语表的人名、地名、舰名保留英文原文，不要自行音译。
不要补全原文没有出现的全名、头衔或身份；原文说“Princess Kainda”，不要写成“凯因达·沃尔特斯·福斯特·墨菲公主”。
Deal Hunter 作为船名、AI 名或书名保留英文；Hunter 作为简称保留英文；Umbra Audio 保留英文。
对话要像{target.chinese_name}人物在说话；叙述要流畅、克制、有画面感。
严格使用给出的术语表。未列入术语表的人名、地名、舰名保留原文，不要自行音译。
不要补全原文没有出现的全名、头衔或身份。
只输出 JSON，不要解释，不要 Markdown 代码围栏。"""
SYSTEM_CRITIC = """你是一位严格的双语审校。对照英文原文和中文译文，只找意义、指代、语气、术语和遗漏方面的问题。
def system_critic(target: TargetLanguage) -> str:
return f"""你是一位严格的双语审校。对照英文原文和{target.prompt_name}译文，只找意义、指代、语气、术语和遗漏方面的问题。
不要因为“风格不同”而报错；只有当读者会因此理解错、或人物声音被抹平、或事实被改变时才报。
不要改写全文，只报告问题并给出最小修正建议。
只输出 JSON，不要解释，不要 Markdown 代码围栏。"""
@@ -152,11 +249,11 @@ def _paragraph_lines(paragraphs: list[dict]) -> str:
lines = []
for p in paragraphs:
if p.get("type") == "scene_break":
lines.append(f'[{p["id"]}] (场景分隔)')
lines.append(f"[{p['id']}] (场景分隔)")
continue
speaker = p.get("speaker") or "narrator"
ptype = p.get("type") or "narration"
lines.append(f'[{p["id"]}] ({ptype}; {speaker}) {p["text"]}')
lines.append(f"[{p['id']}] ({ptype}; {speaker}) {p['text']}")
return "\n".join(lines)
@@ -171,7 +268,7 @@ def _draft_paragraph_lines(paragraphs: list[dict]) -> str:
for p in paragraphs:
if p.get("type") == "scene_break":
continue
lines.append(f'[{p["id"]}] {p["text"]}')
lines.append(f"[{p['id']}] {p['text']}")
return "\n".join(lines)
@@ -184,6 +281,7 @@ def analysis_prompt(
paragraphs: list[dict],
previous_summary: str,
glossary: dict,
target: TargetLanguage,
) -> str:
speakers = {p.get("speaker") or "narrator" for p in paragraphs}
text = " ".join(p.get("text", "") for p in paragraphs)
@@ -192,17 +290,17 @@ def analysis_prompt(
场景 {scene_index + 1}，第 {chunk_index + 1}/{chunks_in_scene} 段
上一段场景摘要：{previous_summary or "（无，本章开头）"}
{glossary_block(glossary, speakers, text)}
{glossary_block(glossary, speakers, text, target)}
{style_block(glossary)}
请分析下面这段英文。输出 JSON：
{{
"summary_zh": "1-2句中文（80字以内），概括本段发生了什么、人物想要什么、语气如何；供下一段翻译参考",
"summary": "1-2句{target.chinese_name}（80字以内），概括本段发生了什么、人物想要什么、语气如何；供下一段翻译参考",
"register": "正式/中性/口语（15字以内）",
"tone": "例如：紧张、戏谑、疲惫、克制（15字以内）",
"participants": ["本段实际说话或被点名的人物"],
"proposed_terms": [{{"en": "术语表里没有的专名", "zh": "建议译名", "why": "简短理由"}}],
"proposed_terms": [{{"en": "术语表里没有的专名", "{target.glossary_key}": "建议译名", "why": "简短理由"}}],
"risks": [{{"id": "段落id", "issue": "直译会错在哪里", "guidance": "译者应注意什么"}}]
}}
@@ -223,6 +321,7 @@ def draft_prompt(
analysis: dict,
previous_summary: str,
glossary: dict,
target: TargetLanguage,
) -> str:
speakers = {p.get("speaker") or "narrator" for p in paragraphs}
text = " ".join(p.get("text", "") for p in paragraphs)
@@ -232,31 +331,29 @@ def draft_prompt(
场景 {scene_index + 1}，第 {chunk_index + 1}/{chunks_in_scene} 段
上一段场景摘要：{previous_summary or "（无，本章开头）"}
{glossary_block(glossary, speakers, text)}
{glossary_block(glossary, speakers, text, target)}
{style_block(glossary)}
翻译前的场景分析（必须遵守其中的语气、人物关系和风险提示）：
{analysis_text}
请把下面英文译为简体中文。输出 JSON：
{{"paragraphs": [{{"id": "段落id", "zh": "中文译文"}}]}}
请把下面英文译为{target.prompt_name}。输出 JSON：
{{"paragraphs": [{{"id": "段落id", "{target.glossary_key}": "{target.chinese_name}译文"}}]}}
规则：
- 段落 id 必须与原文一一对应，数量和顺序不变。
- 每个 id 只输出一段中文；不要把两段合并成一个 id。
- 每个 id 只输出一段译文；不要把两段合并成一个 id。
- **每个 id 的译文只能覆盖该 id 自己的英文。** 原文常在句子中间被切成多段
（例如一段是"The chair in the middle has"，下一段是'"Captain"'），译文也必须
照这个切分来，哪怕某一段的中文读起来不完整。绝对不要把相邻段落的内容提前
照这个切分来，哪怕某一段的译文读起来不完整。绝对不要把相邻段落的内容提前
写进前一个 id，也不要把本段的内容挪到下一个 id。宁可让某段译文是半句。
- 只输出译文。绝对不要把段落 id、说话人、类型、括号注释写进译文。
- 不要输出英文原文，不要输出解释。
- 不要把英文单词留在译文里（术语表允许保留的专名和书名除外）；每个英文实词都必须译成中文。
- 对话用中文“”，并保证每个引号成对；不要把英文直引号 " 原样搬进译文。叙述中不要滥用引号。
- 如果原文有省略号、破折号、打断，保留其节奏；省略号用……，破折号用——。
- 专名严格按术语表；术语表没有的名字保留英文。
- 指恒星系时，system / star system 译作“星系”（如 Albert system = 阿尔伯特星系；236 systems = 236 个星系）。
- 指船载、音频、计算机等设备时，system 译作“系统”（如 ship systems = 飞船系统；Umbra Audio system = Umbra Audio 系统）。不要一律译作“星系”。
- 不要把英文单词留在译文里（术语表允许保留的专名和书名除外）；每个英文实词都必须译成{target.chinese_name}。
- 对话用{target.quote_open}{target.quote_close}，并保证每个引号成对；不要把英文直引号 " 原样搬进译文。叙述中不要滥用引号。
- 如果原文有省略号、破折号、打断，保留其节奏；省略号用{target.ellipsis}，破折号用{target.dash}。
- 专名严格按术语表；术语表没有的名字保留原文。
英文原文：
{_draft_paragraph_lines(paragraphs)}"""
@@ -268,6 +365,7 @@ def critic_prompt(
paragraphs: list[dict],
draft: dict[str, str],
glossary: dict,
target: TargetLanguage,
) -> str:
speakers = {p.get("speaker") or "narrator" for p in paragraphs}
text = " ".join(p.get("text", "") for p in paragraphs)
@@ -281,17 +379,15 @@ def critic_prompt(
"speaker": p.get("speaker"),
"type": p.get("type"),
"en": p["text"],
"zh": draft.get(p["id"], ""),
target.glossary_key: draft.get(p["id"], ""),
}
)
return f"""书名：Deal Hunter（交易猎人）
本章：{chapter_title}
{glossary_block(glossary, speakers, text)}
{glossary_block(glossary, speakers, text, target)}
请逐段对照英文与中文，只报告会让读者理解错、或改变人物声音、或丢失/增加事实的问题。
注意：原文 “236 systems in the Dial space sector” 中的 systems 指恒星系，译作“星系”是正确的，不要改成“系统”。ship systems / audio system 才译作“系统”。
请逐段对照英文与{target.chinese_name}译文，只报告会让读者理解错、或改变人物声音、或丢失/增加事实的问题。
输出 JSON：
{{
@@ -301,9 +397,9 @@ def critic_prompt(
"id": "段落id",
"severity": "block|major|minor",
"type": "meaning|pronoun|irony|term|omission|addition|register|misalignment",
"problem": "中文说明问题",
"problem": "{target.chinese_name}说明问题",
"source_excerpt": "英文片段（不超过30词）",
"target_excerpt": "中文片段（不超过40字）",
"target_excerpt": "{target.chinese_name}片段（不超过40字）",
"fix": "最小修正建议，不要重译整段"
}}
]
@@ -336,18 +432,19 @@ def consistency_prompt(
draft: dict[str, str],
glossary: dict,
character_notes: str,
target: TargetLanguage,
) -> str:
speakers = {p.get("speaker") or "narrator" for p in paragraphs}
text = " ".join(p.get("text", "") for p in paragraphs)
pairs = [
{"id": p["id"], "speaker": p.get("speaker"), "zh": draft.get(p["id"], "")}
{"id": p["id"], "speaker": p.get("speaker"), target.glossary_key: draft.get(p["id"], "")}
for p in paragraphs
if p.get("type") != "scene_break"
]
return f"""书名：Deal Hunter（交易猎人）
本章：{chapter_title}
{glossary_block(glossary, speakers, text)}
{glossary_block(glossary, speakers, text, target)}
人物声音约束：
{character_notes}

91
tools/translation/qa.py
View File

@@ -1,11 +1,16 @@
# Copyright 2026 Umbra Tools (umbratools.com)
# SPDX-License-Identifier: Apache-2.0
"""Mechanical QA for the Deal Hunter Chinese translation.
"""Mechanical QA for a translated chapter.
This does not judge literary quality. It catches the defects that can be
detected without a human: missing paragraphs, leaked prompt metadata, stray
English, unbalanced quotes, duplicated punctuation and forbidden title forms.
The checks are the target language's own. A Spanish draft is full of Latin
words, so the "stray English" check cannot run there - it would flag the
whole book. The quote characters, the punctuation and the script count come
from the same registry the prompts and the renderer use.
"""
from __future__ import annotations
@@ -18,14 +23,40 @@ from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from book import SUBTITLES
from languages import SCRIPT_LATIN, TargetLanguage, resolve, script_counts # noqa: E402
from pipeline import latin_residue # noqa: E402
CHINESE = resolve("zh-Hans")
def forbidden_titles(target: TargetLanguage) -> list[str]:
"""The translated book title, in the target's own book-title marks.
The rule is "the book title stays in English", so the form that must not
appear in the text is the *translated* title. The Chinese table is the
one the pipeline has always used; every other language derives its own
from the subtitle table, and a language with no subtitle has nothing to
forbid.
"""
if target.base == "zh":
return ["《交易猎人》"]
subtitle = SUBTITLES.get(target.base, "")
if not subtitle:
return []
return [f"{target.title_open}{subtitle}{target.title_close}"]
def _load(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8"))
def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]:
def check_chapter(
chapter: dict,
data: dict,
target: TargetLanguage | None = None,
) -> tuple[list[dict], dict]:
target = target or CHINESE
drafts: dict[str, str] = data.get("paragraphs", {})
by_id = {p["id"]: p for p in chapter["paragraphs"] if p.get("type") != "scene_break"}
expected = list(by_id)
@@ -38,40 +69,46 @@ def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]:
if pid not in by_id:
issues.append({"id": pid, "level": "major", "kind": "unknown_id", "detail": "id not in source"})
# A target written in Latin script has Latin words by definition, so the
# residue check only applies where Latin script is foreign.
check_latin = SCRIPT_LATIN not in target.scripts
forbidden = forbidden_titles(target)
double_punctuation = r"[。，！？；：]{2,}" if target.base == "zh" else r"[.!?;]{2,}"
chars = 0
for pid, zh in drafts.items():
zh = str(zh)
chars += len(re.findall(r"[\u4e00-\u9fff]", zh))
counts = script_counts(zh)
chars += sum(counts.get(script, 0) for script in target.scripts)
if re.search(r"^\[ch-[^\]]+\]", zh) or re.search(r"（(?:对话|旁白|叙述)(?:；[^）]*)?）", zh):
issues.append({"id": pid, "level": "block", "kind": "metadata_leak", "detail": zh[:40]})
if '"' in zh:
issues.append({"id": pid, "level": "major", "kind": "straight_quote", "detail": zh[:60]})
if zh.count("“") != zh.count("”"):
opens = zh.count(target.quote_open)
closes = zh.count(target.quote_close)
if opens != closes:
issues.append(
{
"id": pid,
"level": "major",
"kind": "quote_balance",
"detail": f"{zh.count('“')} open / {zh.count('”')} close",
"detail": f"{opens} open / {closes} close",
}
)
if re.search(r"[。，！？；：]{2,}", zh):
if re.search(double_punctuation, zh):
issues.append({"id": pid, "level": "minor", "kind": "double_punctuation", "detail": zh[:60]})
if "《交易猎人》" in zh:
for form in forbidden:
if form in zh:
issues.append({"id": pid, "level": "major", "kind": "title_translated", "detail": zh[:60]})
break
if check_latin:
common, proper = latin_residue(zh)
if common:
issues.append(
{"id": pid, "level": "major", "kind": "english_residue", "detail": ", ".join(common)}
)
issues.append({"id": pid, "level": "major", "kind": "english_residue", "detail": ", ".join(common)})
if proper:
issues.append(
{"id": pid, "level": "info", "kind": "proper_name_review", "detail": ", ".join(proper)}
)
issues.append({"id": pid, "level": "info", "kind": "proper_name_review", "detail": ", ".join(proper)})
source = by_id.get(pid, {}).get("text", "")
star_system = re.search(
r"\bstar system\b|\b(?!The\b)[A-Z][a-z]+\s+system\b|\b\d+\s+systems\b", source
)
star_system = re.search(r"\bstar system\b|\b(?!The\b)[A-Z][a-z]+\s+system\b|\b\d+\s+systems\b", source)
if star_system:
phrase = star_system.group(0).lower()
if any(
@@ -89,7 +126,11 @@ def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]:
)
):
star_system = None
if star_system and "系统" in zh:
# The mistranslation this catches is a Chinese one: a star system
# rendered with the word for an installed system. In another language
# the wrong word is a different word and the critic, not this check,
# is what finds it.
if star_system and target.base == "zh" and "系统" in zh:
issues.append(
{"id": pid, "level": "major", "kind": "system_mistranslated", "detail": "star system -> 系统"}
)
@@ -98,23 +139,26 @@ def check_chapter(chapter: dict, data: dict) -> tuple[list[dict], dict]:
"chapter_id": chapter["id"],
"chapter_index": chapter["index"],
"title": chapter["title"],
"target_language": target.code,
"expected": len(expected),
"translated": len(drafts),
"chinese_chars": chars,
"target_chars": chars,
"issues": len(issues),
}
return issues, metrics
def main() -> None:
ap = argparse.ArgumentParser(description="Mechanical QA for the Chinese translation.")
ap = argparse.ArgumentParser(description="Mechanical QA for a translated chapter.")
ap.add_argument("--book", type=Path, required=True)
ap.add_argument("--run-dir", type=Path, required=True)
ap.add_argument("--target-language", default=None)
ap.add_argument("--json-out", type=Path)
ap.add_argument("--strict", action="store_true", help="exit 1 on any block/major issue")
args = ap.parse_args()
book = _load(args.book)
target = resolve(args.target_language or book.get("target_language") or "zh-Hans")
all_issues: list[dict] = []
metrics: list[dict] = []
@@ -123,7 +167,7 @@ def main() -> None:
if not path.exists():
continue
data = _load(path)
issues, m = check_chapter(chapter, data)
issues, m = check_chapter(chapter, data, target)
for issue in issues:
issue["chapter_id"] = chapter["id"]
issue["chapter_title"] = chapter["title"]
@@ -136,11 +180,12 @@ def main() -> None:
by_level[issue["level"]] = by_level.get(issue["level"], 0) + 1
by_kind[issue["kind"]] = by_kind.get(issue["kind"], 0) + 1
print(f"TARGET: {target.label}")
print("CHAPTERS")
for m in metrics:
print(
f" {m['chapter_id']:6} {m['title'][:28]:28} "
f"{m['translated']:>4}/{m['expected']:<4} paras {m['chinese_chars']:>6} chars {m['issues']:>3} issues"
f"{m['translated']:>4}/{m['expected']:<4} paras {m['target_chars']:>6} chars {m['issues']:>3} issues"
)
print()
print("ISSUES BY LEVEL:", dict(sorted(by_level.items())))
@@ -148,9 +193,7 @@ def main() -> None:
print()
for issue in all_issues:
if issue["level"] in {"block", "major"}:
print(
f" [{issue['level']:5}] {issue['kind']:20} {issue['chapter_id']}/{issue['id']}: {issue['detail']}"
)
print(f" [{issue['level']:5}] {issue['kind']:20} {issue['chapter_id']}/{issue['id']}: {issue['detail']}")
elif issue["level"] == "minor":
print(f" [minor] {issue['kind']:20} {issue['chapter_id']}/{issue['id']}: {issue['detail']}")
else:

284
tools/translation/render.py
View File

@@ -7,6 +7,12 @@ The AsciiDoc file is the canonical text output. The ODT is produced from a
Flat ODF (.fodt) document so that styles, page breaks and the table of
contents are under our control, then converted with LibreOffice. The TOC is
updated through UNO, because a plain ``--convert-to`` leaves an empty index.
The typography is the target language's: its fonts, its language and country
codes, its chapter-numbering convention, its quotation marks, and the fixed
editorial strings the renderer writes around the translation. A Spanish
edition typeset in Chinese fonts with Chinese chapter headings is the same
defect as a Chinese draft under a Spanish name, one level down.
"""
from __future__ import annotations
@@ -23,10 +29,11 @@ import time
from pathlib import Path
from xml.sax.saxutils import escape
from book import SUBTITLES
from languages import SCRIPT_LATIN, TargetLanguage, resolve
from llm import DEFAULT_ANALYST, DEFAULT_CRITIC, DEFAULT_MODEL
FONT_SERIF = "Noto Serif CJK SC"
FONT_SANS = "Noto Sans CJK SC"
CHINESE = resolve("zh-Hans")
def _esc(text: str) -> str:
@@ -36,14 +43,21 @@ def _esc(text: str) -> str:
LATIN_RUN = re.compile(r"[A-Za-z][A-Za-z0-9'’.\-·]*(?:\s+[A-Za-z][A-Za-z0-9'’.\-·]*)*")
def _rich_esc(text: str) -> str:
def _rich_esc(text: str, target: TargetLanguage | None = None) -> str:
"""Escape text and mark Latin runs as English.
A Chinese paragraph tells LibreOffice to use Chinese line-breaking for the
whole run, which can split an English name in the middle of a word
("Deal Hu / nter"). A character style with fo:language="en" keeps the
Latin run on Western line-breaking rules without changing its font.
For a target written in Latin script this is not applied at all: the whole
text is Latin, and marking it as English would tell the renderer that a
Spanish paragraph is English.
"""
target = target or CHINESE
if SCRIPT_LATIN in target.scripts:
return _esc(text)
out: list[str] = []
last = 0
for m in LATIN_RUN.finditer(text):
@@ -82,18 +96,38 @@ def _ordered_chapters(book: dict, run_dir: Path | None, include_missing: bool =
return [c for c in out if chapter_drafts(run_dir, c["id"])]
def _title_of(chapter: dict) -> str:
"""The chapter's title in the target language.
def _cn_num(n: int) -> str:
digits = "零一二三四五六七八九"
if n <= 0:
return str(n)
if n <= 10:
return "十" if n == 10 else digits[n]
if n < 20:
return "十" + digits[n - 10]
if n < 100:
return digits[n // 10] + "十" + (digits[n % 10] if n % 10 else "")
return str(n)
The field was called ``title_zh`` while the pipeline was Chinese-only
and is now called ``title_target``. The old name is still read, so a book
file written by an earlier version keeps working.
"""
return chapter.get("title_target") or chapter.get("title_zh") or chapter.get("title") or ""
def _subtitle_of(book: dict, target: TargetLanguage | None = None) -> str:
"""The book's subtitle in the target language.
The old book format had no subtitle field: it stored the subtitle in the
book-level ``title_zh``, so that is read as a fallback. The table is the
last resort, and a language with no entry in it gets no subtitle rather
than another language's.
"""
target = target or CHINESE
return (
book.get("subtitle_target") or book.get("subtitle_zh") or book.get("title_zh") or SUBTITLES.get(target.base, "")
)
def _book_title(book: dict) -> str:
"""The book's title.
The title stays in English for every language: "Deal Hunter" is also the
ship's name and the AI's name, so translating it would rename a
character.
"""
return book.get("title_target") or book.get("title") or "Deal Hunter"
def _body_chapter_number(book: dict, chapter: dict) -> int:
@@ -124,24 +158,34 @@ def _is_author_note_code(text: str) -> bool:
# --------------------------------------------------------------------- AsciiDoc
def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str = "") -> str:
def build_asciidoc(
book: dict,
run_dir: Path,
*,
target: TargetLanguage | None = None,
sample: bool = False,
date: str = "",
) -> str:
target = target or CHINESE
chapters = _ordered_chapters(book, run_dir)
title = _book_title(book)
subtitle = _subtitle_of(book, target)
lines: list[str] = []
lines.append("= Deal Hunter: 交易猎人")
lines.append(f"= {title}: {subtitle}" if subtitle else f"= {title}")
lines.append("B. D. Murphy")
lines.append(":doctype: book")
lines.append(":lang: zh-CN")
lines.append(f":lang: {target.asciidoc_lang}")
lines.append(":toc: left")
lines.append(":toclevels: 2")
lines.append(f":revdate: {date or time.strftime('%Y-%m-%d')}")
if sample:
lines.append(":revremark: 审阅样本")
lines.append(f":revremark: {target.string('review_sample')}")
lines.append(":icons: font")
lines.append("")
lines.append("// 译者：Umbra Tools")
lines.append(f"// {target.string('translator_credit')}")
if sample:
lines.append("")
lines.append("NOTE: 本文件是简体中文翻译的审阅样本。已译部分见目录；未列出的章节将在全文运行中补齐。")
lines.append(f"NOTE: {target.string('sample_note')}")
lines.append("")
seen_audio = False
@@ -154,15 +198,15 @@ def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str
continue
seen_audio = True
lines.append(f"// source: {c['id']} (audio front matter)")
lines.append("== 音频版序言")
lines.append(f"== {target.string('audio_preface')}")
lines.append("")
lines.append("以下是《Deal Hunter》有声版的开场白。")
lines.append(target.string("audio_preface_intro"))
lines.append("")
for ac in [x for x in book["chapters"] if x["matter"] == "front_audio"]:
adrafts = chapter_drafts(run_dir, ac["id"])
if not adrafts:
continue
lines.append(f"=== {ac['title_zh']}")
lines.append(f"=== {_title_of(ac)}")
lines.append("")
for p in ac["paragraphs"]:
zh = adrafts.get(p["id"])
@@ -178,9 +222,9 @@ def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str
if c["matter"] == "body":
n = _body_chapter_number(book, c)
heading = f"第{_cn_num(n)}章　{c['title_zh']}"
heading = target.chapter_heading(n, _title_of(c))
else:
heading = c["title_zh"]
heading = _title_of(c)
lines.append(f"// source: {c['id']} (database row {c['row']}), {len(c['paragraphs'])} paragraphs")
lines.append(f"== {heading}")
lines.append("")
@@ -206,15 +250,25 @@ def build_asciidoc(book: dict, run_dir: Path, *, sample: bool = False, date: str
# --------------------------------------------------------------------- Flat ODF
STYLES = f"""
def build_styles(target: TargetLanguage | None = None) -> str:
"""The document stylesheet for the target language.
The fonts, the language and country codes and the writing direction are
the target's. A document that declares ``fo:language="zh"`` while its
text is Spanish tells LibreOffice to hyphenate and shape it as Chinese,
which is how a Latin-script edition ends up with the wrong line breaks.
"""
target = target or CHINESE
direction = ' fo:direction="rtl"' if target.rtl else ""
return f"""
<office:font-face-decls>
<style:font-face style:name="{FONT_SERIF}" svg:font-family="'{FONT_SERIF}'" style:font-family-generic="roman" style:font-pitch="variable"/>
<style:font-face style:name="{FONT_SANS}" svg:font-family="'{FONT_SANS}'" style:font-family-generic="swiss" style:font-pitch="variable"/>
<style:font-face style:name="{target.font_serif}" svg:font-family="'{target.font_serif}'" style:font-family-generic="roman" style:font-pitch="variable"/>
<style:font-face style:name="{target.font_sans}" svg:font-family="'{target.font_sans}'" style:font-family-generic="swiss" style:font-pitch="variable"/>
</office:font-face-decls>
<office:styles>
<style:default-style style:family="paragraph">
<style:paragraph-properties fo:orphans="2" fo:widows="2" fo:hyphenation-ladder-count="no-limit"/>
<style:text-properties style:font-name="{FONT_SERIF}" fo:font-size="11pt" fo:language="zh" fo:country="CN"/>
<style:text-properties style:font-name="{target.font_serif}" fo:font-size="11pt" fo:language="{target.odf_language}" fo:country="{target.odf_country}"{direction}/>
</style:default-style>
<style:style style:name="Title" style:family="paragraph" style:parent-style-name="Standard" style:class="text" style:master-page-name="FirstPage">
<style:paragraph-properties fo:text-align="center" fo:margin-top="2.1in" fo:margin-bottom="0.15in"/>
@@ -294,31 +348,48 @@ STYLES = f"""
"""
def _odt_paragraph(style: str, text: str, outline: int | None = None) -> str:
def _odt_paragraph(
style: str,
text: str,
outline: int | None = None,
target: TargetLanguage | None = None,
) -> str:
body = _rich_esc(text, target)
if outline:
return f'<text:h text:style-name="{style}" text:outline-level="{outline}">{_rich_esc(text)}</text:h>'
return f'<text:p text:style-name="{style}">{_rich_esc(text)}</text:p>'
return f'<text:h text:style-name="{style}" text:outline-level="{outline}">{body}</text:h>'
return f'<text:p text:style-name="{style}">{body}</text:p>'
def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = "") -> str:
def build_fodt(
book: dict,
run_dir: Path,
*,
target: TargetLanguage | None = None,
sample: bool = False,
date: str = "",
) -> str:
target = target or CHINESE
chapters = _ordered_chapters(book, run_dir)
body: list[str] = []
title = _book_title(book)
subtitle = _subtitle_of(book, target)
# Title page
body.append(_odt_paragraph("Title", "Deal Hunter"))
body.append(_odt_paragraph("Subtitle", "交易猎人"))
body.append(_odt_paragraph("Author", "B·D·墨菲 著"))
body.append(_odt_paragraph("Small", "Umbra Tools 译"))
body.append(_odt_paragraph("Title", title, target=target))
if subtitle:
body.append(_odt_paragraph("Subtitle", subtitle, target=target))
body.append(_odt_paragraph("Author", target.string("author_line"), target=target))
body.append(_odt_paragraph("Small", target.string("translator_line"), target=target))
if sample:
body.append(_odt_paragraph("Small", "简体中文译本 · 审阅样本"))
body.append(_odt_paragraph("Small", date or time.strftime("%Y-%m-%d")))
body.append(_odt_paragraph("Small", target.string("edition_label"), target=target))
body.append(_odt_paragraph("Small", date or time.strftime("%Y-%m-%d"), target=target))
body.append('<text:p text:style-name="PageBreak"/>')
# Table of contents
body.append(
'<text:table-of-content text:name="TableOfContents" text:style-name="Contents1" text:protected="false">'
'<text:table-of-content-source text:outline-level="1">'
'<text:index-title-template text:style-name="TOCTitle">目录</text:index-title-template>'
f'<text:index-title-template text:style-name="TOCTitle">{target.string("toc_title")}</text:index-title-template>'
'<text:table-of-content-entry-template text:outline-level="1" text:style-name="Contents1">'
"<text:index-entry-text/>"
'<text:index-entry-tab-stop style:type="right" style:leader-char="."/>'
@@ -329,19 +400,9 @@ def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = "
)
if sample:
body.append(_odt_paragraph("ChapterHeading", "审阅样本说明", 1))
body.append(
_odt_paragraph(
"FrontText",
"本文件是《Deal Hunter》简体中文翻译的审阅样本。已译部分见目录；未列出的章节将在全文运行中补齐。",
)
)
body.append(
_odt_paragraph(
"FrontText",
"翻译流程：段落重建 → 意图分析 → 初译 → 双语审校。术语见 glossary.json，审校意见见 review.html。",
)
)
body.append(_odt_paragraph("ChapterHeading", target.string("sample_heading"), 1, target=target))
body.append(_odt_paragraph("FrontText", target.string("sample_note"), target=target))
body.append(_odt_paragraph("FrontText", target.string("sample_process"), target=target))
seen_audio = False
for c in chapters:
@@ -352,46 +413,41 @@ def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = "
if seen_audio:
continue
seen_audio = True
body.append(_odt_paragraph("ChapterHeading", "音频版序言", 1))
body.append(_odt_paragraph("FrontText", "以下是《Deal Hunter》有声版的开场白。"))
body.append(_odt_paragraph("ChapterHeading", target.string("audio_preface"), 1, target=target))
body.append(_odt_paragraph("FrontText", target.string("audio_preface_intro"), target=target))
for ac in [x for x in book["chapters"] if x["matter"] == "front_audio"]:
adrafts = chapter_drafts(run_dir, ac["id"])
if not adrafts:
continue
body.append(_odt_paragraph("SectionHeading", ac["title_zh"], 2))
body.append(_odt_paragraph("SectionHeading", _title_of(ac), 2, target=target))
for p in ac["paragraphs"]:
zh = adrafts.get(p["id"])
if not zh:
continue
if p.get("type") == "scene_break":
body.append(_odt_paragraph("SceneBreak", "* * *"))
body.append(_odt_paragraph("SceneBreak", "* * *", target=target))
else:
body.append(_odt_paragraph("FrontText", zh))
body.append(_odt_paragraph("FrontText", zh, target=target))
continue
if c["matter"] == "body":
n = _body_chapter_number(book, c)
heading = f"第{_cn_num(n)}章　{c['title_zh']}"
heading = target.chapter_heading(n, _title_of(c))
else:
heading = c["title_zh"]
body.append(_odt_paragraph("ChapterHeading", heading, 1))
heading = _title_of(c)
body.append(_odt_paragraph("ChapterHeading", heading, 1, target=target))
for p in c["paragraphs"]:
zh = drafts.get(p["id"])
if not zh:
continue
if p.get("type") == "scene_break":
body.append(_odt_paragraph("SceneBreak", "* * *"))
body.append(_odt_paragraph("SceneBreak", "* * *", target=target))
continue
body.append(_odt_paragraph(_paragraph_style(p, c["matter"]), zh))
body.append(_odt_paragraph(_paragraph_style(p, c["matter"]), zh, target=target))
if sample:
body.append(_odt_paragraph("ChapterHeading", "样本结束", 1))
body.append(
_odt_paragraph(
"FrontText",
"以上为审阅样本。全文运行将覆盖其余章节；审校问题清单会随各章翻译记录一并提供。",
)
)
body.append(_odt_paragraph("ChapterHeading", target.string("sample_end"), 1, target=target))
body.append(_odt_paragraph("FrontText", target.string("sample_end_note"), target=target))
return f"""<?xml version="1.0" encoding="UTF-8"?>
<office:document xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0"
@@ -407,7 +463,7 @@ def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = "
xmlns:svg="urn:oasis:names:tc:opendocument:xmlns:svg-compatible:1.0"
xmlns:of="urn:oasis:names:tc:opendocument:xmlns:of:1.2"
office:version="1.3" office:mimetype="application/vnd.oasis.opendocument.text">
{STYLES}
{build_styles(target)}
<office:automatic-styles>
<style:page-layout style:name="pm1">
<style:page-layout-properties fo:page-width="8.27in" fo:page-height="11.69in"
@@ -419,7 +475,7 @@ def build_fodt(book: dict, run_dir: Path, *, sample: bool = False, date: str = "
</office:automatic-styles>
<office:master-styles>
<style:master-page style:name="Standard" style:page-layout-name="pm1" style:next-style-name="Standard">
<style:header><text:p text:style-name="Header">Deal Hunter · 交易猎人</text:p></style:header>
<style:header><text:p text:style-name="Header">{_esc(title + (f" · {subtitle}" if subtitle else ""))}</text:p></style:header>
<style:footer><text:p text:style-name="Footer"><text:page-number text:select-page="current">1</text:page-number></text:p></style:footer>
</style:master-page>
<style:master-page style:name="FirstPage" style:page-layout-name="pm1"/>
@@ -526,17 +582,21 @@ def render(
sample: bool = False,
date: str = "",
pdf: bool = True,
target_language: str = "zh-Hans",
) -> dict[str, Path]:
target = resolve(target_language)
book = json.loads(book_path.read_text(encoding="utf-8"))
out_dir.mkdir(parents=True, exist_ok=True)
stem = "deal-hunter-zh-sample" if sample else "deal-hunter-zh"
stem = f"deal-hunter-{target.base}-sample" if sample else f"deal-hunter-{target.base}"
asciidoc = out_dir / f"{stem}.asciidoc"
fodt = out_dir / f"{stem}.fodt"
odt = out_dir / f"{stem}.odt"
pdf_path = out_dir / f"{stem}.pdf" if pdf else None
asciidoc.write_text(build_asciidoc(book, run_dir, sample=sample, date=date), encoding="utf-8")
fodt.write_text(build_fodt(book, run_dir, sample=sample, date=date), encoding="utf-8")
asciidoc.write_text(
build_asciidoc(book, run_dir, target=target, sample=sample, date=date), encoding="utf-8"
)
fodt.write_text(build_fodt(book, run_dir, target=target, sample=sample, date=date), encoding="utf-8")
convert_with_uno(fodt, odt, pdf_path)
out = {"asciidoc": asciidoc, "fodt": fodt, "odt": odt}
if pdf_path:
@@ -545,17 +605,23 @@ def render(
# Enhance EPUB with cover image and metadata
epub_path = out_dir / f"{stem}.epub"
cover_path = book_path.parent / "covers" / "cover.png"
enhance_epub(epub_path, book, cover_path if cover_path.exists() else None)
enhance_epub(epub_path, book, cover_path if cover_path.exists() else None, target)
out["epub"] = epub_path
return out
def enhance_epub(epub_path: Path, book: dict, cover_path: Path | None = None) -> None:
def enhance_epub(
epub_path: Path,
book: dict,
cover_path: Path | None = None,
target: TargetLanguage | None = None,
) -> None:
"""Add cover image, ISBN, publisher, and translator metadata to EPUB.
Pandoc's ODT->EPUB conversion loses some metadata. This function post-processes
the EPUB to add cover image and complete metadata.
"""
target = target or CHINESE
import zipfile
import tempfile
import shutil
@@ -625,26 +691,41 @@ def enhance_epub(epub_path: Path, book: dict, cover_path: Path | None = None) ->
cover_meta.set("name", "cover")
cover_meta.text = cover_id
# Add CJK-optimized CSS
# Add the target language's typography to the stylesheet
css_path = tmpdir / "EPUB" / "styles" / "stylesheet1.css"
if css_path.exists():
css_content = css_path.read_text(encoding="utf-8")
cjk_css = '''
/* CJK typography for Chinese body text */
html {
font-family: "Noto Serif CJK SC", "Source Han Serif SC", "WenQuanYi Micro Hei", serif;
}
html[lang="zh-CN"], body[lang="zh-CN"] {
font-family: "Noto Serif CJK SC", "Source Han Serif SC", "WenQuanYi Micro Hei", serif;
lang = target.asciidoc_lang
font_stack = target.epub_font_stack
if target.cjk:
css_extra = f"""
/* CJK typography for {lang} body text */
html {{
font-family: {font_stack};
}}
html[lang="{lang}"], body[lang="{lang}"] {{
font-family: {font_stack};
line-break: strict;
word-break: normal;
}
p[lang="zh-CN"] {
font-family: "Noto Serif CJK SC", "Source Han Serif SC", "WenQuanYi Micro Hei", serif;
}}
p[lang="{lang}"] {{
font-family: {font_stack};
line-break: strict;
word-break: normal;
}
}}"""
else:
css_extra = f"""
/* Typography for {lang} body text */
html {{
font-family: {font_stack};
}}
html[lang="{lang}"], body[lang="{lang}"] {{
font-family: {font_stack};
}}
p[lang="{lang}"] {{
font-family: {font_stack};
}}"""
target_css = css_extra + '''
/* Dialogue styling */
.utterance__text {
font-size: 1.1em;
@@ -666,7 +747,7 @@ h1, h2 {
font-size: 0.8em;
}
'''
css_path.write_text(css_content + chr(10) + cjk_css, encoding="utf-8")
css_path.write_text(css_content + chr(10) + target_css, encoding="utf-8")
# Add publisher metadata
metadata = root.find("opf:metadata", ns)
@@ -677,6 +758,8 @@ h1, h2 {
translator = etree.SubElement(metadata, "{http://purl.org/dc/elements/1.1/}contributor")
translator.set("{http://www.idpf.org/2007/opf}role", "trl")
translator.text = "Umbra Tools Translation Team"
language = etree.SubElement(metadata, "{http://purl.org/dc/elements/1.1/}language")
language.text = target.asciidoc_lang
tree.write(str(opf_path), xml_declaration=True, encoding="UTF-8", standalone=True)
@@ -721,11 +804,13 @@ def write_manifest(
out_dir: Path,
sample: bool,
files: dict[str, Path],
target: TargetLanguage | None = None,
) -> Path:
"""Write the reproducibility manifest next to the rendered files.
It contains hashes, counts and model names, never the book text.
"""
target = target or CHINESE
book = json.loads(book_path.read_text(encoding="utf-8"))
chapters: list[dict] = []
total = translated = issues = scene_breaks = 0
@@ -746,7 +831,7 @@ def write_manifest(
"index": c["index"],
"matter": c["matter"],
"title": c["title"],
"title_zh": c["title_zh"],
"title_target": _title_of(c),
"paragraphs_expected": expected,
"paragraphs_translated": n,
"critic_issues": chapter_issues,
@@ -755,7 +840,9 @@ def write_manifest(
glossary = run_dir.parent / "glossary" / "glossary.json"
manifest = {
"title": book.get("title"),
"title_zh": book.get("title_zh"),
"title_target": _book_title(book),
"subtitle_target": _subtitle_of(book, target),
"target_language": target.code,
"author": book.get("author"),
"generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
"sample": sample,
@@ -788,14 +875,21 @@ def write_manifest(
def main() -> None:
ap = argparse.ArgumentParser(description="Render Deal Hunter Chinese AsciiDoc + ODT.")
ap = argparse.ArgumentParser(description="Render Deal Hunter AsciiDoc + ODT in the target language.")
ap.add_argument("--book", type=Path, required=True)
ap.add_argument("--run-dir", type=Path, required=True)
ap.add_argument("--out-dir", type=Path, required=True)
ap.add_argument(
"--target-language",
default=None,
help="the language the translations are in. Defaults to the book's own.",
)
ap.add_argument("--sample", action="store_true")
ap.add_argument("--date", default="")
ap.add_argument("--no-pdf", action="store_true")
args = ap.parse_args()
book = json.loads(args.book.read_text(encoding="utf-8"))
target_language = args.target_language or book.get("target_language") or "zh-Hans"
out = render(
book_path=args.book,
run_dir=args.run_dir,
@@ -803,6 +897,7 @@ def main() -> None:
sample=args.sample,
date=args.date,
pdf=not args.no_pdf,
target_language=target_language,
)
out["manifest"] = write_manifest(
book_path=args.book,
@@ -810,6 +905,7 @@ def main() -> None:
out_dir=args.out_dir,
sample=args.sample,
files=out,
target=resolve(target_language),
)
for k, v in out.items():
print(f"{k:9} {v}")

56
tools/translation/tests/test_align.py
View File

@@ -31,8 +31,13 @@ from align import ( # noqa: E402
repartition,
source_utterances,
split_target_sentences,
system_aligner,
verify_translation,
)
from languages import resolve # noqa: E402
CHINESE = resolve("zh-Hans")
SPANISH = resolve("es-ES")
def bundle_fixture() -> dict[str, dict]:
@@ -840,5 +845,56 @@ class DriftVerificationTests(unittest.TestCase):
self.assertEqual(report["needs_review"], 2)
class TargetLanguageTests(unittest.TestCase):
"""The alignment pass must work in the language it is aligning."""
def test_the_aligner_is_told_the_target_language(self) -> None:
self.assertIn("中文", system_aligner(CHINESE))
self.assertIn("西班牙语", system_aligner(SPANISH))
self.assertNotIn("中文", system_aligner(SPANISH))
def test_a_spanish_sentence_splits_on_the_spanish_terminators(self) -> None:
text = "La silla está sucia. ¿Who está ahí? ¡Ven!"
sentences = split_target_sentences(text, SPANISH)
self.assertEqual(len(sentences), 3)
self.assertTrue(sentences[0].endswith("sucia."))
self.assertTrue(sentences[1].endswith("ahí?"))
def test_a_chinese_sentence_splits_on_the_chinese_terminators(self) -> None:
text = "椅子很脏。谁在那里？快来！"
self.assertEqual(len(split_target_sentences(text, CHINESE)), 3)
def test_the_mechanical_split_uses_the_target_clause_marks(self) -> None:
"""A Spanish translation merges sentences with a comma, not a full stop."""
text = "La silla estaba sucia, y el suelo tenía polvo"
pieces = mechanical_split(text, 2, SPANISH)
self.assertIsNotNone(pieces)
assert pieces is not None
self.assertEqual(len(pieces), 2)
# The contract is whitespace-insensitive: the split never rewrites
# the translation, it only divides it.
self.assertEqual("".join(pieces).replace(" ", ""), text.replace(" ", ""))
def test_the_delivery_carries_the_base_language_code(self) -> None:
alignment = align_paragraph(
{
"id": "ch-1-p0000",
"type": "narration",
"text": "The ship was large.",
"source_indices": [0],
"source_ids": ["utt-100"],
},
"El barco era grande.",
bundle_fixture(),
)
delivery = build_delivery(
{"target_language": "es-ES"},
[({"id": "ch-1", "paragraphs": []}, [alignment])],
bundle_fixture(),
target=SPANISH,
)
self.assertEqual(delivery["target_language"], "es")
if __name__ == "__main__":
unittest.main()

158
tools/translation/tests/test_render.py Normal file
View File

@@ -0,0 +1,158 @@
# Copyright 2026 Umbra Tools (umbratools.com)
# SPDX-License-Identifier: Apache-2.0
"""Tests for the renderer.
The renderer writes the book the reader receives, so its typography is part
of the translation: the fonts, the language codes, the chapter-numbering
convention and the fixed editorial strings all have to be the target
language's. A Spanish edition typeset with Chinese fonts and Chinese chapter
headings is the same defect as a Chinese draft filed under a Spanish name,
one level down and much easier to miss.
"""
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
from languages import resolve # noqa: E402
from render import build_asciidoc, build_fodt, build_styles # noqa: E402
CHINESE = resolve("zh-Hans")
SPANISH = resolve("es-ES")
ARABIC = resolve("ar-SA")
def _book(target: str = "zh-Hans", subtitle: str = "交易猎人") -> dict:
return {
"title": "Deal Hunter",
"title_target": "Deal Hunter",
"subtitle_target": subtitle,
"target_language": target,
"chapters": [
{
"id": "ch-6",
"index": 6,
"row": 6,
"matter": "body",
"title": "Salvage",
"title_target": "Salvage",
"paragraphs": [
{"id": "ch-6-p0000", "type": "narration", "text": "The chair has"},
{"id": "ch-6-p0001", "type": "dialogue", "text": "Hello."},
],
},
],
}
def _run_dir(root: Path, text: str) -> Path:
chapter = root / "ch-6"
chapter.mkdir(parents=True, exist_ok=True)
(chapter / "translation.json").write_text(
json.dumps({"paragraphs": {"ch-6-p0000": text, "ch-6-p0001": text}}),
encoding="utf-8",
)
return root
class RenderTypographyTests(unittest.TestCase):
"""The renderer's typography is the target language's."""
def test_the_asciidoc_language_is_the_target_language(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "La silla estaba sucia.")
asciidoc = build_asciidoc(_book("es-ES", "Cazador de Ofertas"), run, target=SPANISH)
self.assertIn(":lang: es-ES", asciidoc)
self.assertNotIn("zh-CN", asciidoc)
def test_the_stylesheet_carries_the_target_language_codes(self) -> None:
styles = build_styles(SPANISH)
self.assertIn('fo:language="es"', styles)
self.assertIn('fo:country="ES"', styles)
self.assertIn('style:font-name="Noto Serif"', styles)
self.assertNotIn('fo:language="zh"', styles)
def test_an_rtl_target_gets_a_right_to_left_direction(self) -> None:
self.assertIn('fo:direction="rtl"', build_styles(ARABIC))
self.assertNotIn('fo:direction="rtl"', build_styles(SPANISH))
def test_the_chapter_heading_follows_the_target_convention(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "La silla estaba sucia.")
spanish = build_asciidoc(_book("es-ES"), run, target=SPANISH)
chinese = build_asciidoc(_book(), run, target=CHINESE)
self.assertIn("== Capítulo 1 Salvage", spanish)
self.assertIn("== 第一章　Salvage", chinese)
def test_a_latin_target_does_not_mark_its_text_as_english(self) -> None:
"""The Latin-run span exists so Chinese line-breaking leaves names alone.
For a Spanish paragraph the whole text is Latin, and marking it as
English would tell the renderer that the paragraph is English.
"""
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "La silla estaba sucia.")
spanish = build_fodt(_book("es-ES"), run, target=SPANISH)
chinese = build_fodt(_book(), run, target=CHINESE)
self.assertNotIn('text:style-name="Latin"', spanish)
self.assertIn('text:style-name="Latin"', chinese)
def test_the_fixed_strings_are_the_target_language_s(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "La silla estaba sucia.")
spanish = build_asciidoc(_book("es-ES"), run, target=SPANISH, sample=True)
chinese = build_asciidoc(_book(), run, target=CHINESE, sample=True)
self.assertIn("Muestra de revisión", spanish)
self.assertIn("Traducido por Umbra Tools", spanish)
self.assertNotIn("审阅样本", spanish)
self.assertIn("审阅样本", chinese)
def test_the_contents_heading_is_the_target_language_s(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "La silla estaba sucia.")
spanish = build_fodt(_book("es-ES"), run, target=SPANISH)
self.assertIn(">Índice</text:index-title-template>", spanish)
self.assertNotIn(">目录</text:index-title-template>", spanish)
def test_the_running_header_carries_the_target_subtitle(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "La silla estaba sucia.")
spanish = build_fodt(_book("es-ES", "Cazador de Ofertas"), run, target=SPANISH)
self.assertIn("Deal Hunter · Cazador de Ofertas", spanish)
def test_the_old_book_format_still_renders(self) -> None:
"""A book file written before the subtitle field existed must still work.
The old format stored the subtitle in the book-level ``title_zh``.
"""
book = _book()
del book["subtitle_target"]
book["title_zh"] = "交易猎人"
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "中间的椅子旁。")
asciidoc = build_asciidoc(book, run, target=CHINESE)
fodt = build_fodt(book, run, target=CHINESE)
self.assertIn("= Deal Hunter: 交易猎人", asciidoc)
self.assertIn("交易猎人", fodt)
def test_a_chapter_with_no_target_title_falls_back_to_the_english_one(self) -> None:
"""A missing translation is visible; borrowing another language's is not."""
book = _book("es-ES")
del book["chapters"][0]["title_target"]
with tempfile.TemporaryDirectory() as tmp:
run = _run_dir(Path(tmp), "La silla estaba sucia.")
asciidoc = build_asciidoc(book, run, target=SPANISH)
self.assertIn("Salvage", asciidoc)
self.assertNotIn("打捞", asciidoc)
if __name__ == "__main__":
unittest.main()

312
tools/translation/tests/test_translation.py
View File

@@ -10,17 +10,24 @@ residue detection, JSON extraction and the mechanical QA checks.
from __future__ import annotations
import json
import sys
import tempfile
import unittest
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
from llm import parse_json # noqa: E402
from pipeline import latin_residue, sanitize # noqa: E402
from prompts import critic_prompt, draft_prompt # noqa: E402
from qa import check_chapter # noqa: E402
from book import SUBTITLES, _title_for # noqa: E402
from languages import SCRIPT_LATIN, TargetLanguage, by_code, resolve # noqa: E402
from llm import LLMError, parse_json # noqa: E402
from pipeline import Runner, latin_residue, sanitize # noqa: E402
from prompts import critic_prompt, draft_prompt, glossary_block, system_critic, system_drafter # noqa: E402
from qa import check_chapter, forbidden_titles # noqa: E402
CHINESE = resolve("zh-Hans")
SPANISH = resolve("es-ES")
class PromptInstructionTests(unittest.TestCase):
@@ -34,24 +41,26 @@ class PromptInstructionTests(unittest.TestCase):
``--verify`` stage.
"""
def _draft(self) -> str:
def _draft(self, target: TargetLanguage | None = None) -> str:
return draft_prompt(
chapter_title="Salvage",
scene_index=0,
chunk_index=0,
chunks_in_scene=1,
paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}],
analysis={"summary_zh": "摘要"},
analysis={"summary": "摘要"},
previous_summary="",
glossary={},
target=target or CHINESE,
)
def _critic(self) -> str:
def _critic(self, target: TargetLanguage | None = None) -> str:
return critic_prompt(
chapter_title="Salvage",
paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}],
draft={"ch-6-p0000": "中间的椅子旁"},
glossary={},
target=target or CHINESE,
)
def test_the_drafter_is_told_to_keep_each_id_to_its_own_english(self) -> None:
@@ -73,6 +82,249 @@ class PromptInstructionTests(unittest.TestCase):
self.assertIn("段落错位", prompt)
class TargetLanguageTests(unittest.TestCase):
"""The pipeline must produce the language it was told to produce.
Measured on oldbox: the production directory ``deal-hunter-es`` declared
``target_language: es-ES`` and held 3,038 paragraphs of Chinese, because
the drafter, the critic and the glossary lookup were all hard-coded to
Simplified Chinese. These tests pin the parameterisation that fixes it.
"""
def test_a_language_is_found_by_tag_base_or_name(self) -> None:
self.assertEqual(by_code("es-ES").base, "es")
self.assertEqual(by_code("es").base, "es")
self.assertEqual(by_code("es_ES").base, "es")
self.assertEqual(by_code("Spanish").base, "es")
self.assertEqual(by_code("简体中文").base, "zh")
def test_an_unknown_language_falls_back_loudly(self) -> None:
# The fallback prints a warning; the point is that it does not
# silently pretend the language was recognised.
self.assertEqual(resolve("xx-YY").base, "zh")
def test_the_drafter_is_told_the_target_language(self) -> None:
prompt = draft_prompt(
chapter_title="Salvage",
scene_index=0,
chunk_index=0,
chunks_in_scene=1,
paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}],
analysis={"summary": "摘要"},
previous_summary="",
glossary={},
target=SPANISH,
)
self.assertIn("西班牙语", prompt)
self.assertIn("español", prompt)
self.assertNotIn("简体中文", prompt)
def test_the_drafter_asks_for_the_target_language_key(self) -> None:
prompt = draft_prompt(
chapter_title="Salvage",
scene_index=0,
chunk_index=0,
chunks_in_scene=1,
paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}],
analysis={"summary": "摘要"},
previous_summary="",
glossary={},
target=SPANISH,
)
self.assertIn('"es":', prompt)
self.assertNotIn('"zh":', prompt)
def test_the_critic_is_told_the_target_language(self) -> None:
prompt = critic_prompt(
chapter_title="Salvage",
paragraphs=[{"id": "ch-6-p0000", "speaker": "narrator", "text": "The chair has"}],
draft={"ch-6-p0000": "la silla"},
glossary={},
target=SPANISH,
)
self.assertIn("西班牙语", prompt)
self.assertNotIn("中文译文", prompt)
def test_the_glossary_reads_the_target_language_key(self) -> None:
glossary = {
"title": {"en": "Deal Hunter"},
"characters": {"Kainda": {"zh": "凯因达", "es": "Kainda"}},
"terms": {"bounty": {"zh": "赏金", "es": "recompensa"}},
}
spanish = glossary_block(glossary, set(), "bounty", SPANISH)
self.assertIn("recompensa", spanish)
self.assertNotIn("赏金", spanish)
def test_a_glossary_value_for_another_language_is_not_borrowed(self) -> None:
"""A Chinese-only glossary must not leak Chinese into a Spanish run."""
glossary = {
"characters": {"Kainda": {"zh": "凯因达"}},
"terms": {},
}
spanish = glossary_block(glossary, set(), "", SPANISH)
self.assertNotIn("凯因达", spanish)
def test_the_chinese_rules_are_kept_for_chinese(self) -> None:
"""The existing production must not lose its hard rules."""
glossary = {"characters": {}, "terms": {}, "places": {}}
chinese = glossary_block(glossary, set(), "jump", CHINESE)
self.assertIn("恒星系 system / star system = 星系", chinese)
self.assertIn("不译成中文", chinese)
def test_every_target_gets_the_books_own_rules(self) -> None:
"""The rules about this book are not Chinese's and not Spanish's.
*Deal Hunter* is the ship's name, the AI's name and the book's title,
so it stays in English in every edition. The Chinese table carries
its own wording of this rule, so a language with no table must not
lose it.
"""
glossary = {"characters": {}, "terms": {}, "places": {}}
for target in (CHINESE, SPANISH):
block = glossary_block(glossary, set(), "", target)
self.assertIn("Deal Hunter", block)
self.assertIn("保留英文", block)
self.assertIn("Hunter 作为 AI 的简称保留英文", block)
self.assertIn("Umbra Audio 保留英文", block)
def test_a_language_without_a_table_gets_no_other_languages_terms(self) -> None:
"""A Spanish run must not inherit the Chinese hard vocabulary."""
glossary = {"characters": {}, "terms": {}, "places": {}}
spanish = glossary_block(glossary, set(), "jump", SPANISH)
self.assertNotIn("星系", spanish)
self.assertNotIn("跃迁", spanish)
def test_a_glossary_can_supply_its_own_rules(self) -> None:
glossary = {
"characters": {},
"terms": {},
"places": {},
"rules": ["- Deal Hunter se conserva en inglés"],
}
spanish = glossary_block(glossary, set(), "", SPANISH)
self.assertIn("Deal Hunter se conserva en inglés", spanish)
def test_the_system_prompts_name_the_target_language(self) -> None:
self.assertIn("简体中文", system_drafter(CHINESE))
self.assertIn("西班牙语", system_drafter(SPANISH))
self.assertIn("Spanish", system_critic(SPANISH))
self.assertNotIn("中文译文", system_critic(SPANISH))
def test_quotes_are_normalised_to_the_target_language(self) -> None:
self.assertEqual(sanitize('"hola"', SPANISH), "“hola”")
self.assertEqual(sanitize('"hallo"', resolve("de-DE")), "„hallo“")
self.assertEqual(sanitize('"bonjour"', resolve("fr-FR")), "«bonjour»")
def test_the_ellipsis_is_the_target_language_s(self) -> None:
self.assertEqual(sanitize("...", CHINESE), "……")
self.assertEqual(sanitize("...", SPANISH), "…")
def test_the_residue_check_is_only_meaningful_for_a_non_latin_target(self) -> None:
"""A Spanish draft is full of Latin words; they are not residue."""
self.assertIn("silla", latin_residue("la silla del capitán")[0])
self.assertEqual(latin_residue("中间那把椅子")[0], [])
# So the repair is skipped for a Latin-script target: the check would
# otherwise flag the whole Spanish draft and "repair" it into
# nonsense.
self.assertIn(SCRIPT_LATIN, SPANISH.scripts)
self.assertNotIn(SCRIPT_LATIN, CHINESE.scripts)
class RunDirectoryGuardTests(unittest.TestCase):
"""A run directory belongs to one language.
Measured on oldbox: the production directory ``deal-hunter-es`` declared
``target_language: es-ES`` and held 3,038 paragraphs of Chinese, because
the pipeline caches by file existence and the run directory is chosen by
the caller. A Spanish run pointed at that directory would have skipped
drafting entirely and published the Chinese under a Spanish name.
The guard refuses that. It checks the manifest first (a claim) and then
measures the text that is actually there (the evidence).
"""
def _runner(self, run_dir: Path, target: str, **kwargs: object) -> Runner:
book = run_dir / "book.json"
glossary = run_dir / "glossary.json"
book.write_text(
json.dumps({"target_language": target, "chapters": []}),
encoding="utf-8",
)
glossary.write_text(json.dumps({}), encoding="utf-8")
return Runner(
book_path=book,
glossary_path=glossary,
run_dir=run_dir / "runs",
target_language=target,
**kwargs,
)
def test_a_run_directory_records_its_language(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
self._runner(root, "es-ES")
manifest = json.loads((root / "runs" / "run.json").read_text(encoding="utf-8"))
self.assertEqual(manifest["target_language"], "es-ES")
self.assertEqual(manifest["glossary_key"], "es")
def test_a_spanish_run_refuses_a_chinese_run_directory(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
self._runner(root, "zh-Hans")
with self.assertRaises(SystemExit) as caught:
self._runner(root, "es-ES")
self.assertIn("zh-Hans", str(caught.exception))
self.assertIn("es-ES", str(caught.exception))
def test_a_spanish_run_refuses_a_directory_holding_chinese_text(self) -> None:
"""No manifest - an older directory - so the text itself is measured."""
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
self._runner(root, "zh-Hans")
chapter = root / "runs" / "ch-1"
chapter.mkdir(parents=True, exist_ok=True)
(chapter / "translation.json").write_text(
json.dumps({"paragraphs": {"ch-1-p0000": "凯因达走到门前。"}}),
encoding="utf-8",
)
(root / "runs" / "run.json").unlink()
with self.assertRaises(SystemExit) as caught:
self._runner(root, "es-ES")
self.assertIn("not Spanish", str(caught.exception))
def test_a_spanish_run_accepts_a_spanish_run_directory(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
self._runner(root, "es-ES")
chapter = root / "runs" / "ch-1"
chapter.mkdir(parents=True, exist_ok=True)
(chapter / "translation.json").write_text(
json.dumps({"paragraphs": {"ch-1-p0000": "Kainda se acercó a la puerta."}}),
encoding="utf-8",
)
runner = self._runner(root, "es-ES")
self.assertEqual(runner.target.code, "es-ES")
def test_force_overrides_the_manifest(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
self._runner(root, "zh-Hans")
runner = self._runner(root, "es-ES", force=True)
self.assertEqual(runner.target.code, "es-ES")
def test_a_draft_in_the_wrong_language_is_refused(self) -> None:
"""The drafter has been known to ignore the target language."""
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
runner = self._runner(root, "es-ES")
with self.assertRaises(LLMError) as caught:
runner._normalise_draft(
{"paragraphs": [{"id": "ch-1-p0000", "es": "凯因达走到门前。"}]},
["ch-1-p0000"],
)
self.assertIn("wrong language", str(caught.exception))
class SanitizeTests(unittest.TestCase):
def test_strips_dialogue_label(self) -> None:
self.assertEqual(sanitize("（对话；凯因达）“我在游艇上。”"), "“我在游艇上。”")
@@ -184,5 +436,51 @@ class QaTests(unittest.TestCase):
self.assertNotIn("system_mistranslated", {i["kind"] for i in issues})
class ChapterTitleTests(unittest.TestCase):
"""The chapter title is the target language's, or nothing else's."""
def test_a_language_with_no_table_falls_back_to_the_english_title(self) -> None:
"""Never to another language's title."""
self.assertEqual(_title_for("Salvage", SPANISH), "Salvage")
self.assertEqual(_title_for("Salvage", resolve("ja-JP")), "Salvage")
def test_a_language_with_a_table_uses_it(self) -> None:
self.assertEqual(_title_for("Salvage", CHINESE), "打捞")
def test_a_chapter_missing_from_the_table_is_marked_in_the_target_language(self) -> None:
"""A missing heading is a visible gap, not an English title that looks done."""
chinese = _title_for("Brand New Chapter", CHINESE)
self.assertIn("待译", chinese)
self.assertIn("Brand New Chapter", chinese)
def test_the_subtitle_is_the_target_language_s(self) -> None:
self.assertEqual(SUBTITLES["zh"], "交易猎人")
self.assertEqual(SUBTITLES["es"], "Cazador de Ofertas")
# A language with no subtitle gets none, not another language's.
self.assertNotIn("en", SUBTITLES)
class ForbiddenTitleTests(unittest.TestCase):
"""The translated title must not appear in the text, in any language."""
def test_the_chinese_form_is_forbidden(self) -> None:
self.assertIn("《交易猎人》", forbidden_titles(CHINESE))
def test_the_spanish_form_is_forbidden(self) -> None:
self.assertIn("«Cazador de Ofertas»", forbidden_titles(SPANISH))
def test_a_language_without_a_subtitle_forbids_nothing(self) -> None:
"""A language with no translated title has no forbidden form."""
language = TargetLanguage(
code="xx-XX",
base="xx",
name="Example",
native="example",
chinese_name="示例语",
scripts=(SCRIPT_LATIN,),
)
self.assertEqual(forbidden_titles(language), [])
if __name__ == "__main__":
unittest.main()

103
umbratrans_web/project_status.py
View File

@@ -11,11 +11,20 @@ For each program, the adapter returns every configured target language plus the
English source row. Languages without a production directory are reported as
``not_started``; languages with a production directory report real chapter
progress from the runs directory.
The chapter count is a claim about the *files*, not about the *text*. A
directory full of Chinese can sit in a directory named for Spanish and be
reported as a complete Spanish translation - measured on oldbox, that is
exactly what ``deal-hunter-es`` was. So the text is measured too, and a
production whose text is not in the language it claims is reported as
``mismatched`` rather than ``complete``.
"""
from __future__ import annotations
import importlib.util
import json
import sys
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
@@ -75,6 +84,60 @@ def _max_mtime(paths: list[Path]) -> str | None:
return datetime.fromtimestamp(max(mtimes)).isoformat()
# The translation language registry, loaded once. It lives in
# ``tools/translation/languages.py``, which is not a distributed package, so
# it is loaded from the repository by path. It is the same file the pipeline
# uses, so the dashboard cannot disagree with it about which script a
# language is written in.
_LANGUAGE_REGISTRY: Any | None = None
_LANGUAGE_REGISTRY_TRIED = False
def _language_registry() -> Any | None:
"""The translation language registry, or None if it cannot be loaded."""
global _LANGUAGE_REGISTRY, _LANGUAGE_REGISTRY_TRIED
if _LANGUAGE_REGISTRY_TRIED:
return _LANGUAGE_REGISTRY
_LANGUAGE_REGISTRY_TRIED = True
path = Path(__file__).resolve().parents[1] / "tools" / "translation" / "languages.py"
try:
if path.is_file():
spec = importlib.util.spec_from_file_location("umbratrans_languages", path)
if spec and spec.loader:
module = importlib.util.module_from_spec(spec)
# The module has to be in sys.modules before it is executed:
# @dataclass resolves its own module through sys.modules.
sys.modules[spec.name] = module
spec.loader.exec_module(module)
_LANGUAGE_REGISTRY = module
return module
from tools.translation import languages # type: ignore[import-not-found]
_LANGUAGE_REGISTRY = languages
return languages
except Exception:
return None
def _language_problem(runs_dir: Path | None, tag: str) -> str | None:
"""Why the text in a production is not in the language it claims.
Returns None when the text is in the declared language. When the check
cannot be performed at all it says so, rather than passing silently: a
production that cannot be verified is not one that can be reported
complete.
"""
if runs_dir is None or not runs_dir.is_dir():
return None
registry = _language_registry()
if registry is None:
return "the language registry could not be loaded, so the text could not be checked"
try:
return registry.check_runs_language(runs_dir, tag)
except Exception as exc: # pragma: no cover - defensive
return f"the language could not be checked: {exc}"
def _count_translated_chapters(runs_dir: Path) -> tuple[int, int]:
"""Return (done, total) chapter counts from the runs directory."""
done = 0
@@ -119,8 +182,6 @@ class ProjectStatusAdapter:
prod_name = _SLUG_TO_PROD.get(getattr(project, "slug", ""))
prod_dir = self.productions_root / prod_name if prod_name else None
runs_dir = prod_dir / "runs" if prod_dir else None
out_progress = prod_dir / "output-progress" if prod_dir else None
out_final = prod_dir / "output" if prod_dir else None
# Chapter totals come from the source book when available; fall back
# to the number of chapter directories found in runs.
@@ -135,7 +196,12 @@ class ProjectStatusAdapter:
"language_label": "English (source)",
"role": "source",
"source_present": True,
"translation": {"state": "complete", "chapters_done": total, "chapters_total": total},
"translation": {
"state": "complete",
"chapters_done": total,
"chapters_total": total,
"language_problem": None,
},
"audio": {"state": "unknown"},
"outputs": {},
"review": {"state": "ok"},
@@ -149,9 +215,10 @@ class ProjectStatusAdapter:
continue # source row already added above
label, role = target_langs[tag]
if prod_dir and prod_dir.is_dir():
# A language-specific production directory would be named
# deal-hunter-<tag> (e.g. deal-hunter-es for Spanish).
# deal-hunter-<tag> (e.g. deal-hunter-es for Spanish). It is
# looked up on its own: a Spanish production exists or it does
# not, regardless of whether the Chinese one is there.
lang_prod_name = f"deal-hunter-{tag.split('-')[0]}"
lang_prod_dir = self.productions_root / lang_prod_name
lang_runs = lang_prod_dir / "runs" if lang_prod_dir.is_dir() else None
@@ -165,24 +232,23 @@ class ProjectStatusAdapter:
if lang_total and lang_done >= lang_total
else ("in_progress" if lang_done else "not_started")
)
outputs = _read_outputs(
lang_out_progress or Path(), lang_out_final or Path()
)
outputs = _read_outputs(lang_out_progress or Path(), lang_out_final or Path())
updated_at = _max_mtime(
[lang_runs / f"ch-{i}" / "translation.json" for i in range(1, 40)]
if lang_runs.is_dir()
else []
[lang_runs / f"ch-{i}" / "translation.json" for i in range(1, 40)] if lang_runs.is_dir() else []
)
else:
state = "not_started"
lang_done, lang_total = 0, 0
outputs = {}
updated_at = None
else:
state = "not_started"
lang_done, lang_total = 0, 0
outputs = {}
updated_at = None
# The chapter count says the files are there. The script says
# which language they are in. A production that declares one
# language and holds another is not complete in the language
# it declares.
language_problem = _language_problem(lang_runs, tag)
if language_problem:
state = "mismatched"
languages.append(
{
@@ -194,12 +260,11 @@ class ProjectStatusAdapter:
"state": state,
"chapters_done": lang_done,
"chapters_total": lang_total,
"language_problem": language_problem,
},
"audio": {"state": "not_started"},
"outputs": outputs,
"review": {
"state": "needs_review" if state == "complete" else "pending"
},
"review": {"state": "needs_review" if state in {"complete", "mismatched"} else "pending"},
"updated_at": updated_at,
}
)

96
umbratrans_web/tests/test_project_status.py Normal file
View File

@@ -0,0 +1,96 @@
# Copyright 2026 Umbra Tools (umbratools.com)
# SPDX-License-Identifier: Apache-2.0
"""Tests for the project-status adapter.
The adapter reports a language's state from the productions directory. The
chapter count is a claim about the *files*; the script check is a measurement
of the *text*. Measured on oldbox, ``deal-hunter-es`` held 3,038 paragraphs
of Chinese and was reported as a complete Spanish translation, so the
measurement is the part that matters.
"""
from __future__ import annotations
import json
from pathlib import Path
from types import SimpleNamespace
from umbratrans_web.project_status import ProjectStatusAdapter
PROJECT = SimpleNamespace(slug="dealhunter-trans", id="11111111-1111-1111-1111-111111111111")
def _production(root: Path, tag: str, chapters: dict[str, str]) -> Path:
"""Write a production directory with the given chapters of text."""
prod = root / f"deal-hunter-{tag}"
for chapter_id, text in chapters.items():
directory = prod / "runs" / chapter_id
directory.mkdir(parents=True, exist_ok=True)
(directory / "translation.json").write_text(
json.dumps({"paragraphs": {"ch-1-p0000": text}}, ensure_ascii=False),
encoding="utf-8",
)
return prod
def _row(adapter: ProjectStatusAdapter, tag: str) -> dict:
status = adapter.status_for(PROJECT)
row = next(r for r in status["languages"] if r["language_tag"] == tag)
return row["translation"]
def test_a_production_in_its_own_language_is_complete(tmp_path: Path) -> None:
_production(tmp_path, "zh", {"ch-1": "凯因达走到门前。"})
_production(tmp_path, "fr", {"ch-1": "Kainda s'approcha de la porte."})
adapter = ProjectStatusAdapter(tmp_path)
assert _row(adapter, "zh-Hans")["state"] == "complete"
assert _row(adapter, "fr-FR")["state"] == "complete"
assert _row(adapter, "fr-FR")["language_problem"] is None
def test_a_mislabelled_production_is_not_reported_complete(tmp_path: Path) -> None:
"""The defect this check exists for: Chinese filed under a Spanish name."""
_production(tmp_path, "es", {"ch-1": "凯因达走到门前。", "ch-2": "门是关着的。"})
adapter = ProjectStatusAdapter(tmp_path)
row = _row(adapter, "es-ES")
assert row["state"] == "mismatched"
assert row["chapters_done"] == 2
assert "not Spanish" in row["language_problem"]
def test_a_production_with_no_runs_is_not_started(tmp_path: Path) -> None:
adapter = ProjectStatusAdapter(tmp_path)
row = _row(adapter, "de-DE")
assert row["state"] == "not_started"
assert row["language_problem"] is None
def test_a_mislabelled_production_needs_review(tmp_path: Path) -> None:
_production(tmp_path, "es", {"ch-1": "凯因达走到门前。"})
adapter = ProjectStatusAdapter(tmp_path)
status = adapter.status_for(PROJECT)
row = next(r for r in status["languages"] if r["language_tag"] == "es-ES")
assert row["review"]["state"] == "needs_review"
def test_a_language_with_no_table_of_its_own_is_not_borrowed(tmp_path: Path) -> None:
"""A Chinese production must not be accepted as Japanese.
Japanese is written in Han characters *and* kana. A text with no kana at
all is Chinese, however many Han characters it has.
"""
_production(tmp_path, "ja", {"ch-1": "凯因达走到门前。"})
adapter = ProjectStatusAdapter(tmp_path)
row = _row(adapter, "ja-JP")
assert row["state"] == "mismatched"
assert "kana" in row["language_problem"]
def test_every_configured_language_is_reported(tmp_path: Path) -> None:
adapter = ProjectStatusAdapter(tmp_path)
status = adapter.status_for(PROJECT)
tags = {r["language_tag"] for r in status["languages"]}
assert {"en", "zh-Hans", "es-ES", "fr-FR"} <= tags

1
web/assets/styles.css
View File

@@ -51,6 +51,7 @@ button:focus-visible, a:focus-visible, input:focus-visible, select:focus-visible
.status-pill--info { color: #bfdbfe; background: #1e3a5f; }
.status-pill--success { color: #a7f3d0; background: #065f46; }
.status-pill--warning { color: #fde68a; background: #78350f; }
.status-pill--danger { color: #fecaca; background: #7f1d1d; }
.status-pill--muted { color: #cbd5e1; background: #334155; }
.status-dot { display: inline-block; width: .55rem; height: .55rem; border-radius: 50%; }
.status-dot--good { background: #34d399; box-shadow: 0 0 0 .25rem rgba(52, 211, 153, .12); }

13
web/assets/umbra-app.js
View File

@@ -292,6 +292,7 @@
if (state === 'complete') return { text: 'Text translated', tone: 'success' };
if (state === 'in_progress') return { text: 'In progress', tone: 'info' };
if (state === 'needs_review') return { text: 'Needs review', tone: 'warning' };
if (state === 'mismatched') return { text: 'Wrong language', tone: 'danger' };
return { text: 'Not started', tone: 'muted' };
}
@@ -458,14 +459,20 @@
chapters = 'Complete';
}
// A production whose text is not in the language it declares says why,
// rather than just looking unfinished.
var problem = tr.language_problem || '';
var badgeNode = el('span', { class: 'status-pill status-pill--' + badge.tone, text: badge.text });
if (problem) {
badgeNode.setAttribute('title', problem);
}
return el('div', { class: 'language-row' + (code === state.languageCode ? ' language-row--selected' : ''), role: 'button', tabIndex: '0' }, [
el('div', { class: 'language-row__label' }, [
el('strong', { text: label }),
el('span', { class: 'language-row__role', text: isSource ? 'Source' : chapters })
]),
el('div', { class: 'language-row__status' }, [
el('span', { class: 'status-pill status-pill--' + badge.tone, text: badge.text })
]),
el('div', { class: 'language-row__status' }, [ badgeNode ]),
el('div', { class: 'language-row__actions' }, languageActions(program, code, status))
]);
}