3f05489e25
The pipeline was hard-coded to Simplified Chinese in three places: the drafter was told to translate into Chinese, the critic was told to compare English with Chinese, and the glossary's translated forms were read from a "zh" key. A production that declared target_language: es-ES was therefore translated into Chinese and filed under a Spanish name - measured on oldbox, deal-hunter-es held 3,038 paragraphs of Chinese and was reported complete. A new registry, tools/translation/languages.py, is the single source of truth for what a target language is: its names, its script, its punctuation, its glossary key, its typography and its chapter-numbering convention. Ten languages are registered. The prompt scaffolding stays in Chinese - that is what the models read best - and the target is named with its Chinese, English and native names so there is no room to misread it. The registry now reaches every stage: - prompts.py: the analyst, drafter, critic and glossary blocks take the target. The hard rules about this book - Deal Hunter is the ship's name, the AI's name and the book's title, so it stays in English - are the language's own table for Chinese and the global rules for everything else, so a language with no table does not lose them. The Chinese rules are byte-for-byte what they were. - pipeline.py: --target-language, a run-directory guard that refuses to reuse a directory belonging to another language (from the manifest, or by measuring the text it already holds), a draft-keyed-by-target reader, and a refusal of a draft that came back in the wrong script. - book.py: per-language chapter-title and subtitle tables. A language with no table falls back to the English title, never to another language's; a language with a table and a missing chapter gets a visible marker in its own language. - qa.py: the target's quote marks, punctuation, script count and forbidden title forms. The Latin-residue check is skipped for a Latin target, where it would flag the whole book. - align.py: the aligner and verifier prompts name the target, and the sentence and clause splitting uses the target's own terminators. - render.py: the fonts, the language and country codes, the writing direction, the chapter headings and the fixed editorial strings around the translation are the target's. The Latin-run span is skipped for a Latin target, where marking the text as English would tell the renderer it is English. - project_status.py: the dashboard measures a sample of a production's text and reports a production whose text is not in the language it declares as "mismatched", not "complete". A chapter count is a claim about the files; the script check is a measurement of the text. Also fixes: the dashboard gated every language's production on the source production existing, so a Spanish production could never be reported even when it was there. The Chinese render is byte-identical to before on the real book, and the Chinese rules are unchanged. Suite: 620 passed, the same 68 pre-existing environment failures, +49 new tests.
905 lines
37 KiB
Python
905 lines
37 KiB
Python
# Copyright 2026 Umbra Tools (umbratools.com)
|
||
# SPDX-License-Identifier: Apache-2.0
|
||
|
||
"""The target languages this pipeline can translate into.
|
||
|
||
The pipeline was written for one language and hard-coded to it: the drafter
|
||
was told to translate into Simplified Chinese, the critic was told to compare
|
||
English with Chinese, the glossary's translated forms were read from a ``zh``
|
||
key, the chapter titles came from a ``TITLE_ZH`` table, and the renderer
|
||
typeset CJK. A production directory that declared ``target_language: es-ES``
|
||
was therefore translated into Chinese and filed under a Spanish name.
|
||
|
||
This module is the single place that says what a target language *is*: its
|
||
name, its script, its punctuation, its typography, and the key its
|
||
translated forms live under in a glossary. Everything else reads it, so a
|
||
second language is a data change rather than a code change.
|
||
|
||
It also carries the script check. A production's ``target_language`` field is
|
||
a claim, and a claim has to be measured: the text in the production directory
|
||
is either written in the declared language or it is not, and the dashboard
|
||
must not report a Chinese draft under a Spanish name as a completed
|
||
Spanish translation.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
from dataclasses import dataclass
|
||
from pathlib import Path
|
||
|
||
# --- scripts ---------------------------------------------------------------
|
||
|
||
SCRIPT_LATIN = "latin"
|
||
SCRIPT_CYRILLIC = "cyrillic"
|
||
SCRIPT_HAN = "han"
|
||
SCRIPT_KANA = "kana"
|
||
SCRIPT_HANGUL = "hangul"
|
||
SCRIPT_ARABIC = "arabic"
|
||
|
||
SCRIPT_NAMES: dict[str, str] = {
|
||
SCRIPT_LATIN: "Latin script",
|
||
SCRIPT_CYRILLIC: "Cyrillic script",
|
||
SCRIPT_HAN: "Han characters",
|
||
SCRIPT_KANA: "Japanese kana",
|
||
SCRIPT_HANGUL: "Hangul",
|
||
SCRIPT_ARABIC: "Arabic script",
|
||
}
|
||
|
||
_SCRIPT_RANGES: tuple[tuple[str, int, int], ...] = (
|
||
(SCRIPT_LATIN, 0x0041, 0x005A),
|
||
(SCRIPT_LATIN, 0x0061, 0x007A),
|
||
(SCRIPT_LATIN, 0x00C0, 0x024F),
|
||
(SCRIPT_CYRILLIC, 0x0400, 0x04FF),
|
||
(SCRIPT_HAN, 0x3400, 0x4DBF),
|
||
(SCRIPT_HAN, 0x4E00, 0x9FFF),
|
||
(SCRIPT_HAN, 0xF900, 0xFAFF),
|
||
(SCRIPT_KANA, 0x3040, 0x309F),
|
||
(SCRIPT_KANA, 0x30A0, 0x30FF),
|
||
(SCRIPT_KANA, 0x31F0, 0x31FF),
|
||
(SCRIPT_HANGUL, 0x1100, 0x11FF),
|
||
(SCRIPT_HANGUL, 0x3130, 0x318F),
|
||
(SCRIPT_HANGUL, 0xAC00, 0xD7AF),
|
||
(SCRIPT_ARABIC, 0x0600, 0x06FF),
|
||
(SCRIPT_ARABIC, 0x0750, 0x077F),
|
||
)
|
||
|
||
|
||
# --- the registry ----------------------------------------------------------
|
||
|
||
|
||
@dataclass(frozen=True)
|
||
class TargetLanguage:
|
||
"""One language the pipeline can translate into.
|
||
|
||
``code`` is the BCP-47 tag the production declares. ``base`` is the
|
||
two-letter code the delivery uses, because the downstream ingest folds
|
||
locale codes to their base (``zh-Hans`` becomes ``zh``).
|
||
"""
|
||
|
||
code: str
|
||
base: str
|
||
name: str
|
||
native: str
|
||
# The language's name in Chinese. The prompts are written in Chinese, so
|
||
# "译成西班牙语" reads as a Chinese instruction while "译成Spanish" reads
|
||
# as a typo; the English and native names ride along so the model has no
|
||
# room to misread which language is meant.
|
||
chinese_name: str
|
||
scripts: tuple[str, ...]
|
||
# A script that has to be present for the text to be this language at all:
|
||
# Japanese without kana is Chinese, and Korean without Hangul is Chinese.
|
||
required_script: str | None = None
|
||
# Scripts that would mean the delivery is a different language.
|
||
forbidden_scripts: tuple[str, ...] = ()
|
||
# The key a glossary uses for the translated form of a name or term.
|
||
glossary_key: str = ""
|
||
# Typography.
|
||
quote_open: str = '"'
|
||
quote_close: str = '"'
|
||
ellipsis: str = "..."
|
||
dash: str = "—"
|
||
paren_open: str = "("
|
||
paren_close: str = ")"
|
||
title_open: str = '"'
|
||
title_close: str = '"'
|
||
font_serif: str = "Noto Serif"
|
||
font_sans: str = "Noto Sans"
|
||
# The font stack used for the EPUB stylesheet, which has to name real
|
||
# installed families rather than the two the FODT declares.
|
||
epub_font_stack: str = "Noto Serif, serif"
|
||
# The language tag for the AsciiDoc header (``:lang:``) and for the
|
||
# EPUB's ``lang`` attribute.
|
||
asciidoc_lang: str = "en"
|
||
# The language and country codes ODF wants (``fo:language``,
|
||
# ``fo:country``). They are separate from the BCP-47 tag: ODF wants
|
||
# ``zh`` and ``CN``, not ``zh-Hans``.
|
||
odf_language: str = "en"
|
||
odf_country: str = "US"
|
||
# Chapter numbering: "第 N 章" is a CJK convention, not a universal one.
|
||
chapter_prefix: str = "Chapter "
|
||
# The character that closes a chapter number in a CJK language: 章 in
|
||
# Chinese and Japanese, 장 in Korean.
|
||
chapter_suffix: str = ""
|
||
# "arabic" (1, 2, 3) or "chinese" (一, 二, 三). Japanese uses Arabic
|
||
# numerals with the 第 prefix; Chinese uses Chinese numerals.
|
||
chapter_numeral: str = "arabic"
|
||
rtl: bool = False
|
||
# The characters that end a sentence in this language.
|
||
sentence_terminators: str = ".!?…"
|
||
# The characters that can close a sentence after its terminator: the
|
||
# target's quotation marks and brackets.
|
||
closing_marks: str = "\"'”’)]}»"
|
||
# The clause boundaries the mechanical fallback splits at, strongest
|
||
# first. A translation merges English sentences with these rather than
|
||
# with a full stop, so they are where a paragraph can be divided.
|
||
clause_marks: tuple[str, ...] = ("—", "–", ":", ";", ",")
|
||
# True for the languages whose line-breaking and punctuation rules differ
|
||
# enough from Latin that the renderer needs to be told.
|
||
cjk: bool = False
|
||
|
||
@property
|
||
def prompt_name(self) -> str:
|
||
"""How to name this language inside a prompt."""
|
||
return f"{self.chinese_name}（{self.name} / {self.native}）"
|
||
|
||
@property
|
||
def label(self) -> str:
|
||
"""A short label for a document header."""
|
||
return f"{self.name} ({self.code})"
|
||
|
||
def chapter_number(self, n: int) -> str:
|
||
"""The chapter number as this language writes it."""
|
||
if self.chapter_numeral == "chinese":
|
||
return _chinese_numeral(n)
|
||
return str(n)
|
||
|
||
def chapter_heading(self, n: int, title: str) -> str:
|
||
"""The full chapter heading, in this language's convention.
|
||
|
||
Chinese writes 第一章 打捞; Spanish writes Capítulo 1 Salvage. The
|
||
convention is the language's, not the renderer's.
|
||
"""
|
||
if self.cjk:
|
||
return f"{self.chapter_prefix}{self.chapter_number(n)}{self.chapter_suffix} {title}"
|
||
return f"{self.chapter_prefix}{self.chapter_number(n)} {title}"
|
||
|
||
def string(self, key: str, default: str = "") -> str:
|
||
"""A fixed editorial string in this language.
|
||
|
||
The strings the renderer writes around the translated text - the
|
||
contents heading, the review-sample notice, the translator credit -
|
||
are part of the book, so they cannot be left in Chinese when the book
|
||
is not Chinese. A language without a table gets the English string,
|
||
never another language's.
|
||
"""
|
||
table = RENDER_STRINGS.get(self.base) or RENDER_STRINGS["en"]
|
||
return table.get(key) or default
|
||
|
||
|
||
_CN_DIGITS = "零一二三四五六七八九"
|
||
|
||
|
||
def _chinese_numeral(n: int) -> str:
|
||
"""The Chinese numeral for *n*: 1 is 一, 10 is 十, 21 is 二十一."""
|
||
if n <= 0:
|
||
return str(n)
|
||
if n <= 10:
|
||
return "十" if n == 10 else _CN_DIGITS[n]
|
||
if n < 20:
|
||
return "十" + _CN_DIGITS[n - 10]
|
||
if n < 100:
|
||
return _CN_DIGITS[n // 10] + "十" + (_CN_DIGITS[n % 10] if n % 10 else "")
|
||
return str(n)
|
||
|
||
|
||
# The ten target languages the dashboard offers. The codes match the tags the
|
||
# web dashboard already uses, so a production directory named for a tag and
|
||
# the language the pipeline was told to produce are the same thing.
|
||
TARGET_LANGUAGES: tuple[TargetLanguage, ...] = (
|
||
TargetLanguage(
|
||
code="zh-Hans",
|
||
base="zh",
|
||
name="Simplified Chinese",
|
||
native="简体中文",
|
||
chinese_name="简体中文",
|
||
scripts=(SCRIPT_HAN,),
|
||
forbidden_scripts=(SCRIPT_KANA, SCRIPT_HANGUL),
|
||
glossary_key="zh",
|
||
quote_open="“",
|
||
quote_close="”",
|
||
ellipsis="……",
|
||
dash="——",
|
||
paren_open="（",
|
||
paren_close="）",
|
||
title_open="《",
|
||
title_close="》",
|
||
font_serif="Noto Serif CJK SC",
|
||
font_sans="Noto Sans CJK SC",
|
||
chapter_prefix="第",
|
||
chapter_suffix="章",
|
||
cjk=True,
|
||
sentence_terminators="。！？!?；;…",
|
||
closing_marks="”’」』）)】》\"'",
|
||
clause_marks=("——", "—", "：", "，", "、", "；", ",", ";"),
|
||
asciidoc_lang="zh-CN",
|
||
odf_language="zh",
|
||
odf_country="CN",
|
||
epub_font_stack="Noto Serif CJK SC, Source Han Serif SC, WenQuanYi Micro Hei, serif",
|
||
chapter_numeral="chinese",
|
||
),
|
||
TargetLanguage(
|
||
code="es-ES",
|
||
base="es",
|
||
name="Spanish",
|
||
native="español",
|
||
chinese_name="西班牙语",
|
||
scripts=(SCRIPT_LATIN,),
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
|
||
glossary_key="es",
|
||
quote_open="“",
|
||
quote_close="”",
|
||
ellipsis="…",
|
||
title_open="«",
|
||
title_close="»",
|
||
asciidoc_lang="es-ES",
|
||
odf_language="es",
|
||
odf_country="ES",
|
||
chapter_prefix="Capítulo ",
|
||
),
|
||
TargetLanguage(
|
||
code="fr-FR",
|
||
base="fr",
|
||
name="French",
|
||
native="français",
|
||
chinese_name="法语",
|
||
scripts=(SCRIPT_LATIN,),
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
|
||
glossary_key="fr",
|
||
quote_open="«",
|
||
quote_close="»",
|
||
ellipsis="…",
|
||
title_open="«",
|
||
title_close="»",
|
||
asciidoc_lang="fr-FR",
|
||
odf_language="fr",
|
||
odf_country="FR",
|
||
chapter_prefix="Chapitre ",
|
||
),
|
||
TargetLanguage(
|
||
code="de-DE",
|
||
base="de",
|
||
name="German",
|
||
native="Deutsch",
|
||
chinese_name="德语",
|
||
scripts=(SCRIPT_LATIN,),
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
|
||
glossary_key="de",
|
||
quote_open="„",
|
||
quote_close="“",
|
||
ellipsis="…",
|
||
title_open="»",
|
||
title_close="«",
|
||
asciidoc_lang="de-DE",
|
||
odf_language="de",
|
||
odf_country="DE",
|
||
chapter_prefix="Kapitel ",
|
||
),
|
||
TargetLanguage(
|
||
code="it-IT",
|
||
base="it",
|
||
name="Italian",
|
||
native="italiano",
|
||
chinese_name="意大利语",
|
||
scripts=(SCRIPT_LATIN,),
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
|
||
glossary_key="it",
|
||
quote_open="«",
|
||
quote_close="»",
|
||
ellipsis="…",
|
||
title_open="«",
|
||
title_close="»",
|
||
asciidoc_lang="it-IT",
|
||
odf_language="it",
|
||
odf_country="IT",
|
||
chapter_prefix="Capitolo ",
|
||
),
|
||
TargetLanguage(
|
||
code="pt-BR",
|
||
base="pt",
|
||
name="Portuguese",
|
||
native="português",
|
||
chinese_name="葡萄牙语",
|
||
scripts=(SCRIPT_LATIN,),
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
|
||
glossary_key="pt",
|
||
quote_open="“",
|
||
quote_close="”",
|
||
ellipsis="…",
|
||
title_open="«",
|
||
title_close="»",
|
||
asciidoc_lang="pt-BR",
|
||
odf_language="pt",
|
||
odf_country="BR",
|
||
chapter_prefix="Capítulo ",
|
||
),
|
||
TargetLanguage(
|
||
code="ru-RU",
|
||
base="ru",
|
||
name="Russian",
|
||
native="русский",
|
||
chinese_name="俄语",
|
||
scripts=(SCRIPT_CYRILLIC,),
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL),
|
||
glossary_key="ru",
|
||
quote_open="«",
|
||
quote_close="»",
|
||
ellipsis="…",
|
||
title_open="«",
|
||
title_close="»",
|
||
asciidoc_lang="ru-RU",
|
||
odf_language="ru",
|
||
odf_country="RU",
|
||
chapter_prefix="Глава ",
|
||
),
|
||
TargetLanguage(
|
||
code="ja-JP",
|
||
base="ja",
|
||
name="Japanese",
|
||
native="日本語",
|
||
chinese_name="日语",
|
||
scripts=(SCRIPT_HAN, SCRIPT_KANA),
|
||
required_script=SCRIPT_KANA,
|
||
forbidden_scripts=(SCRIPT_HANGUL,),
|
||
glossary_key="ja",
|
||
quote_open="「",
|
||
quote_close="」",
|
||
ellipsis="……",
|
||
dash="——",
|
||
paren_open="（",
|
||
paren_close="）",
|
||
title_open="『",
|
||
title_close="』",
|
||
font_serif="Noto Serif CJK JP",
|
||
font_sans="Noto Sans CJK JP",
|
||
chapter_prefix="第",
|
||
chapter_suffix="章",
|
||
cjk=True,
|
||
sentence_terminators="。！？!?；;…",
|
||
closing_marks="”’」』）)】》\"'",
|
||
clause_marks=("——", "—", "：", "、", "，", "；", ",", ";"),
|
||
asciidoc_lang="ja-JP",
|
||
odf_language="ja",
|
||
odf_country="JP",
|
||
epub_font_stack="Noto Serif CJK JP, Source Han Serif JP, serif",
|
||
),
|
||
TargetLanguage(
|
||
code="ko-KR",
|
||
base="ko",
|
||
name="Korean",
|
||
native="한국어",
|
||
chinese_name="韩语",
|
||
scripts=(SCRIPT_HANGUL,),
|
||
required_script=SCRIPT_HANGUL,
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA),
|
||
glossary_key="ko",
|
||
quote_open="“",
|
||
quote_close="”",
|
||
ellipsis="…",
|
||
dash="—",
|
||
paren_open="(",
|
||
paren_close=")",
|
||
title_open="『",
|
||
title_close="』",
|
||
font_serif="Noto Serif CJK KR",
|
||
font_sans="Noto Sans CJK KR",
|
||
cjk=True,
|
||
sentence_terminators="。！？!?…",
|
||
closing_marks="”’」』）)】》\"'",
|
||
clause_marks=("—", "–", ":", ";", ",", "、", "，"),
|
||
asciidoc_lang="ko-KR",
|
||
odf_language="ko",
|
||
odf_country="KR",
|
||
epub_font_stack="Noto Serif CJK KR, Source Han Serif KR, serif",
|
||
chapter_prefix="제",
|
||
chapter_suffix="장",
|
||
),
|
||
TargetLanguage(
|
||
code="ar-SA",
|
||
base="ar",
|
||
name="Arabic",
|
||
native="العربية",
|
||
chinese_name="阿拉伯语",
|
||
scripts=(SCRIPT_ARABIC,),
|
||
required_script=SCRIPT_ARABIC,
|
||
forbidden_scripts=(SCRIPT_HAN, SCRIPT_KANA, SCRIPT_HANGUL, SCRIPT_CYRILLIC),
|
||
glossary_key="ar",
|
||
quote_open="”",
|
||
quote_close="“",
|
||
ellipsis="…",
|
||
dash="—",
|
||
title_open="«",
|
||
title_close="»",
|
||
font_serif="Noto Naskh Arabic",
|
||
font_sans="Noto Sans Arabic",
|
||
rtl=True,
|
||
asciidoc_lang="ar-SA",
|
||
odf_language="ar",
|
||
odf_country="SA",
|
||
epub_font_stack="Noto Naskh Arabic, Noto Sans Arabic, serif",
|
||
chapter_prefix="الفصل ",
|
||
# Arabic has its own question mark and semicolon, and its own comma.
|
||
sentence_terminators=".!?…؟؛",
|
||
closing_marks="”‘)]}»؟؛\"'",
|
||
clause_marks=("—", "–", ":", "؛", "،", ";", ","),
|
||
),
|
||
)
|
||
|
||
_BY_CODE: dict[str, TargetLanguage] = {lang.code: lang for lang in TARGET_LANGUAGES}
|
||
_BY_BASE: dict[str, TargetLanguage] = {lang.base: lang for lang in TARGET_LANGUAGES}
|
||
_BY_ALIAS: dict[str, TargetLanguage] = {}
|
||
for _lang in TARGET_LANGUAGES:
|
||
_BY_ALIAS[_lang.code.casefold()] = _lang
|
||
_BY_ALIAS[_lang.base.casefold()] = _lang
|
||
_BY_ALIAS[_lang.name.casefold()] = _lang
|
||
_BY_ALIAS[_lang.native.casefold()] = _lang
|
||
|
||
|
||
def by_code(value: str | None) -> TargetLanguage | None:
|
||
"""Look up a language by tag, base code, English name or native name.
|
||
|
||
``'es'``, ``'es-ES'``, ``'es_ES'`` and ``'Spanish'`` all resolve to the
|
||
same entry, so a caller does not have to know which form a production
|
||
directory happens to use.
|
||
"""
|
||
if not value:
|
||
return None
|
||
raw = str(value).strip()
|
||
if not raw:
|
||
return None
|
||
language = _BY_ALIAS.get(raw.casefold())
|
||
if language is not None:
|
||
return language
|
||
# 'es_ES' and 'zh_Hans' are common on disk; fold the separator.
|
||
folded = raw.replace("_", "-")
|
||
language = _BY_ALIAS.get(folded.casefold())
|
||
if language is not None:
|
||
return language
|
||
# A tag we do not carry, but whose base we do ('pt-PT' -> Portuguese).
|
||
base = folded.split("-", 1)[0].casefold()
|
||
return _BY_BASE.get(base)
|
||
|
||
|
||
def resolve(value: str | None) -> TargetLanguage:
|
||
"""The language for *value*, or the Chinese default with a warning.
|
||
|
||
Falling back silently would recreate the defect this module exists to
|
||
prevent: a production that declares a language the pipeline does not
|
||
support would be translated into Chinese and filed under the wrong name.
|
||
So the fallback is loud.
|
||
"""
|
||
language = by_code(value)
|
||
if language is not None:
|
||
return language
|
||
print(
|
||
f"warning: target language {value!r} is not in the registry; falling back to Simplified Chinese",
|
||
flush=True,
|
||
)
|
||
return _BY_BASE["zh"]
|
||
|
||
|
||
def script_counts(text: str) -> dict[str, int]:
|
||
"""How many letters of *text* belong to each script.
|
||
|
||
Only letters count. Punctuation, digits and whitespace say nothing about
|
||
which language a passage is written in, and a name kept in English inside
|
||
a Chinese paragraph must not pull the count towards Latin.
|
||
"""
|
||
counts: dict[str, int] = {}
|
||
for ch in text:
|
||
point = ord(ch)
|
||
for script, low, high in _SCRIPT_RANGES:
|
||
if low <= point <= high:
|
||
counts[script] = counts.get(script, 0) + 1
|
||
break
|
||
return counts
|
||
|
||
|
||
def script_mismatch(code: str | None, text: str) -> str | None:
|
||
"""Why *text* cannot be a translation into *code*, or None when it can.
|
||
|
||
The check is deliberately about the *script*, not the language. It cannot
|
||
tell Spanish from Portuguese, but it can tell Chinese from Spanish, which
|
||
is the mistake that actually happens - a pipeline configured for one
|
||
language that silently produces another.
|
||
"""
|
||
language = by_code(code)
|
||
if language is None:
|
||
return None
|
||
|
||
counts = script_counts(text)
|
||
total = sum(counts.values())
|
||
if total == 0:
|
||
return None
|
||
|
||
for forbidden in language.forbidden_scripts:
|
||
if counts.get(forbidden, 0) / total >= 0.10:
|
||
return f"the text is written in {SCRIPT_NAMES[forbidden]}, so it is not {language.name}"
|
||
|
||
if language.required_script and not counts.get(language.required_script):
|
||
return f"the text has no {SCRIPT_NAMES[language.required_script]}, so it is not {language.name}"
|
||
|
||
acceptable = sum(counts.get(script, 0) for script in language.scripts)
|
||
if acceptable / total < 0.5:
|
||
dominant = max(counts, key=lambda key: counts[key])
|
||
return f"the text is mostly {SCRIPT_NAMES[dominant]}, so it is not {language.name}"
|
||
return None
|
||
|
||
|
||
# --- text helpers ----------------------------------------------------------
|
||
|
||
# How many chapters the dashboard samples when it measures a production's
|
||
# language. A handful is enough to know which script a book is written in and
|
||
# cheap enough to run on every page load.
|
||
SAMPLE_CHAPTERS = 8
|
||
|
||
|
||
def sample_run_text(runs_dir: Path, max_chapters: int = SAMPLE_CHAPTERS) -> str:
|
||
"""The translated text of a sample of chapters in a run directory.
|
||
|
||
The sample is spread across the book rather than taken from the front:
|
||
the first chapter is often front matter, and a front-matter chapter can be
|
||
mostly numbers and names, which say nothing about which language the book
|
||
is in.
|
||
"""
|
||
if not runs_dir.is_dir():
|
||
return ""
|
||
chapters = sorted(p for p in runs_dir.glob("ch-*") if (p / "translation.json").is_file())
|
||
if not chapters:
|
||
return ""
|
||
if len(chapters) <= max_chapters:
|
||
chosen = chapters
|
||
else:
|
||
step = len(chapters) / max_chapters
|
||
chosen = [chapters[int(i * step)] for i in range(max_chapters)]
|
||
|
||
parts: list[str] = []
|
||
for chapter in chosen:
|
||
try:
|
||
data = json.loads((chapter / "translation.json").read_text(encoding="utf-8"))
|
||
except (OSError, json.JSONDecodeError):
|
||
continue
|
||
paragraphs = data.get("paragraphs") or {}
|
||
if isinstance(paragraphs, dict):
|
||
parts.extend(str(value) for value in paragraphs.values() if isinstance(value, str))
|
||
return " ".join(parts)
|
||
|
||
|
||
def check_runs_language(runs_dir: Path, code: str) -> str | None:
|
||
"""Why the translations in *runs_dir* cannot be in *code*, or None.
|
||
|
||
The dashboard reports a language's state from the presence and chapter
|
||
count of its run directory. That is a claim about the *files*, not about
|
||
the *text*: a directory full of Chinese can sit in a directory named for
|
||
Spanish and be reported as a complete Spanish translation. Measured on
|
||
oldbox, that is exactly what ``deal-hunter-es`` was - 3,038 paragraphs
|
||
of Chinese under a Spanish name, reported complete.
|
||
|
||
So the text is measured, not the file names.
|
||
"""
|
||
language = by_code(code)
|
||
if language is None:
|
||
return None
|
||
sample = sample_run_text(runs_dir)
|
||
if not sample.strip():
|
||
return None
|
||
return script_mismatch(language.code, sample)
|
||
|
||
|
||
# The clause boundary a language uses to join two sentences into one. Chinese
|
||
def punctuation_hint(language: TargetLanguage) -> str:
|
||
"""How to describe the target's punctuation, for a prompt."""
|
||
return (
|
||
f"引号用{language.quote_open}{language.quote_close}成对；省略号用{language.ellipsis}；破折号用{language.dash}"
|
||
)
|
||
|
||
|
||
# --- the strings the renderer writes ---------------------------------------
|
||
|
||
# The renderer puts a small amount of fixed text around the translation: the
|
||
# contents heading, the review-sample notice, the translator credit, the
|
||
# chapter heading convention. That text is part of the book the reader
|
||
# receives, so it has to be in the book's language. Leaving it in Chinese is
|
||
# the same defect as a Chinese chapter title in a Spanish edition, one level
|
||
# down and much easier to miss.
|
||
#
|
||
# A language with no entry here gets the English string, never another
|
||
# language's.
|
||
RENDER_STRINGS: dict[str, dict[str, str]] = {
|
||
"en": {
|
||
"missing_title": "[Untranslated] {title}",
|
||
"toc_title": "Table of Contents",
|
||
"review_sample": "Review sample",
|
||
"translator_credit": "Translated by Umbra Tools",
|
||
"sample_note": (
|
||
"This file is a review sample of the English translation. The translated "
|
||
"chapters are listed in the contents; the remaining chapters will be "
|
||
"completed in the full run."
|
||
),
|
||
"audio_preface": "Audio Preface",
|
||
"audio_preface_intro": "The following is the opening of the Deal Hunter audiobook.",
|
||
"sample_heading": "About this review sample",
|
||
"sample_process": (
|
||
"Translation process: paragraph reconstruction, intent analysis, first "
|
||
"draft, bilingual review. Terms are in glossary.json; the reviewer's "
|
||
"notes are in review.html."
|
||
),
|
||
"sample_end": "End of sample",
|
||
"sample_end_note": (
|
||
"That is the review sample. The full run will cover the remaining "
|
||
"chapters; the reviewer's issue list is provided with each chapter's "
|
||
"translation record."
|
||
),
|
||
"author_line": "B. D. Murphy",
|
||
"translator_line": "Translated by Umbra Tools",
|
||
"edition_label": "English edition · Review sample",
|
||
},
|
||
"zh": {
|
||
"missing_title": "[待译] {title}",
|
||
"toc_title": "目录",
|
||
"review_sample": "审阅样本",
|
||
"translator_credit": "译者：Umbra Tools",
|
||
"sample_note": "本文件是简体中文翻译的审阅样本。已译部分见目录；未列出的章节将在全文运行中补齐。",
|
||
"audio_preface": "音频版序言",
|
||
"audio_preface_intro": "以下是《Deal Hunter》有声版的开场白。",
|
||
"sample_heading": "审阅样本说明",
|
||
"sample_process": (
|
||
"翻译流程：段落重建 → 意图分析 → 初译 → 双语审校。术语见 glossary.json，审校意见见 review.html。"
|
||
),
|
||
"sample_end": "样本结束",
|
||
"sample_end_note": ("以上为审阅样本。全文运行将覆盖其余章节；审校问题清单会随各章翻译记录一并提供。"),
|
||
"author_line": "B·D·墨菲 著",
|
||
"translator_line": "Umbra Tools 译",
|
||
"edition_label": "简体中文译本 · 审阅样本",
|
||
},
|
||
"es": {
|
||
"missing_title": "[Sin traducir] {title}",
|
||
"toc_title": "Índice",
|
||
"review_sample": "Muestra de revisión",
|
||
"translator_credit": "Traducido por Umbra Tools",
|
||
"sample_note": (
|
||
"Este archivo es una muestra de revisión de la traducción al español. "
|
||
"Los capítulos traducidos aparecen en el índice; los capítulos "
|
||
"restantes se completarán en la ejecución completa."
|
||
),
|
||
"audio_preface": "Prefacio del audiolibro",
|
||
"audio_preface_intro": "Lo que sigue es la introducción del audiolibro de Deal Hunter.",
|
||
"sample_heading": "Sobre esta muestra de revisión",
|
||
"sample_process": (
|
||
"Proceso de traducción: reconstrucción de párrafos, análisis de intención, "
|
||
"primer borrador, revisión bilingüe. Los términos están en glossary.json; "
|
||
"las notas del revisor están en review.html."
|
||
),
|
||
"sample_end": "Fin de la muestra",
|
||
"sample_end_note": (
|
||
"Esta es la muestra de revisión. La ejecución completa cubrirá los "
|
||
"capítulos restantes; la lista de incidencias del revisor se entrega "
|
||
"junto con el registro de traducción de cada capítulo."
|
||
),
|
||
"author_line": "B. D. Murphy",
|
||
"translator_line": "Traducido por Umbra Tools",
|
||
"edition_label": "Edición en español · Muestra de revisión",
|
||
},
|
||
"fr": {
|
||
"missing_title": "[Non traduit] {title}",
|
||
"toc_title": "Table des matières",
|
||
"review_sample": "Échantillon de relecture",
|
||
"translator_credit": "Traduit par Umbra Tools",
|
||
"sample_note": (
|
||
"Ce fichier est un échantillon de relecture de la traduction française. "
|
||
"Les chapitres traduits figurent dans la table des matières ; les "
|
||
"chapitres restants seront complétés lors de l'exécution complète."
|
||
),
|
||
"audio_preface": "Préface de l'audiobook",
|
||
"audio_preface_intro": "Voici l'introduction de l'audiobook de Deal Hunter.",
|
||
"sample_heading": "À propos de cet échantillon",
|
||
"sample_process": (
|
||
"Processus de traduction : reconstruction des paragraphes, analyse de "
|
||
"l'intention, premier jet, relecture bilingue. Les termes sont dans "
|
||
"glossary.json ; les notes du relecteur sont dans review.html."
|
||
),
|
||
"sample_end": "Fin de l'échantillon",
|
||
"sample_end_note": (
|
||
"Voici l'échantillon de relecture. L'exécution complète couvrira les "
|
||
"chapitres restants ; la liste des remarques du relecteur est fournie "
|
||
"avec le dossier de traduction de chaque chapitre."
|
||
),
|
||
"author_line": "B. D. Murphy",
|
||
"translator_line": "Traduit par Umbra Tools",
|
||
"edition_label": "Édition française · Échantillon de relecture",
|
||
},
|
||
"de": {
|
||
"missing_title": "[Nicht übersetzt] {title}",
|
||
"toc_title": "Inhaltsverzeichnis",
|
||
"review_sample": "Korrekturdurchschlag",
|
||
"translator_credit": "Übersetzt von Umbra Tools",
|
||
"sample_note": (
|
||
"Diese Datei ist ein Korrekturdurchschlag der deutschen Übersetzung. "
|
||
"Die übersetzten Kapitel stehen im Inhaltsverzeichnis; die restlichen "
|
||
"Kapitel werden im vollständigen Lauf ergänzt."
|
||
),
|
||
"audio_preface": "Hörbuch-Vorwort",
|
||
"audio_preface_intro": "Es folgt die Einleitung des Hörbuchs zu Deal Hunter.",
|
||
"sample_heading": "Über diesen Korrekturdurchschlag",
|
||
"sample_process": (
|
||
"Übersetzungsablauf: Absatzrekonstruktion, Intentionsanalyse, Rohtext, "
|
||
"zweisprachige Korrektur. Begriffe stehen in glossary.json; die "
|
||
"Anmerkungen der Korrektorin stehen in review.html."
|
||
),
|
||
"sample_end": "Ende des Durchschlags",
|
||
"sample_end_note": (
|
||
"Das ist der Korrekturdurchschlag. Der vollständige Lauf deckt die "
|
||
"restlichen Kapitel ab; die Liste der Anmerkungen liegt dem "
|
||
"Übersetzungsprotokoll jedes Kapitels bei."
|
||
),
|
||
"author_line": "B. D. Murphy",
|
||
"translator_line": "Übersetzt von Umbra Tools",
|
||
"edition_label": "Deutsche Ausgabe · Korrekturdurchschlag",
|
||
},
|
||
"it": {
|
||
"missing_title": "[Non tradotto] {title}",
|
||
"toc_title": "Indice",
|
||
"review_sample": "Bozza di revisione",
|
||
"translator_credit": "Tradotto da Umbra Tools",
|
||
"sample_note": (
|
||
"Questo file è una bozza di revisione della traduzione italiana. I "
|
||
"capitoli tradotti compaiono nell'indice; i restanti saranno completati "
|
||
"nella corsione completa."
|
||
),
|
||
"audio_preface": "Prefazione dell'audiolibro",
|
||
"audio_preface_intro": "Quello che segue è l'introduzione dell'audiolibro di Deal Hunter.",
|
||
"sample_heading": "Informazioni su questa bozza",
|
||
"sample_process": (
|
||
"Processo di traduzione: ricostruzione dei paragrafi, analisi "
|
||
"dell'intento, prima stesura, revisione bilingue. I termini sono in "
|
||
"glossary.json; le note del revisore sono in review.html."
|
||
),
|
||
"sample_end": "Fine bozza",
|
||
"sample_end_note": (
|
||
"Questa è la bozza di revisione. La corsione completa coprirà i "
|
||
"capitoli restanti; l'elenco delle osservazioni del revisore "
|
||
"accompagna il registro di traduzione di ogni capitolo."
|
||
),
|
||
"author_line": "B. D. Murphy",
|
||
"translator_line": "Tradotto da Umbra Tools",
|
||
"edition_label": "Edizione italiana · Bozza di revisione",
|
||
},
|
||
"pt": {
|
||
"missing_title": "[Não traduzido] {title}",
|
||
"toc_title": "Sumário",
|
||
"review_sample": "Amostra de revisão",
|
||
"translator_credit": "Traduzido por Umbra Tools",
|
||
"sample_note": (
|
||
"Este arquivo é uma amostra de revisão da tradução para português. Os "
|
||
"capítulos traduzidos estão no sumário; os capítulos restantes serão "
|
||
"completados na execução completa."
|
||
),
|
||
"audio_preface": "Prefácio do audiolivro",
|
||
"audio_preface_intro": "Segue-se a introdução do audiolivro de Deal Hunter.",
|
||
"sample_heading": "Sobre esta amostra de revisão",
|
||
"sample_process": (
|
||
"Processo de tradução: reconstrução de parágrafos, análise de intenção, "
|
||
"primeira redação, revisão bilíngue. Os termos estão em glossary.json; "
|
||
"as notas do revisor estão em review.html."
|
||
),
|
||
"sample_end": "Fim da amostra",
|
||
"sample_end_note": (
|
||
"Esta é a amostra de revisão. A execução completa cobrirá os capítulos "
|
||
"restantes; a lista de observações do revisor acompanha o registro de "
|
||
"tradução de cada capítulo."
|
||
),
|
||
"author_line": "B. D. Murphy",
|
||
"translator_line": "Traduzido por Umbra Tools",
|
||
"edition_label": "Edição em português · Amostra de revisão",
|
||
},
|
||
"ru": {
|
||
"missing_title": "[Не переведено] {title}",
|
||
"toc_title": "Оглавление",
|
||
"review_sample": "Пробная вычитка",
|
||
"translator_credit": "Перевод: Umbra Tools",
|
||
"sample_note": (
|
||
"Этот файл — пробная вычитка русского перевода. Переведённые главы "
|
||
"перечислены в оглавлении; остальные главы будут дополнены при полном "
|
||
"прогоне."
|
||
),
|
||
"audio_preface": "Предисловие к аудиокниге",
|
||
"audio_preface_intro": "Ниже следует вступление к аудиокниге «Deal Hunter».",
|
||
"sample_heading": "Об этой пробной вычитке",
|
||
"sample_process": (
|
||
"Порядок перевода: сборка абзацев, анализ замысла, черновик, двуязычная "
|
||
"вычитка. Термины — в glossary.json, замечания редактора — в review.html."
|
||
),
|
||
"sample_end": "Конец пробы",
|
||
"sample_end_note": (
|
||
"Это пробная вычитка. Полный прогон охватит остальные главы; список "
|
||
"замечаний редактора прилагается к протоколу перевода каждой главы."
|
||
),
|
||
"author_line": "Б. Д. Морфи",
|
||
"translator_line": "Перевод: Umbra Tools",
|
||
"edition_label": "Русское издание · Пробная вычитка",
|
||
},
|
||
"ja": {
|
||
"missing_title": "[未翻訳] {title}",
|
||
"toc_title": "目次",
|
||
"review_sample": "査読用サンプル",
|
||
"translator_credit": "翻訳：Umbra Tools",
|
||
"sample_note": (
|
||
"このファイルは日本語訳の査読用サンプルです。翻訳済みの章は目次に記載されています。"
|
||
"残りの章は全文実行で補完されます。"
|
||
),
|
||
"audio_preface": "オーディオブック前書き",
|
||
"audio_preface_intro": "以下は『Deal Hunter』オーディオブックの冒頭です。",
|
||
"sample_heading": "この査読用サンプルについて",
|
||
"sample_process": (
|
||
"翻訳の流れ：段落の再構成、意図の分析、初稿、二言語査読。用語は "
|
||
"glossary.json、査読者の指摘は review.html にあります。"
|
||
),
|
||
"sample_end": "サンプル終了",
|
||
"sample_end_note": (
|
||
"以上が査読用サンプルです。全文実行が残りの章をすべて扱い、査読者の指摘一覧は各章の翻訳記録に添付されます。"
|
||
),
|
||
"author_line": "B・D・マーフィ 著",
|
||
"translator_line": "Umbra Tools 訳",
|
||
"edition_label": "日本語版 · 査読用サンプル",
|
||
},
|
||
"ko": {
|
||
"missing_title": "[미번역] {title}",
|
||
"toc_title": "목차",
|
||
"review_sample": "교정용 샘플",
|
||
"translator_credit": "번역: Umbra Tools",
|
||
"sample_note": (
|
||
"이 파일은 한국어 번역의 교정용 샘플입니다. 번역된 장은 목차에 나열되어 있고, "
|
||
"남은 장은 전체 실행에서 완성됩니다."
|
||
),
|
||
"audio_preface": "오디오북 머리말",
|
||
"audio_preface_intro": "다음은 『Deal Hunter』 오디오북의 도입부입니다.",
|
||
"sample_heading": "이 교정용 샘플에 대하여",
|
||
"sample_process": (
|
||
"번역 절차: 문단 재구성, 의도 분석, 초안, 이중 언어 교정. 용어는 "
|
||
"glossary.json, 교정자 메모는 review.html에 있습니다."
|
||
),
|
||
"sample_end": "샘플 끝",
|
||
"sample_end_note": (
|
||
"이것이 교정용 샘플입니다. 전체 실행이 남은 장을 모두 다루며, 교정자의 지적 "
|
||
"목록은 각 장의 번역 기록과 함께 제공됩니다."
|
||
),
|
||
"author_line": "B. D. 머피 지음",
|
||
"translator_line": "Umbra Tools 옮김",
|
||
"edition_label": "한국어판 · 교정용 샘플",
|
||
},
|
||
"ar": {
|
||
"missing_title": "[لم يُترجم] {title}",
|
||
"toc_title": "المحتويات",
|
||
"review_sample": "عيّنة مراجعة",
|
||
"translator_credit": "ترجمة: Umbra Tools",
|
||
"sample_note": (
|
||
"هذا الملف عيّنة مراجعة للترجمة العربية. الفصول المترجمة مدرجة في المحتويات، "
|
||
"وتُستكمل بقية الفصول في التشغيل الكامل."
|
||
),
|
||
"audio_preface": "تمهيد الكتاب الصوتي",
|
||
"audio_preface_intro": "فيما يلي مقدمة الكتاب الصوتي لـ Deal Hunter.",
|
||
"sample_heading": "عن هذه عيّنة المراجعة",
|
||
"sample_process": (
|
||
"مسار الترجمة: إعادة بناء الفقرات، تحليل القصد، المسودة الأولى، المراجعة "
|
||
"ثنائية اللغة. المصطلحات في glossary.json، وملاحظات المراجع في review.html."
|
||
),
|
||
"sample_end": "نهاية العيّنة",
|
||
"sample_end_note": (
|
||
"هذه هي عيّنة المراجعة. سيغطي التشغيل الكامل بقية الفصول، وتُرفق قائمة ملاحظات المراجع مع سجل ترجمة كل فصل."
|
||
),
|
||
"author_line": "ب. د. مورفي",
|
||
"translator_line": "ترجمة: Umbra Tools",
|
||
"edition_label": "الطبعة العربية · عيّنة مراجعة",
|
||
},
|
||
}
|