Refuse a delivery whose text is not written in the language it claims

The production directory oldbox calls "deal-hunter-es" declares
target_language "es-ES", its glossary says Spanish, and its 32-chapter run
finished - and all 3,038 translated paragraphs are in Chinese. It is a second
Chinese draft filed under a Spanish name.

The cause is in the pipeline: the drafter's instructions are hard-coded to
Simplified Chinese in three places (SYSTEM_DRAFTER, draft_prompt,
SYSTEM_CRITIC), so the glossary's target language was ignored. No non-Chinese
translation has ever been produced by that pipeline.

The danger at this end is that the rows would be stored under "es", the review
lane would look populated, and a reviewer would approve Chinese as Spanish.
So the ingest now checks the script of the delivered text against the
declared language, and refuses the delivery before writing a single row.

The check is about the script, not the language: it cannot tell Spanish from
Portuguese, but it can tell Chinese from Spanish, which is the mistake that
actually happens. Japanese needs its kana and Korean its Hangul, and a
Chinese delivery carrying kana is a Japanese translation filed in the wrong
lane. An unknown language is not checked - there is no expectation to check
it against.

Proven on the real data: the es delivery is refused with "the text is mostly
Han characters, so it is not Spanish", and the 4,300-row Chinese delivery
still imports unchanged.

19 new tests. Suite: 909 passed, 8 failed (the same 8 pre-existing failures).
This commit is contained in:
Umbra Tools
2026-11-06 20:06:59 -08:00
parent 81915b4ec1
commit 3db6ac3b8b
4 changed files with 400 additions and 6 deletions

View File

@@ -185,6 +185,50 @@ the mechanical split, 0 partition mismatches**. The one fallback was a model
answer that did not reproduce the translated text; the content check caught it
and the paragraph was flagged rather than accepted.
### The pipeline is Chinese-only, and one production says otherwise
Measured on oldbox (2026-11-06), the second production directory is
`productions/deal-hunter-es`. It declares `target_language: es-ES`, its
glossary says `"script": "Spanish"` with Spanish style rules and the Spanish
title `Cazador de Ofertas`, and it has a complete 32-chapter run (39,083 s).
**Its text is Chinese.** All 3,038 translated paragraphs are Chinese, in the
same proportion as the Chinese production (3,034 with Han characters, 4
Latin-only). It is a second Chinese draft filed under a Spanish name - a
different Chinese wording from the `deal-hunter-zh` run (2,792 of 3,038
paragraphs differ), but Chinese.
The cause is in the pipeline, not in the run: the drafter's instructions are
**hard-coded to Simplified Chinese** in three places.
- `prompts.py` `SYSTEM_DRAFTER`: *"把英文太空冒险小说译成简体中文"* - translate the
English space adventure novel into Simplified Chinese.
- `prompts.py` `draft_prompt`: *"请把下面英文译为简体中文"* and a JSON key of
`"zh"`.
- `prompts.py` `SYSTEM_CRITIC`: *"对照英文原文和中文译文"*.
The glossary is passed into the prompt, but the *instruction* says Chinese, so
the model followed the instruction and ignored the glossary's target. The
critic is hard-coded the same way, so it checked the Chinese text and found
nothing wrong - there was nothing wrong with it, for the language it was
actually asked about.
Two consequences, both measured rather than inferred:
1. **No non-Chinese translation has ever been produced by this pipeline.** The
nine other Qwen3-TTS languages exist only as the single-pass sample files
described above.
2. **The English translation output would be wrong for any other language**,
because the analyst, the drafter and the critic are all told they are
working on Chinese.
The pipeline needs a target-language parameter before it can be run for
anything but Chinese. Until then, a production directory's name and its
`target_language` field are a *claim*, not a measurement - which is why the
delivery ingest now checks the script of the delivered text against the
declared language and refuses a delivery whose text is written in another
language (`umbraaudio/languages.py:script_mismatch`).
## Open questions
1. Which model runs Passes 1-4? This is where the local LLM question returns,