The production directory oldbox calls "deal-hunter-es" declares
target_language "es-ES", its glossary says Spanish, and its 32-chapter run
finished - and all 3,038 translated paragraphs are in Chinese. It is a second
Chinese draft filed under a Spanish name.
The cause is in the pipeline: the drafter's instructions are hard-coded to
Simplified Chinese in three places (SYSTEM_DRAFTER, draft_prompt,
SYSTEM_CRITIC), so the glossary's target language was ignored. No non-Chinese
translation has ever been produced by that pipeline.
The danger at this end is that the rows would be stored under "es", the review
lane would look populated, and a reviewer would approve Chinese as Spanish.
So the ingest now checks the script of the delivered text against the
declared language, and refuses the delivery before writing a single row.
The check is about the script, not the language: it cannot tell Spanish from
Portuguese, but it can tell Chinese from Spanish, which is the mistake that
actually happens. Japanese needs its kana and Korean its Hangul, and a
Chinese delivery carrying kana is a Japanese translation filed in the wrong
lane. An unknown language is not checked - there is no expectation to check
it against.
Proven on the real data: the es delivery is refused with "the text is mostly
Han characters, so it is not Spanish", and the 4,300-row Chinese delivery
still imports unchanged.
19 new tests. Suite: 909 passed, 8 failed (the same 8 pre-existing failures).
The remote had one commit this branch did not (03f8a85), which adds the
Apache-2.0 LICENSE, the NOTICE and SPDX headers to every .py and .sh
file. The merge was clean - the headers sit at the top of each file and
my changes are further down.
The new files this branch added (languages.py, translations.py, the
cluster package, the new tests and scripts) were written after the
merge base, so the licence commit never saw them. They now carry the
same header, so the convention holds across the whole tree.
Nothing else changed. 883 tests passed / 8 failed - the same 8
pre-existing failures.
The review UI's language dropdown did nothing: every selection kept
rendering the English text. Two independent causes, both fixed.
1. The translations were never being read. The pipeline's
`utterance_translations` table holds one row in Deal Hunter, and that
row stores the *English* text under a `zh-Hans` code. The real
translations live in the on-disk sample files under
`samples/translations/`. `umbraaudio/translations.py` now resolves both
sources and merges them by (chapter_id, utterance_index): the table wins
where it holds a real translation, the sample files fill the gaps. A
candidate whose text is identical to the source is a placeholder, not a
translation, so it is ignored - otherwise the page would show English
under a Chinese heading and claim it was translated.
2. The query that read them was broken. `language_routes.py` selected
`u.index` and `u.speaker`; the columns are `utterance_index` and
`speaker_name`, so every read raised `sqlite3.OperationalError: near
"index": syntax error` and `/language-mockup` returned 500. It also
opened `<project>/project.db` - a 0-byte decoy - instead of
`<project>/.umbra/project.db`, the file that actually holds the data.
Also in this commit:
* The chapter review page honours `?lang=`, remembers the choice in a
cookie (set by a middleware, so it survives on every page), and shows the
selected language's *name* in the "has not been translated yet" message.
* The page now shows the translation, not the source, and the edit box is
bound to the source text so saving cannot overwrite the English with the
Chinese.
* The stray language selector that sat after `</html>` in base.html is
gone; the chapters-list page's dropdown had zero options because the
route never passed the language list.
* The language tag is wired into generation: `utterance.language_code`
(NULL = project source language) reaches `engine.generate()` and the
staleness marker, so a retagged line is re-rendered instead of keeping
English phonology behind an unchanged text. The marker records the
language; a marker written before the tag existed means English, so the
~4,300 existing clips are not invalidated.
* `project.source_language` (spec 31 section 13.1) records the book's own
language, with a migration for existing project DBs.
* A scene-break marker deliberately records no language: it is silence,
identical in every language, and tagging it would claim a phonology the
audio does not have.
Tests: 883 passed / 8 failed - the same 8 pre-existing failures
(2 test_pronunciation, 6 voice_profile_manager accent). 26 new tests
cover the resolver and the generation wiring. ruff introduces no new
findings.
