Ingest the translation delivery, and stop mis-reporting locale codes
UmbraTrans delivers one row per utterance per target language, in the shape translations.import_translations already accepts. There was no script to run that ingest, so the delivery side of the contract existed only as a function. scripts/import_translations.py runs it: report-only by default, --write to store, --confirm to store as confirmed rather than draft. It prints the coverage before and after, so a delivery can be checked without writing it. Two bugs in the ingest, both found by running a real delivery: * import_translations folded the target language with lang.normalize, which does not know script subtags. A delivery labelled "zh-Hans" - which is what UmbraTrans writes - matched nothing, so every row was dropped and reported as a problem. It now folds with canonical_code, and an unknown language is still stored: the text is reviewable even when the render will be refused. * coverage and not_voiced_rows compared the stored code to the requested code as strings, so a row stored as "zh-Hans" read as "nothing delivered" when the reviewer asked about "zh". The coverage report was wrong in the direction that makes a reviewer distrust it. Measured on the real Deal Hunter Chinese delivery: 4,300 rows ingested, the review UI's resolver then renders Chinese for 12/12 utterances of chapter 1 and 415/416 of chapter 6 - the missing one being the spoken chapter announcement, which is deliberately not part of the reading edition. spec 27 records what the shipped samples actually are: single-pass drafts made against a hand-cleaned paraphrase of the English, matching the speaker-verified database text in 0 of 12 utterances. The production Chinese edition on oldbox does use the intent-based process; what it lacked was the alignment back onto the utterances, which is now implemented in UmbraTrans. 7 new tests. Suite: 890 passed, 8 failed (the same 8 pre-existing failures).
This commit is contained in:
@@ -115,6 +115,76 @@ They are useful as a **bad-case corpus**: Pass 4 run over them should find real
|
||||
errors. If the critic cannot find errors in known-imperfect text, it is not
|
||||
doing its job.
|
||||
|
||||
### Measured (2026-11-06): the sample files are not the book
|
||||
|
||||
The sample files under `samples/translations/` were measured against the
|
||||
speaker-verified English in the project's database. **0 of 12 utterances match
|
||||
the database text exactly**; the sample English is a hand-cleaned paraphrase
|
||||
of it, with the source's typos silently repaired ("the AI the runs the ship"
|
||||
-> "the AI that runs the ship", "Kianda's aunt" -> "Kainda's aunt", "I trade
|
||||
salvage" -> "I trade salvage"). The sample files also carry `Kainda` where the
|
||||
database carries the canonical `Kainda Walters Foster Murphy`, and `Narrator`
|
||||
where the database carries `NULL`.
|
||||
|
||||
So those files are wrong twice over: they are single-pass, and they were
|
||||
translated against a different English base than the one the audiobook speaks.
|
||||
They must not be treated as a translation of anything. The only other
|
||||
translation in the system is one placeholder row whose `text` is the English
|
||||
source under a `zh-Hans` code, which the resolver correctly ignores.
|
||||
|
||||
### The real Deal Hunter Chinese edition
|
||||
|
||||
The production translation is not in this repository. It is built by UmbraTrans
|
||||
on oldbox (`productions/deal-hunter-zh`), and it **does** use the intent-based
|
||||
process:
|
||||
|
||||
| Pass | What it does | Model |
|
||||
|---|---|---|
|
||||
| 0 | Paragraph/scene/chunk assembly, source indices kept | none |
|
||||
| 1 | Scene intent: summary, register, tone, participants, risks | `lfm2.5-2.6b` |
|
||||
| 2 | Context-aware draft, given glossary, character voices, scene analysis, prior chunk summary | `qwen3.5-9b` |
|
||||
| 3 | Chapter voice/term consistency (optional) | critic model |
|
||||
| 4 | Bilingual critic: meaning, pronoun, irony, term, omission | `deepseek-r1-8b` |
|
||||
|
||||
Its Pass 1 output carries per-paragraph risks with guidance, which is intent
|
||||
analysis, not word mapping. Its critic runs on a *different* model from the
|
||||
drafter, which is the point of Pass 4.
|
||||
|
||||
### The missing pass, and why it matters
|
||||
|
||||
That pipeline works on **paragraphs**, because epub and pdf are paragraph
|
||||
media. The audiobook needs **utterances**: this project has verified who
|
||||
speaks every one of them, and each clip is voiced separately by that
|
||||
speaker's voice. A paragraph-level translation cannot be spoken.
|
||||
|
||||
Measured on the real book: 3,047 paragraphs cover 4,317 source utterances.
|
||||
2,322 paragraphs map to one utterance, 180 to two and 545 to three. All 725
|
||||
multi-utterance paragraphs are narration; dialogue is already one paragraph
|
||||
per speech turn.
|
||||
|
||||
So the missing piece is an alignment pass that divides each translated
|
||||
paragraph between the utterances it was built from. It is implemented in
|
||||
UmbraTrans as `tools/translation/align.py` (Pass 5), and its output is a
|
||||
delivery payload in the shape this project's `translations.import_translations`
|
||||
already accepts.
|
||||
|
||||
Two rules the alignment must not break:
|
||||
|
||||
- a source utterance may only be absorbed into another utterance **by the same
|
||||
speaker** - a line must not be handed to a different character;
|
||||
- the partition must reproduce the translated text exactly (modulo
|
||||
whitespace), so the alignment can never rewrite the translation.
|
||||
|
||||
A source utterance the target does not voice is delivered as
|
||||
`voiced = false` with the reason. The row is kept, so the gap stays visible;
|
||||
it is not deleted.
|
||||
|
||||
Measured on the real book (2026-11-06), 42 sampled multi-utterance
|
||||
paragraphs across seven chapters: **41 aligned by the model, 1 fell back to
|
||||
the mechanical split, 0 partition mismatches**. The one fallback was a model
|
||||
answer that did not reproduce the translated text; the content check caught it
|
||||
and the paragraph was flagged rather than accepted.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Which model runs Passes 1-4? This is where the local LLM question returns,
|
||||
|
||||
Reference in New Issue
Block a user