Record the translation alignment session, and fix the summary header

The "Current Session Summary" heading had run into the first session's
heading on one line, so the first session read as part of the header and
never appeared in the outline. Split them, and add this session's entry
above.
This commit is contained in:
Umbra Tools
2026-11-06 19:44:41 -08:00
parent 301ff17f3e
commit 81915b4ec1

106
AGENTS.md
View File

@@ -209,7 +209,111 @@ Each pass has quality gates. Quality is the primary concern; generation speed is
- Never add a dependency if stdlib suffices
- Prefer: stdlib → third-party → custom implementation
## Current Session Summary### Session: Login gate on the review UI (replaces the open surface)
## Current Session Summary
### Session: Intent-based translation is real; the missing link was the utterance alignment
- **Question answered with evidence: the production Chinese translation DOES use the
intent-based, chapter-level, speaker-mapped process - but the audiobook link was missing.**
Three independent findings:
1. The 10 `samples/translations/*.json` files self-declare
`provenance: "LLM-produced sample, single pass, NOT the multi-pass pipeline"` - spec 27
calls them "Pass 2 with Pass 1 skipped". They also were translated against a
**hand-cleaned paraphrase** of the English: **0 of 12** utterances match the
speaker-verified DB text (typos silently repaired, e.g. "the AI the runs the ship" ->
"the AI that runs the ship"), and they carry `Kainda`/`Narrator` where the DB carries
`Kainda Walters Foster Murphy`/`NULL`. **They are not a translation of anything.**
2. The **real** production translation lives in UmbraTrans on oldbox
(`/home/umbra/src/UmbraTrans/productions/deal-hunter-zh`) and DOES use the process:
Pass 0 paragraph assembly -> Pass 1 scene intent (summary/register/tone/participants/
**per-paragraph risks + guidance**) -> Pass 2 context-aware draft (glossary + character
voices + scene analysis + prior-chunk summary) -> Pass 4 bilingual critic on a
**different** model (`deepseek-r1-0528` vs drafter `qwen3.5-9b`, analyst `lfm2.5-2.6b`).
Outputs `deal-hunter-zh.asciidoc/.epub/.fodt/.odt/.pdf` (Oct 25).
3. No intent artefacts exist in this repo - no `propositions`/`speech_act`/`depends_on`/
`literal_risk` anywhere; spec 27's status was "design, not implemented".
- **The missing link, quantified.** The pipeline works on **paragraphs**; the audiobook needs
**utterances**. Measured on the real book: 3,047 paragraphs cover 4,317 source utterances -
2,322 paragraphs -> 1 utterance, 180 -> 2, 545 -> 3; all 725 multi-utterance paragraphs are
narration (dialogue is already 1 paragraph per speech turn). UmbraTrans's own
`tools/translation/README.md` said it: *"It does not produce a 1:1 utterance alignment for
TTS ... splitting or aligning back to the 4,319 utterances is a separate pass."*
- **Alignment lives in UmbraTrans, not here** - spec 18 §1 makes UmbraAudio an upstream
read-only provider, and `book.py` already anticipated "a later audio-alignment step can
still map back". New `tools/translation/align.py` = **Pass 5**. Committed `996fe37`, then
`f0c2317`; pushed to `ssh://git@mini-eq:2222/umbragit/UmbraTrans.git` (master `f0c2317`).
- **Alignment is by meaning, not position.** A model gets the source utterances + the
translation and says which part renders which utterance. Two rules it cannot break:
a source utterance may only be absorbed into another **by the same speaker**; and the
partition must reproduce the translated text exactly (whitespace-insensitive), so the
alignment **never rewrites the translation**. A source utterance the target absorbed is
delivered as `voiced=false` with a reason (UmbraAudio's `utterance_translations.voiced`):
row kept, gap visible, no clip generated.
- **Scene breaks are silence** and have no translation. The spoken chapter announcement
("Deal Hunter. Chapter one, Salvage.") is deliberately excluded from the reading edition.
- **Mechanical clause-boundary split is the no-model fallback only**; always flagged
"needs review". It is also the only thing that can *fail*, and the failures are real
translation defects (see below), not alignment defects.
- **Three real bugs found and fixed in the aligner** (by running it on the real book):
the splitter only tried the *longest* piece, which may be the unsplittable one; clause
splitting ignored `:`/`——` and marks at a piece's end (Chinese merges English sentences
with a comma/colon/dash, not a full stop); and **the CLI assigned `llm.cache_dir` after
construction**, so the directory was never created - every model answer would error and
the fallback would silently turn the whole book into a mechanical split **while still
reporting success**. Now created by the CLI and pinned by a test.
- **Two real ingest bugs fixed in `umbraaudio/translations.py`:** `import_translations` folded
the target with `lang.normalize`, which does not know script subtags, so a delivery labelled
`zh-Hans` was **dropped entirely**; and `coverage`/`not_voiced_rows` compared stored vs
requested codes as strings, so a row stored as `zh-Hans` read as "nothing delivered" when
the reviewer asked about `zh`. Locale codes now fold to base codes on ingest and on every
coverage query.
- **Delivery contract is the existing `import_translations` payload shape** - neither side
invents a second contract. `scripts/import_translations.py` added: report-only by default,
`--write`, `--confirm`, `--replace` (clears the language's **draft** rows only; confirmed
rows are never deleted - a confirmed translation is an approval).
- **Bundle verified in sync with the live DB:** 428/428 ch1+ch6 bundle rows match the current
UmbraAudio DB on text *and* index; 0 missing; 0 differing.
- **End-to-end delivery proven.** 4,300 rows ingested; the review UI's `resolved_translations`
then renders Chinese for **12/12** utterances of ch1 and **415/416** of ch6 (the missing one
is the spoken chapter announcement). Live UI measured after import: `/chapters-audio/ch1?lang=zh`
-> 12 x "✓ Chinese", 354 CJK chars, 0 "not translated"; `/chapters-audio/ch6?lang=zh` ->
415 x "✓ Chinese", 8,231 CJK chars, 1 "not translated" (the chapter announcement);
`/chapters-audio/ch6?lang=ru` -> 416 "not translated"; `?lang=en` -> English.
- **Model alignment validated on real data:** 42 sampled multi-utterance paragraphs across 7
chapters -> **41 LLM, 1 mechanical fallback, 0 partition mismatches**; the one fallback was a
model answer that did not reproduce the translation, caught by the content check. Chapter 6
full run: **232 paragraphs, 131 one_to_one, 101 llm, 0 needing review**, delivery 415
utterances (412 voiced, **3 collapsed**) - e.g. `utt-88245` "It was first launched over 450
years ago." absorbed into `utt-88244`, whose Chinese is "Deal Hunter 是一艘老船,早在四百五十
年前就已首航。"
- **Spec 27 updated** with the measured findings, the production pipeline table, and the
alignment-pass description. UmbraAudio commits `c476cfe` + `301ff17`, pushed to
`minieq` (`ssh://git@mini-eq:2222/umbragit/UmbraAudio.git`).
- **UmbraAudio suite: 890 passed / 8 failed / 6 skipped** - the same 8 pre-existing failures
(2 `test_pronunciation`, 6 `voice_profile_manager` accent); +9 new tests, no regressions.
Web server restarted; all pages return 200 (auth gate intact, `/` -> 303 `/login`).
- **IN FLIGHT:** full-book model alignment running (`--chapters all`, output
`zhprod/delivery-full-llm.json`); progress at ch-9 of 32, **0 needing review** so far, 184
align cache entries. Auto-ingest watcher armed (`ingest-after-align.sh` waits for the
process to exit, then runs `scripts/import_translations.py --write --replace` so the
model-aligned delivery replaces the mechanical rows automatically).
- **Two production translation defects found, NOT fixed** (translation-quality, not code):
`ch-15-p0100` source is a passage about dust on a console but the Chinese is "是控制台上的";
`ch-25-p0089` source describes cuboid shapes but the Chinese is "怎么才能接通电源?". Both
flagged "cannot divide" - the aligner is correctly reporting a broken upstream paragraph.
- **Third alignment failure is legitimate:** `ch-32-p0001` - source ["Kainda stops.", "The
servant says,"], Chinese "凯因达停下脚步。" - a genuine collapse needing the model to mark
source #2 not voiced.
- **Blocked / next:** oldbox cluster (PostgreSQL listens only on `127.0.0.1:5432`; needs
`listen_addresses='*'`, pg_hba for `192.168.123.0/24`, db `umbratrans` + roles
`umbraaudio`/`umbraformat`/`umbra_web`, PgBouncer on 6432, NFS exports, firewall) - **sudo
on oldbox needs a password**, so `postgresql.conf`/`pg_hba.conf` cannot be edited
non-interactively. Controller Stage 1 still waiting on a `compute_nodes` row + credential
with `compute:read` + `compute:heartbeat`. LM Studio at `192.168.123.16:1234` is down from
both this Mac and oldbox; alignment runs against **this Mac's** `http://127.0.0.1:1234/v1`
(model `qwen/qwen3.5-9b` present, plus `translategemma-12b-it`, `gemma-4-12b-qat`,
`google/gemma-4-e4b`). Other languages (`de/es/fr/it/pt/ja/ko`) still need production runs;
`productions/deal-hunter-es` also exists on oldbox.
### Session: Login gate on the review UI (replaces the open surface)
- **The review UI is now authenticated.** Until 2026-10-09 it had **no auth at all**:
`active_project` is an unsigned cookie and the server binds `0.0.0.0`, so every page -
including mutation endpoints and audio downloads - was reachable from the LAN and,