Compare commits

...

2 Commits

Author SHA1 Message Date
Umbra Tools
25605e2cad Record the paragraph-boundary drift findings and the verification stage
Also records the negative result from the independent cross-check attempt:
names are not a drift signal, and the glossary-anchor detector conflates
drift with ordinary glossary non-compliance. Both would otherwise be
tried again.
2026-11-06 21:40:37 -08:00
Umbra Tools
31cd9f7ce4 Record the paragraph-boundary drift and how it is now caught
The production Chinese re-flows the passage across paragraph ids in the
places where Pass 0 split the original mid-sentence. Measured: 439 of
3,038 translated paragraphs (14.5%) are sentence fragments, and in ch-15
the drift runs p0094-p0104 - eleven paragraphs whose Chinese is good but
attached to the wrong paragraph.

The alignment pass cannot see this by construction: a one-to-one paragraph
is a pure copy, so there is nothing to divide and nothing to disagree.

The verification stage finds it, and the spec records how it was made
precise. The first version asked two yes/no questions per paragraph and
was wrong in both directions - five of fifteen flags were paragraphs it
claimed were missing an adverb that was present, and it missed two
paragraphs that are drifted. The stage now asks which paragraph a
translation renders, shows the model its neighbours, and judges the
paragraphs as a window rather than one at a time.

On the real ch-15 window it now finds exactly the drift: ten paragraphs
dropped, p0094 kept and flagged because its Chinese correctly renders its
first two sentences.

Also recorded: the cost of the choice (a gap where a fragment inside a
broken window is dropped on an unreliable verdict) and the root cause
(the drafter is not told that each id must cover only its own English).
2026-11-06 21:26:21 -08:00
2 changed files with 113 additions and 0 deletions

View File

@@ -287,6 +287,49 @@ Each pass has quality gates. Quality is the primary concern; generation speed is
- **Spec 27 updated** with the measured findings, the production pipeline table, and the
alignment-pass description. UmbraAudio commits `c476cfe` + `301ff17`, pushed to
`minieq` (`ssh://git@mini-eq:2222/umbragit/UmbraAudio.git`).
- **THE PRODUCTION CHINESE HAS A PARAGRAPH-BOUNDARY DRIFT, and the alignment pass
cannot see it by construction.** Pass 0 splits the original at sentence boundaries, so
many paragraphs are **sentence fragments** ("The chair in the middle has", `"operations"`,
"on the console."): measured **439 of 3,038 translated paragraphs (14.5%)**. A fragment is
exactly where an intent-based drafter is told to write natural target-language prose, and
once it does that the id-to-text correspondence drifts - the Chinese is good, but attached
to the wrong paragraph. **In ch-15 the drift runs p0094-p0104 (11 paragraphs):** p0102's
Chinese ("上一次有人登上这艘舰桥...") is p0101's translation, and p0104's Chinese opens with
p0103's sentence. The alignment pass *cannot* see it: a one-to-one paragraph is a pure copy,
so there is nothing to divide and nothing to disagree.
- **The verification stage (`align.py --verify`) finds it, but only after two rounds of
measurement.** The first version asked two yes/no questions per paragraph ("is anything
missing", "is anything extra") and was wrong in **both** directions: it flagged 5 paragraphs
that were fine (it called the adverb `then` missing from a translation that has it, and
`Enthusiastically` missing from one that has it - the model was reading the *English* and
not the Chinese), and it missed 2 paragraphs that are drifted. **5 of 15 flags were wrong.**
The stage now asks **which paragraph does this translation render** (aligned / incomplete /
drift), shows the model its **neighbouring** paragraphs, and judges the paragraphs as a
**window** rather than one at a time - because the model calls the two paragraphs *before* a
real drift "aligned" when their Chinese is demonstrably the neighbour's, and the start of a
drift is not knowable from any single paragraph. On the real ch-15 window it now finds
exactly the drift: **10 paragraphs dropped (p0095-p0104)**, p0094 kept and flagged because
its Chinese correctly renders its first two sentences. Only `drift` deletes a line.
**The cost is explicit:** a fragment inside a broken window can be dropped on an unreliable
verdict, so the audiobook has a gap there. The alternative is a line that belongs to a
different sentence, and a gap is visible while a wrong line is silent.
- **A dropped paragraph still gets a row:** `voiced=false` + the reason, so the gap says *why*
rather than looking like a paragraph nobody translated. Verified end-to-end: the ch-15
delivery imports 154 rows, 13 of them not-voiced with the reason in `review_reason`.
- **An independent deterministic cross-check was tried and FAILED - names are not a drift
signal.** A glossary-anchor detector found 179 candidate paragraphs, and a name-anchor
detector found **1**, which was a false positive (word order: 公主凯因达 vs 凯因达公主).
The ch-15 drift window contains no names at all. Do not repeat the glossary-anchor
detector: it conflates drift with ordinary glossary non-compliance (the model renders
"Warbots" as 战博特 as often as 战斗机器人).
- **The root cause is upstream and is now fixed in the prompts** (UmbraTrans `5433653`): the
drafter is told each id may cover only its own English, even where the source is cut
mid-sentence, and must not move a neighbour's content in or its own content out; the critic
is asked to check the id-to-text correspondence (new type `misalignment`, severity
`block`). The critic cannot repair the draft - the pipeline records critic issues rather
than revising - but the drift becomes visible in the run record instead of silent.
- **UmbraTrans commits pushed to `minieq`:** `acab79d` (window judging), `11fd8d8` (gap rows
keep the reason), `5433653` (drafter + critic prompts). Master = `5433653`.
- **UmbraAudio suite: 890 passed / 8 failed / 6 skipped** - the same 8 pre-existing failures
(2 `test_pronunciation`, 6 `voice_profile_manager` accent); +9 new tests, no regressions.
Web server restarted; all pages return 200 (auth gate intact, `/` -> 303 `/login`).

View File

@@ -185,6 +185,76 @@ the mechanical split, 0 partition mismatches**. The one fallback was a model
answer that did not reproduce the translated text; the content check caught it
and the paragraph was flagged rather than accepted.
### The drafter re-flows the passage, and the alignment cannot see it
There is a defect in the production Chinese that the alignment pass is
structurally unable to find, and it has to be stated plainly because it
changes what the delivery means.
Pass 0 splits the original passage at sentence boundaries, so many
paragraphs are **sentence fragments** - `The chair in the middle has`,
`"operations"`, `on the console.` Measured: **439 of 3,038 translated
paragraphs (14.5%) are fragments**. A fragment is exactly where an
intent-based drafter is told to write natural target-language prose, and
once it does that, the id-to-text correspondence drifts: the Chinese is
good, but it is attached to the wrong paragraph.
In ch-15 the drift runs from p0094 to p0104 - eleven paragraphs. The Chinese
for p0102 (`"Yes, Captain."`) is the translation of p0101, and the Chinese
for p0104 opens with p0103's sentence. The audiobook would say the right
words at the wrong place.
The alignment pass could not see it. A one-to-one paragraph is a pure copy,
so there is nothing to divide and nothing to disagree; the drift is invisible
to an alignment check by construction.
The verification stage (`align.py --verify`) finds it. It starts at a
sentence fragment - the only place the drift is possible - and follows the
run until the passage re-synchronises. The verdict is one of three:
| verdict | meaning | action |
|---|---|---|
| `aligned` | the translation renders this paragraph | keep |
| `incomplete` | it renders this paragraph but leaves part out | keep, flag for review |
| `drift` | it renders another paragraph's content | drop, with a reason |
Only `drift` deletes a line, and the paragraphs are judged as a **window**
rather than one at a time. Both of those choices are measured, not assumed:
- The first version asked two yes/no questions per paragraph ("is anything
missing", "is anything extra") and was wrong in both directions. It
flagged five paragraphs that were fine - it reported the adverb `then` as
missing from a translation that contains it, and `Enthusiastically` as
missing from a translation that contains it - and it missed two paragraphs
that are drifted, because those paragraphs are short and their Chinese
looks plausible when seen alone. **Five of fifteen flags were wrong.**
- A re-flowed passage cannot be judged one paragraph at a time. The model
calls the two paragraphs *before* a real drift "aligned" when their Chinese
is demonstrably the neighbouring paragraph's, and the start of a drift is
not knowable from any single paragraph. So inside a window that contains a
confirmed drift, a fragment the model called `aligned` is dropped as well.
On the real ch-15 window this now finds exactly the drift: **ten paragraphs
dropped (p0095-p0104)**, and p0094 kept and flagged because its Chinese is a
correct translation of its first two sentences. An earlier version dropped
p0094 too - losing two good lines - and kept the two drifted paragraphs the
model wrongly called aligned.
The cost of the choice is explicit. A fragment inside a broken window can be
dropped on an unreliable verdict, so the audiobook has a gap there. The
alternative is a line in the audiobook that belongs to a different sentence,
and a gap is visible while a wrong line is silent.
A dropped paragraph still produces a row: `voiced = false` with the reason,
so the gap says *why* it is there rather than looking like a paragraph
nobody translated.
**The root cause is upstream.** The drafter should be told that each id's
translation must cover that id's English only, even where that leaves the
Chinese reading as a fragment. Fixing the drafter is a re-translation; until
then the verification stage is what stands between the drifted Chinese and
the audiobook.
### The pipeline is Chinese-only, and one production says otherwise
Measured on oldbox (2026-11-06), the second production directory is