umbragit/UmbraAudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Analysis-quality plan: why the ch6 speakers were missed (spec 24)

Browse Source 
The operator reviewed ch6 with author knowledge and found real defects the
duplicate scan could not see. Two corrections and one structural diagnosis.

I was wrong about two pairs. I wrote "probably one person" about The Officer /
Officer, and "probably two different riders" about Border Soldier / Soldier
(Rider). Both are two different characters. Measured, The Officer (113.3 Hz) and
Officer (104.5 Hz) are 1.4 SEMITONES apart - two distinct characters with
near-identical voices, which is a genuine cast-clarity risk that a merge would
have hidden. Border Soldier and Soldier (Rider) are 12.2 semitones apart, so
their risk is narrative proximity, not vocal.

The real lesson is the principle behind it: because the author wrote the book he
knows these facts, and on a book he did not write nobody does. So every merge
decision must be resolvable from evidence in the text, and the default when
evidence is absent must be SEPARATION. Keeping two speakers separate costs one
redundant voice; merging two people costs a wrong character for the rest of the
book. Reworded the scan so a blocked group is reported as "same name-space,
different characters unproven" and never as a probable merge.

Why the scan missed ch6: it can only compare names that already exist. It cannot
see a line attributed to the wrong EXISTING speaker. ch6 utt29 - "The leader on
the right for Hallr says," / "We want to be the ones to claim we killed the last
dwarf" - is a scum boy attributed to Hallr. No duplicate existed to find. The
scum group was also missed on identity: only the case variants were caught, and
nothing compares a designation against the text that produced it.

Diagnosis, measured book-wide: a role speech-tag must not resolve to a distinct
named character. 20 dialogue utterances violate it. True positives include ch11
utt191-193 where ONE tag ("The young man replies,") was split across Kaleb,
Tobias and Hallr, and ch13 utt45 where "The girl replies," became The King. But
40% are false positives because ROLE_WORDS is incomplete - royal, king, witch,
doctor and others are missing - and a gate that noisy would be ignored. The
vocabulary must come from definitions.py, not a hand-written list.

Plan, model-independent first: (A) mechanical evidence-coverage gate, which
catches this class with certainty; (B) constrained assignment - choose from the
evidence-supported candidates only, not all 93 names, which is the actual cause
of utt29; (C) role-phrase preservation so "the leader ON THE RIGHT" stops
acquiring a new name per mention; (D) tag-run consistency, never split one
introduction across three characters; (E) a segmentation gate for narration typed
as dialogue (ch6 utt26); (F) confusability signals, a casting concern distinct
from identity; (G) an evaluation harness with a ch6 gold set and five metrics,
because without a number "use a better model" is a guess.

On a better model, honestly: 12b cannot fit in 16 GB, and the GPU box at
192.168.123.14:1234 refused a connection today (curl exit 7), so it is blocked.
And B removes the search space first, which is where the error actually was.

Full suite unchanged at 575 passed / 8 failed, the same 8 pre-existing failures.
This commit is contained in:
B.D. Murphy
2026-10-04 20:01:05 -07:00
parent 216bc78a3a
commit 60564cc056
3 changed files with 279 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

275
specs/24-analysis-quality-plan.md Normal file
View File

@@ -0,0 +1,275 @@
# Spec 24 — Analysis Quality: Why the Scan Missed the ch6 Speakers, and How the Analysis Gets Better
- **Status:** proposal. Two corrections to spec 23, one structural diagnosis, and a ranked plan.
- **Date:** 2026-10-04
- **Trigger:** operator review of `Last Dwarf` chapter 6 with author knowledge.
- **Depends on:** `specs/23` (duplicate detection), `specs/17` (clarity), `manifest.md`.
---
## 1. Two corrections to spec 23, and the principle behind them
### 1.1 I was wrong about two pairs
| group | I wrote | The operator, who wrote the book | Measured consequence |
|---|---|---|---|
| `The Officer` / `Officer` | "probably one person" | **two different characters** | their voices are **113.3 Hz vs 104.5 Hz — 1.4 semitones apart**. Two distinct characters that are acoustically near-identical. Merging them would have destroyed a real distinction and hidden a genuine cast-clarity risk. |
| `Border Soldier` / `Soldier (Rider)` | "probably two different riders — my guess is they should *not* merge" | **two different characters**, and "close enough in the book that a listener may be confused" | **12.2 semitones apart** (107.5 vs 217.6 Hz). Acoustically fine. The confusion risk is **narrative**, not vocal. |
The lesson is not "trust the author over the scan". It is:
> **A duplicate scan proposes identity. It never decides it. Any suggestion phrased as
> "probably one person" is a claim the system cannot support from the text, and must not be
> phrased that way.**
Every `blocked` group should be reported as *"same name-space, different characters
unproven"* — never as a probable merge. I will reword the scan's output accordingly.
### 1.2 The principle, which is the real issue
> *"Because I wrote the book I know this information. If I put in a book I didn't write this
> could be a problem."*
This is the most important sentence in the review, and it invalidates the workflow I built.
Spec 23 ends with five questions addressed to the operator. On this book the author answers
them from memory in one pass. On a book he did not write, **nobody can**, and the questions
are unanswerable by anyone.
So the design requirement is not "make the scan better". It is:
> **Every merge decision must be resolvable from evidence in the text, or it must be
> defaulted to "keep them separate".**
Keeping two speakers separate costs one redundant voice. Merging two people costs a wrong
character for the rest of the book. **The default when evidence is absent is separation.** The
scan already blocks; it must also stop *leaning*, in wording and in survivor heuristics.
One legitimate exception the operator supplied: *"separated by multiple chapters the voice can
be shared."* A designation that appears once in chapter 6 and the same designation in chapter
30 is plausibly one person, because the author is consistent about anonymous roles within a
scene and inconsistent across scenes. Chapter separation is therefore **positive** evidence
for sameness, which is the opposite of what my "single-chapter" heuristic assumed. Encoded in
§3.4.
## 2. Why the scan did not flag the ch6 speakers
Not a threshold problem. A structural one:
> **The duplicate scan can only compare names that already exist. It cannot see a line
> attributed to the wrong *existing* speaker.**
Chapter 6, the scene the operator flagged:
| utt | narration tag | assigned speaker | should be |
|---|---|---|---|
| 23–25 | "The apparent leader bends down… He says," | `scum leader` | correct |
| **29** | **"The leader on the right for Hallr says,"** | **`Hallr`** | **a scum boy — missed** |
| 31–32 | "The leader says," | `scum leader` | correct |
| 33–34 | "The one on the left says," | `Scum guy1` | correct (hand-created) |
| 35–36 | "The big one says," | `scum country boy` | name does not match the tag |
| 26 | *(narration, typed as dialogue)* | `Narrator` | segmentation defect |
**utt 29** is the finding. "We want to be the ones to claim we killed the last dwarf" is spoken
by a scum boy, not by Hallr. `Hallr` is an existing profile, so no duplicate existed to find —
the line was mis-assigned to a real character and the dedup scan is blind to that by
construction.
The scum group was also a partial miss on identity: `scum leader` (lowercase) and
`scum country boy` were only flagged as **case variants**. That they are all one group of
unnamed boys — or at least that "country boy" is the wrong name for "the big one" — was not
detected, because nothing compares a **designation against the text that produced it**.
## 3. The measurement, and what it generalises to
I ran one mechanical rule across the book: **a role speech-tag must not resolve to a distinct
named character.** Result: **20 dialogue utterances violate it.**
### True positives — real defects
| location | tag | assigned | why it is wrong |
|---|---|---|---|
| ch11 utt 191, 192, 193 | "The young man replies," | `Kaleb`, `Tobias`, `Hallr` | **one tag, three different named characters.** A single introduction cannot introduce three people. |
| ch13 utt 45 | "The girl replies," | `The King` | a girl speaking, attributed to the King |
| ch10 utt 300 | "The woman says," | `Hallr` | Hallr is not "the woman" |
| ch7 utt 200 | "The man asks," | `Hallr` | plausible, but resolvable only from context |
| ch8 utt 264, 265 | "says the oldest / The boy says," | `Feliu`, `Merial` | the utterance *is* the name being called, so the speaker is the caller, not the callee |
### False positives — the rule needs a complete role vocabulary
| location | tag | assigned | actually fine because |
|---|---|---|---|
| ch14 utt 31 | "The guard says to Tobias," | `Royal Guard` | "Royal Guard" **is** a role; my role list had no `royal` |
| ch23 utt 42, 43 | "The guard says," | `Guard (Scene 2)` | same class of false positive |
| ch42 utt 12 | "The voice asks," | `Voice (Coven)` | `Voice (Coven)` is a role |
So the rule is sound but **noisy at a 40% false-positive rate**, because `ROLE_WORDS` is
incomplete (`king`, `royal`, `queen`, `witch`, `doctor`, `dragon`, `troll` … are missing). A
gate that is 40% noisy will be ignored, which is worse than no gate. The vocabulary must come
from `definitions.py` (spec 21 §5.4) rather than a hand-written list.
### What this tells us about the process
The attributor already has a verify pass — but **it asks the same class of model whether its
own guess is right.** That is circular. A guess from the full cast of 93 names, checked by the
same model, will look confident. Every defect above passed that check.
> **The gate has to be mechanical and it has to be independent of the model.** A rule that
> cannot be argued with beats a second opinion from the same mind.
## 4. The plan
Ordered by value per unit of effort. Items A–C need no model and no infrastructure; the
biggest single improvement (a larger model) is **currently unavailable** — see §5.
### A. Mechanical evidence-coverage gate — *do first*
Every dialogue line must be supported by exactly one of:
1. a **name tag** that matches its speaker, or
2. a **role tag** whose designation matches the speaker's name or an alias, or
3. a **pronoun tag** resolvable to exactly **one** candidate in the scene.
Anything else is `needs_review` with `reason: unsupported_attribution`, and **may not be
silently assigned to a named character.**
This is the rule that catches ch6 utt29, ch10 utt300 and ch13 utt45 with certainty, before any
model is consulted. `specs/17` §3.3 already has the pronoun-uniqueness half; the missing half is
"name tag / role tag must support the assignment".
- Extends `umbraaudio/analysis/attribution_qa.py`; a new issue type
`role_tag_named_speaker`.
- Vocabulary comes from `definitions.py`, not a literal.
- **Acceptance:** the 20 candidates above are classified, and every false positive is explained
by a missing role word rather than by a wrong call.
### B. Constrained assignment — *do second, biggest real win*
The attributor currently chooses from **all 93 names**. That is why a scum boy's line became
Hallr: Hallr was available.
Change it to choose from the **evidence-supported candidate set only**. For a role tag, the
candidate set is the speakers whose designation matches the tag; for a pronoun tag, the
in-scene candidates of that gender; for a name tag, that one name. If the set is empty, the
line is flagged as **unassigned** — it does not get a guess.
This is more effective than a bigger model, because the error was an unconstrained search, not
a weak one. It also makes the failure mode honest: "I don't know who this is" instead of a
confident wrong answer.
### C. Role-phrase preservation and crowd inventory
- The designation scanner must capture the **whole** role phrase, including qualifiers:
"the leader **on the right**", "the one **on the left**", "the big one". It currently drops
the qualifier, so the same character acquires a new name per mention style.
- Per scene, enumerate the distinct role phrases that introduce speech. That list **is** the
scene's unnamed cast, and it is mechanical.
- **Coverage assertion:** every dialogue line in the scene maps either to a named character or
to a member of that scene's role list. A line that maps to neither is the defect class.
### D. Tag-run consistency
When one narration tag introduces **consecutive** lines ("The young man replies," ×3), either
all of them resolve to one character or the text is ambiguous — but they must **never** be split
across three different named characters (ch11 utt191–193). Splitting a single introduction is a
hard error, not a judgement call.
### E. Segmentation gate for narration typed as dialogue
ch6 utt26 is narration stored with `type='dialogue'` (", Hallr, with his better low-light
vision, is looking at four young humans…"). It carries no quotation marks and reads as
narration. A mechanical check — `dialogue` rows must contain quoted or contracted speech —
flags it. Currently it also consumes an attribution decision, which is how the passage got
mis-aligned in the first place.
### F. A listener-confusability signal — *new, and distinct from duplicates*
The operator raised a different problem: two characters that are **correctly separate** but
that a listener may confuse. Measured on `Last Dwarf`:
| pair | verdict |
|---|---|
| `The Officer` (113.3 Hz) / `Officer` (104.5 Hz) | **1.4 semitones — real risk.** Two different characters, near-identical voices. |
| `Border Soldier` / `Soldier (Rider)` | 12.2 semitones — acoustically fine; the risk is narrative co-presence. |
Two new classes, both distinct from duplicate detection:
- `cast_acoustic_conflict` — different characters whose median pitch is within ~2 semitones.
Action: re-cast one, or accept with an explicit note.
- `cast_narrative_proximity` — different characters of the same role that appear in the same
scene. Action: a reviewer note, and a check that the text distinguishes them clearly.
Neither is a merge proposal. Both are **casting** decisions.
### G. Evaluation harness — the thing that makes the next iteration real
> *"If the analysis process is evaluated, the model used considered the next evaluation needs
> to be the improvement of the analysis."*
Correct, and the missing piece is a **metric**. There is no number today, so "use a better
model" is a guess and "the prompt got worse" is indistinguishable from noise.
Build a gold-set evaluation:
- **Fixture:** chapter 6 (`Tattered Dwarf`) with the author's known cast and the known
utterance→speaker map. Cheap to produce, high signal, and it is the chapter that exposed the
defect.
- **Metrics:**
- *coverage* — fraction of dialogue lines whose speaker is a known cast member;
- *missed-speaker rate* — lines belonging to a character with no profile (the ch6 failure);
- *precision* — of the lines assigned, how many are correct;
- *unsupported-assignment rate* — how many assignments fail gate A (§4A);
- *duplicate rate* — profiles that should be one person (spec 23), reported separately so it
cannot mask a coverage failure.
- **Every prompt, model, or pipeline change is scored against it.** A change ships only if it
improves coverage **without** reducing precision.
- A second fixture chapter should be added before trusting any number, or the harness will
overfit to one chapter's crowd.
## 5. On using a better model — honest status
| option | status |
|---|---|
| `gemma-4-12b` locally | **Impossible.** Needs ~16.9 GB; the machine has 16 GB unified. |
| A larger model on the GPU box | **Blocked right now.** `192.168.123.14:1234` refused a connection on 2026-10-04 (`curl` exit 7). Per `AGENTS.md`, that box's services are manually started and currently down. |
| Bigger model generally | Worth it *after* B. Constrained assignment removes the search space; a larger model on a constrained task is a modest gain, while on the current unconstrained task it is the whole problem. |
**Recommendation: do A–E first, and re-measure.** If gate A produces a clean unsupported list,
that list *is* the work a bigger model would be doing — and it is work a bigger model would
still get wrong without the evidence constraint.
## 6. Reviewer independence, stated as an acceptance criterion
For any book, authored or not:
1. every dialogue line resolves to a known speaker **or** is visibly unassigned;
2. no two profiles are merged on a guess — the default is separation;
3. every surviving "same person?" question is answerable **from the text** or is left open;
4. a reviewer with no author knowledge can clear the queue.
Criterion 4 is the one to test, and the only honest way to test it is to run the harness on a
book nobody in the project has read.
## 7. What I would do first, concretely
| # | item | effort | blocks |
|---|---|---|---|
| 1 | Gate A: `role_tag_named_speaker` using `definitions.py` vocabulary | 3–4 h | everything else, because it produces the metric |
| 2 | Harness: chapter 6 gold set + the five metrics | 2–3 h | any claim that a change helped |
| 3 | Constrained assignment (B) | 1–2 days | the actual miss rate |
| 4 | Role-phrase preservation (C) | 1 day | future crowd scenes |
| 5 | Tag-run consistency (D) | 2 h | ch11-style splits |
| 6 | Segmentation gate (E) | 2 h | ch6 utt26 and its neighbours |
| 7 | Confusability signals (F) | 3 h | casting, not identity |
| 8 | Score A–E; only then revisit the model | — | §5 |
## 8. Open, for the operator
1. **Scum group.** Is "the apparent leader", "the leader on the right", "the one on the left"
and "the big one" one person or four? The text is ambiguous; your call is authoritative and
becomes the ch6 gold label.
2. **"scum country boy" for "the big one"** — wrong name, or is that the boy who later goes home
to his country? The tag and the name disagree.
3. **ch8 utt264/265** — "Feliu." / "Merial." are the oldest boy calling the others by name.
Should a line that *is* a name always be attributed to the caller?
4. **The Officer / Officer** — 1.4 semitones apart, two different characters. Re-cast one, or
accept?

1
specs/README.md
View File

@@ -25,6 +25,7 @@ Index of all specification documents for the UmbraAudio audiobook generation sys
| [21-umbra-system-nodes-tasks-and-changes.md](21-umbra-system-nodes-tasks-and-changes.md) | System | Worker nodes, task assignment, cross-product change management, UI topology |
| [22-releases-and-the-publish-flag.md](22-releases-and-the-publish-flag.md) | Release | The publish flag, pinned artifact sets, post-publication change, and the Deal Hunter lesson |
| [23-speaker-duplicate-detection.md](23-speaker-duplicate-detection.md) | Speakers | Duplicate-speaker scan, approval-safe merging, and the crowd veto |
| [24-analysis-quality-plan.md](24-analysis-quality-plan.md) | Analysis | Why the ch6 speakers were missed, and a ranked plan to improve attribution |
## Quick Reference

4
umbraaudio/analysis/speaker_dedup.py
View File

@@ -454,7 +454,9 @@ def classify(group: list[SpeakerFacts], score: float) -> tuple[str, list[str]]:
if len(speaking) > 1:
if all(f.is_designation for f in speaking):
reasons.append(
"two designations both have lines; likely one person, but both were heard"
"two designations both have lines. SAME NAME-SPACE, DIFFERENT CHARACTERS "
"UNPROVEN - both were heard, so keep them separate unless the text says "
"otherwise (spec 24 section 1)"
)
else:
reasons.append("both members have lines and are not both designations")