UmbraAudio/specs/24-analysis-quality-plan.md

Spec 24 — Analysis Quality: Why the Scan Missed the ch6 Speakers, and How the Analysis Gets Better

• Status: proposal. Two corrections to spec 23, one structural diagnosis, and a ranked plan.
• Date: 2026-10-04
• Trigger: operator review of Last Dwarf chapter 6 with author knowledge.
• Depends on: specs/23 (duplicate detection), specs/17 (clarity), manifest.md.

---

1. Two corrections to spec 23, and the principle behind them

1.1 I was wrong about two pairs

group	I wrote	The operator, who wrote the book	Measured consequence
The Officer / Officer	"probably one person"	two different characters	their voices are 113.3 Hz vs 104.5 Hz — 1.4 semitones apart. Two distinct characters that are acoustically near-identical. Merging them would have destroyed a real distinction and hidden a genuine cast-clarity risk.
Border Soldier / Soldier (Rider)	"probably two different riders — my guess is they should not merge"	two different characters, and "close enough in the book that a listener may be confused"	12.2 semitones apart (107.5 vs 217.6 Hz). Acoustically fine. The confusion risk is narrative, not vocal.

The lesson is not "trust the author over the scan". It is:

A duplicate scan proposes identity. It never decides it. Any suggestion phrased as "probably one person" is a claim the system cannot support from the text, and must not be phrased that way.

Every blocked group should be reported as "same name-space, different characters unproven" — never as a probable merge. I will reword the scan's output accordingly.

1.2 The principle, which is the real issue

"Because I wrote the book I know this information. If I put in a book I didn't write this could be a problem."

This is the most important sentence in the review, and it invalidates the workflow I built. Spec 23 ends with five questions addressed to the operator. On this book the author answers them from memory in one pass. On a book he did not write, nobody can, and the questions are unanswerable by anyone.

So the design requirement is not "make the scan better". It is:

Every merge decision must be resolvable from evidence in the text, or it must be defaulted to "keep them separate".

Keeping two speakers separate costs one redundant voice. Merging two people costs a wrong character for the rest of the book. The default when evidence is absent is separation. The scan already blocks; it must also stop leaning, in wording and in survivor heuristics.

One legitimate exception the operator supplied: "separated by multiple chapters the voice can be shared." A designation that appears once in chapter 6 and the same designation in chapter 30 is plausibly one person, because the author is consistent about anonymous roles within a scene and inconsistent across scenes. Chapter separation is therefore positive evidence for sameness, which is the opposite of what my "single-chapter" heuristic assumed. Encoded in §3.4.

2. Why the scan did not flag the ch6 speakers

Not a threshold problem. A structural one:

The duplicate scan can only compare names that already exist. It cannot see a line attributed to the wrong existing speaker.

Chapter 6, the scene the operator flagged:

utt	narration tag	assigned speaker	should be
23–25	"The apparent leader bends down… He says,"	scum leader	correct
29	"The leader on the right for Hallr says,"	Hallr	a scum boy — missed
31–32	"The leader says,"	scum leader	correct
33–34	"The one on the left says,"	Scum guy1	correct (hand-created)
35–36	"The big one says,"	scum country boy	name does not match the tag
26	(narration, typed as dialogue)	Narrator	segmentation defect

utt 29 is the finding. "We want to be the ones to claim we killed the last dwarf" is spoken by a scum boy, not by Hallr. Hallr is an existing profile, so no duplicate existed to find — the line was mis-assigned to a real character and the dedup scan is blind to that by construction.

The scum group was also a partial miss on identity: scum leader (lowercase) and scum country boy were only flagged as case variants. That they are all one group of unnamed boys — or at least that "country boy" is the wrong name for "the big one" — was not detected, because nothing compares a designation against the text that produced it.

3. The measurement, and what it generalises to

I ran one mechanical rule across the book: a role speech-tag must not resolve to a distinct named character. Result: 20 dialogue utterances violate it.

True positives — real defects

location	tag	assigned	why it is wrong
ch11 utt 191, 192, 193	"The young man replies,"	Kaleb, Tobias, Hallr	one tag, three different named characters. A single introduction cannot introduce three people.
ch13 utt 45	"The girl replies,"	The King	a girl speaking, attributed to the King
ch10 utt 300	"The woman says,"	Hallr	Hallr is not "the woman"
ch7 utt 200	"The man asks,"	Hallr	plausible, but resolvable only from context
ch8 utt 264, 265	"says the oldest / The boy says,"	Feliu, Merial	the utterance is the name being called, so the speaker is the caller, not the callee

False positives — the rule needs a complete role vocabulary

location	tag	assigned	actually fine because
ch14 utt 31	"The guard says to Tobias,"	Royal Guard	"Royal Guard" is a role; my role list had no royal
ch23 utt 42, 43	"The guard says,"	Guard (Scene 2)	same class of false positive
ch42 utt 12	"The voice asks,"	Voice (Coven)	Voice (Coven) is a role

So the rule is sound but noisy at a 40% false-positive rate, because ROLE_WORDS is incomplete (king, royal, queen, witch, doctor, dragon, troll … are missing). A gate that is 40% noisy will be ignored, which is worse than no gate. The vocabulary must come from definitions.py (spec 21 §5.4) rather than a hand-written list.

What this tells us about the process

The attributor already has a verify pass — but it asks the same class of model whether its own guess is right. That is circular. A guess from the full cast of 93 names, checked by the same model, will look confident. Every defect above passed that check.

The gate has to be mechanical and it has to be independent of the model. A rule that cannot be argued with beats a second opinion from the same mind.

4. The plan

Ordered by value per unit of effort. Items A–C need no model and no infrastructure; the biggest single improvement (a larger model) is currently unavailable — see §5.

A. Mechanical evidence-coverage gate — do first

Every dialogue line must be supported by exactly one of:

1. a name tag that matches its speaker, or
2. a role tag whose designation matches the speaker's name or an alias, or
3. a pronoun tag resolvable to exactly one candidate in the scene.

Anything else is needs_review with reason: unsupported_attribution, and may not be silently assigned to a named character.

This is the rule that catches ch6 utt29, ch10 utt300 and ch13 utt45 with certainty, before any model is consulted. specs/17 §3.3 already has the pronoun-uniqueness half; the missing half is "name tag / role tag must support the assignment".

• Extends umbraaudio/analysis/attribution_qa.py; a new issue type role_tag_named_speaker.
• Vocabulary comes from definitions.py, not a literal.
• Acceptance: the 20 candidates above are classified, and every false positive is explained by a missing role word rather than by a wrong call.

B. Constrained assignment — do second, biggest real win

The attributor currently chooses from all 93 names. That is why a scum boy's line became Hallr: Hallr was available.

Change it to choose from the evidence-supported candidate set only. For a role tag, the candidate set is the speakers whose designation matches the tag; for a pronoun tag, the in-scene candidates of that gender; for a name tag, that one name. If the set is empty, the line is flagged as unassigned — it does not get a guess.

This is more effective than a bigger model, because the error was an unconstrained search, not a weak one. It also makes the failure mode honest: "I don't know who this is" instead of a confident wrong answer.

C. Role-phrase preservation and crowd inventory

• The designation scanner must capture the whole role phrase, including qualifiers: "the leader on the right", "the one on the left", "the big one". It currently drops the qualifier, so the same character acquires a new name per mention style.
• Per scene, enumerate the distinct role phrases that introduce speech. That list is the scene's unnamed cast, and it is mechanical.
• Coverage assertion: every dialogue line in the scene maps either to a named character or to a member of that scene's role list. A line that maps to neither is the defect class.

D. Tag-run consistency

When one narration tag introduces consecutive lines ("The young man replies," ×3), either all of them resolve to one character or the text is ambiguous — but they must never be split across three different named characters (ch11 utt191–193). Splitting a single introduction is a hard error, not a judgement call.

E. Segmentation gate for narration typed as dialogue

ch6 utt26 is narration stored with type='dialogue' (", Hallr, with his better low-light vision, is looking at four young humans…"). It carries no quotation marks and reads as narration. A mechanical check — dialogue rows must contain quoted or contracted speech — flags it. Currently it also consumes an attribution decision, which is how the passage got mis-aligned in the first place.

F. A listener-confusability signal — new, and distinct from duplicates

The operator raised a different problem: two characters that are correctly separate but that a listener may confuse. Measured on Last Dwarf:

pair	verdict
The Officer (113.3 Hz) / Officer (104.5 Hz)	1.4 semitones — real risk. Two different characters, near-identical voices.
Border Soldier / Soldier (Rider)	12.2 semitones — acoustically fine; the risk is narrative co-presence.

Two new classes, both distinct from duplicate detection:

• cast_acoustic_conflict — different characters whose median pitch is within ~2 semitones. Action: re-cast one, or accept with an explicit note.
• cast_narrative_proximity — different characters of the same role that appear in the same scene. Action: a reviewer note, and a check that the text distinguishes them clearly.

Neither is a merge proposal. Both are casting decisions.

G. Evaluation harness — the thing that makes the next iteration real

"If the analysis process is evaluated, the model used considered the next evaluation needs to be the improvement of the analysis."

Correct, and the missing piece is a metric. There is no number today, so "use a better model" is a guess and "the prompt got worse" is indistinguishable from noise.

Build a gold-set evaluation:

• Fixture: chapter 6 (Tattered Dwarf) with the author's known cast and the known utterance→speaker map. Cheap to produce, high signal, and it is the chapter that exposed the defect.
• Metrics:
  • coverage — fraction of dialogue lines whose speaker is a known cast member;
  • missed-speaker rate — lines belonging to a character with no profile (the ch6 failure);
  • precision — of the lines assigned, how many are correct;
  • unsupported-assignment rate — how many assignments fail gate A (§4A);
  • duplicate rate — profiles that should be one person (spec 23), reported separately so it cannot mask a coverage failure.
• Every prompt, model, or pipeline change is scored against it. A change ships only if it improves coverage without reducing precision.
• A second fixture chapter should be added before trusting any number, or the harness will overfit to one chapter's crowd.

5. On using a better model — honest status

option	status
gemma-4-12b locally	Impossible. Needs ~16.9 GB; the machine has 16 GB unified.
A larger model on the GPU box	Blocked right now. 192.168.123.14:1234 refused a connection on 2026-10-04 (curl exit 7). Per AGENTS.md, that box's services are manually started and currently down.
Bigger model generally	Worth it after B. Constrained assignment removes the search space; a larger model on a constrained task is a modest gain, while on the current unconstrained task it is the whole problem.

Recommendation: do A–E first, and re-measure. If gate A produces a clean unsupported list, that list is the work a bigger model would be doing — and it is work a bigger model would still get wrong without the evidence constraint.

6. Reviewer independence, stated as an acceptance criterion

For any book, authored or not:

1. every dialogue line resolves to a known speaker or is visibly unassigned;
2. no two profiles are merged on a guess — the default is separation;
3. every surviving "same person?" question is answerable from the text or is left open;
4. a reviewer with no author knowledge can clear the queue.

Criterion 4 is the one to test, and the only honest way to test it is to run the harness on a book nobody in the project has read.

7. What I would do first, concretely

#	item	effort	blocks
1	Gate A: role_tag_named_speaker using definitions.py vocabulary	3–4 h	everything else, because it produces the metric
2	Harness: chapter 6 gold set + the five metrics	2–3 h	any claim that a change helped
3	Constrained assignment (B)	1–2 days	the actual miss rate
4	Role-phrase preservation (C)	1 day	future crowd scenes
5	Tag-run consistency (D)	2 h	ch11-style splits
6	Segmentation gate (E)	2 h	ch6 utt26 and its neighbours
7	Confusability signals (F)	3 h	casting, not identity
8	Score A–E; only then revisit the model	—	§5

8. Open, for the operator

1. Scum group. Is "the apparent leader", "the leader on the right", "the one on the left" and "the big one" one person or four? The text is ambiguous; your call is authoritative and becomes the ch6 gold label.
2. "scum country boy" for "the big one" — wrong name, or is that the boy who later goes home to his country? The tag and the name disagree.
3. ch8 utt264/265 — "Feliu." / "Merial." are the oldest boy calling the others by name. Should a line that is a name always be attributed to the caller?
4. The Officer / Officer — 1.4 semitones apart, two different characters. Re-cast one, or accept?

---

9. Addendum — item A is built and measured (2026-10-04)

umbraaudio/analysis/speech_evidence.py and scripts/report_speech_evidence.py implement §4A. Read-only, model-free, 44 tests. Measured on Last Dwarf, all 49 chapters:

	lines	share
supported	1,844	47.1%
unconstrained (no tag applies — mid-conversation, action or thought narration)	2,032	51.9%
contradicted	34	0.87%
unsupported (a tag names someone with no profile — the missed-speaker signal)	8	0.20%
review queue	42	1.07%

The design decision that made it usable

The first working version put 60% of the book in the queue, because it reported every line with no adjacent speech tag as a finding. Most dialogue in a novel has no tag: once a conversation starts, only the first line is introduced. So "no tag" was split into two verdicts that are not findings:

• unconstrained — the line sits inside a conversation, or the preceding narration is an action or a thought ("He looks at the dried meat, thinking,").
• unsupported — a tag exists and names somebody the cast does not contain.

And ambiguous was removed as a verdict. It was hiding the important case: a line assigned to Hallr under the tag "The leader on the right" is wrong no matter how many leaders exist, so the verdict is contradicted whenever the assignment is outside a non-empty candidate set. How many candidates exist now rides along as candidate_count, where it decides whether the suggested fix is confident or a question — which is where it belonged.

What it caught

The chapter 6 defect, exactly:

chapter_id=6 (index 5) 'Tattered Dwarf'
  utt29  CONTRADICTED  speaker='Hallr'
  narration : The leader on the right for Hallr says,
  line      : "We want to be the ones to claim we killed the last dwarf."
  why       : tag 'The leader on the right' does not support 'Hallr';
              2 possible speaker(s): Scum leader, The Leader

Nothing else in the system flagged this line. The existing attribution_qa reported 11 findings book-wide; this reports 42, and its chapter 6 finding is not among the 11.

Corrections to §3, made by measuring

• §3 said 20 role-tag violations at a ~40% false-positive rate. The real number is 11, produced by the existing attribution_qa.role_tag_guessed. My ad-hoc probe was worse than the checker I was about to replace.
• Coverage of identity is already 100%. Every one of the 3,918 dialogue lines resolves to a profile or an alias. The failure is not a missing speaker; it is a line attributed to the wrong existing speaker. That is why §4A is an evidence gate and not a profile-completeness check.
• The verifier was circular. The attributor's verify pass asks the same class of model whether its own guess is right. Confirmed by every defect surviving it.

The six extractor bugs the measurement exposed

Each of these produced a wrong tag on real text, and each is now a regression test:

narration	tag before the fix	tag now
The leader on the right for Hallr says,	no tag at all	The leader on the right, addressee Hallr
The kid licks his lips, with a breath, says,	The kid licks his lips	The kid
Sheyla shakes her head, and Tanisha says,	Sheyla	Tanisha
, Hallr chuckles, and Milfred says,	Hallr	Milfred
The woman\u2019s voice says,	The woman\u2019s voice → matched Voice (Coven)	The woman
One of the six men says,	One of the six	One of the six men, head man

Two generalisable rules came out of them:

1. A speech tag's subject is a noun phrase, and a prepositional qualifier is still part of it. "The leader on the right" is one reference. Enumerating verbs to find where a reference stops never ends; the head noun is the boundary.
2. A coordinated predicate speaks as its last subject, and a subordinating conjunction opens a clause of its own — "The dragon swallows when Hallr says" is Hallr.

Still open

• ~8 of the 42 are residual false positives from hard narration: "Sheyla's Past Hallr says,", "He hurries to the horse's head, close to the dragon, and says,". Each is a real line to read, so they are not harmful — the queue is simply not 100% precise.
• The gate is not wired into attribution_qa or the review UI yet. It is a report.
• unsupported is the metric to watch: 8 lines book-wide name a speaker with no profile. That is the "missing speaker" count the operator asked about, and it is small here because earlier work already created most of them.
• §4B (constrained assignment) is still the biggest open win. The gate now measures the problem; it does not prevent it.

A measurement note worth keeping

Chapter references in every report carry chapter_id and the title, because chapter_id and chapter_index differ by one in every project in this repo. Reporting a bare "ch8 utt264" sent the operator to the wrong chapter: chapter 8 ("After the Battle") utt264 is Hallr and was never flagged; the flagged line is chapter 9 ("Peace With the Enemy") utt264.