UmbraAudio/specs/17-speaker-confidence-review.md

Spec 17 — Speaker Identification Confidence, Orphan Aliases, and Utterance Clarity Review

Status: Phase 1 implemented (2026-09-23) — backend clarity signals live; eval measured 4.4% queue on Last Dwarf. Phases 2-5 still planned. Date: 2026-09-23 Consumer: translation pipeline (requires highly accurate speaker identity data); audiobook voice assignment; future books by authors other than the user.

1. Goal

Design a review workflow where the system auto-accepts everything it can prove, and surfaces a small, high-value review queue (~1–5% of utterances) for the human. The reviewer should be able to fix 10 items, not re-review every utterance and change 50%. All identity/attribution data must be exportable with per-item confidence for a downstream translation system.

Requirements driving this design:

• Identify all speakers and all aliases of each speaker.
• A "review" section on the Speakers tab for any speaker that isn't high confidence.
• Utterance clarity: "He said" with an unclear referent should be flagged for review.
• The system finds all high-confidence attributions; the human helps with the rest.
• Future books are NOT written by the user — the reviewer cannot be assumed to know the cast. The pipeline must work when the reviewer has zero prior book knowledge.
• Data must be highly accurate for another system (translations).

2. Current-State Evaluation

2.1 What already exists (and works)

Capability	Where	Notes
4-pass LLM character discovery	character_discovery.py discover_speakers	speaking + non-speaking chars, aliases with alias_evidence, unidentified + unidentified_reason, chapters_appeared
Alias validation	validate_aliases	checks aliases appear in chapter text
Alias merge gating	_merge_chapter_results	_has_two_proper_names + _alias_has_evidence prevent garbage aliases
Mechanical narration evidence	llm_attributor.py	narr_before, tag, tag_kind (name/role/pronoun), addressees — extracted from the manuscript, not LLM guesses
Multi-pass attribution	_verify_attributions	up to 2 verify rounds, batched 20/call, invented-name gate, _normalize_speaker
Mechanical QA	attribution_qa.py	6 issue types; fix_high_confidence writes back only resolved exact-name mismatches
QA accept (durable)	utterances.qa_accepted + qa_accepted_sig	signature = hash(speaker,text), so edits invalidate acceptance
Per-utterance confidence	attribution_confidence	stored, but see §2.2 weaknesses
Orphaned utterances page	/orphaned-utterances	utterance-level: speaker_name not in known names → reassign
Exception review table+page	exception_reviews table, /exception-review	filters by ambiguity type + confidence, resolve modal, export CSV, quality gate
Speakers tab	/speakers, _speakers_list.html, _speaker_row.html	flag badge, add/remove alias, sample dialogue, per-row voice gen

2.2 What is weak or missing

1. Confidence is not trustworthy as-is. Distribution over 3,918 dialogue utterances (Last Dwarf): 1.0 → 2480 (63%), 0.9 → 879, 0.85 → 13, 0.8 → 173, 0.7 → 58, 0.6 → 17, ≤0.5 → 4. 94% sit at ≥0.9, and the LLM self-reports the number. There is no calibration against actual correctness, no decomposition into why it is confident (mechanical name tag vs. dialogue-content guess).

2. The pass-1 uncertain flag is transient. attribution_uncertain is set on in-memory dicts during attribution (llm_attributor.py:538) and consumed only by the verify pass (:682). It is never persisted — the DB has no such column, so the review UI cannot see which lines "had no mechanical evidence to lean on".

3. Speakers have no identification confidence. The only flag is _is_unknown_speaker (no vocal_traits + gender Unknown + empty description + Casual register) — a crude heuristic. Discovery already produces unidentified, unidentified_reason, alias_evidence, chapters_appeared, but none are surfaced as a confidence/certainty signal or as a speaker-level accept/review workflow. 7 of 66 Last Dwarf speakers are flagged unidentified, and many have zero aliases — no UI to resolve them.

4. No orphan-ALIAS concept. The orphaned-utterances page handles utterances whose speaker_name is unknown. There is no list of name tokens that appear in the text but are not attached to any speaker as primary name or alias (e.g., a name mentioned in narration that is never attributed a line, or a role title that has no profile). Discovery's non_speaking_characters capture some of this but they are not reconciled with the alias map in the UI.

5. No clarity classification per utterance. The system never labels an utterance as name-tagged / role-tagged / pronoun-tagged / content-guess at persistence time. "He said" with a pronoun that cannot be resolved to a unique recent character is a first-class ambiguity case the reviewer cares about, but nothing mechanically flags it.

6. Exception review page is orphaned from navigation. /exception-review exists but is not linked in base.html; its populate endpoint only inserts rows for attribution_confidence < 0.7, and it sniffs attribution_reasoning text for the words "pronoun"/"role" to classify ambiguity. It never auto-populates during attribution, and confidence <0.7 is 0.2% of the book — so it currently finds almost nothing.

7. DB default is dangerous. attribution_confidence REAL DEFAULT 1.0 means un-attributed rows silently look perfect.

8. Alias unification is a one-way street. Discovery creates speakers + aliases; manual alias add exists. But there is no flow to merge "this name is actually an alias of that speaker" driven by evidence (the same name token to which the LLM attributed lines in multiple chapters), and no automatic fix-up of attributed speaker_name when aliases are added/merged.

3. Design

3.1 Two axes of confidence

Keep them separate — they answer different questions and flow to the translation system differently.

A. Speaker identification confidence (speaker-level; stored on speakers.json): how sure we are that this speaker exists and that their alias set is complete. Derived from:

• unidentified / unidentified_reason (discovery)
• alias evidence strength: alias_evidence entries, chapters where the alias is attested
• chapters appeared with dialogue vs. only mentioned
• speaker-level attribution health: number of utterances, mean attribution confidence, count of needs_review utterances attributed to them
• soundex/word-overlap collision warnings: two speakers whose names/aliases are near-dups

B. Utterance attribution confidence (utterance-level; persisted in DB): composite of

• mechanical tag strength: name tag > role tag > pronoun tag > no tag (content guess). This is the strongest signal and is manuscript-derived, not LLM.
• LLM self-reported confidence (pass 1)
• verify-pass outcome: agreed / corrected / still-uncertain
• mechanical QA issues (attribution_qa)
• pronoun resolvability (see §3.3)

Composite → one of: HIGH (auto-accept), MEDIUM (review batch), LOW (must review). The review queue = MEDIUM + LOW ≈ the ~1–5% the user wants to see, not 50%.

3.2 Speaker page changes

Layout (all on /speakers):

1. Review section (top). "Speakers needing review" — any speaker with:

   • unidentified: true, or
   • identification confidence < threshold, or
   • alias-set incompleteness (name tokens in text that look like they belong to them — fed by the orphan-alias scanner §3.4), or
   • attribution-health problems (many LOW/MEDIUM utterances, tag-vs-speaker mismatches). Each row: speaker name, reason (machine-readable + short human text), sample dialogue, and actions:
   • Accept (sets reviewed: true, clears the flag; the tool remembers the decision)
   • Edit (existing speaker edit form)
   • Create speaker from an orphan alias / merge alias into this speaker (§3.4). Auto-accept is never silent: accepted speakers are recorded so a re-run does not re-flag.

2. Main list (existing table, enriched). Add per-speaker columns:

   • confidence bar (identification confidence)
   • utterance count + % attributed at HIGH
   • alias count + "alias coverage" (of name tokens attested for them in the text)
   • a small ✓/! review-status badge.

3. Orphan aliases panel (§3.4) — same page, below the speaker table.

3.3 Utterance clarity: confidence assessment + review-needed flagging

At attribution time, persist four new fields per utterance:

• attribution_evidence_kind: name | role | pronoun | address_only | none
• attribution_tag_name: the exact tag string from narration (e.g. "the lead guard")
• attribution_uncertain: the pass-1 flag, now persisted (0/1)
• needs_review: computed composite flag + review_reason TEXT

Mechanical determinism (no LLM needed):

• pronoun tag (He said/She said): resolve against the dialogue positions just before the tag, within the scene. If exactly one candidate of that gender is present in the recent window → assignment is HIGH-confidence (LLM still sets the name, but the machine has corroborated uniqueness). If zero or multiple candidates → flag needs_review with reason pronoun: N candidates. This is the "He said" case the user explicitly called out.

Role/name tags that match a profile/alias → HIGH; role tags with no matching profile → MEDIUM (review; likely a missing speaker → orphan alias / create speaker).

none / address_only (content-guess, incl. "The voice calls again") → MEDIUM only when confidence < 0.75 (implemented _NO_EVIDENCE_CONFIDENCE_FLOOR); verify-confirmed content-guess lines at 0.75-0.9 stay auto-accepted to keep the queue near 1-5%.

QA issues from attribution_qa.py (speech_tag_mismatch, narrator_dialogue, address_mismatch, narration_speaker_mismatch, etc.) push to needs_review regardless of confidence. long_monologue is deliberately excluded from the queue (weak 0.5 heuristic; intentional 4+ line speeches are common) but still reported by mechanical QA.

attribution_uncertain (pass-1 no-narration-evidence flag) → review only when confidence did not recover to ≥ 0.9 after the verify pass.

A MEDIUM/LOW item still keeps its attributed speaker — the reviewer corrects only when wrong; the queue is a triage list, not a "blank" list.

3.4 Orphan aliases: display and assignment

New derived concept: unresolved name tokens — proper-noun tokens appearing in chapter text/narration (from the existing segmentation, tag fields, and discovery pass) that are not in _all_speaker_names_and_aliases() and are not stopwords. Grouped by token:

Name token          | Occurrences | Chapters | Sample line             | Action
Trader Marek        | 12          | 14, 15   | "Trader Marek nodded." | [Assign ▾] [Create speaker]

Actions per token:

• Assign to existing speaker — dropdown (searchable) of speakers; on save: add token to that speaker's aliases with the evidence lines, then re-normalize every utterance whose speaker_name equals the token → the speaker's primary_name (updates speaker_name + bumps attribution_reasoning; audio markers already key on speaker so regen is automatic via staleness).
• Create new speaker — pre-filled modal (primary_name = token, aliases = [token], sample dialogue = first attributed line). Speaker enters the normal voice-design flow.
• Dismiss — mark as non-speaking / not-a-character (e.g., a place or a once-mentioned person). Recorded so it does not reappear.
• Merge two speakers — the reverse of assign: two profiles are one person. Keep the longer alias set, merge aliases, re-normalize utterances from the removed primary to the kept primary.

Feeds the speaker-review section: if an unresolved token looks like a variant of an existing speaker's name (word-overlap / soundex / prefix match ≥ threshold), surface it directly as "suggested alias of X" so the reviewer can accept in one click.

Everything recorded in speakers.json (aliases + alias_evidence) and in a new name_tokens table (token, status: assigned|created|dismissed|unresolved, target speaker_id, evidence lines) so re-runs are stable and the translation export can prove "every name in the text resolves to a speaker or was dismissed".

3.5 Review workflow & persistence

• Extend the existing exception_reviews table as the review queue (it already has the shape: chapter, utterance, current_speaker, confidence, ambiguity_type, evidence, alternatives, status). Re-populate it automatically at the end of attribution from needs_review utterances (no manual populate step), and link the page in base.html.
• Add speaker_name changes through the review UI → same re-normalization + regen path as §3.4.
• qa_accepted_sig already invalidates acceptance when speaker/text changes — reuse.
• The queue should be small by construction: only pronoun-ambiguity, role-no-profile, no-tag, address-only, and mechanical-QA-flagged lines. That is the user's "review 10, not 50%".

3.6 Translation export

New endpoint/CLI: export identity_export.json:

{
  "speaker_id": "trader-marek",           // stable slug, not display name
  "canonical_name": "Trader Marek",
  "aliases": [{"name": "Marek", "evidence_lines": [...]}],
  "identification_confidence": 0.96,
  "review_status": "accepted",            // accepted | needs_review | auto_high
  "utterances": [
    {"chapter": 14, "index": 41, "text": "...",
     "effective_speaker": "trader-marek",
     "confidence": 0.97, "evidence_kind": "name", "review_status": "auto_high"}
  ]
}

Rules:

• Every utterance maps to a stable speaker_id (never a raw display string).
• Every speaker carries identification confidence + review status.
• Every name token in the book is resolved or explicitly dismissed (the name_tokens table proves it).
• A benchmark/precision metric: per evidence_kind, the fraction of reviewer-confirmed items that were correct — lets the user see "name-tagged lines are 99% right, content-guesses are 60%".

3.7 New columns / tables

• utterances: attribution_evidence_kind TEXT, attribution_tag_name TEXT, attribution_uncertain INTEGER DEFAULT 0, needs_review INTEGER DEFAULT 0, review_reason TEXT (nullable), and fix attribution_confidence default → NULL.
• speakers.json per-speaker: identification_confidence REAL, review_status TEXT (auto_high|needs_review|accepted), reviewed_at TEXT, alias_coverage (derived, not stored).
• New table name_tokens (token, status, target_speaker, evidence_json, dismissed).
• Reuse exception_reviews as the review-queue table; auto-populate.

4. Implementation Plan

Phase 1 — Persist clarity signals (backend) ✅ implemented

1. DB: 5 columns added (attribution_evidence_kind, attribution_tag_name, attribution_uncertain, needs_review, review_reason); attribution_confidence default → NULL. Migrations idempotent in database.py._migrate, routes.py _ensure_utterance_columns, and scripts/reattribute_all.py._ensure_utterance_columns (the raw-sqlite driver used for chapter re-runs must add columns itself).
2. llm_attributor.py: attribute_speakers accumulates scene evidence across batches into evidence_by_idx and stamps attribution_evidence_kind (via clarity.evidence_kind_for) + attribution_tag_name onto returned utterance dicts.
3. New module umbraaudio/analysis/clarity.py:
   • extract_mechanical_evidence / evidence_kind_for: deterministic evidence kind (name|role|pronoun|address_only|none); addressee only counts when it resolves to a known name/title (guards against "But where…" false positives).
   • pronoun resolvability: _recent_candidates walks back in-scene (gender-filtered via speakers.json demographics); exactly 1 candidate → resolvable, else → review.
   • composite needs_review — triggers: any QA issue (long_monologue excluded), pronoun non-unique, role tag with no profile (speaker not in alias map), no/address-only evidence below 0.75 confidence, attribution_uncertain below 0.9, narrator-left dialogue, confidence < 0.7. review_reason is a "; "-joined readable summary.
4. Drivers persist everything: reattribute_all.py runs assess_utterance_clarity after attribution and writes all 8 columns; run_llm_attribution.py same via update_utterance_speaker; both web attribution paths in routes.py updated. _populate_exceptions now keys off needs_review (was confidence<0.7 + keyword sniff) and maps reason → ambiguity_type; fixed a latent NameError (pid not in scope).
5. Eval on Last Dwarf (pre-re-run confidences, 3918 dialogue): review queue = 173 (4.42%) — top signals: none-evidence low-conf (73), pronoun ambiguity (53), narration-speaker mismatch (18), address mismatch (5), role-no-profile (6), narrator-left (3). Long-monologue runs (43) deliberately NOT auto-queued (weak 0.5 heuristic; 4+ line speeches are commonly intentional) — still visible in QA reports.

Tests: tests/test_clarity.py — 25 tests (evidence kinds, address-only false-positive guard, pronoun 0/1/many candidates, role-with/without profile, QA coupling, uncertain flag persistence, narrator-left, low-confidence, non-dialogue passthrough, no-mutation). ruff + mypy clean on clarity.py, llm_attributor.py deltas, reattribute_all.py; routes.py unchanged baseline (73 pre-existing). Full suite 380 passed / 6 pre-existing voice_profile_manager accent failures. Migration smoke-tested against a copy of the real project DB. To refresh real data: re-run scripts/reattribute_all.py (adds columns + fills clarity fields for the reviewed chapters; qa_accepted rows are skipped as today).

Phase 2 — Orphan alias scanner + assignment (backend)

1. scripts/scan_name_tokens.py + umbraaudio/analysis/name_tokens.py: extract proper-noun tokens from chapter text/tags; diff against alias map; write name_tokens table; fuzzy-suggest existing speakers (word-overlap/soundex).
2. Assignment APIs: assign-to-speaker (add alias + evidence + re-normalize utterances), create-speaker, dismiss, merge-speakers. Reuse routes' _all_speaker_names_and_aliases and the alias-modal machinery.

Tests: token extraction precision/recall on a fixture chapter; assign→re-normalize→stale marker; merge idempotence.

Phase 3 — UI (web)

1. Speakers tab: review section (reason + Accept/Edit/Create/Assign), enriched columns, orphan-aliases panel with inline assign/create/dismiss.
2. Link /exception-review in base.html nav; row actions: speaker dropdown (searchable), context, resolve (reuse existing modals), auto-regen wiring.
3. Chapter view: per-utterance clarity badge (name/role/pronoun/none) + "why flagged" tooltip; mark needs_review items with the same amber styling as QA flags.

Manual verification: load a Last Dwarf chapter; confirm review queue ≈ expected small set; confirm every orphan alias action updates speakers.json + DB + markers.

Phase 4 — Translation-oriented metrics + export

1. identity_export endpoint + CLI (scripts/export_identity.py).
2. Precision-by-evidence-kind metric from reviewed items (requires reviewer accept counts — starts as "n/a" until first review batch lands).

Tests: export schema golden file; every utterance ✓ effective_speaker id; every token resolved-or-dismissed assertion.

Phase 5 — Validation on Last Dwarf

1. Re-run attribution for ch11–49 with clarity persistence (existing workflow, markers allow regen on output change; a full re-attribute run only writes DB fields, no audio regen unless speaker changes).
2. Report: review queue size, per-evidence-kind distribution, unresolved-token count, speaker identification confidences — vs. this document's targets (queue ~1–5%).
3. Commit cadence per AGENTS.md (Sunday commits; scripts/commit_check.sh).

5. Success Criteria

• Review queue: ≤5% of dialogue utterances, ≥80% of reviewer-corrections correspond to items the system flagged (precision), and ≥90% of genuine errors are in the queue (recall — measured against a gold set of the user's known books as label source).
• 100% of name tokens in the text resolve to a speaker or are explicitly dismissed.
• Translation export: deterministic, stable speaker_ids, no raw-name drift.
• Works with zero prior book knowledge: the pipeline can run end-to-end on a fixture book with no user corrections and still produce a usable export (some review items, none fatal).

6. Non-Goals / Deferred

• Automated resolution of pronoun ambiguity by extra LLM passes (verify rounds already cover it; keep the human loop).
• Auto-merging speakers without a human decision (only suggestions).
• Per-utterance audio regeneration automation beyond existing staleness markers.