UmbraAudio/specs/14-two-phase-segmentation.md

Spec: Two-Phase Segmentation (Pass 2)

Overview

Pass 2 segments chapter text into utterances (narration and dialogue) and attributes speakers using a two-phase approach: a mechanical segmenter handles text splitting, and an LLM attributes speakers to all dialogue. This produces a 0.05% dialogue-as-narrator error rate (1/1,882 dialogue utterances) across all 35 chapters.

Architecture

Raw chapter text
    │
    ▼
┌─────────────────────────────────────────────────┐
│  PHASE 1: MECHANICAL SEGMENTER                  │
│  mapping/segmenter.py                           │
│                                                 │
│  - Sentence splitting at .!? (not commas)       │
│  - Dialogue detection via quotes ("'""')        │
│  - Tag-based attribution (forward/backward)     │
│  - Scene break (<<<>>>) handling                 │
│  - Chapter title detection                      │
└─────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────┐
│  PHASE 2: LLM SPEAKER ATTRIBUTION               │
│  analysis/llm_attributor.py                     │
│                                                 │
│  - Attributes ALL dialogue (overrides guesses)  │
│  - Resolves pronouns using context              │
│  - Uses character profiles from speakers.json   │
│  - Batches of 40 utterances for efficiency      │
│  - Normalizes names (aliases/first-word,        │
│    narrator lowercase, pronoun guard)           │
└─────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────┐
│  OUTPUT: Utterances table                       │
│  4,339 utterances across 35 chapters            │
│  0.05% dialogue-as-narrator (1/1,882)           │
└─────────────────────────────────────────────────┘

Components

1. Mechanical Segmenter (mapping/segmenter.py)

Class: Segmenter

class Segmenter:
    def segment(
        self,
        text: str,
        chapter_id: int,
        character_names: list[str] | None = None,
    ) -> list[dict[str, Any]]

Returns:

[{
    "index": int,
    "text": str,
    "type": "narration" | "dialogue",
    "character_name": str | None,
}, ...]

Segmentation Pipeline:

Raw text
    │
    ├── 1. _split_segments(text) → list[str]
    │       Character-by-character parser
    │       Detects quote boundaries and sentence endings
    │
    ├── 2. _build_utterances(segments) → list[dict]
    │       Tags each segment as "dialogue" or "narration"
    │       based on leading quote character
    │
    └── 3. _attribute_speakers(utterances, character_names) → list[dict]
            Forward-tracking speaker attribution
            Regex matching against character names

Quote handling: a quoted dialogue span is extracted whole and kept as one utterance even with multiple sentences. _restore_quote_wrap re-wraps it with typographic double quotes and only treats double-quote closes (", \u201d) as outer closes — trailing inner single-quotes (’) are preserved (e.g. ...named ‘Nanny.’). Previously the inner text was split back into sentences and each was re-quoted, which fragmented multi-sentence speech and mangled inner single-quotes.

Quote Detection:

• Open: ", ', \u201c (left double), \u2018 (left single)
• Close: ", ', \u201d (right double), \u2019 (right single)
• Matching pairs: " → ", \u201c → \u201d, etc.

Sentence Splitting:

• Split at ., !, ? followed by whitespace/quote/dash
• Minimum 2 words per segment (avoids fragments)
• Quote boundaries always create segment breaks

Speaker Attribution Algorithm:

def _attribute_speakers(self, utterances, character_names) -> list[dict]

1. Tag Detection: Find NAME said/asked/replied patterns
2. Forward Tracking: When tag found, set next_speaker = NAME
3. Backward Reference: When no tag, look back for said NAME patterns
4. Name Normalization: Case-insensitive matching, alias resolution

Speech Verbs (42 total):

said, asked, replied, whispered, shouted, muttered, called,
exclaimed, answered, began, continued, added, finished, noted,
stated, remarked, yelled, screamed, laughed, sighed, says,
tells, told, orders, ordered, demands, demanded, insists,
insisted, begs, begged, pleads, pleaded, commands, commanded,
snapped, snarled, hissed, growled, barked, roared, croaked,
grunted, murmured, purred, intoned, declared, announced,
proclaimed, responds, repeats, mentions, offers

2. LLM Speaker Attribution (analysis/llm_attributor.py)

Class: LLMAttributor

class LLMAttributor:
    def __init__(self, config: LLMConfig, model: str | None = None) -> None
    
    def attribute_speakers(
        self,
        utterances: list[dict[str, Any]],
        chapter_title: str,
        chapter_id: int,
    ) -> list[dict[str, Any]]

Key Insight: The LLM attributes all dialogue utterances. The mechanical attribution from the segmenter is preserved only as a current_speaker hint (Rule 8) — the LLM's verdict takes precedence so that ambiguous or incorrectly guessed lines can be corrected. This differs from the original design where only speaker_name == "narrator" dialogue was attributed.

Attribution Flow:

1. Filter: Collect all dialogue utterances (type == "dialogue")
2. Batch: Group into batches of 40 utterances
3. Context: Split into scenes (<<<>>>); include per-scene narration names + last-12 narration lines as context
4. LLM Call: Send batch with character profiles and scene context to LLM
5. Parse Response: Extract JSON array with index → speaker mapping
6. Post-Process: Strip speech verb suffixes ("says", "said", etc.); run _normalize_speaker (aliases + first-word-of-primary → primary; narrator/the-narrator → lowercase narrator; pronouns → narrator guard)
7. Update: Apply LLM verdicts to all dialogue utterances
8. Fallback: Re-attribute any dialogue still un-attributed via speech-tag regex on the 3 preceding utterances, then speaker carry-forward (scene breaks reset the carried speaker)

LLM Prompt (inline):

You are a dialogue attribution expert for audiobook production.

Given utterances that need speaker attribution, identify who is speaking each line.

## Character Profiles
{character_profiles}

## Chapter Title
{chapter_title}

## Scene Context
This chapter has {scene_count} scene(s) separated by scene breaks (<<<>>>).
You are attributing dialogue for: {scene_label}

{scene_speakers_context}

## Utterances to Attribute
{utterances_json}

## Rules
1. Each utterance is already correctly split — do NOT merge or split them
2. EVERY utterance MUST be attributed to a speaker. Return EXACTLY one entry per utterance index. Do not skip any.
3. ALL these utterances are dialogue (in quotes) — they are spoken by a CHARACTER, almost never "Narrator". Only use "Narrator" if the line is truly un-attributable.
3b. STRONG HINT: when a preceding narration line says a character IS telling/explaining/speaking to someone (e.g. "Nanny is telling the princess about what will happen."), the following quoted lines are that character's dialogue, until another character is indicated.
3c. A multi-line speech block from one character continues until dialogue tags, action, or the responder's lines indicate a change.
3d. A single quoted dialogue utterance may contain MULTIPLE sentences (e.g. one paragraph of speech) — every sentence within it belongs to the SAME speaker. Do not split the speaker across sentences within one utterance.
4. Use dialogue tags (X says/replies/asks) to identify speakers
5. Use context: who is present, who was speaking before, who is addressed
6. Resolve pronouns: "she says" refers to the most recently mentioned female character
7. If a character has aliases, use their EXACT full primary_name shown in the profiles (e.g. "Kainda Walters Foster Murphy", NOT "Kainda" or "Princess Kainda")
8. The "current_speaker" field shows who the mechanical segmenter guessed — use it as a hint but correct if wrong
9. In a conversation, speakers usually alternate. If one line is Character A, the next line is likely a different character (often the one addressed).
10. CRITICAL — only return the EXACT utterance indices provided in the input. Do NOT invent or hallucinate extra indices.
11. CRITICAL — every returned speaker must be the ACTUAL character speaking. Assigning the wrong speaker ruins the audiobook.
12. AI-1 is a specific AI character voice. Only attribute dialogue to AI-1 if the text explicitly indicates AI-1 is speaking. Do NOT default undetermined utterances to AI-1.

## Scene Attribution Rules (CRITICAL)
- Chapters can contain multiple scenes separated by scene breaks (<<<>>>)
- Each scene is an independent unit. Speakers identified in one scene may NOT appear in other scenes
- Within a scene, once a speaker is identified, subsequent dialogue by the same person continues until another speaker is identified
- "captain" in one scene may refer to a different person than "captain" in another scene — there can be multiple characters with the same title
- Only attribute a speaker to dialogue if there is evidence within THAT SCENE that they are speaking
- If a scene introduces new characters via narration, those characters are candidates for dialogue in that scene
- A speaker who appears in scene 1 may be entirely absent from scene 2 — do NOT carry speakers across scene boundaries
- When ambiguous ("he says", "she says"), resolve using the most recent speaker mention within the SAME SCENE

## Output Format
Return ONLY a JSON array of objects, one per utterance, each with:
- "index": the utterance index (number)
- "speaker": the correct speaker name (use primary_name from profiles, MUST be a character name, NOT "Narrator")

Return ONLY the JSON array, no other text.

Performance:

• ~5 seconds per batch of 40 utterances with google/gemma-4-12b-qat (~34 tok/s on the remote LM Studio server)
• The model must be loaded with context_length: 8192 (default 262k-token context requests ~44.9 GB of memory and blocks loading)
• ~20-30 minutes for all 35 chapters; every dialogue utterance is processed (not just unattributed ones)

3. Scene Break Handling

Scene Breaks (<<<>>>):

• 51 scene breaks across the book
• Marked as special utterances: {"speaker": "scene_break", "text": "<<<>>>"}
• Within a scene: track which characters are present
• At a scene break: re-evaluate present characters
• Chapter context carries across scene breaks

LLM Context:

• Full chapter text with <<<>>> preserved
• LLM maintains context across scene breaks
• Can alter character presence between scenes

4. Sentence Splitting Rules

Correct Behavior:

• Split at sentence endings: . ! ? (narration only)
• Do NOT split at commas
• Narration: each sentence = separate utterance
• Quoted dialogue: the whole quoted span = ONE utterance, even when it contains multiple sentences (all sentences share the same speaker — Rule 3d)

Examples:

"I'm on a yacht." → ["I'm on a yacht."]
"That is not something..." → ["That is not something..."]
"Help! Open the latch." → ["Help! Open the latch."]  # one utterance, one speaker

5. Speaker Name Resolution

Every LLM-returned speaker passes through _normalize_speaker before being stored:

Aliases → Primary Names:

• "Princess", "Kainda", "the princess" → "Kainda Walters Foster Murphy"
• "Captain", "Captain Wake" → "Willamina Wake"
• "Hunter", "Ship AI", "The Deal Hunter" → "Deal Hunter" (the ship's AI — one speaker)

First-word-of-primary mapping: the first word of each primary name maps to the full primary name when it is not already an alias (e.g. "kainda" → "Kainda Walters Foster Murphy").

Narrator lowercase: "narrator" / "the narrator" → canonical lowercase narrator (the DB and generation code check this exact case).

Pronoun guard: pronouns ("she", "he", "they", "her", etc.) → narrator; a bare pronoun is an LLM failure, never a character.

LLM Post-Processing:

• Strip speech verb suffixes: "Kainda says" → "Kainda"
• Handle group speakers: "Management Group", "Guard Detail Members"
• Handle AI speakers: "Deal Hunter" (alias "Hunter"/"Ship AI"), "AI-1"
• Unknown names are kept as-is and surfaced on the orphaned-utterances page for review

Output

Utterances are saved to the utterances table in project.db:

INSERT INTO utterances (
    chapter_id,
    utterance_index,
    text,
    type,  -- "narration" | "dialogue"
    speaker_name  -- "narrator" | "Princess Kainda" | "scene_break" | etc.
) VALUES (?, ?, ?, ?, ?)

Quality Metrics

Metric	Value
Total utterances	4,339
Dialogue utterances	1,882
Narration utterances	2,453
Dialogue-as-narrator errors	1 (0.05%)
Scene breaks	51
Chapters processed	35

Error Analysis

Remaining 0.05% Error:

• 1 dialogue utterance still attributed to narrator (ch35 "About the author" back matter — expected, no speaker is present)
• Edge cases where the LLM and mechanical segmenter both fail
• Acceptable for production use

Common Failure Modes:

1. Missing dialogue tags with no context
2. Multiple speakers in rapid succession
3. Internal thoughts (should be narrator, sometimes misattributed)
4. Indirect speech ("she said that...") vs direct speech
5. Bare pronoun speakers ("She") — caught by the pronoun guard, which maps them to narrator

Validation

Run after any re-segment or re-attribute to confirm the quality gates hold:

-- Dialogue left as narrator (expect only the ch35 back-matter exception)
SELECT chapter_id, COUNT(*) FROM utterances
WHERE type = 'dialogue' AND lower(speaker_name) = 'narrator'
GROUP BY chapter_id;

-- Pronoun speakers (must be 0 — guard maps these to narrator)
SELECT COUNT(*) FROM utterances
WHERE type = 'dialogue' AND lower(speaker_name) IN
  ('i','me','my','you','your','he','him','his','she','her',
   'they','them','their','we','us','our','it','its','one');

-- Narrator case (must be lowercase 'narrator', never 'Narrator')
SELECT COUNT(*) FROM utterances WHERE speaker_name = 'Narrator';

-- Unknown speakers (cross-check against speakers.json aliases/first-words;
-- leftovers appear on the /orphaned-utterances page)
SELECT DISTINCT speaker_name FROM utterances
WHERE type = 'dialogue' AND lower(speaker_name) != 'narrator';

Additional checks:

• ch13 golden reference: 44 utterances — 15 Kainda / 15 Nanny / 14 narrator (verified against the manuscript, e.g. "Fifty light years" is Kainda, "Don't help Princess Kainda" is Nanny)
• Multi-sentence quotes: spot-check that a paragraph of quoted speech is ONE dialogue utterance with a single speaker (Rule 3d)
• Alternation: dialogue should alternate between the characters present; a long run of one speaker with no tag is suspicious

Scripts

resegment_all.py

# Run on all chapters with LLM attribution
python scripts/resegment_all.py --model google/gemma-4-12b-qat

# Run on specific chapters
python scripts/resegment_all.py --chapters 5,6,7 --model google/gemma-4-12b-qat

# Mechanical-only (no LLM)
python scripts/resegment_all.py --mechanical-only

reattribute_all.py

Re-runs LLM speaker attribution on existing utterances without re-segmenting (preserves utterance IDs and mechanical segmentation; only updates speaker_name):

# All chapters (--sleep paces requests to the remote server)
python scripts/reattribute_all.py --model google/gemma-4-12b-qat --sleep 0.2

# Specific chapters
python scripts/reattribute_all.py --chapters 5,6,7 --model google/gemma-4-12b-qat

Note: before running attribution, ensure the remote LM Studio server has the model loaded with context_length: 8192:

curl -X POST http://192.168.123.14:1234/api/v1/models/load \
  -H 'Content-Type: application/json' \
  -d '{"model": "google/gemma-4-12b-qat", "context_length": 8192, "flash_attention": true, "echo_load_config": true}'

Key Design Decisions

1. Two-phase over one-phase: Mechanical segmenter handles text splitting reliably; LLM only does attribution
2. LLM attributes all dialogue: The LLM's verdict overrides the mechanical current_speaker guess so ambiguous lines get corrected (not just the unattributed ones)
3. Quotes kept whole: Multi-sentence quoted speech is one utterance; splitting it loses the single-speaker evidence
4. Inner-quote preservation: _restore_quote_wrap only recognizes double-quote closes so trailing inner single-quotes (’) survive re-wrapping
5. Name normalization: Aliases and first-word-of-primary map to canonical primary names; narrator stays lowercase; pronouns map to narrator (no fabricated speakers)
6. Deterministic fallback: Speech-tag regex + speaker carry-forward guarantee un-attributed dialogue still gets a speaker
7. Batch processing: 40 utterances per LLM call for efficiency
8. Scene break preservation: <<<>>> markers maintained as special utterances; they reset the carried speaker
9. google/gemma-4-12b-qat model: ~34 tok/s on the remote LM Studio server, must be loaded with context_length: 8192 (~44.9 GB context otherwise blocks loading)