UmbraAudio/specs/04-mapping.md

Spec: Mapping — Two-Phase Segmentation (Pass 2)

Overview

Pass 2 segments chapter text into utterances (narration and dialogue) and attributes speakers using a two-phase approach: a mechanical segmenter handles text splitting, and an LLM attributes speakers to all dialogue.

This pass handles:

• Sentence splitting at . ! ? (not commas)
• Dialogue detection via quotes ("'"")
• Speaker attribution via dialogue tags and context
• Scene break (<<<>>>) handling
• Chapter title detection

Components

1. Mechanical Segmenter (mapping/segmenter.py)

Class: Segmenter

class Segmenter:
    def segment(
        self,
        text: str,
        chapter_id: int,
        character_names: list[str] | None = None,
        chapter_title: str | None = None,
    ) -> list[dict[str, Any]]

Special chapter handling by chapter_title: chapters titled "copyright", "author's note", "back matter", and "title" are force-attributed to the narrator.

Returns:

[{
    "index": int,
    "text": str,
    "type": "narration" | "dialogue",
    "character_name": str | None,
}, ...]

Segmentation Pipeline:

Raw chapter text
    │
    ├── 1. _split_segments(text) → list[str]
    │       Character-by-character parser
    │       Detects quote boundaries and sentence endings
    │
    ├── 2. _build_utterances(segments) → list[dict]
    │       Tags each segment as "dialogue" or "narration"
    │       based on leading quote character
    │
    └── 3. _attribute_speakers(utterances, character_names) → list[dict]
            Forward-tracking speaker attribution
            Regex matching against character names

Quote Preservation:

• A quoted dialogue span is extracted whole (_extract_quote scans to the closing quote) and kept as a single utterance even when it contains multiple sentences. The inner text is never split back into per-sentence re-quoted fragments — that split is what caused multi-sentence speech to lose its single-speaker evidence.
• _restore_quote_wrap re-wraps the stripped inner text with typographic double quotes. It only treats double-quote closes (", \u201d) as outer closes, so a trailing inner single-quote (’, e.g. ...named ‘Nanny.’) is preserved instead of being mistaken for the outer closing quote.

Quote Detection:

• Open: ", ', \u201c (left double), \u2018 (left single)
• Close: ", ', \u201d (right double), \u2019 (right single)

Sentence Splitting:

• Split at ., !, ? followed by whitespace/quote/dash (narration only)
• Quote boundaries always create segment breaks
• Quoted dialogue is never split by sentence endings — the whole quote is one utterance

Speaker Attribution Algorithm:

1. Tag Detection: Find NAME said/asked/replied patterns
2. Forward Tracking: When tag found, set next_speaker = NAME
3. Backward Reference: When no tag, look back for said NAME patterns
4. Name Normalization: Case-insensitive matching, alias resolution

Speech Verbs (53 total):

said, asked, replied, whispered, shouted, muttered, called,
exclaimed, answered, began, continued, added, finished, noted,
stated, remarked, yelled, screamed, laughed, sighed, says,
tells, told, orders, ordered, demands, demanded, insists,
insisted, begs, begged, pleads, pleaded, commands, commanded,
snapped, snarled, hissed, growled, barked, roared, croaked,
grunted, murmured, purred, intoned, declared, announced,
proclaimed, responds, repeats, mentions, offers

2. LLM Speaker Attribution (analysis/llm_attributor.py)

Class: LLMAttributor

class LLMAttributor:
    def __init__(self, config: LLMConfig, model: str | None = None) -> None
    
    def attribute_speakers(
        self,
        utterances: list[dict[str, Any]],
        chapter_title: str,
        chapter_id: int,
    ) -> list[dict[str, Any]]

Key Insight: The LLM attributes all dialogue utterances. The mechanical attribution from the segmenter is preserved only as a current_speaker hint for the LLM (rule 8) — the LLM's verdict takes precedence so that ambiguous or incorrectly guessed lines can be corrected. This differs from the original design where only speaker_name == "narrator" dialogue was attributed; the current implementation lets the LLM override mechanical guesses.

Attribution Flow:

1. Filter: Collect all dialogue utterances (type == "dialogue")
2. Batch: Group into batches of 40 utterances
3. Context: Include surrounding utterances for context
4. LLM Call: Send batch with character profiles
5. Parse Response: Extract JSON array with index → speaker mapping
6. Post-Process: Strip speech verb suffixes ("says", "said", etc.) and run _normalize_speaker (alias/first-word resolution, narrator lowercase, pronoun guard)
7. Update: Apply LLM verdicts to all dialogue utterances
8. Fallback: Re-attribute any dialogue still un-attributed via
   • Speech-tag detection on the 3 preceding utterances (regex NAME says/said/asked/...)
   • Speaker carry-forward from the last active speaker within the same scene
   • Scene breaks (<<<>>>) reset the carried speaker

Performance:

• ~5 seconds per batch of 40 utterances with gemma-4-12b-it-qat (~34 tok/s on the remote LM Studio server)
• The model must be loaded with context_length: 8192; the default 262k-token context requests ~44.9 GB of memory and blocks loading
• Processes all dialogue utterances; deterministic fallback recovers any that the LLM misses (target 100% dialogue attribution)

3. Emotion Mapping (mapping/emotion.py)

Class: EmotionMapper

class EmotionMapper:
    def __init__(self, llm_client: Any = None, model: str = "") -> None
    def analyze_utterances(self, utterances, chapter_context="") -> list[dict]
    def batch_generate(self, utterances, chapter_context="") -> list[dict]

Maps each utterance to TTS voice-direction metadata: emotion, pace, emphasis_words (max 3), instruct, pause_before_ms, pause_after_ms.

Batching: utterances are processed in batches of 30 per LLM call. The output JSON array must match the input order; a length mismatch falls back to deterministic defaults for the whole batch.

Question Intonation (required):

• Any utterance ending with ? (including ?") is a question.
• The LLM prompt must instruct rising question intonation for question-ending utterances.
• The deterministic fallback (_default_result) also emits instruct = "Speak naturally with rising question intonation at the end." for question-ending dialogue when no LLM is available or a batch fails.
• This ensures every question has rising intonation even if the LLM is unavailable.

Defaults:

• Narrator text → emotion="neutral", pace="moderate", instruct="Read steadily and evenly. Low emphasis, consistent tone."
• Non-question dialogue → emotion="neutral", pace="moderate", instruct="Speak naturally."
• All results default to pause_before_ms=500, pause_after_ms=500.

LLM Settings: temperature 0.2, max_tokens 8192, 3 attempts with exponential backoff, robust JSON extraction (strip fenced blocks, brace/bracket trimming).

4. Scene Break Handling

Scene Breaks (<<<>>>):

• 51 scene breaks across the book
• Marked as special utterances: {"speaker": "scene_break", "text": "<<<>>>"}
• Within a scene: track which characters are present
• At a scene break: re-evaluate present characters
• Chapter context carries across scene breaks

5. Name Normalization

All LLM-returned speakers pass through _normalize_speaker before being stored:

Aliases → Primary Names (from speakers.json):

• "Princess", "Kainda", "the princess" → "Kainda Walters Foster Murphy"
• "Captain", "Captain Wake" → "Willamina Wake"
• "Hunter", "Ship AI", "The Deal Hunter" → "Deal Hunter" (the ship's AI — one speaker)

First-word-of-primary mapping: the first word of each primary name maps to the full primary name when it is not already an alias (e.g. "kainda" → "Kainda Walters Foster Murphy"). This fixes attributions that only match the primary name's first word.

Narrator lowercase: "narrator" / "the narrator" resolve to the canonical lowercase narrator (the DB and generation code check this exact case).

Pronoun guard: pronouns ("she", "he", "they", "her", etc.) resolve to narrator instead of fabricating a speaker — a bare pronoun (e.g. "She") is a real LLM failure mode, never a character name.

Resolution:

• Case-insensitive matching
• Alias map from speakers.json
• Fallback: unknown names are kept as-is and surfaced on the orphaned-utterances page for review

Output

Utterances saved to utterances table:

INSERT INTO utterances (
    chapter_id,
    utterance_index,
    text,
    type,
    speaker_name
) VALUES (?, ?, ?, ?, ?)

Quality Metrics

Metric	Value
Total utterances	4,339
Dialogue utterances	1,882
Narration utterances	2,453
Dialogue-as-narrator errors	1 (0.05%)
Scene breaks	51
Chapters	35

Scripts

resegment_all.py

# Run on all chapters with LLM attribution
python scripts/resegment_all.py --model gemma-4-12b-it-qat

# Run on specific chapters
python scripts/resegment_all.py --chapters 5,6,7

# Mechanical-only (no LLM)
python scripts/resegment_all.py --mechanical-only

reattribute_all.py

Re-runs LLM speaker attribution on existing utterances without re-segmenting (preserves utterance IDs and mechanical segmentation; only updates speaker_name):

# All chapters
python scripts/reattribute_all.py --model gemma-4-12b-it-qat

# Specific chapters (--sleep adds a delay between chapters to pace the server)
python scripts/reattribute_all.py --chapters 5,6,7 --sleep 0.2

analyze_emotions.py

Re-runs emotion mapping (including question intonation) on all utterances in the database.

Key Design Decisions

1. Two-phase over one-phase: Mechanical handles splitting reliably; LLM does attribution
2. Quotes kept whole: A quoted dialogue span stays one utterance even with multiple sentences (Rule 3d) — splitting it would lose the single-speaker evidence
3. Inner-quote preservation: _restore_quote_wrap only recognizes double-quote closes so a trailing inner single-quote (’) survives re-wrapping
4. LLM attributes all dialogue: The LLM's verdict overrides the mechanical current_speaker guess so ambiguous lines get corrected
5. Deterministic fallback: Speech-tag regex + speaker carry-forward guarantee un-attributed dialogue still gets a speaker
6. Batch processing: 40 utterances per LLM call for efficiency
7. Scene break preservation: <<<>>> markers maintained as special utterances; they reset the carried speaker
8. Speaker name normalization: Aliases and first-word-of-primary resolve to canonical primary names; narrator stays lowercase; pronouns map to narrator
9. Question intonation is mandatory: every ?-ending utterance carries rising question intonation in its TTS instruct text, enforced in both the LLM prompt and the deterministic fallback