UmbraAudio/specs/02-analysis.md

Spec: Analysis — BookNLP & Character Profiling (Pass 1b)

Overview

Pass 1b extracts characters, dialogue, and coreferences from chapter text using BookNLP, then generates detailed voice profiles for each character using an LLM. Unusual terms get pronunciation guides.

Critically, this pass is also where voice characteristics — soft-spoken, assertive, gravelly, and so on — are converted from free-text description into a small, fixed vocabulary (the voice attribute taxonomy). Pass 2 (Voice Design) and Pass 3 (Mapping) both consume this taxonomy directly rather than re-interpreting prose. This is what makes character voices reproducible and what lets the system check, mechanically, whether two characters are actually going to sound different from each other.

The narrator is handled as a special case throughout this pass — see "Narrator Profile" below — because the narrator must not inherit the LLM's free-ranging interpretation that named characters get.

Components

1. BookNLP Engine (analysis/booknlp_engine.py)

Class: BookNLPEngine

class BookNLPEngine:
    def __init__(self) -> None
    async def analyze(self, text: str, output_dir: Path) -> dict

Initialization:

• Lazy-loads BookNLP("en", model_params={"model": "small", "pipeline": "entity,coref,quote"})
• Model params: entity recognition + coreference resolution + quote attribution

Analysis Flow:

1. Writes text to {output_dir}/input.txt
2. Runs self._nlp.process(input_file, output_dir, idd="input")
3. Parses output files:
   • input.entities — TSV: entity mentions (PER persons with coref IDs)
   • input.quotes — TSV: dialogue with speaker attribution

Output Format:

{
    "characters": [
        {
            "name": str,          # Character name
            "mention_count": int,  # How many times mentioned
            "coref_id": int,       # Coreference cluster ID
            "context": str,        # First 200 chars of chapter text
        }
    ],
    "quotes": [
        {
            "speaker": str,        # Speaker name (from BookNLP)
            "text": str,           # Quoted dialogue text
            "char_id": int,        # Character ID
        }
    ]
}

Patching Required:

• booknlp/english/entity_tagger.py: load_state_dict(..., strict=False) for transformers 5.x
• booknlp/english/litbank_coref.py: load_state_dict(..., strict=False)
• booknlp/english/bert_qa.py: load_state_dict(..., strict=False)

2. Voice Attribute Taxonomy

Every profiled character (and the narrator, via fixed defaults below) is described by a small set of enumerated dimensions. These are the only values Pass 2 and Pass 3 are allowed to branch on — free-text voice_description is for human review and TTS conditioning text, not for program logic.

Dimension	Allowed Values	Drives
pitch_register	very_low, low, mid, high, very_high	Sample-text style, embedding-collision grouping
pace_baseline	very_slow, slow, moderate, brisk, fast	Default pace in manifest before punctuation adjustment
energy	subdued, soft_spoken, measured, assertive, commanding, volatile	TTS exaggeration/cfg_weight range (Pass 2); how strongly punctuation cues scale (Pass 3)
timbre	warm, breathy, nasal, gravelly, clear, resonant, thin	Sample-text selection; collision grouping
texture	smooth, raspy, clipped, melodic, monotone	Sample-text selection
formality	casual, conversational, formal, archaic	Speech-pattern conditioning text
emotional_baseline	free-form short tag, e.g. neutral_calm, weary_cynical, anxious, stoic, warm_optimistic	Default instruct framing in Pass 3
distinguishing_trait	one short phrase, e.g. "drawls vowels", "clips sentences short", "trails off when uncertain"	Human-review summary; tie-breaker in collision review

accent_marker is optional free text (e.g. "received_pronunciation", "none") and is only ever populated when the text itself gives clear evidence (e.g. explicit dialect spelling, a stated nationality with described speech). It is never inferred from a character's name alone, and the profiler must not invent accents that play into ethnic or national stereotyping without direct textual support — if evidence is thin, accent_marker stays "none".

3. Character Profiler (analysis/character_profiler.py)

Class: CharacterProfiler

class CharacterProfiler:
    def __init__(self, llm_config: LLMConfig) -> None
    async def profile_character(self, name, aliases, gender, context) -> CharacterProfile
    async def check_distinctiveness(
        self, profile: CharacterProfile, existing: list[CharacterProfile]
    ) -> DistinctivenessResult
    async def generate_pronunciation_guide(self, terms, context) -> dict[str, str]

LLM Provider Selection (auto mode):

1. Try LM Studio at config.llm.lmstudio_base_url/models (default http://192.168.123.14:1234/v1, overridable via LLMConfig.lmstudio_base_url); the model is config.llm.lmstudio_model (default gemma-4-12b-it-qat — must be loaded with context_length: 8192), and the configured model is preferred over whatever the server has loaded
2. Fall back to OpenAI (requires OPENAI_API_KEY)
3. Raise RuntimeError if neither available

Lifecycle note: the lmstudio_lifecycle module resolves the same URL from config via load_config().llm.lmstudio_base_url for server/status operations, with http://localhost:1234/v1 only as a hardcoded fallback when config cannot be loaded. Standalone scripts (reattribute_all.py, resegment_all.py, analyze_emotions.py) still hardcode the LAN URL (http://192.168.123.14:1234/v1) directly; keep the config default and the script constant in sync.

Character Profile Prompt:

You are an expert audiobook director. Analyze the following character
from a novel and create a detailed voice profile for text-to-speech generation.

Character name: {name}
Known aliases: {aliases}
Gender (inferred): {gender}
Context from the book:
{context}

Already-cast voices in this book (avoid sounding like any of these):
{existing_cast_summary}

Create a JSON profile with these fields:
- personality: Brief personality description (1-2 sentences)
- voice_description: Detailed voice description for TTS voice design
  (include gender, age, accent, tone, pitch characteristics)
- emotion_tendencies: List of 3-5 emotions this character commonly expresses
- speech_patterns: How this character speaks (formal/casual, sentence structure, etc.)
- narrator_style: If this character were narrating, what style would they use?
- pitch_register: one of [very_low, low, mid, high, very_high]
- pace_baseline: one of [very_slow, slow, moderate, brisk, fast]
- energy: one of [subdued, soft_spoken, measured, assertive, commanding, volatile]
- timbre: one of [warm, breathy, nasal, gravelly, clear, resonant, thin]
- texture: one of [smooth, raspy, clipped, melodic, monotone]
- formality: one of [casual, conversational, formal, archaic]
- emotional_baseline: short tag, e.g. "weary_cynical"
- distinguishing_trait: one short phrase
- accent_marker: only set if the text gives direct evidence; otherwise "none"

Return ONLY valid JSON, no other text.

existing_cast_summary is a compact list of {name, pitch_register, energy, timbre} for every character already profiled in this book (built incrementally as profiling proceeds), so the LLM has a chance to avoid an obvious collision before one is even generated.

LLM Settings:

• Model: GPT-4o (or local LM Studio model)
• Temperature: 0.3
• Max tokens: 4096
• Context truncated to 2000 chars

Response Parsing:

1. Try full JSON parse
2. Fallback: extract first {...} block from response
3. Enum validation: any taxonomy field whose value is not in its allowed set is coerced to a defined fallback (the dimension's middle/neutral value, e.g. energy → measured, pace_baseline → moderate) and a warning is logged with character name + bad value. The pipeline never raises on this — a bad enum degrades to a safe default rather than blocking the run.
4. Returns CharacterProfile dataclass

Cast-Aware Distinctiveness Check

This is a text-level pre-check before any audio is generated; it catches the cheap, obvious collisions before Pass 2's more expensive embedding-based check runs.

@dataclass
class DistinctivenessResult:
    is_distinct: bool
    colliding_with: list[str]          # character names
    matched_dimensions: list[str]      # which taxonomy fields matched

Algorithm:

1. Compare the new profile's (pitch_register, pace_baseline, energy, timbre, texture, formality) tuple against every existing profiled character who co-occurs with this one — defined as: both have at least one BookNLP-attributed quote within the same chapter.
2. If 4 or more of the 6 dimensions match exactly with any co-occurring character, flag a collision.
3. On collision: re-prompt the LLM once with an explicit instruction — "Character X already has {colliding dimensions}; revise this character so it differs in at least two of: pitch_register, energy, timbre, texture, while staying consistent with the text evidence." Maximum 2 retries.
4. If still colliding after retries, accept the profile but mark characters.status = 'collision_pending' — this surfaces in the Pass 2 quality gate (see 03-voice-design.md) rather than blocking Pass 1b.

Characters who never share a chapter with attributed dialogue are not compared against each other — two minor characters in different parts of the book sounding similar is not a defect worth flagging.

4. Narrator Profile — Required Defaults

The narrator's CharacterProfile is not produced by the generic LLM character-profiling call above. It is constructed directly with fixed values, because the narrator's job is to disappear behind the prose, not to have a distinct dramatic personality:

NARRATOR_DEFAULT_PROFILE = CharacterProfile(
    name="narrator",
    is_narrator=True,
    pitch_register="mid",
    pace_baseline="moderate",
    energy="measured",
    timbre="clear",
    texture="smooth",
    formality="conversational",
    emotional_baseline="neutral_calm",
    distinguishing_trait="even, unhurried, minimal emphasis",
    accent_marker="none",
    voice_description=(
        "A calm, clear, even-toned narrator. Minimal emotional coloring, "
        "no shouting, no dramatic swings in pace or pitch. Reads "
        "descriptive prose steadily regardless of how dramatic the "
        "events being described are."
    ),
)

This profile is never sent through check_distinctiveness against itself, and is excluded from the LLM prompt's "already-cast voices" list (the narrator is an anchor, not a competing cast member).

First-person exception: if BookNLP/coref analysis indicates the book is narrated in first person by a named character (the narrator is a character in the story, e.g. a memoirist or first-person protagonist), that character still gets a full, normally-profiled CharacterProfile with is_narrator=True — they are allowed a real personality. However, the calm/measured ceiling above still applies specifically to their narration-type utterances (descriptive/expository prose); their dialogue-type utterances (lines they speak as a character within remembered scenes) use their full taxonomy-driven range. This split is enforced in Pass 3 (see 04-mapping.md, "Narrator Constraint"), not here — Pass 1b's job is only to set is_narrator=True correctly.

5. Character Profile Dataclass (analysis/__init__.py)

@dataclass
class CharacterProfile:
    name: str
    aliases: list[str] = field(default_factory=list)
    gender: str | None = None
    age_range: str | None = None
    is_narrator: bool = False
    personality: str | None = None
    voice_description: str | None = None
    emotion_tendencies: list[str] = field(default_factory=list)
    speech_patterns: str | None = None
    narrator_style: str | None = None

    # Voice attribute taxonomy (see "Voice Attribute Taxonomy" above)
    pitch_register: str = "mid"
    pace_baseline: str = "moderate"
    energy: str = "measured"
    timbre: str = "clear"
    texture: str = "smooth"
    formality: str = "conversational"
    emotional_baseline: str = "neutral_calm"
    distinguishing_trait: str | None = None
    accent_marker: str = "none"

Defaults match the narrator baseline deliberately — an unprofiled character (e.g. a one-line walk-on the profiler skipped) renders as plain and unremarkable rather than as an error or as something randomly dramatic.

Taxonomy Revision (Post-Hoc)

A character's taxonomy is not frozen once Pass 1b completes. It can be revised later — specifically when a human selects "Always" on a detected speech descriptor during Pass 3's review queue (e.g. a line tagged "with his raspy voice" gets promoted from a one-off note to a permanent trait). See 04-mapping.md, "Speech Descriptor Extractor & Review Queue."

That revision is never a direct field write. It always routes through the same apply_edit_prompt() mechanism described in 03-voice-design.md:

1. The detected phrase seeds an edit prompt (human can refine the wording before submitting)
2. The LLM translates it into specific taxonomy deltas (e.g. timbre: clear → gravelly)
3. The character's voice reference is regenerated against the new taxonomy
4. The cast-aware distinctiveness check re-runs against the full cast — not just a sanity check, but a requirement: changing one character's timbre from clear to gravelly can create a brand-new collision with a different character who was previously safely distinct
5. characters.status drops from voiced_approved back to a pending-review state for the duration of this loop, exactly as if Pass 2 were running for the first time on this character
6. Only on re-approval does the revised taxonomy become the character's new permanent baseline

This keeps the system honest about a real trade-off: a transient state ("her voice was hoarse from crying in this one scene") should never silently become a permanent trait. Promoting something to "Always" costs exactly as much process as the original profiling did, on purpose.

6. Pronunciation Generator (analysis/pronunciation.py)

Class: PronunciationGenerator

class PronunciationGenerator:
    def __init__(self, llm_config: Any) -> None
    def find_unusual_terms(self, text: str) -> list[str]
    def phonetic_to_ssml(self, word: str, phonetic: str) -> str

Term Detection Algorithm (3 regex passes):

1. Proper nouns: [A-Z][a-z]+(\s[A-Z][a-z]+)* — filters common words (The, And, But, For, Not)
2. Foreign words: Words containing accented characters (é, ñ, ü, etc.)
3. Hyphenated words: Words containing hyphens

Note — not the same as shouted ALL-CAPS: this generator is for unusual/foreign pronunciation, not for vocal emphasis. In-line ALL-CAPS words inside dialogue (e.g. "NO!") are a Pass 3 prosody signal (see 04-mapping.md), not a pronunciation-guide entry. Do not route ALL-CAPS dialogue words into find_unusual_terms.

Pronunciation via LLM:

• Prompt: "Give IPA pronunciation for: {terms}"
• Temperature: 0.1 (low for accuracy)
• Context truncated to 1000 chars

SSML Output:

<phoneme alphabet="ipa" ph="{phonetic}">{word}</phoneme>

Orchestrator Flow

# In orchestrator.py run_pipeline():
async def _run_analysis(db, project_dir, chapters_db):
    # 0. Assign the fixed narrator profile (no LLM call)
    narrator_profile = NARRATOR_DEFAULT_PROFILE
    await db.upsert_character_profile("narrator", narrator_profile)

    # 1. BookNLP per chapter
    for ch in chapters_db:
        result = booknlp.analyze(ch["source_text"], analysis_dir)

    # 2. Merge characters across chapters
    merged = _merge_characters(raw_characters, all_quotes)

    # 3. Profile each character via LLM, checking distinctiveness as we go
    cast_so_far: list[CharacterProfile] = []
    for char in merged[:30]:
        profile = await profiler.profile_character(...)
        result = await profiler.check_distinctiveness(profile, cast_so_far)
        if not result.is_distinct:
            profile = await profiler.profile_character(..., retry_hint=result)
        cast_so_far.append(profile)
        await db.update_character_profile(char_id, profile)

    # 4. Pronunciation guide for unusual terms
    guide = await profiler.generate_pronunciation_guide(terms, context)

Character Merging Algorithm

def _merge_characters(raw_characters: list[dict], quotes: list[dict]) -> list[dict]

1. Filter: Remove pronouns (I, you, he, she, it, we, they...) and NOISE_CHARACTERS (people, everyone, someone, pirates, warbots...)
2. Filter: Remove single-character names
3. Merge by lowercase: Group variants (e.g., "Princess", "princess", "PRINCESS" → one entry)
4. Keep highest mention count as primary
5. Build alias groups: Case-variant duplicates become aliases
6. Attach quote context: First 200 chars from associated quotes
7. Sort by mention count (descending)

Noise Characters (Excluded from Voice Design)

NOISE_CHARACTERS = {
    "people", "everyone", "someone", "anyone", "nobody", "something",
    "nothing", "everything", "the fighters", "the fugitives", "the pirates",
    "Warbots", "the Warbots", "pirates",
}

Database Updates

Stage	Table	Fields Set
Narrator assigned	characters	fixed taxonomy fields, is_narrator=True, status='profiled' (skips 'identified')
Characters extracted	characters	name, aliases, gender, age_range, status='identified'
Profiled via LLM	characters	profile_json, voice_description, taxonomy fields, status='profiled'
Distinctiveness flagged	characters	status='collision_pending' (surfaced again in Pass 2 gate)
Pronunciation	pronunciation_guide	term, phonetic, notes
Voice designed	characters	voice_ref_path, speaker_embedding, status='voiced'