UmbraAudio/specs/03-voice-design.md

Spec: Voice Design (Pass 2)

Overview

Pass 2 generates unique voice reference audio for each character using Chatterbox TTS. Each character's TTS parameters (exaggeration, cfg_weight) are now derived in two steps rather than one: first, the character's voice attribute taxonomy (from Pass 1b) selects a parameter range appropriate to how that character is actually described; second, an MD5 hash of the name picks a deterministic point within that range, so re-running the pipeline always reproduces the same voice while two characters who share the same energy value still land at different points in the range.

The narrator is the one character whose parameters are not range-and-hash derived at all — they are fixed constants, because the narrator must sound the same in every project, every run, forever (see "Narrator Parameters" below).

This pass also adds a hard quality gate: after every voice reference is generated, the system checks whether any two co-occurring characters are acoustically too similar, and blocks the handoff to Pass 3 until that's resolved or a human explicitly overrides it.

Components

Voice Designer (voices/voice_designer.py)

class VoiceDesigner:
    def __init__(self, config: TTSConfig) -> None

    def generate_voice_reference(
        self,
        character: CharacterProfile,
        output_dir: Path,
        sample_text: str | None = None,
    ) -> Path

    def generate_all_references(
        self,
        characters: list[CharacterProfile],
        output_dir: Path,
    ) -> dict[str, Path]

    def check_cast_distinctiveness(
        self,
        characters: list[CharacterProfile],
        embeddings: dict[str, bytes],
        cooccurrence: dict[str, set[str]],
    ) -> list[VoiceCollision]

Voice Reference Generation Flow

CharacterProfile
    │
    ├── _character_params(character) → (exaggeration, cfg_weight)
    │       energy → base range (ATTRIBUTE_PARAM_MAP)
    │       MD5 hash of name → deterministic position within that range
    │       narrator → fixed constants, taxonomy/hash ignored entirely
    │
    ├── _generate_sample_text(character) → str
    │       Selected from taxonomy fields (energy, timbre, pitch_register),
    │       falling back to voice_description keyword sniffing only for
    │       legacy profiles created before the taxonomy existed
    │
    └── ChatterboxEngine.generate(
            text=sample_text,
            output_path={name}_ref.wav,
            voice_ref=None,           # No prior ref for initial generation
            instruct=voice_description,  # Text conditioning (not forwarded to model)
            exaggeration=exag,
            cfg_weight=cfg,
        )
            │
            └── ChatterboxTTS.generate(text, audio_prompt_path=None, exaggeration, cfg_weight)
                    │
                    └── Perth watermarker (if available)
                            │
                            └── Write WAV via soundfile.write()

Attribute-Driven Parameter Ranges

energy is the primary driver of expressive intensity. cfg_weight runs inversely to exaggeration: higher cfg_weight means the model adheres closely to a neutral, reference-like delivery; lower cfg_weight paired with higher exaggeration gives a more emphatic, expressive delivery. A calm, measured character should have low exaggeration and high cfg_weight; a volatile, commanding character should have the opposite.

ATTRIBUTE_PARAM_MAP: dict[str, tuple[tuple[float, float], tuple[float, float]]] = {
    # energy:        (exaggeration_range,  cfg_weight_range)
    "subdued":        ((0.15, 0.30),        (0.55, 0.70)),
    "soft_spoken":     ((0.20, 0.35),        (0.45, 0.60)),
    "measured":        ((0.30, 0.45),        (0.40, 0.55)),
    "assertive":       ((0.45, 0.60),        (0.30, 0.45)),
    "commanding":      ((0.55, 0.70),        (0.20, 0.35)),
    "volatile":        ((0.60, 0.80),        (0.15, 0.30)),
}

Per-Character Parameter Derivation

def _character_params(self, character: CharacterProfile) -> tuple[float, float]:
    if character.is_narrator and character.name == "narrator":
        return NARRATOR_EXAGGERATION, NARRATOR_CFG_WEIGHT

    exag_range, cfg_range = ATTRIBUTE_PARAM_MAP[character.energy]
    name_hash = int(hashlib.md5(character.name.encode()).hexdigest()[:8], 16)

    exaggeration = exag_range[0] + ((name_hash % 100) / 100.0) * (exag_range[1] - exag_range[0])
    cfg_weight = cfg_range[0] + (((name_hash >> 8) % 100) / 100.0) * (cfg_range[1] - cfg_range[0])
    return exaggeration, cfg_weight

Example Values:

Character	energy	MD5 Prefix	Exaggeration	Cfg Weight
narrator	— (fixed)	—	0.25 (fixed)	0.65 (fixed)
Princess Kainda	assertive	a3f7...	0.512	0.341
Captain	commanding	4b2e...	0.598	0.247
Deal Hunter	measured	8c1d...	0.368	0.452
Nanny	soft_spoken	f5a3...	0.248	0.512
The robot	subdued	7e9b...	0.198	0.612
Aunt Vicky	measured	2d4c...	0.402	0.481

Values are illustrative; exact figures depend on the live MD5 digest, but every character's pair now falls inside the band their energy attribute defines — an "assertive" character can no longer accidentally land in subdued-character territory the way pure name-hashing allowed.

Narrator Parameters

NARRATOR_EXAGGERATION = 0.25
NARRATOR_CFG_WEIGHT = 0.65

These are constants, not derived from anything character-specific — explicitly so the narrator sounds identical across every book and every pipeline run, which is the acoustic anchor the rest of the cast is heard against. Do not route the narrator through ATTRIBUTE_PARAM_MAP or the hash function, even though NARRATOR_DEFAULT_PROFILE.energy == "measured" would otherwise resolve to a valid range — the explicit override always wins for character.name == "narrator".

Sample Text Generation

def _generate_sample_text(self, character: CharacterProfile) -> str

Primary path: select sample-text style from (character.energy, character.timbre, character.pitch_register) directly — deterministic, no string matching.

energy	timbre	Sample Text Style
subdued / soft_spoken	any	Quiet, understated first-person line
measured	gravelly, nasal, monotone-adjacent	Wise elder / contemplative narration
subdued	monotone, clear	Flat, mechanical/robotic delivery
volatile / commanding	any	Assertive dramatic narration
any, pitch_register=high + youthful context	—	Enthusiastic, energetic dialogue
narrator (fixed)	—	Descriptive landscape prose, even pace

Legacy fallback (only used when a profile predates the taxonomy and taxonomy fields are still at their defaults): substring-match voice_description for keywords (female/woman/girl/she, old/elder/gruff, robot/machine/synth, child/young, narrator), same behavior as before.

Output

• File: {output_dir}/{character_name_lower}_ref.wav
• Format: WAV, 24kHz sample rate, mono
• Size: Typically 100KB–500KB depending on text length

Voice Reference vs Chapter Generation

Aspect	Voice Reference	Chapter Generation
voice_ref param	None	Path to reference WAV
Purpose	Create initial voice identity	Clone voice to new text
exaggeration/cfg_weight	From taxonomy range + name hash (or narrator constants)	Same values, reused
instruct param	voice_description	None (dropped)
Watermark	Applied	Applied

Speaker Embedding Extraction

After voice reference generation, an embedding is extracted:

embedding = tts.extract_embedding(ref_path)
# Returns: bytes (numpy array serialized via .tobytes())
# Stored in: characters.speaker_embedding column

Uses Chatterbox's internal encode_speaker() or encode_spk() method.

Cast Distinctiveness Quality Gate

This is the acoustic counterpart to Pass 1b's text-level distinctiveness pre-check, and it is the gate that's allowed to actually block the pipeline.

@dataclass
class VoiceCollision:
    character_a: str
    character_b: str
    similarity: float
    remediation_attempts: int
    resolved: bool

Algorithm:

1. After all references are generated and embeddings extracted, compute pairwise cosine similarity across every character pair, including the narrator as a fixed comparison point (no character should sound like the narrator either).
2. For any pair with similarity > 0.92 and who co-occur (per the same co-occurrence definition used in Pass 1b — share an attributed-dialogue chapter): flag a VoiceCollision.
3. Automatic remediation (up to 3 attempts per flagged character): re-derive parameters using a disambiguated hash input — hashlib.md5(f"{character.name}#{attempt}".encode()) — keeping the same ATTRIBUTE_PARAM_MAP range, regenerate the reference, re-extract the embedding, re-check.
4. If still colliding after 3 attempts: write an entry to .umbra/defects.json (type: "voice_collision", both names, similarity score) and set both characters' status to collision_pending if not already.
5. Pipeline behavior: the orchestrator does not proceed to Pass 3 while any collision_pending status remains, unless the operator passes --allow-voice-collisions (logged in the project's run history) or resolves it via the web review UI.

This mirrors the project's "quality over speed" principle (00-project-overview.md): an unresolved collision between two characters who actually talk in the same scene is treated as a release blocker, not a cosmetic issue.

Human Approval Workflow (Required Gate)

Passing the distinctiveness gate makes a voice safe to listen to — it doesn't make it right. A character described as "gruff and commanding" can still get a reference that technically sits in the right parameter range and is acoustically distinct from everyone else, while simply not sounding like what the author had in mind. So every character's reference — after the distinctiveness gate, not instead of it — goes through an explicit human checkpoint before it's allowed to become the voice used anywhere else in the pipeline.

class VoiceReviewSession:
    def submit_for_review(self, character: CharacterProfile, ref_path: Path) -> ReviewItem
    def approve(self, character_name: str) -> None
    def reject(self, character_name: str) -> Path
    async def edit_with_prompt(self, character_name: str, prompt: str) -> CharacterProfile

Flow, per character:

1. Generate — as above: taxonomy-ranged params, hash-disambiguated, embedding extracted, distinctiveness-gated.
2. Present — queued in the web review UI: character name, voice_description, taxonomy summary, a playable reference WAV, and three actions.
3. Human chooses one:

Action	Effect
Approve	status = 'voiced_approved'. Parameters are locked as this character's "normal" baseline. Nothing else in the pipeline may change them except a future taxonomy revision (02-analysis.md, "Taxonomy Revision").
Reject	Regenerate using a disambiguated hash variant (name#attempt, same mechanism as collision remediation) — same ATTRIBUTE_PARAM_MAP range, same described character, just a different roll within it. Re-extract embedding, re-queue. This is for "wrong roll of the dice," not "wrong direction."
Edit with prompt	Human supplies free text describing the desired change — "make it gruffer and slower," "less nasal, lower the pitch." Routed through apply_edit_prompt() below.

async def apply_edit_prompt(character: CharacterProfile, prompt: str) -> CharacterProfile:
    """LLM translates a free-text director's note into concrete taxonomy
    deltas. Returns a candidate updated profile; does not mutate the
    stored profile until the human approves the regenerated result."""

You are adjusting a TTS voice profile based on director feedback.

Current profile:
{taxonomy_json}

Director's note: "{prompt}"

Return ONLY a JSON object containing the fields that should change, from:
pitch_register, pace_baseline, energy, timbre, texture, formality,
emotional_baseline, distinguishing_trait. Only include a field if the
note actually justifies changing it. Do not invent unrelated changes.

4. On an edit prompt: apply the returned deltas to a candidate profile → re-derive (exaggeration, cfg_weight) via _character_params() → regenerate the reference and sample text → re-extract the embedding → re-run the cast distinctiveness check (an edit can introduce a collision that didn't exist before — it isn't optional just because this is round two) → re-present per step 2. Loop continues until Approve.
5. Pipeline gate: Pass 3's manifest finalization and Pass 5 generation will not bind or render against any character whose status != 'voiced_approved'. Pass 3's text-level segmentation/attribution can still run in parallel since it doesn't touch audio — but nothing downstream of voice selection proceeds for an unapproved character.

Table	Fields
voice_review_history	id, character_name, action (generated/approved/rejected/edit_requested), prompt_text (nullable), resulting_exaggeration, resulting_cfg_weight, taxonomy_snapshot_json, reviewed_by, reviewed_at

This table is what later lets a Pass 3 "Always" promotion (04-mapping.md, "Speech Descriptor Extractor & Review Queue") reopen exactly this same loop on an already-approved character, with full history of how the voice got to where it is.

Once approved, this is the "normal" voice referenced by everything downstream: Pass 3's punctuation-driven prosody hints modulate around it, and human-reviewed per-line descriptor overrides modulate around it too. Neither layer ever resets the baseline itself without going back through this same approval loop.

Approved Voice = Reference WAV + Captured Vmatrix

An approval now records two artifacts for the character (this is the requirement: a wav to listen to + the numbers inside the TTS that are the voice):

Artifact	Path	Purpose
Reference WAV	.umbra/voices/{slug}_ref.wav	Listenable sample of the approved voice
Vmatrix (.pt)	.umbra/voice_prompts/{slug}.pt	The captured voice identity used for consistency

The vmatrix is what Qwen3-TTS actually conditions on to reproduce a voice:

• ref_code — the ICL "8k matrix": a (T, 16) tensor of codec tokens produced by re-speaking the reference WAV through the model's codec encoder.
• ref_spk_embedding — the 2048-dim extracted speaker x-vector.

It is captured once (server-side POST /v1/voice-prompt, x_vector_only=0) and cached under voice_prompts/{slug}.pt. POST /api/speakers/voice-status approve calls build_voice_prompt(...) and records vmatrix_path in the voice registry next to voice_ref_path. Every subsequent utterance passes prompt_slug={slug} so it reuses the exact same numbers — a character's voice can never drift or "change" between chapters. If the reference WAV is ever replaced, the .pt must be rebuilt (approve again, or force=True) or generation will silently use the stale voice.

Note on modes: x_vector_only=True (embedding-only) is the legacy-coarse capture and must NOT be used for production consistency. The full vmatrix (x_vector_only=False) is the canonical mode and the engine default.

Skip Logic

Voice references are not regenerated if:

1. character.voice_ref_path is set in DB
2. The referenced file exists on disk
3. character.status == 'voiced_approved' — anything else (collision_pending, mid-review, or rolled back by a Pass 3 "Always" promotion) is always regenerated, since skipping would silently re-ship a voice that's either known-broken or mid-revision

This allows incremental pipeline runs.

GPU Requirements

• Chatterbox model: ~3.2 GB VRAM
• One voice ref: ~3-8 seconds to generate
• 29 characters: ~2-4 minutes of GPU time total (plus remediation regenerations for any flagged collisions)
• The human approval loop adds wall-clock time, not GPU load, beyond what's already accounted for — each reject/edit cycle is just another single-reference regeneration
• Requires LM Studio to be unloaded first (VRAM conflict)