UmbraAudio/specs/05-generation.md

Spec: Audio Generation (Pass 3)

Overview

Pass 3 generates audio for every utterance in every chapter using Qwen3-TTS (primary engine) with per-character voice reference samples (*_ref.wav). Each utterance gets its own WAV file. Generation resolves each utterance's speaker to a voice reference, then clones that voice for the utterance. Edge-TTS remains available for test samples and as a fallback path inside the voice designer.

Components

Qwen3-TTS Engine (voices/qwen_tts.py)

class QwenTTSEngine:
    def __init__(self, device="cuda", sample_rate=24000, qwen_tts_url: str) -> None
    def load_remote_model(self) -> None
    def generate(self, text: str, output_path: Path, voice_ref: Path | None = None, instruct: str = "", prompt_slug: str | None = None) -> Path
    def build_voice_prompt(self, voice_ref: Path, ref_text: str | None = None, slug: str | None = None, x_vector_only: bool = False, force: bool = False) -> str
    def unload_all(self) -> None

Qwen3-TTS Features:

• Remote/on-prem synthesis via qwen_tts_url (LM Studio / Qwen server)
• Voice cloning from a reference WAV per character
• Optional instruct text for prosody/emotion direction (e.g. rising question intonation)
• Vmatrix capture (build_voice_prompt): computes the voice's "8k matrix" (ICL ref_code codec-token matrix) + speaker embedding, caches it server-side as voice_prompts/{slug}.pt, returns the prompt_slug to pass to generate().
• prompt_slug generation (the consistency path): every utterance of a character passes the same slug, so the exact same cached vmatrix is reused — voice identity can never drift between utterances or chapters.
• WAV output at 24kHz sample rate

x_vector_only default is False (full vmatrix). The full capture (ref_code + embedding) is the canonical mode; the embed-only mode (True) is legacy/experimental and caused the historical "voice changed between chapters" problem.

Vmatrix Capture in the Generation Flow

Because build_voice_prompt defaults to /v1/voice-prompt server semantics, the bulk generator captures each distinct speaker's vmatrix once per run (before any utterance is generated) and reuses the slug for all of that speaker's utterances:

for each distinct speaker with a missing .pt:
    prompt_slug = build_voice_prompt(vref, ref_text=VOICE_REFERENCE_TEXT, x_vector_only=False)
    # server writes/updates voice_prompts/{slug}.pt = {ref_code (T,16), ref_spk_embedding (2048,), icl_mode: True}

The .pt cache is the same filesystem location the server writes to (PROMPT_DIR, default .umbra/voice_prompts), so it persists across runs and is shipped to the GPU host via tts_server/sync_voice_data.sh.

Edge-TTS (voices/edge_tts.py) — Test Samples & Fallback

class EdgeTTSEngine:
    def __init__(self) -> None
    def generate(self, text: str, output_path: Path, voice: str, pitch_shift: int = 0) -> Path

• Used by scripts/generate_test_samples.py (verification phrase per speaker) and as the _generate_edge_fallback path in the voice designer.
• 27 distinct base voices (US, UK, AU, IE, IN, CA) with pitch shifting for variety.

Voice Reference Resolution

def resolve_reference(
    speaker_name: str,
    voice_refs: dict[str, Path],
    alias_map: dict[str, str],
    narrator_ref: Path | None = None,
) -> Path | None

voice_refs is built by scanning .umbra/voices/*_ref.wav; the key is the filename stem with underscores converted to spaces (e.g. captain_weston_ref.wav → captain weston).

Resolution Order:

1. Direct match against voice_refs keys (speaker primary name)
2. Case-insensitive match against reference names
3. Alias map lookup (primary names + aliases from speakers.json, all lowercased)
4. Fall back to narrator_ref (the narrator's reference WAV); if no narrator ref exists, fall back to the Kainda reference

Fallback when no reference resolves at all: the utterance is generated with a plain instruction ("A calm, neutral voice speaking clearly.") instead of a cloned voice.

Generation Flow

For each chapter:
    │
    ├── 0. Capture vmatrix for each distinct speaker (once per run)
    │       prompt_slug = engine.build_voice_prompt(vref, ref_text=VOICE_REFERENCE_TEXT,
    │                                              x_vector_only=False)   # full "8k matrix" + embedding
    │       # → voice_prompts/{slug}.pt (cached; missing only when the ref changed)
    │
    ├── 1. Load utterances from DB
    │       SELECT id, speaker_name, text, utterance_index
    │       FROM utterances WHERE chapter_id = ? ORDER BY utterance_index
    │
    ├── 2. Create output directory
    │       .umbra/chapter_audio/ch{id}/
    │
    └── 3. For each utterance:
            │
            ├── Skip empty text
            ├── Skip if file exists
            │       if output_path.exists(): continue
            │
            ├── Resolve voice reference → prompt_slug
            │       ref_path = resolve_reference(speaker, voice_refs, alias_map, narrator_ref)
            │       slug = engine.slug_for_ref(ref_path)
            │
            ├── Generate audio (Qwen3-TTS) — identical cached vmatrix for the speaker
            │       engine.generate(text, output_path, prompt_slug=slug)
            │       # no ref → engine.generate(text, output_path, instruct=neutral)
            │
            ├── Verify output file exists (else count as failed)
            │
            └── Handle errors
                    Log error, continue to next utterance

Note the two-layer design: step 0 captures the vmatrix (a set of numbers) once per speaker; step 3 reuses that identical capture for every utterance via prompt_slug. This is what guarantees a character's voice never drifts across a chapter or the book.

Engine mode: the engine uses remote synthesis when cfg.tts.qwen_tts_url is set (the default deployment). The remote server holds the VoiceDesign and Base models loaded and answers /v1/tts; the local QwenTTSEngine only loads models itself when no URL is configured.

Output Structure

.umbra/chapter_audio/
├── ch5/
│   ├── utt_0000.wav
│   ├── utt_0001.wav
│   └── ... (one WAV per utterance)
├── ch6/
│   ├── utt_0000.wav
│   └── ...
└── ch7/
    ├── utt_0000.wav
    └── ...

Each WAV is named utt_{utterance_index:04d}.wav, so filenames are stable across re-runs and regeneration after utterance edits deletes and recreates the matching file.

Remote Server Coordination

• The engine checks VRAM on start (nvidia-smi) and logs used/free memory; the remote model is expected to be the only heavy VRAM consumer on the serving host.
• The remote server keeps the VoiceDesign model loaded so a single synthesis call can switch between voice-cloning (ref_audio) and description-driven (voice_description) generation without a model swap — the per-utterance Clone is what drives bulk generation, so the Base model stays resident.
• On completion, unload_all() is called only on the client side (remote models are left for the server's lifecycle manager).

Scripts

generate_all_audio.py (bulk generation)

# Background chapter audio generation
nohup python scripts/generate_all_audio.py > .umbra/generation.log 2>&1 &

Behavior:

• Loads voice references from .umbra/voices/*_ref.wav and speakers/aliases from .umbra/speakers.json
• Iterates chapters in chapter_index order, generating each missing utterance WAV
• Logs per-chapter progress (+N gen, +M fail), running total, and elapsed minutes
• Emits progress estimates assuming ~6.4s per utterance
• Exits with a final COMPLETE: N generated, M failed summary
• Accepts optional status_callback and generation_manager hooks so the web UI can display live progress (see 09-web-review-ui.md)

Resume semantics: because existing WAVs are skipped, re-running the script after a crash or interruption resumes exactly where it left off.

Voice Library

Voice References (.umbra/voices/*_ref.wav)

Each speaker gets a reference WAV synthesized from the verification phrase "When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors..." (the Rainbow Passage — a phonetically balanced paragraph covering all English phonemes) using their taxonomy-derived voice description (see 03-voice-design.md).

• Narrator → narrator_ref.wav
• General/misc speakers (from GENERAL_SPEAKER_NAMES) → general_ref.wav
• Named speakers → {name_lower_spaces_to_underscores}_ref.wav

Reference count in current deployment: 92 speakers in speakers.json, ~26 unique reference WAVs generated (many characters intentionally share a reference or fall back to the narrator reference rather than each getting a distinct clone).

Voice Variety Strategy

1. Distinct voice descriptions: each character's taxonomy (gender, age, timbre, pitch_register, energy, texture, formality, accent, distinguishing_trait) is rendered into a unique natural-language description via build_voice_description()
2. Voice cloning: the speaker's reference WAV is the identity anchor; clone synthesis preserves that identity across all utterances
3. No pitch-shifting: variety comes from the reference voice itself, not post-hoc pitch shifts

Error Handling

• Empty Text: Skip utterances with no text
• File Exists: Skip if output file already exists
• No Reference Resolved: Fall back to narrator reference (or Kainda), else neutral instruction
• TTS Failure (QwenTTSUnavailableError — connect refused, timeout after 300s, non-200 HTTP, or response without audio): log and continue to next utterance
• Per-utterance exception: caught, logged, counted as failed — never aborts the chapter run

Performance

Metric	Value
Rate	~6.4 seconds/utterance (remote synthesis)
Full book (6,861 utterances)	~12 hours
Progress logging	Every 50 utterances, plus per-chapter summary
Resume	Idempotent via file-existence skip

Success rate is reported per run as generated / failed; failed utterances can be regenerated by re-running the script (only the missing files are produced).

Quality Considerations

• All voices tested with verification phrase (test samples reviewed in the web UI)
• Web UI allows human review of generated audio per chapter (/chapters-audio)
• Utterance text/speaker edits in the web UI delete the stale WAV and trigger background regeneration
• Voice references can be regenerated individually or en masse from the web UI

Key Design Decisions

1. Qwen3-TTS over Edge-TTS: cloning from per-character reference samples gives stable, character-accurate voices; Edge-TTS is retained only for test samples and the _generate_edge_fallback path in the voice designer
2. Remote synthesis: the Qwen server (qwen_tts_url) hosts the models, so bulk generation doesn't fight the LLM (LM Studio) for local GPU/VRAM
3. Vmatrix identity from references: identity comes from the captured vmatrix of each *_ref.wav (ICL codec-token "8k matrix" + speaker embedding, cached as voice_prompts/{slug}.pt), not from voice_assignments.json voice strings with pitch shifts, and not from a coarse embedding-only capture. Every utterance reuses the same .pt via prompt_slug.
4. Sequential generation: avoids overloading the remote server and simplifies resumability
5. File-based caching: skip already-generated files; regeneration after edits is file-targeted
6. Error resilience: individual utterance failures never abort a chapter run; a final summary counts successes and failures