UmbraAudio/specs/05-generation.md

Spec: Audio Generation (Pass 3)

Overview

Pass 3 generates audio for every utterance in every chapter using Qwen3-TTS (primary engine) with per-character voice reference samples (*_ref.wav). Each utterance gets its own WAV file. Generation resolves each utterance's speaker to a voice reference, then clones that voice for the utterance. Edge-TTS remains available for test samples and as a fallback path inside the voice designer.

Components

Qwen3-TTS Engine (voices/qwen_tts.py)

class QwenTTSEngine:
    def __init__(self, device="cuda", sample_rate=24000, qwen_tts_url: str) -> None
    def load_remote_model(self) -> None
    def generate(self, text: str, output_path: Path, voice_ref: Path | None = None, instruct: str = "") -> Path
    def unload_all(self) -> None

Qwen3-TTS Features:

• Remote/on-prem synthesis via qwen_tts_url (LM Studio / Qwen server)
• Voice cloning from a reference WAV per character
• Optional instruct text for prosody/emotion direction (e.g. rising question intonation)
• WAV output at 24kHz sample rate

Edge-TTS (voices/edge_tts.py) — Test Samples & Fallback

class EdgeTTSEngine:
    def __init__(self) -> None
    def generate(self, text: str, output_path: Path, voice: str, pitch_shift: int = 0) -> Path

• Used by scripts/generate_test_samples.py (verification phrase per speaker) and as the _generate_edge_fallback path in the voice designer.
• 27 distinct base voices (US, UK, AU, IE, IN, CA) with pitch shifting for variety.

Voice Reference Resolution

def resolve_reference(
    speaker_name: str,
    voice_refs: dict[str, Path],
    alias_map: dict[str, str],
    narrator_ref: Path | None = None,
) -> Path | None

voice_refs is built by scanning .umbra/voices/*_ref.wav; the key is the filename stem with underscores converted to spaces (e.g. captain_weston_ref.wav → captain weston).

Resolution Order:

1. Direct match against voice_refs keys (speaker primary name)
2. Case-insensitive match against reference names
3. Alias map lookup (primary names + aliases from speakers.json, all lowercased)
4. Fall back to narrator_ref (the narrator's reference WAV); if no narrator ref exists, fall back to the Kainda reference

Fallback when no reference resolves at all: the utterance is generated with a plain instruction ("A calm, neutral voice speaking clearly.") instead of a cloned voice.

Generation Flow

For each chapter:
    │
    ├── 1. Load utterances from DB
    │       SELECT id, speaker_name, text, utterance_index
    │       FROM utterances WHERE chapter_id = ? ORDER BY utterance_index
    │
    ├── 2. Create output directory
    │       .umbra/chapter_audio/ch{id}/
    │
    └── 3. For each utterance:
            │
            ├── Skip empty text
            ├── Skip if file exists
            │       if output_path.exists(): continue
            │
            ├── Resolve voice reference
            │       ref_path = resolve_reference(speaker, voice_refs, alias_map, narrator_ref)
            │
            ├── Generate audio (Qwen3-TTS)
            │       engine.generate(text, output_path, voice_ref=ref_path)
            │       # no ref → engine.generate(text, output_path, instruct=neutral)
            │
            ├── Verify output file exists (else count as failed)
            │
            └── Handle errors
                    Log error, continue to next utterance

Engine mode: the engine uses remote synthesis when cfg.tts.qwen_tts_url is set (the default deployment). The remote server holds the VoiceDesign and Base models loaded and answers /v1/tts; the local QwenTTSEngine only loads models itself when no URL is configured.

Output Structure

.umbra/chapter_audio/
├── ch5/
│   ├── utt_0000.wav
│   ├── utt_0001.wav
│   └── ... (one WAV per utterance)
├── ch6/
│   ├── utt_0000.wav
│   └── ...
└── ch7/
    ├── utt_0000.wav
    └── ...

Each WAV is named utt_{utterance_index:04d}.wav, so filenames are stable across re-runs and regeneration after utterance edits deletes and recreates the matching file.

Remote Server Coordination

• The engine checks VRAM on start (nvidia-smi) and logs used/free memory; the remote model is expected to be the only heavy VRAM consumer on the serving host.
• The remote server keeps the VoiceDesign model loaded so a single synthesis call can switch between voice-cloning (ref_audio) and description-driven (voice_description) generation without a model swap — the per-utterance Clone is what drives bulk generation, so the Base model stays resident.
• On completion, unload_all() is called only on the client side (remote models are left for the server's lifecycle manager).

Scripts

generate_all_audio.py (bulk generation)

# Background chapter audio generation
nohup python scripts/generate_all_audio.py > .umbra/generation.log 2>&1 &

Behavior:

• Loads voice references from .umbra/voices/*_ref.wav and speakers/aliases from .umbra/speakers.json
• Iterates chapters in chapter_index order, generating each missing utterance WAV
• Logs per-chapter progress (+N gen, +M fail), running total, and elapsed minutes
• Emits progress estimates assuming ~6.4s per utterance
• Exits with a final COMPLETE: N generated, M failed summary
• Accepts optional status_callback and generation_manager hooks so the web UI can display live progress (see 09-web-review-ui.md)

Resume semantics: because existing WAVs are skipped, re-running the script after a crash or interruption resumes exactly where it left off.

Voice Library

Voice References (.umbra/voices/*_ref.wav)

Each speaker gets a reference WAV synthesized from the verification phrase "When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors..." (the Rainbow Passage — a phonetically balanced paragraph covering all English phonemes) using their taxonomy-derived voice description (see 03-voice-design.md).

• Narrator → narrator_ref.wav
• General/misc speakers (from GENERAL_SPEAKER_NAMES) → general_ref.wav
• Named speakers → {name_lower_spaces_to_underscores}_ref.wav

Reference count in current deployment: 92 speakers in speakers.json, ~26 unique reference WAVs generated (many characters intentionally share a reference or fall back to the narrator reference rather than each getting a distinct clone).

Voice Variety Strategy

1. Distinct voice descriptions: each character's taxonomy (gender, age, timbre, pitch_register, energy, texture, formality, accent, distinguishing_trait) is rendered into a unique natural-language description via build_voice_description()
2. Voice cloning: the speaker's reference WAV is the identity anchor; clone synthesis preserves that identity across all utterances
3. No pitch-shifting: variety comes from the reference voice itself, not post-hoc pitch shifts

Error Handling

• Empty Text: Skip utterances with no text
• File Exists: Skip if output file already exists
• No Reference Resolved: Fall back to narrator reference (or Kainda), else neutral instruction
• TTS Failure (QwenTTSUnavailableError — connect refused, timeout after 300s, non-200 HTTP, or response without audio): log and continue to next utterance
• Per-utterance exception: caught, logged, counted as failed — never aborts the chapter run

Performance

Metric	Value
Rate	~6.4 seconds/utterance (remote synthesis)
Full book (6,861 utterances)	~12 hours
Progress logging	Every 50 utterances, plus per-chapter summary
Resume	Idempotent via file-existence skip

Success rate is reported per run as generated / failed; failed utterances can be regenerated by re-running the script (only the missing files are produced).

Quality Considerations

• All voices tested with verification phrase (test samples reviewed in the web UI)
• Web UI allows human review of generated audio per chapter (/chapters-audio)
• Utterance text/speaker edits in the web UI delete the stale WAV and trigger background regeneration
• Voice references can be regenerated individually or en masse from the web UI

Key Design Decisions

1. Qwen3-TTS over Edge-TTS: cloning from per-character reference samples gives stable, character-accurate voices; Edge-TTS is retained only for test samples and the _generate_edge_fallback path in the voice designer
2. Remote synthesis: the Qwen server (qwen_tts_url) hosts the models, so bulk generation doesn't fight the LLM (LM Studio) for local GPU/VRAM
3. Voice references over voice strings: identity comes from *_ref.wav cloning instead of voice_assignments.json voice strings with pitch shifts
4. Sequential generation: avoids overloading the remote server and simplifies resumability
5. File-based caching: skip already-generated files; regeneration after edits is file-targeted
6. Error resilience: individual utterance failures never abort a chapter run; a final summary counts successes and failures