UmbraAudio/specs/05-generation.md

# Spec: Audio Generation (Pass 3)

## Overview

Pass 3 generates audio for every utterance in every chapter using **Qwen3-TTS** (primary engine) with per-character **voice reference samples** (`*_ref.wav`). Each utterance gets its own WAV file. Generation resolves each utterance's speaker to a voice reference, then clones that voice for the utterance. Edge-TTS remains available for test samples and as a fallback path inside the voice designer.

## Components

### Qwen3-TTS Engine (`voices/qwen_tts.py`)

```python
class QwenTTSEngine:
    def __init__(self, device="cuda", sample_rate=24000, qwen_tts_url: str) -> None
    def load_remote_model(self) -> None
    def generate(self, text: str, output_path: Path, voice_ref: Path | None = None, instruct: str = "", prompt_slug: str | None = None) -> Path
    def build_voice_prompt(self, voice_ref: Path, ref_text: str | None = None, slug: str | None = None, x_vector_only: bool = False, force: bool = False) -> str
    def unload_all(self) -> None
```

**Qwen3-TTS Features:**
- Remote/on-prem synthesis via `qwen_tts_url` (LM Studio / Qwen server)
- Voice cloning from a reference WAV per character
- Optional `instruct` text for prosody/emotion direction (e.g. rising question intonation)
- **Vmatrix capture** (`build_voice_prompt`): computes the voice's "8k matrix" (ICL `ref_code` codec-token matrix) + speaker embedding, caches it server-side as `voice_prompts/{slug}.pt`, returns the `prompt_slug` to pass to `generate()`.
- **`prompt_slug` generation** (the consistency path): every utterance of a character passes the same slug, so the exact same cached vmatrix is reused — voice identity can never drift between utterances or chapters.
- WAV output at 24kHz sample rate

> **`x_vector_only` default is `False` (full vmatrix).** The full capture (ref_code + embedding) is the canonical mode; the embed-only mode (`True`) is legacy/experimental and caused the historical "voice changed between chapters" problem.

### Vmatrix Capture in the Generation Flow

Because `build_voice_prompt` defaults to `/v1/voice-prompt` server semantics, the bulk
generator captures each distinct speaker's vmatrix once per run (before any utterance is
generated) and reuses the slug for all of that speaker's utterances:

```
for each distinct speaker with a missing .pt:
    prompt_slug = build_voice_prompt(vref, ref_text=VOICE_REFERENCE_TEXT, x_vector_only=False)
    # server writes/updates voice_prompts/{slug}.pt = {ref_code (T,16), ref_spk_embedding (2048,), icl_mode: True}
```

The `.pt` cache is the same filesystem location the server writes to (`PROMPT_DIR`,
default `.umbra/voice_prompts`), so it persists across runs and is shipped to the GPU
host via `tts_server/sync_voice_data.sh`.

### Edge-TTS (`voices/edge_tts.py`) — Test Samples & Fallback

```python
class EdgeTTSEngine:
    def __init__(self) -> None
    def generate(self, text: str, output_path: Path, voice: str, pitch_shift: int = 0) -> Path
```

- Used by `scripts/generate_test_samples.py` (verification phrase per speaker) and as the `_generate_edge_fallback` path in the voice designer.
- 27 distinct base voices (US, UK, AU, IE, IN, CA) with pitch shifting for variety.

### Voice Reference Resolution

```python
def resolve_reference(
    speaker_name: str,
    voice_refs: dict[str, Path],
    alias_map: dict[str, str],
    narrator_ref: Path | None = None,
) -> Path | None
```

`voice_refs` is built by scanning `.umbra/voices/*_ref.wav`; the key is the filename stem with underscores converted to spaces (e.g. `captain_weston_ref.wav` → `captain weston`).

**Resolution Order:**
1. Direct match against `voice_refs` keys (speaker primary name)
2. Case-insensitive match against reference names
3. Alias map lookup (primary names + aliases from `speakers.json`, all lowercased)
4. Fall back to `narrator_ref` (the narrator's reference WAV); if no narrator ref exists, fall back to the Kainda reference

**Fallback when no reference resolves at all:** the utterance is generated with a plain instruction (`"A calm, neutral voice speaking clearly."`) instead of a cloned voice.

### Generation Flow

```
For each chapter:
    │
    ├── 0. Capture vmatrix for each distinct speaker (once per run)
    │       prompt_slug = engine.build_voice_prompt(vref, ref_text=VOICE_REFERENCE_TEXT,
    │                                              x_vector_only=False)   # full "8k matrix" + embedding
    │       # → voice_prompts/{slug}.pt (cached; missing only when the ref changed)
    │
    ├── 1. Load utterances from DB
    │       SELECT id, speaker_name, text, utterance_index
    │       FROM utterances WHERE chapter_id = ? ORDER BY utterance_index
    │
    ├── 2. Create output directory
    │       .umbra/chapter_audio/ch{id}/
    │
    └── 3. For each utterance:
            │
            ├── Skip empty text
            ├── Skip if file exists
            │       if output_path.exists(): continue
            │
            ├── Resolve voice reference → prompt_slug
            │       ref_path = resolve_reference(speaker, voice_refs, alias_map, narrator_ref)
            │       slug = engine.slug_for_ref(ref_path)
            │
            ├── Generate audio (Qwen3-TTS) — identical cached vmatrix for the speaker
            │       engine.generate(text, output_path, prompt_slug=slug)
            │       # no ref → engine.generate(text, output_path, instruct=neutral)
            │
            ├── Verify output file exists (else count as failed)
            │
            └── Handle errors
                    Log error, continue to next utterance
```

Note the two-layer design: step 0 captures the vmatrix (a set of numbers) **once** per
speaker; step 3 reuses that identical capture for every utterance via `prompt_slug`.
This is what guarantees a character's voice never drifts across a chapter or the book.

**Engine mode:** the engine uses remote synthesis when `cfg.tts.qwen_tts_url` is set (the default deployment). The remote server holds the VoiceDesign and Base models loaded and answers `/v1/tts`; the local `QwenTTSEngine` only loads models itself when no URL is configured.

### Output Structure

```
.umbra/chapter_audio/
├── ch5/
│   ├── utt_0000.wav
│   ├── utt_0001.wav
│   └── ... (one WAV per utterance)
├── ch6/
│   ├── utt_0000.wav
│   └── ...
└── ch7/
    ├── utt_0000.wav
    └── ...
```

Each WAV is named `utt_{utterance_index:04d}.wav`, so filenames are stable across re-runs and regeneration after utterance edits deletes and recreates the matching file.

### Remote Server Coordination

- The engine checks VRAM on start (`nvidia-smi`) and logs used/free memory; the remote model is expected to be the only heavy VRAM consumer on the serving host.
- The remote server keeps the **VoiceDesign model** loaded so a *single* synthesis call can switch between voice-cloning (`ref_audio`) and description-driven (`voice_description`) generation without a model swap — the per-utterance Clone is what drives bulk generation, so the Base model stays resident.
- On completion, `unload_all()` is called only on the client side (remote models are left for the server's lifecycle manager).

### Scripts

#### generate_all_audio.py (bulk generation)

```bash
# Background chapter audio generation
nohup python scripts/generate_all_audio.py > .umbra/generation.log 2>&1 &
```

**Behavior:**
- Loads voice references from `.umbra/voices/*_ref.wav` and speakers/aliases from `.umbra/speakers.json`
- Iterates chapters in `chapter_index` order, generating each missing utterance WAV
- Logs per-chapter progress (`+N gen, +M fail`), running total, and elapsed minutes
- Emits progress estimates assuming ~6.4s per utterance
- Exits with a final `COMPLETE: N generated, M failed` summary
- Accepts optional `status_callback` and `generation_manager` hooks so the web UI can display live progress (see `09-web-review-ui.md`)

**Resume semantics:** because existing WAVs are skipped, re-running the script after a crash or interruption resumes exactly where it left off.

## Voice Library

### Voice References (`.umbra/voices/*_ref.wav`)

Each speaker gets a reference WAV synthesized from the verification phrase *"When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors..."* (the Rainbow Passage — a phonetically balanced paragraph covering all English phonemes) using their taxonomy-derived voice description (see `03-voice-design.md`).

- Narrator → `narrator_ref.wav`
- General/misc speakers (from `GENERAL_SPEAKER_NAMES`) → `general_ref.wav`
- Named speakers → `{name_lower_spaces_to_underscores}_ref.wav`

**Reference count in current deployment:** 92 speakers in `speakers.json`, ~26 unique reference WAVs generated (many characters intentionally share a reference or fall back to the narrator reference rather than each getting a distinct clone).

### Voice Variety Strategy

1. **Distinct voice descriptions**: each character's taxonomy (`gender`, `age`, `timbre`, `pitch_register`, `energy`, `texture`, `formality`, `accent`, `distinguishing_trait`) is rendered into a unique natural-language description via `build_voice_description()`
2. **Voice cloning**: the speaker's reference WAV is the identity anchor; clone synthesis preserves that identity across all utterances
3. **No pitch-shifting**: variety comes from the reference voice itself, not post-hoc pitch shifts

## Error Handling

- **Empty Text**: Skip utterances with no text
- **File Exists**: Skip if output file already exists
- **No Reference Resolved**: Fall back to narrator reference (or Kainda), else neutral instruction
- **TTS Failure** (`QwenTTSUnavailableError` — connect refused, timeout after 300s, non-200 HTTP, or response without audio): log and continue to next utterance
- **Per-utterance exception**: caught, logged, counted as failed — never aborts the chapter run

## Performance

| Metric | Value |
|--------|-------|
| Rate | ~6.4 seconds/utterance (remote synthesis) |
| Full book (6,861 utterances) | ~12 hours |
| Progress logging | Every 50 utterances, plus per-chapter summary |
| Resume | Idempotent via file-existence skip |

Success rate is reported per run as `generated / failed`; failed utterances can be regenerated by re-running the script (only the missing files are produced).

## Quality Considerations

- All voices tested with verification phrase (test samples reviewed in the web UI)
- Web UI allows human review of generated audio per chapter (`/chapters-audio`)
- Utterance text/speaker edits in the web UI delete the stale WAV and trigger background regeneration
- Voice references can be regenerated individually or en masse from the web UI

## Key Design Decisions

1. **Qwen3-TTS over Edge-TTS**: cloning from per-character reference samples gives stable, character-accurate voices; Edge-TTS is retained only for test samples and the `_generate_edge_fallback` path in the voice designer
2. **Remote synthesis**: the Qwen server (`qwen_tts_url`) hosts the models, so bulk generation doesn't fight the LLM (LM Studio) for local GPU/VRAM
3. **Vmatrix identity from references**: identity comes from the **captured vmatrix** of each `*_ref.wav` (ICL codec-token "8k matrix" + speaker embedding, cached as `voice_prompts/{slug}.pt`), not from `voice_assignments.json` voice strings with pitch shifts, and not from a coarse embedding-only capture. Every utterance reuses the same `.pt` via `prompt_slug`.
4. **Sequential generation**: avoids overloading the remote server and simplifies resumability
5. **File-based caching**: skip already-generated files; regeneration after edits is file-targeted
6. **Error resilience**: individual utterance failures never abort a chapter run; a final summary counts successes and failures