UmbraAudio/Umbra_Voice_generation.md

# UmbraAudio Voice Generation Specification

## Overview

UmbraAudio generates multi-voice audiobooks from English manuscripts. Each character
gets a unique, consistent voice. The voice identity is stored as a **vmatrix** — the
set of numbers inside Qwen3-TTS that *is* the voice:

1. the **ICL "8k matrix"** (`ref_code`) — a `(T, 16)` tensor of codec tokens rebuilt by
   re-speaking the reference WAV through the model's codec encoder, AND
2. the **speaker embedding** (`ref_spk_embedding`) — a 2048-dim float32/bfloat16 vector
   extracted from Qwen3-TTS's speaker encoder.

The vmatrix is captured ONCE per approved voice and cached server-side as a `.pt` prompt
(in `voice_prompts/{slug}.pt`). Every utterance reuses those exact cached numbers, so the
voice is bit-identical in identity across every utterance and chapter. **An approved voice
is therefore two artifacts: a WAV to listen to + the captured vmatrix for consistency.**

> Historical note: an earlier pipeline stored ONLY the 2048-dim speaker embedding
> (`x_vector_only=True`) as the single source of truth. That was the "nearly identical /
> drift-prone" coarse capture. The canonical capture is the **full vmatrix** (ref_code +
> embedding, `x_vector_only=False`).

## Architecture

### Voice Identity Pipeline

```
Approved reference WAV  (per speaker: the author-approved "*_ref.wav")
        │
        ▼
POST /v1/voice-prompt   (x_vector_only=0)
        │   model re-speaks the reference: codec encoder + speaker encoder
        ▼
vmatrix  { ref_code:          (T,16) int64    ICL "8k matrix"
           ref_spk_embedding: (2048,)         speaker x-vector
           x_vector_only_mode: False, icl_mode: True }
        │
        ▼
cached as .pt  under  voice_prompts/{slug}.pt  (PROMPT_DIR)
        │
        ▼
every utterance → generate(prompt_slug={slug}) reuses the SAME cached vmatrix
```

### Key Insight

The previous approach (WAV ref → extract a bare x-vector at generation time, or use the
embedding-only `x_vector_only=True` mode) produced **drift-prone / coarse** voices,
because neither kept the exact numbers the model conditions on. The **vmatrix** captures
the exact ICL codec-token matrix (`ref_code`) **plus** the speaker embedding the model
uses, caches it once, and reuses it for every utterance. This gives:

- **Perfect consistency**: every utterance of a character uses the exact same cached
  numbers → the voice can never drift or "change" between chapters
- **Faithful identity**: regeneration re-speaks the reference, so the voice is the
  reference voice, not a lossy summary of it
- **One-time capture**: the `.pt` is built once per approved voice (on Approve) and at
  the start of each generation run if missing
- **Distinct voices**: narrator / nanny / kainda measured F0s remain distinct

## File Structure

```
.umbra/
├── voice_prompts/              # VMXATRIX cache (.pt): the canonical voice identity
│   ├── kainda_walters_foster_murphy.pt   # ref_code (T,16) + ref_spk_embedding (2048,)
│   ├── nanny.pt
│   ├── narrator.pt
│   └── ...                     # one .pt per approved voice
├── voice_embeddings/           # LEGACY speaker-embedding cache (x_vector_only mode)
│   ├── narrator.npy            # 2048-dim float32 (8KB)
│   └── ...                     # kept only for fallback/experiments
├── voices/                     # Approved reference WAVs (the "listen to" artifact)
│   ├── narrator_ref.wav
│   ├── nanny_ref.wav
│   └── ...
├── speakers.json               # Character metadata
├── voice_assignments.json      # Edge-TTS voice mapping
├── voice_profiles.json         # LLM-generated voice profiles
└── project.db                  # Utterances, chapters, attributions
```

## Generation Modes

# Mode 1 is the CANONICAL mode (since the vmatrix fix):
# client captures the FULL vmatrix once, then generates via prompt_slug.

### Mode 1: Cached Vmatrix via prompt_slug (Preferred — Canonical)

Full voice identity (ICL "8k matrix" ref_code + speaker embedding) captured once and
cached as `.pt`. Every utterance reuses the same cached numbers.

```python
# Client captures the vmatrix once per approved voice (equivalent to Approve):
prompt_slug = engine.build_voice_prompt(
    ref_path,                          # the approved *_ref.wav
    ref_text=VOICE_REFERENCE_TEXT,     # Rainbow Passage
    x_vector_only=False,               # FULL vmatrix (ref_code + embedding)
    force=False,                       # True only if the ref wav changed
)

# Server: POST /v1/voice-prompt → cached voice_prompts/{prompt_slug}.pt
# containing { ref_code: (T,16), ref_spk_embedding: (2048,),
#              x_vector_only_mode: False, icl_mode: True }

# Every utterance then references the same identity:
engine.generate(text, output_path, prompt_slug=prompt_slug)
```

**Consistency**: every utterance of a character uses the exact same cached vmatrix → the
voice is identical across all chapters and can never silently change.

### Mode 2: Embedding-only (x_vector_only=1)

Captures ONLY `ref_spk_embedding` (`ref_code=None`). Coarse, drift-prone; this was the
pre-fix default and caused the "voice changed / inconsistent" problem. Keep for
experiments or machines without ICL VRAM headroom.

```python
POST /v1/voice-prompt  x_vector_only=1
# → cached voice_prompts/{slug}.pt with ref_code=None, icl_mode=False
```

### Mode 3: WAV Reference (fallback)

Used only when no cached prompt exists and no slug is passed — server extracts the
x-vector at generation time.

```python
POST /v1/tts
  ref_audio: <WAV file upload>  # 50-200KB
  text: "The captain nodded."
  temperature: 0.5
```

### Mode 4: VoiceDesign (New Voice Creation)

Used to create new character voices from text descriptions (also used to generate the
initial `*_ref.wav` for each character).

```python
POST /v1/tts
  voice_description: "A deep male voice with Australian accent"
  text: "The captain nodded."
  temperature: 0.5
```

## Voice Key Generation

### From Existing WAV Refs

```bash
python scripts/extract_embeddings.py --project DealHunterShip_4.0e_whole-to-audio
```

Loads Base model once, iterates all `*_ref.wav` files, extracts embeddings, saves as `.npy`.
Takes ~6 seconds for 94 speakers.

### From VoiceDesign Instruct Text

```bash
python scripts/extract_embeddings.py \
    --design "A young female voice with slight accent, speaking softly" \
    --slug new_character \
    --seed 42
```

Generates audio via VoiceDesign → extracts embedding via Base model → saves as `.npy`.

### Regenerating All Voice Samples

```bash
python scripts/extract_embeddings.py --force  # Re-extract from WAV refs
```

## Rainbow Passage

All voice references are generated using the Rainbow Passage — a phonetically balanced
paragraph covering all English phonemes. This ensures the speaker embedding captures the
full range of vocal characteristics.

```
When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow.
The rainbow is a division of white light into many beautiful colors.
These take the shape of a long round arch, with its path high above,
and its two ends apparently beyond the horizon.
There is, according to legend, a boiling pot of gold at one end.
People look, but no one ever finds it.
When a man looks for something beyond his reach,
his friends say he is looking for the pot of gold at the end of the rainbow.
```

## Server API

### POST /v1/tts

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| text | str | required | Text to synthesize |
| sample_rate | int | 24000 | Audio sample rate |
| voice_description | str | None | VoiceDesign instruct text |
| ref_audio | File | None | Voice reference WAV (fallback mode) |
| ref_text | str | None | Reference text for ICL mode |
| speaker_embedding | str | None | JSON-encoded float list (2048-dim) |
| temperature | float | 0.9 | Sampling temperature (lower = more consistent) |
| do_sample | bool | True | Enable sampling |
| seed | int | None | Random seed for reproducibility |

### POST /load

Load a model into memory.

```json
{"model_type": "base"}  // or "voicedesign"
```

### POST /unload

Unload a model to free memory.

```json
{"model_type": "all"}  // or "base" or "voicedesign"
```

### GET /health

Returns server status and loaded models.

## Consistency Guarantees

### Per-Utterance Consistency

Every utterance of a character passes the **same cached vmatrix** (`prompt_slug` → the
`.pt` with ref_code + embedding) → identical voice identity. Variation comes only from:
- Different text (codec-token sequence)
- Temperature (controlled per-chapter)

### Cross-Chapter Consistency

Same cached vmatrix used across all chapters → same voice identity.
The narrator always sounds like the narrator. Kainda always sounds like Kainda —
the voice can never silently change between chapters.

### Capture Discipline

A voice is only the SAME voice while its cached `.pt` matches its approved reference WAV.
When a `*_ref.wav` is regenerated/approved, the `.pt` MUST be rebuilt (the Approve flow
does this; `build_voice_prompt(..., force=True)` forces it) — otherwise chapter
generation silently uses the stale old voice. This is why "Kainda still had the old
voice": the `.pt` was built Aug 27, before the Aug 31 reference replacements.

## Chapter Generation Flow

1. **Preflight**: Verify TTS server is running, Base model loaded
2. **Validate**: Check which utterances need (re)generation
3. **Capture vmatrix**: For each distinct speaker with a missing/stale `.pt`,
   `build_voice_prompt(vref, x_vector_only=False)` → cached `voice_prompts/{slug}.pt`
4. **Generate**: For each utterance:
   - Resolve speaker → prompt_slug (the cached vmatrix)
   - Send to server with text + temperature + seed + prompt_slug
   - Save WAV to `.umbra/chapter_audio/ch{N}/utt{M}.wav`
5. **Stitch**: Concatenate utterance WAVs into chapter WAV
6. **Export**: Convert chapter WAV to MP3 via ffmpeg
7. **Verify**: Run quality checks

## Hardware Constraints

- **Mac 16 GB MPS**: One model resident at a time (Base OR VoiceDesign)
- **Base model**: ~3.5 GB VRAM, generates ~10s/utterance
- **VoiceDesign model**: ~3.5 GB VRAM, generates ~10s/utterance
- **ICL mode**: ~11 minutes/utterance on MPS (not viable for production)
- **x-vector mode**: ~10s/utterance on MPS (production mode)

## Model Paths

| Model | Path | Purpose |
|-------|------|---------|
| Base | `models/Qwen3-TTS-12Hz-1.7B-Base` | Voice cloning + embedding extraction |
| VoiceDesign | `models/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | New voice creation from instruct text |
| Tokenizer | `models/Qwen3-TTS-Tokenizer-12Hz` | Shared tokenizer |

## Configuration

### speakers.json

```json
{
  "speakers": [
    {
      "primary_name": "Narrator",
      "gender": "male",
      "accent": "american",
      "voice_status": "approved",
      "vocal_traits": {
        "pitch": "low",
        "energy": "measured",
        "timbre": "warm"
      }
    }
  ]
}
```

### voice_assignments.json

Maps character names to Edge-TTS voice identifiers (used for WAV ref generation).

```json
{
  "Narrator": "en-US-GuyNeural:-10",
  "Nanny": "en-US-AvaNeural",
  "Kainda Walters Foster Murphy": "en-US-JennyNeural"
}
```

## Troubleshooting

### Voice sounds wrong / "it's still the old voice"

1. Check the voice is approved and has a vmatrix:
   `ls .umbra/voice_prompts/{slug}.pt` (slug = ref filename minus `_ref.wav`)
2. If the `.pt` is missing or older than the ref WAV, rebuild it:
   approve again via the web UI, or force: `build_voice_prompt(ref, ref_text=Rainbow, x_vector_only=False, force=True)`
3. Verify the approved reference WAV (`.umbra/voices/{slug}_ref.wav` — listen to it)
4. Only then regenerate the character's chapter utterances

### TTS server not responding

```bash
curl http://localhost:8881/health
# If down: restart with qwen_tts_server.py
```

### Memory pressure

```bash
curl -X POST http://localhost:8881/unload -d '{"model_type": "all"}'
```

### Inconsistent voices across utterances

Solved by the vmatrix: every utterance passes `prompt_slug` → the same cached `.pt`
(ref_code + embedding). If a character's voice still changes, the `.pt` is being
rebuilt per utterance or is stale vs the approved ref — force-rebuild it once and
ensure generation uses `x_vector_only=False` (check logs for cache hits of the slug).