UmbraAudio/Umbra_Voice_generation.md

UmbraAudio Voice Generation Specification

Overview

UmbraAudio generates multi-voice audiobooks from English manuscripts. Each character gets a unique, consistent voice. The voice identity is stored as a vmatrix — the set of numbers inside Qwen3-TTS that is the voice:

1. the ICL "8k matrix" (ref_code) — a (T, 16) tensor of codec tokens rebuilt by re-speaking the reference WAV through the model's codec encoder, AND
2. the speaker embedding (ref_spk_embedding) — a 2048-dim float32/bfloat16 vector extracted from Qwen3-TTS's speaker encoder.

The vmatrix is captured ONCE per approved voice and cached server-side as a .pt prompt (in voice_prompts/{slug}.pt). Every utterance reuses those exact cached numbers, so the voice is bit-identical in identity across every utterance and chapter. An approved voice is therefore two artifacts: a WAV to listen to + the captured vmatrix for consistency.

Historical note: an earlier pipeline stored ONLY the 2048-dim speaker embedding (x_vector_only=True) as the single source of truth. That was the "nearly identical / drift-prone" coarse capture. The canonical capture is the full vmatrix (ref_code + embedding, x_vector_only=False).

Architecture

Voice Identity Pipeline

Approved reference WAV  (per speaker: the author-approved "*_ref.wav")
        │
        ▼
POST /v1/voice-prompt   (x_vector_only=0)
        │   model re-speaks the reference: codec encoder + speaker encoder
        ▼
vmatrix  { ref_code:          (T,16) int64    ICL "8k matrix"
           ref_spk_embedding: (2048,)         speaker x-vector
           x_vector_only_mode: False, icl_mode: True }
        │
        ▼
cached as .pt  under  voice_prompts/{slug}.pt  (PROMPT_DIR)
        │
        ▼
every utterance → generate(prompt_slug={slug}) reuses the SAME cached vmatrix

Key Insight

The previous approach (WAV ref → extract a bare x-vector at generation time, or use the embedding-only x_vector_only=True mode) produced drift-prone / coarse voices, because neither kept the exact numbers the model conditions on. The vmatrix captures the exact ICL codec-token matrix (ref_code) plus the speaker embedding the model uses, caches it once, and reuses it for every utterance. This gives:

• Perfect consistency: every utterance of a character uses the exact same cached numbers → the voice can never drift or "change" between chapters
• Faithful identity: regeneration re-speaks the reference, so the voice is the reference voice, not a lossy summary of it
• One-time capture: the .pt is built once per approved voice (on Approve) and at the start of each generation run if missing
• Distinct voices: narrator / nanny / kainda measured F0s remain distinct

File Structure

.umbra/
├── voice_prompts/              # VMXATRIX cache (.pt): the canonical voice identity
│   ├── kainda_walters_foster_murphy.pt   # ref_code (T,16) + ref_spk_embedding (2048,)
│   ├── nanny.pt
│   ├── narrator.pt
│   └── ...                     # one .pt per approved voice
├── voice_embeddings/           # LEGACY speaker-embedding cache (x_vector_only mode)
│   ├── narrator.npy            # 2048-dim float32 (8KB)
│   └── ...                     # kept only for fallback/experiments
├── voices/                     # Approved reference WAVs (the "listen to" artifact)
│   ├── narrator_ref.wav
│   ├── nanny_ref.wav
│   └── ...
├── speakers.json               # Character metadata
├── voice_assignments.json      # Edge-TTS voice mapping
├── voice_profiles.json         # LLM-generated voice profiles
└── project.db                  # Utterances, chapters, attributions

Generation Modes

Mode 1 is the CANONICAL mode (since the vmatrix fix):

client captures the FULL vmatrix once, then generates via prompt_slug.

Mode 1: Cached Vmatrix via prompt_slug (Preferred — Canonical)

Full voice identity (ICL "8k matrix" ref_code + speaker embedding) captured once and cached as .pt. Every utterance reuses the same cached numbers.

# Client captures the vmatrix once per approved voice (equivalent to Approve):
prompt_slug = engine.build_voice_prompt(
    ref_path,                          # the approved *_ref.wav
    ref_text=VOICE_REFERENCE_TEXT,     # Rainbow Passage
    x_vector_only=False,               # FULL vmatrix (ref_code + embedding)
    force=False,                       # True only if the ref wav changed
)

# Server: POST /v1/voice-prompt → cached voice_prompts/{prompt_slug}.pt
# containing { ref_code: (T,16), ref_spk_embedding: (2048,),
#              x_vector_only_mode: False, icl_mode: True }

# Every utterance then references the same identity:
engine.generate(text, output_path, prompt_slug=prompt_slug)

Consistency: every utterance of a character uses the exact same cached vmatrix → the voice is identical across all chapters and can never silently change.

Mode 2: Embedding-only (x_vector_only=1)

Captures ONLY ref_spk_embedding (ref_code=None). Coarse, drift-prone; this was the pre-fix default and caused the "voice changed / inconsistent" problem. Keep for experiments or machines without ICL VRAM headroom.

POST /v1/voice-prompt  x_vector_only=1
# → cached voice_prompts/{slug}.pt with ref_code=None, icl_mode=False

Mode 3: WAV Reference (fallback)

Used only when no cached prompt exists and no slug is passed — server extracts the x-vector at generation time.

POST /v1/tts
  ref_audio: <WAV file upload>  # 50-200KB
  text: "The captain nodded."
  temperature: 0.5

Mode 4: VoiceDesign (New Voice Creation)

Used to create new character voices from text descriptions (also used to generate the initial *_ref.wav for each character).

POST /v1/tts
  voice_description: "A deep male voice with Australian accent"
  text: "The captain nodded."
  temperature: 0.5

Voice Key Generation

From Existing WAV Refs

python scripts/extract_embeddings.py --project DealHunterShip_4.0e_whole-to-audio

Loads Base model once, iterates all *_ref.wav files, extracts embeddings, saves as .npy. Takes ~6 seconds for 94 speakers.

From VoiceDesign Instruct Text

python scripts/extract_embeddings.py \
    --design "A young female voice with slight accent, speaking softly" \
    --slug new_character \
    --seed 42

Generates audio via VoiceDesign → extracts embedding via Base model → saves as .npy.

Regenerating All Voice Samples

python scripts/extract_embeddings.py --force  # Re-extract from WAV refs

Rainbow Passage

All voice references are generated using the Rainbow Passage — a phonetically balanced paragraph covering all English phonemes. This ensures the speaker embedding captures the full range of vocal characteristics.

When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow.
The rainbow is a division of white light into many beautiful colors.
These take the shape of a long round arch, with its path high above,
and its two ends apparently beyond the horizon.
There is, according to legend, a boiling pot of gold at one end.
People look, but no one ever finds it.
When a man looks for something beyond his reach,
his friends say he is looking for the pot of gold at the end of the rainbow.

Server API

POST /v1/tts

Parameter	Type	Default	Description
text	str	required	Text to synthesize
sample_rate	int	24000	Audio sample rate
voice_description	str	None	VoiceDesign instruct text
ref_audio	File	None	Voice reference WAV (fallback mode)
ref_text	str	None	Reference text for ICL mode
speaker_embedding	str	None	JSON-encoded float list (2048-dim)
temperature	float	0.9	Sampling temperature (lower = more consistent)
do_sample	bool	True	Enable sampling
seed	int	None	Random seed for reproducibility

POST /load

Load a model into memory.

{"model_type": "base"}  // or "voicedesign"

POST /unload

Unload a model to free memory.

{"model_type": "all"}  // or "base" or "voicedesign"

GET /health

Returns server status and loaded models.

Consistency Guarantees

Per-Utterance Consistency

Every utterance of a character passes the same cached vmatrix (prompt_slug → the .pt with ref_code + embedding) → identical voice identity. Variation comes only from:

• Different text (codec-token sequence)
• Temperature (controlled per-chapter)

Cross-Chapter Consistency

Same cached vmatrix used across all chapters → same voice identity. The narrator always sounds like the narrator. Kainda always sounds like Kainda — the voice can never silently change between chapters.

Capture Discipline

A voice is only the SAME voice while its cached .pt matches its approved reference WAV. When a *_ref.wav is regenerated/approved, the .pt MUST be rebuilt (the Approve flow does this; build_voice_prompt(..., force=True) forces it) — otherwise chapter generation silently uses the stale old voice. This is why "Kainda still had the old voice": the .pt was built Aug 27, before the Aug 31 reference replacements.

Chapter Generation Flow

1. Preflight: Verify TTS server is running, Base model loaded
2. Validate: Check which utterances need (re)generation
3. Capture vmatrix: For each distinct speaker with a missing/stale .pt, build_voice_prompt(vref, x_vector_only=False) → cached voice_prompts/{slug}.pt
4. Generate: For each utterance:
   • Resolve speaker → prompt_slug (the cached vmatrix)
   • Send to server with text + temperature + seed + prompt_slug
   • Save WAV to .umbra/chapter_audio/ch{N}/utt{M}.wav
5. Stitch: Concatenate utterance WAVs into chapter WAV
6. Export: Convert chapter WAV to MP3 via ffmpeg
7. Verify: Run quality checks

Hardware Constraints

• Mac 16 GB MPS: One model resident at a time (Base OR VoiceDesign)
• Base model: ~3.5 GB VRAM, generates ~10s/utterance
• VoiceDesign model: ~3.5 GB VRAM, generates ~10s/utterance
• ICL mode: ~11 minutes/utterance on MPS (not viable for production)
• x-vector mode: ~10s/utterance on MPS (production mode)

Model Paths

Model	Path	Purpose
Base	models/Qwen3-TTS-12Hz-1.7B-Base	Voice cloning + embedding extraction
VoiceDesign	models/Qwen3-TTS-12Hz-1.7B-VoiceDesign	New voice creation from instruct text
Tokenizer	models/Qwen3-TTS-Tokenizer-12Hz	Shared tokenizer

Configuration

speakers.json

{
  "speakers": [
    {
      "primary_name": "Narrator",
      "gender": "male",
      "accent": "american",
      "voice_status": "approved",
      "vocal_traits": {
        "pitch": "low",
        "energy": "measured",
        "timbre": "warm"
      }
    }
  ]
}

voice_assignments.json

Maps character names to Edge-TTS voice identifiers (used for WAV ref generation).

{
  "Narrator": "en-US-GuyNeural:-10",
  "Nanny": "en-US-AvaNeural",
  "Kainda Walters Foster Murphy": "en-US-JennyNeural"
}

Troubleshooting

Voice sounds wrong / "it's still the old voice"

1. Check the voice is approved and has a vmatrix: ls .umbra/voice_prompts/{slug}.pt (slug = ref filename minus _ref.wav)
2. If the .pt is missing or older than the ref WAV, rebuild it: approve again via the web UI, or force: build_voice_prompt(ref, ref_text=Rainbow, x_vector_only=False, force=True)
3. Verify the approved reference WAV (.umbra/voices/{slug}_ref.wav — listen to it)
4. Only then regenerate the character's chapter utterances

TTS server not responding

curl http://localhost:8881/health
# If down: restart with qwen_tts_server.py

Memory pressure

curl -X POST http://localhost:8881/unload -d '{"model_type": "all"}'

Inconsistent voices across utterances

Solved by the vmatrix: every utterance passes prompt_slug → the same cached .pt (ref_code + embedding). If a character's voice still changes, the .pt is being rebuilt per utterance or is stale vs the approved ref — force-rebuild it once and ensure generation uses x_vector_only=False (check logs for cache hits of the slug).