UmbraAudio/Umbra_Voice_generation.md

UmbraAudio Voice Generation Specification

Overview

UmbraAudio generates multi-voice audiobooks from English manuscripts. Each character gets a unique, consistent voice. The voice identity is stored as a speaker embedding — a 2048-dimensional float32 vector extracted from Qwen3-TTS's internal speaker encoder. This embedding is the single source of truth for a character's voice.

Architecture

Voice Identity Pipeline

Edge-TTS WAV ref  ──or──  VoiceDesign instruct text
        │                          │
        ▼                          ▼
  Base model               VoiceDesign model
  extract_speaker_          generate_voice_design()
  embedding()                      │
        │                          ▼
        ▼                    Base model
  Speaker embedding        extract_speaker_embedding()
  (2048-dim float32)             │
        │                          ▼
        ▼                    Speaker embedding
  Save as .npy                   │
  (8KB per speaker)              ▼
                           Save as .npy

Key Insight

The old approach (WAV ref → extract x-vector at generation time) produced nearly identical voices (MFCC cosine similarity 0.99+) because x-vector extraction is lossy. The new approach stores the exact embedding that the model uses, bypassing extraction entirely. This gives:

• Perfect determinism: same embedding + same seed = identical waveform
• Distinct voices: narrator 275 Hz, nanny 176 Hz, kainda 227 Hz (measured F0)
• Tiny storage: 8KB per speaker vs megabytes of WAV refs
• No drift: embedding is fixed once saved, never re-extracted

File Structure

.umbra/
├── voice_embeddings/           # Speaker identity cache
│   ├── narrator.npy            # 2048-dim float32 (8KB)
│   ├── nanny.npy
│   ├── kainda_walters_foster_murphy.npy
│   ├── deal_hunter.npy
│   └── ...                     # 94 speakers total
├── voices/                     # Legacy WAV refs (kept for fallback)
│   ├── narrator_ref.wav
│   ├── nanny_ref.wav
│   └── ...
├── speakers.json               # Character metadata
├── voice_assignments.json      # Edge-TTS voice mapping
├── voice_profiles.json         # LLM-generated voice profiles
└── project.db                  # Utterances, chapters, attributions

Generation Modes

Mode 1: Cached Embedding (Preferred)

Fastest and most consistent. Used when a .npy embedding exists for the speaker.

# Client loads cached embedding
emb = np.load("voice_embeddings/narrator.npy")  # 2048-dim float32

# Sends to server as JSON (8KB payload)
POST /v1/tts
  speaker_embedding: "[0.123, -0.456, ...]"  # JSON float list
  text: "The captain nodded."
  temperature: 0.5
  seed: 42

# Server passes embedding directly to model
model.generate_voice_clone(
    text=text,
    voice_clone_prompt={
        "ref_code": [None],
        "ref_spk_embedding": [spk_tensor],
        "x_vector_only_mode": [True],
        "icl_mode": [False],
    },
    language="english",
    temperature=0.5,
)

Determinism: Same embedding + same seed = identical waveform (verified: mean abs diff = 0.000000).

Mode 2: WAV Reference (Fallback)

Used when no cached embedding exists. Extracts x-vector at generation time.

POST /v1/tts
  ref_audio: <WAV file upload>  # 50-200KB
  text: "The captain nodded."
  temperature: 0.5

Mode 3: VoiceDesign (New Voice Creation)

Used to create new character voices from text descriptions.

POST /v1/tts
  voice_description: "A deep male voice with Australian accent"
  text: "The captain nodded."
  temperature: 0.5

Voice Key Generation

From Existing WAV Refs

python scripts/extract_embeddings.py --project DealHunterShip_4.0e_whole-to-audio

Loads Base model once, iterates all *_ref.wav files, extracts embeddings, saves as .npy. Takes ~6 seconds for 94 speakers.

From VoiceDesign Instruct Text

python scripts/extract_embeddings.py \
    --design "A young female voice with slight accent, speaking softly" \
    --slug new_character \
    --seed 42

Generates audio via VoiceDesign → extracts embedding via Base model → saves as .npy.

Regenerating All Voice Samples

python scripts/extract_embeddings.py --force  # Re-extract from WAV refs

Rainbow Passage

All voice references are generated using the Rainbow Passage — a phonetically balanced paragraph covering all English phonemes. This ensures the speaker embedding captures the full range of vocal characteristics.

When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow.
The rainbow is a division of white light into many beautiful colors.
These take the shape of a long round arch, with its path high above,
and its two ends apparently beyond the horizon.
There is, according to legend, a boiling pot of gold at one end.
People look, but no one ever finds it.
When a man looks for something beyond his reach,
his friends say he is looking for the pot of gold at the end of the rainbow.

Server API

POST /v1/tts

Parameter	Type	Default	Description
text	str	required	Text to synthesize
sample_rate	int	24000	Audio sample rate
voice_description	str	None	VoiceDesign instruct text
ref_audio	File	None	Voice reference WAV (fallback mode)
ref_text	str	None	Reference text for ICL mode
speaker_embedding	str	None	JSON-encoded float list (2048-dim)
temperature	float	0.9	Sampling temperature (lower = more consistent)
do_sample	bool	True	Enable sampling
seed	int	None	Random seed for reproducibility

POST /load

Load a model into memory.

{"model_type": "base"}  // or "voicedesign"

POST /unload

Unload a model to free memory.

{"model_type": "all"}  // or "base" or "voicedesign"

GET /health

Returns server status and loaded models.

Consistency Guarantees

Per-Utterance Consistency

Each utterance uses the same embedding + same seed → identical voice identity. Variation comes only from:

• Different text (phoneme sequence)
• Temperature (controlled per-chapter)

Cross-Chapter Consistency

Same speaker embedding used across all chapters → same voice identity. The narrator always sounds like the narrator. Kainda always sounds like Kainda.

Determinism

With seed=42 and the same embedding:

• Run 1 and Run 2 produce identical waveforms (mean abs diff = 0.000000)
• Different seeds produce different phrasings but same voice identity

Chapter Generation Flow

1. Preflight: Verify TTS server is running, Base model loaded
2. Validate: Check which utterances need (re)generation
3. Generate: For each utterance:
   • Load speaker embedding from .npy cache
   • Send to server with text + temperature + seed
   • Save WAV to .umbra/chapter_audio/ch{N}/utt{M}.wav
4. Stitch: Concatenate utterance WAVs into chapter WAV
5. Export: Convert chapter WAV to MP3 via ffmpeg
6. Verify: Run quality checks

Hardware Constraints

• Mac 16 GB MPS: One model resident at a time (Base OR VoiceDesign)
• Base model: ~3.5 GB VRAM, generates ~10s/utterance
• VoiceDesign model: ~3.5 GB VRAM, generates ~10s/utterance
• ICL mode: ~11 minutes/utterance on MPS (not viable for production)
• x-vector mode: ~10s/utterance on MPS (production mode)

Model Paths

Model	Path	Purpose
Base	models/Qwen3-TTS-12Hz-1.7B-Base	Voice cloning + embedding extraction
VoiceDesign	models/Qwen3-TTS-12Hz-1.7B-VoiceDesign	New voice creation from instruct text
Tokenizer	models/Qwen3-TTS-Tokenizer-12Hz	Shared tokenizer

Configuration

speakers.json

{
  "speakers": [
    {
      "primary_name": "Narrator",
      "gender": "male",
      "accent": "american",
      "voice_status": "approved",
      "vocal_traits": {
        "pitch": "low",
        "energy": "measured",
        "timbre": "warm"
      }
    }
  ]
}

voice_assignments.json

Maps character names to Edge-TTS voice identifiers (used for WAV ref generation).

{
  "Narrator": "en-US-GuyNeural:-10",
  "Nanny": "en-US-AvaNeural",
  "Kainda Walters Foster Murphy": "en-US-JennyNeural"
}

Troubleshooting

Voice sounds wrong

1. Check if embedding exists: ls .umbra/voice_embeddings/{slug}.npy
2. Regenerate embedding: python scripts/extract_embeddings.py --speaker {name} --force
3. Verify with Rainbow Passage sample

TTS server not responding

curl http://localhost:8881/health
# If down: restart with qwen_tts_server.py

Memory pressure

curl -X POST http://localhost:8881/unload -d '{"model_type": "all"}'

Inconsistent voices across utterances

This was the old x-vector extraction problem. Now solved by cached embeddings. If still occurring, ensure the caller is using cached embeddings (check logs for "Using cached embedding for ...").