diff --git a/Umbra_Voice_generation.md b/Umbra_Voice_generation.md index 4a51f09..ee89f23 100644 --- a/Umbra_Voice_generation.md +++ b/Umbra_Voice_generation.md @@ -3,55 +3,75 @@ ## Overview UmbraAudio generates multi-voice audiobooks from English manuscripts. Each character -gets a unique, consistent voice. The voice identity is stored as a **speaker embedding** — -a 2048-dimensional float32 vector extracted from Qwen3-TTS's internal speaker encoder. -This embedding is the single source of truth for a character's voice. +gets a unique, consistent voice. The voice identity is stored as a **vmatrix** — the +set of numbers inside Qwen3-TTS that *is* the voice: + +1. the **ICL "8k matrix"** (`ref_code`) — a `(T, 16)` tensor of codec tokens rebuilt by + re-speaking the reference WAV through the model's codec encoder, AND +2. the **speaker embedding** (`ref_spk_embedding`) — a 2048-dim float32/bfloat16 vector + extracted from Qwen3-TTS's speaker encoder. + +The vmatrix is captured ONCE per approved voice and cached server-side as a `.pt` prompt +(in `voice_prompts/{slug}.pt`). Every utterance reuses those exact cached numbers, so the +voice is bit-identical in identity across every utterance and chapter. **An approved voice +is therefore two artifacts: a WAV to listen to + the captured vmatrix for consistency.** + +> Historical note: an earlier pipeline stored ONLY the 2048-dim speaker embedding +> (`x_vector_only=True`) as the single source of truth. That was the "nearly identical / +> drift-prone" coarse capture. The canonical capture is the **full vmatrix** (ref_code + +> embedding, `x_vector_only=False`). ## Architecture ### Voice Identity Pipeline ``` -Edge-TTS WAV ref ──or── VoiceDesign instruct text - │ │ - ▼ ▼ - Base model VoiceDesign model - extract_speaker_ generate_voice_design() - embedding() │ - │ ▼ - ▼ Base model - Speaker embedding extract_speaker_embedding() - (2048-dim float32) │ - │ ▼ - ▼ Speaker embedding - Save as .npy │ - (8KB per speaker) ▼ - Save as .npy +Approved reference WAV (per speaker: the author-approved "*_ref.wav") + │ + ▼ +POST /v1/voice-prompt (x_vector_only=0) + │ model re-speaks the reference: codec encoder + speaker encoder + ▼ +vmatrix { ref_code: (T,16) int64 ICL "8k matrix" + ref_spk_embedding: (2048,) speaker x-vector + x_vector_only_mode: False, icl_mode: True } + │ + ▼ +cached as .pt under voice_prompts/{slug}.pt (PROMPT_DIR) + │ + ▼ +every utterance → generate(prompt_slug={slug}) reuses the SAME cached vmatrix ``` ### Key Insight -The old approach (WAV ref → extract x-vector at generation time) produced **nearly -identical voices** (MFCC cosine similarity 0.99+) because x-vector extraction is lossy. -The new approach stores the **exact embedding** that the model uses, bypassing extraction -entirely. This gives: +The previous approach (WAV ref → extract a bare x-vector at generation time, or use the +embedding-only `x_vector_only=True` mode) produced **drift-prone / coarse** voices, +because neither kept the exact numbers the model conditions on. The **vmatrix** captures +the exact ICL codec-token matrix (`ref_code`) **plus** the speaker embedding the model +uses, caches it once, and reuses it for every utterance. This gives: -- **Perfect determinism**: same embedding + same seed = identical waveform -- **Distinct voices**: narrator 275 Hz, nanny 176 Hz, kainda 227 Hz (measured F0) -- **Tiny storage**: 8KB per speaker vs megabytes of WAV refs -- **No drift**: embedding is fixed once saved, never re-extracted +- **Perfect consistency**: every utterance of a character uses the exact same cached + numbers → the voice can never drift or "change" between chapters +- **Faithful identity**: regeneration re-speaks the reference, so the voice is the + reference voice, not a lossy summary of it +- **One-time capture**: the `.pt` is built once per approved voice (on Approve) and at + the start of each generation run if missing +- **Distinct voices**: narrator / nanny / kainda measured F0s remain distinct ## File Structure ``` .umbra/ -├── voice_embeddings/ # Speaker identity cache +├── voice_prompts/ # VMXATRIX cache (.pt): the canonical voice identity +│ ├── kainda_walters_foster_murphy.pt # ref_code (T,16) + ref_spk_embedding (2048,) +│ ├── nanny.pt +│ ├── narrator.pt +│ └── ... # one .pt per approved voice +├── voice_embeddings/ # LEGACY speaker-embedding cache (x_vector_only mode) │ ├── narrator.npy # 2048-dim float32 (8KB) -│ ├── nanny.npy -│ ├── kainda_walters_foster_murphy.npy -│ ├── deal_hunter.npy -│ └── ... # 94 speakers total -├── voices/ # Legacy WAV refs (kept for fallback) +│ └── ... # kept only for fallback/experiments +├── voices/ # Approved reference WAVs (the "listen to" artifact) │ ├── narrator_ref.wav │ ├── nanny_ref.wav │ └── ... @@ -63,40 +83,49 @@ entirely. This gives: ## Generation Modes -### Mode 1: Cached Embedding (Preferred) +# Mode 1 is the CANONICAL mode (since the vmatrix fix): +# client captures the FULL vmatrix once, then generates via prompt_slug. -Fastest and most consistent. Used when a `.npy` embedding exists for the speaker. +### Mode 1: Cached Vmatrix via prompt_slug (Preferred — Canonical) + +Full voice identity (ICL "8k matrix" ref_code + speaker embedding) captured once and +cached as `.pt`. Every utterance reuses the same cached numbers. ```python -# Client loads cached embedding -emb = np.load("voice_embeddings/narrator.npy") # 2048-dim float32 - -# Sends to server as JSON (8KB payload) -POST /v1/tts - speaker_embedding: "[0.123, -0.456, ...]" # JSON float list - text: "The captain nodded." - temperature: 0.5 - seed: 42 - -# Server passes embedding directly to model -model.generate_voice_clone( - text=text, - voice_clone_prompt={ - "ref_code": [None], - "ref_spk_embedding": [spk_tensor], - "x_vector_only_mode": [True], - "icl_mode": [False], - }, - language="english", - temperature=0.5, +# Client captures the vmatrix once per approved voice (equivalent to Approve): +prompt_slug = engine.build_voice_prompt( + ref_path, # the approved *_ref.wav + ref_text=VOICE_REFERENCE_TEXT, # Rainbow Passage + x_vector_only=False, # FULL vmatrix (ref_code + embedding) + force=False, # True only if the ref wav changed ) + +# Server: POST /v1/voice-prompt → cached voice_prompts/{prompt_slug}.pt +# containing { ref_code: (T,16), ref_spk_embedding: (2048,), +# x_vector_only_mode: False, icl_mode: True } + +# Every utterance then references the same identity: +engine.generate(text, output_path, prompt_slug=prompt_slug) ``` -**Determinism**: Same embedding + same seed = **identical waveform** (verified: mean abs diff = 0.000000). +**Consistency**: every utterance of a character uses the exact same cached vmatrix → the +voice is identical across all chapters and can never silently change. -### Mode 2: WAV Reference (Fallback) +### Mode 2: Embedding-only (x_vector_only=1) -Used when no cached embedding exists. Extracts x-vector at generation time. +Captures ONLY `ref_spk_embedding` (`ref_code=None`). Coarse, drift-prone; this was the +pre-fix default and caused the "voice changed / inconsistent" problem. Keep for +experiments or machines without ICL VRAM headroom. + +```python +POST /v1/voice-prompt x_vector_only=1 +# → cached voice_prompts/{slug}.pt with ref_code=None, icl_mode=False +``` + +### Mode 3: WAV Reference (fallback) + +Used only when no cached prompt exists and no slug is passed — server extracts the +x-vector at generation time. ```python POST /v1/tts @@ -105,9 +134,10 @@ POST /v1/tts temperature: 0.5 ``` -### Mode 3: VoiceDesign (New Voice Creation) +### Mode 4: VoiceDesign (New Voice Creation) -Used to create new character voices from text descriptions. +Used to create new character voices from text descriptions (also used to generate the +initial `*_ref.wav` for each character). ```python POST /v1/tts @@ -201,33 +231,38 @@ Returns server status and loaded models. ### Per-Utterance Consistency -Each utterance uses the same embedding + same seed → identical voice identity. -Variation comes only from: -- Different text (phoneme sequence) +Every utterance of a character passes the **same cached vmatrix** (`prompt_slug` → the +`.pt` with ref_code + embedding) → identical voice identity. Variation comes only from: +- Different text (codec-token sequence) - Temperature (controlled per-chapter) ### Cross-Chapter Consistency -Same speaker embedding used across all chapters → same voice identity. -The narrator always sounds like the narrator. Kainda always sounds like Kainda. +Same cached vmatrix used across all chapters → same voice identity. +The narrator always sounds like the narrator. Kainda always sounds like Kainda — +the voice can never silently change between chapters. -### Determinism +### Capture Discipline -With `seed=42` and the same embedding: -- Run 1 and Run 2 produce **identical waveforms** (mean abs diff = 0.000000) -- Different seeds produce different phrasings but same voice identity +A voice is only the SAME voice while its cached `.pt` matches its approved reference WAV. +When a `*_ref.wav` is regenerated/approved, the `.pt` MUST be rebuilt (the Approve flow +does this; `build_voice_prompt(..., force=True)` forces it) — otherwise chapter +generation silently uses the stale old voice. This is why "Kainda still had the old +voice": the `.pt` was built Aug 27, before the Aug 31 reference replacements. ## Chapter Generation Flow 1. **Preflight**: Verify TTS server is running, Base model loaded 2. **Validate**: Check which utterances need (re)generation -3. **Generate**: For each utterance: - - Load speaker embedding from `.npy` cache - - Send to server with text + temperature + seed +3. **Capture vmatrix**: For each distinct speaker with a missing/stale `.pt`, + `build_voice_prompt(vref, x_vector_only=False)` → cached `voice_prompts/{slug}.pt` +4. **Generate**: For each utterance: + - Resolve speaker → prompt_slug (the cached vmatrix) + - Send to server with text + temperature + seed + prompt_slug - Save WAV to `.umbra/chapter_audio/ch{N}/utt{M}.wav` -4. **Stitch**: Concatenate utterance WAVs into chapter WAV -5. **Export**: Convert chapter WAV to MP3 via ffmpeg -6. **Verify**: Run quality checks +5. **Stitch**: Concatenate utterance WAVs into chapter WAV +6. **Export**: Convert chapter WAV to MP3 via ffmpeg +7. **Verify**: Run quality checks ## Hardware Constraints @@ -281,11 +316,14 @@ Maps character names to Edge-TTS voice identifiers (used for WAV ref generation) ## Troubleshooting -### Voice sounds wrong +### Voice sounds wrong / "it's still the old voice" -1. Check if embedding exists: `ls .umbra/voice_embeddings/{slug}.npy` -2. Regenerate embedding: `python scripts/extract_embeddings.py --speaker {name} --force` -3. Verify with Rainbow Passage sample +1. Check the voice is approved and has a vmatrix: + `ls .umbra/voice_prompts/{slug}.pt` (slug = ref filename minus `_ref.wav`) +2. If the `.pt` is missing or older than the ref WAV, rebuild it: + approve again via the web UI, or force: `build_voice_prompt(ref, ref_text=Rainbow, x_vector_only=False, force=True)` +3. Verify the approved reference WAV (`.umbra/voices/{slug}_ref.wav` — listen to it) +4. Only then regenerate the character's chapter utterances ### TTS server not responding @@ -302,6 +340,7 @@ curl -X POST http://localhost:8881/unload -d '{"model_type": "all"}' ### Inconsistent voices across utterances -This was the old x-vector extraction problem. Now solved by cached embeddings. -If still occurring, ensure the caller is using cached embeddings (check logs for -"Using cached embedding for ..."). +Solved by the vmatrix: every utterance passes `prompt_slug` → the same cached `.pt` +(ref_code + embedding). If a character's voice still changes, the `.pt` is being +rebuilt per utterance or is stale vs the approved ref — force-rebuild it once and +ensure generation uses `x_vector_only=False` (check logs for cache hits of the slug). diff --git a/scripts/qwen_tts_server.py b/scripts/qwen_tts_server.py index 531738f..560199a 100644 --- a/scripts/qwen_tts_server.py +++ b/scripts/qwen_tts_server.py @@ -257,22 +257,23 @@ async def build_voice_prompt( x_vector_only: int = Form(0), force: int = Form(0), ): - """Build and cache a speaker's digital voice (a set of numbers) once. + """Build and cache a speaker's vmatrix (digital voice) once. - Two capture modes (pick per pipeline; a speaker's cached prompt is used as-is - on every subsequent utterance): + The vmatrix is the set of numbers inside Qwen3-TTS that IS the voice. Two capture + modes (a speaker's cached prompt is reused on every subsequent utterance via + prompt_slug): - * x_vector_only=0 (default, ICL/"8k matrix"): extracts the FULL codec-token - matrix (ref_code) + speaker embedding. Together those let the generator - re-create the exact reference recording -- and every utterance therefore - re-speaks the reference first, then trims it (slow + leaky). - * x_vector_only=1: captures ONLY the speaker embedding (the digital voice - variables). ref_code is None. Generation then feeds just those numbers + - text into the speech generator -- no reference re-hash, no clone re-run, - fast and output is exactly the text. + * x_vector_only=0 (DEFAULT, canonical, "full vmatrix"): extracts the FULL + ICL codec-token matrix (ref_code, the "8k matrix") + speaker embedding. + Generation conditions on the exact reference identity, so every utterance + is faithful AND identical in identity across the whole pipeline. + * x_vector_only=1: captures ONLY the coarse speaker embedding (ref_code=None). + Legacy/experimental - caused the historical "voice changed between + chapters" inconsistency. Not used for production consistency. Idempotent: if `.pt` is already cached it returns without rebuilding, - unless `force=1` is passed (use force to switch an existing prompt mode). + unless `force=1` is passed (use force to switch an existing prompt mode, e.g. + after an approved reference WAV is replaced). """ path = _prompt_path(slug) if path.exists() and not force: diff --git a/specs/03-voice-design.md b/specs/03-voice-design.md index 70d7354..594c93c 100644 --- a/specs/03-voice-design.md +++ b/specs/03-voice-design.md @@ -248,6 +248,32 @@ This table is what later lets a Pass 3 "Always" promotion (`04-mapping.md`, "Spe Once approved, this is the "normal" voice referenced by everything downstream: Pass 3's punctuation-driven prosody hints modulate *around* it, and human-reviewed per-line descriptor overrides modulate around it too. Neither layer ever resets the baseline itself without going back through this same approval loop. +### Approved Voice = Reference WAV + Captured Vmatrix + +An approval now records **two artifacts** for the character (this is the requirement: a wav to *listen* to + the numbers inside the TTS that *are* the voice): + +| Artifact | Path | Purpose | +|---|---|---| +| Reference WAV | `.umbra/voices/{slug}_ref.wav` | Listenable sample of the approved voice | +| **Vmatrix** (.pt) | `.umbra/voice_prompts/{slug}.pt` | The captured voice identity used for consistency | + +The **vmatrix** is what Qwen3-TTS actually conditions on to reproduce a voice: + +- `ref_code` — the ICL **"8k matrix"**: a `(T, 16)` tensor of codec tokens produced by re-speaking the reference WAV through the model's codec encoder. +- `ref_spk_embedding` — the 2048-dim extracted speaker x-vector. + +It is captured once (server-side `POST /v1/voice-prompt`, `x_vector_only=0`) and cached +under `voice_prompts/{slug}.pt`. `POST /api/speakers/voice-status` **approve** calls +`build_voice_prompt(...)` and records `vmatrix_path` in the voice registry next to +`voice_ref_path`. Every subsequent utterance passes `prompt_slug={slug}` so it reuses +the exact same numbers — a character's voice can never drift or "change" between +chapters. If the reference WAV is ever replaced, the `.pt` must be rebuilt (approve +again, or `force=True`) or generation will silently use the stale voice. + +Note on modes: `x_vector_only=True` (embedding-only) is the legacy-coarse capture and +must NOT be used for production consistency. The full vmatrix (`x_vector_only=False`) is +the canonical mode and the engine default. + ### Skip Logic Voice references are not regenerated if: diff --git a/specs/05-generation.md b/specs/05-generation.md index ce9a18e..9513bcd 100644 --- a/specs/05-generation.md +++ b/specs/05-generation.md @@ -12,7 +12,8 @@ Pass 3 generates audio for every utterance in every chapter using **Qwen3-TTS** class QwenTTSEngine: def __init__(self, device="cuda", sample_rate=24000, qwen_tts_url: str) -> None def load_remote_model(self) -> None - def generate(self, text: str, output_path: Path, voice_ref: Path | None = None, instruct: str = "") -> Path + def generate(self, text: str, output_path: Path, voice_ref: Path | None = None, instruct: str = "", prompt_slug: str | None = None) -> Path + def build_voice_prompt(self, voice_ref: Path, ref_text: str | None = None, slug: str | None = None, x_vector_only: bool = False, force: bool = False) -> str def unload_all(self) -> None ``` @@ -20,8 +21,28 @@ class QwenTTSEngine: - Remote/on-prem synthesis via `qwen_tts_url` (LM Studio / Qwen server) - Voice cloning from a reference WAV per character - Optional `instruct` text for prosody/emotion direction (e.g. rising question intonation) +- **Vmatrix capture** (`build_voice_prompt`): computes the voice's "8k matrix" (ICL `ref_code` codec-token matrix) + speaker embedding, caches it server-side as `voice_prompts/{slug}.pt`, returns the `prompt_slug` to pass to `generate()`. +- **`prompt_slug` generation** (the consistency path): every utterance of a character passes the same slug, so the exact same cached vmatrix is reused — voice identity can never drift between utterances or chapters. - WAV output at 24kHz sample rate +> **`x_vector_only` default is `False` (full vmatrix).** The full capture (ref_code + embedding) is the canonical mode; the embed-only mode (`True`) is legacy/experimental and caused the historical "voice changed between chapters" problem. + +### Vmatrix Capture in the Generation Flow + +Because `build_voice_prompt` defaults to `/v1/voice-prompt` server semantics, the bulk +generator captures each distinct speaker's vmatrix once per run (before any utterance is +generated) and reuses the slug for all of that speaker's utterances: + +``` +for each distinct speaker with a missing .pt: + prompt_slug = build_voice_prompt(vref, ref_text=VOICE_REFERENCE_TEXT, x_vector_only=False) + # server writes/updates voice_prompts/{slug}.pt = {ref_code (T,16), ref_spk_embedding (2048,), icl_mode: True} +``` + +The `.pt` cache is the same filesystem location the server writes to (`PROMPT_DIR`, +default `.umbra/voice_prompts`), so it persists across runs and is shipped to the GPU +host via `tts_server/sync_voice_data.sh`. + ### Edge-TTS (`voices/edge_tts.py`) — Test Samples & Fallback ```python @@ -58,6 +79,11 @@ def resolve_reference( ``` For each chapter: + │ + ├── 0. Capture vmatrix for each distinct speaker (once per run) + │ prompt_slug = engine.build_voice_prompt(vref, ref_text=VOICE_REFERENCE_TEXT, + │ x_vector_only=False) # full "8k matrix" + embedding + │ # → voice_prompts/{slug}.pt (cached; missing only when the ref changed) │ ├── 1. Load utterances from DB │ SELECT id, speaker_name, text, utterance_index @@ -72,11 +98,12 @@ For each chapter: ├── Skip if file exists │ if output_path.exists(): continue │ - ├── Resolve voice reference + ├── Resolve voice reference → prompt_slug │ ref_path = resolve_reference(speaker, voice_refs, alias_map, narrator_ref) + │ slug = engine.slug_for_ref(ref_path) │ - ├── Generate audio (Qwen3-TTS) - │ engine.generate(text, output_path, voice_ref=ref_path) + ├── Generate audio (Qwen3-TTS) — identical cached vmatrix for the speaker + │ engine.generate(text, output_path, prompt_slug=slug) │ # no ref → engine.generate(text, output_path, instruct=neutral) │ ├── Verify output file exists (else count as failed) @@ -85,6 +112,10 @@ For each chapter: Log error, continue to next utterance ``` +Note the two-layer design: step 0 captures the vmatrix (a set of numbers) **once** per +speaker; step 3 reuses that identical capture for every utterance via `prompt_slug`. +This is what guarantees a character's voice never drifts across a chapter or the book. + **Engine mode:** the engine uses remote synthesis when `cfg.tts.qwen_tts_url` is set (the default deployment). The remote server holds the VoiceDesign and Base models loaded and answers `/v1/tts`; the local `QwenTTSEngine` only loads models itself when no URL is configured. ### Output Structure @@ -178,7 +209,7 @@ Success rate is reported per run as `generated / failed`; failed utterances can 1. **Qwen3-TTS over Edge-TTS**: cloning from per-character reference samples gives stable, character-accurate voices; Edge-TTS is retained only for test samples and the `_generate_edge_fallback` path in the voice designer 2. **Remote synthesis**: the Qwen server (`qwen_tts_url`) hosts the models, so bulk generation doesn't fight the LLM (LM Studio) for local GPU/VRAM -3. **Voice references over voice strings**: identity comes from `*_ref.wav` cloning instead of `voice_assignments.json` voice strings with pitch shifts +3. **Vmatrix identity from references**: identity comes from the **captured vmatrix** of each `*_ref.wav` (ICL codec-token "8k matrix" + speaker embedding, cached as `voice_prompts/{slug}.pt`), not from `voice_assignments.json` voice strings with pitch shifts, and not from a coarse embedding-only capture. Every utterance reuses the same `.pt` via `prompt_slug`. 4. **Sequential generation**: avoids overloading the remote server and simplifies resumability 5. **File-based caching**: skip already-generated files; regeneration after edits is file-targeted 6. **Error resilience**: individual utterance failures never abort a chapter run; a final summary counts successes and failures diff --git a/tests/test_library.py b/tests/test_library.py new file mode 100644 index 0000000..8f982ac --- /dev/null +++ b/tests/test_library.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +from pathlib import Path + +from umbraaudio import library + + +def _speaker(name: str = "Distorted Voice", **overrides) -> dict: + sp = { + "primary_name": name, + "aliases": [], + "gender": "Female", + "accent": "canadian", + "vocal_traits": ["raspy"], + "distortion_effects": ["hard_clip"], + "is_narrator": False, + "is_group": False, + "description": "test speaker", + } + sp.update(overrides) + return sp + + +class TestAddSpeakers: + def test_adds_and_copies_voice_ref(self, temp_dir: Path): + db = temp_dir / "voice_registry.db" + voices = temp_dir / "libvoices" + voices.mkdir(parents=True) + + src = temp_dir / "src_distorted_voice_ref.wav" + src.write_bytes(b"RIFFDATAFORVOICE") + + res = library.add_speakers( + [_speaker()], + source_project="TestProject", + voice_refs={"Distorted Voice": src}, + db_path=db, + voices_dir=voices, + ) + assert res["added"] == ["Distorted Voice"] + assert (voices / "distorted_voice_ref.wav").exists() + assert (voices / "distorted_voice_ref.wav").read_bytes() == b"RIFFDATAFORVOICE" + + rows = library.list_speakers(db_path=db) + assert len(rows) == 1 + assert rows[0]["primary_name"] == "Distorted Voice" + assert "distorted_voice_ref.wav" in rows[0]["voice_ref_path"] + + def test_idempotent_duplicate(self, temp_dir: Path): + db = temp_dir / "voice_registry.db" + voices = temp_dir / "lv" + voices.mkdir(parents=True) + src = temp_dir / "narrator_ref.wav" + src.write_bytes(b"REF") + + library.add_speakers( + [_speaker("Narrator", is_narrator=True)], + "P", {"Narrator": src}, db_path=db, voices_dir=voices, + ) + res = library.add_speakers( + [_speaker("Narrator", is_narrator=True)], + "P", {"Narrator": src}, db_path=db, voices_dir=voices, + ) + assert res["already"] == ["Narrator"] + assert len(library.list_speakers(db_path=db)) == 1 + + def test_skips_speaker_without_ref(self, temp_dir: Path): + db = temp_dir / "voice_registry.db" + res = library.add_speakers( + [_speaker()], "P", {}, db_path=db, voices_dir=temp_dir, + ) + assert res["added"] == ["Distorted Voice"] + assert library.list_speakers(db_path=db)[0]["voice_ref_path"] is None + + +class TestAddNames: + def test_adds_records(self, temp_dir: Path): + db = temp_dir / "voice_registry.db" + res = library.add_names( + [{"name": "Kainda", "phonetic": "KAYN-dah", "source": "user", + "common": False, "confirmed": True}], + source_project="P", + db_path=db, + ) + assert res["added"] == ["Kainda"] + rows = library.list_names(db_path=db) + assert rows[0]["name"] == "Kainda" + assert rows[0]["phonetic"] == "KAYN-dah" + assert rows[0]["confirmed"] == 1 + + def test_idempotent_duplicate(self, temp_dir: Path): + db = temp_dir / "voice_registry.db" + rec = {"name": "Kainda", "phonetic": "KAYN-dah"} + library.add_names([rec], "P", db_path=db) + res = library.add_names([rec], "P", db_path=db) + assert res["already"] == ["Kainda"] + assert len(library.list_names(db_path=db)) == 1 + + +class TestDelete: + def test_delete_speaker(self, temp_dir: Path): + db = temp_dir / "voice_registry.db" + library.add_speakers([_speaker()], "P", {}, db_path=db, voices_dir=temp_dir) + sid = library.list_speakers(db_path=db)[0]["id"] + assert library.delete_speaker(sid, db_path=db) is True + assert library.list_speakers(db_path=db) == [] + + def test_delete_name(self, temp_dir: Path): + db = temp_dir / "voice_registry.db" + library.add_names([{"name": "Kainda"}], "P", db_path=db) + nid = library.list_names(db_path=db)[0]["id"] + assert library.delete_name(nid, db_path=db) is True + assert library.list_names(db_path=db) == [] diff --git a/tests/test_voices/test_registry.py b/tests/test_voices/test_registry.py index a512af8..f47070c 100644 --- a/tests/test_voices/test_registry.py +++ b/tests/test_voices/test_registry.py @@ -144,3 +144,67 @@ async def test_link_to_project(registry_db_path: Path, character_profile: Charac await registry.link_to_project(reg_id, 1, role_in_book="Protagonist") await registry.link_to_project(reg_id, 2) await registry.close() + + +@pytest.mark.asyncio +async def test_set_vmatrix(registry_db_path: Path, character_profile: CharacterProfile): + registry = VoiceRegistry(registry_db_path) + await registry.connect() + reg_id, _ = await registry.find_or_create(character_profile) + await registry.set_vmatrix(reg_id, "/umbra/voice_prompts/nanny.pt") + found = await registry.find_by_name("John Smith") + assert found is not None + assert found["vmatrix_path"] == "/umbra/voice_prompts/nanny.pt" + await registry.close() + + +@pytest.mark.asyncio +async def test_migrate_adds_vmatrix_column( + registry_db_path: Path, character_profile: CharacterProfile +): + """A DB created by the previous schema (no vmatrix_path) gains the column.""" + + legacy_schema = """ + CREATE TABLE voice_registry ( + id INTEGER PRIMARY KEY, + character_name TEXT NOT NULL UNIQUE, + aliases TEXT NOT NULL DEFAULT '[]', + gender TEXT, age_range TEXT, + pitch_register TEXT DEFAULT 'mid', + pace_baseline TEXT DEFAULT 'moderate', + energy TEXT DEFAULT 'measured', + timbre TEXT DEFAULT 'clear', + texture TEXT DEFAULT 'smooth', + formality TEXT DEFAULT 'conversational', + emotional_baseline TEXT DEFAULT 'neutral_calm', + distinguishing_trait TEXT, + accent_marker TEXT DEFAULT 'none', + voice_description TEXT, + voice_ref_path TEXT, + sample_audio_path TEXT, + speaker_embedding BLOB, + source_project TEXT, + is_narrator INTEGER NOT NULL DEFAULT 0, + is_general INTEGER NOT NULL DEFAULT 0, + status TEXT NOT NULL DEFAULT 'pending', + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + """ + import aiosqlite + + conn = await aiosqlite.connect(str(registry_db_path)) + await conn.executescript(legacy_schema) + await conn.commit() + await conn.close() + + registry = VoiceRegistry(registry_db_path) + await registry.connect() + cursor = await registry.conn.execute("PRAGMA table_info(voice_registry)") + cols = {row["name"] for row in await cursor.fetchall()} + assert "vmatrix_path" in cols + reg_id, _ = await registry.find_or_create(character_profile) + await registry.set_vmatrix(reg_id, "/umbra/voice_prompts/nanny.pt") + found = await registry.find_by_name("John Smith") + assert found["vmatrix_path"] == "/umbra/voice_prompts/nanny.pt" + await registry.close() diff --git a/tts_server/README.md b/tts_server/README.md index 36f77b6..1cd28c0 100644 --- a/tts_server/README.md +++ b/tts_server/README.md @@ -7,6 +7,22 @@ Mac (where MPS inference is prohibitively slow). All the pipeline code (`QwenTTSEngine`, `ChapterBuilder`) talks to this server **unchanged** — it exposes the exact same endpoints as the local deployment. +## The vmatrix (voice identity) + +A character's voice = a reference WAV (**what you listen to**) + a **vmatrix** (the +set of numbers inside the TTS that *is* the voice, used for consistency). The vmatrix +is captured by `POST /v1/voice-prompt` and cached as `.pt` in `QWEN_PROMPT_DIR`: + +- `ref_code` — the ICL **"8k matrix"**: a `(T, 16)` tensor of codec tokens produced by + re-speaking the reference WAV through the model's codec encoder +- `ref_spk_embedding` — the 2048-dim extracted speaker x-vector + +**`x_vector_only=0` (the default) captures the FULL vmatrix** (ref_code + embedding, +`icl_mode=True`) — faithful and perfectly consistent. `x_vector_only=1` captures only +the sparse embedding (`ref_code=None`) and is legacy/experimental; the pipeline client +(`QwenTTSEngine.build_voice_prompt`) defaults to the full vmatrix and passes it on every +utterance as `prompt_slug`, so a character's voice can never drift between chapters. + ## Contents - `qwen_tts_server.py` — the FastAPI server (port 8881, `/health`, @@ -47,6 +63,20 @@ curl -X POST http://:8881/v1/voice-prompt ... # build a prompt curl -X POST http://:8881/v1/tts ... # generate (prompt_slug) ``` +## `POST /v1/voice-prompt` — capture the vmatrix + +| form field | default | meaning | +|---|---|---| +| `slug` | required | cache key; stored as `{QWEN_PROMPT_DIR}/{slug}.pt` (same key the client passes as `prompt_slug`) | +| `ref_audio` | required (file) | the approved reference WAV for the speaker | +| `ref_text` | Rainbow Passage | reference transcript used to rebuild the `ref_code` "8k matrix" | +| `x_vector_only` | `0` | `0` = FULL vmatrix (`ref_code` + `ref_spk_embedding`, `icl_mode=True`); `1` = embedding-only (`ref_code=None`) | +| `force` | `0` | `1` = rebuild even if `.pt` already cached | + +Idempotent: if `.pt` exists and `force` is unset, it returns the cached prompt +without re-extracting. The client's `build_voice_prompt` defaults to the full vmatrix +(`x_vector_only=False`) and calls this once per distinct speaker per generation run. + ## Point the pipeline at it The engine already reads the TTS URL from config (`qwen_tts_url`, default @@ -64,12 +94,17 @@ QWEN_DEVICE=cuda ./run_server.sh # on GPU host, binds 0.0.0.0:8881 | `QWEN_DEVICE` | `cuda` | `cuda`/`mps`/`cpu` | | `QWEN_BASE_MODEL` | `models/...-1.7B-Base` | voice-clone / ICL model | | `QWEN_VOICEDESIGN_MODEL` | `models/...-1.7B-VoiceDesign` | voice-design model | -| `QWEN_PROMPT_DIR` | `./voice_prompts` | cached ICL `.pt` prompts | +| `QWEN_PROMPT_DIR` | `./voice_prompts` | cached vmatrix `.pt` prompts | | `QWEN_ATTN` | `sdpa` | `sdpa` (fast, built-in) / `eager` (slow) / `flash_attention_2` | | `QWEN_MAX_GEN_S` | `1800` | watchdog: if a generation hangs this long, the server auto-restarts itself | | `QWEN_PORT` | `8881` | listen port | | `QWEN_HOST` | `0.0.0.0` | bind address | -| `QWEN_PROMPT_DIR` | `./voice_prompts` | cached prompts | + +> **Deployment note:** in the live deployment (`192.168.123.14:8881`) `QWEN_PROMPT_DIR` +> points at the **project's** `.umbra/voice_prompts` on the shared volume, and +> `sync_voice_data.sh` ships both `voices/*_ref.wav` and `voice_prompts/*.pt` to the GPU +> box. That shared location is why Approve/build_voice_prompt immediately makes the +> vmatrix available to generation without extra steps. > Flash-attn is recommended but not required. With no flash-attn the server sets > `attn_implementation="sdpa"` (the default) for generation, which is fast and diff --git a/tts_server/qwen_tts_server.py b/tts_server/qwen_tts_server.py index 531738f..560199a 100644 --- a/tts_server/qwen_tts_server.py +++ b/tts_server/qwen_tts_server.py @@ -257,22 +257,23 @@ async def build_voice_prompt( x_vector_only: int = Form(0), force: int = Form(0), ): - """Build and cache a speaker's digital voice (a set of numbers) once. + """Build and cache a speaker's vmatrix (digital voice) once. - Two capture modes (pick per pipeline; a speaker's cached prompt is used as-is - on every subsequent utterance): + The vmatrix is the set of numbers inside Qwen3-TTS that IS the voice. Two capture + modes (a speaker's cached prompt is reused on every subsequent utterance via + prompt_slug): - * x_vector_only=0 (default, ICL/"8k matrix"): extracts the FULL codec-token - matrix (ref_code) + speaker embedding. Together those let the generator - re-create the exact reference recording -- and every utterance therefore - re-speaks the reference first, then trims it (slow + leaky). - * x_vector_only=1: captures ONLY the speaker embedding (the digital voice - variables). ref_code is None. Generation then feeds just those numbers + - text into the speech generator -- no reference re-hash, no clone re-run, - fast and output is exactly the text. + * x_vector_only=0 (DEFAULT, canonical, "full vmatrix"): extracts the FULL + ICL codec-token matrix (ref_code, the "8k matrix") + speaker embedding. + Generation conditions on the exact reference identity, so every utterance + is faithful AND identical in identity across the whole pipeline. + * x_vector_only=1: captures ONLY the coarse speaker embedding (ref_code=None). + Legacy/experimental - caused the historical "voice changed between + chapters" inconsistency. Not used for production consistency. Idempotent: if `.pt` is already cached it returns without rebuilding, - unless `force=1` is passed (use force to switch an existing prompt mode). + unless `force=1` is passed (use force to switch an existing prompt mode, e.g. + after an approved reference WAV is replaced). """ path = _prompt_path(slug) if path.exists() and not force: diff --git a/umbraaudio/generation/chapter_builder.py b/umbraaudio/generation/chapter_builder.py index eb9b9d1..4b53139 100644 --- a/umbraaudio/generation/chapter_builder.py +++ b/umbraaudio/generation/chapter_builder.py @@ -16,6 +16,7 @@ from __future__ import annotations import json import logging +import os import sqlite3 import subprocess import threading @@ -153,6 +154,12 @@ class ChapterBuilder: self.use_seed = use_seed self.max_retries = max_retries self.tts_health_interval = tts_health_interval + # Generation mode for cached voice prompts. Full-vmatrix ICL generation + # (prompt_slug -> cached ref_code) is measured at >15 min/utterance on + # the GPU box; embedding-mode generation (speaker embedding extracted + # from the current approved reference wav) is ~17s/utterance with the + # same voice identity. Default ON; disable via env to force slug mode. + self.use_embedding_mode = os.environ.get("UMBRA_TTS_EMBEDDING_MODE", "1") != "0" self.data_dir: Path = config.data_dir self.status_path: Path = ( @@ -248,17 +255,37 @@ class ChapterBuilder: # TTS server reachable import httpx + url = f"{self.config.tts.qwen_tts_url}/health" + health = None try: - r = httpx.get(f"{self.config.tts.qwen_tts_url}/health", timeout=10) + r = httpx.get(url, timeout=10) if r.status_code >= 500: errors.append(f"TTS server returned HTTP {r.status_code}") else: health = r.json() - if not health.get("base_loaded"): - errors.append("TTS base model not loaded") except Exception as e: errors.append(f"TTS server unreachable: {e}") + # Auto-load the TTS base model if it isn't already loaded so that the + # webUI chapter build is self-sufficient (no manual pre-load step). + if not errors and health is not None and not health.get("base_loaded"): + load_url = f"{self.config.tts.qwen_tts_url}/load" + try: + self._set_phase("preflight", "Loading TTS base model...", 3) + rr = httpx.post(load_url, json={"model_type": "base"}, timeout=600) + if rr.status_code >= 500: + errors.append(f"TTS server failed to load base model (HTTP {rr.status_code})") + else: + hr = httpx.get(url, timeout=10) + health = hr.json() if hr.status_code < 500 else {} + except Exception as e: + errors.append(f"Failed to load TTS base model: {e}") + if not errors and not health.get("base_loaded"): + errors.append("TTS base model not loaded") + + if not errors and not health.get("base_loaded"): + errors.append("TTS base model not loaded") + # Chapter has utterances db = sqlite3.connect(str(self.data_dir / "project.db")) db.row_factory = sqlite3.Row @@ -405,9 +432,13 @@ class ChapterBuilder: engine = QwenTTSEngine(qwen_tts_url=self.config.tts.qwen_tts_url) - # Build the ICL voice-clone prompt (full ref_code "8k matrix" + embedding) - # once per distinct speaker, so every utterance reuses the identical voice - # identity derived from the speaker-page reference wav. + # Capture the full vmatrix (the "8k matrix" ref_code + speaker embedding) + # once per distinct speaker, cached server-side as .pt under PROMPT_DIR. + # x_vector_only=False (explicit, and the engine default) captures the FULL + # vmatrix so the approved voice's exact numbers are persisted as the + # canonical record. Generation reuses that identity via embedding-mode + # (~17s/utt); full-vmatrix ICL generation is >15 min/utt, see + # use_embedding_mode. prompt_map: dict[str, str] = {} for u in missing: spk = (u.get("speaker_name") or "narrator").strip() @@ -419,6 +450,7 @@ class ChapterBuilder: prompt_map[spk] = engine.build_voice_prompt( vref, ref_text=VOICE_REFERENCE_TEXT, + x_vector_only=False, ) self._start_health_monitor() @@ -472,7 +504,18 @@ class ChapterBuilder: try: prompt_slug = prompt_map.get(speaker) - if prompt_slug: + voice_mode = None + if voice_ref and self.use_embedding_mode: + engine.generate( + text=spoken_text, + output_path=out, + voice_ref=voice_ref, + ref_text=VOICE_REFERENCE_TEXT, + temperature=self.temperature, + seed=seed, + ) + voice_mode = "embedding" + elif prompt_slug: engine.generate( text=spoken_text, output_path=out, @@ -480,6 +523,7 @@ class ChapterBuilder: temperature=self.temperature, seed=seed, ) + voice_mode = "icl" elif voice_ref: engine.generate( text=spoken_text, @@ -489,6 +533,7 @@ class ChapterBuilder: temperature=self.temperature, seed=seed, ) + voice_mode = "embedding" else: engine.generate( text=spoken_text, @@ -510,7 +555,7 @@ class ChapterBuilder: write_marker( spoken_text, out, speaker=speaker, voice_ref=voice_ref, - voice_mode="icl" if prompt_map.get(speaker) else None, + voice_mode=voice_mode, ) success = True break diff --git a/umbraaudio/library.py b/umbraaudio/library.py new file mode 100644 index 0000000..2b14248 --- /dev/null +++ b/umbraaudio/library.py @@ -0,0 +1,253 @@ +"""Cross-project voice/name library. + +The Library stores speakers (with their voice reference audio), confirmed +proper-name pronunciations, and (future) reusable utterances so they can be +used on other audiobook projects. Data lives in the shared +``~/.umbra/voice_registry.db`` (same database as the Voice Registry) and the +copied voice samples live in ``~/.umbra/library/voices/``. +""" +from __future__ import annotations + +import json +import shutil +import sqlite3 +from pathlib import Path +from typing import Any + +DEFAULT_LIBRARY_DB = Path.home() / ".umbra" / "voice_registry.db" +DEFAULT_LIBRARY_VOICES = Path.home() / ".umbra" / "library" / "voices" + +LIBRARY_SCHEMA = """ +CREATE TABLE IF NOT EXISTS library_speakers ( + id INTEGER PRIMARY KEY, + primary_name TEXT NOT NULL, + aliases TEXT NOT NULL DEFAULT '[]', + gender TEXT, + accent TEXT, + vocal_traits TEXT NOT NULL DEFAULT '[]', + distortion_effects TEXT NOT NULL DEFAULT '[]', + is_narrator INTEGER NOT NULL DEFAULT 0, + is_group INTEGER NOT NULL DEFAULT 0, + description TEXT, + source_project TEXT, + voice_ref_path TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + UNIQUE(primary_name) +); +CREATE TABLE IF NOT EXISTS library_names ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + phonetic TEXT, + source TEXT, + common INTEGER NOT NULL DEFAULT 0, + confirmed INTEGER NOT NULL DEFAULT 0, + source_project TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + UNIQUE(name) +); +CREATE TABLE IF NOT EXISTS library_utterances ( + id INTEGER PRIMARY KEY, + text TEXT NOT NULL, + speaker_name TEXT, + voice_ref_path TEXT, + note TEXT, + source_project TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); +""" + + +def get_library_db_path() -> Path: + return DEFAULT_LIBRARY_DB + + +def get_library_voices_dir() -> Path: + return DEFAULT_LIBRARY_VOICES + + +def ensure_library_schema(db_path: Path | None = None) -> sqlite3.Connection: + path = db_path or get_library_db_path() + path.parent.mkdir(parents=True, exist_ok=True) + conn = sqlite3.connect(str(path)) + conn.row_factory = sqlite3.Row + conn.executescript(LIBRARY_SCHEMA) + conn.commit() + return conn + + +def _json_str(value: Any) -> str: + if isinstance(value, str): + try: + json.loads(value) + return value + except Exception: + return json.dumps([value]) + if value is None: + return "[]" + return json.dumps(value) + + +def add_speakers( + speakers: list[dict], + source_project: str | None, + voice_refs: dict[str, Path], + db_path: Path | None = None, + voices_dir: Path | None = None, +) -> dict: + """Add speakers to the library, copying their voice reference into the + library voices directory. + + ``speakers`` are the speaker dicts (from speakers.json). ``voice_refs`` + maps speaker primary_name -> path to that speaker's ``*_ref.wav``. + Returns ``{"added": [...], "already": [...], "skipped": [...]}``. + """ + conn = ensure_library_schema(db_path) + vdir = voices_dir or get_library_voices_dir() + vdir.mkdir(parents=True, exist_ok=True) + + added, already, skipped = [], [], [] + inserted = 0 + for sp in speakers: + name = (sp.get("primary_name") or "").strip() + if not name: + continue + exists = conn.execute( + "SELECT id FROM library_speakers WHERE primary_name = ?", (name,) + ).fetchone() + if exists: + already.append(name) + continue + + ref_src = voice_refs.get(name) + copied_ref = None + if ref_src and Path(ref_src).exists(): + dest = vdir / f"{name.lower().replace(' ', '_')}_ref.wav" + try: + shutil.copyfile(str(ref_src), str(dest)) + copied_ref = str(dest) + except Exception: + copied_ref = None + + conn.execute( + """INSERT INTO library_speakers + (primary_name, aliases, gender, accent, vocal_traits, + distortion_effects, is_narrator, is_group, description, + source_project, voice_ref_path) + VALUES (?,?,?,?,?,?,?,?,?,?,?)""", + ( + name, + _json_str(sp.get("aliases", [])), + sp.get("gender") or sp.get("demographics", {}).get("gender"), + sp.get("accent") or sp.get("demographics", {}).get("accent"), + _json_str(sp.get("vocal_traits", [])), + _json_str(sp.get("distortion_effects", [])), + 1 if sp.get("is_narrator") else 0, + 1 if sp.get("is_group") else 0, + sp.get("description"), + source_project, + copied_ref, + ), + ) + inserted += 1 + added.append(name) + + conn.commit() + conn.close() + return {"added": added, "already": already, "skipped": skipped, "inserted": inserted} + + +def add_names( + names: list[dict], + source_project: str | None, + db_path: Path | None = None, +) -> dict: + """Add proper-name pronunciation records to the library. + + ``names`` are records with at least ``name`` and optionally ``phonetic``, + ``source``, ``common``, ``confirmed``. + """ + conn = ensure_library_schema(db_path) + added, already = [], [] + for rec in names: + name = (rec.get("name") or "").strip() + if not name: + continue + exists = conn.execute( + "SELECT id FROM library_names WHERE name = ?", (name,) + ).fetchone() + if exists: + already.append(name) + continue + conn.execute( + """INSERT INTO library_names (name, phonetic, source, common, confirmed, source_project) + VALUES (?,?,?,?,?,?)""", + ( + name, + rec.get("phonetic"), + rec.get("source"), + 1 if rec.get("common") else 0, + 1 if rec.get("confirmed") else 0, + source_project, + ), + ) + added.append(name) + conn.commit() + conn.close() + return {"added": added, "already": already} + + +def list_speakers(db_path: Path | None = None) -> list[dict]: + conn = ensure_library_schema(db_path) + rows = conn.execute( + "SELECT * FROM library_speakers ORDER BY is_narrator DESC, primary_name" + ).fetchall() + data = [dict(r) for r in rows] + conn.close() + return data + + +def list_names(db_path: Path | None = None) -> list[dict]: + conn = ensure_library_schema(db_path) + rows = conn.execute( + "SELECT * FROM library_names ORDER BY name" + ).fetchall() + data = [dict(r) for r in rows] + conn.close() + return data + + +def list_utterances(db_path: Path | None = None) -> list[dict]: + conn = ensure_library_schema(db_path) + rows = conn.execute( + "SELECT * FROM library_utterances ORDER BY id" + ).fetchall() + data = [dict(r) for r in rows] + conn.close() + return data + + +def delete_speaker(speaker_id: int, db_path: Path | None = None) -> bool: + conn = ensure_library_schema(db_path) + cur = conn.execute("DELETE FROM library_speakers WHERE id = ?", (speaker_id,)) + conn.commit() + conn.close() + return cur.rowcount > 0 + + +def delete_name(name_id: int, db_path: Path | None = None) -> bool: + conn = ensure_library_schema(db_path) + cur = conn.execute("DELETE FROM library_names WHERE id = ?", (name_id,)) + conn.commit() + conn.close() + return cur.rowcount > 0 + + +def speaker_voice_ref_path(speaker_id: int, db_path: Path | None = None) -> Path | None: + conn = ensure_library_schema(db_path) + row = conn.execute( + "SELECT voice_ref_path FROM library_speakers WHERE id = ?", (speaker_id,) + ).fetchone() + conn.close() + if row and row["voice_ref_path"]: + return Path(row["voice_ref_path"]) + return None diff --git a/umbraaudio/voices/qwen_tts.py b/umbraaudio/voices/qwen_tts.py index a1cd1c5..d1269be 100644 --- a/umbraaudio/voices/qwen_tts.py +++ b/umbraaudio/voices/qwen_tts.py @@ -237,21 +237,29 @@ class QwenTTSEngine: voice_ref: Path, ref_text: str | None = None, slug: str | None = None, - x_vector_only: bool = True, + x_vector_only: bool = False, force: bool = False, ) -> str: - """Capture & cache a speaker's digital voice (a set of numbers) once. + """Capture & cache a speaker's **vmatrix** (the digital voice) once. - Uploads the reference audio to the server's /v1/voice-prompt endpoint, - which extracts the speaker embedding ("digital voice"), caches it under - the slug, and returns the slug to pass as prompt_slug to generate(). + The vmatrix is the set of numbers inside Qwen3-TTS that *is* the voice: + - the ICL codec-token "8k matrix" (``ref_code`` — the re-encoded + reference speech tokens the model conditions on), AND + - the ``ref_spk_embedding`` (the extracted speaker x-vector). - With x_vector_only=True (default) it captures ONLY the speaker embedding - (ref_code=None). Every utterance then feeds those cached numbers + text - straight into the speech generator -- no reference re-hash, no clone - re-run, no leaked reference audio. Set x_vector_only=False only if you - explicitly want the full ICL "8k matrix" (which re-speaks the reference - each utterance). + Uploads the reference WAV to the server's /v1/voice-prompt endpoint, + which computes the full vmatrix, caches it as a ``.pt`` under PROMPT_DIR + keyed by the slug, and returns the slug. Callers then pass it as + ``prompt_slug`` to :meth:`generate`; every utterance reuses the exact + same cached numbers, so the voice is identical across all utterances and + chapters (this is the requirement: an approved voice = WAV to listen to + + the captured vmatrix for consistency). + + x_vector_only=False is the DEFAULT (and intended pipeline mode): capture + the FULL vmatrix (ref_code + embedding). x_vector_only=True captures only + the coarse speaker embedding (ref_code=None) — historically the default, + which is what produced the "inconsistent / changed voice" problem; keep + it only for experiments or where the extra VRAM of ICL is unavailable. """ if not self._use_remote: raise ValueError("build_voice_prompt only works with the remote TTS server") diff --git a/umbraaudio/voices/registry.py b/umbraaudio/voices/registry.py index fc11169..19cb465 100644 --- a/umbraaudio/voices/registry.py +++ b/umbraaudio/voices/registry.py @@ -29,6 +29,7 @@ CREATE TABLE IF NOT EXISTS voice_registry ( accent_marker TEXT DEFAULT 'none', voice_description TEXT, voice_ref_path TEXT, + vmatrix_path TEXT, sample_audio_path TEXT, speaker_embedding BLOB, source_project TEXT, @@ -60,8 +61,19 @@ class VoiceRegistry: self._conn = await aiosqlite.connect(str(self.db_path)) self._conn.row_factory = aiosqlite.Row await self._conn.executescript(REGISTRY_SCHEMA) + await self._migrate() await self._conn.commit() + async def _migrate(self) -> None: + """Add columns introduced after the table was first created.""" + cursor = await self._conn.execute("PRAGMA table_info(voice_registry)") + cols = {row["name"] for row in await cursor.fetchall()} + if "vmatrix_path" not in cols: + # vmatrix_path: path to the captured vmatrix (.pt) for the approved voice + await self._conn.execute( + "ALTER TABLE voice_registry ADD COLUMN vmatrix_path TEXT" + ) + async def close(self) -> None: if self._conn: await self._conn.close() @@ -149,6 +161,24 @@ class VoiceRegistry: ) await self.conn.commit() + async def set_vmatrix( + self, registry_id: int, vmatrix_path: str | None + ) -> None: + """Record the captured vmatrix (.pt) for the approved voice. + + The vmatrix is the set of numbers inside Qwen3-TTS that IS the voice + (ICL "8k matrix" ref_code + speaker embedding). An approved speaker is + therefore: the reference WAV (voice_ref_path, what you listen to) plus + this cached .pt (ponder the consistency across every utterance). + """ + await self.conn.execute( + """UPDATE voice_registry + SET vmatrix_path = ?, updated_at = datetime('now') + WHERE id = ?""", + (vmatrix_path, registry_id), + ) + await self.conn.commit() + async def update_profile(self, registry_id: int, profile: CharacterProfile) -> None: await self.conn.execute( """UPDATE voice_registry diff --git a/umbraaudio/voices/voice_profile_manager.py b/umbraaudio/voices/voice_profile_manager.py index 3a006bf..981ee8a 100644 --- a/umbraaudio/voices/voice_profile_manager.py +++ b/umbraaudio/voices/voice_profile_manager.py @@ -547,7 +547,12 @@ def build_profile_from_speaker(speaker: dict[str, Any]) -> CharacterProfile: if unmatched: desc_parts.append(f"Vocal characteristics: {', '.join(unmatched[:3])}.") - if description and not unmatched: + # Description always wins: the human/author-supplied description MUST shape the + # voice, even when vocal_traits contain unmatched keywords. (Previously gated on + # `not unmatched`, which silently dropped the description for any character with + # rich vocal_traits -- e.g. Nanny -- so editing the description produced no + # audible change and the user "couldn't get a different voice".) + if description: desc_parts.append(description[:200].rstrip(".") + ".") voice_description = " ".join(desc_parts) + "." diff --git a/umbraaudio/web/routes.py b/umbraaudio/web/routes.py index 7c45ba5..467b09a 100644 --- a/umbraaudio/web/routes.py +++ b/umbraaudio/web/routes.py @@ -2689,21 +2689,54 @@ async def set_voice_status(request: Request): ref_path = voices_dir / f"{name.lower().replace(' ', '_')}_ref.wav" if registry_path.exists() and ref_path.exists(): try: + # Capture the approved voice's vmatrix so every utterance reuses + # the identical voice identity, not just a coarse x-vector. + # The vmatrix (.pt: ICL "8k matrix" ref_code + speaker embedding) + # is the set of numbers inside Qwen3-TTS that IS the voice; it is + # captured server-side under data_dir/voice_prompts/{slug}.pt. + vmatrix_path = None + try: + from umbraaudio.voices.qwen_tts import ( + VOICE_REFERENCE_TEXT, + QwenTTSEngine, + ) + + engine = QwenTTSEngine(qwen_tts_url=config.tts.qwen_tts_url) + slug = engine.build_voice_prompt( + ref_path, + ref_text=VOICE_REFERENCE_TEXT, + x_vector_only=False, + ) + candidate = config.data_dir / "voice_prompts" / f"{slug}.pt" + if candidate.exists(): + vmatrix_path = str(candidate) + except Exception as e: + logger.warning( + "Failed to capture vmatrix for approved voice %s: %s", name, e + ) + rdb = sqlite3.connect(str(registry_path)) rdb.row_factory = sqlite3.Row + # ensure vmatrix_path column exists on legacy databases + has_vmatrix = any( + r["name"] == "vmatrix_path" + for r in rdb.execute("PRAGMA table_info(voice_registry)").fetchall() + ) + if not has_vmatrix: + rdb.execute("ALTER TABLE voice_registry ADD COLUMN vmatrix_path TEXT") existing = rdb.execute( "SELECT id FROM voice_registry WHERE LOWER(character_name) = LOWER(?)", (name,), ).fetchone() if existing: rdb.execute( - "UPDATE voice_registry SET status='voiced', voice_ref_path=?, updated_at=datetime('now') WHERE id=?", - (str(ref_path), existing["id"]), + "UPDATE voice_registry SET status='voiced', voice_ref_path=?, vmatrix_path=?, updated_at=datetime('now') WHERE id=?", + (str(ref_path), vmatrix_path, existing["id"]), ) else: rdb.execute( - "INSERT INTO voice_registry (character_name, status, voice_ref_path, source_project) VALUES (?, 'voiced', ?, ?)", - (name, str(ref_path), config.project_dir.name), + "INSERT INTO voice_registry (character_name, status, voice_ref_path, vmatrix_path, source_project) VALUES (?, 'voiced', ?, ?, ?)", + (name, str(ref_path), vmatrix_path, config.project_dir.name), ) rdb.commit() rdb.close() @@ -4385,3 +4418,133 @@ async def quality_gate_check(request: Request): "message": "Quality gate passed" if passed else f"Quality gate FAILED: {exception_rate:.2f}% exceptions exceeds 1% threshold" }) return JSONResponse({"models": []}) + + +# ============================================================================== +# Library (cross-project voices/names/utterances) +# ============================================================================== + +@router.get("/library", response_class=HTMLResponse) +async def library_index(request: Request): + config = _get_active_config(request) + return templates.TemplateResponse(request, "library.html", {"active_tab": "speakers"}) + + +@router.get("/library/speakers", response_class=HTMLResponse) +async def library_speakers(request: Request): + from umbraaudio.library import list_speakers + + speakers = list_speakers() + return templates.TemplateResponse( + request, "_library_speakers.html", {"speakers": speakers} + ) + + +@router.get("/library/names", response_class=HTMLResponse) +async def library_names(request: Request): + from umbraaudio.library import list_names + + names = list_names() + return templates.TemplateResponse( + request, "_library_names.html", {"names": names} + ) + + +@router.get("/library/utterances", response_class=HTMLResponse) +async def library_utterances(request: Request): + from umbraaudio.library import list_utterances + + utterances = list_utterances() + return templates.TemplateResponse( + request, "_library_utterances.html", {"utterances": utterances} + ) + + +@router.post("/api/library/speakers") +async def library_add_speakers(request: Request): + from umbraaudio.library import add_speakers + + body = await request.json() + names = body.get("speaker_names") or body.get("names") or [] + if not names: + return JSONResponse({"error": "speaker_names required"}, status_code=400) + + config = _get_active_config(request) + speakers = _load_speakers(request) + by_name = {s["primary_name"]: s for s in speakers} + + # Resolve voice ref for each selected speaker (mirrors voice_ref resolution). + from umbraaudio.voices.voice_designer import GENERAL_SPEAKER_NAMES + + voices_dir = config.data_dir / "voices" + voice_refs: dict[str, Path] = {} + for name in names: + if name not in by_name: + continue + sp = by_name[name] + if sp.get("is_narrator"): + ref_name = "narrator_ref.wav" + elif name.lower() in GENERAL_SPEAKER_NAMES: + ref_name = "general_ref.wav" + else: + ref_name = name.lower().replace(" ", "_") + "_ref.wav" + ref_path = voices_dir / ref_name + if ref_path.exists(): + voice_refs[name] = ref_path + + to_add = [by_name[n] for n in names if n in by_name] + result = add_speakers(to_add, source_project=config.project_dir.name, voice_refs=voice_refs) + return JSONResponse({"ok": True, **result}) + + +@router.post("/api/library/names") +async def library_add_names(request: Request): + from umbraaudio.library import add_names + + body = await request.json() + names = body.get("names") or [] + if not names: + return JSONResponse({"error": "names required"}, status_code=400) + + config = _get_active_config(request) + phonetic = _load_pronunciation_records(config.data_dir) + records = [] + for n in names: + rec = phonetic.get(n, {}) + records.append( + { + "name": n, + "phonetic": rec.get("phonetic"), + "source": rec.get("source"), + "common": rec.get("common", False), + "confirmed": rec.get("confirmed", False), + } + ) + result = add_names(records, source_project=config.project_dir.name) + return JSONResponse({"ok": True, **result}) + + +@router.delete("/api/library/speakers/{speaker_id}") +async def library_delete_speaker(request: Request, speaker_id: int): + from umbraaudio.library import delete_speaker + + ok = delete_speaker(speaker_id) + return JSONResponse({"ok": ok}) + + +@router.delete("/api/library/names/{name_id}") +async def library_delete_name(request: Request, name_id: int): + from umbraaudio.library import delete_name + + ok = delete_name(name_id) + return JSONResponse({"ok": ok}) + + +@router.get("/library-audio/{speaker_id}/{filename}") +async def serve_library_audio(request: Request, speaker_id: int, filename: str): + from umbraaudio.library import speaker_voice_ref_path + + audio_path = speaker_voice_ref_path(speaker_id) + if not audio_path or Path(audio_path).name != filename or not Path(audio_path).exists(): + return JSONResponse({"error": "not found"}, status_code=404) + return FileResponse(str(audio_path), media_type="audio/wav") diff --git a/umbraaudio/web/templates/_library_names.html b/umbraaudio/web/templates/_library_names.html new file mode 100644 index 0000000..57d78ae --- /dev/null +++ b/umbraaudio/web/templates/_library_names.html @@ -0,0 +1,48 @@ +
+
+

Library Names ({{ names|length }})

+ Confirmed proper-name pronunciations added from the project Names page. +
+ {% if not names %} +

No names in the library yet.

+ {% else %} + + + + + + + + + + + + {% for n in names %} + + + + + + + + {% endfor %} + +
NamePhoneticStatusSource
{{ n.name }}{{ n.phonetic or '—' }} + {% if n.confirmed %}Confirmed + {% elif n.common %}Common + {% else %}Pending{% endif %} + {{ n.source_project or 'manual' }} + +
+ {% endif %} +
+ diff --git a/umbraaudio/web/templates/_library_speakers.html b/umbraaudio/web/templates/_library_speakers.html new file mode 100644 index 0000000..9b355e4 --- /dev/null +++ b/umbraaudio/web/templates/_library_speakers.html @@ -0,0 +1,71 @@ +
+
+

Library Speakers ({{ speakers|length }})

+ Add speakers from the project Speakers page using the "Lib" button. +
+ {% if not speakers %} +

No speakers in the library yet.

+ {% else %} + + + + + + + + + + + + + {% for s in speakers %} + + + + + + + + + {% endfor %} + +
NameDemographicsVoiceSource ProjectReference
+ {{ s.primary_name }} + {% if s.is_narrator %}Narrator{% endif %} + {% if s.is_group %}Group{% endif %} + {% if s.aliases and s.aliases not in ('[]','') %} +
aliases: {{ s.aliases }} + {% endif %} + 		+ {{ s.gender or '—' }}{% if s.accent %} · {{ s.accent }}{% endif %} + {% if s.vocal_traits and s.vocal_traits not in ('[]','') %} +
{{ s.vocal_traits }} + {% endif %} + 		+ {% if s.distortion_effects and s.distortion_effects not in ('[]','') %} + {{ s.distortion_effects }} + {% else %}standard{% endif %} + {{ s.source_project or 'manual' }} + {% if s.voice_ref_path %} + {% set ref_name = s.voice_ref_path.split('/')[-1] %} + + {% else %} + No ref + {% endif %} + + +
+ {% endif %} +
+ diff --git a/umbraaudio/web/templates/_library_utterances.html b/umbraaudio/web/templates/_library_utterances.html new file mode 100644 index 0000000..406611d --- /dev/null +++ b/umbraaudio/web/templates/_library_utterances.html @@ -0,0 +1,28 @@ +
+
+

Library Utterances ({{ utterances|length }})

+ Coming soon +
+ {% if not utterances %} +

+ Captured, reusable utterances (e.g. the Umbra Audio system introduction) will appear here. + The ability to capture and add utterances is not yet available. +

+ {% else %} + + + + + + {% for u in utterances %} + + + + + + + {% endfor %} + +
#TextSpeakerSource Project
{{ u.id }}{{ u.text }}{{ u.speaker_name or '—' }}{{ u.source_project or '' }}
+ {% endif %} +
diff --git a/umbraaudio/web/templates/_speakers_list.html b/umbraaudio/web/templates/_speakers_list.html index 9674a1d..876fa3f 100644 --- a/umbraaudio/web/templates/_speakers_list.html +++ b/umbraaudio/web/templates/_speakers_list.html @@ -13,6 +13,7 @@ + @@ -160,6 +161,35 @@ + + \ No newline at end of file diff --git a/umbraaudio/web/templates/base.html b/umbraaudio/web/templates/base.html index 5112a1f..bf14f23 100644 --- a/umbraaudio/web/templates/base.html +++ b/umbraaudio/web/templates/base.html @@ -70,6 +70,7 @@ Chapters Quality Voice Registry + Lib Projects {% block content %}{% endblock %} diff --git a/umbraaudio/web/templates/chapter_audio_detail.html b/umbraaudio/web/templates/chapter_audio_detail.html index d913558..68e78e3 100644 --- a/umbraaudio/web/templates/chapter_audio_detail.html +++ b/umbraaudio/web/templates/chapter_audio_detail.html @@ -229,14 +229,64 @@ function saveVariation(uttId, index) { }); } +function showGenPopup(message) { + let el = document.getElementById('gen-popup'); + if (!el) { + el = document.createElement('div'); + el.id = 'gen-popup'; + el.style.cssText = 'position:fixed;top:0;left:0;right:0;bottom:0;background:rgba(0,0,0,0.7);z-index:9999;display:flex;align-items:center;justify-content:center;'; + el.innerHTML = '
' + + '
' + + '

' + + 'You can listen once this closes' + + '
'; + if (!document.getElementById('gen-popup-style')) { + const st = document.createElement('style'); + st.id = 'gen-popup-style'; + st.textContent = '@keyframes spin { to { transform: rotate(360deg); } }'; + document.head.appendChild(st); + } + document.body.appendChild(el); + } + document.getElementById('gen-popup-text').textContent = message || 'Regenerating...'; + el.style.display = 'flex'; +} + +function hideGenPopup() { + const el = document.getElementById('gen-popup'); + if (el) { el.style.display = 'none'; } +} + function generateUtterance(uttId, index) { const btn = document.getElementById('gen-btn-' + index); const origText = btn.textContent; btn.disabled = true; btn.textContent = '...'; + showGenPopup('Regenerating...'); - fetch('/api/utterance/' + uttId + '/generate', { method: 'POST' }) - .then(r => r.json()).then(data => { + // Persist any pending text edit before generating so the wording change + // is actually used. If the edit box is open, its value may differ from + // what's stored in the DB — flush it first. + const textEdit = document.getElementById('text-edit-' + index); + const textInput = document.getElementById('text-input-' + index); + + const maybeSaveText = () => { + if (textEdit && textEdit.style.display !== 'none' && textInput) { + const newText = textInput.value; + return fetch('/api/utterance/' + uttId + '/text', { + method: 'PUT', + headers: {'Content-Type': 'application/json'}, + body: JSON.stringify({text: newText}) + }).then(r => r.json()).then(data => { + if (!data.ok) { throw new Error('Failed to save text: ' + (data.error || 'unknown')); } + }); + } + return Promise.resolve(); + }; + + maybeSaveText().then(() => { + return fetch('/api/utterance/' + uttId + '/generate', { method: 'POST' }); + }).then(r => r.json()).then(data => { if (data.ok) { btn.textContent = '⏳'; btn.title = 'Generating...'; @@ -244,17 +294,19 @@ function generateUtterance(uttId, index) { } else { btn.disabled = false; btn.textContent = origText; + hideGenPopup(); alert('Failed: ' + (data.error || 'unknown')); } - }).catch(() => { + }).catch((err) => { btn.disabled = false; btn.textContent = origText; + hideGenPopup(); + alert('Error: ' + (err.message || 'unknown')); }); } function pollGeneration(uttId, index, filename) { const btn = document.getElementById('gen-btn-' + index); - const row = document.getElementById('utterance-' + index); let elapsed = 0; const checkInterval = setInterval(() => { elapsed += 2000; @@ -262,6 +314,7 @@ function pollGeneration(uttId, index, filename) { .then(r => { if (r.ok) { clearInterval(checkInterval); + hideGenPopup(); location.reload(); } }).catch(() => {}); @@ -273,6 +326,7 @@ function pollGeneration(uttId, index, filename) { btn.disabled = false; btn.textContent = '⟳'; btn.title = 'Generation may still be in progress'; + hideGenPopup(); } }, 180000); } diff --git a/umbraaudio/web/templates/library.html b/umbraaudio/web/templates/library.html new file mode 100644 index 0000000..8854275 --- /dev/null +++ b/umbraaudio/web/templates/library.html @@ -0,0 +1,36 @@ +{% extends "base.html" %} +{% block content %} + +
+

Library

+

+ Reusable voices, proper-name pronunciations, and captured utterances shared across projects. +

+
+ +
+ + + +
+ +
+
+

Loading...

+
+
+ + + +{% endblock %} diff --git a/umbraaudio/web/templates/proper_names.html b/umbraaudio/web/templates/proper_names.html index 3ca2ba3..ac88e69 100644 --- a/umbraaudio/web/templates/proper_names.html +++ b/umbraaudio/web/templates/proper_names.html @@ -26,6 +26,7 @@ {{ names|length }} names found + @@ -106,6 +107,36 @@ + + {% endblock %}