commit c86e323d7a2dc1387a5944e0e12dd838bfa84356 Author: B.D. Murphy <161863146+bdmurphy73@users.noreply.github.com> Date: Sun Aug 30 07:58:24 2026 -0700 Initial commit: UmbraAudio audiobook generation program Quality-first multi-pass audiobook generation from English manuscripts. Includes: - umbraaudio/ core package (pipeline, config, voices, web review UI) - scripts/ CLI utilities + model download script - tts_server/ remote TTS server - specs/ design docs, tests/, pyproject.toml + uv.lock - Qwen3-TTS git submodule (pinned 022e286) with local demo.py UI patch saved at scripts/qwen3_tts_demo_umbra_patch.patch - .env with runtime config/secrets Excluded (see .gitignore): models/ (downloaded via scripts/download_models.sh), output/, logs/, .venv, .umbra runtime data, and book projects (DealHunterShip_*), which are committed separately. diff --git a/.env b/.env new file mode 100644 index 0000000..79b53af --- /dev/null +++ b/.env @@ -0,0 +1,44 @@ +# UmbraAudio Environment Configuration +# Copy this to .env and modify as needed + +# --- Project --- +UMBRA_PROJECT_DIR=. +UMBRA_DATA_DIR=.umbra + +# --- TTS --- +UMBRA_TTS__ENGINE=qwen +UMBRA_TTS__DEVICE=mps +UMBRA_TTS__SAMPLE_RATE=24000 +UMBRA_TTS__MAX_RETRIES=3 +UMBRA_TTS__RETRY_DELAY=2.0 +UMBRA_TTS__VOICEDESIGN_MODEL=/Volumes/1TB_APL_Storage/UmbraAudio/models/Qwen3-TTS-12Hz-1.7B-VoiceDesign +UMBRA_TTS__BASE_MODEL=/Volumes/1TB_APL_Storage/UmbraAudio/models/Qwen3-TTS-12Hz-1.7B-Base + +# --- LLM --- +UMBRA_LLM__PROVIDER=auto +UMBRA_LLM__LMSTUDIO_BASE_URL=http://localhost:1234/v1 +UMBRA_LLM__LMSTUDIO_MODEL=qwen/qwen3.5-9b +UMBRA_LLM__OPENAI_MODEL=gpt-4o +UMBRA_LLM__TEMPERATURE=0.3 +UMBRA_LLM__MAX_TOKENS=4096 + +# --- Web --- +UMBRA_WEB__HOST=0.0.0.0 +UMBRA_WEB__PORT=8420 +UMBRA_WEB__AUTO_OPEN=false + +# --- Quality --- +UMBRA_QUALITY__MIN_MOS=3.8 +UMBRA_QUALITY__MIN_TEST_MOS=4.0 +UMBRA_QUALITY__MAX_WER=0.03 +UMBRA_QUALITY__MIN_SPEAKER_SIMILARITY=0.80 +UMBRA_QUALITY__MIN_TEST_SPEAKER_SIMILARITY=0.85 +UMBRA_QUALITY__MIN_ARTIFACT_SCORE=3.5 + +# --- Audio --- +UMBRA_AUDIO__TARGET_RMS_DB=-20.0 +UMBRA_AUDIO__PEAK_LIMIT_DB=-3.0 +UMBRA_AUDIO__NOISE_FLOOR_DB=-60.0 +UMBRA_AUDIO__SILENCE_BETWEEN_UTTERANCES_MS=500 +UMBRA_AUDIO__SILENCE_BETWEEN_PARAGRAPHS_MS=2000 +UMBRA_AUDIO__SILENCE_BETWEEN_CHAPTERS_MS=4000 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..10aa68e --- /dev/null +++ b/.gitignore @@ -0,0 +1,39 @@ +# macOS +.DS_Store + +# Python +__pycache__/ +*.py[cod] +*.egg-info/ +.mypy_cache/ +.pytest_cache/ +.ruff_cache/ + +# Virtual environments & toolchains +.venv/ +.python/ + +# HuggingFace cache +.hf_home/ + +# Downloaded models (see scripts/download_models.sh) +/models/ + +# Generated output (audio artifacts) +/output/ + +# Logs +/logs/ + +# Build artifacts +*.zip +umbraaudio_package.zip + +# Local TTS server runtime +/tts_server/__pycache__/ + +# Book projects are committed separately (not part of this program repo) +/DealHunterShip_4.0e_whole-to-audio/ + +# App runtime data (per-project book data; projects committed separately) +/.umbra/ diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..0da0689 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "Qwen3-TTS"] + path = Qwen3-TTS + url = https://github.com/QwenLM/Qwen3-TTS.git diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..b604e45 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,200 @@ +# UmbraAudio — Development Conventions + +## Project Overview +Quality-first multi-pass audiobook generation from English manuscripts. +Each pass has quality gates. Quality is the primary concern; generation speed is secondary. + +## Architecture +- **5-pass pipeline**: Analysis → Voice Design → Mapping → Test → Generate +- **Multi-engine TTS**: Edge-TTS (primary for voice variety), Chatterbox (secondary for cloning) +- **LLM-enhanced**: gemma-4-12b-it-qat for character discovery and profiling +- **Web review**: FastAPI + HTMX at checkpoints for human approval + +## Voice System +- **Narrator**: Dedicated soft-spoken male voice (en-US-GuyNeural:-10), deep, radio speaker style +- **Characters**: 59 unique voices from Edge-TTS with pitch shifting +- **Voice Assignment**: Stored in `.umbra/voice_assignments.json` +- **Test Samples**: Verification phrase uses the Rainbow Passage (phonetically balanced paragraph covering all English phonemes) + +## Tech Stack +- Python 3.11+, asyncio throughout +- SQLite via aiosqlite for project state +- FastAPI + HTMX for web review UI +- Click + Rich for CLI +- Edge-TTS for voice generation +- LM Studio (`google/gemma-4-e4b`) for character discovery and speaker attribution + +## Commands +- **Lint:** `ruff check umbraaudio/ scripts/` +- **Format:** `ruff format umbraaudio/ scripts/` +- **Type check:** `mypy umbraaudio/` +- **Test:** `pytest tests/ -v` +- **Run:** `python -m umbraaudio.cli` +- **Web:** `python -c "from pathlib import Path; from umbraaudio.web import create_app; from umbraaudio.config import load_config; import uvicorn; config = load_config(Path('.')); app = create_app(config); uvicorn.run(app, host='0.0.0.0', port=8420)"` +- **Mechanical re-segment:** `python scripts/resegment_all.py --mechanical-only` +- **LLM speaker attribution:** `python scripts/reattribute_all.py --model google/gemma-4-e4b` +- **LLM character discovery:** 4-pass pipeline via `/api/analyze` endpoint (see specs/02-analysis.md) + +## Code Conventions +- Type hints on all function signatures +- Docstrings on public classes/methods +- `from __future__ import annotations` in every file +- No comments unless explaining *why*, not *what* +- Use `logging` module, never `print()` for debug output +- Async for all I/O-bound operations +- Dataclasses or Pydantic models for structured data +- Paths via `pathlib.Path`, never string paths + +## Testing +- Tests mirror source structure: `umbraaudio/parsers/` → `tests/test_parsers.py` +- Use pytest fixtures for database, config, mock TTS engine +- Mock external services (LLM APIs, TTS engines) in tests +- Integration tests for full pipeline with sample manuscript + +## File Naming +- Modules: `snake_case.py` +- Classes: `PascalCase` +- Functions: `snake_case` +- Constants: `UPPER_SNAKE_CASE` +- Private: `_leading_underscore` + +## Dependencies +- Check `pyproject.toml` before adding new dependencies +- Never add a dependency if stdlib suffices +- Prefer: stdlib → third-party → custom implementation + +## Current Session Summary + +### Goal +Fix speaker-attribution errors so the user can review/accept voices with confidence. Reported +failures in chapter "Salvage" (ch6): utt35 "We have strict orders…" (after "she asks the lead guard… +He replies") was tagged Kabir instead of the lead guard, and utt156–157 "The voice calls again" / +"Help me, please." was tagged Deal Hunter instead of the female space-survivor (Kainda). + +### Root Cause Analysis +1. **No rule for unnamed role tags**: narration "the lead guard", "the robot" → LLM guessed a named + character present in the scene (Kabir) instead of the role. +2. **Old prompt rule 3e over-applied**: any "voice"/comm → Deal Hunter, so a human voice ("Help me, + please.") became the ship AI. +3. **Context window too narrow**: `_build_position_context` only looked 8 utterances back → "It + sounds female," (13 lines before) fell outside, so a female survivor's voice went to Deal Hunter. +4. **No verification pass**: attribution was trust-on-faith; wrong speakers were never caught + mechanically. +5. **`ModelCoordinator.load_lm` was not idempotent**: re-entering CHAPTER_ANALYSIS always re-POSTed + `/v1/models/load` → a second copy of a 9 GB model failed to allocate on the 16 GB Mac. + +### Fixes Implemented (this session) +- **`umbraaudio/analysis/llm_attributor.py` prompt**: new rule 3e (Deal Hunter/AI-1 ONLY when narration + explicitly names a computer/AI/comm system; a bare "a voice is heard"/"the voice calls again" over + radio is HUMAN — use gender cues like "it sounds female"), 3f (pronoun coreference; never resolve to + an absent character or a named char when the antecedent is an unnamed role), 3g (unnamed roles → the + role speaker's EXACT primary_name e.g. "Chief of the Guard", else a Title Case role name; NEVER a + random named character), 3h (repeated lines — "The voice calls again, 'Help me, please.'" — keep the + first occurrence's speaker). Context widened: `_extract_scene_narration` 12→20 lines; + `_build_position_context` 8→40 utterances back, labelled dialogue, 4200-char / 45-part caps. +- **New `umbraaudio/analysis/attribution_qa.py`** (mechanical cross-check): `check_attribution()` + flags `speech_tag_mismatch` (conf 0.9 resolved / 0.7 unresolved-name), `narrator_dialogue` (0.95), + `role_tag_guessed` (0.5). Tag applied only from the IMMEDIATELY preceding narration (never across a + dialogue line — prevents shared-tag bleed like "Kainda asks," + reply). Pronoun subjects skipped + (coreference is the LLM's job). `_names_match()` falls back to word-overlap for unresolved titles + ("Wakam" vs "Trader Wakam", "Admiral Wake" vs "Admiral Wilbur Wake"). `fix_high_confidence()` writes + back only resolved exact-name mismatches. Role regex capped at 2 role words and rejects trailing + conjunctions ("the robot and tells"). +- **New `scripts/qa_attribution.py`** CLI: `--chapters N`, `--fix`, `--type`. `_build_alias_map` + includes primary, aliases, and first-word mappings. +- **`umbraaudio/voices/model_coordinator.py`**: `load_lm` (sync + async) now no-ops when the requested + model already has a loaded instance — re-entering CHAPTER_ANALYSIS no longer tries to allocate a + second copy. +- **`.umbra/speakers.json` aliases added**: `Admiral Wake`/`The Admiral`→Admiral Wilbur Wake, + `Captain Antlion`/`Antlion`→Captain Wolfgang Antlion, `Wakam`/`Wakam the Trader`→Trader Wakam, + `Vicky`→Duchess Victoria, `Pirate Leader`/`The Pirate Leader`→Pirate Clan Leader, + `Niñera`/`Nini`/`The Nanny`→Nanny (Niñera = the nanny robot, Support Robot 74532). This session: + `Lead Guard`→Chief of the Guard, `Lead Trainer`→Training Officer, `Bill`→Worker Robot (Bill), + `Captain Alec Weston`→Captain Weston. + +### Completed Work +- **Full 35-chapter LLM re-attribute** with the new prompt (`google/gemma-4-e4b`, ~21 tok/s at ~1.3 + min/batch; total ~2 h): 1878 dialogue utterances, **0 left as narrator**. Chapters 0–4 and 33–35 are + front/back matter with no dialogue (skipped as "no utterances" or 0 dialogue). +- **QA + auto-fix loop**: book-wide QA fell from 38 → 10 issues. 21 auto-fixes applied (resolved + tag mismatches) plus 5 evidence-driven role fixes. +- **Fixed speakers**: ch6 utt157→Kainda Walters Foster Murphy ("Help me, please." = female survivor); + ch6 utt44/49/52/56/211→Nanny (the "robot"/Niñera who rushes in and explains the attack is the nanny + bot); ch8 utt18/33→Trader Wakam; ch9 utt22→Duchess Victoria (tag "Vicky says assertively"); + ch16 utt120→Pirate Clan Leader (the "modified voice" on the call); ch17 utt94→Deal Hunter; ch18 + utt154→Xu Ming, utt168→Kainda; ch20 utt230/utt345→Warbots / Pirate Clan Leader; ch23 utt63→Deal + Hunter; ch24 utt15→Kainda; ch25 utt131/utt224→Kainda, utt133→Warbots; ch30 utt82/115→Admiral + Wilbur Wake, utt113/133→Duchess Victoria, utt162/170→Kabir, utt194/241→Captain Wolfgang Antlion; + ch32 utt68/87→Captain Wolfgang Antlion. +- **Ghost-speaker sweep** (post-review, driven by "Kabir isn't in ch6"): script cross-checked every + assigned speaker against all name/alias tokens present in the chapter text (stopwords filtered, + alias-aware — Willamina's "Captain" alias counts). Found and fixed: + - ch6 utt38/40→Nanny (were `Kabir`; the "service robot … named Niñera" who enters with the suit + says these; Kabir appears only once in ch6, in narration as the off-screen brother). + - ch9 utt2/8/11/13→Training Officer (were `Admiral Wilbur Wake`; narration: "She is talking to the + lead trainer for the young royals" / "the argument with the training officer"; profile: [9]). + - ch16 utt122/123/124→Pirate Clan Leader (were `Admiral Wilbur Wake`; the "modified voice" scolding + the failing pirate, same voice as utt120). + - ch24 utt16→Pilot (Freighter) (were `Alec`; same freighter pilot as utt18/20; Alec is a ch8-only + Wakam associate). + - ch26 utt128/139→Frigate Captain (were `Admiral Wilbur Wake`; "the pirate captain … yells" on the + frigate bridge), ch26 utt135→Frigate Captain (were Willamina; "Seal the bridge" order after the + captain sees the video). Profile: [24,26]. ch26 utt194→Kainda Walters Foster Murphy (were Admiral + Wilbur Wake; she's the one who made it to 12.5G in the pilot chair — Warbots reply "Yes, Captain"). +- **Remaining QA flags** are deliberate assignments the checker can't see intent behind (Nanny=robot, + Kainda="voice", Warbots, Pirate Clan Leader) — treat as clean. +- pytest: **181 passed, 3 warnings** (17 in `tests/test_attribution_qa.py` covering tag mismatch, + alias resolution, tag isolation, pronouns, word-overlap fallback, conjunction rejection, auto-fix + gating). `ruff check` + `mypy` clean on `attribution_qa.py`, `qa_attribution.py`, + `model_coordinator.py`, tests. +- **Hardware reality (16 GB Mac)**: gemma-4-12b needs 16.88 GB → cannot fit; qwen3.5-9b fits but + generates ~1 tok/s under swap (~15 h/book) → not viable; gemma-4-e4b is the practical best + (~21 tok/s). LLM and Qwen TTS must not be resident simultaneously (coordinator enforces). + +### Flagged for User Review (ambiguous, left as-is) +| Loc | Line | Currently | Question | +|---|---|---|---| +| ch16 utt121/125 | "The ship landed with a different designation…" / "Yes." | Adonai Maul | The unnamed pirate subordinate failing the Clan Leader (Adonai Maul is a ch27 female bounty — wrong). No role profile; suggest a new "Pirate Commander" | +| ch20 utt223 | "You are taking out multiple air pressure areas…" | Deal Hunter | Ship AI warning — likely already correct | +| ch24 utt108 | "We don't know. Checking." | Deal Hunter | Probably a bridge/crew voice, not Deal Hunter | +| ch26 utt129 | "Captain, it is a real alarm. Warbots just boarded…" | Willamina | Frigate bridge crew reporting to the pirate captain (not Willamina, she's on Deal Hunter); no crew role profile | +| ch26 utt155 | "Four minutes and fifty seconds." | Warbots | Who is "the speaker"? A leader/announcer? | +| ch26 utt158 | "Captain, they have taken the bridge…" | Willamina | Narration: "Status info is radioed to Deal Hunter, who tells Kainda" — could be Deal Hunter the AI, not Willamina | + +### Known Pre-existing Bugs (reported, not fixed) +- `scripts/generate_chapter_audio.py:132` references undefined `phonetic` (NameError on first + utterance; superseded by `generate_all_audio.py`). +- `start_server.sh` launches the Gradio **qwen-tts-demo** on port 8000 — NOT the API server + `scripts/qwen_tts_server.py` (port 8881) the app actually uses. Use `qwen_tts_server.py` for restarts. +- **Orphaned Qwen generations**: a client that times out does NOT cancel the server-side thread; the + model keeps generating under `_model_lock` for minutes, blocking subsequent requests. Recovery = + `kill -9` the qwen server PID and restart `scripts/qwen_tts_server.py`. +- dictionaryapi.dev free tier rate-limits; residual bad records no longer matter for audio + (dictionaryapi records are not applied to speech), would need bulk re-lookup to clean review panel. + +### Audio-Generation Prep (voice resolution + staleness) +- **`umbraaudio/generation/staleness.py`**: markers are now JSON `{"speaker", "text"}` instead of + plain text. `is_current(text, wav, speaker=...)` only returns True when BOTH match. Legacy plain-text + markers are ALWAYS stale → any existing ch1–7 wavs regenerate with corrected attribution on the next + run. All call sites updated (`generate_all_audio.py`, `routes.py` ×3), speaker default normalized to + `"narrator"`. +- **`scripts/generate_all_audio.py`**: `load_voice_references` now uses `rglob` (finds nested refs like + `human_staff_person/messenger_ref.wav`); `resolve_reference` now does case-insensitive canonical + alias lookup, unique first-word match, and longest-substring fallback (was exact/casefold only → + silent narrator fallback). **50/50 distinct speakers now resolve to a voice ref (0 fallback).** +- **New speakers added** (speakers.json + voice_profiles.json + generated refs): `Crew Member` + (ch24 bridge crew, male, clipped urgency; `crew_member_ref.wav`) and `Supervisor` (ch10 female + production supervisor, stressed/assertive; `supervisor_ref.wav`). Both generated via + `scripts/generate_voice_references.py`. +- All 184 pytest tests pass; ruff clean on `staleness.py`, `generate_all_audio.py`, tests. +- **Next**: run `scripts/generate_all_audio.py` (nohup) — Qwen TTS server up (port 8881); all + 4406 utterances will regenerate since legacy markers are stale (~6.4 s/utt, multi-hour run). + +### Remaining Edge Cases +Chapter-5 audio verified end-to-end (files, markers, stitch, ASR). A/B samples: +`/tmp/author_note_before.wav`, `/tmp/author_note_natural.wav`, `/tmp/ch5_combined.wav`. Optional +future tooling: a per-utterance Whisper-based QA transcription pass over `ch*_full.wav`. + +### Previous Session (chapter-5 audio fix) +Complete — ch5 audio (markers, staleness, pronunciation, curly punctuation) verified. History below +this section documents the model-coordinator session and speaker-attribution completion. diff --git a/FixFeedback.txt.rtf b/FixFeedback.txt.rtf new file mode 100644 index 0000000..80ca352 --- /dev/null +++ b/FixFeedback.txt.rtf @@ -0,0 +1,32 @@ +{\rtf1\ansi\ansicpg1252\cocoartf2870 +\cocoatextscaling0\cocoaplatform0{\fonttbl\f0\fswiss\fcharset0 Helvetica;} +{\colortbl;\red255\green255\blue255;\red249\green249\blue249;\red17\green17\blue17;} +{\*\expandedcolortbl;;\cssrgb\c98039\c98039\c98039;\cssrgb\c8627\c8627\c8627;} +\margl1440\margr1440\vieww11520\viewh8400\viewkind0 +\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0 + +\f0\fs24 \cf0 In the umbra audio system we are processing to validate utterances are done correctly. Chapter 6 Salvage\ +Was generated and listened to. Issues became clear that need to be fixed.\ +\ +The system has a proper names section where the correct way to pronounce names is verified. Is that verified pronunciation used with every utterance?\ +\ +\pard\pardeftab720\partightenfactor0 + +\fs26 \cf2 \cb3 \expnd0\expndtw0\kerning0 +Checking the chapter Salvage, utterance 56 isn't correct. It is actually three sentences and should be three utterances with different speakers. \ +\ +Based on context the utterance 145 should be modified to use a pleading tone. \ +\ +Utterances starting at 186 uses the Deal Hunter voice. When the narative clearly talks about a robot. The speaker should be the robot that we know from later in this section (from the context) is Nanny. \ +\ +Utterance 186 follow "She asks", clearly the rescued girl. From the context - Kainda has red hair, This person rescued from the yaht debre has red hair - it is Kainda. The speaker assigned is again deal hunter which is wrong. \ +\ +At utterance 211 suddently shifts to nanny as the speaker - this is correct, but it should have been nanny before. \ +\ +The utterance 218 is "<<<>>>" a section break and should be removed is is unspoken. \ +\ +utterance 221 speaker is nanny and correct. utterance 222 is a different person (Kainda) but the system assigns the speaker as nanny. \ +\ +Utterance 353 clearly has the captain yells, so utterance 354 should be yelling. Utterance 254 is correctly identified as captain willamina wake, but not yelling. \ +\ +These are all examples of the analysis section not working properly. The system needs to be improved. To be clear it is acceptable to have multiple passes to determine different parts of the product. This project is about quality not speed.} \ No newline at end of file diff --git a/Qwen3-TTS b/Qwen3-TTS new file mode 160000 index 0000000..022e286 --- /dev/null +++ b/Qwen3-TTS @@ -0,0 +1 @@ +Subproject commit 022e286b98fbec7e1e916cb940cdf532cd9f488e diff --git a/README.md b/README.md new file mode 100644 index 0000000..ded02d7 --- /dev/null +++ b/README.md @@ -0,0 +1,59 @@ +# UmbraAudio + +Quality-first multi-pass audiobook generation from English manuscripts. + +## Features + +- 5-pass pipeline: Analysis → Voice Design → Mapping → Test → Generate +- Multi-engine TTS: Chatterbox (primary), Qwen3-TTS (secondary) +- LLM-enhanced: BookNLP for NLP, LLM for profiling/emotion/pronunciation +- Web review: FastAPI + HTMX at checkpoints for human approval + +## Quick Start + +```bash +python -m umbraaudio.cli init --name 'My Book' +python -m umbraaudio.cli analyze +python -m umbraaudio.cli voices +python -m umbraaudio.cli review +``` + +## Requirements + +- Python 3.11+ +- NVIDIA GPU with CUDA support (RTX 3060 or better recommended) +- ffmpeg + +## Development + +```bash +pip install -e ".[dev]" +ruff check umbraaudio/ +pytest tests/ -v +``` + +## Rebuilding from source + +The repo tracks source only; large runtime data is reconstructed as needed. + +1. **Models**: weights are not committed (see `.gitignore`). Download them: + ```bash + bash scripts/download_models.sh # ~9 GB, targets ./models + ``` + +2. **Qwen3-TTS** is a git submodule pinned to upstream commit `022e286`: + ```bash + git submodule update --init --recursive + ``` + The `tts_server`/web-demo backend includes a local UI customization to + `qwen_tts/cli/demo.py`. It is preserved as a patch — reapply it after + checking out the submodule: + ```bash + cd Qwen3-TTS && git apply ../scripts/qwen3_tts_demo_umbra_patch.patch + ``` + +3. **Environment**: copy/keep `.env` (contains secrets) and set `UMBRA_TTS__*` + model paths under `models/`. + +Book projects (e.g. `DealHunterShip_...`) are committed separately under their +own project directories and are not part of this program repository. diff --git a/UmbraAudio_Blue_2.png b/UmbraAudio_Blue_2.png new file mode 100755 index 0000000..d21a205 Binary files /dev/null and b/UmbraAudio_Blue_2.png differ diff --git a/UmbraAudio_icon.jpeg b/UmbraAudio_icon.jpeg new file mode 100644 index 0000000..5f2abb2 Binary files /dev/null and b/UmbraAudio_icon.jpeg differ diff --git a/UmbraAudio_icon.png b/UmbraAudio_icon.png new file mode 100644 index 0000000..d21a205 Binary files /dev/null and b/UmbraAudio_icon.png differ diff --git a/Umbra_Voice_generation.md b/Umbra_Voice_generation.md new file mode 100644 index 0000000..4a51f09 --- /dev/null +++ b/Umbra_Voice_generation.md @@ -0,0 +1,307 @@ +# UmbraAudio Voice Generation Specification + +## Overview + +UmbraAudio generates multi-voice audiobooks from English manuscripts. Each character +gets a unique, consistent voice. The voice identity is stored as a **speaker embedding** — +a 2048-dimensional float32 vector extracted from Qwen3-TTS's internal speaker encoder. +This embedding is the single source of truth for a character's voice. + +## Architecture + +### Voice Identity Pipeline + +``` +Edge-TTS WAV ref ──or── VoiceDesign instruct text + │ │ + ▼ ▼ + Base model VoiceDesign model + extract_speaker_ generate_voice_design() + embedding() │ + │ ▼ + ▼ Base model + Speaker embedding extract_speaker_embedding() + (2048-dim float32) │ + │ ▼ + ▼ Speaker embedding + Save as .npy │ + (8KB per speaker) ▼ + Save as .npy +``` + +### Key Insight + +The old approach (WAV ref → extract x-vector at generation time) produced **nearly +identical voices** (MFCC cosine similarity 0.99+) because x-vector extraction is lossy. +The new approach stores the **exact embedding** that the model uses, bypassing extraction +entirely. This gives: + +- **Perfect determinism**: same embedding + same seed = identical waveform +- **Distinct voices**: narrator 275 Hz, nanny 176 Hz, kainda 227 Hz (measured F0) +- **Tiny storage**: 8KB per speaker vs megabytes of WAV refs +- **No drift**: embedding is fixed once saved, never re-extracted + +## File Structure + +``` +.umbra/ +├── voice_embeddings/ # Speaker identity cache +│ ├── narrator.npy # 2048-dim float32 (8KB) +│ ├── nanny.npy +│ ├── kainda_walters_foster_murphy.npy +│ ├── deal_hunter.npy +│ └── ... # 94 speakers total +├── voices/ # Legacy WAV refs (kept for fallback) +│ ├── narrator_ref.wav +│ ├── nanny_ref.wav +│ └── ... +├── speakers.json # Character metadata +├── voice_assignments.json # Edge-TTS voice mapping +├── voice_profiles.json # LLM-generated voice profiles +└── project.db # Utterances, chapters, attributions +``` + +## Generation Modes + +### Mode 1: Cached Embedding (Preferred) + +Fastest and most consistent. Used when a `.npy` embedding exists for the speaker. + +```python +# Client loads cached embedding +emb = np.load("voice_embeddings/narrator.npy") # 2048-dim float32 + +# Sends to server as JSON (8KB payload) +POST /v1/tts + speaker_embedding: "[0.123, -0.456, ...]" # JSON float list + text: "The captain nodded." + temperature: 0.5 + seed: 42 + +# Server passes embedding directly to model +model.generate_voice_clone( + text=text, + voice_clone_prompt={ + "ref_code": [None], + "ref_spk_embedding": [spk_tensor], + "x_vector_only_mode": [True], + "icl_mode": [False], + }, + language="english", + temperature=0.5, +) +``` + +**Determinism**: Same embedding + same seed = **identical waveform** (verified: mean abs diff = 0.000000). + +### Mode 2: WAV Reference (Fallback) + +Used when no cached embedding exists. Extracts x-vector at generation time. + +```python +POST /v1/tts + ref_audio: # 50-200KB + text: "The captain nodded." + temperature: 0.5 +``` + +### Mode 3: VoiceDesign (New Voice Creation) + +Used to create new character voices from text descriptions. + +```python +POST /v1/tts + voice_description: "A deep male voice with Australian accent" + text: "The captain nodded." + temperature: 0.5 +``` + +## Voice Key Generation + +### From Existing WAV Refs + +```bash +python scripts/extract_embeddings.py --project DealHunterShip_4.0e_whole-to-audio +``` + +Loads Base model once, iterates all `*_ref.wav` files, extracts embeddings, saves as `.npy`. +Takes ~6 seconds for 94 speakers. + +### From VoiceDesign Instruct Text + +```bash +python scripts/extract_embeddings.py \ + --design "A young female voice with slight accent, speaking softly" \ + --slug new_character \ + --seed 42 +``` + +Generates audio via VoiceDesign → extracts embedding via Base model → saves as `.npy`. + +### Regenerating All Voice Samples + +```bash +python scripts/extract_embeddings.py --force # Re-extract from WAV refs +``` + +## Rainbow Passage + +All voice references are generated using the Rainbow Passage — a phonetically balanced +paragraph covering all English phonemes. This ensures the speaker embedding captures the +full range of vocal characteristics. + +``` +When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. +The rainbow is a division of white light into many beautiful colors. +These take the shape of a long round arch, with its path high above, +and its two ends apparently beyond the horizon. +There is, according to legend, a boiling pot of gold at one end. +People look, but no one ever finds it. +When a man looks for something beyond his reach, +his friends say he is looking for the pot of gold at the end of the rainbow. +``` + +## Server API + +### POST /v1/tts + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| text | str | required | Text to synthesize | +| sample_rate | int | 24000 | Audio sample rate | +| voice_description | str | None | VoiceDesign instruct text | +| ref_audio | File | None | Voice reference WAV (fallback mode) | +| ref_text | str | None | Reference text for ICL mode | +| speaker_embedding | str | None | JSON-encoded float list (2048-dim) | +| temperature | float | 0.9 | Sampling temperature (lower = more consistent) | +| do_sample | bool | True | Enable sampling | +| seed | int | None | Random seed for reproducibility | + +### POST /load + +Load a model into memory. + +```json +{"model_type": "base"} // or "voicedesign" +``` + +### POST /unload + +Unload a model to free memory. + +```json +{"model_type": "all"} // or "base" or "voicedesign" +``` + +### GET /health + +Returns server status and loaded models. + +## Consistency Guarantees + +### Per-Utterance Consistency + +Each utterance uses the same embedding + same seed → identical voice identity. +Variation comes only from: +- Different text (phoneme sequence) +- Temperature (controlled per-chapter) + +### Cross-Chapter Consistency + +Same speaker embedding used across all chapters → same voice identity. +The narrator always sounds like the narrator. Kainda always sounds like Kainda. + +### Determinism + +With `seed=42` and the same embedding: +- Run 1 and Run 2 produce **identical waveforms** (mean abs diff = 0.000000) +- Different seeds produce different phrasings but same voice identity + +## Chapter Generation Flow + +1. **Preflight**: Verify TTS server is running, Base model loaded +2. **Validate**: Check which utterances need (re)generation +3. **Generate**: For each utterance: + - Load speaker embedding from `.npy` cache + - Send to server with text + temperature + seed + - Save WAV to `.umbra/chapter_audio/ch{N}/utt{M}.wav` +4. **Stitch**: Concatenate utterance WAVs into chapter WAV +5. **Export**: Convert chapter WAV to MP3 via ffmpeg +6. **Verify**: Run quality checks + +## Hardware Constraints + +- **Mac 16 GB MPS**: One model resident at a time (Base OR VoiceDesign) +- **Base model**: ~3.5 GB VRAM, generates ~10s/utterance +- **VoiceDesign model**: ~3.5 GB VRAM, generates ~10s/utterance +- **ICL mode**: ~11 minutes/utterance on MPS (not viable for production) +- **x-vector mode**: ~10s/utterance on MPS (production mode) + +## Model Paths + +| Model | Path | Purpose | +|-------|------|---------| +| Base | `models/Qwen3-TTS-12Hz-1.7B-Base` | Voice cloning + embedding extraction | +| VoiceDesign | `models/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | New voice creation from instruct text | +| Tokenizer | `models/Qwen3-TTS-Tokenizer-12Hz` | Shared tokenizer | + +## Configuration + +### speakers.json + +```json +{ + "speakers": [ + { + "primary_name": "Narrator", + "gender": "male", + "accent": "american", + "voice_status": "approved", + "vocal_traits": { + "pitch": "low", + "energy": "measured", + "timbre": "warm" + } + } + ] +} +``` + +### voice_assignments.json + +Maps character names to Edge-TTS voice identifiers (used for WAV ref generation). + +```json +{ + "Narrator": "en-US-GuyNeural:-10", + "Nanny": "en-US-AvaNeural", + "Kainda Walters Foster Murphy": "en-US-JennyNeural" +} +``` + +## Troubleshooting + +### Voice sounds wrong + +1. Check if embedding exists: `ls .umbra/voice_embeddings/{slug}.npy` +2. Regenerate embedding: `python scripts/extract_embeddings.py --speaker {name} --force` +3. Verify with Rainbow Passage sample + +### TTS server not responding + +```bash +curl http://localhost:8881/health +# If down: restart with qwen_tts_server.py +``` + +### Memory pressure + +```bash +curl -X POST http://localhost:8881/unload -d '{"model_type": "all"}' +``` + +### Inconsistent voices across utterances + +This was the old x-vector extraction problem. Now solved by cached embeddings. +If still occurring, ensure the caller is using cached embeddings (check logs for +"Using cached embedding for ..."). diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..5be0a9e --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,99 @@ +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[project] +name = "umbraaudio" +version = "0.1.0" +description = "Quality-first multi-pass audiobook generation from manuscripts" +readme = "README.md" +license = "MIT" +requires-python = ">=3.11" +authors = [{ name = "UmbraAudio" }] +keywords = ["audiobook", "tts", "text-to-speech", "narration"] +classifiers = [ + "Development Status :: 3 - Alpha", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Topic :: Multimedia :: Sound/Audio :: Speech", +] + +dependencies = [ + # CLI + "click>=8.1", + "rich>=13.0", + + # Database + "aiosqlite>=0.19", + + # Parsers + "ebooklib>=0.18", + "beautifulsoup4>=4.12", + "PyMuPDF>=1.23", + "python-docx>=1.0", + + # NLP / Analysis + "booknlp>=1.0", + + # LLM clients + "openai>=1.10", + "httpx>=0.25", + + # TTS engines + "qwen-tts>=0.1", + "edge-tts>=6.1", + "torch>=2.6", + "torchaudio>=2.6", + + # Audio processing + "librosa>=0.10", + "soundfile>=0.12", + "pydub>=0.25", + + # Quality metrics + "speechmos>=0.0.1", + "openai-whisper>=20231117", + "speechbrain>=0.5", + + # Web review UI + "fastapi>=0.104", + "uvicorn[standard]>=0.24", + "jinja2>=3.1", + "python-multipart>=0.0.6", + + # Utilities + "numpy>=1.24", + "pydantic>=2.5", + "pydantic-settings>=2.1", +] + +[project.optional-dependencies] +dev = [ + "pytest>=7.4", + "pytest-asyncio>=0.21", + "pytest-cov>=4.1", + "ruff>=0.1", + "mypy>=1.7", +] + +[project.scripts] +umbraaudio = "umbraaudio.cli:main" + +[tool.hatch.build.targets.wheel] +packages = ["umbraaudio"] + +[tool.ruff] +target-version = "py311" +line-length = 100 + +[tool.ruff.lint] +select = ["E", "F", "I", "N", "W", "UP", "B", "A", "SIM", "TCH"] + +[tool.mypy] +python_version = "3.11" +strict = true +plugins = ["pydantic.mypy"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +asyncio_mode = "auto" diff --git a/scripts/analyze_emotions.py b/scripts/analyze_emotions.py new file mode 100644 index 0000000..a395dbb --- /dev/null +++ b/scripts/analyze_emotions.py @@ -0,0 +1,118 @@ +#!/usr/bin/env python3 +"""Run LLM emotion/tone analysis on existing utterances without re-segmenting.""" +from __future__ import annotations + +import sqlite3 +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config +from umbraaudio.mapping.emotion import EmotionMapper +from umbraaudio.voices.model_coordinator import ModelCoordinator, PipelineStage + +MODEL = "google/gemma-4-e4b" + + +def main(): + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument( + "--project", + type=str, + default=None, + help="Project name or path", + ) + parser.add_argument( + "--chapters", + type=str, + default=None, + help="Comma-separated chapter IDs (e.g. '5,6,7')", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Print stats without writing to DB", + ) + args = parser.parse_args() + + cfg = load_config(args.project) + coordinator = ModelCoordinator(cfg) + if not coordinator.switch_to( + PipelineStage.CHAPTER_ANALYSIS, lm_model=MODEL + ).get("ok"): + print("Failed to switch models to CHAPTER_ANALYSIS stage") + sys.exit(1) + + from openai import OpenAI + + client = OpenAI(base_url=cfg.llm.lmstudio_base_url, api_key="lm-studio") + mapper = EmotionMapper(llm_client=client, model=MODEL) + + db_path = cfg.db_path + db = sqlite3.connect(str(db_path)) + db.row_factory = sqlite3.Row + + if args.chapters: + chapter_ids = [int(x.strip()) for x in args.chapters.split(",")] + placeholders = ",".join("?" * len(chapter_ids)) + chapters = db.execute( + f"SELECT id, title, chapter_index, source_text FROM chapters " + f"WHERE id IN ({placeholders}) ORDER BY chapter_index", + chapter_ids, + ).fetchall() + else: + chapters = db.execute( + "SELECT id, title, chapter_index, source_text FROM chapters ORDER BY chapter_index" + ).fetchall() + + print(f"Found {len(chapters)} chapters to analyze") + print(f"Model: {MODEL}") + + total = 0 + for ch in chapters: + ch_id = ch["id"] + title = ch["title"] or f"Chapter {ch['chapter_index']}" + + utts = db.execute( + "SELECT id, utterance_index, text, type, speaker_name " + "FROM utterances WHERE chapter_id=? ORDER BY utterance_index", + (ch_id,), + ).fetchall() + if not utts: + print(f" {title}: no utterances, skipping") + continue + + utt_list = [dict(u) for u in utts] + context = (ch["source_text"] or "")[:500] + emotions = mapper.analyze_utterances(utt_list, context) + + if args.dry_run: + print(f" {title} (id={ch_id}): {len(emotions)} emotions returned") + total += len(emotions) + continue + + for u, emo in zip(utt_list, emotions, strict=False): + db.execute( + "UPDATE utterances SET emotion=?, pace=?, instruct=? WHERE id=?", + ( + emo.get("emotion", "neutral"), + emo.get("pace", "moderate"), + emo.get("instruct", ""), + u["id"], + ), + ) + db.commit() + total += len(emotions) + print(f" {title} (id={ch_id}): analyzed {len(emotions)} utterances") + + db.close() + coordinator.switch_to(PipelineStage.IDLE) + print(f"\nTotal: {total} utterances analyzed") + print("Done!") + + +if __name__ == "__main__": + main() diff --git a/scripts/assign_voices.py b/scripts/assign_voices.py new file mode 100644 index 0000000..c2db1ff --- /dev/null +++ b/scripts/assign_voices.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +"""Assign unique Edge-TTS voices to all speakers.""" +from __future__ import annotations + +import hashlib +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config + +# Edge-TTS voices organized by gender and pitch +VOICES = { + "male_low": [ + "en-US-GuyNeural", + "en-GB-ThomasNeural", + "en-NG-AbeoNeural", + ], + "male_mid": [ + "en-US-ChristopherNeural", + "en-US-JasonNeural", + "en-US-TonyNeural", + "en-GB-RyanNeural", + "en-AU-WilliamMultilingualNeural", + "en-IN-PrabhatNeural", + "en-IE-ConnorNeural", + "en-CA-LiamNeural", + "en-KE-ChilembaNeural", + ], + "female_mid": [ + "en-US-JennyNeural", + "en-US-AvaNeural", + "en-US-MichelleNeural", + "en-GB-SoniaNeural", + "en-AU-NatashaNeural", + "en-IN-NeerjaNeural", + "en-IE-EmilyNeural", + "en-CA-ClaraNeural", + "en-NG-EzinneNeural", + "en-KE-AsiliaNeural", + ], + "female_high": [ + "en-US-SaraNeural", + "en-GB-LibbyNeural", + "en-HK-YanNeural", + ], + "child": [ + "en-US-AnaNeural", + ], +} + +# All voices in a flat list for fallback +ALL_VOICES = [] +for voice_list in VOICES.values(): + ALL_VOICES.extend(voice_list) + + +def get_voice_category(speaker: dict) -> str: + """Determine which voice category a speaker should use.""" + demos = speaker.get("demographics", {}) + gender = demos.get("gender", "Unknown").lower() + stype = demos.get("type", "Human").lower() + + # Check for AI/Robot/Computer types + if stype in ("robot", "ai", "computer", "machine", "system"): + return "male_mid" # Robots get a neutral mid voice + + # Check for child + if "child" in stype.lower(): + return "child" + + # Gender-based assignment + if gender == "male": + return "male_mid" + elif gender == "female": + return "female_mid" + else: + # Unknown gender - use mid voices + return "male_mid" + + +def assign_voices(speakers: list[dict]) -> dict[str, str]: + """Assign unique voices to all speakers.""" + voice_assignments = {} + used_voices = set() + + # Track available voices by category + available = {cat: list(voices) for cat, voices in VOICES.items()} + + # Sort speakers by chapter count (more chapters = higher priority) + sorted_speakers = sorted( + speakers, + key=lambda s: len(s.get("chapters_appeared", [])), + reverse=True + ) + + for speaker in sorted_speakers: + name = speaker["primary_name"] + category = get_voice_category(speaker) + + # Try to find an unused voice in the preferred category + assigned = False + if available.get(category): + for voice in available[category]: + if voice not in used_voices: + voice_assignments[name] = voice + used_voices.add(voice) + available[category].remove(voice) + assigned = True + break + + # If no voice in preferred category, try other categories + if not assigned: + for cat, voices in available.items(): + for voice in voices: + if voice not in used_voices: + voice_assignments[name] = voice + used_voices.add(voice) + available[cat].remove(voice) + assigned = True + break + if assigned: + break + + # If still no voice, use a hash-based fallback with pitch shifting + if not assigned: + name_hash = int(hashlib.md5(name.encode()).hexdigest()[:4], 16) + base_voice = ALL_VOICES[name_hash % len(ALL_VOICES)] + # Add pitch shift indicator + pitch_shift = (name_hash % 200) - 100 # -100 to +100 + voice_assignments[name] = f"{base_voice}:{pitch_shift:+d}" + + return voice_assignments + + +def main(): + import argparse + + parser = argparse.ArgumentParser(description="Assign unique Edge-TTS voices to speakers") + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + args = parser.parse_args() + + cfg = load_config(args.project) + + # Load speakers + speakers_path = cfg.data_dir / "speakers.json" + with open(speakers_path) as f: + data = json.load(f) + + speakers = data["speakers"] + print(f"Assigning voices to {len(speakers)} speakers...") + + # Assign voices + assignments = assign_voices(speakers) + + # Save assignments + output_path = cfg.data_dir / "voice_assignments.json" + with open(output_path, "w") as f: + json.dump(assignments, f, indent=2) + + print(f"\nVoice assignments saved to {output_path}") + + # Print summary + print("\n=== Voice Assignments ===") + for name, voice in sorted(assignments.items()): + print(f" {name:35} -> {voice}") + + # Count unique base voices + base_voices = set() + for voice in assignments.values(): + if ":" in voice: + base_voices.add(voice.split(":")[0]) + else: + base_voices.add(voice) + + print("\n=== Summary ===") + print(f"Total speakers: {len(speakers)}") + print(f"Unique base voices: {len(base_voices)}") + print(f"Speakers with pitch shift: {sum(1 for v in assignments.values() if ':' in v)}") + + +if __name__ == "__main__": + main() diff --git a/scripts/download_models.sh b/scripts/download_models.sh new file mode 100644 index 0000000..3916a60 --- /dev/null +++ b/scripts/download_models.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Downloads the Qwen3-TTS model weights required by UmbraAudio. +# +# These models are stock HuggingFace downloads (~9 GB). They are NOT tracked +# in git; run this script once after cloning to reconstruct the models/ tree. +# +# Usage: +# bash scripts/download_models.sh [models_dir] +# +# Defaults to ./models relative to the UmbraAudio repo root. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +MODELS_DIR="${1:-$REPO_ROOT/models}" + +mkdir -p "$MODELS_DIR" + +# Qwen3-TTS models used by the configured engines (see umbraaudio/config.py and .env). +huggingface-cli download "Qwen/Qwen3-TTS-12Hz-0.6B-Base" --local-dir "$MODELS_DIR/Qwen3-TTS-12Hz-0.6B-Base" +huggingface-cli download "Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice" --local-dir "$MODELS_DIR/Qwen3-TTS-12Hz-0.6B-CustomVoice" +huggingface-cli download "Qwen/Qwen3-TTS-12Hz-1.7B-Base" --local-dir "$MODELS_DIR/Qwen3-TTS-12Hz-1.7B-Base" +huggingface-cli download "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice" --local-dir "$MODELS_DIR/Qwen3-TTS-12Hz-1.7B-CustomVoice" +huggingface-cli download "Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign" --local-dir "$MODELS_DIR/Qwen3-TTS-12Hz-1.7B-VoiceDesign" +huggingface-cli download "Qwen/Qwen3-TTS-Tokenizer-12Hz" --local-dir "$MODELS_DIR/Qwen3-TTS-Tokenizer-12Hz" + +echo "Models downloaded to $MODELS_DIR" diff --git a/scripts/extract_embeddings.py b/scripts/extract_embeddings.py new file mode 100644 index 0000000..02ac3a5 --- /dev/null +++ b/scripts/extract_embeddings.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +"""Extract Qwen TTS speaker embeddings from existing voice reference WAVs. + +Usage: + python scripts/extract_embeddings.py + python scripts/extract_embeddings.py --speaker narrator + python scripts/extract_embeddings.py --force # re-extract all + python scripts/extract_embeddings.py --design "A deep male voice" --slug new_speaker + +Loads the Base model once, iterates over voice ref WAVs in .umbra/voices/, +extracts the 2048-dim speaker embedding via Qwen TTS's internal encoder, +and saves each as a .npy file in .umbra/voice_embeddings/. +""" +from __future__ import annotations + +import argparse +import os +import sys +import time +from pathlib import Path + +import numpy as np +import soundfile as sf +import torch + +project_root = Path(__file__).resolve().parent.parent +if str(project_root) not in sys.path: + sys.path.insert(0, str(project_root)) + +from umbraaudio.config import load_config # noqa: E402 +from umbraaudio.voices.qwen_embeddings import QwenEmbeddingCache # noqa: E402 + +BASE_MODEL_PATH = os.getenv( + "QWEN_BASE_MODEL", + "/Volumes/1TB_APL_Storage/UmbraAudio/models/Qwen3-TTS-12Hz-1.7B-Base", +) +VOICEDESIGN_MODEL_PATH = os.getenv( + "QWEN_VOICEDESIGN_MODEL", + "/Volumes/1TB_APL_Storage/UmbraAudio/models/Qwen3-TTS-12Hz-1.7B-VoiceDesign", +) + + +def extract_all_embeddings( + project_name: str, + speaker_filter: str | None = None, + force: bool = False, +) -> None: + config = load_config(project_name) + voices_dir = config.data_dir / "voices" + embeddings_dir = config.data_dir / "voice_embeddings" + + if not voices_dir.is_dir(): + print(f"ERROR: No voices directory at {voices_dir}") + return + + cache = QwenEmbeddingCache(embeddings_dir) + wav_files = sorted(voices_dir.glob("*_ref.wav")) + print(f"Found {len(wav_files)} voice ref WAVs in {voices_dir}") + + if speaker_filter: + wav_files = [w for w in wav_files if speaker_filter.lower() in w.stem.lower()] + print(f"Filtered to {len(wav_files)} WAVs matching '{speaker_filter}'") + + to_extract = [] + for wav_path in wav_files: + slug = wav_path.stem.replace("_ref", "") + if force or not cache.has(slug): + to_extract.append((slug, wav_path)) + + if not to_extract: + print("All embeddings already cached. Use --force to re-extract.") + return + + print(f"Will extract {len(to_extract)} embeddings") + + print("Loading Base model...") + from qwen_tts import Qwen3TTSModel + + model = Qwen3TTSModel.from_pretrained( + BASE_MODEL_PATH, + device_map="mps", + dtype=torch.bfloat16, + ) + print("Base model loaded") + + extracted = 0 + failed = 0 + t0 = time.time() + + for slug, wav_path in to_extract: + try: + audio, sr = sf.read(str(wav_path), dtype="float32") + if sr != 24000: + print(f" SKIP {slug}: sample rate {sr} != 24000") + failed += 1 + continue + + emb = cache.extract_and_save( + slug, audio, sr, model.model + ) + print( + f" OK {slug}: shape={emb.shape}, " + f"norm={np.linalg.norm(emb):.2f}" + ) + extracted += 1 + except Exception as e: + print(f" FAIL {slug}: {e}") + failed += 1 + + elapsed = time.time() - t0 + print(f"\nDone: {extracted} extracted, {failed} failed, {elapsed:.1f}s") + print(f"Cached embeddings: {cache.count()} total in {embeddings_dir}") + + +def create_embedding_from_design( + project_name: str, + instruct: str, + slug: str, + seed: int = 42, + force: bool = False, +) -> None: + """Generate a voice via VoiceDesign, extract its embedding, and cache it.""" + config = load_config(project_name) + embeddings_dir = config.data_dir / "voice_embeddings" + cache = QwenEmbeddingCache(embeddings_dir) + + if cache.has(slug) and not force: + print(f"Embedding for '{slug}' already exists. Use --force to regenerate.") + return + + print("Loading VoiceDesign model...") + from qwen_tts import Qwen3TTSModel + + vd_model = Qwen3TTSModel.from_pretrained( + VOICEDESIGN_MODEL_PATH, + device_map="mps", + dtype=torch.bfloat16, + ) + print("VoiceDesign model loaded") + + test_text = "When the sunlight strikes raindrops in the air, they act as a prism." + print(f"Generating voice with instruct: {instruct[:80]}...") + wavs, sr = vd_model.generate_voice_design( + text=test_text, + instruct=instruct, + language="english", + temperature=0.5, + do_sample=True, + seed=seed, + ) + audio = wavs[0] + if isinstance(audio, list): + audio = audio[0] + audio_np = np.array(audio) + print(f"Generated {len(audio_np)/sr:.2f}s of audio") + + del vd_model + torch.mps.empty_cache() + + print("Loading Base model for embedding extraction...") + base_model = Qwen3TTSModel.from_pretrained( + BASE_MODEL_PATH, + device_map="mps", + dtype=torch.bfloat16, + ) + print("Base model loaded") + + emb = cache.extract_and_save(slug, audio_np, sr, base_model.model) + print(f"Saved embedding for '{slug}': shape={emb.shape}, norm={np.linalg.norm(emb):.2f}") + + wav_path = embeddings_dir / f"{slug}_design_ref.wav" + sf.write(str(wav_path), audio_np, sr) + print(f"Saved reference audio to {wav_path}") + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Extract Qwen TTS speaker embeddings from voice refs" + ) + parser.add_argument( + "--project", + default="DealHunterShip_4.0e_whole-to-audio", + help="Project name (default: DealHunterShip_4.0e_whole-to-audio)", + ) + parser.add_argument("--speaker", help="Extract only this speaker (substring match)") + parser.add_argument("--force", action="store_true", help="Re-extract even if cached") + parser.add_argument("--design", help="VoiceDesign instruct text (create new voice)") + parser.add_argument("--slug", help="Speaker slug for --design mode") + parser.add_argument("--seed", type=int, default=42, help="Random seed for --design") + args = parser.parse_args() + + if args.design: + if not args.slug: + parser.error("--slug is required with --design") + create_embedding_from_design(args.project, args.design, args.slug, args.seed, args.force) + else: + extract_all_embeddings(args.project, args.speaker, args.force) + + +if __name__ == "__main__": + main() diff --git a/scripts/fix_chapter6.py b/scripts/fix_chapter6.py new file mode 100644 index 0000000..f8cc3e1 --- /dev/null +++ b/scripts/fix_chapter6.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +"""Fix Chapter 6 (Salvage) utterance issues: +1. Split utterance 56 into 3 utterances with correct speakers +2. Fix utterance 145 - Kainda with pleading tone +3. Fix utterances 181-190 - Nanny/Kainda not Deal Hunter +4. Fix utterance 222 - Kainda not Nanny +5. Remove utterance 218 (scene break <<<>>>) +6. Fix utterance 354 - Willamina Wake with yelling tone +""" + +from __future__ import annotations + +import argparse +import sqlite3 +import sys +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config + +CHAPTER_ID = 6 + + +def get_utterances(conn: sqlite3.Connection) -> list[dict[str, Any]]: + rows = conn.execute( + "SELECT id, utterance_index, text, type, speaker_name, emotion, pace, instruct " + "FROM utterances WHERE chapter_id=? ORDER BY utterance_index", + (CHAPTER_ID,), + ).fetchall() + return [dict(r) for r in rows] + + +def print_utterances(utts: list[dict[str, Any]], start: int = 0, end: int | None = None) -> None: + for u in utts[start:end]: + print(f" utt{u['utterance_index']:03d} [{u['type']}] {u['speaker_name']}: {u['text'][:80]}...") + + +def fix_chapter6(db_path: Path) -> None: + conn = sqlite3.connect(str(db_path)) + conn.row_factory = sqlite3.Row + + print("=== Current Chapter 6 utterances (key range) ===") + utts = get_utterances(conn) + print_utterances(utts, 50, 70) + print_utterances(utts, 140, 160) + print_utterances(utts, 180, 200) + print_utterances(utts, 210, 230) + print_utterances(utts, 350, 360) + + print("\n=== Applying fixes ===") + + # Fix 1: Split utterance 56 into 3 utterances + # Original utt 56: "The captain is changing course, and that was a missile he is firing at the pirates. + # As Niñera helps Kainda, they feel the impacts more, and the air pressure alarm sounds. + # The computer voice announces, 'Hull breach. Please put on emergency space suits.'" + # Should be: + # - utt 56: "The captain is changing course, and that was a missile he is firing at the pirates." -> Nanny + # - utt 57: "As Niñera helps Kainda, they feel the impacts more, and the air pressure alarm sounds." -> narration + # - utt 58: "The computer voice announces, 'Hull breach. Please put on emergency space suits.'" -> Deal Hunter (AI) + + print("\n1. Splitting utterance 56...") + utt56 = conn.execute( + "SELECT id, text FROM utterances WHERE chapter_id=? AND utterance_index=56", + (CHAPTER_ID,), + ).fetchone() + if utt56: + original_text = utt56["text"] + print(f" Original: {original_text[:100]}...") + + # Split into three parts + part1 = "The captain is changing course, and that was a missile he is firing at the pirates." + part2 = "As Niñera helps Kainda, they feel the impacts more, and the air pressure alarm sounds." + part3 = 'The computer voice announces, "Hull breach. Please put on emergency space suits."' + + # Update utt 56 to be part1 (Nanny) + conn.execute( + "UPDATE utterances SET text=?, speaker_name=?, type=? WHERE id=?", + (part1, "Nanny", "dialogue", utt56["id"]), + ) + + # Insert part2 as narration at index 57 + conn.execute( + "INSERT INTO utterances (chapter_id, utterance_index, text, type, speaker_name, status) " + "VALUES (?, 57, ?, 'narration', 'narrator', 'pending')", + (CHAPTER_ID, part2), + ) + + # Insert part3 as dialogue (Deal Hunter - AI voice) at index 58 + conn.execute( + "INSERT INTO utterances (chapter_id, utterance_index, text, type, speaker_name, status) " + "VALUES (?, 58, ?, 'dialogue', 'Deal Hunter', 'pending')", + (CHAPTER_ID, part3), + ) + + # Shift all utterances >= 57 up by 2 + conn.execute( + "UPDATE utterances SET utterance_index = utterance_index + 2 " + "WHERE chapter_id=? AND utterance_index >= 57", + (CHAPTER_ID,), + ) + print(" Split into 3 utterances: 56 (Nanny), 57 (narration), 58 (Deal Hunter)") + + # Fix 2: Utterance 145 -> now at index 147 after shift, Kainda with pleading tone + print("\n2. Fixing utterance 145 (now ~147) - Kainda pleading...") + conn.execute( + "UPDATE utterances SET speaker_name=?, emotion=?, instruct=? " + "WHERE chapter_id=? AND text LIKE '%Help me, please.%' " + "AND utterance_index BETWEEN 145 AND 150", + ("Kainda Walters Foster Murphy", "pleading", "Speak with desperate pleading. Urgent, emotional.", + CHAPTER_ID), + ) + + # Fix 3: Utterances 181-190 (now shifted by 2) - Nanny/Kainda not Deal Hunter + print("\n3. Fixing utterances 181-190 (now ~183-192) - Nanny/Kainda...") + + # utt 181: "Hello." -> Nanny + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE chapter_id=? AND text='Hello.' AND utterance_index BETWEEN 180 AND 195", + ("Nanny", CHAPTER_ID), + ) + + # utt 186: "What ship is this?" -> Kainda + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE chapter_id=? AND text='What ship is this?' AND utterance_index BETWEEN 180 AND 195", + ("Kainda Walters Foster Murphy", CHAPTER_ID), + ) + + # utt 188: "You are on the Deal Hunter." -> Nanny + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE chapter_id=? AND text='You are on the Deal Hunter.' AND utterance_index BETWEEN 180 AND 195", + ("Nanny", CHAPTER_ID), + ) + + # utt 189: "Is there a rescue ship in the area looking for me?" -> Kainda + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE chapter_id=? AND text LIKE '%rescue ship%' AND utterance_index BETWEEN 180 AND 195", + ("Kainda Walters Foster Murphy", CHAPTER_ID), + ) + + # utt 190: "There are no rescue ships..." -> Nanny + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE chapter_id=? AND text LIKE '%no rescue ships%' AND utterance_index BETWEEN 180 AND 195", + ("Nanny", CHAPTER_ID), + ) + + # Fix 4: Utterance 211 (now ~213) - already Nanny, good. But 181-210 should be Nanny + print("\n4. Ensuring Nanny consistency from 181-210...") + # The robot says at 211: "That was a missile trying to disable our engines. Hold on." - already Nanny + # The robot says at 181 (Hello), 188 (You are on...), 190 (There are no rescue...), 211 (missile) + # Also need to fix 183-184, 186-187 if they are Deal Hunter + conn.execute( + "UPDATE utterances SET speaker_name='Nanny' WHERE chapter_id=? AND utterance_index BETWEEN 181 AND 210 " + "AND speaker_name='Deal Hunter' AND type='dialogue'", + (CHAPTER_ID,), + ) + + # Fix 5: Remove utterance 218 (scene break <<<>>>) - now at ~220 + print("\n5. Removing scene break utterance (<<<>>>)...") + conn.execute( + "DELETE FROM utterances WHERE chapter_id=? AND text='<<<>>>' AND type='scene_break'", + (CHAPTER_ID,), + ) + + # Fix 6: Utterance 222 (now shifted) - should be Kainda not Nanny + print("\n6. Fixing utterance 222 - Kainda not Nanny...") + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE chapter_id=? AND text LIKE '%Why did you accelerate%'", + ("Kainda Walters Foster Murphy", CHAPTER_ID), + ) + + # Fix 7: Utterance 354 (now shifted) - Willamina Wake with yelling + print("\n7. Fixing utterance 354 - Willamina Wake yelling...") + conn.execute( + "UPDATE utterances SET speaker_name=?, emotion=?, instruct=? " + "WHERE chapter_id=? AND text LIKE '%Get the princess on board!%'", + ("Willamina Wake", "yelling", "Speak with authority and command. Firm, unwavering tone. LOUD.", + CHAPTER_ID), + ) + + # Also fix the "The Doctor" utterances + print("\n8. Fixing 'The Doctor' utterances...") + conn.execute( + "UPDATE utterances SET speaker_name='The Doctor' WHERE chapter_id=? AND text LIKE '%robots are all wiped%'", + (CHAPTER_ID,), + ) + + conn.commit() + + print("\n=== Verifying fixes ===") + utts = get_utterances(conn) + print_utterances(utts, 50, 70) + print_utterances(utts, 145, 160) + print_utterances(utts, 180, 200) + print_utterances(utts, 210, 230) + print_utterances(utts, 350, 365) + + conn.close() + print("\n=== Done ===") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Fix Chapter 6 (Salvage) utterance issues") + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + args = parser.parse_args() + cfg = load_config(args.project) + fix_chapter6(cfg.db_path) diff --git a/scripts/generate_all_audio.py b/scripts/generate_all_audio.py new file mode 100644 index 0000000..5230cd9 --- /dev/null +++ b/scripts/generate_all_audio.py @@ -0,0 +1,429 @@ +#!/usr/bin/env python3 +"""Background chapter audio generation with Qwen3-TTS. +Run as: nohup python scripts/generate_all_audio.py [--project NAME] > generation.log 2>&1 & +""" + +from __future__ import annotations + +import json +import logging +import sqlite3 +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config +from umbraaudio.generation.audio_effects import ( + apply_effects_to_file, + apply_effects_to_file_by_name, + load_speaker_distortion_effects, + load_speaker_traits, +) +from umbraaudio.generation.staleness import is_current, write_marker +from umbraaudio.pronunciation import apply_phonetics, load_phonetic_names +from umbraaudio.voices.lmstudio_lifecycle import get_vram_usage_mb +from umbraaudio.voices.qwen_tts import VOICE_REFERENCE_TEXT, QwenTTSEngine + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(message)s", + handlers=[ + logging.StreamHandler(), + ], +) +logger = logging.getLogger(__name__) + + +def load_voice_references(voices_dir: Path) -> dict[str, Path]: + refs: dict[str, Path] = {} + if not voices_dir.exists(): + return refs + for f in voices_dir.rglob("*_ref.wav"): + name = f.stem.replace("_ref", "").replace("_", " ") + if name not in refs: + refs[name] = f + return refs + + +def load_speakers(data_dir: Path) -> dict[str, dict]: + path = data_dir / "speakers.json" + if not path.exists(): + return {} + with open(path) as f: + data = json.load(f) + return {s["primary_name"]: s for s in data.get("speakers", [])} + + +def get_speaker_name_map(speakers: dict[str, dict]) -> dict[str, str]: + alias_map: dict[str, str] = {} + for name, info in speakers.items(): + alias_map[name.lower()] = name + for alias in info.get("aliases", []): + if alias: + alias_map[alias.lower()] = name + return alias_map + + +def resolve_reference( + speaker_name: str, + voice_refs: dict[str, Path], + alias_map: dict[str, str], + narrator_ref: Path | None = None, +) -> Path | None: + if speaker_name in voice_refs: + return voice_refs[speaker_name] + low = speaker_name.lower() + for name, path in voice_refs.items(): + if name.lower() == low: + return path + canonical = alias_map.get(low) + if canonical: + for name, path in voice_refs.items(): + if name.lower() == canonical.lower(): + return path + first_word = low.split()[0] if low.split() else "" + if first_word: + matches = [p for n, p in voice_refs.items() if n.lower().split()[:1] == [first_word]] + if len(matches) == 1: + return matches[0] + best = None + best_len = -1 + for name, path in voice_refs.items(): + nlow = name.lower() + if (nlow in low or low in nlow) and len(nlow) > best_len: + best = path + best_len = len(nlow) + if best is not None: + return best + return narrator_ref + + +def generate_chapter( + chapter_id: int, + engine: QwenTTSEngine, + voice_refs: dict[str, Path], + alias_map: dict[str, str], + output_base: Path, + db_path: Path, + narrator_ref: Path | None = None, + phonetic: dict[str, str] | None = None, + speaker_traits: dict[str, list[str]] | None = None, + distortion_effects: dict[str, list[str]] | None = None, + utterance_callback=None, + progress_callback=None, + progress_context=None, +) -> tuple[int, int]: + db = sqlite3.connect(str(db_path)) + db.row_factory = sqlite3.Row + + chapter = db.execute("SELECT * FROM chapters WHERE id = ?", (chapter_id,)).fetchone() + if not chapter: + return 0, 0 + + title = chapter["title"] or f"Chapter {chapter['chapter_index'] + 1}" + + utterances = db.execute( + "SELECT id, speaker_name, text, utterance_index FROM utterances " + "WHERE chapter_id = ? ORDER BY utterance_index", + (chapter_id,), + ).fetchall() + db.close() + + chapter_dir = output_base / f"ch{chapter_id}" + chapter_dir.mkdir(parents=True, exist_ok=True) + + total = len(utterances) + skipped = 0 + generated = 0 + failed = 0 + + for utt in utterances: + speaker = utt["speaker_name"] or "narrator" + text = utt["text"] + idx = utt["utterance_index"] + + if not text or not text.strip(): + skipped += 1 + if utterance_callback: + utterance_callback() + continue + + if text.strip() == "<<<>>>": + skipped += 1 + if utterance_callback: + utterance_callback() + continue + + output_path = chapter_dir / f"utt_{idx:04d}.wav" + + ref_path = resolve_reference(speaker, voice_refs, alias_map, narrator_ref) + + spoken_text = apply_phonetics(text, phonetic or {}) + + if is_current(spoken_text, output_path, speaker=speaker, voice_ref=ref_path): + skipped += 1 + if utterance_callback: + utterance_callback() + continue + + utt_success = False + try: + if ref_path: + engine.generate( + text=spoken_text, + output_path=output_path, + voice_ref=ref_path, + ref_text=VOICE_REFERENCE_TEXT, + ) + else: + engine.generate( + text=spoken_text, + output_path=output_path, + instruct="A calm, neutral voice speaking clearly.", + ) + + if output_path.exists(): + explicit = (distortion_effects or {}).get(speaker, []) + if explicit: + apply_effects_to_file_by_name(output_path, explicit) + elif speaker_traits: + traits = speaker_traits.get(speaker, []) + if traits: + apply_effects_to_file(output_path, traits) + write_marker(spoken_text, output_path, speaker=speaker, voice_ref=ref_path) + generated += 1 + utt_success = True + else: + failed += 1 + + except Exception as e: + logger.error(" FAILED utt_%04d (%s): %s", idx, speaker, e) + failed += 1 + + if utterance_callback: + utterance_callback(success=utt_success) + + if progress_callback and progress_context: + progress_callback(progress_context["completed"], progress_context["total"], title) + + if (generated + failed) % 50 == 0 and generated > 0: + elapsed_min = (generated + failed) * 6.4 / 60 + logger.info( + " Progress: %d/%d done (%d gen, %d fail), ETA %.0f min", + generated + failed, + total, + generated, + failed, + elapsed_min, + ) + + return generated, failed + + +def main(status_callback=None, gen_manager=None, project: str | None = None): + cfg = load_config(project) + data_dir = cfg.data_dir + + file_handler = logging.FileHandler(data_dir / "generation.log") + file_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s")) + logger.addHandler(file_handler) + + voices_dir = cfg.voices_dir + output_base = data_dir / "chapter_audio" + output_base.mkdir(parents=True, exist_ok=True) + + voice_refs = load_voice_references(voices_dir) + speakers = load_speakers(data_dir) + alias_map = get_speaker_name_map(speakers) + phonetic = load_phonetic_names(data_dir) + speaker_traits = load_speaker_traits(data_dir / "speakers.json") + distortion_effects = load_speaker_distortion_effects( + data_dir / "speakers.json" + ) + + narrator_ref = None + for name, path in voice_refs.items(): + if name.lower() == "narrator": + narrator_ref = path + break + if not narrator_ref: + for name, path in voice_refs.items(): + if name.lower() == "kainda": + narrator_ref = path + break + + logger.info("=== Qwen3-TTS Background Generation ===") + logger.info("Voice references: %d", len(voice_refs)) + logger.info("Speakers: %d", len(speakers)) + logger.info("Confirmed phonetic overrides: %d", len(phonetic)) + + from umbraaudio.voices.model_coordinator import ModelCoordinator, PipelineStage + + coordinator = ModelCoordinator(cfg) + if not coordinator.switch_to(PipelineStage.UTTERANCE_GENERATION).get("ok"): + logger.error("Failed to switch models to UTTERANCE_GENERATION stage") + sys.exit(1) + + vram = get_vram_usage_mb() + logger.info("VRAM: %dMB used, %dMB free", vram["used_mb"], vram["free_mb"]) + + db = sqlite3.connect(str(cfg.db_path)) + db.row_factory = sqlite3.Row + chapters = db.execute( + "SELECT id, title, chapter_index FROM chapters WHERE project_id=1 ORDER BY chapter_index" + ).fetchall() + total_utterances = db.execute("SELECT COUNT(*) as cnt FROM utterances").fetchone()["cnt"] + db.close() + + total_chapters = len(chapters) + + engine = QwenTTSEngine(device="cuda", sample_rate=24000, qwen_tts_url=cfg.tts.qwen_tts_url) + + engine.load_remote_model(model_type="base") + + total_gen = 0 + total_fail = 0 + completed_utterances = 0 + start = time.time() + + if status_callback: + status_callback( + "generate", + "running", + "Starting audio generation...", + 0, + total_chapters=total_chapters, + chapters_completed=[], + ) + + def _utterance_done(success=True): + nonlocal completed_utterances + completed_utterances += 1 + if gen_manager is not None: + gen_manager.update_item(f"utt:{completed_utterances}", success=success) + + def _progress_tick(completed, total, ch_title): + if status_callback: + progress_pct = int((completed_utterances / max(total_utterances, 1)) * 100) + status_callback( + "generate", + "running", + f"Chapter {i + 1}/{total_chapters}: {ch_title}", + progress_pct, + total_chapters=total_chapters, + chapters_completed=list(range(i)), + current_chapter=chapters[i]["id"], + completed_utterances=completed_utterances, + total_utterances=total_utterances, + ) + + batch_total = total_utterances + _gen_ctx = None + if gen_manager is not None: + _gen_ctx = gen_manager.serialize("audio", batch_total, "audio:generate") + _gen_ctx.__enter__() + + try: + for i, ch in enumerate(chapters): + ch_id = ch["id"] + ch_title = ch["title"] or f"Ch{ch['chapter_index'] + 1}" + + if status_callback: + progress_pct = int((completed_utterances / max(total_utterances, 1)) * 100) + status_callback( + "generate", + "running", + f"Chapter {i + 1}/{total_chapters}: {ch_title}", + progress_pct, + total_chapters=total_chapters, + chapters_completed=list(range(i)), + current_chapter=ch_id, + completed_utterances=completed_utterances, + total_utterances=total_utterances, + ) + + progress_ctx = {"completed": completed_utterances, "total": total_utterances} + gen, fail = generate_chapter( + ch_id, + engine, + voice_refs, + alias_map, + output_base, + cfg.db_path, + narrator_ref, + phonetic=phonetic, + speaker_traits=speaker_traits, + distortion_effects=distortion_effects, + utterance_callback=_utterance_done, + progress_callback=_progress_tick, + progress_context=progress_ctx, + ) + completed_utterances = progress_ctx["completed"] + total_gen += gen + total_fail += fail + + ch_dir = output_base / f"ch{ch_id}" + full_wav = output_base / f"ch{ch_id}_full.wav" + utt_files = sorted(ch_dir.glob("utt_*.wav")) if ch_dir.exists() else [] + needs_stitch = True + if full_wav.exists() and utt_files: + newest_utt = max(f.stat().st_mtime for f in utt_files) + needs_stitch = full_wav.stat().st_mtime < newest_utt + if utt_files and needs_stitch: + try: + from umbraaudio.assembly import AudioStitcher + + stitcher = AudioStitcher() + stitcher.stitch_utterances(utt_files, full_wav) + n_files = len(utt_files) + logger.info(" Stitched ch%d -> %s (%d files)", ch_id, full_wav.name, n_files) + except Exception as e: + logger.error(" Stitch failed for ch%d: %s", ch_id, e) + + elapsed = (time.time() - start) / 60 + logger.info( + "Ch %d (%s): +%d gen, +%d fail | Total: %d gen, %d fail | %.1f min elapsed", + ch_id, + ch_title, + gen, + fail, + total_gen, + total_fail, + elapsed, + ) + finally: + if _gen_ctx is not None: + _gen_ctx.__exit__(None, None, None) + + engine.unload_all() + coordinator.switch_to(PipelineStage.IDLE) + elapsed = (time.time() - start) / 60 + logger.info( + "=== COMPLETE: %d generated, %d failed, %.1f min total ===", + total_gen, + total_fail, + elapsed, + ) + + if status_callback: + status_callback( + "generate", + "completed", + f"Done: {total_gen} generated, {total_fail} failed in {elapsed:.1f} min", + 100, + total_chapters=total_chapters, + chapters_completed=list(range(total_chapters)), + ) + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description="Background chapter audio generation") + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + args = parser.parse_args() + main(project=args.project) diff --git a/scripts/generate_chapter_audio.py b/scripts/generate_chapter_audio.py new file mode 100644 index 0000000..5f765f4 --- /dev/null +++ b/scripts/generate_chapter_audio.py @@ -0,0 +1,258 @@ +#!/usr/bin/env python3 +"""Generate audio for chapters using Qwen3-TTS with voice cloning from references.""" +from __future__ import annotations + +import json +import logging +import sqlite3 +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config +from umbraaudio.pronunciation import load_phonetic_names +from umbraaudio.voices.lmstudio_lifecycle import get_loaded_model, get_vram_usage_mb, is_running +from umbraaudio.voices.model_coordinator import ModelCoordinator, PipelineStage +from umbraaudio.voices.qwen_tts import VOICE_REFERENCE_TEXT, QwenTTSEngine + +logger = logging.getLogger(__name__) + + +def load_voice_references(voices_dir: Path) -> dict[str, Path]: + """Load voice reference files, mapping character name -> ref audio path.""" + refs: dict[str, Path] = {} + if not voices_dir.exists(): + return refs + for f in voices_dir.glob("*_ref.wav"): + name = f.stem.replace("_ref", "").replace("_", " ") + refs[name] = f + return refs + + +def load_speakers(data_dir: Path) -> dict[str, dict]: + """Load speakers from JSON file.""" + path = data_dir / "speakers.json" + if not path.exists(): + return {} + with open(path) as f: + data = json.load(f) + return {s["primary_name"]: s for s in data.get("speakers", [])} + + +def get_speaker_name_map(speakers: dict[str, dict]) -> dict[str, str]: + """Build a mapping from all aliases to primary name.""" + alias_map: dict[str, str] = {} + for name, info in speakers.items(): + alias_map[name.lower()] = name + for alias in info.get("aliases", []): + alias_map[alias.lower()] = name + return alias_map + + +def _apply_phonetics(text: str, phonetic: dict[str, str]) -> str: + if not phonetic: + return text + for name, phonetic_spelling in sorted(phonetic.items(), key=lambda x: -len(x[0])): + text = text.replace(name, phonetic_spelling) + return text + + +def resolve_reference( + speaker_name: str, + voice_refs: dict[str, Path], + alias_map: dict[str, str], + narrator_ref: Path | None = None, +) -> Path | None: + """Find the voice reference file for a speaker.""" + if speaker_name in voice_refs: + return voice_refs[speaker_name] + for name, path in voice_refs.items(): + if name.lower() == speaker_name.lower(): + return path + canonical = alias_map.get(speaker_name.lower()) + if canonical and canonical in voice_refs: + return voice_refs[canonical] + return narrator_ref + + +def generate_chapter_audio( + chapter_id: int, + engine: QwenTTSEngine, + voice_refs: dict[str, Path], + alias_map: dict[str, str], + output_base: Path, + db_path: Path, + narrator_ref: Path | None = None, +) -> int: + """Generate audio for all utterances in a chapter using voice cloning.""" + db = sqlite3.connect(str(db_path)) + db.row_factory = sqlite3.Row + + chapter = db.execute("SELECT * FROM chapters WHERE id = ?", (chapter_id,)).fetchone() + if not chapter: + logger.error("Chapter %d not found", chapter_id) + return 0 + + title = chapter["title"] or f"Chapter {chapter['chapter_index']+1}" + logger.info("Generating audio for: %s", title) + + utterances = db.execute( + """ + SELECT u.id, u.speaker_name, u.text, u.utterance_index + FROM utterances u + WHERE u.chapter_id = ? + ORDER BY u.utterance_index + """, + (chapter_id,), + ).fetchall() + logger.info("Found %d utterances", len(utterances)) + + chapter_dir = output_base / f"ch{chapter_id}" + chapter_dir.mkdir(parents=True, exist_ok=True) + + generated = 0 + for utt in utterances: + utt_id = utt["id"] + speaker = utt["speaker_name"] or "narrator" + text = utt["text"] + idx = utt["utterance_index"] + + if not text or not text.strip(): + continue + + ref_path = resolve_reference(speaker, voice_refs, alias_map, narrator_ref) + + filename = f"utt_{idx:04d}.wav" + output_path = chapter_dir / filename + + if output_path.exists(): + generated += 1 + continue + + spoken_text = _apply_phonetics(text, phonetic) + + try: + if ref_path: + engine.generate( + text=spoken_text, + output_path=output_path, + voice_ref=ref_path, + ref_text=VOICE_REFERENCE_TEXT, + ) + else: + engine.generate( + text=text, + output_path=output_path, + instruct="A calm, neutral voice speaking clearly.", + ) + + if output_path.exists(): + generated += 1 + if generated % 20 == 0: + logger.info(" Generated %d/%d utterances", generated, len(utterances)) + else: + logger.warning(" Failed: utt_%04d (%s)", idx, speaker) + + except Exception as e: + logger.error(" Error: utt_%04d (%s): %s", idx, speaker, e) + + logger.info("Generated %d/%d utterances for %s", generated, len(utterances), title) + db.close() + return generated + + +def main(): + import argparse + + parser = argparse.ArgumentParser(description="Generate chapter audio with Qwen3-TTS") + parser.add_argument("chapters", nargs="*", type=int, help="Chapter IDs to generate") + parser.add_argument("--force", action="store_true", help="Overwrite existing audio") + parser.add_argument("--all", action="store_true", help="Generate all chapters") + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + args = parser.parse_args() + + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") + + cfg = load_config(args.project) + + coordinator = ModelCoordinator(cfg) + if not coordinator.switch_to(PipelineStage.UTTERANCE_GENERATION).get("ok"): + logger.error("Failed to switch models to UTTERANCE_GENERATION stage") + sys.exit(1) + + voices_dir = cfg.voices_dir + + voice_refs = load_voice_references(voices_dir) + speakers = load_speakers(cfg.data_dir) + alias_map = get_speaker_name_map(speakers) + + phonetic = load_phonetic_names(cfg.data_dir) + logger.info("Loaded %d confirmed phonetic name mappings", len(phonetic)) + + narrator_ref = None + for name, path in voice_refs.items(): + if name.lower() == "narrator": + narrator_ref = path + break + if not narrator_ref: + for name, path in voice_refs.items(): + if name.lower() == "kainda": + narrator_ref = path + break + + logger.info("Loaded %d voice references", len(voice_refs)) + logger.info("Loaded %d speakers", len(speakers)) + + vram = get_vram_usage_mb() + logger.info("VRAM: %dMB used, %dMB free", vram["used_mb"], vram["free_mb"]) + + if is_running(): + model = get_loaded_model() + if model: + logger.warning("LM Studio has model loaded: %s", model) + logger.warning("TTS needs ~4GB VRAM. If OOM occurs, unload the model in LM Studio UI.") + + engine = QwenTTSEngine(device="cuda", sample_rate=24000) + + output_base = cfg.data_dir / "chapter_audio" + output_base.mkdir(parents=True, exist_ok=True) + + if args.all: + db = sqlite3.connect(str(cfg.db_path)) + db.row_factory = sqlite3.Row + chapters = db.execute( + "SELECT id FROM chapters WHERE project_id=1 ORDER BY chapter_index" + ).fetchall() + db.close() + chapter_ids = [c["id"] for c in chapters] + elif args.chapters: + chapter_ids = args.chapters + else: + chapter_ids = [5] + + if args.force: + for ch_id in chapter_ids: + ch_dir = output_base / f"ch{ch_id}" + if ch_dir.exists(): + for f in ch_dir.glob("*.wav"): + f.unlink() + logger.info("Cleared existing audio for chapter %d", ch_id) + + total_generated = 0 + for ch_id in chapter_ids: + count = generate_chapter_audio( + ch_id, engine, voice_refs, alias_map, output_base, cfg.db_path, narrator_ref + ) + total_generated += count + + engine.unload_all() + coordinator.switch_to(PipelineStage.IDLE) + + logger.info("Total generated: %d utterances across %d chapters", total_generated, len(chapter_ids)) + + +if __name__ == "__main__": + main() diff --git a/scripts/generate_long_voice_refs.py b/scripts/generate_long_voice_refs.py new file mode 100644 index 0000000..1407b22 --- /dev/null +++ b/scripts/generate_long_voice_refs.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Generate longer voice references (30+ seconds) for speakers using VoiceDesign.""" + +from __future__ import annotations + +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +import io + +import httpx +import soundfile as sf + +from umbraaudio.config import load_config + +TTS_URL = "http://localhost:8881" + +# Rainbow Passage - phonetically balanced paragraph for voice samples +# Source: Grant Fairbanks, Voice and Articulation Drillbook (1960) +# Covers all English phonemes including r-colored vowels +# Repeated to generate 30+ seconds of speech for voice cloning references +VOICE_REFERENCE_TEXT = """ +When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors. These take the shape of a long round arch, with its path high above, and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look, but no one ever finds it. When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. + +When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors. These take the shape of a long round arch, with its path high above, and its two ends apparently beyond the horizon. There is, according to legend, a boiling pot of gold at one end. People look, but no one ever finds it. When a man looks for something beyond his reach, his friends say he is looking for the pot of gold at the end of the rainbow. +""".strip() + +# Speaker descriptions for VoiceDesign +SPEAKER_DESCRIPTIONS = { + "narrator": ( + "A professional male audiobook narrator with a deep, clear, " + "authoritative voice. Calm and measured delivery." + ), + "captain_weston": ( + "A confident male ship captain in his 40s with a commanding but " + "warm voice. Authoritative yet approachable." + ), + "brutus": ("A gruff male voice with a deep, gravelly tone. Tough and no-nonsense demeanor."), + "alec": ( + "A young male voice in his 20s with an energetic and friendly tone. Slightly higher pitch." + ), + "admiral_wilbur_wake": ( + "An older male voice with a distinguished, formal tone. Commanding presence with wisdom." + ), + "captain_wolfgang_antlion": ( + "A stern male voice with a German accent. Precise and formal military bearing." + ), + "ai-2": ( + "A synthesized female AI voice. Clear, precise, and slightly " + "robotic with good pronunciation." + ), + "aunt_vicky": ("A warm, motherly female voice in her 50s. Caring and gentle tone."), + "duchess_victoria": ( + "An elegant female voice with refined pronunciation. Aristocratic and poised." + ), + "gisfrid": ( + "A middle-aged male voice with a scholarly tone. Intelligent and measured speaking style." + ), +} + + +def generate_voice_reference( + speaker_name: str, + description: str, + output_path: Path, + text: str = VOICE_REFERENCE_TEXT, +) -> bool: + """Generate a voice reference using VoiceDesign.""" + print(f"Generating voice reference for: {speaker_name}") + + try: + response = httpx.post( + f"{TTS_URL}/v1/tts", + data={ + "text": text, + "voice_description": description, + "sample_rate": 24000, + }, + timeout=600.0, + ) + + if response.status_code != 200: + print(f" Error: HTTP {response.status_code}") + return False + + # Parse the WAV response properly (server returns a WAV file, not raw PCM) + wav, sr = sf.read(io.BytesIO(response.content), dtype="float32") + sf.write(str(output_path), wav, sr) + + # Check duration + info = sf.info(str(output_path)) + print(f" Generated: {info.duration:.1f}s, {output_path.stat().st_size} bytes") + + return info.duration >= 20.0 # At least 20 seconds + + except Exception as e: + print(f" Error: {e}") + return False + + +def main(): + import argparse + + parser = argparse.ArgumentParser(description="Generate longer voice references") + parser.add_argument( + "--project", + type=str, + default=None, + help="Project name or path", + ) + parser.add_argument( + "--speakers", + type=str, + default=None, + help="Comma-separated speaker names (e.g. 'narrator,captain_weston')", + ) + parser.add_argument( + "--all", action="store_true", help="Generate for all speakers with descriptions" + ) + args = parser.parse_args() + + cfg = load_config(args.project) + voices_dir = cfg.voices_dir + voices_dir.mkdir(parents=True, exist_ok=True) + + if args.all: + speakers = SPEAKER_DESCRIPTIONS + elif args.speakers: + names = [s.strip() for s in args.speakers.split(",")] + speakers = {n: SPEAKER_DESCRIPTIONS.get(n, f"A clear voice for {n}") for n in names} + else: + print("Specify --speakers or --all") + return + + success = 0 + failed = 0 + + for speaker_name, description in speakers.items(): + output_path = voices_dir / f"{speaker_name}_ref.wav" + + # Skip if already exists and is long enough + if output_path.exists(): + info = sf.info(str(output_path)) + if info.duration >= 25.0: + print(f"Skipping {speaker_name}: already {info.duration:.1f}s") + success += 1 + continue + + if generate_voice_reference(speaker_name, description, output_path): + success += 1 + else: + failed += 1 + + time.sleep(1) # Rate limiting + + print(f"\nDone: {success} succeeded, {failed} failed") + + +if __name__ == "__main__": + main() diff --git a/scripts/generate_test_samples.py b/scripts/generate_test_samples.py new file mode 100644 index 0000000..cf6eae6 --- /dev/null +++ b/scripts/generate_test_samples.py @@ -0,0 +1,223 @@ +#!/usr/bin/env python3 +"""Generate test samples for all speakers using Edge-TTS with speaker attributes.""" +from __future__ import annotations + +import asyncio +import json +import sys +import time +from pathlib import Path + +# Add project root to path +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config +from umbraaudio.voices.edge_tts import EdgeTTSEngine + +# Rainbow Passage - phonetically balanced paragraph for voice samples +# Source: Grant Fairbanks, Voice and Articulation Drillbook (1960) +# Covers all English phonemes including r-colored vowels +VERIFICATION_PHRASE = ( + "When the sunlight strikes raindrops in the air, they act as a prism " + "and form a rainbow. The rainbow is a division of white light into many " + "beautiful colors. These take the shape of a long round arch, with its " + "path high above, and its two ends apparently beyond the horizon. There " + "is, according to legend, a boiling pot of gold at one end. People look, " + "but no one ever finds it. When a man looks for something beyond his " + "reach, his friends say he is looking for the pot of gold at the end of " + "the rainbow." +) + + +def _map_traits_to_energy(traits: list[str]) -> str: + """Map vocal traits to energy level.""" + traits_lower = [t.lower() for t in traits] + if any(t in traits_lower for t in ["explosive", "thunderous", "shouting"]): + return "explosive" + if any(t in traits_lower for t in ["volatile", "intense", "agitated", "emotional"]): + return "volatile" + if any(t in traits_lower for t in ["commanding", "authoritative", "powerful", "dominant"]): + return "commanding" + if any(t in traits_lower for t in ["assertive", "confident", "firm", "strong"]): + return "assertive" + if any(t in traits_lower for t in ["measured", "even", "controlled", "steady"]): + return "measured" + if any(t in traits_lower for t in ["soft", "gentle", "soft_spoken", "quiet"]): + return "soft_spoken" + if any(t in traits_lower for t in ["subdued", "muted", "restrained", "hushed"]): + return "subdued" + if any(t in traits_lower for t in ["whispered", "whisper", "barely_audible"]): + return "whispered" + return "measured" + + +def _map_traits_to_pitch(traits: list[str], gender: str) -> str: + """Map vocal traits to pitch register.""" + traits_lower = [t.lower() for t in traits] + if any(t in traits_lower for t in ["very_high", "ultra_high", "squeaky", "falsetto"]): + return "very_high" if gender == "Male" else "ultra_high" + if any(t in traits_lower for t in ["high", "high_pitched", "soprano"]): + return "high" + if any(t in traits_lower for t in ["mid_high", "slightly_high"]): + return "mid_high" + if any(t in traits_lower for t in ["mid", "medium", "moderate", "average"]): + return "mid" + if any(t in traits_lower for t in ["mid_low", "slightly_low"]): + return "mid_low" + if any(t in traits_lower for t in ["low", "deep", "bass", "baritone"]): + return "low" + if any(t in traits_lower for t in ["very_low", "very_deep"]): + return "very_low" + # Default based on gender + return "mid" if gender == "Male" else "mid_high" + + +def _extract_accent(traits: list[str], accent_field: str | None) -> str: + """Extract accent from traits or accent field. Default to Canadian.""" + if accent_field and accent_field.lower() not in ("none", "unknown", ""): + return accent_field.lower() + traits_lower = [t.lower() for t in traits] + accent_keywords = { + "british": ["british", "uk", "english", "cockney", "rp"], + "american": ["american", "us", "general_american"], + "australian": ["australian", "aussie"], + "indian": ["indian", "south_asian"], + "irish": ["irish", "irish_accent"], + "canadian": ["canadian", "canada"], + "nigerian": ["nigerian", "west_african"], + "kenyan": ["kenyan", "east_african"], + "hong_kong": ["hong_kong", "hk", "cantonese"], + } + for accent, keywords in accent_keywords.items(): + if any(k in " ".join(traits_lower) for k in keywords): + return accent + return "canadian" # Default accent + + +def _generate_with_retry(engine: EdgeTTSEngine, text: str, output_path: Path, **kwargs) -> bool: + """Generate audio with retry logic for transient Edge-TTS failures.""" + max_retries = 3 + base_delay = 2.0 + + for attempt in range(max_retries): + try: + engine.generate_voice_reference( + text=text, + output_path=output_path, + **kwargs, + ) + if output_path.exists(): + return True + except Exception as e: + if attempt == max_retries - 1: + raise + delay = base_delay * (2 ** attempt) + print(f" Retry {attempt + 1}/{max_retries} after {delay:.1f}s: {e}") + time.sleep(delay) + return False + + +async def generate_test_samples(project: str | None = None): + """Generate test samples for all speakers using their vocal traits.""" + cfg = load_config(project) + + # Load voice assignments + assignments_path = cfg.data_dir / "voice_assignments.json" + with open(assignments_path) as f: + assignments = json.load(f) + + # Load speakers for vocal traits + speakers_path = cfg.data_dir / "speakers.json" + with open(speakers_path) as f: + speakers_data = json.load(f) + speakers = {s["primary_name"]: s for s in speakers_data["speakers"]} + + print(f"Generating test samples for {len(assignments)} speakers...") + print(f"Verification phrase: \"{VERIFICATION_PHRASE}\"") + print() + + # Create output directory + output_dir = cfg.data_dir / "voice_tests" + output_dir.mkdir(exist_ok=True) + + # Initialize Edge-TTS engine + engine = EdgeTTSEngine() + + # Track results + success_count = 0 + fail_count = 0 + + for speaker_name, voice_assignment in sorted(assignments.items()): + speaker = speakers.get(speaker_name, {}) + demographics = speaker.get("demographics", {}) + gender = demographics.get("gender", "Unknown") + stype = demographics.get("type", "Human") + traits = speaker.get("vocal_traits", []) + accent_field = speaker.get("accent") + + # Map traits to Edge-TTS parameters + energy = _map_traits_to_energy(traits) + pitch = _map_traits_to_pitch(traits, gender) + accent = _extract_accent(traits, accent_field) + + # Parse voice assignment for pitch shift (fallback) + if ":" in voice_assignment: + base_voice, pitch_shift = voice_assignment.split(":") + pitch_shift = int(pitch_shift) + else: + base_voice = voice_assignment + pitch_shift = 0 + + # Create safe filename + safe_name = speaker_name.replace(" ", "_").replace("/", "_") + output_path = output_dir / f"{safe_name}.wav" + + # Skip if already exists + if output_path.exists(): + success_count += 1 + print(f"⊙ {speaker_name:35} -> already exists") + continue + + try: + # Generate audio using speaker attributes (accent defaults to Canadian) + success = _generate_with_retry( + engine=engine, + text=VERIFICATION_PHRASE, + output_path=output_path, + energy=energy, + pitch=pitch, + gender=gender, + character_name=speaker_name, + accent=accent, + ) + + if success and output_path.exists(): + success_count += 1 + print(f"✓ {speaker_name:35} -> energy={energy}, pitch={pitch}, gender={gender}, accent={accent}") + else: + fail_count += 1 + print(f"✗ {speaker_name:35} -> (file not created)") + + except Exception as e: + fail_count += 1 + print(f"✗ {speaker_name:35} -> (error: {e})") + + # Small delay between requests to avoid rate limiting + time.sleep(0.5) + + print("\n=== Summary ===") + print(f"Success: {success_count}") + print(f"Failed: {fail_count}") + print(f"Total: {len(assignments)}") + print(f"\nTest samples saved to: {output_dir}") + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser(description="Generate test samples for all speakers") + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + args = parser.parse_args() + asyncio.run(generate_test_samples(project=args.project)) diff --git a/scripts/generate_voice_references.py b/scripts/generate_voice_references.py new file mode 100644 index 0000000..f1d5800 --- /dev/null +++ b/scripts/generate_voice_references.py @@ -0,0 +1,179 @@ +from __future__ import annotations + +import json +import logging +from pathlib import Path + +from umbraaudio.config import load_config +from umbraaudio.voices.model_coordinator import ModelCoordinator, PipelineStage +from umbraaudio.voices.qwen_tts import QwenTTSEngine, build_voice_description + +logger = logging.getLogger(__name__) + + +def _load_speakers(data_dir: Path) -> list[dict]: + path = data_dir / "speakers.json" + if not path.exists(): + raise FileNotFoundError(f"speakers.json not found: {path}") + with open(path) as f: + data = json.load(f) + return data.get("speakers", []) + + +def _load_profiles(data_dir: Path) -> dict[str, dict]: + """Load voice profiles from voice_profiles.json, falling back to project.db. + + voice_profiles.json is the canonical source; it contains the full + voice_description for every speaker. The DB characters table may be + empty for some projects. + """ + profiles_path = data_dir / "voice_profiles.json" + if profiles_path.exists(): + with open(profiles_path) as f: + data = json.load(f) + if data: + result = {} + for name, profile in data.items(): + result[name] = { + "profile_json": json.dumps(profile), + "voice_description": profile.get("voice_description"), + } + return result + + import sqlite3 + db_path = data_dir / "project.db" + if not db_path.exists(): + return {} + db = sqlite3.connect(str(db_path)) + db.row_factory = sqlite3.Row + rows = db.execute( + "SELECT name, profile_json, voice_description FROM characters WHERE project_id=1" + ).fetchall() + db.close() + result = {} + for r in rows: + result[r["name"]] = { + "profile_json": r["profile_json"], + "voice_description": r["voice_description"], + } + return result + + +def generate_all_voice_references( + project_dir: Path | None = None, + force: bool = False, +) -> None: + project_dir = project_dir or Path(".") + cfg = load_config(project_dir) + + coordinator = ModelCoordinator(cfg) + if not coordinator.switch_to(PipelineStage.VOICE_DESIGN).get("ok"): + raise RuntimeError("Failed to switch models to VOICE_DESIGN stage") + + speakers = _load_speakers(cfg.data_dir) + db_profiles = _load_profiles(cfg.data_dir) + + voices_dir = cfg.data_dir / "voices" + voices_dir.mkdir(parents=True, exist_ok=True) + + tts = QwenTTSEngine( + device=cfg.tts.device, + sample_rate=cfg.tts.sample_rate, + voicedesign_model=cfg.tts.voicedesign_model, + base_model=cfg.tts.base_model, + qwen_tts_url=cfg.tts.qwen_tts_url, + ) + + narrator_name = None + for s in speakers: + if s.get("is_narrator"): + narrator_name = s["primary_name"] + break + + generated = 0 + for s in speakers: + name = s["primary_name"] + is_narrator = s.get("is_narrator", False) + ref_name = name.lower().replace(" ", "_") + "_ref.wav" + ref_path = voices_dir / ref_name + + if ref_path.exists() and not force: + logger.info("Skipping %s (exists)", name) + continue + + db_info = db_profiles.get(name, {}) + voice_desc_override = db_info.get("voice_description") + + if voice_desc_override and not force: + voice_desc = voice_desc_override + elif is_narrator: + voice_desc = ( + "A deep, resonant male voice with a calm, authoritative tone. " + "Rich baritone timbre, slow and deliberate pacing, " + "like a late-night radio host. Very controlled and even." + ) + else: + profile_data = {} + if db_info.get("profile_json"): + try: + profile_data = json.loads(db_info["profile_json"]) + except (json.JSONDecodeError, TypeError): + pass + voice_desc = build_voice_description( + character_name=name, + profile=profile_data if profile_data else None, + gender=s.get("gender") or (profile_data or {}).get("gender"), + energy=(profile_data or {}).get("energy"), + timbre=(profile_data or {}).get("timbre"), + pitch_register=(profile_data or {}).get("pitch_register"), + ) + + sample_text = _sample_text(name, is_narrator, s.get("demographics", {})) + + logger.info("[%d/%d] %s: %s", generated + 1, len(speakers), name, voice_desc[:70]) + try: + tts.generate_voice_reference( + text=sample_text, + output_path=ref_path, + voice_description=voice_desc, + ) + generated += 1 + logger.info(" -> %s", ref_path.name) + except Exception as e: + logger.error(" FAILED %s: %s", name, e) + + tts.unload_all() + coordinator.switch_to(PipelineStage.IDLE) + logger.info("Generated %d/%d voice references", generated, len(speakers)) + + +# Rainbow Passage - phonetically balanced paragraph for voice samples +# Source: Grant Fairbanks, Voice and Articulation Drillbook (1960) +# Covers all English phonemes including r-colored vowels +RAINBOW_PASSAGE = ( + "When the sunlight strikes raindrops in the air, they act as a prism " + "and form a rainbow. The rainbow is a division of white light into many " + "beautiful colors. These take the shape of a long round arch, with its " + "path high above, and its two ends apparently beyond the horizon. There " + "is, according to legend, a boiling pot of gold at one end. People look, " + "but no one ever finds it. When a man looks for something beyond his " + "reach, his friends say he is looking for the pot of gold at the end of " + "the rainbow." +) + + +def _sample_text(name: str, is_narrator: bool, demographics: dict) -> str: + return RAINBOW_PASSAGE + + +if __name__ == "__main__": + import argparse + + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") + parser = argparse.ArgumentParser(description="Generate voice references for all speakers") + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + parser.add_argument("--force", action="store_true", help="Regenerate existing references") + args = parser.parse_args() + generate_all_voice_references(args.project, force=args.force) diff --git a/scripts/generate_voice_samples.py b/scripts/generate_voice_samples.py new file mode 100644 index 0000000..3cbe062 --- /dev/null +++ b/scripts/generate_voice_samples.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +"""Generate Rainbow Passage samples from cached speaker embeddings. + +For each speaker in the given chapter, generates a WAV file of the Rainbow Passage +spoken in that character's voice. Used for human review/approval before full generation. + +Usage: + python scripts/generate_voice_samples.py --chapter 15 + python scripts/generate_voice_samples.py --speakers narrator,nanny,kainda + python scripts/generate_voice_samples.py --all # all speakers with embeddings +""" +from __future__ import annotations + +import argparse +import json +import sqlite3 +import sys +import time +from pathlib import Path + +import numpy as np +import soundfile as sf + +project_root = Path(__file__).resolve().parent.parent +if str(project_root) not in sys.path: + sys.path.insert(0, str(project_root)) + +from umbraaudio.config import load_config # noqa: E402 +from umbraaudio.voices.qwen_embeddings import QwenEmbeddingCache # noqa: E402 + +RAINBOW_PASSAGE = ( + "When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. " + "The rainbow is a division of white light into many beautiful colors. " + "These take the shape of a long round arch, with its path high above, " + "and its two ends apparently beyond the horizon. " + "There is, according to legend, a boiling pot of gold at one end. " + "People look, but no one ever finds it. " + "When a man looks for something beyond his reach, " + "his friends say he is looking for the pot of gold at the end of the rainbow. " + "When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. " + "The rainbow is a division of white light into many beautiful colors. " + "These take the shape of a long round arch, with its path high above, " + "and its two ends apparently beyond the horizon. " + "There is, according to legend, a boiling pot of gold at one end. " + "People look, but no one ever finds it. " + "When a man looks for something beyond his reach, " + "his friends say he is looking for the pot of gold at the end of the rainbow." +) + + +def name_to_slug(name: str) -> str: + return name.lower().replace(' ', '_').replace('.', '').replace("'", '').replace('(', '').replace(')', '').replace('-', '_').replace('/', '_') + + +def get_chapter_speakers(project_name: str, chapter_id: int) -> list[str]: + config = load_config(project_name) + db_path = config.data_dir / "project.db" + db = sqlite3.connect(str(db_path)) + rows = db.execute(''' + SELECT DISTINCT speaker_name FROM utterances + WHERE chapter_id=? AND speaker_name IS NOT NULL + ''', (chapter_id,)).fetchall() + db.close() + return [r[0] for r in rows] + + +def generate_samples( + project_name: str, + speaker_names: list[str], + output_dir: Path, + temperature: float = 0.5, + seed: int = 42, +) -> list[Path]: + from umbraaudio.voices.qwen_tts import QwenTTSEngine + + config = load_config(project_name) + embeddings_dir = config.data_dir / "voice_embeddings" + cache = QwenEmbeddingCache(embeddings_dir) + + output_dir.mkdir(parents=True, exist_ok=True) + engine = QwenTTSEngine(qwen_tts_url="http://localhost:8881") + + generated = [] + for name in speaker_names: + slug = name_to_slug(name) + if not cache.has(slug): + print(f" SKIP {name}: no embedding cached (slug={slug})") + continue + + out_path = output_dir / f"{slug}_rainbow.wav" + t0 = time.time() + # Use a dummy ref path — the engine will find the cached embedding + dummy_ref = config.data_dir / "voices" / f"{slug}_ref.wav" + if not dummy_ref.exists(): + dummy_ref = config.data_dir / "voices" / "narrator_ref.wav" + engine.generate( + text=RAINBOW_PASSAGE, + output_path=out_path, + voice_ref=dummy_ref, + temperature=temperature, + seed=seed, + ) + elapsed = time.time() - t0 + size_kb = out_path.stat().st_size / 1024 + print(f" OK {name:40s} -> {out_path.name:40s} ({elapsed:.1f}s, {size_kb:.0f}KB)") + generated.append(out_path) + + return generated + + +def main() -> None: + parser = argparse.ArgumentParser(description="Generate voice samples for review") + parser.add_argument("--project", default="DealHunterShip_4.0e_whole-to-audio") + parser.add_argument("--chapter", type=int, help="Generate samples for all speakers in this chapter") + parser.add_argument("--speakers", help="Comma-separated speaker names") + parser.add_argument("--all", action="store_true", help="All speakers with embeddings") + parser.add_argument("--temperature", type=float, default=0.5) + parser.add_argument("--seed", type=int, default=42) + parser.add_argument("--output-dir", default=None, help="Output directory") + args = parser.parse_args() + + if args.chapter: + speakers = get_chapter_speakers(args.project, args.chapter) + out_dir = Path(args.output_dir or f"/tmp/voice_samples_ch{args.chapter}") + print(f"Chapter {args.chapter}: {len(speakers)} speakers") + elif args.speakers: + speakers = [s.strip() for s in args.speakers.split(",")] + out_dir = Path(args.output_dir or "/tmp/voice_samples") + elif args.all: + config = load_config(args.project) + cache = QwenEmbeddingCache(config.data_dir / "voice_embeddings") + speakers = [s.replace("_", " ").title() for s in cache.list_cached()] + out_dir = Path(args.output_dir or "/tmp/voice_samples_all") + else: + parser.error("Specify --chapter, --speakers, or --all") + + print(f"Output: {out_dir}") + t0 = time.time() + files = generate_samples(args.project, speakers, out_dir, args.temperature, args.seed) + elapsed = time.time() - t0 + print(f"\nDone: {len(files)} samples in {elapsed:.1f}s") + + +if __name__ == "__main__": + main() diff --git a/scripts/lmstudio_control.py b/scripts/lmstudio_control.py new file mode 100644 index 0000000..c225b4b --- /dev/null +++ b/scripts/lmstudio_control.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python3 +"""LM Studio model control script for UmbraAudio. + +Usage: + python scripts/lmstudio_control.py status # Show LM Studio status + python scripts/lmstudio_control.py start # Start server + python scripts/lmstudio_control.py stop # Stop server + python scripts/lmstudio_control.py load [MODEL] # Load a model + python scripts/lmstudio_control.py unload # Unload current model +""" +from __future__ import annotations + +import sys + +from umbraaudio.voices.lmstudio_lifecycle import ( + get_loaded_model, + is_running, + load_model, + server_status, + start_server, + stop_server, + unload_all, +) + +PREFERRED_MODELS = [ + "gemma-4-12b-it-qat", + "google/gemma-4-e4b", + "qwen3.5-9b", +] + + +def cmd_start(): + if is_running(): + print("LM Studio is already running") + return + start_server() + print("LM Studio server started") + + +def cmd_stop(): + if not is_running(): + print("LM Studio is not running") + return + stop_server() + print("LM Studio server stopped") + + +def cmd_unload(): + if not is_running(): + print("LM Studio is not running") + return + loaded = get_loaded_model() + if not loaded: + print("No model loaded") + return + print(f"Unloading {loaded}...") + if unload_all(): + print("Unloaded successfully") + else: + print("Unload failed") + + +def cmd_load(model: str | None = None): + if not is_running(): + print("LM Studio is not running. Use 'start' first.") + return + + current = get_loaded_model() + if current: + if current == model or (not model and current in PREFERRED_MODELS): + print(f"Model already loaded: {current}") + return + print(f"Unloading {current}...") + unload_all() + + if not model: + import json + import subprocess + + result = subprocess.run( + ["lms", "ls", "--json"], + capture_output=True, text=True, timeout=30, + ) + if result.returncode == 0: + available = [ + m["modelKey"] for m in json.loads(result.stdout) + if m.get("type") == "llm" + ] + for preferred in PREFERRED_MODELS: + if preferred in available: + model = preferred + break + if not model: + print("No preferred model found and no model specified") + return + + print(f"Loading {model}...") + if load_model(model): + print(f"Loaded successfully: {model}") + else: + print(f"Load failed: {model}") + + +def cmd_status(): + status = server_status() + if not status["api_responding"]: + print("LM Studio is NOT running") + print("\nStart it with:") + print(" python scripts/lmstudio_control.py start") + return + + print("LM Studio is running") + print(f" Server: {'running' if status['server_running'] else 'stopped'}") + + models = status["loaded_models"] + if models: + print(f" Loaded: {models[0]}") + else: + print(" No model loaded") + + import json + import subprocess + + result = subprocess.run( + ["lms", "ls", "--json"], + capture_output=True, text=True, timeout=30, + ) + if result.returncode == 0: + available = [ + m["modelKey"] for m in json.loads(result.stdout) + if m.get("type") == "llm" + ] + if available: + print(f"\nAvailable models ({len(available)}):") + for m in available: + marker = " *" if models and m == models[0] else "" + preferred = " (recommended)" if m in PREFERRED_MODELS else "" + print(f" {m}{marker}{preferred}") + + +def main(): + if len(sys.argv) < 2: + print(__doc__) + cmd_status() + return + + cmd = sys.argv[1].lower() + + if cmd == "status": + cmd_status() + elif cmd == "start": + cmd_start() + elif cmd == "stop": + cmd_stop() + elif cmd == "unload": + cmd_unload() + elif cmd == "load": + model = sys.argv[2] if len(sys.argv) > 2 else None + cmd_load(model) + else: + print(f"Unknown command: {cmd}") + print(__doc__) + + +if __name__ == "__main__": + main() diff --git a/scripts/model_coordinator.py b/scripts/model_coordinator.py new file mode 100644 index 0000000..8fec00c --- /dev/null +++ b/scripts/model_coordinator.py @@ -0,0 +1,115 @@ +"""CLI for the model coordinator. + +Ensures only the model needed by the current pipeline stage is resident in +memory, freeing maximum GPU/RAM for whichever model is about to work. + +Usage: + python scripts/model_coordinator.py status + python scripts/model_coordinator.py switch \\ + voice_design|chapter_analysis|utterance_generation|idle + python scripts/model_coordinator.py unload-all +""" + +from __future__ import annotations + +import argparse +import json +import logging +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from rich.console import Console +from rich.table import Table + +from umbraaudio.config import load_config +from umbraaudio.voices.model_coordinator import ModelCoordinator, PipelineStage + +logger = logging.getLogger(__name__) +console = Console() + + +def _render_status(state: dict) -> None: + qwen_model = state.get("qwen_model") or "—" + lm_loaded = state.get("lm_loaded") or [] + lm_str = ", ".join(lm_loaded) if lm_loaded else "—" + table = Table(title="Model Coordinator Status") + table.add_column("Server", style="bold cyan") + table.add_column("State", style="bold green") + table.add_row("Qwen TTS server", state.get("qwen_server", "unknown")) + table.add_row("Qwen current model", qwen_model) + table.add_row(" voicedesign", "loaded" if state.get("voicedesign_loaded") else "unloaded") + table.add_row(" base", "loaded" if state.get("base_loaded") else "unloaded") + table.add_row("LM Studio loaded", lm_str) + console.print(table) + + +def cmd_status(coord: ModelCoordinator, _args: argparse.Namespace) -> int: + _render_status(coord.status()) + return 0 + + +def cmd_switch(coord: ModelCoordinator, args: argparse.Namespace) -> int: + try: + stage = PipelineStage(args.stage) + except ValueError: + console.print( + f"[red]Unknown stage '{args.stage}'. " + f"Choose from: {', '.join(s.value for s in PipelineStage)}[/red]" + ) + return 2 + console.print(f"Switching to stage [bold]{stage.value}[/bold]...") + result = coord.switch_to(stage) + if not result.get("ok"): + console.print(f"[red]Failed: {result.get('error', 'unknown error')}[/red]") + console.print("Before:", json.dumps(result.get("before", {}), indent=2)) + return 1 + console.print("[green]OK[/green]") + _render_status(result["after"]) + return 0 + + +def cmd_unload_all(coord: ModelCoordinator, _args: argparse.Namespace) -> int: + console.print("Unloading all models...") + result = coord.switch_to(PipelineStage.IDLE) + if not result.get("ok"): + console.print(f"[red]Failed: {result.get('error', 'unknown error')}[/red]") + return 1 + console.print("[green]All models unloaded[/green]") + _render_status(result["after"]) + return 0 + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Switch which models are resident for the current pipeline stage." + ) + sub = parser.add_subparsers(dest="command", required=True) + + p_status = sub.add_parser("status", help="Show which models are currently loaded") + p_status.set_defaults(func=cmd_status) + + p_switch = sub.add_parser("switch", help="Switch to a pipeline stage's model set") + p_switch.add_argument( + "stage", + choices=[s.value for s in PipelineStage], + help="Which stage's models should be resident", + ) + p_switch.set_defaults(func=cmd_switch) + + p_unload = sub.add_parser("unload-all", help="Unload all models from both servers") + p_unload.set_defaults(func=cmd_unload_all) + + return parser + + +def main(argv: list[str] | None = None) -> int: + logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(message)s") + args = build_parser().parse_args(argv) + coord = ModelCoordinator(load_config()) + return args.func(coord, args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/nas_lifecycle.py b/scripts/nas_lifecycle.py new file mode 100644 index 0000000..b603d62 --- /dev/null +++ b/scripts/nas_lifecycle.py @@ -0,0 +1,224 @@ +#!/usr/bin/env python3 +"""NAS sync + local prune lifecycle for UmbraAudio. + +Primary store: local SSD. Secondary: NAS mirror. On each run: + 1. ensure the NAS is mounted + 2. rsync local -> NAS (mirror everything) + 3. prune locally only what is verified present on the NAS, + respecting age and free-space targets + +Fetch fallback for pruned files lives in umbraaudio.storage. +""" + +from __future__ import annotations + +import argparse +import json +import logging +import shutil +import subprocess +import sys +import time +from pathlib import Path + +DEFAULT_LOCAL_ROOT = Path("/Volumes/1TB_APL_Storage/UmbraAudio") +DEFAULT_NAS_ROOT = Path("/Volumes/Umbra/UmbraAudio") +MOUNT_SCRIPT = Path.home() / ".umbra/mount_nas.sh" +CONFIG_PATH = Path.home() / ".umbra/nas_config.json" + +RSYNC_EXCLUDES = [".git/", "node_modules/", ".idea/", "__pycache__/", ".venv/", ".python/"] + +# Never pruned in normal mode: live code, config, and active TTS models +PROTECTED_DIRS = {"umbraaudio", "scripts", "tests", "specs", "Qwen3-TTS", + ".umbra", ".git", ".venv", ".python", ".pytest_cache"} + +DEFAULT_POLICY = { + "min_age_days": 30, # only prune files untouched this long + "free_target_gb": 150, # prune until at least this much is free (normal mode) + "emergency_free_gb": 40, # below this, protected dirs may be pruned too +} + +logger = logging.getLogger("nas_lifecycle") +LOCK_FILE = Path("/tmp/umbra_nas_lifecycle.lock") + + +class Locked(RuntimeError): + pass + + +def load_policy() -> dict: + policy = dict(DEFAULT_POLICY) + try: + cfg = json.loads(CONFIG_PATH.read_text()) + policy.update(cfg.get("prune_policy", {})) + except (OSError, json.JSONDecodeError): + pass + return policy + + +def nas_mounted(nas_root: Path = DEFAULT_NAS_ROOT) -> bool: + return nas_root.is_dir() and any(nas_root.iterdir()) + + +def ensure_mounted() -> bool: + if nas_mounted(): + return True + logger.info("NAS not mounted; running %s", MOUNT_SCRIPT) + try: + subprocess.run(["bash", str(MOUNT_SCRIPT)], check=True, + capture_output=True, timeout=120) + except (subprocess.SubprocessError, OSError) as e: + logger.error("mount failed: %s", e) + return False + return nas_mounted() + + +def sync(local_root: Path = DEFAULT_LOCAL_ROOT, + nas_root: Path = DEFAULT_NAS_ROOT) -> bool: + cmd = ["rsync", "-a", "--delete"] + for exc in RSYNC_EXCLUDES: + cmd += ["--exclude", exc] + cmd += [f"{local_root}/", f"{nas_root}/"] + result = subprocess.run(cmd, capture_output=True, text=True) + if result.returncode != 0: + logger.error("rsync failed (%d): %s", result.returncode, result.stderr[-500:]) + return False + return True + + +def verified_on_nas(local_file: Path, local_root: Path, + nas_root: Path) -> bool: + mirror = nas_root / local_file.relative_to(local_root) + try: + ls, ln = local_file.stat(), mirror.stat() + except OSError: + return False + return ls.st_size == ln.st_size and int(ls.st_mtime) == int(ln.st_mtime) + + +def free_bytes(path: Path) -> int: + return shutil.disk_usage(path).free + + +def prune_plan(local_root: Path, nas_root: Path, policy: dict, + emergency: bool = False) -> list[Path]: + """Files safe to delete now, largest-first among eligible.""" + min_age_s = policy["min_age_days"] * 86400 + cutoff = time.time() - min_age_s + candidates: list[tuple[int, Path]] = [] + for p in local_root.rglob("*"): + if not p.is_file() or p.is_symlink(): + continue + rel_parents = set(p.relative_to(local_root).parts[:-1]) + if not emergency and rel_parents & PROTECTED_DIRS: + continue + st = p.stat() + if st.st_mtime > cutoff: + continue + if not verified_on_nas(p, local_root, nas_root): + continue + candidates.append((st.st_size, p)) + candidates.sort(reverse=True) + return [p for _, p in candidates] + + +def prune(local_root: Path = DEFAULT_LOCAL_ROOT, + nas_root: Path = DEFAULT_NAS_ROOT, + policy: dict | None = None) -> dict: + policy = policy or load_policy() + target = policy["free_target_gb"] * 1024**3 + emergency_at = policy["emergency_free_gb"] * 1024**3 + freed = 0 + deleted = 0 + emergency = free_bytes(local_root) < emergency_at + for p in prune_plan(local_root, nas_root, policy, emergency=emergency): + if free_bytes(local_root) >= target and freed > 0: + break + size = p.stat().st_size + try: + p.unlink() + except OSError as e: + logger.warning("could not unlink %s: %s", p, e) + continue + freed += size + deleted += 1 + parent = p.parent + if parent != local_root and not any(parent.iterdir()): + parent.rmdir() + stats = {"deleted": deleted, "freed_gb": round(freed / 1024**3, 2), + "emergency": emergency, + "free_gb": round(free_bytes(local_root) / 1024**3, 1)} + logger.info("prune: %s", stats) + return stats + + +def status(local_root: Path = DEFAULT_LOCAL_ROOT, + nas_root: Path = DEFAULT_NAS_ROOT) -> dict: + lf = lb = 0 + for p in local_root.rglob("*"): + if p.is_file() and not p.is_symlink(): + try: + lf += 1 + lb += p.stat().st_size + except OSError: + pass + usage = shutil.disk_usage(nas_root) if nas_mounted(nas_root) else None + return { + "local": {"files": lf, "gb": round(lb / 1024**3, 2), + "free_gb": round(free_bytes(local_root) / 1024**3, 1)}, + "nas": {"mounted": nas_mounted(nas_root), + "free_gb": round(usage.free / 1024**3, 1) if usage else None}, + } + + +def _acquire_lock() -> int: + import os + fd = os.open(LOCK_FILE, os.O_CREAT | os.O_EXCL | os.O_WRONLY) + os.write(fd, str(os.getpid()).encode()) + return fd + + +def run_cycle(local_root: Path = DEFAULT_LOCAL_ROOT, + nas_root: Path = DEFAULT_NAS_ROOT) -> bool: + if not ensure_mounted(): + return False + busy = subprocess.run(["pgrep", "-f", + f"rsync .*{nas_root}"], capture_output=True) + if busy.returncode == 0: + logger.info("another rsync to the NAS is running; deferring cycle") + return True + try: + _acquire_lock() + except FileExistsError: + logger.info("cycle already in progress; skipping") + return True + try: + if not sync(): + return False + prune() + finally: + LOCK_FILE.unlink(missing_ok=True) + return True + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("command", choices=["sync", "prune", "status", "cycle"]) + args = parser.parse_args(argv) + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") + + if args.command == "status": + print(json.dumps(status(), indent=2)) + return 0 + if args.command == "sync": + ok = ensure_mounted() and sync() + elif args.command == "prune": + print(json.dumps(prune(), indent=2)) + return 0 + else: + ok = run_cycle() + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/qa_attribution.py b/scripts/qa_attribution.py new file mode 100644 index 0000000..22ce62e --- /dev/null +++ b/scripts/qa_attribution.py @@ -0,0 +1,152 @@ +#!/usr/bin/env python3 +"""Mechanical QA cross-check of speaker attribution against speech tags. + +Usage: + python scripts/qa_attribution.py # check all chapters + python scripts/qa_attribution.py --chapters 6 # check chapter id 6 + python scripts/qa_attribution.py --fix # auto-fix exact name-tag mismatches + python scripts/qa_attribution.py --type role_tag_guessed # only one issue type +""" + +from __future__ import annotations + +import argparse +import json +import sqlite3 +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.analysis.attribution_qa import ( + QAIssue, + check_attribution, + fix_high_confidence, + verify_conversation_flow, +) +from umbraaudio.config import load_config + + +def _build_alias_map(speakers_path: Path) -> dict[str, str]: + alias_map: dict[str, str] = {} + if not speakers_path.exists(): + return alias_map + with open(speakers_path) as f: + data = json.load(f) + for s in data.get("speakers", []): + primary = s.get("primary_name", "") + if not primary: + continue + alias_map[primary.lower()] = primary + for alias in s.get("aliases", []): + if alias: + alias_map[alias.lower()] = primary + first = primary.split()[0].lower() if primary.split() else "" + if first and first not in alias_map: + alias_map[first] = primary + return alias_map + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + parser.add_argument("--chapters", type=str, default=None, help="Comma-separated chapter IDs") + parser.add_argument("--fix", action="store_true", help="Auto-fix exact speech-tag mismatches") + parser.add_argument( + "--type", + dest="issue_type", + type=str, + default=None, + help=( + "Only report one issue type " + "(speech_tag_mismatch, narrator_dialogue, role_tag_guessed, " + "address_mismatch, narration_speaker_mismatch, long_monologue)" + ), + ) + args = parser.parse_args() + + cfg = load_config(args.project) + db_path = cfg.db_path + if not db_path.exists(): + print(f"Database not found: {db_path}") + return + + alias_map = _build_alias_map(cfg.data_dir / "speakers.json") + db = sqlite3.connect(str(db_path)) + db.row_factory = sqlite3.Row + + if args.chapters: + chapter_ids = [int(x.strip()) for x in args.chapters.split(",")] + placeholders = ",".join("?" * len(chapter_ids)) + chapters = db.execute( + f"SELECT id, title, chapter_index FROM chapters " + f"WHERE id IN ({placeholders}) ORDER BY chapter_index", + chapter_ids, + ).fetchall() + else: + chapters = db.execute( + "SELECT id, title, chapter_index FROM chapters ORDER BY chapter_index" + ).fetchall() + + totals = { + "speech_tag_mismatch": 0, + "narrator_dialogue": 0, + "role_tag_guessed": 0, + "address_mismatch": 0, + "narration_speaker_mismatch": 0, + "long_monologue": 0, + } + all_issues: list[QAIssue] = [] + + for ch in chapters: + ch_id = ch["id"] + rows = db.execute( + "SELECT id, utterance_index, text, type, speaker_name " + "FROM utterances WHERE chapter_id=? ORDER BY utterance_index", + (ch_id,), + ).fetchall() + utts = [dict(r) for r in rows] + + result = check_attribution(utts, ch_id, alias_map) + + flow_result = verify_conversation_flow(utts, ch_id, alias_map) + result.issues.extend(flow_result.issues) + + if args.fix: + fixed = fix_high_confidence(utts, alias_map) + for orig, new in zip(utts, fixed): + if orig.get("speaker_name") != new.get("speaker_name"): + db.execute( + "UPDATE utterances SET speaker_name=? WHERE id=?", + (new["speaker_name"], orig["id"]), + ) + print( + f" FIX ch{ch_id} utt{new.get('utterance_index')}: " + f"{orig.get('speaker_name')} -> {new['speaker_name']}" + ) + db.commit() + + for issue in result.issues: + if args.issue_type and issue.issue_type != args.issue_type: + continue + totals[issue.issue_type] += 1 + all_issues.append(issue) + print( + f"[{issue.issue_type}] ch{issue.chapter_id} utt{issue.utterance_index} " + f"speaker={issue.speaker!r} expected={issue.expected!r} conf={issue.confidence}" + ) + if issue.evidence: + print(f" evidence: {issue.evidence}") + print(f" text: {issue.text}") + + db.close() + + print(f"\nTotals: {totals}") + if all_issues: + print(f"Total issues: {len(all_issues)} (review or fix before accepting voices)") + + +if __name__ == "__main__": + main() diff --git a/scripts/qwen3_tts_demo_umbra_patch.patch b/scripts/qwen3_tts_demo_umbra_patch.patch new file mode 100644 index 0000000..e1a5750 --- /dev/null +++ b/scripts/qwen3_tts_demo_umbra_patch.patch @@ -0,0 +1,111 @@ +diff --git a/qwen_tts/cli/demo.py b/qwen_tts/cli/demo.py +index e267861..686c3f6 100644 +--- a/qwen_tts/cli/demo.py ++++ b/qwen_tts/cli/demo.py +@@ -18,9 +18,11 @@ A gradio demo for Qwen3 TTS models. + """ + + import argparse ++import base64 + import os + import tempfile + from dataclasses import asdict ++from pathlib import Path + from typing import Any, Dict, List, Optional, Tuple + + import gradio as gr +@@ -59,6 +61,31 @@ def _maybe(v): + return v if v is not None else gr.update() + + ++def _find_icon() -> Optional[Path]: ++ candidates = [ ++ Path(os.environ.get("UMBRA_AUDIO_ICON", "")) if os.environ.get("UMBRA_AUDIO_ICON") else None, ++ Path("/Volumes/1TB_APL_Storage/UmbraAudio/UmbraAudio_icon.png"), ++ Path(__file__).resolve().parents[2] / "assets" / "logo.png", ++ Path("/Volumes/1TB_APL_Storage/UmbraAudio/UmbraAudio_icon.jpeg"), ++ ] ++ for c in candidates: ++ if c is not None and c.is_file(): ++ return c ++ return None ++ ++ ++def _icon_data_uri() -> Optional[str]: ++ icon = _find_icon() ++ if icon is None: ++ return None ++ try: ++ data = base64.b64encode(icon.read_bytes()).decode("ascii") ++ mime = "image/png" if icon.suffix.lower() == ".png" else "image/jpeg" ++ return f"data:{mime};base64,{data}" ++ except OSError: ++ return None ++ ++ + def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="qwen-tts-demo", +@@ -168,6 +195,18 @@ def build_parser() -> argparse.ArgumentParser: + return parser + + ++def _find_favicon() -> Optional[Path]: ++ for name in ("UmbraAudio_icon.png", "logo.png"): ++ p = Path(__file__).resolve().parents[3] / name ++ if p.is_file(): ++ return p ++ alt = Path("/Volumes/1TB_APL_Storage/UmbraAudio") / name ++ if alt.is_file(): ++ return alt ++ icon = _find_icon() ++ return icon if icon is not None and icon.suffix.lower() in (".png", ".gif", ".ico") else None ++ ++ + def _resolve_checkpoint(args: argparse.Namespace) -> str: + ckpt = args.checkpoint or args.checkpoint_pos + if not ckpt: +@@ -275,13 +314,30 @@ def build_demo(tts: Qwen3TTSModel, ckpt: str, gen_kwargs_default: Dict[str, Any] + css = ".gradio-container {max-width: none !important;}" + + with gr.Blocks(theme=theme, css=css) as demo: +- gr.Markdown( +- f""" ++ icon_uri = _icon_data_uri() ++ if icon_uri: ++ with gr.Row(): ++ with gr.Column(scale=1, min_width=120): ++ gr.HTML( ++ f'UmbraAudio' ++ ) ++ with gr.Column(scale=4): ++ gr.Markdown( ++ f""" ++# UmbraAudio Voice Studio ++**Checkpoint:** `{ckpt}` ++**Model Type:** `{model_kind}` ++""" ++ ) ++ else: ++ gr.Markdown( ++ f""" + # Qwen3 TTS Demo + **Checkpoint:** `{ckpt}` + **Model Type:** `{model_kind}` + """ +- ) ++ ) + + if model_kind == "custom_voice": + with gr.Row(): +@@ -621,6 +677,9 @@ def main(argv=None) -> int: + share=args.share, + ssl_verify=True if args.ssl_verify else False, + ) ++ favicon = _find_favicon() ++ if favicon is not None: ++ launch_kwargs["favicon_path"] = str(favicon) + if args.ssl_certfile is not None: + launch_kwargs["ssl_certfile"] = args.ssl_certfile + if args.ssl_keyfile is not None: diff --git a/scripts/qwen_tts_server.py b/scripts/qwen_tts_server.py new file mode 100644 index 0000000..531738f --- /dev/null +++ b/scripts/qwen_tts_server.py @@ -0,0 +1,535 @@ +#!/usr/bin/env python3 +"""Qwen3-TTS FastAPI Server for UmbraAudio. + +Provides REST endpoints for voice design and voice cloning. +Expected by umbraaudio.voices.qwen_tts.QwenTTSEngine. +""" + +from __future__ import annotations + +import asyncio +import io +import json +import logging +import os +import time + +import anyio +import numpy as np +import soundfile as sf +import torch +from dataclasses import asdict +from fastapi import FastAPI, File, Form, HTTPException, UploadFile +from fastapi.responses import Response +from pathlib import Path +from pydantic import BaseModel +from qwen_tts import Qwen3TTSModel +from qwen_tts.inference.qwen3_tts_model import VoiceClonePromptItem + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +# Directory where precomputed voice-clone prompts (the "8k matrix" ref_code + +# speaker embedding) are cached, one .pt file per speaker slug. Populated by +# POST /v1/voice-prompt and consumed by /v1/tts via the prompt_slug field so +# every utterance for a speaker reuses the exact same voice identity. +PROMPT_DIR = Path( + os.getenv( + "QWEN_PROMPT_DIR", + "/Volumes/1TB_APL_Storage/UmbraAudio/DealHunterShip_4.0e_whole-to-audio/.umbra/voice_prompts", + ) +) + +app = FastAPI(title="Qwen3-TTS Server", version="0.1.0") + + +async def _watchdog_loop() -> None: + """Background task: if a generation has been stuck past MAX_GEN, restart.""" + while True: + await asyncio.sleep(min(_MAX_GEN, 30)) + s = _gen_start_at + if s is not None and (time.monotonic() - s) > _MAX_GEN: + _restart_server( + f"generation hung >{_MAX_GEN:.0f}s (poisoned CUDA context?)" + ) + await asyncio.sleep(3600) # never reached after execv + + +@app.on_event("startup") +async def _start_watchdog() -> None: + app.state._wd_task = asyncio.create_task(_watchdog_loop()) + +# Rainbow Passage — used as default ref_text for ICL voice cloning mode. +# All voice refs were generated with this text via Edge-TTS. +VOICE_REFERENCE_TEXT = ( + "When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. " + "The rainbow is a division of white light into many beautiful colors. " + "These take the shape of a long round arch, with its path high above, " + "and its two ends apparently beyond the horizon. " + "There is, according to legend, a boiling pot of gold at one end. " + "People look, but no one ever finds it. " + "When a man looks for something beyond his reach, " + "his friends say he is looking for the pot of gold at the end of the rainbow. " + "When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. " + "The rainbow is a division of white light into many beautiful colors. " + "These take the shape of a long round arch, with its path high above, " + "and its two ends apparently beyond the horizon. " + "There is, according to legend, a boiling pot of gold at one end. " + "People look, but no one ever finds it. " + "When a man looks for something beyond his reach, " + "his friends say he is looking for the pot of gold at the end of the rainbow." +) + +# Global model state +_vd_model: Qwen3TTSModel | None = None +_base_model: Qwen3TTSModel | None = None +_current_model: str | None = None # "voicedesign" or "base" + +# Serializes all model access. Model loads and generations are blocking CPU/GPU +# calls; they run in worker threads (anyio.to_thread) under this lock so the +# event loop stays responsive and concurrent /v1/tts requests cannot wedge it. +_model_lock = asyncio.Lock() + +# Hang watchdog. On CUDA, a kernel error (e.g. OOM) can poison the context and +# wedge the generation thread forever while the event loop stays responsive +# (health works, every /v1/tts hangs). The only safe recovery is to restart the +# process. We track when a generation started and, if it exceeds MAX_GEN_SECONDS, +# we relaunch ourselves via os.execv to get a clean CUDA context. +_MAX_GEN = float(os.getenv("QWEN_MAX_GEN_S", "1800")) +_gen_start_at: float | None = None # time.monotonic() when a generation began + + +def _mark_gen_started() -> None: + global _gen_start_at + _gen_start_at = time.monotonic() + + +def _mark_gen_done() -> None: + global _gen_start_at + _gen_start_at = None + + +def _restart_server(reason: str) -> None: + """Relaunch the current process (os.execv) to clear a wedged GPU context.""" + import sys as _sys + + logger.warning("Restarting TTS server: %s", reason) + try: + for s in (_sys.stdout, _sys.stderr): + try: + s.flush() + except Exception: + pass + except Exception: + pass + try: + _sys.stdout = open(os.devnull, "w") + _sys.stderr = open(os.devnull, "w") + _sys.stdin = open(os.devnull) + os.execv( + _sys.executable, + [_sys.executable, os.path.abspath(__file__), *list(_sys.argv[1:])], + ) + except Exception as e: # pragma: no cover + logger.error("os.execv failed: %s", e) + raise SystemExit(1) + +# Model paths from environment +VOICEDESIGN_PATH = os.getenv( + "QWEN_VOICEDESIGN_MODEL", + "/Volumes/1TB_APL_Storage/UmbraAudio/models/Qwen3-TTS-12Hz-1.7B-VoiceDesign", +) +BASE_PATH = os.getenv( + "QWEN_BASE_MODEL", + "/Volumes/1TB_APL_Storage/UmbraAudio/models/Qwen3-TTS-12Hz-1.7B-Base", +) +DEVICE = os.getenv("QWEN_DEVICE", "mps") +DTYPE = torch.bfloat16 if DEVICE in ("cuda", "mps") else torch.float32 +# Attention implementation. "eager" is very slow for Qwen3-TTS autoregressive +# codec decoding but is the only one that reliably runs create_voice_clone_prompt +# (prompt extraction). "sdpa" (built into torch) is far faster for generation. +# Runtime-switchable via POST /attn (reloads the model); this is read at model +# load time, so switching reloads the active model with the new implementation. +_ATTN_IMPL: str = os.getenv("QWEN_ATTN", "sdpa") + + +class LoadRequest(BaseModel): + model_type: str # "voicedesign" or "base" + + +class GenerateRequest(BaseModel): + text: str + sample_rate: int = 24000 + voice_description: str | None = None + + +def _load_model(model_type: str) -> Qwen3TTSModel: + """Load a model, unloading the other if needed.""" + global _vd_model, _base_model, _current_model + + if _current_model == model_type: + return _vd_model if model_type == "voicedesign" else _base_model + + # Unload current + if _current_model == "voicedesign" and _vd_model is not None: + del _vd_model + _vd_model = None + elif _current_model == "base" and _base_model is not None: + del _base_model + _base_model = None + + if torch.mps.is_available() or torch.cuda.is_available(): + torch.mps.empty_cache() if torch.mps.is_available() else torch.cuda.empty_cache() + + # Load requested + path = VOICEDESIGN_PATH if model_type == "voicedesign" else BASE_PATH + logger.info(f"Loading {model_type} model from {path}") + + model = Qwen3TTSModel.from_pretrained( + path, + device_map=DEVICE, + dtype=DTYPE, + attn_implementation=_ATTN_IMPL, + ) + + if model_type == "voicedesign": + _vd_model = model + else: + _base_model = model + + _current_model = model_type + logger.info(f"{model_type} model loaded successfully") + return model + + +def _prompt_path(slug: str) -> Path: + safe = Path(slug).name + PROMPT_DIR.mkdir(parents=True, exist_ok=True) + return PROMPT_DIR / f"{safe}.pt" + + +def _save_prompt(slug: str, items: list[VoiceClonePromptItem]) -> Path: + path = _prompt_path(slug) + payload = {"items": [asdict(it) for it in items]} + torch.save(payload, str(path)) + logger.info("Saved ICL voice prompt for %s to %s", slug, path) + return path + + +def _load_prompt_items(slug: str) -> list[VoiceClonePromptItem] | None: + path = _prompt_path(slug) + if not path.is_file(): + return None + try: + payload = torch.load(str(path), map_location="cpu", weights_only=True) + except Exception as e: + logger.warning("Failed to load prompt %s: %s", slug, e) + return None + raw_items = payload.get("items", []) + items: list[VoiceClonePromptItem] = [] + for d in raw_items: + ref_code = d.get("ref_code") + if ref_code is not None and not torch.is_tensor(ref_code): + ref_code = torch.tensor(ref_code) + ref_spk = d.get("ref_spk_embedding") + if ref_spk is None: + logger.warning("Prompt %s missing ref_spk_embedding", slug) + return None + if not torch.is_tensor(ref_spk): + ref_spk = torch.tensor(ref_spk) + items.append( + VoiceClonePromptItem( + ref_code=ref_code, + ref_spk_embedding=ref_spk, + x_vector_only_mode=bool(d.get("x_vector_only_mode", False)), + icl_mode=bool(d.get("icl_mode", not bool(d.get("x_vector_only_mode", False)))), + ref_text=d.get("ref_text"), + ) + ) + return items + + +@app.post("/v1/voice-prompt") +async def build_voice_prompt( + slug: str = Form(...), + ref_text: str | None = Form(None), + ref_audio: UploadFile | None = File(None), + x_vector_only: int = Form(0), + force: int = Form(0), +): + """Build and cache a speaker's digital voice (a set of numbers) once. + + Two capture modes (pick per pipeline; a speaker's cached prompt is used as-is + on every subsequent utterance): + + * x_vector_only=0 (default, ICL/"8k matrix"): extracts the FULL codec-token + matrix (ref_code) + speaker embedding. Together those let the generator + re-create the exact reference recording -- and every utterance therefore + re-speaks the reference first, then trims it (slow + leaky). + * x_vector_only=1: captures ONLY the speaker embedding (the digital voice + variables). ref_code is None. Generation then feeds just those numbers + + text into the speech generator -- no reference re-hash, no clone re-run, + fast and output is exactly the text. + + Idempotent: if `.pt` is already cached it returns without rebuilding, + unless `force=1` is passed (use force to switch an existing prompt mode). + """ + path = _prompt_path(slug) + if path.exists() and not force: + return {"ok": True, "slug": slug, "cached": True, "path": str(path)} + + if ref_audio is None: + raise HTTPException(400, "ref_audio is required to build a voice prompt") + ref_bytes = await ref_audio.read() + + ref_text = (ref_text or "").strip() or VOICE_REFERENCE_TEXT + xvec = bool(x_vector_only) + + def _build() -> list[VoiceClonePromptItem]: + model = _load_model("base") + ref_audio_array, ref_sr = sf.read(io.BytesIO(ref_bytes), dtype="float32") + if ref_sr != 24000: + import librosa + ref_audio_array = librosa.resample( + ref_audio_array, orig_sr=ref_sr, target_sr=24000 + ) + items = model.create_voice_clone_prompt( + ref_audio=(ref_audio_array, 24000), + ref_text=ref_text, + x_vector_only_mode=xvec, + ) + return items + + _mark_gen_started() + async with _model_lock: + items = await anyio.to_thread.run_sync(_build) + _mark_gen_done() + + _save_prompt(slug, items) + return {"ok": True, "slug": slug, "path": str(_prompt_path(slug))} + + +@app.get("/health") +async def health(): + return { + "status": "ok", + "voicedesign_loaded": _vd_model is not None, + "base_loaded": _base_model is not None, + "current_model": _current_model, + "device": DEVICE, + } + + +@app.post("/load") +async def load_model(request: LoadRequest): + if request.model_type not in ("voicedesign", "base"): + raise HTTPException(400, "model_type must be 'voicedesign' or 'base'") + + try: + async with _model_lock: + await anyio.to_thread.run_sync(_load_model, request.model_type) + return {"ok": True, "model": request.model_type} + except Exception as e: + logger.exception("Failed to load model") + raise HTTPException(500, f"Failed to load model: {e}") from e + + +def _unload_model(model_type: str) -> None: + global _vd_model, _base_model, _current_model + + if model_type in ("voicedesign", "all") and _vd_model is not None: + del _vd_model + _vd_model = None + if model_type in ("base", "all") and _base_model is not None: + del _base_model + _base_model = None + + if model_type == "all" or _current_model == model_type: + _current_model = None + + if torch.mps.is_available() or torch.cuda.is_available(): + torch.mps.empty_cache() if torch.mps.is_available() else torch.cuda.empty_cache() + + +@app.post("/unload") +async def unload_model(request: LoadRequest): + if request.model_type not in ("voicedesign", "base", "all"): + raise HTTPException(400, "model_type must be 'voicedesign', 'base', or 'all'") + + async with _model_lock: + await anyio.to_thread.run_sync(_unload_model, request.model_type) + + return {"ok": True, "unloaded": request.model_type} + + +class AttnRequest(BaseModel): + impl: str # "eager", "sdpa", "fluent", "flash_attention_2" + + +@app.get("/attn") +async def get_attn(): + return {"ok": True, "impl": _ATTN_IMPL} + + +@app.post("/attn") +async def set_attn(request: AttnRequest): + """Switch the attention implementation and reload the active model. + + The attention implementation is fixed at model-load time, so this unloads + the current model(s) and reloads `base` with the new implementation. Callers + should switch to ``eager`` for prompt extraction (Phase 1) and ``sdpa`` for + generation (Phase 2) without a process restart. + """ + global _ATTN_IMPL + impl = (request.impl or "").strip().lower() + if impl not in ("eager", "sdpa", "fluent", "flash_attention_2"): + raise HTTPException(400, "impl must be eager|sdpa|fluent|flash_attention_2") + + async with _model_lock: + try: + await anyio.to_thread.run_sync(_unload_model, "all") + _ATTN_IMPL = impl + # reload so subsequent use has the new attn; base is the main model + await anyio.to_thread.run_sync(_load_model, "base") + except Exception as e: + logger.exception("Failed to switch attn to %s", impl) + raise HTTPException(500, f"Failed to switch attn: {e}") from e + + return {"ok": True, "impl": _ATTN_IMPL} + + +@app.post("/v1/tts") +async def generate_tts( + text: str = Form(...), + sample_rate: int = Form(24000), + voice_description: str | None = Form(None), + ref_audio: UploadFile | None = File(None), + ref_text: str | None = Form(None), + speaker_embedding: str | None = Form(None), + prompt_slug: str | None = Form(None), + temperature: float = Form(0.9), + do_sample: bool = Form(True), + seed: int | None = Form(None), +): + """Generate TTS audio. + + - voice_description + text -> VoiceDesign generation + - ref_audio + text + ref_text -> VoiceClone generation + - speaker_embedding + text -> Direct embedding generation (fastest, most consistent) + - prompt_slug -> ICL generation from cached voice-clone prompt (most faithful & + consistent; the "8k matrix" built by POST /v1/voice-prompt) + + prompt_slug takes priority: if the cached prompt exists it is used, giving + the speaker the identical reference voice on every utterance. + """ + gen_kwargs: dict = {"temperature": temperature, "do_sample": do_sample} + + if seed is not None: + torch.manual_seed(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) + + _mark_gen_started() + async with _model_lock: + if prompt_slug: + items = _load_prompt_items(prompt_slug) + if items is None: + raise HTTPException( + 404, f"No cached voice prompt for slug '{prompt_slug}'" + ) + + def _icl() -> tuple[np.ndarray, int]: + model = _load_model("base") + wavs, sr = model.generate_voice_clone( + text=text, + voice_clone_prompt=items, + language="english", + **gen_kwargs, + ) + return wavs, sr + + wavs, sr = await anyio.to_thread.run_sync(_icl) + elif speaker_embedding is not None: + emb_list = json.loads(speaker_embedding) + emb_np = np.array(emb_list, dtype=np.float32) + spk_tensor = torch.from_numpy(emb_np) + + def _embed() -> tuple[np.ndarray, int]: + model = _load_model("base") + device = next(model.model.parameters()).device + dtype = next(model.model.parameters()).dtype + spk = spk_tensor.to(device).to(dtype) + + wavs, sr = model.generate_voice_clone( + text=text, + voice_clone_prompt={ + "ref_code": [None], + "ref_spk_embedding": [spk], + "x_vector_only_mode": [True], + "icl_mode": [False], + }, + language="english", + **gen_kwargs, + ) + return wavs, sr + + wavs, sr = await anyio.to_thread.run_sync(_embed) + elif ref_audio is not None: + ref_bytes = await ref_audio.read() + + def _clone() -> tuple[np.ndarray, int]: + model = _load_model("base") + ref_audio_array, ref_sr = sf.read(io.BytesIO(ref_bytes), dtype="float32") + + if ref_sr != sample_rate: + import librosa + + ref_audio_array = librosa.resample( + ref_audio_array, orig_sr=ref_sr, target_sr=sample_rate + ) + + wavs, sr = model.generate_voice_clone( + text=text, + ref_audio=(ref_audio_array, sample_rate), + ref_text=None, + x_vector_only_mode=True, + language="english", + **gen_kwargs, + ) + return wavs, sr + + wavs, sr = await anyio.to_thread.run_sync(_clone) + elif voice_description: + def _design() -> tuple[np.ndarray, int]: + model = _load_model("voicedesign") + wavs, sr = model.generate_voice_design( + text=text, + instruct=voice_description, + language="english", + **gen_kwargs, + ) + return wavs, sr + + wavs, sr = await anyio.to_thread.run_sync(_design) + else: + raise HTTPException(400, "Either voice_description or ref_audio required") + + _mark_gen_done() + if not wavs: + raise HTTPException(500, "Model returned no audio") + + # Return WAV as binary + wav_bytes = io.BytesIO() + sf.write(wav_bytes, wavs[0], sr, format="WAV") + wav_bytes.seek(0) + + return Response( + content=wav_bytes.read(), + media_type="audio/wav", + headers={"Content-Disposition": "attachment; filename=output.wav"}, + ) + + +if __name__ == "__main__": + import uvicorn + uvicorn.run(app, host="0.0.0.0", port=8881) diff --git a/scripts/reattribute_all.py b/scripts/reattribute_all.py new file mode 100644 index 0000000..796239e --- /dev/null +++ b/scripts/reattribute_all.py @@ -0,0 +1,147 @@ +#!/usr/bin/env python3 +"""Re-run LLM speaker attribution on existing utterances without re-segmenting. + +Preserves utterance IDs and mechanical segmentation; only updates speaker_name. +""" +from __future__ import annotations + +import sqlite3 +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.analysis.llm_attributor import LLMAttributor +from umbraaudio.config import AppConfig, LLMConfig +from umbraaudio.voices.model_coordinator import ModelCoordinator, PipelineStage + + +def main(): + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument( + "--project", + type=str, + default=None, + help="Project name or path", + ) + parser.add_argument( + "--chapters", + type=str, + default=None, + help="Comma-separated chapter IDs (e.g. '5,6,7')", + ) + parser.add_argument( + "--model", + type=str, + default="google/gemma-4-e4b", + help="LLM model name", + ) + parser.add_argument( + "--sleep", + type=float, + default=1.0, + help="Seconds to sleep between LLM batches", + ) + args = parser.parse_args() + + from umbraaudio.config import load_config + + cfg = load_config(args.project) + config = cfg.llm + + coordinator = ModelCoordinator(cfg) + if not coordinator.switch_to( + PipelineStage.CHAPTER_ANALYSIS, lm_model=args.model + ).get("ok"): + print("Failed to switch models to CHAPTER_ANALYSIS stage") + sys.exit(1) + + db_path = cfg.db_path + if not db_path.exists(): + print(f"Database not found: {db_path}") + return + + db = sqlite3.connect(str(db_path)) + db.row_factory = sqlite3.Row + + if args.chapters: + chapter_ids = [int(x.strip()) for x in args.chapters.split(",")] + placeholders = ",".join("?" * len(chapter_ids)) + chapters = db.execute( + f"SELECT id, title, chapter_index FROM chapters " + f"WHERE id IN ({placeholders}) ORDER BY chapter_index", + chapter_ids, + ).fetchall() + else: + chapters = db.execute( + "SELECT id, title, chapter_index FROM chapters ORDER BY chapter_index" + ).fetchall() + + print(f"Found {len(chapters)} chapters to attribute") + print(f"Model: {args.model}") + + attributor = LLMAttributor(config, model=args.model, data_dir=cfg.data_dir) + total_dialogue = 0 + total_narrator_left = 0 + + for ch in chapters: + ch_id = ch["id"] + title = ch["title"] or f"Chapter {ch['chapter_index']}" + + utts = db.execute( + "SELECT id, utterance_index, text, type, speaker_name " + "FROM utterances WHERE chapter_id=? ORDER BY utterance_index", + (ch_id,), + ).fetchall() + if not utts: + print(f" {title}: no utterances, skipping") + continue + + utt_list = [dict(u) for u in utts] + dialogue_before = sum(1 for u in utt_list if u.get("type") == "dialogue") + narrator_before = sum( + 1 + for u in utt_list + if u.get("type") == "dialogue" + and (u.get("speaker_name") or "narrator").lower() == "narrator" + ) + + attributed = attributor.attribute_speakers(utt_list, title, ch_id) + time.sleep(0.5) + + for u in attributed: + db.execute( + "UPDATE utterances SET speaker_name=? WHERE id=?", + (u.get("speaker_name", "Narrator"), u["id"]), + ) + db.commit() + + narrator_after = sum( + 1 + for u in attributed + if u.get("type") == "dialogue" + and (u.get("speaker_name") or "narrator").lower() == "narrator" + ) + total_dialogue += dialogue_before + total_narrator_left += narrator_after + print( + f" {title} (id={ch_id}): dialogue {dialogue_before}, " + f"narrator-left {narrator_before} -> {narrator_after}" + ) + + db.close() + + coordinator.switch_to(PipelineStage.IDLE) + + print( + f"\nTotal: {total_dialogue} dialogue utterances, " + f"{total_narrator_left} still attributed to narrator" + ) + print("Done!") + + +if __name__ == "__main__": + main() diff --git a/scripts/reindex_chapter6.py b/scripts/reindex_chapter6.py new file mode 100644 index 0000000..d492080 --- /dev/null +++ b/scripts/reindex_chapter6.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Re-index utterance_index values and fix remaining issues for Chapter 6.""" + +from __future__ import annotations + +import argparse +import sqlite3 +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config + +CHAPTER_ID = 6 + + +def reindex_and_fix(db_path: Path) -> None: + conn = sqlite3.connect(str(db_path)) + conn.row_factory = sqlite3.Row + + # Get all utterances for chapter 6 ordered by current utterance_index + utts = conn.execute( + "SELECT id, utterance_index, text, type, speaker_name, emotion, instruct " + "FROM utterances WHERE chapter_id=? ORDER BY utterance_index", + (CHAPTER_ID,), + ).fetchall() + + print(f"Found {len(utts)} utterances before re-indexing") + + # Re-index sequentially from 0 + for new_idx, u in enumerate(utts): + conn.execute( + "UPDATE utterances SET utterance_index=? WHERE id=?", + (new_idx, u["id"]), + ) + + conn.commit() + print("Re-indexed utterances 0 to", len(utts) - 1) + + # Now fix specific issues with correct new indices + # Get fresh data + utts = conn.execute( + "SELECT id, utterance_index, text, type, speaker_name, emotion, instruct " + "FROM utterances WHERE chapter_id=? ORDER BY utterance_index", + (CHAPTER_ID,), + ).fetchall() + + print("\n=== After re-indexing ===") + for u in utts: + if u["utterance_index"] in range(50, 70) or u["utterance_index"] in range(140, 160) or \ + u["utterance_index"] in range(180, 200) or u["utterance_index"] in range(210, 230) or \ + u["utterance_index"] in range(350, 370): + print(f" utt{u['utterance_index']:03d} [{u['type']}] {u['speaker_name']}: {u['text'][:80]}...") + + # Fix: "What ship is this?" should be Kainda (not Nanny) + # Find the utterance with this text + for u in utts: + if u["text"] == "What ship is this?" and u["speaker_name"] == "Nanny": + conn.execute( + "UPDATE utterances SET speaker_name=?, emotion=?, instruct=? WHERE id=?", + ("Kainda Walters Foster Murphy", "confused", "Speak with confusion and disorientation. Trembling, uncertain delivery.", + u["id"]), + ) + print(f"\nFixed utt{u['utterance_index']}: 'What ship is this?' -> Kainda Walters Foster Murphy (confused)") + + # Verify "Hello." is Nanny (the robot) + for u in utts: + if u["text"] == "Hello." and u["utterance_index"] > 180 and u["utterance_index"] < 190: + if u["speaker_name"] != "Nanny": + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE id=?", + ("Nanny", u["id"]), + ) + print(f"Fixed utt{u['utterance_index']}: 'Hello.' -> Nanny") + + # Verify "You are on the Deal Hunter." is Nanny + for u in utts: + if u["text"] == "You are on the Deal Hunter." and u["speaker_name"] != "Nanny": + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE id=?", + ("Nanny", u["id"]), + ) + print(f"Fixed utt{u['utterance_index']}: 'You are on the Deal Hunter.' -> Nanny") + + # Verify "There are no rescue ships..." is Nanny + for u in utts: + if "no rescue ships" in u["text"] and u["speaker_name"] != "Nanny": + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE id=?", + ("Nanny", u["id"]), + ) + print(f"Fixed utt{u['utterance_index']}: 'There are no rescue ships...' -> Nanny") + + # Verify utt 211 (now reindexed) "That was a missile..." is Nanny + for u in utts: + if "missile trying to disable" in u["text"] and u["speaker_name"] != "Nanny": + conn.execute( + "UPDATE utterances SET speaker_name=? WHERE id=?", + ("Nanny", u["id"]), + ) + print(f"Fixed utt{u['utterance_index']}: 'That was a missile...' -> Nanny") + + # Verify the scene break was removed (should be no <<<>>> utterances) + scene_breaks = conn.execute( + "SELECT id, utterance_index FROM utterances WHERE chapter_id=? AND text='<<<>>>'", + (CHAPTER_ID,), + ).fetchall() + if scene_breaks: + print(f"\nWARNING: {len(scene_breaks)} scene break utterances still exist!") + for sb in scene_breaks: + print(f" utt{sb['utterance_index']}: {sb['id']}") + else: + print("\nScene breaks removed successfully.") + + # Verify Willamina Wake yelling at "Get the princess on board!" + for u in utts: + if "Get the princess on board" in u["text"]: + emotion = u["emotion"] if u["emotion"] else "" + if u["speaker_name"] != "Willamina Wake" or emotion != "yelling": + conn.execute( + "UPDATE utterances SET speaker_name=?, emotion=?, instruct=? WHERE id=?", + ("Willamina Wake", "yelling", "Speak with authority and command. Firm, unwavering tone. LOUD.", + u["id"]), + ) + print(f"Fixed utt{u['utterance_index']}: 'Get the princess on board!' -> Willamina Wake (yelling)") + else: + print(f"utt{u['utterance_index']}: Willamina Wake yelling - already correct") + + # Verify Kainda pleading at "Help me, please." (the first one, utt ~147) + for u in utts: + if u["text"] == "Help me, please." and u["utterance_index"] < 160 and u["speaker_name"] != "Kainda Walters Foster Murphy": + conn.execute( + "UPDATE utterances SET speaker_name=?, emotion=?, instruct=? WHERE id=?", + ("Kainda Walters Foster Murphy", "pleading", "Speak with desperate pleading. Urgent, emotional.", + u["id"]), + ) + print(f"Fixed utt{u['utterance_index']}: 'Help me, please.' -> Kainda (pleading)") + elif u["text"] == "Help me, please." and u["utterance_index"] < 160: + # Check emotion is set + if u.get("emotion") != "pleading": + conn.execute( + "UPDATE utterances SET emotion=?, instruct=? WHERE id=?", + ("pleading", "Speak with desperate pleading. Urgent, emotional.", u["id"]), + ) + print(f"Fixed utt{u['utterance_index']}: Added pleading emotion") + + conn.commit() + + # Final verification + print("\n=== Final verification of key utterances ===") + utts = conn.execute( + "SELECT id, utterance_index, text, type, speaker_name, emotion, instruct " + "FROM utterances WHERE chapter_id=? ORDER BY utterance_index", + (CHAPTER_ID,), + ).fetchall() + + key_indices = [56, 57, 58, 147, 183, 188, 190, 191, 192, 213, 224, 352] + for u in utts: + if u["utterance_index"] in key_indices: + emotion = u["emotion"] if u["emotion"] else "" + emo = f" [{emotion}]" if emotion else "" + instruct = u["instruct"] if u["instruct"] else "" + inst = f" ({instruct[:40]}...)" if instruct else "" + print(f" utt{u['utterance_index']:03d} [{u['type']}] {u['speaker_name']}{emo}{inst}: {u['text'][:80]}...") + + conn.close() + print("\n=== Done ===") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Re-index and fix Chapter 6 utterances") + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + args = parser.parse_args() + cfg = load_config(args.project) + reindex_and_fix(cfg.db_path) diff --git a/scripts/remigrate_speakers.py b/scripts/remigrate_speakers.py new file mode 100644 index 0000000..e90288a --- /dev/null +++ b/scripts/remigrate_speakers.py @@ -0,0 +1,202 @@ +"""Deterministic re-merge of existing discovery chapter data with fixed logic. + +Regenerates .umbra/speakers.json from .umbra/discovery/discovery_pass1.json +chapter results using the corrected _merge_chapter_results (no non-speaker +promotion, shared-specific-alias dedup, LLM alias evidence respected). + +The discovery pass1 chapter results are unchanged (LLM per-chapter output); +only the deterministic merge/dedup post-processing is re-applied. +""" + +from __future__ import annotations + +import json +import shutil +import sys +from datetime import datetime +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.analysis.character_discovery import ( + ChapterSpeakers, + CharacterDiscoverer, +) +from umbraaudio.config import load_config + + +def main() -> None: + import argparse + + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + args = parser.parse_args() + + cfg = load_config(args.project) + data_dir = cfg.data_dir + pass1_path = data_dir / "discovery" / "discovery_pass1.json" + speakers_path = data_dir / "speakers.json" + + with open(pass1_path) as f: + pass1 = json.load(f) + + crs = [] + for cr in pass1.get("chapter_results", []): + crs.append( + ChapterSpeakers( + chapter_index=cr.get("chapter_index", 0), + chapter_title=cr.get("chapter_title", ""), + speaking_characters=cr.get("speaking_characters", []), + non_speaking_characters=cr.get("non_speaking_characters", []), + validation_notes=cr.get("validation_notes", []), + ) + ) + + cd = CharacterDiscoverer.__new__(CharacterDiscoverer) + cd.TITLE_WORDS = CharacterDiscoverer.TITLE_WORDS + + speakers = cd._merge_chapter_results(crs, pass1.get("book_title", ""), 0) + + has_narrator = any(s.is_narrator for s in speakers) + if not has_narrator: + from umbraaudio.analysis.character_discovery import ( + CommunicationStyle, + Demographics, + Speaker, + ) + + narrator = Speaker( + primary_name="Narrator", + aliases=["narrator", "the narrator"], + demographics=Demographics(type="Human", gender="Male"), + vocal_traits=["deep", "resonant", "calm", "authoritative"], + communication_style=CommunicationStyle( + primary_register="Formal", + style_shifts_or_notes=( + "Steady, even-toned throughout. Never rushes." + ), + ), + description=( + "Deep, resonant male voice. Calm, authoritative tone. " + "Rich baritone timbre, slow and deliberate pacing, " + "like a late-night radio host." + ), + is_narrator=True, + ) + speakers.insert(0, narrator) + + result = { + "speakers": [s.to_dict() for s in speakers], + "book_title": pass1.get("book_title", ""), + "total_chapters": pass1.get("total_chapters", 0), + "total_words": pass1.get("total_words", 0), + } + + if speakers_path.exists(): + backup = ( + data_dir + / "archives" + / "speakers_remigrated" + / f"{datetime.now():%Y%m%d_%H%M%S}.json" + ) + backup.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(speakers_path, backup) + print(f"Backed up previous speakers.json -> {backup}") + _port_enriched_fields(backup, result) + + with open(speakers_path, "w") as f: + json.dump(result, f, indent=2) + + print(f"Wrote {len(speakers)} speakers to {speakers_path}") + for s in speakers: + n = s.primary_name + if "kainda" in n.lower() or "capek" in n.lower(): + print(f" {n}: ch{s.chapters_appeared} aliases={s.aliases}") + + _reconcile_voice_profiles(data_dir, speakers) + + +def _port_enriched_fields(backup: Path, result: dict) -> None: + with open(backup) as f: + old = json.load(f) + + old_by_name = {s["primary_name"].lower(): s for s in old.get("speakers", [])} + old_by_alias: dict[str, dict] = {} + for s in old.get("speakers", []): + for alias in s.get("aliases", []): + old_by_alias.setdefault(alias.lower(), s) + + for speaker in result["speakers"]: + n = speaker["primary_name"] + candidates = [] + if n.lower() in old_by_name: + candidates.append(old_by_name[n.lower()]) + if n.lower() in old_by_alias: + candidates.append(old_by_alias[n.lower()]) + candidates.extend( + s + for key, s in old_by_name.items() + if key != n.lower() and key in n.lower() + ) + old_speaker = next( + (s for s in candidates if s.get("description") or s.get("sample_dialogue")), + candidates[0] if candidates else None, + ) + if not old_speaker: + continue + for field in ("description", "sample_dialogue"): + if not speaker.get(field) and old_speaker.get(field): + speaker[field] = old_speaker[field] + + +def _reconcile_voice_profiles(data_dir: Path, speakers: list) -> None: + profiles_path = data_dir / "voice_profiles.json" + if not profiles_path.exists(): + return + + with open(profiles_path) as f: + profiles = json.load(f) + + current_names = {s.primary_name for s in speakers} + + merged = {} + for s in speakers: + name = s.primary_name + if name in profiles: + merged[name] = profiles[name] + + alias_to_canonical: dict[str, str] = {} + for s in speakers: + for alias in s.aliases: + alias_to_canonical.setdefault(alias.lower(), s.primary_name) + + for old_name in list(profiles.keys()): + if old_name in merged: + continue + canonical = alias_to_canonical.get(old_name.lower()) + if canonical is None: + canonical = next( + ( + n + for n in current_names + if n.lower() != old_name.lower() + and old_name.lower() in n.lower() + ), + None, + ) + if canonical and canonical not in merged: + merged[canonical] = profiles[old_name] + elif canonical: + for key, value in profiles[old_name].items(): + if value and not merged[canonical].get(key): + merged[canonical][key] = value + + with open(profiles_path, "w") as f: + json.dump(merged, f, indent=2) + print(f"Reconciled voice profiles: {len(merged)} kept (was {len(profiles)})") + + +if __name__ == "__main__": + main() diff --git a/scripts/resegment_all.py b/scripts/resegment_all.py new file mode 100644 index 0000000..e4c72fa --- /dev/null +++ b/scripts/resegment_all.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Re-segment all chapters using mechanical splitter + LLM attribution.""" + +from __future__ import annotations + +import json +import sqlite3 +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.analysis.llm_attributor import LLMAttributor +from umbraaudio.config import LLMConfig +from umbraaudio.mapping.segmenter import Segmenter + + +def main(): + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument( + "--project", type=str, default=None, help="Project name or path" + ) + parser.add_argument( + "--chapters", type=str, default=None, help="Comma-separated chapter IDs (e.g. '5,6,7')" + ) + parser.add_argument("--model", type=str, default="gemma-4-12b-it-qat", help="LLM model name") + parser.add_argument( + "--mechanical-only", action="store_true", help="Skip LLM attribution, use mechanical only" + ) + args = parser.parse_args() + + from umbraaudio.config import load_config + + cfg = load_config(args.project) + config = cfg.llm + + db_path = cfg.db_path + if not db_path.exists(): + print(f"Database not found: {db_path}") + return + + db = sqlite3.connect(str(db_path)) + db.row_factory = sqlite3.Row + + if args.chapters: + chapter_ids = [int(x.strip()) for x in args.chapters.split(",")] + placeholders = ",".join("?" * len(chapter_ids)) + chapters = db.execute( + f"SELECT id, title, chapter_index, source_text FROM chapters " + f"WHERE id IN ({placeholders}) ORDER BY chapter_index", + chapter_ids, + ).fetchall() + else: + chapters = db.execute( + "SELECT id, title, chapter_index, source_text FROM chapters ORDER BY chapter_index" + ).fetchall() + + print(f"Found {len(chapters)} chapters to segment") + print(f"Model: {args.model}") + print(f"LLM attribution: {'disabled' if args.mechanical_only else 'enabled'}") + + # Load character names and alias map from speakers.json + speakers_path = cfg.data_dir / "speakers.json" + character_names = [] + alias_map = {} + if speakers_path.exists(): + with open(speakers_path) as f: + data = json.load(f) + for s in data.get("speakers", []): + primary = s.get("primary_name", "") + if not primary: + continue + character_names.append(primary) + for alias in s.get("aliases", []): + alias_map[alias] = primary + + print(f"Loaded {len(character_names)} character names, {len(alias_map)} aliases") + + seg = Segmenter(alias_map=alias_map) + attributor = LLMAttributor(config, model=args.model, data_dir=cfg.data_dir) if not args.mechanical_only else None + + total_utterances = 0 + + for ch in chapters: + ch_id = ch["id"] + title = ch["title"] or f"Chapter {ch['chapter_index']}" + source_text = ch["source_text"] + + old_count = db.execute( + "SELECT COUNT(*) as cnt FROM utterances WHERE chapter_id = ?", (ch_id,) + ).fetchone()["cnt"] + + # Phase 1: Mechanical segmentation + mechanical = seg.segment(source_text, ch_id, character_names, chapter_title=title) + + utterances = [] + for i, u in enumerate(mechanical): + speaker = u.get("character_name", "narrator") + if speaker is None: + speaker = "narrator" + + is_scene_break = u["text"].strip() in ("<<<>>>",) + + utterances.append( + { + "utterance_index": i, + "text": u["text"], + "type": u.get("type", "narration"), + "speaker_name": "scene_break" if is_scene_break else speaker, + } + ) + + # Phase 2: LLM attribution (if enabled) + if attributor and not args.mechanical_only: + utterances = attributor.attribute_speakers(utterances, title, ch_id) + + # Write to database + db.execute("DELETE FROM utterances WHERE chapter_id = ?", (ch_id,)) + + for u in utterances: + db.execute( + "INSERT INTO utterances " + "(chapter_id, utterance_index, text, type, speaker_name) " + "VALUES (?, ?, ?, ?, ?)", + (ch_id, u["utterance_index"], u["text"], u["type"], u["speaker_name"]), + ) + + db.commit() + total_utterances += len(utterances) + + print(f" {title}: {old_count} -> {len(utterances)} utterances") + + db.close() + + print(f"\nTotal: {total_utterances} utterances across {len(chapters)} chapters") + print("Done!") + + +if __name__ == "__main__": + main() diff --git a/scripts/run_llm_attribution.py b/scripts/run_llm_attribution.py new file mode 100644 index 0000000..f60097e --- /dev/null +++ b/scripts/run_llm_attribution.py @@ -0,0 +1,201 @@ +from __future__ import annotations + +import logging +from typing import Any + +from umbraaudio.analysis.llm_attributor import LLMAttributor +from umbraaudio.config import load_config +from umbraaudio.database import Database + +logger = logging.getLogger(__name__) + + +async def run_llm_attribution( + project_dir: str = ".", + chapter_ids: list[int] | None = None, +) -> dict[str, Any]: + """ + Run multi-pass LLM speaker attribution on chapters. + + This replaces the mechanical _attribute_speakers with LLM-based attribution + that includes context analysis, confidence scoring, and exception detection. + """ + from pathlib import Path + config = load_config(Path(project_dir)) + db = Database(config.db_path) + await db.connect() + + try: + # Get chapters to process + all_chapters = await db.get_chapters(1) + if chapter_ids: + chapters = [ch for ch in all_chapters if ch["id"] in chapter_ids] + else: + chapters = all_chapters + + # Initialize LLM attributor + attributor = LLMAttributor(config.llm, data_dir=config.data_dir) + + results = { + "chapters_processed": 0, + "total_utterances": 0, + "attributed": 0, + "exceptions_found": 0, + "quality_gate_passed": True, + } + + for ch in chapters: + chapter_id = ch["id"] + chapter_title = ch.get("title") or f"Chapter {ch['chapter_index'] + 1}" + + logger.info(f"Running LLM attribution for chapter {chapter_id}: {chapter_title}") + + # Get utterances from database + utterances = await db.get_utterances(chapter_id) + if not utterances: + logger.warning(f"Chapter {chapter_id}: no utterances found") + continue + + # Prepare chapter data for context analysis + chapter_data = { + "id": chapter_id, + "title": chapter_title, + "source_text": ch.get("source_text", ""), + "utterances": [dict(u) for u in utterances], + } + + # Run multi-pass LLM attribution + updated_utterances = attributor.attribute_speakers( + utterances=[dict(u) for u in utterances], + chapter_title=chapter_title, + chapter_id=chapter_id, + chapter_data=chapter_data, + ) + + # Update database with new attributions + for u in updated_utterances: + if u.get("type") == "dialogue" and "speaker_name" in u: + await db.update_utterance_speaker( + u["id"], + u["speaker_name"], + u.get("attribution_confidence", 0.5), + u.get("attribution_reasoning", ""), + u.get("emotion"), + u.get("emotion_confidence"), + ) + + # Populate exception reviews from confidence scores + await _populate_exceptions(db, chapter_id, updated_utterances) + + results["chapters_processed"] += 1 + results["total_utterances"] += len([u for u in updated_utterances if u.get("type") == "dialogue"]) + results["attributed"] += len([u for u in updated_utterances if u.get("type") == "dialogue" and u.get("speaker_name") != "narrator"]) + + # Check quality gate + quality_gate = await _check_quality_gate(db) + results["quality_gate_passed"] = quality_gate["passed"] + results["exception_rate"] = quality_gate["exception_rate"] + results["exceptions_found"] = quality_gate["total_exceptions"] + + logger.info(f"LLM attribution complete: {results}") + return results + + finally: + await db.close() + + +async def _populate_exceptions(db: Database, chapter_id: int, utterances: list[dict[str, Any]]) -> int: + """Populate exception_reviews table from low-confidence attributions.""" + import json + + total_added = 0 + + for u in utterances: + if u.get("type") != "dialogue": + continue + + confidence = u.get("attribution_confidence", 1.0) + if confidence >= 0.7: + continue + + # Check if already in exception_reviews + cursor = await db.conn.execute( + "SELECT id FROM exception_reviews WHERE utterance_id = ?", + (u["id"],) + ) + existing = await cursor.fetchone() + if existing: + continue + + # Determine ambiguity type + ambiguity_type = "low_confidence" + reasoning = u.get("attribution_reasoning", "") + if confidence == 0.0: + ambiguity_type = "no_confidence" + elif u.get("speaker_name") == "narrator": + ambiguity_type = "narrator_dialogue" + elif "pronoun" in reasoning.lower(): + ambiguity_type = "pronoun" + elif "role" in reasoning.lower() or "unnamed" in reasoning.lower(): + ambiguity_type = "unnamed_role" + elif "retroactive" in reasoning.lower(): + ambiguity_type = "retroactive_name" + elif "generic" in reasoning.lower(): + ambiguity_type = "generic_role" + + await db.conn.execute(""" + INSERT INTO exception_reviews ( + project_id, chapter_id, utterance_id, utterance_index, + current_speaker, attributed_speaker, confidence, + ambiguity_type, evidence, reasoning, alternatives, text_preview, status + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'pending') + """, ( + 1, chapter_id, u["id"], u["utterance_index"], + u.get("speaker_name", ""), u.get("speaker_name", ""), + confidence, + ambiguity_type, + json.dumps([reasoning] if reasoning else ["Low confidence attribution"]), + reasoning, + json.dumps([]), + u["text"][:200], + )) + total_added += 1 + + return total_added + + +async def _check_quality_gate(db: Database) -> dict[str, Any]: + """Check if exception rate exceeds 1% quality gate.""" + cursor = await db.conn.execute( + "SELECT COUNT(*) as cnt FROM utterances WHERE type = 'dialogue'" + ) + total_utterances = await cursor.fetchone() + cursor = await db.conn.execute( + "SELECT COUNT(*) as cnt FROM exception_reviews WHERE status = 'pending'" + ) + total_exceptions = await cursor.fetchone() + + total_utt = total_utterances["cnt"] if total_utterances else 0 + total_exc = total_exceptions["cnt"] if total_exceptions else 0 + exception_rate = (total_exc / total_utt * 100) if total_utt > 0 else 0 + + return { + "passed": exception_rate <= 1.0, + "exception_rate": exception_rate, + "total_exceptions": total_exc, + "total_utterances": total_utt, + "threshold": 1.0, + } + + +# CLI entry point +if __name__ == "__main__": + import asyncio + import sys + + chapter_ids = None + if len(sys.argv) > 1: + chapter_ids = [int(x) for x in sys.argv[1].split(",")] + + result = asyncio.run(run_llm_attribution(chapter_ids=chapter_ids)) + print(result) \ No newline at end of file diff --git a/scripts/run_voice_analysis.py b/scripts/run_voice_analysis.py new file mode 100644 index 0000000..72b669c --- /dev/null +++ b/scripts/run_voice_analysis.py @@ -0,0 +1,53 @@ +from __future__ import annotations + +import json +import logging +from pathlib import Path + +from umbraaudio.analysis.voice_analysis import VoiceAnalysisEngine +from umbraaudio.config import load_config + +logger = logging.getLogger(__name__) + + +def run_voice_analysis(project_dir: Path | None = None) -> None: + """Compute the objective voice-analysis report for a project.""" + project_dir = project_dir or Path(".") + cfg = load_config(project_dir) + + speakers_path = cfg.data_dir / "speakers.json" + speaker_names: set[str] = set() + if speakers_path.exists(): + with open(speakers_path) as f: + speaker_names = {s["primary_name"] for s in json.load(f).get("speakers", [])} + + engine = VoiceAnalysisEngine( + refs_dir=cfg.data_dir / "voices", + output_dir=cfg.data_dir / "analysis", + target_sample_rate=cfg.tts.sample_rate, + speaker_names=speaker_names, + ) + report = engine.analyze() + summary = report["summary"] + print( + f"Analyzed {summary['total']} voices: " + f"{summary['passed']} passed, {summary['failed']} failed, " + f"{summary['near_duplicate_count']} near-duplicate pairs" + ) + if report["near_duplicates"]: + print("Near-duplicate pairs:") + for pair in report["near_duplicates"]: + print(f" {pair['a']} <-> {pair['b']} ({pair['similarity']})") + if report["errors"]: + print("Errors:") + for name, err in report["errors"].items(): + print(f" {name}: {err}") + print(f"Report written to {engine.report_path}") + + +if __name__ == "__main__": + import sys + + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") + project = Path(sys.argv[1]) if len(sys.argv) > 1 else None + run_voice_analysis(project) diff --git a/scripts/start.sh b/scripts/start.sh new file mode 100755 index 0000000..e3cb2aa --- /dev/null +++ b/scripts/start.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# UmbraAudio Startup Script + +set -e + +echo "=== UmbraAudio Startup ===" + +# Check MySQL +echo "Checking MySQL..." +if ! mysqladmin ping -u root -p'MyQl#26Pas$' --silent 2>/dev/null; then + echo "Starting MySQL..." + sudo systemctl start mysql +fi +echo "MySQL: OK" + +# Check Redis +echo "Checking Redis..." +if ! redis-cli ping 2>/dev/null | grep -q PONG; then + echo "Starting Redis..." + sudo systemctl start redis-server +fi +echo "Redis: OK" + +# Ensure database exists +echo "Ensuring database..." +mysql -u root -p'MyQl#26Pas$' -e "CREATE DATABASE IF NOT EXISTS umbra_audio CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" 2>/dev/null +echo "Database: OK" + +# Run migrations +echo "Running migrations..." +cd /home/bm/Apps/UmbraAudio +alembic upgrade head 2>/dev/null +echo "Migrations: OK" + +echo "" +echo "=== Services Ready ===" +echo "Start API server: python -m umbra_audio.main" +echo "Start Celery worker: celery -A umbra_audio.tasks worker --loglevel=info" diff --git a/scripts/test_parser.py b/scripts/test_parser.py new file mode 100644 index 0000000..c5e69dd --- /dev/null +++ b/scripts/test_parser.py @@ -0,0 +1,329 @@ +#!/usr/bin/env python3 +""" +Test script for document parser and entity extraction. +Tests the parsing of DealHunterShip manuscript. +""" + +import json +import sys +from pathlib import Path + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbra_audio.agents.entity_extractor import ( + extract_characters_from_text, + merge_characters, +) +from umbra_audio.core.document_parser import ( + extract_dialogue_lines, + get_document_stats, + get_section_content_for_audio, + parse_manuscript, + validate_manuscript, +) + + +def test_document_parser(): + """Test the document parser on DealHunterShip.""" + print("=" * 80) + print("TESTING DOCUMENT PARSER") + print("=" * 80) + + # Read the manuscript + manuscript_path = Path(__file__).parent.parent / "DealHunterShip.txt" + if not manuscript_path.exists(): + print(f"ERROR: Manuscript not found at {manuscript_path}") + return False + + with open(manuscript_path, encoding='utf-8') as f: + text = f.read() + + print(f"\nManuscript loaded: {len(text)} characters") + + # Validate manuscript + print("\n1. Validating manuscript structure...") + validation = validate_manuscript(text) + print(f" Valid: {validation['valid']}") + print(f" Errors: {validation['errors']}") + print(f" Warnings: {validation['warnings']}") + print(f" Stats: {json.dumps(validation['stats'], indent=2)}") + + # Parse manuscript + print("\n2. Parsing manuscript...") + manuscript = parse_manuscript(text) + print(f" Title: {manuscript.title}") + print(f" Author: {manuscript.author}") + print(f" Sections found: {len(manuscript.sections)}") + + # Analyze sections + print("\n3. Section analysis:") + chapter_count = 0 + scene_count = 0 + for section in manuscript.sections: + if section.section_type == 'chapter': + chapter_count += 1 + print(f" Chapter {section.chapter_number}: {section.chapter_title[:50]}...") + print(f" Content length: {len(section.content)} chars") + print(f" Scenes: {section.scene_index}") + elif section.section_type == 'scene': + scene_count += 1 + + print(f"\n Total chapters: {chapter_count}") + print(f" Total scenes: {scene_count}") + + # Test audio content extraction + print("\n4. Testing audio content extraction...") + if manuscript.sections: + first_section = manuscript.sections[0] + audio_content = get_section_content_for_audio(first_section) + print(f" First section audio content length: {len(audio_content)} chars") + print(f" Preview: {audio_content[:200]}...") + + # Test dialogue extraction + print("\n5. Testing dialogue extraction...") + all_dialogues = [] + for section in manuscript.sections: + dialogues = extract_dialogue_lines(section) + all_dialogues.extend(dialogues) + + print(f" Total dialogue lines extracted: {len(all_dialogues)}") + + # Show sample dialogues + if all_dialogues: + print("\n Sample dialogues:") + for d in all_dialogues[:5]: + print(f" Speaker: {d['speaker']}") + print(f" Dialogue: {d['dialogue'][:80]}...") + print() + + # Get document stats + print("\n6. Document statistics:") + stats = get_document_stats(text) + print(f" Total characters: {stats['total_chars']}") + print(f" Total lines: {stats['total_lines']}") + print(f" Word count: {stats['word_count']}") + print(f" Chapters: {stats['chapter_count']}") + print(f" Scenes: {stats['scene_count']}") + print(f" Dialogue lines: {stats['dialogue_count']}") + + print("\n Top character mentions:") + for name, count in list(stats['character_mentions'].items())[:10]: + print(f" {name}: {count}") + + return validation['valid'] + + +def test_entity_extraction(): + """Test entity extraction on DealHunterShip.""" + print("\n" + "=" * 80) + print("TESTING ENTITY EXTRACTION") + print("=" * 80) + + # Read the manuscript + manuscript_path = Path(__file__).parent.parent / "DealHunterShip.txt" + if not manuscript_path.exists(): + print(f"ERROR: Manuscript not found at {manuscript_path}") + return False + + with open(manuscript_path, encoding='utf-8') as f: + text = f.read() + + print(f"\nManuscript loaded: {len(text)} characters") + + # Extract characters + print("\n1. Extracting characters...") + characters = extract_characters_from_text(text) + print(f" Characters found: {len(characters)}") + + # Show top characters + print("\n2. Top characters by dialogue count:") + for char in characters[:15]: + if char.dialogue_count > 0: + print(f"\n {char.canonical_name}:") + print(f" Dialogue lines: {char.dialogue_count}") + if char.aliases: + print(f" Aliases: {', '.join(char.aliases[:3])}") + if char.voice_clues: + print(f" Voice clues: {', '.join(char.voice_clues[:2])}") + if char.sample_dialogues: + print(f" Sample: {char.sample_dialogues[0][:60]}...") + + # Merge characters + print("\n3. Merging characters with aliases...") + merged = merge_characters(characters) + print(f" Characters after merge: {len(merged)}") + + # Save results + output_path = Path(__file__).parent.parent / "DealHunterShip_characters.json" + output_data = { + "book_title": "Deal Hunter", + "total_characters_found": len(merged), + "characters": [] + } + + for char in merged: + output_data["characters"].append({ + "canonical_name": char.canonical_name, + "aliases": char.aliases, + "is_speaker": char.is_speaker, + "dialogue_count": char.dialogue_count, + "voice_clues": char.voice_clues, + "sample_dialogues": char.sample_dialogues, + "qwen_voice_prompt": "" # To be filled by voice agent + }) + + with open(output_path, 'w', encoding='utf-8') as f: + json.dump(output_data, f, indent=2, ensure_ascii=False) + + print(f"\n Results saved to: {output_path}") + + return True + + +def test_combined_workflow(): + """Test the combined parser + extraction workflow.""" + print("\n" + "=" * 80) + print("TESTING COMBINED WORKFLOW") + print("=" * 80) + + # Read the manuscript + manuscript_path = Path(__file__).parent.parent / "DealHunterShip.txt" + if not manuscript_path.exists(): + print(f"ERROR: Manuscript not found at {manuscript_path}") + return False + + with open(manuscript_path, encoding='utf-8') as f: + text = f.read() + + # Parse manuscript + print("\n1. Parsing manuscript...") + manuscript = parse_manuscript(text) + print(f" Sections: {len(manuscript.sections)}") + + # Extract characters + print("\n2. Extracting characters...") + characters = extract_characters_from_text(text) + print(f" Characters: {len(characters)}") + + # Match characters to sections + print("\n3. Matching characters to sections...") + character_sections = {} + + for char in characters[:10]: # Top 10 characters + char_sections = [] + for section in manuscript.sections: + if char.canonical_name in section.content or \ + any(alias in section.content for alias in char.aliases): + char_sections.append({ + 'chapter': section.chapter_number, + 'scene': section.scene_index, + 'title': section.chapter_title + }) + character_sections[char.canonical_name] = char_sections + + print("\n Character appearances by chapter:") + for char_name, sections in character_sections.items(): + if sections: + chapter_nums = sorted(set(s['chapter'] for s in sections)) + print(f" {char_name}: Chapters {chapter_nums}") + + # Generate voice prompts + print("\n4. Generating voice prompts...") + voice_prompts = {} + + for char in characters[:10]: + if char.is_speaker and char.voice_clues: + # Simple voice prompt generation based on clues + gender = "female" if any(word in char.canonical_name.lower() for word in ['princess', 'kainda', 'vicky', 'nanny', 'willamina']) else "male" + age = "young" if any(word in char.voice_clues for word in ['young', 'youth']) else "mature" + + voice_prompts[char.canonical_name] = { + "gender": gender, + "age": age, + "clues": char.voice_clues[:3], + "prompt": f"A {age} {gender} voice with {', '.join(char.voice_clues[:2])} characteristics." + } + + print(f" Voice prompts generated: {len(voice_prompts)}") + + # Save combined results + output_path = Path(__file__).parent.parent / "DealHunterShip_analysis.json" + output_data = { + "manuscript": { + "title": manuscript.title, + "author": manuscript.author, + "sections": len(manuscript.sections), + "chapters": len([s for s in manuscript.sections if s.section_type == 'chapter']), + }, + "characters": [ + { + "name": char.canonical_name, + "aliases": char.aliases, + "is_speaker": char.is_speaker, + "dialogue_count": char.dialogue_count, + "voice_clues": char.voice_clues, + "voice_prompt": voice_prompts.get(char.canonical_name, {}) + } + for char in characters[:20] + ], + "character_sections": character_sections + } + + with open(output_path, 'w', encoding='utf-8') as f: + json.dump(output_data, f, indent=2, ensure_ascii=False) + + print(f"\n Analysis saved to: {output_path}") + + return True + + +def main(): + """Run all tests.""" + print("UmbraAudio Document Parser & Entity Extraction Tests") + print("=" * 80) + + results = {} + + try: + results['document_parser'] = test_document_parser() + except Exception as e: + print(f"ERROR in document parser test: {e}") + results['document_parser'] = False + + try: + results['entity_extraction'] = test_entity_extraction() + except Exception as e: + print(f"ERROR in entity extraction test: {e}") + results['entity_extraction'] = False + + try: + results['combined_workflow'] = test_combined_workflow() + except Exception as e: + print(f"ERROR in combined workflow test: {e}") + results['combined_workflow'] = False + + # Summary + print("\n" + "=" * 80) + print("TEST SUMMARY") + print("=" * 80) + + all_passed = True + for test_name, passed in results.items(): + status = "PASS" if passed else "FAIL" + print(f" {test_name}: {status}") + if not passed: + all_passed = False + + print("\n" + "=" * 80) + if all_passed: + print("ALL TESTS PASSED!") + else: + print("SOME TESTS FAILED!") + print("=" * 80) + + return 0 if all_passed else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/test_tts.py b/scripts/test_tts.py new file mode 100644 index 0000000..ad2fe82 --- /dev/null +++ b/scripts/test_tts.py @@ -0,0 +1,309 @@ +#!/usr/bin/env python3 +""" +Test script for Qwen3-TTS voice synthesis. +Tests the TTS server with character voice prompts. +""" + +import sys +import time +from pathlib import Path + +# Add parent directory to path +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbra_audio.core.tts_client import QwenTTSClient + + +def test_tts_connection(): + """Test connection to TTS server.""" + print("=" * 80) + print("TESTING TTS CONNECTION") + print("=" * 80) + + client = QwenTTSClient() + + try: + # Test with a simple text + print("\n1. Testing basic synthesis...") + start_time = time.time() + audio_data = client.synthesize_speech( + text="Hello, this is a test.", + voice_prompt="A clear, neutral voice." + ) + elapsed = time.time() - start_time + + print(f" Success! Audio data: {len(audio_data)} bytes") + print(f" Time taken: {elapsed:.2f} seconds") + + # Save the test audio + output_path = Path(__file__).parent.parent / "test_audio_basic.wav" + with open(output_path, 'wb') as f: + f.write(audio_data) + print(f" Saved to: {output_path}") + + return True + + except Exception as e: + print(f" Error: {e}") + return False + finally: + client.close() + + +def test_character_voices(): + """Test synthesis with character voice prompts.""" + print("\n" + "=" * 80) + print("TESTING CHARACTER VOICES") + print("=" * 80) + + client = QwenTTSClient() + + # Test characters with voice prompts + test_characters = [ + { + "name": "Princess Kainda", + "voice_prompt": "A young adult female in her early 20s with a clear, bright mid-range voice. Speaks with a curious, confident cadence and a refined, royal accent.", + "test_line": "I will not be trapped in my cabin any longer. I want to see what's out there." + }, + { + "name": "Captain Hunter", + "voice_prompt": "A middle-aged male in his 40s with a gravelly, low-pitched baritone. Speaks with a slow, deliberate pace, rough texture, and authoritative delivery.", + "test_line": "Orders, Captain. We need to move fast before the salvagers arrive." + }, + { + "name": "King Kabir", + "voice_prompt": "A senior male in his 60s with a resonant, regal baritone. Speaks with a slow, dignified pace and an authoritative, paternal tone.", + "test_line": "My daughter has always been headstrong. Perhaps that is what we need in these troubled times." + }, + { + "name": "Nanny (Robot)", + "voice_prompt": "A synthesized female voice with a smooth, neutral tone. Speaks with a steady, measured cadence and precise, matter-of-fact delivery.", + "test_line": "Princess, please put this on quickly. We are under attack by pirates." + }, + { + "name": "Aunt Vicky", + "voice_prompt": "A mature female in her 50s with a warm, concerned alto. Speaks with a gentle, maternal cadence and a worried, caring tone.", + "test_line": "Who is the captain of this ship? I need to speak with him immediately." + } + ] + + results = [] + + for i, char in enumerate(test_characters): + print(f"\n{i+1}. Testing {char['name']}...") + + try: + start_time = time.time() + audio_data = client.synthesize_speech( + text=char['test_line'], + voice_prompt=char['voice_prompt'] + ) + elapsed = time.time() - start_time + + # Save audio + safe_name = char['name'].replace(' ', '_').replace('(', '').replace(')', '') + output_path = Path(__file__).parent.parent / f"test_audio_{safe_name}.wav" + with open(output_path, 'wb') as f: + f.write(audio_data) + + results.append({ + "name": char['name'], + "success": True, + "audio_size": len(audio_data), + "time_seconds": elapsed, + "output_path": str(output_path) + }) + + print(f" Success! Audio: {len(audio_data)} bytes, Time: {elapsed:.2f}s") + print(f" Saved to: {output_path}") + + except Exception as e: + results.append({ + "name": char['name'], + "success": False, + "error": str(e) + }) + print(f" Error: {e}") + + client.close() + + # Summary + print("\n" + "=" * 80) + print("CHARACTER VOICE TEST SUMMARY") + print("=" * 80) + + successful = sum(1 for r in results if r['success']) + print(f"Successful: {successful}/{len(results)}") + + for r in results: + status = "PASS" if r['success'] else "FAIL" + print(f" {r['name']}: {status}") + if r['success']: + print(f" Size: {r['audio_size']} bytes, Time: {r['time_seconds']:.2f}s") + else: + print(f" Error: {r.get('error', 'Unknown')}") + + return successful == len(results) + + +def test_dialogue_extraction(): + """Test synthesis with actual dialogue from the manuscript.""" + print("\n" + "=" * 80) + print("TESTING DIALOGUE SYNTHESIS") + print("=" * 80) + + client = QwenTTSClient() + + # Real dialogue from DealHunterShip + dialogues = [ + { + "speaker": "Kainda", + "text": "Where is the ship that brought me? Did you fire on that ship?", + "voice_prompt": "A young adult female in her early 20s with a clear, bright mid-range voice. Speaks with a curious, confident cadence and a refined, royal accent." + }, + { + "speaker": "Hunter", + "text": "Orders, Captain.", + "voice_prompt": "A middle-aged male in his 40s with a gravelly, low-pitched baritone. Speaks with a slow, deliberate pace, rough texture, and authoritative delivery." + }, + { + "speaker": "Kabir", + "text": "Or I can let her run loose and watch the fun.", + "voice_prompt": "A senior male in his 60s with a resonant, regal baritone. Speaks with a slow, dignified pace and an authoritative, paternal tone." + } + ] + + results = [] + + for i, d in enumerate(dialogues): + print(f"\n{i+1}. Synthesizing {d['speaker']}'s dialogue...") + + try: + start_time = time.time() + audio_data = client.synthesize_speech( + text=d['text'], + voice_prompt=d['voice_prompt'] + ) + elapsed = time.time() - start_time + + # Save audio + output_path = Path(__file__).parent.parent / f"test_dialogue_{d['speaker'].lower()}.wav" + with open(output_path, 'wb') as f: + f.write(audio_data) + + results.append({ + "speaker": d['speaker'], + "success": True, + "audio_size": len(audio_data), + "time_seconds": elapsed + }) + + print(f" Success! Audio: {len(audio_data)} bytes, Time: {elapsed:.2f}s") + + except Exception as e: + results.append({ + "speaker": d['speaker'], + "success": False, + "error": str(e) + }) + print(f" Error: {e}") + + client.close() + + # Summary + print("\n" + "=" * 80) + print("DIALOGUE SYNTHESIS SUMMARY") + print("=" * 80) + + successful = sum(1 for r in results if r['success']) + print(f"Successful: {successful}/{len(results)}") + + for r in results: + status = "PASS" if r['success'] else "FAIL" + print(f" {r['speaker']}: {status}") + + return successful == len(results) + + +def test_voice_prompt_validation(): + """Test voice prompt validation.""" + print("\n" + "=" * 80) + print("TESTING VOICE PROMPT VALIDATION") + print("=" * 80) + + client = QwenTTSClient() + + test_prompts = [ + "A young adult female in her early 20s with a clear, bright mid-range voice. Speaks with a curious, confident cadence and a refined, royal accent.", + "A middle-aged male with a gravelly, low-pitched baritone.", + "This is a very long prompt that exceeds the forty word limit and should fail validation because it contains too many words for the voice synthesis system to handle properly.", + "A voice that sounds like a mysterious philosopher from ancient times." + ] + + for i, prompt in enumerate(test_prompts): + print(f"\n{i+1}. Validating prompt: {prompt[:50]}...") + result = client.validate_voice_prompt(prompt) + print(f" Valid: {result['valid']}") + print(f" Word count: {result['word_count']}") + if result['issues']: + print(f" Issues: {result['issues']}") + + client.close() + return True + + +def main(): + """Run all TTS tests.""" + print("UmbraAudio Voice Synthesis Tests") + print("=" * 80) + + results = {} + + try: + results['connection'] = test_tts_connection() + except Exception as e: + print(f"ERROR in connection test: {e}") + results['connection'] = False + + try: + results['character_voices'] = test_character_voices() + except Exception as e: + print(f"ERROR in character voice test: {e}") + results['character_voices'] = False + + try: + results['dialogue'] = test_dialogue_extraction() + except Exception as e: + print(f"ERROR in dialogue test: {e}") + results['dialogue'] = False + + try: + results['validation'] = test_voice_prompt_validation() + except Exception as e: + print(f"ERROR in validation test: {e}") + results['validation'] = False + + # Final summary + print("\n" + "=" * 80) + print("FINAL TEST SUMMARY") + print("=" * 80) + + all_passed = True + for test_name, passed in results.items(): + status = "PASS" if passed else "FAIL" + print(f" {test_name}: {status}") + if not passed: + all_passed = False + + print("\n" + "=" * 80) + if all_passed: + print("ALL TESTS PASSED!") + else: + print("SOME TESTS FAILED!") + print("=" * 80) + + return 0 if all_passed else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/specs/00-project-overview.md b/specs/00-project-overview.md new file mode 100644 index 0000000..4810757 --- /dev/null +++ b/specs/00-project-overview.md @@ -0,0 +1,172 @@ +# UmbraAudio — Project Overview & Architecture + +## Purpose + +UmbraAudio is a quality-first, multi-pass audiobook generation system that converts English manuscripts (EPUB, PDF, TXT, MD) into fully produced audiobooks using neural TTS. Each pass has quality gates. Quality is the primary concern; generation speed is secondary. + +Two acoustic-quality invariants run through every pass and are non-negotiable for release: + +1. **Narrator consistency** — the narrator's voice stays calm, even, and low-emphasis for the entire book, independent of how dramatic the underlying scene text is. +2. **Cast distinctiveness** — every character who shares dialogue proximity with another character must be *audibly* distinguishable, not just textually different. + +A third principle governs delivery: **punctuation and typography are prosody instructions, not decoration.** A question mark, an ellipsis, an em-dash interruption, or a shouted ALL-CAPS word must change how a line is spoken, not just how it reads. + +A book is a project. Its source, analysis, intermitent files, and outputs are kept seperate from other projects. + +## High-Level Pipeline + +``` +EPUB/PDF/TXT/MD + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ PASS 1a: PARSE │ +│ Parse manuscript into chapters and scense "<<<>>>"│ +│ Preserves emphasis markup + punctuation verbatim│ +│ Output: chapters table (source_text) │ +└─────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ PASS 1b: CHARACTER DISCOVERY │ +│ LLM chapter-by-chapter character extraction │ +│ Merge across chapters, deduplicate aliases │ +│ Narrator identified (not a character) │ +| identify characteristics/voice of each character +| Each voice must be unique for the listener. +│ Voice reference generation via Qwen3-TTS │ +│ Verification samples generated │ +│ Output: speakers.json, voice_assignments.json │ +└─────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ PASS 2: TWO-PHASE SEGMENTATION │ +│ Phase 1: Mechanical segmenter splits text │ +│ - Sentence splitting at .!? │ +│ - Dialogue detection via quotes │ +│ - Tag-based attribution (forward/backward) │ +│ - Scene break (<<<>>>) handling │ +│ Phase 2: LLM speaker attribution │ +│ - Attributes ALL dialogue (overrides guesses)│ +│ - Resolves pronouns and context │ +│ - Uses character profiles from speakers.json │ +| Uterances are narrator "Bob said," and then Bob speaking. +│ Output: utterances table │ +└─────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ PASS 3: AUDIO GENERATION │ +│ Qwen3-TTS Base → voice cloning from references │ +| Each character has a unique voice the listener can decern +│ Voice assignment from voice_assignments.json │ +│ Retry logic for rate limiting │ +│ Output: chapter_audio/ch{id}/*.wav │ +└─────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ PASS 4: REVIEW │ +│ Web UI for human review of: │ +│ - Character profiles and voice assignments │ +│ - Voice test samples │ +│ - Chapter audio with per-utterance playback │ +│ - Speaker attribution accuracy │ +│ Output: approved chapters │ +└─────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────┐ +│ EXPORT │ +│ Stitch chapter audio → full audiobook │ +│ Output: export/*.m4b, export/*.mp3 │ +└─────────────────────────────────────────────────┘ +``` + +## Runtime Data Layout + +``` +.umbra/ # Runtime data directory +├── project.db # SQLite database +├── speakers.json # Character profiles (61 speakers) +├── voice_assignments.json # Voice assignments (60 mappings) +├── voice_tests/ # Verification audio samples +│ ├── {speaker_name}.wav +│ └── ... +├── chapter_audio/ # Generated chapter audio +│ ├── ch5/ +│ │ ├── utt_0000.wav +│ │ └── ... +│ ├── ch6/ +│ └── ch7/ +├── analysis/ +│ └── input.txt # Extracted manuscript text +└── defects.json # Quality defect log +``` + +## Technology Stack + +| Layer | Technology | Version | +|-------|-----------|---------| +| Language | Python | 3.13+ | +| CLI | Click + Rich | 8.1+ / 13.0+ | +| Database | SQLite (aiosqlite) | 0.19+ | +| LLM | OpenAI-compatible (LM Studio) | 1.10+ | +| TTS | Qwen3-TTS 1.7B (local GPU) | 0.1+ | +| LLM Model | gemma-4-12b-it-qat (remote LM Studio, context_length=8192) | remote | +| Web | FastAPI + HTMX + Jinja2 | 0.104+ | +| Parsing | ebooklib, beautifulsoup4, PyMuPDF | 0.18+ / 4.0+ / 1.23+ | + +## GPU Requirements + +- **Qwen3-TTS 1.7B Base**: ~3.9GB VRAM (voice cloning) +- **Qwen3-TTS 1.7B VoiceDesign**: ~3.9GB VRAM (voice reference generation) +- **LM Studio (gemma-4-12b-it-qat)**: remote server; load with `context_length: 8192` (speaker attribution) +- **Note**: Only one Qwen3-TTS model can be loaded at a time (auto-swaps). LM Studio must be unloaded before TTS generation if VRAM is tight. + +## Module Structure + +``` +umbraaudio/ +├── cli.py # Click CLI (all commands) +├── config.py # Pydantic settings (AppConfig tree) +├── database.py # SQLite schema + async CRUD +├── orchestrator.py # Full pipeline coordinator +├── parsers/ # Manuscript parsing (EPUB, PDF, TXT, MD) +├── analysis/ # LLM character discovery, profiling +│ ├── character_discovery.py # Chapter-by-chapter LLM extraction +│ ├── llm_attributor.py # Two-phase speaker attribution +│ ├── llm_segmenter.py # LLM-based segmentation (legacy) +│ └── CHARACTER_DISCOVERY_PROMPT.md +├── mapping/ # Mechanical segmentation, speaker attribution +│ └── segmenter.py # Text splitting + tag-based attribution +├── voices/ # TTS engine and voice management +│ ├── qwen_tts.py # Qwen3-TTS engine (voice design + cloning) +│ ├── lmstudio_lifecycle.py # LM Studio VRAM management +│ ├── edge_tts.py # Edge-TTS wrapper (legacy) +│ ├── voice_designer.py # Voice reference generation pipeline +│ └── embeddings.py # Speaker embedding extraction +├── generation/ # Audio generation +├── web/ # FastAPI review UI +└── scripts/ # Standalone scripts + ├── resegment_all.py # Batch segmentation + attribution + ├── generate_voice_references.py # Voice ref generation + ├── generate_chapter_audio.py # Per-chapter audio + └── generate_all_audio.py # Background full generation +``` + +## Key Design Decisions + +1. **Quality over speed**: Each pass has quality gates; the pipeline can stop and wait for human review +2. **Two-phase segmentation**: Mechanical segmenter handles text splitting (narration by sentence; quoted dialogue kept whole), LLM handles speaker attribution — this division of labor produces a 0.05% dialogue-as-narrator error rate (1/1,882) +3. **LLM attributes all dialogue**: The LLM attributes every dialogue utterance, using the mechanical `current_speaker` as a hint and overriding it when wrong — ambiguous or incorrectly guessed lines get corrected +4. **Narrator is not a character**: Dedicated soft-spoken male voice (en-US-GuyNeural:-10), never participates in character discovery +5. **59 unique voices via Qwen3-TTS**: VoiceDesign generates unique voice profiles from character descriptions; VoiceClone maintains consistency across utterances using reference audio +6. **Character profiles in JSON**: speakers.json stores all character data; voice_assignments.json maps characters to voice references +7. **Scene break handling**: <<<>>> markers preserved as special utterances, LLM maintains context across scene breaks +8. **Web review at checkpoints**: HTMX-powered UI for human approval of voices, speaker assignments, and chapter audio +9. **Punctuation as prosody**: Question marks, ellipses, em-dashes, and ALL-CAPS words change how lines are spoken +10. **Cast-wide distinctiveness gate**: Characters must be audibly distinguishable, not just textually different +11. **Verification phrase**: Rainbow Passage (phonetically balanced paragraph covering all English phonemes) used for voice testing +12. **Author watermark**: "Author B.D. Murphy" embedded in final audiobook diff --git a/specs/01-parsing.md b/specs/01-parsing.md new file mode 100644 index 0000000..b413415 --- /dev/null +++ b/specs/01-parsing.md @@ -0,0 +1,113 @@ +# Spec: EPUB Parsing (Pass 1a) + +## Overview + +The parser converts manuscripts in EPUB, PDF, TXT, or MD format into a list of chapter objects with title and source text. All parsing logic lives in `umbraaudio/parsers/__init__.py`. + +## Entry Point + +```python +def parse_manuscript(path: Path, fmt: str) -> list[dict[str, str | None]] +``` + +Returns: `[{"title": str | None, "text": str}, ...]` + +## Format Detection + +```python +def detect_format(path: Path) -> str +``` + +| Extension | Format | Status | +|-----------|--------|--------| +| `.txt` | `"txt"` | Implemented | +| `.md`, `.markdown` | `"md"` | Implemented | +| `.epub` | `"epub"` | Implemented | +| `.pdf` | `"pdf"` | Implemented | + +Raises `ValueError` for unsupported formats. + +## EPUB Parser + +```python +def _parse_epub(path: Path) -> list[dict] +``` + +- Uses `ebooklib` to iterate over `ITEM_DOCUMENT` items +- Extracts text via `BeautifulSoup.get_text()` +- Title detection: first `h1`, `h2`, or `h3` tag in each document +- Groups consecutive items under the same heading into one chapter +- Falls back to item index if no heading found + +### Dependencies +- `ebooklib >= 0.18` +- `beautifulsoup4` + +## TXT Parser + +```python +def _parse_txt(path: Path) -> list[dict] +``` + +- Reads file as UTF-8 text +- Splits on newlines +- **Chapter detection**: Lines that are < 100 characters AND entirely UPPERCASE +- Groups text under each detected header into a chapter +- Title = the uppercase header line + +## MD Parser + +```python +def _parse_md(path: Path) -> list[dict] +``` + +- Reads file as UTF-8 text +- Splits on lines starting with `#` (any heading level) +- Title = text after stripping leading `#` characters and whitespace +- Groups content between headings into chapter text + +## PDF Parser + +```python +def _parse_pdf(path: Path) -> list[dict] +``` + +- Uses PyMuPDF (`fitz`) for text extraction +- Same ALL-UPPERCASE header detection as TXT parser +- Additional constraint: header must have ≤ 8 words +- Groups text between headers into chapters + +### Dependencies +- `PyMuPDF >= 1.23` + +## Integration with CLI + +The `init` command calls `parse_manuscript()`: + +```python +@main.command() +@click.argument("manuscript", type=click.Path(exists=True)) +@click.option("--name", "-n", help="Project name") +def init(ctx, manuscript, name): + fmt = detect_format(path) + chapters = parse_manuscript(path, fmt) + # Insert into DB: chapters table +``` + +## Output Schema + +Each chapter dict: +```python +{ + "title": str | None, # Chapter title (from heading) + "text": str, # Full chapter text +} +``` + +## Edge Cases + +- EPUBs with no headings: falls back to document item index as title +- TXT/PDF with no uppercase headers: entire file treated as one chapter +- Empty chapters: filtered out (no empty source_text) +- Encoding: UTF-8 assumed; no fallback for other encodings +- Nested headings (MD): only first `#` level triggers chapter break diff --git a/specs/02-analysis.md b/specs/02-analysis.md new file mode 100644 index 0000000..08f40db --- /dev/null +++ b/specs/02-analysis.md @@ -0,0 +1,332 @@ +# Spec: Analysis — BookNLP & Character Profiling (Pass 1b) + +## Overview + +Pass 1b extracts characters, dialogue, and coreferences from chapter text using BookNLP, then generates detailed voice profiles for each character using an LLM. Unusual terms get pronunciation guides. + +Critically, this pass is also where voice *characteristics* — soft-spoken, assertive, gravelly, and so on — are converted from free-text description into a small, fixed vocabulary (the **voice attribute taxonomy**). Pass 2 (Voice Design) and Pass 3 (Mapping) both consume this taxonomy directly rather than re-interpreting prose. This is what makes character voices reproducible and what lets the system check, mechanically, whether two characters are actually going to sound different from each other. + +The narrator is handled as a special case throughout this pass — see "Narrator Profile" below — because the narrator must not inherit the LLM's free-ranging interpretation that named characters get. + +## Components + +### 1. BookNLP Engine (`analysis/booknlp_engine.py`) + +#### Class: `BookNLPEngine` + +```python +class BookNLPEngine: + def __init__(self) -> None + async def analyze(self, text: str, output_dir: Path) -> dict +``` + +**Initialization:** +- Lazy-loads `BookNLP("en", model_params={"model": "small", "pipeline": "entity,coref,quote"})` +- Model params: entity recognition + coreference resolution + quote attribution + +**Analysis Flow:** +1. Writes text to `{output_dir}/input.txt` +2. Runs `self._nlp.process(input_file, output_dir, idd="input")` +3. Parses output files: + - `input.entities` — TSV: entity mentions (PER persons with coref IDs) + - `input.quotes` — TSV: dialogue with speaker attribution + +**Output Format:** +```python +{ + "characters": [ + { + "name": str, # Character name + "mention_count": int, # How many times mentioned + "coref_id": int, # Coreference cluster ID + "context": str, # First 200 chars of chapter text + } + ], + "quotes": [ + { + "speaker": str, # Speaker name (from BookNLP) + "text": str, # Quoted dialogue text + "char_id": int, # Character ID + } + ] +} +``` + +**Patching Required:** +- `booknlp/english/entity_tagger.py`: `load_state_dict(..., strict=False)` for transformers 5.x +- `booknlp/english/litbank_coref.py`: `load_state_dict(..., strict=False)` +- `booknlp/english/bert_qa.py`: `load_state_dict(..., strict=False)` + +### 2. Voice Attribute Taxonomy + +Every profiled character (and the narrator, via fixed defaults below) is described by a small set of enumerated dimensions. These are the only values Pass 2 and Pass 3 are allowed to branch on — free-text `voice_description` is for human review and TTS conditioning text, not for program logic. + +| Dimension | Allowed Values | Drives | +|---|---|---| +| `pitch_register` | `very_low`, `low`, `mid`, `high`, `very_high` | Sample-text style, embedding-collision grouping | +| `pace_baseline` | `very_slow`, `slow`, `moderate`, `brisk`, `fast` | Default `pace` in manifest before punctuation adjustment | +| `energy` | `subdued`, `soft_spoken`, `measured`, `assertive`, `commanding`, `volatile` | TTS `exaggeration`/`cfg_weight` range (Pass 2); how strongly punctuation cues scale (Pass 3) | +| `timbre` | `warm`, `breathy`, `nasal`, `gravelly`, `clear`, `resonant`, `thin` | Sample-text selection; collision grouping | +| `texture` | `smooth`, `raspy`, `clipped`, `melodic`, `monotone` | Sample-text selection | +| `formality` | `casual`, `conversational`, `formal`, `archaic` | Speech-pattern conditioning text | +| `emotional_baseline` | free-form short tag, e.g. `neutral_calm`, `weary_cynical`, `anxious`, `stoic`, `warm_optimistic` | Default `instruct` framing in Pass 3 | +| `distinguishing_trait` | one short phrase, e.g. `"drawls vowels"`, `"clips sentences short"`, `"trails off when uncertain"` | Human-review summary; tie-breaker in collision review | + +`accent_marker` is optional free text (e.g. `"received_pronunciation"`, `"none"`) and is only ever populated when the text itself gives clear evidence (e.g. explicit dialect spelling, a stated nationality with described speech). It is never inferred from a character's name alone, and the profiler must not invent accents that play into ethnic or national stereotyping without direct textual support — if evidence is thin, `accent_marker` stays `"none"`. + +### 3. Character Profiler (`analysis/character_profiler.py`) + +#### Class: `CharacterProfiler` + +```python +class CharacterProfiler: + def __init__(self, llm_config: LLMConfig) -> None + async def profile_character(self, name, aliases, gender, context) -> CharacterProfile + async def check_distinctiveness( + self, profile: CharacterProfile, existing: list[CharacterProfile] + ) -> DistinctivenessResult + async def generate_pronunciation_guide(self, terms, context) -> dict[str, str] +``` + +**LLM Provider Selection (auto mode):** +1. Try LM Studio at `config.llm.lmstudio_base_url/models` (default `http://192.168.123.14:1234/v1`, overridable via `LLMConfig.lmstudio_base_url`); the model is `config.llm.lmstudio_model` (default `gemma-4-12b-it-qat` — must be loaded with `context_length: 8192`), and the configured model is preferred over whatever the server has loaded +2. Fall back to OpenAI (requires `OPENAI_API_KEY`) +3. Raise `RuntimeError` if neither available + +**Lifecycle note:** the `lmstudio_lifecycle` module resolves the same URL from config via `load_config().llm.lmstudio_base_url` for server/status operations, with `http://localhost:1234/v1` only as a hardcoded fallback when config cannot be loaded. Standalone scripts (`reattribute_all.py`, `resegment_all.py`, `analyze_emotions.py`) still hardcode the LAN URL (`http://192.168.123.14:1234/v1`) directly; keep the config default and the script constant in sync. + +**Character Profile Prompt:** +``` +You are an expert audiobook director. Analyze the following character +from a novel and create a detailed voice profile for text-to-speech generation. + +Character name: {name} +Known aliases: {aliases} +Gender (inferred): {gender} +Context from the book: +{context} + +Already-cast voices in this book (avoid sounding like any of these): +{existing_cast_summary} + +Create a JSON profile with these fields: +- personality: Brief personality description (1-2 sentences) +- voice_description: Detailed voice description for TTS voice design + (include gender, age, accent, tone, pitch characteristics) +- emotion_tendencies: List of 3-5 emotions this character commonly expresses +- speech_patterns: How this character speaks (formal/casual, sentence structure, etc.) +- narrator_style: If this character were narrating, what style would they use? +- pitch_register: one of [very_low, low, mid, high, very_high] +- pace_baseline: one of [very_slow, slow, moderate, brisk, fast] +- energy: one of [subdued, soft_spoken, measured, assertive, commanding, volatile] +- timbre: one of [warm, breathy, nasal, gravelly, clear, resonant, thin] +- texture: one of [smooth, raspy, clipped, melodic, monotone] +- formality: one of [casual, conversational, formal, archaic] +- emotional_baseline: short tag, e.g. "weary_cynical" +- distinguishing_trait: one short phrase +- accent_marker: only set if the text gives direct evidence; otherwise "none" + +Return ONLY valid JSON, no other text. +``` + +`existing_cast_summary` is a compact list of `{name, pitch_register, energy, timbre}` for every character already profiled in this book (built incrementally as profiling proceeds), so the LLM has a chance to avoid an obvious collision before one is even generated. + +**LLM Settings:** +- Model: GPT-4o (or local LM Studio model) +- Temperature: 0.3 +- Max tokens: 4096 +- Context truncated to 2000 chars + +**Response Parsing:** +1. Try full JSON parse +2. Fallback: extract first `{...}` block from response +3. **Enum validation:** any taxonomy field whose value is not in its allowed set is coerced to a defined fallback (the dimension's middle/neutral value, e.g. `energy → measured`, `pace_baseline → moderate`) and a warning is logged with character name + bad value. The pipeline never raises on this — a bad enum degrades to a safe default rather than blocking the run. +4. Returns `CharacterProfile` dataclass + +#### Cast-Aware Distinctiveness Check + +This is a text-level pre-check before any audio is generated; it catches the cheap, obvious collisions before Pass 2's more expensive embedding-based check runs. + +```python +@dataclass +class DistinctivenessResult: + is_distinct: bool + colliding_with: list[str] # character names + matched_dimensions: list[str] # which taxonomy fields matched +``` + +Algorithm: +1. Compare the new profile's `(pitch_register, pace_baseline, energy, timbre, texture, formality)` tuple against every existing profiled character who **co-occurs** with this one — defined as: both have at least one BookNLP-attributed quote within the same chapter. +2. If **4 or more of the 6 dimensions match exactly** with any co-occurring character, flag a collision. +3. On collision: re-prompt the LLM once with an explicit instruction — "Character X already has {colliding dimensions}; revise this character so it differs in at least two of: pitch_register, energy, timbre, texture, while staying consistent with the text evidence." Maximum 2 retries. +4. If still colliding after retries, accept the profile but mark `characters.status = 'collision_pending'` — this surfaces in the Pass 2 quality gate (see `03-voice-design.md`) rather than blocking Pass 1b. + +Characters who never share a chapter with attributed dialogue are not compared against each other — two minor characters in different parts of the book sounding similar is not a defect worth flagging. + +### 4. Narrator Profile — Required Defaults + +The narrator's `CharacterProfile` is **not** produced by the generic LLM character-profiling call above. It is constructed directly with fixed values, because the narrator's job is to disappear behind the prose, not to have a distinct dramatic personality: + +```python +NARRATOR_DEFAULT_PROFILE = CharacterProfile( + name="narrator", + is_narrator=True, + pitch_register="mid", + pace_baseline="moderate", + energy="measured", + timbre="clear", + texture="smooth", + formality="conversational", + emotional_baseline="neutral_calm", + distinguishing_trait="even, unhurried, minimal emphasis", + accent_marker="none", + voice_description=( + "A calm, clear, even-toned narrator. Minimal emotional coloring, " + "no shouting, no dramatic swings in pace or pitch. Reads " + "descriptive prose steadily regardless of how dramatic the " + "events being described are." + ), +) +``` + +This profile is **never** sent through `check_distinctiveness` against itself, and is excluded from the LLM prompt's "already-cast voices" list (the narrator is an anchor, not a competing cast member). + +**First-person exception:** if BookNLP/coref analysis indicates the book is narrated in first person by a named character (the narrator *is* a character in the story, e.g. a memoirist or first-person protagonist), that character still gets a full, normally-profiled `CharacterProfile` with `is_narrator=True` — they are allowed a real personality. However, the calm/measured ceiling above still applies specifically to their **narration-type** utterances (descriptive/expository prose); their **dialogue-type** utterances (lines they speak as a character within remembered scenes) use their full taxonomy-driven range. This split is enforced in Pass 3 (see `04-mapping.md`, "Narrator Constraint"), not here — Pass 1b's job is only to set `is_narrator=True` correctly. + +### 5. Character Profile Dataclass (`analysis/__init__.py`) + +```python +@dataclass +class CharacterProfile: + name: str + aliases: list[str] = field(default_factory=list) + gender: str | None = None + age_range: str | None = None + is_narrator: bool = False + personality: str | None = None + voice_description: str | None = None + emotion_tendencies: list[str] = field(default_factory=list) + speech_patterns: str | None = None + narrator_style: str | None = None + + # Voice attribute taxonomy (see "Voice Attribute Taxonomy" above) + pitch_register: str = "mid" + pace_baseline: str = "moderate" + energy: str = "measured" + timbre: str = "clear" + texture: str = "smooth" + formality: str = "conversational" + emotional_baseline: str = "neutral_calm" + distinguishing_trait: str | None = None + accent_marker: str = "none" +``` + +Defaults match the narrator baseline deliberately — an unprofiled character (e.g. a one-line walk-on the profiler skipped) renders as plain and unremarkable rather than as an error or as something randomly dramatic. + +#### Taxonomy Revision (Post-Hoc) + +A character's taxonomy is not frozen once Pass 1b completes. It can be revised later — specifically when a human selects **"Always"** on a detected speech descriptor during Pass 3's review queue (e.g. a line tagged "with his raspy voice" gets promoted from a one-off note to a permanent trait). See `04-mapping.md`, "Speech Descriptor Extractor & Review Queue." + +That revision is never a direct field write. It always routes through the same `apply_edit_prompt()` mechanism described in `03-voice-design.md`: + +1. The detected phrase seeds an edit prompt (human can refine the wording before submitting) +2. The LLM translates it into specific taxonomy deltas (e.g. `timbre: clear → gravelly`) +3. The character's voice reference is regenerated against the new taxonomy +4. The **cast-aware distinctiveness check re-runs against the full cast** — not just a sanity check, but a requirement: changing one character's `timbre` from `clear` to `gravelly` can create a brand-new collision with a different character who was previously safely distinct +5. `characters.status` drops from `voiced_approved` back to a pending-review state for the duration of this loop, exactly as if Pass 2 were running for the first time on this character +6. Only on re-approval does the revised taxonomy become the character's new permanent baseline + +This keeps the system honest about a real trade-off: a transient state ("her voice was hoarse from crying in this one scene") should never silently become a permanent trait. Promoting something to "Always" costs exactly as much process as the original profiling did, on purpose. + +### 6. Pronunciation Generator (`analysis/pronunciation.py`) + +#### Class: `PronunciationGenerator` + +```python +class PronunciationGenerator: + def __init__(self, llm_config: Any) -> None + def find_unusual_terms(self, text: str) -> list[str] + def phonetic_to_ssml(self, word: str, phonetic: str) -> str +``` + +**Term Detection Algorithm (3 regex passes):** +1. **Proper nouns:** `[A-Z][a-z]+(\s[A-Z][a-z]+)*` — filters common words (The, And, But, For, Not) +2. **Foreign words:** Words containing accented characters (é, ñ, ü, etc.) +3. **Hyphenated words:** Words containing hyphens + +**Note — not the same as shouted ALL-CAPS:** this generator is for unusual/foreign *pronunciation*, not for vocal emphasis. In-line ALL-CAPS words inside dialogue (e.g. `"NO!"`) are a Pass 3 prosody signal (see `04-mapping.md`), not a pronunciation-guide entry. Do not route ALL-CAPS dialogue words into `find_unusual_terms`. + +**Pronunciation via LLM:** +- Prompt: "Give IPA pronunciation for: {terms}" +- Temperature: 0.1 (low for accuracy) +- Context truncated to 1000 chars + +**SSML Output:** +```xml +{word} +``` + +## Orchestrator Flow + +```python +# In orchestrator.py run_pipeline(): +async def _run_analysis(db, project_dir, chapters_db): + # 0. Assign the fixed narrator profile (no LLM call) + narrator_profile = NARRATOR_DEFAULT_PROFILE + await db.upsert_character_profile("narrator", narrator_profile) + + # 1. BookNLP per chapter + for ch in chapters_db: + result = booknlp.analyze(ch["source_text"], analysis_dir) + + # 2. Merge characters across chapters + merged = _merge_characters(raw_characters, all_quotes) + + # 3. Profile each character via LLM, checking distinctiveness as we go + cast_so_far: list[CharacterProfile] = [] + for char in merged[:30]: + profile = await profiler.profile_character(...) + result = await profiler.check_distinctiveness(profile, cast_so_far) + if not result.is_distinct: + profile = await profiler.profile_character(..., retry_hint=result) + cast_so_far.append(profile) + await db.update_character_profile(char_id, profile) + + # 4. Pronunciation guide for unusual terms + guide = await profiler.generate_pronunciation_guide(terms, context) +``` + +## Character Merging Algorithm + +```python +def _merge_characters(raw_characters: list[dict], quotes: list[dict]) -> list[dict] +``` + +1. **Filter**: Remove pronouns (I, you, he, she, it, we, they...) and NOISE_CHARACTERS (people, everyone, someone, pirates, warbots...) +2. **Filter**: Remove single-character names +3. **Merge by lowercase**: Group variants (e.g., "Princess", "princess", "PRINCESS" → one entry) +4. **Keep highest mention count** as primary +5. **Build alias groups**: Case-variant duplicates become aliases +6. **Attach quote context**: First 200 chars from associated quotes +7. **Sort by mention count** (descending) + +## Noise Characters (Excluded from Voice Design) + +```python +NOISE_CHARACTERS = { + "people", "everyone", "someone", "anyone", "nobody", "something", + "nothing", "everything", "the fighters", "the fugitives", "the pirates", + "Warbots", "the Warbots", "pirates", +} +``` + +## Database Updates + +| Stage | Table | Fields Set | +|-------|-------|------------| +| Narrator assigned | `characters` | fixed taxonomy fields, `is_narrator=True`, `status='profiled'` (skips `'identified'`) | +| Characters extracted | `characters` | `name`, `aliases`, `gender`, `age_range`, `status='identified'` | +| Profiled via LLM | `characters` | `profile_json`, `voice_description`, taxonomy fields, `status='profiled'` | +| Distinctiveness flagged | `characters` | `status='collision_pending'` (surfaced again in Pass 2 gate) | +| Pronunciation | `pronunciation_guide` | `term`, `phonetic`, `notes` | +| Voice designed | `characters` | `voice_ref_path`, `speaker_embedding`, `status='voiced'` | diff --git a/specs/03-voice-design.md b/specs/03-voice-design.md new file mode 100644 index 0000000..70d7354 --- /dev/null +++ b/specs/03-voice-design.md @@ -0,0 +1,266 @@ +# Spec: Voice Design (Pass 2) + +## Overview + +Pass 2 generates unique voice reference audio for each character using Chatterbox TTS. Each character's TTS parameters (`exaggeration`, `cfg_weight`) are now derived in two steps rather than one: first, the character's **voice attribute taxonomy** (from Pass 1b) selects a parameter *range* appropriate to how that character is actually described; second, an MD5 hash of the name picks a deterministic point *within* that range, so re-running the pipeline always reproduces the same voice while two characters who share the same `energy` value still land at different points in the range. + +The narrator is the one character whose parameters are **not** range-and-hash derived at all — they are fixed constants, because the narrator must sound the same in every project, every run, forever (see "Narrator Parameters" below). + +This pass also adds a hard quality gate: after every voice reference is generated, the system checks whether any two co-occurring characters are acoustically too similar, and blocks the handoff to Pass 3 until that's resolved or a human explicitly overrides it. + +## Components + +### Voice Designer (`voices/voice_designer.py`) + +```python +class VoiceDesigner: + def __init__(self, config: TTSConfig) -> None + + def generate_voice_reference( + self, + character: CharacterProfile, + output_dir: Path, + sample_text: str | None = None, + ) -> Path + + def generate_all_references( + self, + characters: list[CharacterProfile], + output_dir: Path, + ) -> dict[str, Path] + + def check_cast_distinctiveness( + self, + characters: list[CharacterProfile], + embeddings: dict[str, bytes], + cooccurrence: dict[str, set[str]], + ) -> list[VoiceCollision] +``` + +### Voice Reference Generation Flow + +``` +CharacterProfile + │ + ├── _character_params(character) → (exaggeration, cfg_weight) + │ energy → base range (ATTRIBUTE_PARAM_MAP) + │ MD5 hash of name → deterministic position within that range + │ narrator → fixed constants, taxonomy/hash ignored entirely + │ + ├── _generate_sample_text(character) → str + │ Selected from taxonomy fields (energy, timbre, pitch_register), + │ falling back to voice_description keyword sniffing only for + │ legacy profiles created before the taxonomy existed + │ + └── ChatterboxEngine.generate( + text=sample_text, + output_path={name}_ref.wav, + voice_ref=None, # No prior ref for initial generation + instruct=voice_description, # Text conditioning (not forwarded to model) + exaggeration=exag, + cfg_weight=cfg, + ) + │ + └── ChatterboxTTS.generate(text, audio_prompt_path=None, exaggeration, cfg_weight) + │ + └── Perth watermarker (if available) + │ + └── Write WAV via soundfile.write() +``` + +### Attribute-Driven Parameter Ranges + +`energy` is the primary driver of expressive intensity. `cfg_weight` runs inversely to `exaggeration`: higher `cfg_weight` means the model adheres closely to a neutral, reference-like delivery; lower `cfg_weight` paired with higher `exaggeration` gives a more emphatic, expressive delivery. A calm, measured character should have low exaggeration and high cfg_weight; a volatile, commanding character should have the opposite. + +```python +ATTRIBUTE_PARAM_MAP: dict[str, tuple[tuple[float, float], tuple[float, float]]] = { + # energy: (exaggeration_range, cfg_weight_range) + "subdued": ((0.15, 0.30), (0.55, 0.70)), + "soft_spoken": ((0.20, 0.35), (0.45, 0.60)), + "measured": ((0.30, 0.45), (0.40, 0.55)), + "assertive": ((0.45, 0.60), (0.30, 0.45)), + "commanding": ((0.55, 0.70), (0.20, 0.35)), + "volatile": ((0.60, 0.80), (0.15, 0.30)), +} +``` + +### Per-Character Parameter Derivation + +```python +def _character_params(self, character: CharacterProfile) -> tuple[float, float]: + if character.is_narrator and character.name == "narrator": + return NARRATOR_EXAGGERATION, NARRATOR_CFG_WEIGHT + + exag_range, cfg_range = ATTRIBUTE_PARAM_MAP[character.energy] + name_hash = int(hashlib.md5(character.name.encode()).hexdigest()[:8], 16) + + exaggeration = exag_range[0] + ((name_hash % 100) / 100.0) * (exag_range[1] - exag_range[0]) + cfg_weight = cfg_range[0] + (((name_hash >> 8) % 100) / 100.0) * (cfg_range[1] - cfg_range[0]) + return exaggeration, cfg_weight +``` + +**Example Values:** + +| Character | `energy` | MD5 Prefix | Exaggeration | Cfg Weight | +|-----------|----------|-----------|--------------|------------| +| narrator | — (fixed) | — | **0.25 (fixed)** | **0.65 (fixed)** | +| Princess Kainda | assertive | a3f7... | 0.512 | 0.341 | +| Captain | commanding | 4b2e... | 0.598 | 0.247 | +| Deal Hunter | measured | 8c1d... | 0.368 | 0.452 | +| Nanny | soft_spoken | f5a3... | 0.248 | 0.512 | +| The robot | subdued | 7e9b... | 0.198 | 0.612 | +| Aunt Vicky | measured | 2d4c... | 0.402 | 0.481 | + +Values are illustrative; exact figures depend on the live MD5 digest, but every character's pair now falls inside the band their `energy` attribute defines — an "assertive" character can no longer accidentally land in subdued-character territory the way pure name-hashing allowed. + +### Narrator Parameters + +```python +NARRATOR_EXAGGERATION = 0.25 +NARRATOR_CFG_WEIGHT = 0.65 +``` + +These are constants, not derived from anything character-specific — explicitly so the narrator sounds identical across every book and every pipeline run, which is the acoustic anchor the rest of the cast is heard against. Do not route the narrator through `ATTRIBUTE_PARAM_MAP` or the hash function, even though `NARRATOR_DEFAULT_PROFILE.energy == "measured"` would otherwise resolve to a valid range — the explicit override always wins for `character.name == "narrator"`. + +### Sample Text Generation + +```python +def _generate_sample_text(self, character: CharacterProfile) -> str +``` + +Primary path: select sample-text style from `(character.energy, character.timbre, character.pitch_register)` directly — deterministic, no string matching. + +| `energy` | `timbre` | Sample Text Style | +|---|---|---| +| `subdued` / `soft_spoken` | any | Quiet, understated first-person line | +| `measured` | `gravelly`, `nasal`, `monotone`-adjacent | Wise elder / contemplative narration | +| `subdued` | `monotone`, `clear` | Flat, mechanical/robotic delivery | +| `volatile` / `commanding` | any | Assertive dramatic narration | +| any, `pitch_register=high` + youthful context | — | Enthusiastic, energetic dialogue | +| narrator (fixed) | — | Descriptive landscape prose, even pace | + +Legacy fallback (only used when a profile predates the taxonomy and taxonomy fields are still at their defaults): substring-match `voice_description` for keywords (`female/woman/girl/she`, `old/elder/gruff`, `robot/machine/synth`, `child/young`, `narrator`), same behavior as before. + +### Output + +- **File**: `{output_dir}/{character_name_lower}_ref.wav` +- **Format**: WAV, 24kHz sample rate, mono +- **Size**: Typically 100KB–500KB depending on text length + +### Voice Reference vs Chapter Generation + +| Aspect | Voice Reference | Chapter Generation | +|--------|----------------|-------------------| +| `voice_ref` param | `None` | Path to reference WAV | +| Purpose | Create initial voice identity | Clone voice to new text | +| `exaggeration`/`cfg_weight` | From taxonomy range + name hash (or narrator constants) | Same values, reused | +| `instruct` param | `voice_description` | `None` (dropped) | +| Watermark | Applied | Applied | + +### Speaker Embedding Extraction + +After voice reference generation, an embedding is extracted: + +```python +embedding = tts.extract_embedding(ref_path) +# Returns: bytes (numpy array serialized via .tobytes()) +# Stored in: characters.speaker_embedding column +``` + +Uses Chatterbox's internal `encode_speaker()` or `encode_spk()` method. + +### Cast Distinctiveness Quality Gate + +This is the acoustic counterpart to Pass 1b's text-level distinctiveness pre-check, and it is the gate that's allowed to actually block the pipeline. + +```python +@dataclass +class VoiceCollision: + character_a: str + character_b: str + similarity: float + remediation_attempts: int + resolved: bool +``` + +Algorithm: +1. After all references are generated and embeddings extracted, compute pairwise cosine similarity across every character pair, **including the narrator** as a fixed comparison point (no character should sound like the narrator either). +2. For any pair with `similarity > 0.92` **and** who co-occur (per the same co-occurrence definition used in Pass 1b — share an attributed-dialogue chapter): flag a `VoiceCollision`. +3. **Automatic remediation** (up to 3 attempts per flagged character): re-derive parameters using a disambiguated hash input — `hashlib.md5(f"{character.name}#{attempt}".encode())` — keeping the same `ATTRIBUTE_PARAM_MAP` range, regenerate the reference, re-extract the embedding, re-check. +4. If still colliding after 3 attempts: write an entry to `.umbra/defects.json` (`type: "voice_collision"`, both names, similarity score) and set both characters' `status` to `collision_pending` if not already. +5. **Pipeline behavior**: the orchestrator does not proceed to Pass 3 while any `collision_pending` status remains, unless the operator passes `--allow-voice-collisions` (logged in the project's run history) or resolves it via the web review UI. + +This mirrors the project's "quality over speed" principle (`00-project-overview.md`): an unresolved collision between two characters who actually talk in the same scene is treated as a release blocker, not a cosmetic issue. + +### Human Approval Workflow (Required Gate) + +Passing the distinctiveness gate makes a voice *safe to listen to* — it doesn't make it *right*. A character described as "gruff and commanding" can still get a reference that technically sits in the right parameter range and is acoustically distinct from everyone else, while simply not sounding like what the author had in mind. So every character's reference — after the distinctiveness gate, not instead of it — goes through an explicit human checkpoint before it's allowed to become the voice used anywhere else in the pipeline. + +```python +class VoiceReviewSession: + def submit_for_review(self, character: CharacterProfile, ref_path: Path) -> ReviewItem + def approve(self, character_name: str) -> None + def reject(self, character_name: str) -> Path + async def edit_with_prompt(self, character_name: str, prompt: str) -> CharacterProfile +``` + +**Flow, per character:** + +1. **Generate** — as above: taxonomy-ranged params, hash-disambiguated, embedding extracted, distinctiveness-gated. +2. **Present** — queued in the web review UI: character name, `voice_description`, taxonomy summary, a playable reference WAV, and three actions. +3. **Human chooses one:** + +| Action | Effect | +|---|---| +| **Approve** | `status = 'voiced_approved'`. Parameters are locked as this character's "normal" baseline. Nothing else in the pipeline may change them except a future taxonomy revision (`02-analysis.md`, "Taxonomy Revision"). | +| **Reject** | Regenerate using a disambiguated hash variant (`name#attempt`, same mechanism as collision remediation) — same `ATTRIBUTE_PARAM_MAP` range, same described character, just a different roll within it. Re-extract embedding, re-queue. This is for "wrong roll of the dice," not "wrong direction." | +| **Edit with prompt** | Human supplies free text describing the desired change — "make it gruffer and slower," "less nasal, lower the pitch." Routed through `apply_edit_prompt()` below. | + +```python +async def apply_edit_prompt(character: CharacterProfile, prompt: str) -> CharacterProfile: + """LLM translates a free-text director's note into concrete taxonomy + deltas. Returns a candidate updated profile; does not mutate the + stored profile until the human approves the regenerated result.""" +``` + +``` +You are adjusting a TTS voice profile based on director feedback. + +Current profile: +{taxonomy_json} + +Director's note: "{prompt}" + +Return ONLY a JSON object containing the fields that should change, from: +pitch_register, pace_baseline, energy, timbre, texture, formality, +emotional_baseline, distinguishing_trait. Only include a field if the +note actually justifies changing it. Do not invent unrelated changes. +``` + +4. **On an edit prompt:** apply the returned deltas to a candidate profile → re-derive `(exaggeration, cfg_weight)` via `_character_params()` → regenerate the reference and sample text → re-extract the embedding → **re-run the cast distinctiveness check** (an edit can introduce a collision that didn't exist before — it isn't optional just because this is round two) → re-present per step 2. Loop continues until **Approve**. +5. **Pipeline gate:** Pass 3's manifest finalization and Pass 5 generation will not bind or render against any character whose `status != 'voiced_approved'`. Pass 3's text-level segmentation/attribution can still run in parallel since it doesn't touch audio — but nothing downstream of voice selection proceeds for an unapproved character. + +| Table | Fields | +|---|---| +| `voice_review_history` | `id`, `character_name`, `action` (`generated`/`approved`/`rejected`/`edit_requested`), `prompt_text` (nullable), `resulting_exaggeration`, `resulting_cfg_weight`, `taxonomy_snapshot_json`, `reviewed_by`, `reviewed_at` | + +This table is what later lets a Pass 3 "Always" promotion (`04-mapping.md`, "Speech Descriptor Extractor & Review Queue") reopen exactly this same loop on an already-approved character, with full history of how the voice got to where it is. + +Once approved, this is the "normal" voice referenced by everything downstream: Pass 3's punctuation-driven prosody hints modulate *around* it, and human-reviewed per-line descriptor overrides modulate around it too. Neither layer ever resets the baseline itself without going back through this same approval loop. + +### Skip Logic + +Voice references are not regenerated if: +1. `character.voice_ref_path` is set in DB +2. The referenced file exists on disk +3. `character.status == 'voiced_approved'` — anything else (`collision_pending`, mid-review, or rolled back by a Pass 3 "Always" promotion) is always regenerated, since skipping would silently re-ship a voice that's either known-broken or mid-revision + +This allows incremental pipeline runs. + +### GPU Requirements + +- Chatterbox model: ~3.2 GB VRAM +- One voice ref: ~3-8 seconds to generate +- 29 characters: ~2-4 minutes of GPU time total (plus remediation regenerations for any flagged collisions) +- The human approval loop adds wall-clock time, not GPU load, beyond what's already accounted for — each reject/edit cycle is just another single-reference regeneration +- Requires LM Studio to be unloaded first (VRAM conflict) diff --git a/specs/04-mapping.md b/specs/04-mapping.md new file mode 100644 index 0000000..4552793 --- /dev/null +++ b/specs/04-mapping.md @@ -0,0 +1,264 @@ +# Spec: Mapping — Two-Phase Segmentation (Pass 2) + +## Overview + +Pass 2 segments chapter text into utterances (narration and dialogue) and attributes speakers using a two-phase approach: a mechanical segmenter handles text splitting, and an LLM attributes speakers to all dialogue. + +This pass handles: +- Sentence splitting at `.` `!` `?` (not commas) +- Dialogue detection via quotes (`"'""`) +- Speaker attribution via dialogue tags and context +- Scene break (`<<<>>>`) handling +- Chapter title detection + +## Components + +### 1. Mechanical Segmenter (`mapping/segmenter.py`) + +#### Class: `Segmenter` + +```python +class Segmenter: + def segment( + self, + text: str, + chapter_id: int, + character_names: list[str] | None = None, + chapter_title: str | None = None, + ) -> list[dict[str, Any]] +``` + +Special chapter handling by `chapter_title`: chapters titled "copyright", "author's note", "back matter", and "title" are force-attributed to the narrator. + +**Returns:** +```python +[{ + "index": int, + "text": str, + "type": "narration" | "dialogue", + "character_name": str | None, +}, ...] +``` + +**Segmentation Pipeline:** + +``` +Raw chapter text + │ + ├── 1. _split_segments(text) → list[str] + │ Character-by-character parser + │ Detects quote boundaries and sentence endings + │ + ├── 2. _build_utterances(segments) → list[dict] + │ Tags each segment as "dialogue" or "narration" + │ based on leading quote character + │ + └── 3. _attribute_speakers(utterances, character_names) → list[dict] + Forward-tracking speaker attribution + Regex matching against character names +``` + +**Quote Preservation:** + +- A quoted dialogue span is extracted whole (`_extract_quote` scans to the closing quote) + and kept as a **single utterance even when it contains multiple sentences**. The inner + text is never split back into per-sentence re-quoted fragments — that split is what caused + multi-sentence speech to lose its single-speaker evidence. +- `_restore_quote_wrap` re-wraps the stripped inner text with typographic double quotes. It + only treats double-quote closes (`"`, `\u201d`) as outer closes, so a trailing **inner + single-quote** (`’`, e.g. `...named ‘Nanny.’`) is preserved instead of being mistaken for + the outer closing quote. + +**Quote Detection:** +- Open: `"`, `'`, `\u201c` (left double), `\u2018` (left single) +- Close: `"`, `'`, `\u201d` (right double), `\u2019` (right single) + +**Sentence Splitting:** +- Split at `.`, `!`, `?` followed by whitespace/quote/dash (narration only) +- Quote boundaries always create segment breaks +- Quoted dialogue is never split by sentence endings — the whole quote is one utterance + +**Speaker Attribution Algorithm:** + +1. **Tag Detection**: Find `NAME said/asked/replied` patterns +2. **Forward Tracking**: When tag found, set `next_speaker = NAME` +3. **Backward Reference**: When no tag, look back for `said NAME` patterns +4. **Name Normalization**: Case-insensitive matching, alias resolution + +**Speech Verbs (53 total):** +``` +said, asked, replied, whispered, shouted, muttered, called, +exclaimed, answered, began, continued, added, finished, noted, +stated, remarked, yelled, screamed, laughed, sighed, says, +tells, told, orders, ordered, demands, demanded, insists, +insisted, begs, begged, pleads, pleaded, commands, commanded, +snapped, snarled, hissed, growled, barked, roared, croaked, +grunted, murmured, purred, intoned, declared, announced, +proclaimed, responds, repeats, mentions, offers +``` + +### 2. LLM Speaker Attribution (`analysis/llm_attributor.py`) + +#### Class: `LLMAttributor` + +```python +class LLMAttributor: + def __init__(self, config: LLMConfig, model: str | None = None) -> None + + def attribute_speakers( + self, + utterances: list[dict[str, Any]], + chapter_title: str, + chapter_id: int, + ) -> list[dict[str, Any]] +``` + +**Key Insight:** The LLM attributes **all** dialogue utterances. The mechanical attribution from the segmenter is preserved only as a `current_speaker` hint for the LLM (rule 8) — the LLM's verdict takes precedence so that ambiguous or incorrectly guessed lines can be corrected. This differs from the original design where only `speaker_name == "narrator"` dialogue was attributed; the current implementation lets the LLM override mechanical guesses. + +**Attribution Flow:** + +1. **Filter**: Collect all dialogue utterances (`type == "dialogue"`) +2. **Batch**: Group into batches of 40 utterances +3. **Context**: Include surrounding utterances for context +4. **LLM Call**: Send batch with character profiles +5. **Parse Response**: Extract JSON array with index → speaker mapping +6. **Post-Process**: Strip speech verb suffixes ("says", "said", etc.) and run `_normalize_speaker` (alias/first-word resolution, narrator lowercase, pronoun guard) +7. **Update**: Apply LLM verdicts to all dialogue utterances +8. **Fallback**: Re-attribute any dialogue still un-attributed via + - **Speech-tag detection** on the 3 preceding utterances (regex `NAME says/said/asked/...`) + - **Speaker carry-forward** from the last active speaker within the same scene + - Scene breaks (`<<<>>>`) reset the carried speaker + +**Performance:** +- ~5 seconds per batch of 40 utterances with `gemma-4-12b-it-qat` (~34 tok/s on the remote LM Studio server) +- The model must be loaded with `context_length: 8192`; the default 262k-token context requests ~44.9 GB of memory and blocks loading +- Processes all dialogue utterances; deterministic fallback recovers any that the LLM misses (target 100% dialogue attribution) + +### 3. Emotion Mapping (`mapping/emotion.py`) + +#### Class: `EmotionMapper` + +```python +class EmotionMapper: + def __init__(self, llm_client: Any = None, model: str = "") -> None + def analyze_utterances(self, utterances, chapter_context="") -> list[dict] + def batch_generate(self, utterances, chapter_context="") -> list[dict] +``` + +Maps each utterance to TTS voice-direction metadata: `emotion`, `pace`, `emphasis_words` (max 3), `instruct`, `pause_before_ms`, `pause_after_ms`. + +**Batching:** utterances are processed in batches of 30 per LLM call. The output JSON array must match the input order; a length mismatch falls back to deterministic defaults for the whole batch. + +**Question Intonation (required):** +- Any utterance ending with `?` (including `?"`) is a question. +- The LLM prompt must instruct rising question intonation for question-ending utterances. +- The deterministic fallback (`_default_result`) also emits `instruct` = "Speak naturally with rising question intonation at the end." for question-ending dialogue when no LLM is available or a batch fails. +- This ensures every question has rising intonation even if the LLM is unavailable. + +**Defaults:** +- Narrator text → `emotion="neutral"`, `pace="moderate"`, `instruct="Read steadily and evenly. Low emphasis, consistent tone."` +- Non-question dialogue → `emotion="neutral"`, `pace="moderate"`, `instruct="Speak naturally."` +- All results default to `pause_before_ms=500`, `pause_after_ms=500`. + +**LLM Settings:** temperature 0.2, max_tokens 8192, 3 attempts with exponential backoff, robust JSON extraction (strip fenced blocks, brace/bracket trimming). + +### 4. Scene Break Handling + +**Scene Breaks (`<<<>>>`):** +- 51 scene breaks across the book +- Marked as special utterances: `{"speaker": "scene_break", "text": "<<<>>>"}` +- Within a scene: track which characters are present +- At a scene break: re-evaluate present characters +- Chapter context carries across scene breaks + +### 5. Name Normalization + +All LLM-returned speakers pass through `_normalize_speaker` before being stored: + +**Aliases → Primary Names (from `speakers.json`):** +- "Princess", "Kainda", "the princess" → "Kainda Walters Foster Murphy" +- "Captain", "Captain Wake" → "Willamina Wake" +- "Hunter", "Ship AI", "The Deal Hunter" → "Deal Hunter" (the ship's AI — one speaker) + +**First-word-of-primary mapping:** the first word of each primary name maps to the full +primary name when it is not already an alias (e.g. "kainda" → "Kainda Walters Foster +Murphy"). This fixes attributions that only match the primary name's first word. + +**Narrator lowercase:** "narrator" / "the narrator" resolve to the canonical lowercase +`narrator` (the DB and generation code check this exact case). + +**Pronoun guard:** pronouns ("she", "he", "they", "her", etc.) resolve to `narrator` +instead of fabricating a speaker — a bare pronoun (e.g. "She") is a real LLM failure +mode, never a character name. + +**Resolution:** +- Case-insensitive matching +- Alias map from `speakers.json` +- Fallback: unknown names are kept as-is and surfaced on the orphaned-utterances page for review + +## Output + +Utterances saved to `utterances` table: +```sql +INSERT INTO utterances ( + chapter_id, + utterance_index, + text, + type, + speaker_name +) VALUES (?, ?, ?, ?, ?) +``` + +## Quality Metrics + +| Metric | Value | +|--------|-------| +| Total utterances | 4,339 | +| Dialogue utterances | 1,882 | +| Narration utterances | 2,453 | +| Dialogue-as-narrator errors | 1 (0.05%) | +| Scene breaks | 51 | +| Chapters | 35 | + +## Scripts + +### resegment_all.py + +```bash +# Run on all chapters with LLM attribution +python scripts/resegment_all.py --model gemma-4-12b-it-qat + +# Run on specific chapters +python scripts/resegment_all.py --chapters 5,6,7 + +# Mechanical-only (no LLM) +python scripts/resegment_all.py --mechanical-only +``` + +### reattribute_all.py + +Re-runs LLM speaker attribution on existing utterances **without re-segmenting** (preserves utterance IDs and mechanical segmentation; only updates `speaker_name`): + +```bash +# All chapters +python scripts/reattribute_all.py --model gemma-4-12b-it-qat + +# Specific chapters (--sleep adds a delay between chapters to pace the server) +python scripts/reattribute_all.py --chapters 5,6,7 --sleep 0.2 +``` + +### analyze_emotions.py + +Re-runs emotion mapping (including question intonation) on all utterances in the database. + +## Key Design Decisions + +1. **Two-phase over one-phase**: Mechanical handles splitting reliably; LLM does attribution +2. **Quotes kept whole**: A quoted dialogue span stays one utterance even with multiple sentences (Rule 3d) — splitting it would lose the single-speaker evidence +3. **Inner-quote preservation**: `_restore_quote_wrap` only recognizes double-quote closes so a trailing inner single-quote (`’`) survives re-wrapping +4. **LLM attributes all dialogue**: The LLM's verdict overrides the mechanical `current_speaker` guess so ambiguous lines get corrected +5. **Deterministic fallback**: Speech-tag regex + speaker carry-forward guarantee un-attributed dialogue still gets a speaker +6. **Batch processing**: 40 utterances per LLM call for efficiency +7. **Scene break preservation**: `<<<>>>` markers maintained as special utterances; they reset the carried speaker +8. **Speaker name normalization**: Aliases and first-word-of-primary resolve to canonical primary names; narrator stays lowercase; pronouns map to narrator +9. **Question intonation is mandatory**: every `?`-ending utterance carries rising question intonation in its TTS instruct text, enforced in both the LLM prompt and the deterministic fallback diff --git a/specs/05-generation.md b/specs/05-generation.md new file mode 100644 index 0000000..ce9a18e --- /dev/null +++ b/specs/05-generation.md @@ -0,0 +1,184 @@ +# Spec: Audio Generation (Pass 3) + +## Overview + +Pass 3 generates audio for every utterance in every chapter using **Qwen3-TTS** (primary engine) with per-character **voice reference samples** (`*_ref.wav`). Each utterance gets its own WAV file. Generation resolves each utterance's speaker to a voice reference, then clones that voice for the utterance. Edge-TTS remains available for test samples and as a fallback path inside the voice designer. + +## Components + +### Qwen3-TTS Engine (`voices/qwen_tts.py`) + +```python +class QwenTTSEngine: + def __init__(self, device="cuda", sample_rate=24000, qwen_tts_url: str) -> None + def load_remote_model(self) -> None + def generate(self, text: str, output_path: Path, voice_ref: Path | None = None, instruct: str = "") -> Path + def unload_all(self) -> None +``` + +**Qwen3-TTS Features:** +- Remote/on-prem synthesis via `qwen_tts_url` (LM Studio / Qwen server) +- Voice cloning from a reference WAV per character +- Optional `instruct` text for prosody/emotion direction (e.g. rising question intonation) +- WAV output at 24kHz sample rate + +### Edge-TTS (`voices/edge_tts.py`) — Test Samples & Fallback + +```python +class EdgeTTSEngine: + def __init__(self) -> None + def generate(self, text: str, output_path: Path, voice: str, pitch_shift: int = 0) -> Path +``` + +- Used by `scripts/generate_test_samples.py` (verification phrase per speaker) and as the `_generate_edge_fallback` path in the voice designer. +- 27 distinct base voices (US, UK, AU, IE, IN, CA) with pitch shifting for variety. + +### Voice Reference Resolution + +```python +def resolve_reference( + speaker_name: str, + voice_refs: dict[str, Path], + alias_map: dict[str, str], + narrator_ref: Path | None = None, +) -> Path | None +``` + +`voice_refs` is built by scanning `.umbra/voices/*_ref.wav`; the key is the filename stem with underscores converted to spaces (e.g. `captain_weston_ref.wav` → `captain weston`). + +**Resolution Order:** +1. Direct match against `voice_refs` keys (speaker primary name) +2. Case-insensitive match against reference names +3. Alias map lookup (primary names + aliases from `speakers.json`, all lowercased) +4. Fall back to `narrator_ref` (the narrator's reference WAV); if no narrator ref exists, fall back to the Kainda reference + +**Fallback when no reference resolves at all:** the utterance is generated with a plain instruction (`"A calm, neutral voice speaking clearly."`) instead of a cloned voice. + +### Generation Flow + +``` +For each chapter: + │ + ├── 1. Load utterances from DB + │ SELECT id, speaker_name, text, utterance_index + │ FROM utterances WHERE chapter_id = ? ORDER BY utterance_index + │ + ├── 2. Create output directory + │ .umbra/chapter_audio/ch{id}/ + │ + └── 3. For each utterance: + │ + ├── Skip empty text + ├── Skip if file exists + │ if output_path.exists(): continue + │ + ├── Resolve voice reference + │ ref_path = resolve_reference(speaker, voice_refs, alias_map, narrator_ref) + │ + ├── Generate audio (Qwen3-TTS) + │ engine.generate(text, output_path, voice_ref=ref_path) + │ # no ref → engine.generate(text, output_path, instruct=neutral) + │ + ├── Verify output file exists (else count as failed) + │ + └── Handle errors + Log error, continue to next utterance +``` + +**Engine mode:** the engine uses remote synthesis when `cfg.tts.qwen_tts_url` is set (the default deployment). The remote server holds the VoiceDesign and Base models loaded and answers `/v1/tts`; the local `QwenTTSEngine` only loads models itself when no URL is configured. + +### Output Structure + +``` +.umbra/chapter_audio/ +├── ch5/ +│ ├── utt_0000.wav +│ ├── utt_0001.wav +│ └── ... (one WAV per utterance) +├── ch6/ +│ ├── utt_0000.wav +│ └── ... +└── ch7/ + ├── utt_0000.wav + └── ... +``` + +Each WAV is named `utt_{utterance_index:04d}.wav`, so filenames are stable across re-runs and regeneration after utterance edits deletes and recreates the matching file. + +### Remote Server Coordination + +- The engine checks VRAM on start (`nvidia-smi`) and logs used/free memory; the remote model is expected to be the only heavy VRAM consumer on the serving host. +- The remote server keeps the **VoiceDesign model** loaded so a *single* synthesis call can switch between voice-cloning (`ref_audio`) and description-driven (`voice_description`) generation without a model swap — the per-utterance Clone is what drives bulk generation, so the Base model stays resident. +- On completion, `unload_all()` is called only on the client side (remote models are left for the server's lifecycle manager). + +### Scripts + +#### generate_all_audio.py (bulk generation) + +```bash +# Background chapter audio generation +nohup python scripts/generate_all_audio.py > .umbra/generation.log 2>&1 & +``` + +**Behavior:** +- Loads voice references from `.umbra/voices/*_ref.wav` and speakers/aliases from `.umbra/speakers.json` +- Iterates chapters in `chapter_index` order, generating each missing utterance WAV +- Logs per-chapter progress (`+N gen, +M fail`), running total, and elapsed minutes +- Emits progress estimates assuming ~6.4s per utterance +- Exits with a final `COMPLETE: N generated, M failed` summary +- Accepts optional `status_callback` and `generation_manager` hooks so the web UI can display live progress (see `09-web-review-ui.md`) + +**Resume semantics:** because existing WAVs are skipped, re-running the script after a crash or interruption resumes exactly where it left off. + +## Voice Library + +### Voice References (`.umbra/voices/*_ref.wav`) + +Each speaker gets a reference WAV synthesized from the verification phrase *"When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. The rainbow is a division of white light into many beautiful colors..."* (the Rainbow Passage — a phonetically balanced paragraph covering all English phonemes) using their taxonomy-derived voice description (see `03-voice-design.md`). + +- Narrator → `narrator_ref.wav` +- General/misc speakers (from `GENERAL_SPEAKER_NAMES`) → `general_ref.wav` +- Named speakers → `{name_lower_spaces_to_underscores}_ref.wav` + +**Reference count in current deployment:** 92 speakers in `speakers.json`, ~26 unique reference WAVs generated (many characters intentionally share a reference or fall back to the narrator reference rather than each getting a distinct clone). + +### Voice Variety Strategy + +1. **Distinct voice descriptions**: each character's taxonomy (`gender`, `age`, `timbre`, `pitch_register`, `energy`, `texture`, `formality`, `accent`, `distinguishing_trait`) is rendered into a unique natural-language description via `build_voice_description()` +2. **Voice cloning**: the speaker's reference WAV is the identity anchor; clone synthesis preserves that identity across all utterances +3. **No pitch-shifting**: variety comes from the reference voice itself, not post-hoc pitch shifts + +## Error Handling + +- **Empty Text**: Skip utterances with no text +- **File Exists**: Skip if output file already exists +- **No Reference Resolved**: Fall back to narrator reference (or Kainda), else neutral instruction +- **TTS Failure** (`QwenTTSUnavailableError` — connect refused, timeout after 300s, non-200 HTTP, or response without audio): log and continue to next utterance +- **Per-utterance exception**: caught, logged, counted as failed — never aborts the chapter run + +## Performance + +| Metric | Value | +|--------|-------| +| Rate | ~6.4 seconds/utterance (remote synthesis) | +| Full book (6,861 utterances) | ~12 hours | +| Progress logging | Every 50 utterances, plus per-chapter summary | +| Resume | Idempotent via file-existence skip | + +Success rate is reported per run as `generated / failed`; failed utterances can be regenerated by re-running the script (only the missing files are produced). + +## Quality Considerations + +- All voices tested with verification phrase (test samples reviewed in the web UI) +- Web UI allows human review of generated audio per chapter (`/chapters-audio`) +- Utterance text/speaker edits in the web UI delete the stale WAV and trigger background regeneration +- Voice references can be regenerated individually or en masse from the web UI + +## Key Design Decisions + +1. **Qwen3-TTS over Edge-TTS**: cloning from per-character reference samples gives stable, character-accurate voices; Edge-TTS is retained only for test samples and the `_generate_edge_fallback` path in the voice designer +2. **Remote synthesis**: the Qwen server (`qwen_tts_url`) hosts the models, so bulk generation doesn't fight the LLM (LM Studio) for local GPU/VRAM +3. **Voice references over voice strings**: identity comes from `*_ref.wav` cloning instead of `voice_assignments.json` voice strings with pitch shifts +4. **Sequential generation**: avoids overloading the remote server and simplifies resumability +5. **File-based caching**: skip already-generated files; regeneration after edits is file-targeted +6. **Error resilience**: individual utterance failures never abort a chapter run; a final summary counts successes and failures diff --git a/specs/06-normalization-watermarking.md b/specs/06-normalization-watermarking.md new file mode 100644 index 0000000..7bd81fc --- /dev/null +++ b/specs/06-normalization-watermarking.md @@ -0,0 +1,204 @@ +# Spec: Normalization & Watermarking + +## Overview + +Post-generation processing includes audio normalization (RMS leveling, peak limiting) and neural watermarking (author identity embedded in audio). These run as part of the generation pipeline per-utterance. + +## Components + +### 1. Audio Normalizer (`assembly/normalize.py`) + +```python +class AudioNormalizer: + def __init__( + self, + target_rms_db: float = -20.0, + peak_limit_db: float = -3.0, + noise_floor_db: float = -60.0, + sample_rate: int = 24000, + ) -> None + + def normalize_file(self, input_path: Path, output_path: Path) -> dict + def check_acx_compliance(self, audio_path: Path) -> dict +``` + +#### Normalization Algorithm + +```python +def normalize_file(self, input_path: Path, output_path: Path) -> dict: + # 1. Read audio + audio, sr = sf.read(input_path) + + # 2. Mono downmix if stereo + if audio.ndim > 1: + audio = audio.mean(axis=1) + + # 3. Compute current RMS in dB + rms = np.sqrt(np.mean(audio ** 2)) + current_rms_db = 20 * np.log10(rms + 1e-10) + + # 4. Calculate gain + gain_db = self.target_rms_db - current_rms_db + gain = 10 ** (gain_db / 20) + audio = audio * gain + + # 5. Peak limiting + peak = np.max(np.abs(audio)) + peak_db = 20 * np.log10(peak + 1e-10) + if peak_db > self.peak_limit_db: + limiting_gain = 10 ** ((self.peak_limit_db - peak_db) / 20) + audio = audio * limiting_gain + + # 6. Clip to [-1.0, 1.0] + audio = np.clip(audio, -1.0, 1.0) + + # 7. Write output + sf.write(output_path, audio, self.sample_rate) +``` + +**Returns:** +```python +{ + "input_path": str, + "output_path": str, + "original_rms_db": float, + "normalized_rms_db": float, + "peak_db": float, + "gain_applied_db": float, +} +``` + +#### ACX Compliance Check + +```python +def check_acx_compliance(self, audio_path: Path) -> dict +``` + +**ACX Requirements:** +| Metric | Range | +|--------|-------| +| RMS level | -23.0 dB to -18.0 dB | +| Peak level | Below -3.0 dB | + +**Returns:** +```python +{ + "rms_db": float, + "peak_db": float, + "rms_in_range": bool, # -23.0 <= rms <= -18.0 + "peak_below_limit": bool, # peak < -3.0 + "compliant": bool, # rms_in_range AND peak_below_limit +} +``` + +### 2. Watermarking + +#### Author Watermark Bits + +```python +def _author_watermark_bits(author_name: str, length: int = 128) -> np.ndarray +``` + +**Algorithm:** +1. SHA-256 hash of author name string +2. Convert 128 bits from the hash digest to a float array +3. Each byte → 8 bits, take first `length` bits +4. Return as `np.float32` array + +**Example:** +```python +author_name = "B.D. Murphy" +digest = sha256(b"B.D. Murphy").digest() # 32 bytes = 256 bits +bits = first 128 bits → np.float32 array of 128 values (0.0 or 1.0) +``` + +#### Perth Neural Watermarker + +```python +class ChatterboxEngine: + def _get_watermarker(self) -> watermarker | None + def apply_watermark(self, wav, watermark_bits, sample_rate) -> torch.Tensor +``` + +**Watermarker Selection:** +1. Try `PerthImplicitWatermarker(device="cuda")` — neural watermarking +2. Fall back to `DummyWatermarker()` — no-op pass-through +3. Return `None` if neither available + +**Application:** +```python +def apply_watermark(self, wav, watermark_bits, sample_rate=None): + if self._watermarker is None: + return wav # No watermarking available + + try: + audio_np = wav.squeeze(0).cpu().numpy() + if audio_np.ndim > 1: + audio_np = audio_np.mean(axis=0) + + watermarked = self._watermarker.embed(audio_np, watermark_bits) + return torch.from_numpy(watermarked).unsqueeze(0) + except Exception: + return wav # Fallback: return unwatermarked +``` + +**PerthImplicitWatermarker:** +- Located at: `~/.cache/torch/hub/snakers4_perth` +- Model: `perth_net` — implicit neural audio watermarking +- Embeds 128-bit payload into audio signal +- Robust to common audio transformations (compression, resampling) +- VRAM usage: minimal (~50MB) + +**DummyWatermarker:** +- No-op implementation +- Returns audio unchanged +- Used as fallback when Perth is unavailable + +### Integration with Generation + +Watermarking runs inside `ChatterboxEngine.generate()`: + +```python +def generate(self, text, output_path, voice_ref, instruct, watermark_bits, ...): + wav = self._model.generate(text, ...) + + if watermark_bits is not None: + wav = self.apply_watermark(wav, watermark_bits) + + # Write to file + sf.write(str(output_path), audio_np, self.sample_rate) +``` + +### Integration with Normalization + +Normalization runs inside `ChapterGenerator.generate_chapter()`: + +```python +# After successful generation: +if self.normalizer: + normalized_path = normalized_dir / out_path.name + self.normalizer.normalize_file(out_path, normalized_path) +``` + +### Quality Defect Tracking + +ACX compliance is checked per-file in the orchestrator: + +```python +# After generation: +result = normalizer.check_acx_compliance(audio_path) +if not result["compliant"]: + db.log_defect(project_id, "quality", "WARNING", + f"ACX non-compliance: RMS={result['rms_db']:.1f}dB, Peak={result['peak_db']:.1f}dB", + chapter_id=ch["id"]) +``` + +## Default Parameters + +| Parameter | Default | ACX Requirement | +|-----------|---------|-----------------| +| `target_rms_db` | -20.0 dB | -23.0 to -18.0 dB | +| `peak_limit_db` | -3.0 dB | Below -3.0 dB | +| `noise_floor_db` | -60.0 dB | N/A | +| `sample_rate` | 24000 Hz | N/A | +| `watermark_length` | 128 bits | N/A | diff --git a/specs/07-assembly-export.md b/specs/07-assembly-export.md new file mode 100644 index 0000000..31a5922 --- /dev/null +++ b/specs/07-assembly-export.md @@ -0,0 +1,162 @@ +# Spec: Assembly & Export + +## Overview + +The assembly module stitches normalized audio files into final audiobook formats (M4B, MP3, WAV) using FFmpeg and Python audio libraries. + +## Components + +### 1. Audio Stitcher (`assembly/__init__.py`) + +```python +class AudioStitcher: + def __init__( + self, + silence_between_utterances_ms: int = 500, + silence_between_paragraphs_ms: int = 2000, + silence_between_chapters_ms: int = 4000, + sample_rate: int = 24000, + ) -> None + + def stitch_utterances(self, audio_files: list[Path], output_path: Path) -> Path + def stitch_chapters(self, chapter_files: list[Path], output_path: Path) -> Path +``` + +#### Silence Generation + +```python +def _make_silence(self, duration_ms: int) -> np.ndarray: + num_samples = int(self.sample_rate * duration_ms / 1000) + return np.zeros(num_samples, dtype=np.float32) +``` + +#### Utterance Stitching + +Concatenates audio files with configurable silence between them: +- Each utterance followed by `silence_between_utterances_ms` (500ms default) +- Mono downmix if any file is stereo +- Output: single WAV file per chapter + +#### Chapter Stitching + +Concatenates chapter files with longer pauses: +- Each chapter followed by `silence_between_chapters_ms` (4000ms default) +- Used for final audiobook assembly + +### 2. Audiobook Exporter (`assembly/export.py`) + +```python +class AudiobookExporter: + def export_m4b( + self, + chapter_files: list[Path], + output_path: Path, + metadata: dict | None = None, + ) -> Path + + def export_mp3( + self, + chapter_files: list[Path], + output_dir: Path, + metadata: dict | None = None, + ) -> list[Path] + + def export_wav( + self, + chapter_files: list[Path], + output_dir: Path, + ) -> list[Path] +``` + +#### M4B Export + +```python +def export_m4b(self, chapter_files, output_path, metadata=None): + # Create FFmpeg concat file + with open(concat_file, "w") as f: + for chapter in chapter_files: + f.write(f"file '{chapter.absolute()}'\n") + + # Run FFmpeg + subprocess.run([ + "ffmpeg", "-f", "concat", "-safe", "0", + "-i", str(concat_file), + "-c", "copy", + str(output_path), + ]) +``` + +**Requirements:** FFmpeg must be installed on the system. + +#### MP3 Export + +```python +def export_mp3(self, chapter_files, output_dir, metadata=None): + for i, chapter in enumerate(chapter_files): + output_path = output_dir / f"chapter_{i+1:03d}.mp3" + subprocess.run([ + "ffmpeg", "-i", str(chapter), + "-codec:a", "libmp3lame", + "-b:a", "192k", + str(output_path), + ]) +``` + +**Settings:** 192kbps CBR MP3 encoding. + +#### WAV Export + +```python +def export_wav(self, chapter_files, output_dir): + for i, chapter in enumerate(chapter_files): + output_path = output_dir / f"chapter_{i+1:03d}.wav" + shutil.copy2(chapter, output_path) # No re-encoding +``` + +### 3. Metadata (`assembly/metadata.py`) + +Currently a stub (empty file). Planned: ID3 tag writing for MP3/M4B metadata (title, author, chapter markers). + +## Export Flow + +``` +Normalized audio files + │ + ├── Stitch per chapter → chapter WAVs + │ + ├── Export formats: + │ ├── M4B: FFmpeg concat copy + │ ├── MP3: FFmpeg 192kbps per chapter + │ └── WAV: Direct file copy + │ + └── Output: .umbra/export/ + ├── *.m4b + ├── *.mp3 + └── *.wav +``` + +## CLI Integration + +```python +@main.command() +@click.option("--format", "-f", "formats", multiple=True, default=["m4b", "mp3"]) +def export(ctx, formats): + # Source: prefers audio_normalized/, falls back to audio/ + # Output: .umbra/export/ +``` + +## Dependencies + +- **FFmpeg**: Required for M4B and MP3 export +- **soundfile**: WAV reading/writing +- **numpy**: Audio array manipulation + +## Default Timing Parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `silence_between_utterances_ms` | 500ms | Gap between sentences | +| `silence_between_paragraphs_ms` | 2000ms | Gap between paragraphs | +| `silence_between_chapters_ms` | 4000ms | Gap between chapters | +| `sample_rate` | 24000 Hz | Audio sample rate | +| `mp3_bitrate` | 192k | MP3 encoding bitrate | diff --git a/specs/08-database.md b/specs/08-database.md new file mode 100644 index 0000000..964c28d --- /dev/null +++ b/specs/08-database.md @@ -0,0 +1,168 @@ +# Spec: Database Schema + +## Overview + +UmbraAudio uses SQLite for all project state. The database tracks the full lifecycle from parsed manuscript through generated audio. Character discovery and voice assignments are stored in JSON files for flexibility. + +## Current Implementation + +### Actual Schema (project.db) + +```sql +-- Project metadata +CREATE TABLE IF NOT EXISTS project ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + source_path TEXT NOT NULL, + source_format TEXT NOT NULL, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')), + status TEXT NOT NULL DEFAULT 'initialized' +); + +-- Chapters with source text +CREATE TABLE IF NOT EXISTS chapters ( + id INTEGER PRIMARY KEY, + project_id INTEGER NOT NULL, + chapter_index INTEGER NOT NULL, + title TEXT, + source_text TEXT NOT NULL, + word_count INTEGER NOT NULL DEFAULT 0, + status TEXT NOT NULL DEFAULT 'parsed', + audio_path TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); + +-- Utterances (narration and dialogue) +CREATE TABLE IF NOT EXISTS utterances ( + id INTEGER PRIMARY KEY, + chapter_id INTEGER NOT NULL, + character_id INTEGER, + utterance_index INTEGER NOT NULL, + text TEXT NOT NULL, + type TEXT NOT NULL DEFAULT 'narration', + context_before TEXT, + context_after TEXT, + emotion TEXT, + pace TEXT, + emphasis_words TEXT, + pause_before_ms INTEGER, + pause_after_ms INTEGER, + instruct TEXT, + speaker_name TEXT, + status TEXT NOT NULL DEFAULT 'pending' +); + +-- Generated audio files +CREATE TABLE IF NOT EXISTS generated_audio ( + id INTEGER PRIMARY KEY, + utterance_id INTEGER NOT NULL, + character_id INTEGER, + audio_path TEXT NOT NULL, + duration_ms INTEGER, + mos_score REAL, + wer_score REAL, + speaker_similarity REAL, + artifact_score REAL, + generation_params TEXT, + status TEXT NOT NULL DEFAULT 'generated', + created_at TEXT NOT NULL DEFAULT (datetime('now')) +); +``` + +### JSON Data Files + +#### speakers.json +```json +{ + "speakers": [ + { + "primary_name": "Kainda", + "aliases": ["Princess Kainda", "Princess", "Captain", "XO Capek"], + "demographics": { + "type": "Human", + "gender": "Female" + }, + "vocal_traits": ["assertive", "clear", "young"], + "communication_style": { + "primary_register": "Formal", + "style_shifts_or_notes": "Shifts to casual with close allies" + }, + "is_narrator": false, + "is_group": false, + "description": "Princess and communication specialist", + "sample_dialogue": "I'm on a yacht.", + "chapters_appeared": [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31] + } + ] +} +``` + +#### voice_assignments.json +```json +{ + "Narrator": "en-US-GuyNeural:-10", + "Kainda": "en-US-JennyNeural", + "Captain Weston": "en-US-ChristopherNeural", + "Hunter": "en-US-JasonNeural", + "Nanny": "en-GB-SoniaNeural", + "Aunt Vicky": "en-US-AvaNeural", + "Admiral Wake": "en-US-AndrewNeural", + "Princess Kainda": "en-US-JennyNeural", + "Doctor": "en-US-GuyNeural", + "Guards": "en-US-GuyNeural:+48" +} +``` + +## Entity Relationships + +``` +project (1) ──< chapters (many) +chapters (1) ──< utterances (many) +utterances (1) ──< generated_audio (many) + +speakers.json ──< (character profiles) +voice_assignments.json ──< (voice mappings) +``` + +## Key Differences from Original Spec + +1. **No characters table**: Character profiles stored in `speakers.json` +2. **No pronunciation_guide table**: Not currently implemented +3. **No quality_reports table**: Quality tracking via web UI +4. **No review_decisions table**: Review via web UI endpoints +5. **speaker_name column**: Added to utterances for speaker attribution +6. **JSON-based configuration**: speakers.json and voice_assignments.json + +## Database Class API + +```python +class Database: + def __init__(self, db_path: Path) -> None + def connect(self) -> None + def close(self) -> None +``` + +### Query Methods + +| Method | Returns | Description | +|--------|---------|-------------| +| `get_chapters()` | `list[dict]` | All chapters ordered by index | +| `get_utterances(chapter_id)` | `list[dict]` | Utterances for a chapter | +| `get_audio_for_chapter(chapter_id)` | `list[dict]` | Generated audio files | + +## Data Flow + +1. **Parsing**: Manuscript → chapters table (source_text) +2. **Character Discovery**: LLM analysis → speakers.json +3. **Voice Assignment**: Manual/automated → voice_assignments.json +4. **Segmentation**: Chapter text + speakers.json → utterances table +5. **Audio Generation**: utterances + voice_assignments.json → chapter_audio/ + +## Indexes + +```sql +CREATE INDEX IF NOT EXISTS idx_utterances_chapter ON utterances(chapter_id); +CREATE INDEX IF NOT EXISTS idx_generated_audio_utterance ON generated_audio(utterance_id); +CREATE INDEX IF NOT EXISTS idx_chapters_project ON chapters(project_id); +``` diff --git a/specs/09-web-review-ui.md b/specs/09-web-review-ui.md new file mode 100644 index 0000000..af82831 --- /dev/null +++ b/specs/09-web-review-ui.md @@ -0,0 +1,300 @@ +# Spec: Web Review UI + +## Overview + +FastAPI + HTMX web interface for human review of generated audiobooks. Provides dashboards, voice review, chapter review with audio playback, and quality reports. + +## Technology Stack + +- **Framework:** FastAPI 0.104+ +- **Templating:** Jinja2 3.1+ +- **Interactive:** HTMX 1.9 (CDN) +- **Styling:** Custom dark theme (no CSS framework) +- **Database:** Synchronous sqlite3 (not aiosqlite — web runs in sync context) + +## App Factory + +```python +# umbraaudio/web/__init__.py +def create_app(config: AppConfig) -> FastAPI +``` + +- Creates FastAPI app (title: "UmbraAudio Review", version: "0.1.0") +- Sets up Jinja2Templates from `templates/` directory +- Mounts static files if `static/` directory exists +- Includes router from `umbraaudio.web.routes` +- Stores config on `app.state.config` +- Singleton pattern via global `_app` + +## Routes + +### HTML Pages (GET) + +| Route | Function | Template | Description | +|-------|----------|----------|-------------| +| `GET /` | `index()` | `index.html` | Dashboard with stats, scan results, pipeline status, export files | +| `GET /voices` | `voice_review()` | `voice_review.html` | Voice review page | +| `GET /voices/partial` | `voice_list_partial()` | `_voice_list.html` | HTMX partial | +| `GET /voices/{name}/history` | `voice_history()` | `_voice_history.html` | Per-character voice review history | +| `GET /speakers` | `speakers_review()` | `speakers.html` | Speaker management (CRUD, aliases, merge) | +| `GET /speakers/partial` | `speakers_list_partial()` | `_speakers_list.html` | HTMX partial | +| `GET /proper-names` | `proper_names_review()` | `proper_names.html` | Proper-noun pronunciation review (`/character-names` is an alias) | +| `GET /orphaned-utterances` | `orphaned_utterances_page()` | `orphaned_utterances.html` | Utterances whose speaker is unknown or `__unassigned__` | +| `GET /voice-tests` | `voice_tests_review()` | `voice_tests.html` | Verification-phrase test samples | +| `GET /voice-tests/partial` | `voice_tests_partial()` | `_voice_tests_list.html` | HTMX partial | +| `GET /chapters/{id}` | `chapter_review()` | `chapter_review.html` | Chapter review | +| `GET /chapters/{id}/partial` | `chapter_utterances_partial()` | `_utterance_list.html` | HTMX partial | +| `GET /chapters-audio` | `chapters_audio_review()` | `chapters_audio.html` | Per-chapter audio listing | +| `GET /chapters-audio/{chapter_dir}` | `chapter_audio_review()` | `chapter_audio_detail.html` | Utterance WAV review with speaker/text editing | +| `GET /quality` | `quality_report()` | `quality_report.html` | Quality dashboard | +| `GET /registry` | `registry_index()` | `registry.html` | Cross-project voice registry | +| `GET /registry/partial` | `registry_list_partial()` | `_registry_list.html` | HTMX partial | +| `GET /registry/search?q=` | `registry_search()` | — (JSON) | Search registry by character or alias | + +### Audio Serving (GET) + +| Route | Description | +|-------|-------------| +| `GET /audio/{chapter_id}/{filename}` | Serve WAV (prefers `audio_normalized`, falls back to `audio`) | +| `GET /voice-audio/{filename}` | Serve voice reference WAVs from `.umbra/voices` | +| `GET /voice-test-audio/{filename}` | Serve test sample (prefers `.umbra/voices`, falls back to `.umbra/voice_tests`) | +| `GET /chapter-audio-file/{chapter_dir}/{filename}` | Serve per-utterance chapter WAV | +| `GET /registry-audio/{registry_id}/{filename}` | Serve a registered voice reference from the registry DB path | + +### API + +| Route | Description | +|-------|-------------| +| `POST /api/upload-manuscript` | Upload `.docx/.epub/.txt/.md/.pdf`, parse chapters, rebuild utterances; archives prior state in `mode=new`, else keeps speakers | +| `POST /api/speakers` | Create speaker (form fields: name, gender, type, aliases, traits, register, description, sample dialogue, narrator/group flags, accent) | +| `PUT /api/speakers/{index}` | Update speaker | +| `DELETE /api/speakers/{index}` | Delete speaker; flags their utterances `__unassigned__` | +| `POST /api/speakers/{index}/alias` | Add alias | +| `DELETE /api/speakers/{index}/alias/{alias}` | Remove alias | +| `POST /api/speakers/{index}/alias/{alias}/move/{target}` | Move alias to another speaker | +| `POST /api/speakers/{index}/alias/{alias}/create` | Promote alias to a new speaker | +| `POST /api/speakers/{source}/merge/{target}` | Merge speakers (attributes, aliases, chapters merged; utterances repointed) | +| `POST /api/speakers/voice-status` | Set `voice_status.json` entry; on `approved`, upsert into `voice_registry` | +| `POST /api/speakers/regenerate-voice` | Regenerate one speaker's voice reference (background) | +| `POST /api/speakers/generate-all-voices` | Generate all missing voice references (background, skips existing) | +| `PUT /api/phonetic/{speaker_name}` | Set/edit phonetic spelling; stored as an unconfirmed user candidate, regenerates preview audio (clearing the value deletes the record + audio) | +| `GET /api/pronunciation/{name}` | Look up the most common pronunciation (dictionaryapi.dev for common words, Wiktionary API for names) and return the candidate without saving | +| `POST /api/pronunciations/{name}/lookup` | Internet lookup → candidate record in `phonetic_names.json` + generated preview audio; dictionary hits auto-confirm as `common`, Wiktionary hits are stored `needs review` | +| `POST /api/pronunciations/{name}/confirm` | Mark a candidate confirmed (only confirmed spellings are applied to TTS) | +| `POST /api/pronunciations/{name}/clear` | Delete the record and its preview audio | +| `GET /api/pronunciations/audio?name=` | Serve the preview WAV from `.umbra/pronunciation_audio/{sanitized}.wav` | +| `GET /api/pronunciations/status` | Read batch progress from `pronunciation_status.json` (`{status, detail, done, total}`) | +| `POST /api/auto-pronunciations` | Background batch lookup over all proper nouns; auto-confirms common words, generates preview audio for Wiktionary hits, re-looks-up legacy records, writes progress to `pronunciation_status.json` | +| `POST /api/voices/{id}/approve` | Approve character voice (records `voice_review_history`) | +| `POST /api/voices/{id}/reject` | Reject character voice | +| `POST /api/voices/{id}/edit` | Queue a voice edit prompt (stored in history) | +| `POST /api/chapters/{id}/approve` | Approve chapter (`review_decisions`) | +| `POST /api/chapters/{id}/regenerate` | Stub (returns `{"status": "queued"}`) | +| `PUT /api/utterance/{utt_id}/speaker` | Reassign utterance speaker; deletes stale WAV and regenerates via Edge fallback | +| `PUT /api/utterance/{utt_id}/text` | Edit utterance text; deletes stale WAV | +| `DELETE /api/utterance/{utt_id}` | Delete utterance; renumber chapter; regenerate chapter WAVs via Qwen3 | +| `POST /api/chapter/{chapter_id}/utterance` | Append utterance; generate its WAV in background | +| `POST /api/orphaned-utterances/{utt_id}/reassign` | Reassign one orphaned utterance | +| `POST /api/orphaned-utterances/reassign-all` | Reassign all utterances that had the given old names | +| `POST /api/analyze` | Run character discovery in background (`analysis_status.json`) | +| `GET /api/analysis-status` | Read `analysis_status.json` | +| `POST /api/segment-utterances` | Run segmentation + LLM attribution + emotion analysis (skips attribution if LM Studio offline) | +| `POST /api/generate-audiobook` | Run full Qwen3-TTS generation in background (`scripts.generate_all_audio.main`) | +| `GET /api/generation/status` | Live generation progress (items, success/failure counts, interrupted flag) | +| `GET /api/pipeline/status` | Pipeline state machine status | +| `POST /api/pipeline/status` | Update pipeline status fields | +| `POST /api/pipeline/start` | Start pipeline (`run_pipeline_background`) | +| `POST /api/pipeline/pause` | Pause pipeline | +| `POST /api/pipeline/resume` | Resume paused pipeline | +| `POST /api/pipeline/cancel` | Cancel pipeline | +| `POST /api/pipeline/reset?step=` | Reset `discover` (speakers/voices/phonetics) / `voices` / `segment` (utterances) | +| `POST /api/pipeline/run-all` | Full pipeline: discover → voices → segment+attribute → generate → quality gate → export M4B/MP3 | +| `POST /api/chapter/{id}/build` | Stitch chapter WAVs into `ch{id}_full.wav` after quality gate | +| `GET /api/quality/check` | Quality-check all chapters (JSON) | +| `GET /api/quality/check/{chapter_id}` | Quality-check one chapter (JSON) | +| `GET /api/export/files` | List export directory files with download URLs | +| `GET /api/export/download/{path}` | Download export file (path-traversal protected) | + +## Pages + +### Dashboard (`/`) + +**Stats displayed:** +- Project name, source path, and format (from `project` row + `scan_results.json`) +- Chapter count, character count (non-narrator), utterance count, audio segment count +- Scene count (`<<<>>>` markers across chapter source text) +- Speaker count (from `speakers.json`) and voice reference count (`*_ref.wav` in `.umbra/voices`) +- Pipeline status (current step, progress, error) +- Export files (name + size, with download links) + +### Voice Review (`/voices`) + +HTMX-powered page that loads character data on demand. + +**`_voice_list.html` partial:** + +| Column | Content | +|--------|---------| +| Name | Character name | +| Description | Truncated `voice_description` JSON | +| Role | Badge: Narrator / Character | +| Status | Badge: voiced_approved / rejected / pending | +| Energy params | Derived `exaggeration` / `cfg_weight` from `ATTRIBUTE_PARAM_MAP[energy]` (deterministic from name hash; narrator uses fixed values) | +| Audio | HTML5 `