43a2853ee3
Survey of cross-lingual voice cloning, as a companion to the speaker-review audit that gates the translation handoff. Recorded as research, not a decision. Terminology correction up front: this is a TTS problem, not an LLM problem. Reproducing a voice in another language is cross-lingual voice cloning; an LLM is still needed for the translation itself but plays no part in voice fidelity. Headline findings: - Qwen3-TTS, already running locally, covers 10 of the 15 languages (en zh ja ko de fr ru pt es it). The gaps are Dutch, Norwegian, Polish, Arabic and Hindi - so no single model does this job. - Licence is the deciding constraint, because the books are sold. XTTS v2 covers 14 of 15 and is the obvious capability choice, but its weights are Coqui Public Model License (non-commercial) and Coqui shut down in Jan 2024, so no commercial licence can be bought. Fish Audio S2 (83 languages) and F5-TTS are likewise non-commercial weights. - Recommendation is a two-engine split: Qwen3-TTS (Apache 2.0, verified from the licence file) for its 10, Chatterbox Multilingual (MIT) for the 5 gaps. Together they cover all 15. The 'voice matrix' question, which the operator raised directly: voice_registry stores two different things. voice_ref_path is the reference WAV and is portable to any clone-from-audio engine; vmatrix_path is a Qwen3-TTS ICL artefact (8k ref_code + speaker embedding, .pt) and is engine-specific and unreadable by anything else. Deal Hunter already has 91 reference WAVs, so voice fidelity across languages needs no retraining and no matrix translation - just handing the same WAV to whichever engine covers that language. Recorded honestly: the Chatterbox language list came from a third-party API wrapper, not Resemble AI's own model card, and that build reports Chinese as disabled. Verification against upstream is listed as open question 1.
UmbraAudio
Quality-first multi-pass audiobook generation from English manuscripts.
Features
- 5-pass pipeline: Analysis → Voice Design → Mapping → Test → Generate
- Multi-engine TTS: Chatterbox (primary), Qwen3-TTS (secondary)
- LLM-enhanced: BookNLP for NLP, LLM for profiling/emotion/pronunciation
- Web review: FastAPI + HTMX at checkpoints for human approval
Quick Start
python -m umbraaudio.cli init <manuscript.epub> --name 'My Book'
python -m umbraaudio.cli analyze
python -m umbraaudio.cli voices
python -m umbraaudio.cli review
Requirements
- Python 3.11+
- NVIDIA GPU with CUDA support (RTX 3060 or better recommended)
- ffmpeg
Development
pip install -e ".[dev]"
ruff check umbraaudio/
pytest tests/ -v
Commit cadence: commit at least once every Sunday. Never leave a week (let alone a month) of work uncommitted — a single accidental
git checkouton 2026-09-20 wiped out a month of changes toumbraaudio/web/routes.py. SeeAGENTS.md→ Commit Cadence.
scripts/commit_check.sh enforces the nudge: it exits non-zero once the tree has been dirty
longer than a week. The LaunchAgent com.umbraaudio.commitcheck runs it every Sunday at
18:00 and posts a macOS notification when there is uncommitted work.
scripts/commit_check.sh # warn if dirty > 7 days
scripts/commit_check.sh --days 3 # tighter threshold
scripts/commit_check.sh --any # warn on any uncommitted change
Rebuilding from source
The repo tracks source only; large runtime data is reconstructed as needed.
-
Models: weights are not committed (see
.gitignore). Download them:bash scripts/download_models.sh # ~9 GB, targets ./models -
Qwen3-TTS is a git submodule pinned to upstream commit
022e286:git submodule update --init --recursiveThe
tts_server/web-demo backend includes a local UI customization toqwen_tts/cli/demo.py. It is preserved as a patch — reapply it after checking out the submodule:cd Qwen3-TTS && git apply ../scripts/qwen3_tts_demo_umbra_patch.patch -
Environment: copy/keep
.env(contains secrets) and setUMBRA_TTS__*model paths undermodels/.
Book projects (e.g. DealHunterShip_...) are committed separately under their
own project directories and are not part of this program repository.