43a2853ee3
Survey of cross-lingual voice cloning, as a companion to the speaker-review audit that gates the translation handoff. Recorded as research, not a decision. Terminology correction up front: this is a TTS problem, not an LLM problem. Reproducing a voice in another language is cross-lingual voice cloning; an LLM is still needed for the translation itself but plays no part in voice fidelity. Headline findings: - Qwen3-TTS, already running locally, covers 10 of the 15 languages (en zh ja ko de fr ru pt es it). The gaps are Dutch, Norwegian, Polish, Arabic and Hindi - so no single model does this job. - Licence is the deciding constraint, because the books are sold. XTTS v2 covers 14 of 15 and is the obvious capability choice, but its weights are Coqui Public Model License (non-commercial) and Coqui shut down in Jan 2024, so no commercial licence can be bought. Fish Audio S2 (83 languages) and F5-TTS are likewise non-commercial weights. - Recommendation is a two-engine split: Qwen3-TTS (Apache 2.0, verified from the licence file) for its 10, Chatterbox Multilingual (MIT) for the 5 gaps. Together they cover all 15. The 'voice matrix' question, which the operator raised directly: voice_registry stores two different things. voice_ref_path is the reference WAV and is portable to any clone-from-audio engine; vmatrix_path is a Qwen3-TTS ICL artefact (8k ref_code + speaker embedding, .pt) and is engine-specific and unreadable by anything else. Deal Hunter already has 91 reference WAVs, so voice fidelity across languages needs no retraining and no matrix translation - just handing the same WAV to whichever engine covers that language. Recorded honestly: the Chatterbox language list came from a third-party API wrapper, not Resemble AI's own model card, and that build reports Chinese as disabled. Verification against upstream is listed as open question 1.
UmbraAudio Specifications
Index of all specification documents for the UmbraAudio audiobook generation system.
Specifications
| File | Topic | Description |
|---|---|---|
| 00-project-overview.md | Architecture | Pipeline overview, tech stack, data layout |
| 01-parsing.md | Pass 1a | EPUB, PDF, TXT, MD manuscript parsing |
| 02-analysis.md | Pass 1b | BookNLP, character profiling, pronunciation |
| 03-voice-design.md | Pass 2 | Per-character voice reference generation |
| 04-mapping.md | Pass 3 | Text segmentation, speaker attribution, manifests |
| 05-generation.md | Pass 5 | Audio generation, caching, retry, CUDA management |
| 06-normalization-watermarking.md | Post-gen | Audio normalization, ACX compliance, Perth watermark |
| 07-assembly-export.md | Export | M4B, MP3, WAV export via FFmpeg |
| 08-database.md | Database | SQLite schema, entity relationships, CRUD API |
| 09-web-review-ui.md | Web UI | FastAPI + HTMX review interface |
| 10-cli-orchestrator.md | CLI | Click commands, full pipeline orchestration |
| 11-configuration-dependencies.md | Config | Pydantic settings, dependencies, build system |
| 12-quality.md | Quality | MOS, WER, consistency, ACX compliance |
| 18-UmbraTrans.md | Multilingual | Meaning-preserving translation, target voices, and multilingual audiobook production |
| 19-enterprise-control-plane.md | Enterprise | Multi-tenant projects, access control, artifact lineage, and commercial voice library |
| 20-enterprise-database-integration.md | Enterprise DB | Shared-PostgreSQL contract, blockers, cutover phases P1-P4 |
| 21-umbra-system-nodes-tasks-and-changes.md | System | Worker nodes, task assignment, cross-product change management, UI topology |
| 22-releases-and-the-publish-flag.md | Release | The publish flag, pinned artifact sets, post-publication change, and the Deal Hunter lesson |
| 23-speaker-duplicate-detection.md | Speakers | Duplicate-speaker scan, approval-safe merging, and the crowd veto |
| 24-analysis-quality-plan.md | Analysis | Why the ch6 speakers were missed, and a ranked plan to improve attribution |
Quick Reference
Pipeline Passes
1a: Parse → chapters table
1b: Analyze → characters, pronunciation_guide
2: Voices → voice_ref WAVs
3: Map → manifests/*.json
4: Test → sample audio (optional)
5: Generate → audio/*.wav, audio_normalized/*.wav
Export → export/*.m4b, *.mp3
Key Entry Points
| Component | File | Class/Function |
|---|---|---|
| CLI | cli.py |
main Click group |
| Pipeline | orchestrator.py |
run_pipeline() |
| TTS | voices/chatterbox.py |
ChatterboxEngine |
| Segmenter | mapping/segmenter.py |
Segmenter |
| Generator | generation/chapter.py |
ChapterGenerator |
| Normalizer | assembly/normalize.py |
AudioNormalizer |
| Web UI | web/__init__.py |
create_app() |
| Database | database.py |
Database |