Research: multilingual TTS models for the 15-language catalogue
Survey of cross-lingual voice cloning, as a companion to the speaker-review audit that gates the translation handoff. Recorded as research, not a decision. Terminology correction up front: this is a TTS problem, not an LLM problem. Reproducing a voice in another language is cross-lingual voice cloning; an LLM is still needed for the translation itself but plays no part in voice fidelity. Headline findings: - Qwen3-TTS, already running locally, covers 10 of the 15 languages (en zh ja ko de fr ru pt es it). The gaps are Dutch, Norwegian, Polish, Arabic and Hindi - so no single model does this job. - Licence is the deciding constraint, because the books are sold. XTTS v2 covers 14 of 15 and is the obvious capability choice, but its weights are Coqui Public Model License (non-commercial) and Coqui shut down in Jan 2024, so no commercial licence can be bought. Fish Audio S2 (83 languages) and F5-TTS are likewise non-commercial weights. - Recommendation is a two-engine split: Qwen3-TTS (Apache 2.0, verified from the licence file) for its 10, Chatterbox Multilingual (MIT) for the 5 gaps. Together they cover all 15. The 'voice matrix' question, which the operator raised directly: voice_registry stores two different things. voice_ref_path is the reference WAV and is portable to any clone-from-audio engine; vmatrix_path is a Qwen3-TTS ICL artefact (8k ref_code + speaker embedding, .pt) and is engine-specific and unreadable by anything else. Deal Hunter already has 91 reference WAVs, so voice fidelity across languages needs no retraining and no matrix translation - just handing the same WAV to whichever engine covers that language. Recorded honestly: the Chatterbox language list came from a third-party API wrapper, not Resemble AI's own model card, and that build reports Chinese as disabled. Verification against upstream is listed as open question 1.
This commit is contained in:
B.D. Murphy
131
specs/26-multilingual-tts-model-survey.md
Normal file
131
specs/26-multilingual-tts-model-survey.md
Normal file
@@ -0,0 +1,131 @@
|
||||
# Multilingual audio production - model survey and recommendation
|
||||
|
||||
Status: research, not yet a decision. Written 2026-10-12.
|
||||
|
||||
Scope: produce Deal Hunter (and the other books) in 15 languages, with each
|
||||
character keeping the same voice identity across every language, in a form a
|
||||
translation system can drive.
|
||||
|
||||
This is a survey of **TTS** models, not LLMs. The requirement - "take an
|
||||
existing voice and have it speak another language" - is *cross-lingual voice
|
||||
cloning*, and that is a text-to-speech capability. An LLM is still needed
|
||||
separately for translation, but it plays no part in reproducing a voice.
|
||||
|
||||
## The 15 target languages
|
||||
|
||||
| # | Language | Code | Qwen3-TTS | Chatterbox ML |
|
||||
|---|----------|------|-----------|---------------|
|
||||
| 1 | English | en | yes | yes |
|
||||
| 2 | Chinese | zh | yes | *disabled in the build checked* |
|
||||
| 3 | Japanese | ja | yes | yes |
|
||||
| 4 | Korean | ko | yes | yes |
|
||||
| 5 | German | de | yes | yes |
|
||||
| 6 | Spanish | es | yes | yes |
|
||||
| 7 | French | fr | yes | yes |
|
||||
| 8 | Russian | ru | yes | yes |
|
||||
| 9 | Italian | it | yes | yes |
|
||||
| 10 | Dutch | nl | **no** | yes |
|
||||
| 11 | Norwegian | no | **no** | yes |
|
||||
| 12 | Polish | pl | **no** | yes |
|
||||
| 13 | Portuguese | pt | yes | yes |
|
||||
| 14 | Arabic | ar | **no** | yes |
|
||||
| 15 | Hindi | hi | **no** | yes |
|
||||
|
||||
**Qwen3-TTS covers 10 of 15. The five gaps are Dutch, Norwegian, Polish,
|
||||
Arabic and Hindi** - which is why one model cannot do this job alone.
|
||||
|
||||
## Licensing is the deciding constraint
|
||||
|
||||
Audiobooks are sold, so *weights* licences matter more than code licences, and
|
||||
the popular multilingual cloning models are almost all non-commercial:
|
||||
|
||||
| Model | Languages | Cloning | Weights licence | Commercial? |
|
||||
|---|---|---|---|---|
|
||||
| **Qwen3-TTS** | 10 | 3 s ref, cross-lingual | Apache 2.0 (Alibaba, 2026) | **Yes** |
|
||||
| **Chatterbox Multilingual** | 22-23 | ~10 s ref, cross-lingual | MIT | **Yes** (output watermarked) |
|
||||
| XTTS v2 (Coqui) | 17 | 6 s ref | Coqui Public Model License - **non-commercial, and Coqui shut down in Jan 2024 so no licence can be bought** | No |
|
||||
| Fish Audio S2 Pro | 83 | 10-30 s ref | Fish Audio Research - non-commercial | No (API terms differ) |
|
||||
| F5-TTS | - | 6 s ref | CC-BY-NC-4.0 weights | No |
|
||||
| CosyVoice 2 | 9 + dialects | cross-lingual | Apache 2.0 | Yes, but same 9-ish languages as Qwen3 |
|
||||
| Kokoro 82M | 8 | **no cloning** | Apache 2.0 | Yes, but cannot reproduce *our* voices |
|
||||
|
||||
XTTS v2 is the obvious candidate on capability - 17 languages, mature, would
|
||||
cover 14 of 15 - and it is **unusable for a commercial audiobook**. It is
|
||||
excluded on licence, not on quality.
|
||||
|
||||
## Recommendation: a two-engine split
|
||||
|
||||
**Engine A - Qwen3-TTS (already running locally, Apache 2.0)**
|
||||
Handles the 10 languages it covers. Its own technical report claims
|
||||
state-of-the-art cross-lingual cloning, and specifically the best results for
|
||||
English->Korean and Chinese->Korean, with roughly 66% lower error than
|
||||
CosyVoice3 on zh->ko. Korean is one of the harder targets, and this is the
|
||||
engine that is strongest at it. No new infrastructure, no new licence review.
|
||||
|
||||
**Engine B - Chatterbox Multilingual (MIT)**
|
||||
Handles the five gaps: **Dutch, Norwegian, Polish, Arabic, Hindi**. MIT permits
|
||||
commercial use and it does zero-shot multilingual cloning from a ~10 s clip,
|
||||
which is the same input we already produce for every speaker.
|
||||
|
||||
Together the two cover all 15.
|
||||
|
||||
Caveat on Engine B: the 22-language list above came from a third-party API
|
||||
wrapper around Resemble AI's `chatterbox-tts` v0.1.4, not from Resemble's own
|
||||
model card, and that build reports Chinese as **disabled**. It must be checked
|
||||
against upstream before relying on it - in particular for the overlap
|
||||
languages, where we would prefer to know whether Chatterbox can be trusted as
|
||||
a second opinion on Korean.
|
||||
|
||||
## The "voice matrix" does not travel - and does not need to
|
||||
|
||||
This is the part most likely to cause pain if it is not understood up front.
|
||||
|
||||
`voice_registry` stores two different kinds of thing:
|
||||
|
||||
- **`voice_ref_path`** - the `*_ref.wav`, 10-30 s of clean speech.
|
||||
**Portable.** Any engine that clones from reference audio can use it.
|
||||
- **`vmatrix_path`** - the Qwen3-TTS ICL artefact: an 8k `ref_code` matrix plus
|
||||
a speaker embedding, saved as a `.pt`. **Engine-specific.** It is a Qwen3
|
||||
internal representation; Chatterbox, XTTS or anything else cannot read it.
|
||||
|
||||
Deal Hunter currently has **91 reference WAVs** and **56 vmatrix files**. The
|
||||
reference WAVs are the real asset and they already exist. The vmatrix is a
|
||||
Qwen3-only cache that should be treated as disposable - regenerable from the
|
||||
WAV at any time.
|
||||
|
||||
So "reproducing voices in each language" does not require translating the
|
||||
voice matrix. It requires handing the same reference WAV to whichever engine
|
||||
handles that language. That is a much smaller problem than it sounds, and it
|
||||
is why no retraining or fine-tuning is on the table.
|
||||
|
||||
## Open questions before this becomes a plan
|
||||
|
||||
1. **Verify Chatterbox upstream.** Confirm the language list against Resemble
|
||||
AI's own model card, and confirm the output watermark is acceptable for a
|
||||
distributed audiobook.
|
||||
2. **Native-speaker listening tests.** Coverage is not quality. Each language
|
||||
needs a native reviewer on pronunciation, prosody and whether the cloned
|
||||
identity survives. Korean and Arabic are the ones most likely to expose
|
||||
problems.
|
||||
3. **Text pipeline first.** Every language needs translated text *in utterance
|
||||
form*, matching the existing `(chapter, utterance_index, speaker, text)`
|
||||
shape, with the per-utterance emotion/pace/instruct preserved. The audio
|
||||
stage is downstream of this and cannot start until it exists.
|
||||
4. **Right-to-left and segmentation.** Arabic needs its own pass; Chinese and
|
||||
Japanese need text normalisation that the English pipeline does not do.
|
||||
5. **Cost.** Two engines is more operational surface than one. If Norwegian
|
||||
and Arabic were dropped, Qwen3-TTS alone would cover 13 of 15 and there
|
||||
would be no second engine at all.
|
||||
|
||||
## Rejected, and why
|
||||
|
||||
- **XTTS v2** - best single-model coverage available (14/15), but
|
||||
non-commercial and unlicensable. Would be the first choice if it were free
|
||||
to sell.
|
||||
- **Fish Audio S2** - 83 languages would solve everything, and the hosted API
|
||||
has a free tier, but the weights are research-licensed and a hosted
|
||||
dependency for a 15-language catalogue is a large recurring cost.
|
||||
- **Commercial APIs (ElevenLabs, Inworld, Cartesia)** - all cover the full
|
||||
list with cloning, and all are per-character pricing. Across 15 languages x N
|
||||
books this is a real recurring bill, and it puts the voice catalogue outside
|
||||
our control. Worth revisiting if a language proves impossible locally.
|
||||
Reference in New Issue
Block a user