UmbraAudio/specs/26-multilingual-tts-model-survey.md

Multilingual audio production - model survey and recommendation

Status: research, not yet a decision. Written 2026-10-12.

Scope: produce Deal Hunter (and the other books) in 15 languages, with each character keeping the same voice identity across every language, in a form a translation system can drive.

This is a survey of TTS models, not LLMs. The requirement - "take an existing voice and have it speak another language" - is cross-lingual voice cloning, and that is a text-to-speech capability. An LLM is still needed separately for translation, but it plays no part in reproducing a voice.

The 15 target languages

#	Language	Code	Qwen3-TTS	Chatterbox ML
1	English	en	yes	yes
2	Chinese	zh	yes	disabled in the build checked
3	Japanese	ja	yes	yes
4	Korean	ko	yes	yes
5	German	de	yes	yes
6	Spanish	es	yes	yes
7	French	fr	yes	yes
8	Russian	ru	yes	yes
9	Italian	it	yes	yes
10	Dutch	nl	no	yes
11	Norwegian	no	no	yes
12	Polish	pl	no	yes
13	Portuguese	pt	yes	yes
14	Arabic	ar	no	yes
15	Hindi	hi	no	yes

Qwen3-TTS covers 10 of 15. The five gaps are Dutch, Norwegian, Polish, Arabic and Hindi - which is why one model cannot do this job alone.

Licensing is the deciding constraint

Audiobooks are sold, so weights licences matter more than code licences, and the popular multilingual cloning models are almost all non-commercial:

Model	Languages	Cloning	Weights licence	Commercial?
Qwen3-TTS	10	3 s ref, cross-lingual	Apache 2.0 (Alibaba, 2026)	Yes
Chatterbox Multilingual	22-23	~10 s ref, cross-lingual	MIT	Yes (output watermarked)
XTTS v2 (Coqui)	17	6 s ref	Coqui Public Model License - non-commercial, and Coqui shut down in Jan 2024 so no licence can be bought	No
Fish Audio S2 Pro	83	10-30 s ref	Fish Audio Research - non-commercial	No (API terms differ)
F5-TTS	-	6 s ref	CC-BY-NC-4.0 weights	No
CosyVoice 2	9 + dialects	cross-lingual	Apache 2.0	Yes, but same 9-ish languages as Qwen3
Kokoro 82M	8	no cloning	Apache 2.0	Yes, but cannot reproduce our voices

XTTS v2 is the obvious candidate on capability - 17 languages, mature, would cover 14 of 15 - and it is unusable for a commercial audiobook. It is excluded on licence, not on quality.

Recommendation: a two-engine split

Engine A - Qwen3-TTS (already running locally, Apache 2.0) Handles the 10 languages it covers. Its own technical report claims state-of-the-art cross-lingual cloning, and specifically the best results for English->Korean and Chinese->Korean, with roughly 66% lower error than CosyVoice3 on zh->ko. Korean is one of the harder targets, and this is the engine that is strongest at it. No new infrastructure, no new licence review.

Engine B - Chatterbox Multilingual (MIT) Handles the five gaps: Dutch, Norwegian, Polish, Arabic, Hindi. MIT permits commercial use and it does zero-shot multilingual cloning from a ~10 s clip, which is the same input we already produce for every speaker.

Together the two cover all 15.

Caveat on Engine B: the 22-language list above came from a third-party API wrapper around Resemble AI's chatterbox-tts v0.1.4, not from Resemble's own model card, and that build reports Chinese as disabled. It must be checked against upstream before relying on it - in particular for the overlap languages, where we would prefer to know whether Chatterbox can be trusted as a second opinion on Korean.

The "voice matrix" does not travel - and does not need to

This is the part most likely to cause pain if it is not understood up front.

voice_registry stores two different kinds of thing:

• voice_ref_path - the *_ref.wav, 10-30 s of clean speech. Portable. Any engine that clones from reference audio can use it.
• vmatrix_path - the Qwen3-TTS ICL artefact: an 8k ref_code matrix plus a speaker embedding, saved as a .pt. Engine-specific. It is a Qwen3 internal representation; Chatterbox, XTTS or anything else cannot read it.

Deal Hunter currently has 91 reference WAVs and 56 vmatrix files. The reference WAVs are the real asset and they already exist. The vmatrix is a Qwen3-only cache that should be treated as disposable - regenerable from the WAV at any time.

So "reproducing voices in each language" does not require translating the voice matrix. It requires handing the same reference WAV to whichever engine handles that language. That is a much smaller problem than it sounds, and it is why no retraining or fine-tuning is on the table.

Measured: Qwen3-TTS speaks Chinese with an English voice reference

Status: research, not yet a decision. Written 2026-10-12. Measured 2026-11-07.

The survey above says Qwen3-TTS covers Chinese. That is a claim from a model card. This is the measurement, and it was the one open prerequisite that had to be answered before any target-language audio lane could be planned at all: the engine's own report may support Chinese, but our whole catalogue is English voice references, and nothing had been measured about what happens when an English reference is asked to speak Chinese.

scripts/probe_cross_language_tts.py answers it by transcription rather than by listening. Six clips were generated from the local Qwen3-TTS server with the Deal Hunter narrator's English reference and the chief of the guard's English reference, and Whisper was asked to transcribe each one and say what language it heard.

clip	text sent	language sent	heard	transcribed as
english-ref-chinese-lang	Chinese	chinese	zh	凯英達站在劍橋上看著鹹窗外的星雲握緊了手裡的搬手
english-ref-english-lang	Chinese	english	zh	凱因达站在建橋上看著嫌窗外的心雲握緊了手裡的搬手
no-ref-chinese-lang	Chinese	chinese	zh	凯因达站在建橋上看着闲窗外的星云握紧了手里的搬手
english-ref-english-text	English	english	en	Kayinda stood on the bridge, watching the nebula through the viewport.
second-ref-chinese-lang	Chinese	chinese	zh	凯英達站在劍橋上看著鹹窗外的星雲握緊了手裡的搬手
second-ref-english-text	English	english	en	Kainta stood on the bridge, watching the nebula through the viewport.

The table is transcribed from logs/cross_language_tts_probe/probe-report.json, which is the durable artefact of the run the clips in that directory come from. Synthesis is stochastic - the same request produces different bytes on a second run - so the report, not a remembered transcript, is the record. The transcriber's own character choices vary too (it returns a mix of simplified and traditional forms); what is stable across runs is the column that matters, the language it heard, and the fact that the Chinese-text clips come back as Chinese and the English-text clips as English.

Finding 1 - the English reference speaks Chinese. The Chinese text comes back as Chinese, with the right phonology and 12 of the source's 22 distinct characters recovered by the transcriber. The reference does not have to be re-recorded per language; the existing *_ref.wav assets are the input.

Finding 2 - the text's script decides the language, not the parameter. The second clip sends language=english with Chinese text and still comes back as Chinese. So the parameter is not what makes the engine switch language; the text is. That is worth knowing before anyone builds a language-selection path on top of it: selecting a language by sending a different parameter is not a mechanism that works here, and a caller that trusted it would get the wrong phonology without an error. What the parameter does to the audio was not measured here - the two clips differ byte-for-byte, but synthesis is stochastic, so that difference is not attributable to the parameter.

Finding 3 - the reference is carried, not ignored. This needed a differential, because a transcript has no timbre. Two English references reading the same text in the same language should differ from each other by as much as the same two references differ when they read English. Measured with ECAPA-TDNN (speechbrain/spkrec-ecapa-voxceleb, cosine similarity):

comparison	similarity
same voice, English (reference vs its own clone)	0.821
different voices, English	0.383
different voices, Chinese	0.369
first reference vs the model's own voice (Chinese)	0.420
second reference vs the model's own voice (Chinese)	0.343

The two references stay about as far apart when they read Chinese (0.369) as when they read English (0.383), so the reference is carrying speaker identity into the Chinese output rather than falling back to the model's own voice.

What this measurement does not establish. ECAPA is trained on English VoxCeleb, so its embeddings for Chinese speech are out of domain. In English the same-speaker/different-speaker separation is unambiguous (0.821 vs 0.383); in Chinese it collapses (0.329 and 0.515 for the same-speaker pairs against 0.369 for the different-speaker pair). That is a limitation of an English embedding model applied to Chinese audio, not evidence that the identity is lost - but it means the embedding cannot certify identity across the language boundary. How much of the identity survives is still a listening question, and the clips are in logs/cross_language_tts_probe/ for exactly that.

This answers the "does it speak the language at all" prerequisite. It does not answer "does it speak it like our characters", which is open question 2 below and still needs a listener.

Open questions before this becomes a plan

1. Verify Chatterbox upstream. Confirm the language list against Resemble AI's own model card, and confirm the output watermark is acceptable for a distributed audiobook.
2. Native-speaker listening tests. Coverage is not quality. Each language needs a native reviewer on pronunciation, prosody and whether the cloned identity survives. Korean and Arabic are the ones most likely to expose problems. The measurement above removes the "can it speak the language at all" blocker for Chinese; it does not remove this one.
3. Text pipeline first. Every language needs translated text in utterance form, matching the existing (chapter, utterance_index, speaker, text) shape, with the per-utterance emotion/pace/instruct preserved. The audio stage is downstream of this and cannot start until it exists. For Chinese this is the delivery UmbraTrans now produces (spec 27, pass 5).
4. Right-to-left and segmentation. Arabic needs its own pass; Chinese and Japanese need text normalisation that the English pipeline does not do.
5. Cost. Two engines is more operational surface than one. If Norwegian and Arabic were dropped, Qwen3-TTS alone would cover 13 of 15 and there would be no second engine at all.
6. Does the identity survive for every speaker? The measurement above used two references. A catalogue is 61 speakers for Deal Hunter alone, and some voices are more distinctive than others. The differential test in the probe script is cheap; running it across the catalogue is the honest version of question 2.

Rejected, and why

• XTTS v2 - best single-model coverage available (14/15), but non-commercial and unlicensable. Would be the first choice if it were free to sell.
• Fish Audio S2 - 83 languages would solve everything, and the hosted API has a free tier, but the weights are research-licensed and a hosted dependency for a 15-language catalogue is a large recurring cost.
• Commercial APIs (ElevenLabs, Inworld, Cartesia) - all cover the full list with cloning, and all are per-character pricing. Across 15 languages x N books this is a real recurring bill, and it puts the voice catalogue outside our control. Worth revisiting if a language proves impossible locally.