umbragit/UmbraAudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Measure whether Qwen3-TTS speaks Chinese with an English voice reference

Browse Source 
This was the one open prerequisite for any target-language audio lane: the
survey in spec 26 says the model covers Chinese, but that is a model-card
claim, and our whole voice catalogue is English references. Nothing had been
measured about what happens when an English reference is asked to speak
Chinese.

scripts/probe_cross_language_tts.py now also takes an optional second English
reference and makes the identity question differential rather than a
transcription question - a transcript has no timbre. Two English references
reading the same text should stay as far apart when they read Chinese as
when they read English; if they do, the reference is carrying identity into
the Chinese output rather than being ignored. The measurement is made with
ECAPA-TDNN cosine similarity.

The result, on the real Deal Hunter narrator and chief-of-the-guard
references, is in logs/cross_language_tts_probe/probe-report.json:

- the English reference speaks Chinese. Whisper hears zh on every
  Chinese-text clip and en on every English-text clip, recovering 12 of the
  source's 22 distinct characters.
- the text's script decides the language, not the `language` parameter.
  Chinese text with language=english still comes back as Chinese, so a
  caller that trusted the parameter to select a language would get the
  wrong phonology without an error.
- the reference is carried: different voices are 0.383 apart reading English
  and 0.369 apart reading Chinese.

What the measurement does not establish is stated in the spec rather than
glossed: ECAPA is an English VoxCeleb model, so its Chinese embeddings are
out of domain and the same-speaker/different-speaker separation collapses in
Chinese. It can show that two clips differ; it cannot certify that two clips
are the same speaker across a language boundary. How much of the identity
survives is still a listening question, and the clips are saved for it.

Spec 26 gains the measurement and a sixth open question - whether the
identity survives for all 61 speakers, not just the two measured.

Spec 27 gains the parameterisation record (UmbraTrans 3f05489) and a
measurement of what the Chinese edition actually contains: 4,305 delivery
rows, of which the 90 not-voiced rows are 44 normal collapses and 46
drift-dropped paragraphs covering 33 paragraphs and 356 source words, about
2.4 minutes of speech. Two of the dropped paragraphs are upstream
translation defects the aligner cannot fix - ch-15-p0100 and ch-25-p0089
carry text that belongs to a different sentence - so fixing them means
re-translating.

Suite: 913 passed, 8 failed (the same pre-existing pronunciation and
voice_profile_manager accent failures), 6 skipped.
This commit is contained in:
B.D. Murphy
2026-11-07 06:27:09 -08:00
parent b259060808
commit 2cafbabe82
3 changed files with 319 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

94
specs/26-multilingual-tts-model-survey.md
View File

@@ -98,6 +98,89 @@ voice matrix. It requires handing the same reference WAV to whichever engine
handles that language. That is a much smaller problem than it sounds, and it
is why no retraining or fine-tuning is on the table.
## Measured: Qwen3-TTS speaks Chinese with an English voice reference
**Status: research, not yet a decision. Written 2026-10-12. Measured 2026-11-07.**
The survey above says Qwen3-TTS covers Chinese. That is a claim from a model
card. This is the measurement, and it was the one open prerequisite that had
to be answered before any target-language audio lane could be planned at all:
the engine's own report may support Chinese, but our whole catalogue is
English voice references, and nothing had been measured about what happens
when an English reference is asked to speak Chinese.
`scripts/probe_cross_language_tts.py` answers it by transcription rather than
by listening. Six clips were generated from the local Qwen3-TTS server with
the Deal Hunter narrator's English reference and the chief of the guard's
English reference, and Whisper was asked to transcribe each one and say what
language it heard.
| clip | text sent | `language` sent | heard | transcribed as |
|---|---|---|---|---|
| `english-ref-chinese-lang` | Chinese | chinese | **zh** | 凯英達站在劍橋上看著鹹窗外的星雲握緊了手裡的搬手 |
| `english-ref-english-lang` | Chinese | english | **zh** | 凱因达站在建橋上看著嫌窗外的心雲握緊了手裡的搬手 |
| `no-ref-chinese-lang` | Chinese | chinese | zh | 凯因达站在建橋上看着闲窗外的星云握紧了手里的搬手 |
| `english-ref-english-text` | English | english | en | Kayinda stood on the bridge, watching the nebula through the viewport. |
| `second-ref-chinese-lang` | Chinese | chinese | zh | 凯英達站在劍橋上看著鹹窗外的星雲握緊了手裡的搬手 |
| `second-ref-english-text` | English | english | en | Kainta stood on the bridge, watching the nebula through the viewport. |
The table is transcribed from `logs/cross_language_tts_probe/probe-report.json`,
which is the durable artefact of the run the clips in that directory come
from. Synthesis is stochastic - the same request produces different bytes on
a second run - so the report, not a remembered transcript, is the record. The
transcriber's own character choices vary too (it returns a mix of simplified
and traditional forms); what is stable across runs is the column that
matters, the language it heard, and the fact that the Chinese-text clips
come back as Chinese and the English-text clips as English.
**Finding 1 - the English reference speaks Chinese.** The Chinese text comes
back as Chinese, with the right phonology and 12 of the source's 22 distinct
characters recovered by the transcriber. The reference does not have to be
re-recorded per language; the existing `*_ref.wav` assets are the input.
**Finding 2 - the text's script decides the language, not the parameter.** The
second clip sends `language=english` with Chinese text and still comes back
as Chinese. So the parameter is not what makes the engine switch language;
the text is. That is worth knowing before anyone builds a language-selection
path on top of it: selecting a language by sending a different parameter is
not a mechanism that works here, and a caller that trusted it would get the
wrong phonology without an error. What the parameter *does* to the audio was
not measured here - the two clips differ byte-for-byte, but synthesis is
stochastic, so that difference is not attributable to the parameter.
**Finding 3 - the reference is carried, not ignored.** This needed a
differential, because a transcript has no timbre. Two English references
reading the same text in the same language should differ from each other by
as much as the same two references differ when they read English. Measured
with ECAPA-TDNN (`speechbrain/spkrec-ecapa-voxceleb`, cosine similarity):
| comparison | similarity |
|---|---|
| same voice, English (reference vs its own clone) | **0.821** |
| different voices, English | 0.383 |
| different voices, Chinese | **0.369** |
| first reference vs the model's own voice (Chinese) | 0.420 |
| second reference vs the model's own voice (Chinese) | 0.343 |
The two references stay about as far apart when they read Chinese (0.369) as
when they read English (0.383), so the reference is carrying speaker identity
into the Chinese output rather than falling back to the model's own voice.
**What this measurement does not establish.** ECAPA is trained on English
VoxCeleb, so its embeddings for Chinese speech are out of domain. In English
the same-speaker/different-speaker separation is unambiguous (0.821 vs
0.383); in Chinese it collapses (0.329 and 0.515 for the same-speaker pairs
against 0.369 for the different-speaker pair). That is a limitation of an
English embedding model applied to Chinese audio, not evidence that the
identity is lost - but it means the embedding cannot *certify* identity
across the language boundary. **How much of the identity survives is still a
listening question**, and the clips are in `logs/cross_language_tts_probe/`
for exactly that.
This answers the "does it speak the language at all" prerequisite. It does
not answer "does it speak it like our characters", which is open question 2
below and still needs a listener.
## Open questions before this becomes a plan
1. **Verify Chatterbox upstream.** Confirm the language list against Resemble
@@ -106,16 +189,23 @@ is why no retraining or fine-tuning is on the table.
2. **Native-speaker listening tests.** Coverage is not quality. Each language
needs a native reviewer on pronunciation, prosody and whether the cloned
identity survives. Korean and Arabic are the ones most likely to expose
problems.
problems. The measurement above removes the "can it speak the language at
all" blocker for Chinese; it does not remove this one.
3. **Text pipeline first.** Every language needs translated text *in utterance
form*, matching the existing `(chapter, utterance_index, speaker, text)`
shape, with the per-utterance emotion/pace/instruct preserved. The audio
stage is downstream of this and cannot start until it exists.
stage is downstream of this and cannot start until it exists. For Chinese
this is the delivery UmbraTrans now produces (spec 27, pass 5).
4. **Right-to-left and segmentation.** Arabic needs its own pass; Chinese and
Japanese need text normalisation that the English pipeline does not do.
5. **Cost.** Two engines is more operational surface than one. If Norwegian
and Arabic were dropped, Qwen3-TTS alone would cover 13 of 15 and there
would be no second engine at all.
6. **Does the identity survive for every speaker?** The measurement above
used two references. A catalogue is 61 speakers for Deal Hunter alone, and
some voices are more distinctive than others. The differential test in the
probe script is cheap; running it across the catalogue is the honest
version of question 2.
## Rejected, and why

74
specs/27-translation-pipeline-multipass.md
View File

@@ -319,6 +319,80 @@ Two further consequences, both measured on oldbox:
under a Spanish name. Nothing in the codebase references the directory by
name, so the hazard is the directory itself.
### Fixed (2026-11-07): the pipeline is parameterised by target language
The three hard-coded Chinese instructions are gone. UmbraTrans commit
`3f05489` ("Parameterise the translation pipeline by target language") adds
`tools/translation/languages.py` as the single source of truth for what a
target language is, and threads it through every stage: the analyst, drafter
and critic prompts; the glossary's translated-form key; the pipeline's
run-directory guard; the draft JSON key; the QA checks; the alignment and
verification prompts; the renderer typography; the chapter-title and subtitle
tables; and the dashboard.
Ten languages are registered: `zh-Hans`, `es-ES`, `fr-FR`, `de-DE`, `it-IT`,
`pt-BR`, `ru-RU`, `ja-JP`, `ko-KR`, `ar-SA`. Three rules make it safe:
1. **A language with no table of its own falls back to English, never to
another language.** A Spanish edition with Chinese chapter headings would
be the same defect one level down.
2. **The prompt scaffolding stays in Chinese.** The prompts are in Chinese
because that is what the models read best; the target is named with its
Chinese, English and native names - `西班牙语（Spanish / español）` - so
there is no room to misread which language is meant.
3. **The script is measured at every boundary where a value is declared.** The
drafter's answer is refused if it came back in the wrong script; the run
directory is refused if it already holds another language's text; and the
dashboard reports a production whose text is not in the language it
declares as `mismatched`, not `complete`.
Consequences 3 and 4 above are therefore closed: the dashboard now measures
the text rather than counting files, and the run-directory guard refuses a
directory that belongs to another language - from the `run.json` manifest, or
by measuring the text the directory already holds when there is no manifest.
The Chinese production is unchanged by this. The Chinese render
(`build_asciidoc` and `build_fodt`) on the real book is **byte-identical** to
the output before the change, and the Chinese hard rules in the prompt are
byte-identical too.
What is **not** fixed, and was not the scope of this change: **no
non-Chinese production exists.** The pipeline can now produce one; it has not
been run. Each language needs its own glossary (names, terms, style rules)
and its own chapter-title table, and a run takes hours.
### Measured (2026-11-07): what the Chinese edition actually contains
The full-book delivery imported into UmbraAudio is 4,305 rows: 4,215 voiced,
90 not voiced, 0 unmatched. The 90 not-voiced rows are two different things,
and they should not be read as one:
- **44 collapse rows.** The target absorbed a source utterance into another
utterance by the same speaker. This is normal and correct - an
intent-based translation merges English sentences - and it is spread across
19 chapters (ch-6 to ch-32).
- **46 drift rows in 9 chapters.** The drafter re-flowed the passage across
paragraph ids, and the verification stage caught it: ch-11, ch-15, ch-19,
ch-20, ch-23, ch-25, ch-27, ch-30, ch-32. These cover **33 paragraphs,
356 source words** - 1.08% of the book's paragraphs, 0.53% of its words,
about 2.4 minutes of speech at 150 wpm.
Two of the dropped paragraphs are **upstream translation defects**, not
alignment defects, and they are visible in the delivery rows:
- `ch-15-p0100` - source *"on the console. She looks around and sees a thin
layer of dust on everything. She asks,"* - Chinese *"是控制台上的"*. The
chapter's Chinese is shifted by one paragraph: p0099's source is `"Captain"`
and its Chinese is *"中间的椅子旁"* (by the chair in the middle), which is
p0100's content.
- `ch-25-p0089` - source about cuboid shapes with a control panel - Chinese
*"怎么才能接通电源？"* (how do I power it up?). The Chinese belongs to a
different sentence entirely.
Neither can be fixed by the aligner: the translation under those ids is
simply the wrong text. Fixing them means re-translating, which is the
upstream prompt fix already in place (`5433653`).
## Open questions
1. Which model runs Passes 1-4? This is where the local LLM question returns,