UmbraAudio/docs/tts/tts-engine.md

TTS engine, measured

Generated: 2026-10-15T19:02:50Z By: scripts/build_tts_facts.py (re-runnable; every value below is measured or read at run time) Document hash: abe913526a4dc330e6cebd06e5dd3381df4053a7681ad971f8a829537db53017

Where a value could not be measured it says unknown and why. No value here is an estimate presented as a fact. "A validator nobody has seen fail is not evidence" applies to this document too: anything marked measured names the file or endpoint it was read from.

Server at time of writing: GET http://localhost:8881/health → {"status": "ok", "voicedesign_loaded": false, "base_loaded": true, "current_model": "base", "device": "mps"}

---

1. Provenance

Upstream	https://github.com/QwenLM/Qwen3-TTS.git
Local HEAD	012e45298d44e8fc6ec748ba8d194bc7d5cc6c94
Upstream base	022e286b98fbec7e1e916cb940cdf532cd9f488e
Commits ahead of upstream	1
Commits behind upstream	0
License	Apache License Version 2.0, January 2004

Commits ahead:

012e452 Add UmbraAudio Voice Studio branding to web demo (icon + favicon)

The fork delta is 1 commit and it is cosmetic. It touches a logo and the web demo (qwen_tts/cli/demo.py). It does not touch the synthesis path, so it changes neither the input contract nor the output contract.

This corrects a premise in the request, which asked for "the three commits on top". There is one. The other recent commits in the log are upstream's own.

The contract that is actually ours is not in this repository. It is umbraaudio/scripts/qwen_tts_server.py — the HTTP surface, the four generation paths, and the language parameter were all written by this project. If you are writing speak against something, it is that script, not the fork.

2. The model

Model id	Weights	sha256 (first 16)
Qwen3-TTS-12Hz-0.6B-Base	1.83 GB	180b3b10eb1c9f1b
Qwen3-TTS-12Hz-0.6B-CustomVoice	no weights on disk	not loadable
Qwen3-TTS-12Hz-1.7B-Base	3.86 GB	38fc7fc51c5e776e
Qwen3-TTS-12Hz-1.7B-CustomVoice	no weights on disk	not loadable
Qwen3-TTS-12Hz-1.7B-VoiceDesign	3.83 GB	391e8db219f292c5

• Which serves which path. -VoiceDesign serves the design path (voice_description only). -Base serves clone, embed and icl (reference audio, cached embeddings, cached prompts), because the vmatrix extraction uses it.
• Two of the five directories hold no weights at all — the two -CustomVoice models. They have config.json and a README and nothing loadable. That is not damage; nothing here uses them.
• Working set. unknown. Not measured. Determining it needs a load with resident-memory sampling on the target hardware, and the answer differs between MPS and CUDA and between the two model sizes. I am not going to put a number on it that I did not watch.
• Weights checksum. sha256 of model.safetensors only, per model, in the JSON. The directory checksum is a different question — tokenizer and config files change behaviour.

For your 48 GB CPU-only node question: the two models actually used total 9.5 GB of weights on disk. Whether that is also the working set is unknown, and that is the number that decides the question: weights on disk are not the memory the process holds, and on CPU the gap between them is what decides whether the box can run it. I did not measure it, so I am not going to reason about it here as though I had.

3. Input contract

POST /v1/tts — all fields are form fields, except ref_audio which is a file upload.

Field	Default	Notes
text	None
sample_rate	24000
language	'english'	was hardcoded to english; a caller must pass it
voice_description	None	VoiceDesign path, and also style direction when combined with a prompt
ref_audio	None
ref_text	None
speaker_embedding	None	path to a cached .npy
prompt_slug	None	cached voice-clone prompt; wins over others
temperature	0.9
do_sample	True
seed	None

Four generation paths, selected by which fields are present; prompt_slug wins if the cached prompt exists:

• icl — prompt_slug -> cached vmatrix (.pt). Most faithful and most consistent.
• embed — speaker_embedding (.npy) -> coarse x-vector. Fastest.
• clone — ref_audio + ref_text -> prompt built per request.
• design — voice_description only -> VoiceDesign model.

Maximum input length: unknown. This server enforces none, and I did not measure the model's own limit. Treat as unknown rather than unbounded — an over-length input is not the same as a long input.

Streaming: whole file, returned in-memory as an audio payload. Not a stream.

4. Output contract

Measured from a file this system produced, not from a config value.

Container / codec	wav / pcm_s16le
Sample rate	24000 Hz
Channels	1
Bit depth	16 bit
Bit rate	PCM; follows from rate x depth x channels
Source	.umbra/chapter_audio/ch1/utt_0000.wav

Renders are 24 kHz. The distribution masters are 44.1 kHz. They are not the same format, and a render produced by speak should be described as the first, not the second.

5. Determinism

Identical by default: False.

"temperature 0.9 with do_sample=True and no seed. Sampling is on, so the same request is expected to differ between calls."

• Seeded call reproducible: unknown — Not measured. torch.manual_seed is called when a seed is supplied, which seeds the global generator, but reproducibility across MPS kernels, across model reloads, and across attention implementations is a separate question that needs a deliberate experiment.
• Seeded when supplied: torch.manual_seed(seed), torch.cuda.manual_seed_all(seed) if cuda
• Not seeded: sampling (temperature 0.9, do_sample True)

A translation re-voiced because one line changed will NOT reproduce the other lines byte-for-byte. Comparing takes needs a perceptual or embedding measure, not a hash.

6. Language coverage

• Declared by upstream: Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian (10 languages)
• Selection: the language form field; 'english' is the server default, 'auto' lets the model detect
• Quality ranking: unknown. No per-language evaluation has been run in this system. There is a quality gate, but it scores MOS/WER/artifact score over the book, not per language. Claiming a ranking would be an estimate.

Known trap, from a real bug here: "English phonology and prosody are applied unless language is passed. This was a real bug here: the field was hardcoded to english in every branch, so German and French text was read with English prosody. Fixed by making it a parameter."

7. Voice identity

The chain:

1. assigned_edge_voice -> an Edge-TTS voice id, e.g. en-CA-ClaraNeural:-100
2. voices/_ref.wav -> the REFERENCE AUDIO, one per speaker
3. voice_prompts/.pt -> the cached 'vmatrix' extracted from that reference
4. utterance render -> Qwen conditions on the vmatrix (prompt_slug)

What the Edge voice actually is: the FALLBACK ENGINE for creating the reference audio.

It is not the engine that renders utterances, not the clone source. Utterances are rendered by Qwen, conditioned on the vmatrix extracted from the reference audio; the Edge voice id never reaches the synthesis call.

generate_voice_reference() tries Qwen VoiceDesign up to max_retries times; only on exhaustion does it call _generate_edge_fallback() and write the Edge-derived audio to the same ref path. The Edge id is also a historical voice assignment from when Edge-TTS was the voice source.

tts_engine_used describes which engine produced the REFERENCE, not which engine rendered the utterance. Reading it as the latter is wrong.

tts_engine_used: "qwen"	this speaker's reference audio was generated by Qwen VoiceDesign
tts_engine_used: null	UNKNOWN - no decision was recorded. Not 'edge'.

Measured on Deal Hunter:

Reference WAVs on disk	93
With recorded provenance	8
With unknown provenance	85
Cached voice prompts (vmatrix)	56
Prompts with no reference WAV	31
Speakers with an Edge voice assigned	61

85 of 93 reference files have no recorded provenance. The ledger was added after those references were generated, so a null here means the system does not know, which is a different statement from 'it was Edge'.

This is a correction to how the mapping reads a field. The bundle's own convention says "null means absent". In tts_engine_used it means unknown, and the mapping should carry it as unknown rather than resolving it to "edge".

31 cached prompts have no matching reference file. A cached vmatrix outlives the audio it was built from, so a replaced reference does not automatically invalidate its prompt; the server takes force=1 to rebuild one deliberately.

8. Failure modes

Condition	What happens	Where it surfaces
Server unreachable	QwenTTSUnavailableError after retries	tts_engine_fallback_reason
Server down, building a reference	Edge audio into the ref path	voice_engines.json
Unknown prompt_slug	HTTP 404	request fails; not regenerated
ref_audio missing on /v1/voice-prompt	HTTP 400	request fails
Generation hangs	watchdog restarts the server process after 1800 s	server log
CUDA OOM / model load failure	unknown	never run on CUDA in this system

Over-length text: unknown — no limit is enforced, and the model's own limit was not measured.

9. Runtime surface

Transport	HTTP, binds 0.0.0.0:8881
Authentication	NONE
Mechanisms searched for	api_token_env, auth_middleware, authorization_header_check, bearer_scheme, fastapi_dependency, oauth2_scheme, security_scheme
Mechanisms found	none
Concurrency	1 (_model_lock)
Watchdog	1800 s
Client timeout	REMOTE_TIMEOUT = 900 s
Endpoints	GET /health, POST /load, POST /unload, POST /attn, POST /v1/tts, POST /v1/voice-prompt

No authentication mechanism is present, and it binds all interfaces. Any client - including a controller-side speak worker - reaches an open synthesis service that will happily spend minutes of CPU per request. This is a property of the deployment, not an oversight in the server code.

Concurrency limit: 1. A single asyncio.Lock serialises generation; the event loop stays responsive and work runs in threads. A speak worker that fans out will serialise here regardless of its own concurrency, and queue behind anything already generating.

---

What I could not measure

Stated once, plainly, rather than repeated as a footnote:

1. Working set / VRAM at load and at inference. Needs resident-memory sampling on the target hardware.
2. Whether a seeded call is reproducible across MPS kernels, model reloads, and attention implementations.
3. Per-language quality. No per-language evaluation exists in this system.
4. The model's own maximum text length.
5. CUDA failure modes. Never exercised here; this system runs MPS.

Each is unknown for a reason, not because it was not worth measuring. If one of them blocks speak, say which and I will run the experiment rather than substitute a guess.