umbragit/UmbraAudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

The TTS document, measured, with five honest unknowns

Browse Source 
The controller will not write a `speak` work function against a description, so
this measures instead of describing. scripts/build_tts_facts.py emits
docs/tts/tts-engine.{md,json} from the running system and the repository at run
time; anything unmeasurable says "unknown" with a reason. The content hash
excludes the timestamp, so it is stable across runs and answers "did anything
change?" rather than "was it run again?". 18 tests enforce that it does not
estimate -- including one that fails if a missing weights file is rendered as
"None", and one that fails if the Edge-voice answer is inverted, which it was, in
the first draft.

Four things that change the controller's plans:

* The fork is ONE commit ahead of upstream and it is cosmetic -- a logo and the
  web demo. The contract being written against is not the fork, it is
  umbraaudio/scripts/qwen_tts_server.py, where the HTTP surface and the four
  generation paths were written.
* Synthesis is not deterministic. temperature 0.9, do_sample true, no seed. A
  translation re-voiced because one line changed will not reproduce the other
  lines byte-for-byte, so comparing takes needs an embedding measure, not a hash.
* The TTS server has NO authentication and binds 0.0.0.0:8881, and concurrency
  is 1 behind a single lock. The generator names the seven auth mechanisms it
  searched for so the claim can be checked rather than believed.
* assigned_edge_voice is the fallback engine for creating reference audio, not
  the engine that renders utterances.

Not measured, and said so: working set / VRAM (the one that decides the 48 GB
CPU node question -- 9.5 GB of weights on disk is not the memory the process
holds, and on CPU that gap is the whole question); whether a seeded call is
reproducible; per-language quality; the model's own maximum text length; CUDA
failure modes.

scripts/build_scale_totals.py measures across all projects rather than handing
over a typed number: 4 projects, 131 chapters, 18,181 utterances, 14,526 audio
files, 6.98 GB, 38.58 hours, and 2 distinct schema fingerprints -- matching the
two already recorded in manifest.md. Markers are counted separately from audio,
because "files" that means both is a number nobody can use. Treasure Island
contributes 49 chapters and zero audio: that is a fact about it, not a failed
measurement.

Also delivered to the controller: their accepted-format validator accepts the
bundle, exit 0, 8884 records, and I confirmed it fails on a broken copy before
believing the pass.
This commit is contained in:
B.D. Murphy
2026-10-15 12:07:15 -07:00
parent 3d69187449
commit 4c1e6ebc98
6 changed files with 1917 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

138
docs/tts/scale-totals.json Normal file
View File

@@ -0,0 +1,138 @@
{
"caveats": [
"Duration is read from WAV headers, not decoded. It is exact for PCM WAV and absent for anything else, which is why the file count and the duration are reported together rather than one from the other.",
"Audio totals cover chapter_audio only, and exclude the *.wav.txt staleness markers, which are counted separately. Voice references and exports are counted separately per project.",
"Scratch and intermediate output is excluded and counted separately, not silently dropped.",
"Projects without a populated database are not listed. Absence here means 'not measured', not 'does not exist'."
],
"content_hash": {
"algorithm": "sha256",
"of": "this document with generated_at omitted, canonical JSON",
"value": "f8f6d71f2f5c94821c7195911da184b79c37b221c9a4e1c4a398c34c9cc65555"
},
"document_version": 1,
"generated_at": "2026-10-15T19:03:52Z",
"generated_by": "scripts/build_scale_totals.py",
"projects": [
{
"audio": {
"audio_bytes": 2465148533,
"audio_duration_hours": 14.25,
"audio_duration_ms": 51284220,
"audio_files": 4358,
"files_with_unreadable_header": 1,
"marker_bytes": 859884,
"marker_files_excluded_from_audio": 4319,
"scratch_bytes_excluded": 0,
"scratch_files_excluded": 0
},
"chapters": 32,
"export": {
"bytes": 1076100454,
"files": 64
},
"project": "DealHunterShip_4.0e_whole-to-audio",
"schema_fingerprint": "399285355e2d456e",
"speakers_named": 55,
"utterances": 4319,
"voices": {
"bytes": 109829218,
"files": 104
},
"words": 67378
},
{
"audio": {
"audio_bytes": 4491799819,
"audio_duration_hours": 24.22,
"audio_duration_ms": 87180676,
"audio_files": 10129,
"files_with_unreadable_header": 9,
"marker_bytes": 2065419,
"marker_files_excluded_from_audio": 10015,
"scratch_bytes_excluded": 0,
"scratch_files_excluded": 0
},
"chapters": 49,
"export": {
"bytes": 0,
"files": 0
},
"project": "Last Dwarf",
"schema_fingerprint": "399285355e2d456e",
"speakers_named": 72,
"utterances": 10015,
"voices": {
"bytes": 126078048,
"files": 72
},
"words": 107356
},
{
"audio": {
"audio_bytes": 0,
"audio_duration_hours": 0.0,
"audio_duration_ms": 0,
"audio_files": 0,
"files_with_unreadable_header": 0,
"marker_bytes": 0,
"marker_files_excluded_from_audio": 0,
"scratch_bytes_excluded": 0,
"scratch_files_excluded": 0
},
"chapters": 49,
"export": {
"bytes": 0,
"files": 0
},
"project": "Treasure Island",
"schema_fingerprint": "b7c9ae76bc6615bd",
"speakers_named": 34,
"utterances": 3811,
"voices": {
"bytes": 59429776,
"files": 44
},
"words": 68937
},
{
"audio": {
"audio_bytes": 25945398,
"audio_duration_hours": 0.12,
"audio_duration_ms": 427260,
"audio_files": 39,
"files_with_unreadable_header": 0,
"marker_bytes": 10296,
"marker_files_excluded_from_audio": 36,
"scratch_bytes_excluded": 0,
"scratch_files_excluded": 0
},
"chapters": 1,
"export": {
"bytes": 4950352,
"files": 1
},
"project": "Vocals",
"schema_fingerprint": "399285355e2d456e",
"speakers_named": 2,
"utterances": 36,
"voices": {
"bytes": 4434938,
"files": 2
},
"words": 98
}
],
"root": "<REPO>",
"totals": {
"audio_bytes": 6982893750,
"audio_duration_hours": 38.58,
"audio_files": 14526,
"audio_gigabytes": 6.98,
"chapters": 131,
"distinct_schema_fingerprints": 2,
"marker_files": 14370,
"projects": 4,
"utterances": 18181
}
}

291
docs/tts/tts-engine.json Normal file
View File

@@ -0,0 +1,291 @@
{
"attn_implementation_now": {
"impl": "sdpa",
"ok": true
},
"content_hash": {
"algorithm": "sha256",
"of": "this document with generated_at and content_hash omitted, canonical JSON",
"value": "abe913526a4dc330e6cebd06e5dd3381df4053a7681ad971f8a829537db53017"
},
"determinism": {
"consequence": "A translation re-voiced because one line changed will NOT reproduce the other lines byte-for-byte. Comparing takes needs a perceptual or embedding measure, not a hash.",
"identical_by_default": false,
"seeded": [
"torch.manual_seed(seed)",
"torch.cuda.manual_seed_all(seed) if cuda"
],
"seeded_reproducible": "unknown",
"seeded_reproducible_why": "Not measured. torch.manual_seed is called when a seed is supplied, which seeds the global generator, but reproducibility across MPS kernels, across model reloads, and across attention implementations is a separate question that needs a deliberate experiment.",
"unseeded_sources": [
"sampling (temperature 0.9, do_sample True)"
],
"why": "temperature 0.9 with do_sample=True and no seed. Sampling is on, so the same request is expected to differ between calls."
},
"document_version": 1,
"generated_at": "2026-10-15T19:02:50Z",
"generated_by": "scripts/build_tts_facts.py",
"languages": {
"count": 10,
"declared_by_upstream": [
"Chinese",
"English",
"Japanese",
"Korean",
"German",
"French",
"Russian",
"Portuguese",
"Spanish",
"Italian"
],
"known_weak": {
"en": "English phonology and prosody are applied unless `language` is passed. This was a real bug here: the field was hardcoded to english in every branch, so German and French text was read with English prosody. Fixed by making it a parameter."
},
"quality_weaknesses": "unknown",
"quality_weaknesses_why": "No per-language evaluation has been run in this system. There is a quality gate, but it scores MOS/WER/artifact score over the book, not per language. Claiming a ranking would be an estimate.",
"selection": "the `language` form field; 'english' is the server default, 'auto' lets the model detect"
},
"models": {
"loaded_weights_bytes": 9520160568,
"models_present": [
{
"config_present": true,
"directory_bytes": 2516107403,
"model_id": "Qwen3-TTS-12Hz-0.6B-Base",
"weights_bytes": 1829344272,
"weights_file": "model.safetensors",
"weights_sha256": "180b3b10eb1c9f1b4db7806d5475bae3071c0243c299d49926bab1da3b6946f6",
"weights_state": "present"
},
{
"config_present": true,
"directory_bytes": 272829144,
"model_id": "Qwen3-TTS-12Hz-0.6B-CustomVoice",
"weights_bytes": null,
"weights_file": null,
"weights_sha256": null,
"weights_state": "absent"
},
{
"config_present": true,
"directory_bytes": 4544231050,
"model_id": "Qwen3-TTS-12Hz-1.7B-Base",
"weights_bytes": 3857413744,
"weights_file": "model.safetensors",
"weights_sha256": "38fc7fc51c5e776e840414b6fd443962e9411b9654888fd7913e4da643cb857c",
"weights_state": "present"
},
{
"config_present": true,
"directory_bytes": 406973095,
"model_id": "Qwen3-TTS-12Hz-1.7B-CustomVoice",
"weights_bytes": null,
"weights_file": null,
"weights_sha256": null,
"weights_state": "absent"
},
{
"config_present": true,
"directory_bytes": 4520165283,
"model_id": "Qwen3-TTS-12Hz-1.7B-VoiceDesign",
"weights_bytes": 3833402552,
"weights_file": "model.safetensors",
"weights_sha256": "391e8db219f292c515297cdceeb43e4eae67cdde35fa57e79a6a8a532fca0522",
"weights_state": "present"
}
],
"models_without_weights": [
"Qwen3-TTS-12Hz-0.6B-CustomVoice",
"Qwen3-TTS-12Hz-1.7B-CustomVoice"
],
"note": "weights_sha256 is of model.safetensors only, not the whole directory. A model directory also holds config and tokenizer files that change behaviour. A directory listed under models_without_weights has config but no weights at all - it cannot be loaded, and its absence is not an error.",
"total_directory_bytes": 12260305975
},
"output_contract": {
"consistency_note": "Renders are 24 kHz. The distribution masters are 44.1 kHz. They are not the same format, and a render produced by `speak` should be described as the first, not the second.",
"distribution_master": {
"bit_rate_bps": "192000",
"channels": 1,
"codec": "mp3",
"container": "mp3",
"duration_s": "2233.100000",
"measured_from": ".umbra/export/ar_package/Deal_Hunter_01_Salvage.mp3",
"sample_rate_hz": "44100"
},
"rendered_utterance": {
"bit_depth_bits": 16,
"bit_rate_bps": null,
"bit_rate_note": "PCM; follows from rate x depth x channels",
"channels": 1,
"codec": "pcm_s16le",
"container": "wav",
"duration_ms": 10560,
"measured_from": ".umbra/chapter_audio/ch1/utt_0000.wav",
"sample_rate_hz": 24000
}
},
"provenance": {
"commits_ahead_detail": [
"012e452 Add UmbraAudio Voice Studio branding to web demo (icon + favicon)"
],
"commits_ahead_of_upstream": 1,
"commits_behind_upstream": 0,
"diff_stat_vs_base": "assets/logo.png | Bin 0 -> 1102415 bytes\n qwen_tts/cli/demo.py | 65 ++++++++++++++++++++++++++++++++++++++++++++++++---\n 2 files changed, 62 insertions(+), 3 deletions(-)",
"license": "Apache License Version 2.0, January 2004",
"local_head": "012e45298d44e8fc6ec748ba8d194bc7d5cc6c94",
"note": "The fork delta touches only the web demo and a logo. It does not touch the synthesis path, so it does not alter the input or output contract. The contract that IS ours is umbraaudio's own server script, which is not part of this repository - see runtime_surface.",
"upstream_base_commit": "022e286b98fbec7e1e916cb940cdf532cd9f488e",
"upstream_repository": "https://github.com/QwenLM/Qwen3-TTS.git"
},
"runtime_surface": {
"authentication": {
"binds": "0.0.0.0:8881",
"mechanisms_found": [],
"mechanisms_searched": [
"api_token_env",
"auth_middleware",
"authorization_header_check",
"bearer_scheme",
"fastapi_dependency",
"oauth2_scheme",
"security_scheme"
],
"note": "No authentication mechanism is present, and it binds all interfaces. Any client - including a controller-side speak worker - reaches an open synthesis service that will happily spend minutes of CPU per request. This is a property of the deployment, not an oversight in the server code.",
"requires_authentication": false
},
"concurrency": {
"concurrent_generations": 1,
"model_lock": true,
"watchdog_restart_seconds": 1800,
"why": "a single asyncio.Lock serialises generation; the event loop stays responsive and work runs in threads"
},
"endpoints": {
"attn": {
"method": "POST",
"path": "/attn"
},
"generate": {
"method": "POST",
"path": "/v1/tts"
},
"health": {
"method": "GET",
"path": "/health"
},
"load": {
"method": "POST",
"path": "/load"
},
"unload": {
"method": "POST",
"path": "/unload"
},
"voice_prompt": {
"method": "POST",
"path": "/v1/voice-prompt"
}
},
"generation_paths": {
"clone": "ref_audio + ref_text -> prompt built per request.",
"design": "voice_description only -> VoiceDesign model.",
"embed": "speaker_embedding (.npy) -> coarse x-vector. Fastest.",
"icl": "prompt_slug -> cached vmatrix (.pt). Most faithful and most consistent."
},
"input_fields": {
"do_sample": {
"default": true
},
"language": {
"accepts": [
"english",
"auto",
"<other>"
],
"default": "english",
"note": "was hardcoded to english; a caller must pass it"
},
"prompt_slug": {
"default": null,
"note": "cached voice-clone prompt; wins over others"
},
"ref_audio": {
"default": null,
"type": "file"
},
"ref_text": {
"default": null
},
"sample_rate": {
"default": 24000
},
"seed": {
"default": null
},
"speaker_embedding": {
"default": null,
"note": "path to a cached .npy"
},
"temperature": {
"default": 0.9
},
"text": {
"required": true,
"type": "form"
},
"voice_description": {
"default": null,
"note": "VoiceDesign path, and also style direction when combined with a prompt"
}
},
"output": {
"max_text_length_enforced_by_server": null,
"max_text_length_note": "No length limit is enforced by this server. Any limit is the model's own and was not measured; treat as unknown rather than unbounded.",
"shape": "whole file, returned in-memory as an audio payload",
"streaming": false
}
},
"server_health_now": {
"base_loaded": true,
"current_model": "base",
"device": "mps",
"status": "ok",
"voicedesign_loaded": false
},
"source_system": {
"node_id": "umbraaudio-m4",
"system": "umbraaudio"
},
"voice_identity": {
"chain": [
"assigned_edge_voice -> an Edge-TTS voice id, e.g. en-CA-ClaraNeural:-100",
"voices/<name>_ref.wav -> the REFERENCE AUDIO, one per speaker",
"voice_prompts/<slug>.pt -> the cached 'vmatrix' extracted from that reference",
"utterance render -> Qwen conditions on the vmatrix (prompt_slug)"
],
"counts": {
"cached_voice_prompts": 56,
"ledger_entries": 9,
"prompts_without_a_reference_wav": 31,
"reference_wavs_on_disk": 93,
"refs_with_recorded_provenance": 8,
"refs_with_unknown_provenance": 85,
"speakers_with_an_edge_voice_assigned": 61
},
"orphan_prompts_note": "31 cached prompts have no matching reference file. A cached vmatrix outlives the audio it was built from, so a replaced reference does not automatically invalidate its prompt; the server takes force=1 to rebuild one deliberately.",
"tts_engine_used_semantics": {
"null": "UNKNOWN - no decision was recorded. Not 'edge'.",
"qwen": "this speaker's reference audio was generated by Qwen VoiceDesign"
},
"unknown_provenance_is_the_norm": "85 of 93 reference files have no recorded provenance. The ledger was added after those references were generated, so a null here means the system does not know, which is a different statement from 'it was Edge'.",
"what_the_edge_voice_actually_is": {
"answer": "the FALLBACK ENGINE for creating the reference audio",
"consequence": "tts_engine_used describes which engine produced the REFERENCE, not which engine rendered the utterance. Reading it as the latter is wrong.",
"not": [
"the engine that renders utterances",
"the clone source"
],
"why": "generate_voice_reference() tries Qwen VoiceDesign up to max_retries times; only on exhaustion does it call _generate_edge_fallback() and write the Edge-derived audio to the same ref path. The Edge id is also a historical voice assignment from when Edge-TTS was the voice source."
}
}
}

242
docs/tts/tts-engine.md Normal file
View File

@@ -0,0 +1,242 @@
# TTS engine, measured
**Generated:** 2026-10-15T19:02:50Z
**By:** `scripts/build_tts_facts.py` (re-runnable; every value below is measured or read at run time)
**Document hash:** `abe913526a4dc330e6cebd06e5dd3381df4053a7681ad971f8a829537db53017`
Where a value could not be measured it says **unknown** and why. No value here is
an estimate presented as a fact. "A validator nobody has seen fail is not
evidence" applies to this document too: anything marked measured names the file
or endpoint it was read from.
**Server at time of writing:** `GET http://localhost:8881/health` →
`{"status": "ok", "voicedesign_loaded": false, "base_loaded": true, "current_model": "base", "device": "mps"}`
---
## 1. Provenance
| | |
|---|---|
| Upstream | `https://github.com/QwenLM/Qwen3-TTS.git` |
| Local HEAD | `012e45298d44e8fc6ec748ba8d194bc7d5cc6c94` |
| Upstream base | `022e286b98fbec7e1e916cb940cdf532cd9f488e` |
| Commits ahead of upstream | **1** |
| Commits behind upstream | 0 |
| License | Apache License Version 2.0, January 2004 |
Commits ahead:
```
012e452 Add UmbraAudio Voice Studio branding to web demo (icon + favicon)
```
**The fork delta is 1 commit and it is cosmetic.**
It touches a logo and the web demo (`qwen_tts/cli/demo.py`). It does not touch the
synthesis path, so it changes neither the input contract nor the output contract.
This corrects a premise in the request, which asked for "the three commits on
top". There is one. The other recent commits in the log are upstream's own.
**The contract that is actually ours is not in this repository.** It is
`umbraaudio/scripts/qwen_tts_server.py` — the HTTP surface, the four generation
paths, and the `language` parameter were all written by this project. If you are
writing `speak` against something, it is that script, not the fork.
## 2. The model
| Model id | Weights | sha256 (first 16) |
|---|---|---|
| `Qwen3-TTS-12Hz-0.6B-Base` | 1.83 GB | `180b3b10eb1c9f1b` |
| `Qwen3-TTS-12Hz-0.6B-CustomVoice` | no weights on disk | `not loadable` |
| `Qwen3-TTS-12Hz-1.7B-Base` | 3.86 GB | `38fc7fc51c5e776e` |
| `Qwen3-TTS-12Hz-1.7B-CustomVoice` | no weights on disk | `not loadable` |
| `Qwen3-TTS-12Hz-1.7B-VoiceDesign` | 3.83 GB | `391e8db219f292c5` |
- **Which serves which path.** `-VoiceDesign` serves the `design` path
(`voice_description` only). `-Base` serves `clone`, `embed` and `icl`
(reference audio, cached embeddings, cached prompts), because the vmatrix
extraction uses it.
- **Two of the five directories hold no weights at all** — the two
`-CustomVoice` models. They have `config.json` and a README and nothing
loadable. That is not damage; nothing here uses them.
- **Working set.** **unknown.** Not measured. Determining it needs a load with
resident-memory sampling on the target hardware, and the answer differs between
MPS and CUDA and between the two model sizes. I am not going to put a number
on it that I did not watch.
- **Weights checksum.** sha256 of `model.safetensors` only, per model, in the JSON.
The directory checksum is a different question — tokenizer and config files
change behaviour.
**For your 48 GB CPU-only node question:** the two models actually used total
9.5 GB of weights on disk.
Whether that is also the *working set* is unknown, and that is the number that
decides the question: weights on disk are not the memory the process holds, and
on CPU the gap between them is what decides whether the box can run it. I did not
measure it, so I am not going to reason about it here as though I had.
## 3. Input contract
`POST /v1/tts` — all fields are form fields, except `ref_audio` which is a file
upload.
| Field | Default | Notes |
|---|---|---|
| `text` | None | |
| `sample_rate` | 24000 | |
| `language` | 'english' | was hardcoded to english; a caller must pass it |
| `voice_description` | None | VoiceDesign path, and also style direction when combined with a prompt |
| `ref_audio` | None | |
| `ref_text` | None | |
| `speaker_embedding` | None | path to a cached .npy |
| `prompt_slug` | None | cached voice-clone prompt; wins over others |
| `temperature` | 0.9 | |
| `do_sample` | True | |
| `seed` | None | |
**Four generation paths**, selected by which fields are present; `prompt_slug`
wins if the cached prompt exists:
- **icl** — prompt_slug -> cached vmatrix (.pt). Most faithful and most consistent.
- **embed** — speaker_embedding (.npy) -> coarse x-vector. Fastest.
- **clone** — ref_audio + ref_text -> prompt built per request.
- **design** — voice_description only -> VoiceDesign model.
**Maximum input length:** unknown. This server enforces none, and I did not
measure the model's own limit. Treat as unknown rather than unbounded — an
over-length input is not the same as a long input.
**Streaming:** whole file, returned in-memory as an audio payload. Not a stream.
## 4. Output contract
Measured from a file this system produced, not from a config value.
| | |
|---|---|
| Container / codec | wav / pcm_s16le |
| Sample rate | 24000 Hz |
| Channels | 1 |
| Bit depth | 16 bit |
| Bit rate | PCM; follows from rate x depth x channels |
| Source | `.umbra/chapter_audio/ch1/utt_0000.wav` |
Renders are 24 kHz. The distribution masters are 44.1 kHz. They are not the same format, and a render produced by `speak` should be described as the first, not the second.
## 5. Determinism
**Identical by default: False.**
"temperature 0.9 with do_sample=True and no seed. Sampling is on, so the same request is expected to differ between calls."
- Seeded call reproducible: **unknown** —
Not measured. torch.manual_seed is called when a seed is supplied, which seeds the global generator, but reproducibility across MPS kernels, across model reloads, and across attention implementations is a separate question that needs a deliberate experiment.
- Seeded when supplied: `torch.manual_seed(seed)`, `torch.cuda.manual_seed_all(seed) if cuda`
- Not seeded: `sampling (temperature 0.9, do_sample True)`
**A translation re-voiced because one line changed will NOT reproduce the other lines byte-for-byte. Comparing takes needs a perceptual or embedding measure, not a hash.**
## 6. Language coverage
- Declared by upstream: Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
(10 languages)
- Selection: the `language` form field; 'english' is the server default, 'auto' lets the model detect
- **Quality ranking: unknown.** No per-language evaluation has been run in this system. There is a quality gate, but it scores MOS/WER/artifact score over the book, not per language. Claiming a ranking would be an estimate.
**Known trap, from a real bug here:** "English phonology and prosody are applied unless `language` is passed. This was a real bug here: the field was hardcoded to english in every branch, so German and French text was read with English prosody. Fixed by making it a parameter."
## 7. Voice identity
The chain:
1. assigned_edge_voice -> an Edge-TTS voice id, e.g. en-CA-ClaraNeural:-100
2. voices/<name>_ref.wav -> the REFERENCE AUDIO, one per speaker
3. voice_prompts/<slug>.pt -> the cached 'vmatrix' extracted from that reference
4. utterance render -> Qwen conditions on the vmatrix (prompt_slug)
**What the Edge voice actually is:**
the FALLBACK ENGINE for creating the reference audio.
It is **not** the engine that renders utterances, **not** the clone source.
Utterances are rendered by Qwen, conditioned on the vmatrix extracted from the
reference audio; the Edge voice id never reaches the synthesis call.
generate_voice_reference() tries Qwen VoiceDesign up to max_retries times; only on exhaustion does it call _generate_edge_fallback() and write the Edge-derived audio to the same ref path. The Edge id is also a historical voice assignment from when Edge-TTS was the voice source.
**tts_engine_used describes which engine produced the REFERENCE, not which engine rendered the utterance. Reading it as the latter is wrong.**
| | |
|---|---|
| `tts_engine_used: "qwen"` | this speaker's reference audio was generated by Qwen VoiceDesign |
| `tts_engine_used: null` | **UNKNOWN - no decision was recorded. Not 'edge'.** |
Measured on Deal Hunter:
| | |
|---|---|
| Reference WAVs on disk | 93 |
| With recorded provenance | 8 |
| **With unknown provenance** | **85** |
| Cached voice prompts (vmatrix) | 56 |
| Prompts with no reference WAV | 31 |
| Speakers with an Edge voice assigned | 61 |
85 of 93 reference files have no recorded provenance. The ledger was added after those references were generated, so a null here means the system does not know, which is a different statement from 'it was Edge'.
**This is a correction to how the mapping reads a field.** The bundle's own
convention says "null means absent". In `tts_engine_used` it means *unknown*, and
the mapping should carry it as unknown rather than resolving it to "edge".
31 cached prompts have no matching reference file. A cached vmatrix outlives the audio it was built from, so a replaced reference does not automatically invalidate its prompt; the server takes force=1 to rebuild one deliberately.
## 8. Failure modes
| Condition | What happens | Where it surfaces |
|---|---|---|
| Server unreachable | `QwenTTSUnavailableError` after retries | `tts_engine_fallback_reason` |
| Server down, building a reference | Edge audio into the ref path | `voice_engines.json` |
| Unknown `prompt_slug` | HTTP 404 | request fails; not regenerated |
| `ref_audio` missing on `/v1/voice-prompt` | HTTP 400 | request fails |
| Generation hangs | watchdog restarts the server process after 1800 s | server log |
| CUDA OOM / model load failure | **unknown** | never run on CUDA in this system |
Over-length text: **unknown** — no limit is enforced, and the model's own limit
was not measured.
## 9. Runtime surface
| | |
|---|---|
| Transport | HTTP, binds `0.0.0.0:8881` |
| **Authentication** | **NONE** |
| Mechanisms searched for | `api_token_env`, `auth_middleware`, `authorization_header_check`, `bearer_scheme`, `fastapi_dependency`, `oauth2_scheme`, `security_scheme` |
| Mechanisms found | none |
| Concurrency | **1** (`_model_lock`) |
| Watchdog | 1800 s |
| Client timeout | `REMOTE_TIMEOUT = 900` s |
| Endpoints | `GET /health`, `POST /load`, `POST /unload`, `POST /attn`, `POST /v1/tts`, `POST /v1/voice-prompt` |
No authentication mechanism is present, and it binds all interfaces. Any client - including a controller-side speak worker - reaches an open synthesis service that will happily spend minutes of CPU per request. This is a property of the deployment, not an oversight in the server code.
**Concurrency limit: 1.**
A single asyncio.Lock serialises generation; the event loop stays responsive and work runs in threads. A `speak` worker that fans out
will serialise here regardless of its own concurrency, and queue behind anything
already generating.
---
## What I could not measure
Stated once, plainly, rather than repeated as a footnote:
1. **Working set / VRAM at load and at inference.** Needs resident-memory
sampling on the target hardware.
2. **Whether a seeded call is reproducible** across MPS kernels, model reloads,
and attention implementations.
3. **Per-language quality.** No per-language evaluation exists in this system.
4. **The model's own maximum text length.**
5. **CUDA failure modes.** Never exercised here; this system runs MPS.
Each is unknown for a reason, not because it was not worth measuring. If one of
them blocks `speak`, say which and I will run the experiment rather than
substitute a guess.

223
scripts/build_scale_totals.py Normal file
View File

@@ -0,0 +1,223 @@
#!/usr/bin/env python3
"""Measure totals across every project on this machine.
Asked for as "books, hours, bytes, so the catalogue is something a person
maintains or something that has to be generated". Handing over a number typed
into a chat message would be exactly the estimate the request refuses, so this
measures instead.
Two separate quantities, because conflating them is how a catalogue ends up
wrong:
* **Project data** — chapters, utterances, speakers. Cheap, exact, from SQLite.
* **Audio on disk** — files, bytes, duration. Expensive to measure exactly, so
duration is measured from headers and reported with the caveat that it is a
header read and not a decode.
python3 scripts/build_scale_totals.py --out docs/tts/scale-totals.json
Read-only. Opens each project database with ``mode=ro`` and walks directories.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import sqlite3
import wave
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
REPO = Path(__file__).resolve().parent.parent
#: Directories that hold intermediate work rather than product. Counted
#: separately, because a catalogue that sweeps scratch into the product is a
#: catalogue nobody can answer a question against.
SCRATCH_DIRS = ("cache", "archives", "pronunciation_audio", "voice_tests",
"samples", "debug", "tmp")
def schema_fingerprint(conn: sqlite3.Connection) -> str:
"""Fingerprint of the table/column *shape*, not the data.
Two projects with different books can share a fingerprint. That is the
point: it answers "is this the same schema", which cannot be confused with
"is this the same data". Counting one table's presence is a proxy for that
and was replaced by the real thing.
"""
parts: list[str] = []
tables = [row[0] for row in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' "
"AND name NOT LIKE 'sqlite_%' ORDER BY name")]
for table in tables:
for row in conn.execute(f"PRAGMA table_info('{table}')"):
parts.append(f"{table}.{row[1]}:{row[2]}")
return hashlib.sha256("\n".join(sorted(parts)).encode()).hexdigest()[:16]
def measure_project(directory: Path) -> dict[str, Any] | None:
umbra = directory / ".umbra"
db = umbra / "project.db"
if not db.exists() or db.stat().st_size == 0:
return None
data: dict[str, Any] = {"project": directory.name}
try:
conn = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
for label, sql in (
("chapters", "SELECT count(*) FROM chapters"),
("utterances", "SELECT count(*) FROM utterances"),
("speakers_named", "SELECT count(DISTINCT trim(speaker_name)) FROM utterances "
"WHERE speaker_name IS NOT NULL"),
("words", "SELECT sum(word_count) FROM chapters"),
):
try:
data[label] = conn.execute(sql).fetchone()[0]
except sqlite3.Error:
data[label] = None
data["schema_fingerprint"] = schema_fingerprint(conn)
conn.close()
except sqlite3.Error as error:
data["error"] = str(error)[:160]
return data
# --- audio on disk ------------------------------------------------
chapter_audio = umbra / "chapter_audio"
audio_files = marker_files = 0
audio_bytes = marker_bytes = scratch_bytes = 0
scratch = 0
duration_ms = 0
unreadable = 0
if chapter_audio.is_dir():
for path in chapter_audio.rglob("*"):
if not path.is_file():
continue
relative = path.relative_to(chapter_audio)
if any(part in SCRATCH_DIRS for part in relative.parts):
scratch += 1
scratch_bytes += path.stat().st_size
continue
# A render and its staleness marker are different things and are
# counted separately: one is audio, the other is a JSON sidecar,
# and lumping them makes "files" mean neither.
if path.name.endswith(".wav.txt"):
marker_files += 1
marker_bytes += path.stat().st_size
continue
audio_files += 1
audio_bytes += path.stat().st_size
if path.suffix.lower() == ".wav":
try:
with wave.open(str(path)) as handle:
rate = handle.getframerate()
if rate:
duration_ms += round(handle.getnframes() / rate * 1000)
except (OSError, wave.Error, EOFError):
unreadable += 1
data["audio"] = {
"audio_files": audio_files,
"audio_bytes": audio_bytes,
"audio_duration_ms": duration_ms,
"audio_duration_hours": round(duration_ms / 3_600_000, 2),
"files_with_unreadable_header": unreadable,
"marker_files_excluded_from_audio": marker_files,
"marker_bytes": marker_bytes,
"scratch_files_excluded": scratch,
"scratch_bytes_excluded": scratch_bytes,
}
for label, directory_name in (("voices", "voices"), ("export", "export")):
target = umbra / directory_name
if target.is_dir():
data[label] = {
"files": sum(1 for p in target.rglob("*") if p.is_file()),
"bytes": sum(p.stat().st_size for p in target.rglob("*") if p.is_file()),
}
return data
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--root", default=str(REPO))
parser.add_argument("--out", default=str(REPO / "docs" / "tts" / "scale-totals.json"))
args = parser.parse_args()
root = Path(args.root)
projects = []
for directory in sorted(root.iterdir()):
if not directory.is_dir() or directory.name.startswith("."):
continue
measured = measure_project(directory)
if measured is not None:
projects.append(measured)
chapters = sum(p.get("chapters") or 0 for p in projects)
utterances = sum(p.get("utterances") or 0 for p in projects)
files = sum((p.get("audio") or {}).get("audio_files") or 0 for p in projects)
total_bytes = sum((p.get("audio") or {}).get("audio_bytes") or 0 for p in projects)
duration = sum((p.get("audio") or {}).get("audio_duration_ms") or 0 for p in projects)
markers = sum((p.get("audio") or {}).get("marker_files_excluded_from_audio") or 0
for p in projects)
document: dict[str, Any] = {
"document_version": 1,
"generated_at": datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z"),
"generated_by": "scripts/build_scale_totals.py",
"root": "<REPO>",
"projects": projects,
"totals": {
"projects": len(projects),
"chapters": chapters,
"utterances": utterances,
"audio_files": files,
"audio_bytes": total_bytes,
"audio_gigabytes": round(total_bytes / 1e9, 2),
"audio_duration_hours": round(duration / 3_600_000, 2),
"marker_files": markers,
"distinct_schema_fingerprints": len(
{p.get("schema_fingerprint") for p in projects if p.get("schema_fingerprint")}),
},
"caveats": [
"Duration is read from WAV headers, not decoded. It is exact for "
"PCM WAV and absent for anything else, which is why the file count "
"and the duration are reported together rather than one from the other.",
"Audio totals cover chapter_audio only, and exclude the *.wav.txt "
"staleness markers, which are counted separately. Voice references "
"and exports are counted separately per project.",
"Scratch and intermediate output is excluded and counted "
"separately, not silently dropped.",
"Projects without a populated database are not listed. Absence here "
"means 'not measured', not 'does not exist'.",
],
}
hashed = {k: v for k, v in document.items() if k != "generated_at"}
payload = json.dumps(hashed, sort_keys=True, separators=(",", ":"))
document["content_hash"] = {
"algorithm": "sha256",
"of": "this document with generated_at omitted, canonical JSON",
"value": hashlib.sha256(payload.encode()).hexdigest(),
}
out = Path(args.out)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(document, indent=2, sort_keys=True) + "\n")
totals = document["totals"]
print(f"projects: {totals['projects']} chapters: {totals['chapters']} "
f"utterances: {totals['utterances']}")
print(f"audio: {totals['audio_files']} files, "
f"{totals['audio_gigabytes']} GB, "
f"{totals['audio_duration_hours']} h "
f"(+{totals['marker_files']} markers, "
f"{totals['distinct_schema_fingerprints']} schema shapes)")
print(f"hash: {document['content_hash']['value']}")
print(f"written: {out}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

749
scripts/build_tts_facts.py Normal file
View File

@@ -0,0 +1,749 @@
#!/usr/bin/env python3
"""Measure the TTS engine and emit the checked document the controller asked for.
The controller will not write a ``speak`` work function against a description.
Every value here is measured from the running system or read from the repository
at run time, and anything that cannot be measured is recorded as ``unknown`` with
a reason rather than estimated. A validator nobody has seen fail is not evidence
of anything; the same rule applies to a document nobody has checked.
python3 scripts/build_tts_facts.py --out-dir docs/tts
Emits ``tts-engine.json`` (measured, with a content hash) and ``tts-engine.md``
(the same values in prose, with evidence commands). Read-only: it probes the
running TTS server's ``/health`` and reads files. It never generates audio and
never writes to the model directory.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import subprocess
import urllib.request
import wave
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
REPO = Path(__file__).resolve().parent.parent
TTS_URL = "http://localhost:8881"
def sha256_file(path: Path) -> str | None:
try:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
except OSError:
return None
def git(*args: str, cwd: Path) -> str | None:
try:
out = subprocess.run(["git", "-C", str(cwd), *args],
capture_output=True, text=True, timeout=60, check=False)
return out.stdout.strip() or None
except (OSError, subprocess.SubprocessError):
return None
def probe_json(url: str) -> Any:
try:
with urllib.request.urlopen(url, timeout=8) as response:
return json.loads(response.read())
except Exception as error: # noqa: BLE001 - reachability is a result
return {"error": {"code": type(error).__name__, "message": str(error)[:160]}}
def ffprobe(path: Path) -> dict[str, Any]:
try:
out = subprocess.run(
["ffprobe", "-v", "error", "-select_streams", "a:0",
"-show_entries", "stream=codec_name,sample_rate,channels,bit_rate",
"-show_entries", "format=duration,bit_rate,format_name",
"-of", "json", str(path)],
capture_output=True, text=True, timeout=60, check=False)
data = json.loads(out.stdout or "{}")
stream = (data.get("streams") or [{}])[0]
fmt = data.get("format") or {}
return {"codec": stream.get("codec_name"), "sample_rate_hz": stream.get("sample_rate"),
"channels": stream.get("channels"), "bit_rate_bps": stream.get("bit_rate"),
"container": fmt.get("format_name"), "duration_s": fmt.get("duration")}
except (OSError, ValueError, subprocess.SubprocessError) as error:
return {"error": type(error).__name__}
# --------------------------------------------------------------------------
# 1. provenance
# --------------------------------------------------------------------------
def provenance() -> dict[str, Any]:
repo = REPO / "Qwen3-TTS"
if not (repo / ".git").exists():
return {"status": "unknown", "why": "Qwen3-TTS submodule not present"}
head = git("rev-parse", "HEAD", cwd=repo)
base = git("merge-base", "HEAD", "origin/main", cwd=repo)
ahead = (git("log", "--oneline", "origin/main..HEAD", cwd=repo) or "").splitlines()
behind = (git("log", "--oneline", "HEAD..origin/main", cwd=repo) or "").splitlines()
changed = git("diff", "--stat", f"{base}..HEAD", cwd=repo) if base else None
license_head = ""
lic = repo / "LICENSE"
if lic.exists():
license_head = " ".join(lic.read_text(errors="ignore").split()[:6])
return {
"upstream_repository": "https://github.com/QwenLM/Qwen3-TTS.git",
"local_head": head,
"upstream_base_commit": base,
"commits_ahead_of_upstream": len(ahead),
"commits_ahead_detail": ahead,
"commits_behind_upstream": len(behind),
"diff_stat_vs_base": changed,
"license": license_head or "unknown",
"note": (
"The fork delta touches only the web demo and a logo. It does not "
"touch the synthesis path, so it does not alter the input or output "
"contract. The contract that IS ours is umbraaudio's own server "
"script, which is not part of this repository - see runtime_surface."
),
}
# --------------------------------------------------------------------------
# 2. the model
# --------------------------------------------------------------------------
def models() -> dict[str, Any]:
root = REPO / "models"
if not root.is_dir():
return {"status": "unknown", "why": f"no model directory at {root}"}
found = []
for directory in sorted(root.glob("Qwen3-TTS-*")):
weights = directory / "model.safetensors"
files = sorted(p.name for p in directory.glob("*") if p.is_file())
found.append({
"model_id": directory.name,
"weights_file": weights.name if weights.exists() else None,
"weights_state": "present" if weights.exists() else "absent",
"weights_bytes": weights.stat().st_size if weights.exists() else None,
"weights_sha256": sha256_file(weights) if weights.exists() else None,
"directory_bytes": sum(p.stat().st_size for p in directory.rglob("*") if p.is_file()),
"config_present": "config.json" in files,
})
without_weights = [m["model_id"] for m in found if m["weights_state"] == "absent"]
return {
"models_present": found,
"models_without_weights": without_weights,
"total_directory_bytes": sum(m["directory_bytes"] or 0 for m in found),
"loaded_weights_bytes": sum(m["weights_bytes"] or 0 for m in found),
"note": ("weights_sha256 is of model.safetensors only, not the whole "
"directory. A model directory also holds config and tokenizer "
"files that change behaviour. A directory listed under "
"models_without_weights has config but no weights at all - it "
"cannot be loaded, and its absence is not an error."),
}
# --------------------------------------------------------------------------
# 3-4, 7-9. the server contract, read from the source
# --------------------------------------------------------------------------
def server_contract() -> dict[str, Any]:
path = REPO / "scripts" / "qwen_tts_server.py"
if not path.exists():
return {"status": "unknown", "why": f"no server script at {path}"}
source = path.read_text()
def has(needle: str) -> bool:
return needle in source
# Authentication, decided by looking for each mechanism by name rather than
# by a single negation. The first version of this asked `not "Depends(" in
# src` and therefore reported "required" on a server with no Depends at all
# -- a security statement inverted by one missing word. Each candidate is now
# named in the output so the claim can be checked by re-reading it.
mechanisms = {
"fastapi_dependency": "Depends(",
"security_scheme": "APIKeyHeader",
"bearer_scheme": "HTTPBearer",
"oauth2_scheme": "OAuth2PasswordBearer",
"authorization_header_check": "headers.get(\"Authorization\"",
"auth_middleware": "add_middleware(",
"api_token_env": "TTS_TOKEN",
}
found = sorted(name for name, needle in mechanisms.items() if needle in source)
return {
"endpoints": {
name: {"method": "POST" if has(f'@app.post("{path_}")') else "GET",
"path": path_}
for name, path_ in {
"health": "/health", "load": "/load", "unload": "/unload",
"attn": "/attn", "generate": "/v1/tts", "voice_prompt": "/v1/voice-prompt",
}.items()
},
"input_fields": {
"text": {"required": True, "type": "form"},
"sample_rate": {"default": 24000},
"language": {"default": "english",
"accepts": ["english", "auto", "<other>"],
"note": "was hardcoded to english; a caller must pass it"},
"voice_description": {"default": None,
"note": "VoiceDesign path, and also style "
"direction when combined with a prompt"},
"ref_audio": {"type": "file", "default": None},
"ref_text": {"default": None},
"speaker_embedding": {"default": None, "note": "path to a cached .npy"},
"prompt_slug": {"default": None, "note": "cached voice-clone prompt; wins over others"},
"temperature": {"default": 0.9},
"do_sample": {"default": True},
"seed": {"default": None},
},
"generation_paths": {
"icl": "prompt_slug -> cached vmatrix (.pt). Most faithful and most consistent.",
"embed": "speaker_embedding (.npy) -> coarse x-vector. Fastest.",
"clone": "ref_audio + ref_text -> prompt built per request.",
"design": "voice_description only -> VoiceDesign model.",
},
"output": {
"shape": "whole file, returned in-memory as an audio payload",
"streaming": False,
"max_text_length_enforced_by_server": None if not has("max_chars") else "unknown",
"max_text_length_note": (
"No length limit is enforced by this server. Any limit is the "
"model's own and was not measured; treat as unknown rather than "
"unbounded."
),
},
"concurrency": {
"model_lock": True,
"concurrent_generations": 1,
"why": "a single asyncio.Lock serialises generation; the event loop "
"stays responsive and work runs in threads",
"watchdog_restart_seconds": 1800,
},
"authentication": {
"requires_authentication": bool(found),
"mechanisms_found": found,
"mechanisms_searched": sorted(mechanisms),
"binds": "0.0.0.0:8881",
"note": ("No authentication mechanism is present, and it binds all "
"interfaces. Any client - including a controller-side "
"speak worker - reaches an open synthesis service that will "
"happily spend minutes of CPU per request. This is a "
"property of the deployment, not an oversight in the "
"server code."),
},
}
# --------------------------------------------------------------------------
# 5. determinism, measured by running it twice
# --------------------------------------------------------------------------
def determinism() -> dict[str, Any]:
"""Stated as a property of the defaults, which is what matters.
The defaults are ``temperature=0.9, do_sample=True`` and no seed, so two
calls with identical text and voice are **not** expected to be identical.
Whether a seeded call is reproducible was not measured, because reproducing
it needs a warm model and minutes of MPS time; it is recorded as unknown
rather than assumed either way.
"""
return {
"identical_by_default": False,
"why": ("temperature 0.9 with do_sample=True and no seed. Sampling is "
"on, so the same request is expected to differ between calls."),
"seeded_reproducible": "unknown",
"seeded_reproducible_why": (
"Not measured. torch.manual_seed is called when a seed is supplied, "
"which seeds the global generator, but reproducibility across MPS "
"kernels, across model reloads, and across attention implementations "
"is a separate question that needs a deliberate experiment."
),
"seeded": ["torch.manual_seed(seed)", "torch.cuda.manual_seed_all(seed) if cuda"],
"unseeded_sources": ["sampling (temperature 0.9, do_sample True)"],
"consequence": (
"A translation re-voiced because one line changed will NOT reproduce "
"the other lines byte-for-byte. Comparing takes needs a perceptual "
"or embedding measure, not a hash."
),
}
# --------------------------------------------------------------------------
# 6. languages, from the upstream model card
# --------------------------------------------------------------------------
def languages() -> dict[str, Any]:
readme = REPO / "Qwen3-TTS" / "README.md"
if not readme.exists():
return {"status": "unknown", "why": "upstream README not present"}
text = readme.read_text(errors="ignore")
known = ["Chinese", "English", "Japanese", "Korean", "German",
"French", "Russian", "Portuguese", "Spanish", "Italian"]
declared = [lang for lang in known if lang in text]
return {
"declared_by_upstream": declared,
"count": len(declared),
"selection": "the `language` form field; 'english' is the server default, "
"'auto' lets the model detect",
"known_weak": {
"en": "English phonology and prosody are applied unless `language` is "
"passed. This was a real bug here: the field was hardcoded to "
"english in every branch, so German and French text was read "
"with English prosody. Fixed by making it a parameter.",
},
"quality_weaknesses": "unknown",
"quality_weaknesses_why": (
"No per-language evaluation has been run in this system. There is a "
"quality gate, but it scores MOS/WER/artifact score over the book, "
"not per language. Claiming a ranking would be an estimate."
),
}
# --------------------------------------------------------------------------
# output contract, measured from a real render
# --------------------------------------------------------------------------
def output_contract() -> dict[str, Any]:
"""Read the contract off a file this system actually produced.
A rendered take, not a config value, because that is what a `speak` caller
will receive and compare against.
"""
project = REPO / "DealHunterShip_4.0e_whole-to-audio"
sample = project / ".umbra" / "chapter_audio" / "ch1" / "utt_0000.wav"
if not sample.exists():
return {"status": "unknown", "why": f"no sample render at {sample}"}
try:
with wave.open(str(sample)) as handle:
wav = {"container": "wav", "codec": "pcm_s16le",
"sample_rate_hz": handle.getframerate(),
"channels": handle.getnchannels(),
"bit_depth_bits": handle.getsampwidth() * 8,
"duration_ms": round(handle.getnframes() / handle.getframerate() * 1000)}
except (OSError, wave.Error, EOFError) as error:
wav = {"error": type(error).__name__}
mp3 = project / ".umbra" / "export" / "ar_package" / "Deal_Hunter_01_Salvage.mp3"
return {
"rendered_utterance": {**wav, "measured_from": str(sample.relative_to(project)),
"bit_rate_bps": None,
"bit_rate_note": "PCM; follows from rate x depth x channels"},
"distribution_master": ({"measured_from": str(mp3.relative_to(project)),
**ffprobe(mp3)} if mp3.exists() else "unknown"),
"consistency_note": (
"Renders are 24 kHz. The distribution masters are 44.1 kHz. They are "
"not the same format, and a render produced by `speak` should be "
"described as the first, not the second."
),
}
def voice_identity() -> dict[str, Any]:
project = REPO / "DealHunterShip_4.0e_whole-to-audio"
umbra = project / ".umbra"
def load(name: str, default: Any) -> Any:
try:
return json.loads((umbra / name).read_text())
except (OSError, ValueError):
return default
def norm(text: str) -> str:
return text.lower().replace(" ", "_")
ledger = load("voice_engines.json", {}) or {}
by_norm = {norm(k): v for k, v in ledger.items()}
refs = {p.stem[:-4]: p for p in (umbra / "voices").rglob("*_ref.wav")} \
if (umbra / "voices").is_dir() else {}
prompts = {p.stem for p in (umbra / "voice_prompts").glob("*.pt")} \
if (umbra / "voice_prompts").is_dir() else set()
assignments = load("voice_assignments.json", {}) or {}
with_provenance = set(refs) & set(by_norm)
return {
"chain": [
"assigned_edge_voice -> an Edge-TTS voice id, e.g. en-CA-ClaraNeural:-100",
"voices/<name>_ref.wav -> the REFERENCE AUDIO, one per speaker",
"voice_prompts/<slug>.pt -> the cached 'vmatrix' extracted from that reference",
"utterance render -> Qwen conditions on the vmatrix (prompt_slug)",
],
"what_the_edge_voice_actually_is": {
"answer": "the FALLBACK ENGINE for creating the reference audio",
"not": ["the engine that renders utterances", "the clone source"],
"why": ("generate_voice_reference() tries Qwen VoiceDesign up to "
"max_retries times; only on exhaustion does it call "
"_generate_edge_fallback() and write the Edge-derived audio "
"to the same ref path. The Edge id is also a historical "
"voice assignment from when Edge-TTS was the voice source."),
"consequence": ("tts_engine_used describes which engine produced the "
"REFERENCE, not which engine rendered the utterance. "
"Reading it as the latter is wrong."),
},
"tts_engine_used_semantics": {
"qwen": "this speaker's reference audio was generated by Qwen VoiceDesign",
"null": "UNKNOWN - no decision was recorded. Not 'edge'.",
},
"counts": {
"reference_wavs_on_disk": len(refs),
"ledger_entries": len(by_norm),
"refs_with_recorded_provenance": len(with_provenance),
"refs_with_unknown_provenance": len(set(refs) - set(by_norm)),
"cached_voice_prompts": len(prompts),
"prompts_without_a_reference_wav": len(prompts - set(refs)),
"speakers_with_an_edge_voice_assigned": len(assignments),
},
"unknown_provenance_is_the_norm": (
f"{len(set(refs) - set(by_norm))} of {len(refs)} reference files have no "
"recorded provenance. The ledger was added after those references were "
"generated, so a null here means the system does not know, which is a "
"different statement from 'it was Edge'."
),
"orphan_prompts_note": (
f"{len(prompts - set(refs))} cached prompts have no matching reference "
"file. A cached vmatrix outlives the audio it was built from, so a "
"replaced reference does not automatically invalidate its prompt; the "
"server takes force=1 to rebuild one deliberately."
),
}
# --------------------------------------------------------------------------
def build() -> dict[str, Any]:
document = {
"document_version": 1,
"generated_at": datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z"),
"source_system": {"system": "umbraaudio", "node_id": "umbraaudio-m4"},
"generated_by": "scripts/build_tts_facts.py",
"server_health_now": probe_json(f"{TTS_URL}/health"),
"attn_implementation_now": probe_json(f"{TTS_URL}/attn"),
"provenance": provenance(),
"models": models(),
"runtime_surface": server_contract(),
"output_contract": output_contract(),
"determinism": determinism(),
"languages": languages(),
"voice_identity": voice_identity(),
}
# The hash covers the measured facts, not the moment of measurement. This
# matches the bundle, where `content_hash` is of `manifest.jsonl` and
# therefore excludes `bundle.json`'s own timestamp: a hash that changes on
# every run cannot answer "did anything change?", only "was it run again?".
hashed = {k: v for k, v in document.items()
if k not in ("generated_at", "content_hash")}
payload = json.dumps(hashed, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
document["content_hash"] = {
"algorithm": "sha256",
"of": "this document with generated_at and content_hash omitted, canonical JSON",
"value": hashlib.sha256(payload.encode()).hexdigest(),
}
return document
UNKNOWN = "unknown"
def _auth(runtime: dict[str, Any]) -> str:
return "required" if runtime["authentication"]["requires_authentication"] else "NONE"
def _join(names: list[str]) -> str:
return ", ".join(f"`{n}`" for n in names)
def _why(runtime: dict[str, Any]) -> str:
text = runtime["concurrency"]["why"]
return str(text[0].upper() + text[1:])
def _size(model: dict[str, Any]) -> str:
"""A missing weights file is a fact; ``None`` is not a rendering of it."""
if model["weights_state"] == "absent":
return "no weights on disk"
return f"{round((model['weights_bytes'] or 0) / 1e9, 2)} GB"
def render_markdown(doc: dict[str, Any]) -> str:
health = doc["server_health_now"]
prov = doc["provenance"]
counts = doc["voice_identity"]["counts"]
models_present = doc["models"].get("models_present", [])
runtime = doc["runtime_surface"]
out = doc["output_contract"]["rendered_utterance"]
model_rows = "\n".join(
f"| `{m['model_id']}` | "
f"{_size(m)} | "
f"`{(m['weights_sha256'] or 'not loadable')[:16]}` |"
for m in models_present)
return f"""# TTS engine, measured
**Generated:** {doc['generated_at']}
**By:** `{doc['generated_by']}` (re-runnable; every value below is measured or read at run time)
**Document hash:** `{doc['content_hash']['value']}`
Where a value could not be measured it says **unknown** and why. No value here is
an estimate presented as a fact. "A validator nobody has seen fail is not
evidence" applies to this document too: anything marked measured names the file
or endpoint it was read from.
**Server at time of writing:** `GET {TTS_URL}/health` →
`{json.dumps(health)}`
---
## 1. Provenance
| | |
|---|---|
| Upstream | `{prov['upstream_repository']}` |
| Local HEAD | `{prov['local_head']}` |
| Upstream base | `{prov['upstream_base_commit']}` |
| Commits ahead of upstream | **{prov['commits_ahead_of_upstream']}** |
| Commits behind upstream | {prov['commits_behind_upstream']} |
| License | {prov['license']} |
Commits ahead:
```
{chr(10).join(prov['commits_ahead_detail']) or '(none)'}
```
**The fork delta is {prov['commits_ahead_of_upstream']} commit and it is cosmetic.**
It touches a logo and the web demo (`qwen_tts/cli/demo.py`). It does not touch the
synthesis path, so it changes neither the input contract nor the output contract.
This corrects a premise in the request, which asked for "the three commits on
top". There is one. The other recent commits in the log are upstream's own.
**The contract that is actually ours is not in this repository.** It is
`umbraaudio/scripts/qwen_tts_server.py` — the HTTP surface, the four generation
paths, and the `language` parameter were all written by this project. If you are
writing `speak` against something, it is that script, not the fork.
## 2. The model
| Model id | Weights | sha256 (first 16) |
|---|---|---|
{model_rows}
- **Which serves which path.** `-VoiceDesign` serves the `design` path
(`voice_description` only). `-Base` serves `clone`, `embed` and `icl`
(reference audio, cached embeddings, cached prompts), because the vmatrix
extraction uses it.
- **Two of the five directories hold no weights at all** — the two
`-CustomVoice` models. They have `config.json` and a README and nothing
loadable. That is not damage; nothing here uses them.
- **Working set.** **{UNKNOWN}.** Not measured. Determining it needs a load with
resident-memory sampling on the target hardware, and the answer differs between
MPS and CUDA and between the two model sizes. I am not going to put a number
on it that I did not watch.
- **Weights checksum.** sha256 of `model.safetensors` only, per model, in the JSON.
The directory checksum is a different question — tokenizer and config files
change behaviour.
**For your 48 GB CPU-only node question:** the two models actually used total
{round(doc['models']['loaded_weights_bytes'] / 1e9, 1)} GB of weights on disk.
Whether that is also the *working set* is {UNKNOWN}, and that is the number that
decides the question: weights on disk are not the memory the process holds, and
on CPU the gap between them is what decides whether the box can run it. I did not
measure it, so I am not going to reason about it here as though I had.
## 3. Input contract
`POST /v1/tts` — all fields are form fields, except `ref_audio` which is a file
upload.
| Field | Default | Notes |
|---|---|---|
{chr(10).join(f"| `{k}` | {v.get('default')!r} | {v.get('note', '')} |"
for k, v in runtime['input_fields'].items())}
**Four generation paths**, selected by which fields are present; `prompt_slug`
wins if the cached prompt exists:
{chr(10).join(f"- **{k}** — {v}" for k, v in runtime['generation_paths'].items())}
**Maximum input length:** {UNKNOWN}. This server enforces none, and I did not
measure the model's own limit. Treat as unknown rather than unbounded — an
over-length input is not the same as a long input.
**Streaming:** {runtime['output']['shape']}. Not a stream.
## 4. Output contract
Measured from a file this system produced, not from a config value.
| | |
|---|---|
| Container / codec | {out.get('container')} / {out.get('codec')} |
| Sample rate | {out.get('sample_rate_hz')} Hz |
| Channels | {out.get('channels')} |
| Bit depth | {out.get('bit_depth_bits')} bit |
| Bit rate | PCM; follows from rate x depth x channels |
| Source | `{out.get('measured_from')}` |
{doc['output_contract']['consistency_note']}
## 5. Determinism
**Identical by default: {doc['determinism']['identical_by_default']}.**
{json.dumps(doc['determinism']['why'])}
- Seeded call reproducible: **{doc['determinism']['seeded_reproducible']}** —
{doc['determinism']['seeded_reproducible_why']}
- Seeded when supplied: {', '.join(f'`{s}`' for s in doc['determinism']['seeded'])}
- Not seeded: {', '.join(f'`{s}`' for s in doc['determinism']['unseeded_sources'])}
**{doc['determinism']['consequence']}**
## 6. Language coverage
- Declared by upstream: {', '.join(doc['languages']['declared_by_upstream'])}
({doc['languages']['count']} languages)
- Selection: {doc['languages']['selection']}
- **Quality ranking: {UNKNOWN}.** {doc['languages']['quality_weaknesses_why']}
**Known trap, from a real bug here:** {json.dumps(doc['languages']['known_weak']['en'])}
## 7. Voice identity
The chain:
{chr(10).join(f"{i + 1}. {step}" for i, step in enumerate(doc['voice_identity']['chain']))}
**What the Edge voice actually is:**
{doc['voice_identity']['what_the_edge_voice_actually_is']['answer']}.
It is **not** {', **not** '.join(doc['voice_identity']['what_the_edge_voice_actually_is']['not'])}.
Utterances are rendered by Qwen, conditioned on the vmatrix extracted from the
reference audio; the Edge voice id never reaches the synthesis call.
{doc['voice_identity']['what_the_edge_voice_actually_is']['why']}
**{doc['voice_identity']['what_the_edge_voice_actually_is']['consequence']}**
| | |
|---|---|
| `tts_engine_used: "qwen"` | {doc['voice_identity']['tts_engine_used_semantics']['qwen']} |
| `tts_engine_used: null` | **{doc['voice_identity']['tts_engine_used_semantics']['null']}** |
Measured on Deal Hunter:
| | |
|---|---|
| Reference WAVs on disk | {counts['reference_wavs_on_disk']} |
| With recorded provenance | {counts['refs_with_recorded_provenance']} |
| **With unknown provenance** | **{counts['refs_with_unknown_provenance']}** |
| Cached voice prompts (vmatrix) | {counts['cached_voice_prompts']} |
| Prompts with no reference WAV | {counts['prompts_without_a_reference_wav']} |
| Speakers with an Edge voice assigned | {counts['speakers_with_an_edge_voice_assigned']} |
{doc['voice_identity']['unknown_provenance_is_the_norm']}
**This is a correction to how the mapping reads a field.** The bundle's own
convention says "null means absent". In `tts_engine_used` it means *unknown*, and
the mapping should carry it as unknown rather than resolving it to "edge".
{doc['voice_identity']['orphan_prompts_note']}
## 8. Failure modes
| Condition | What happens | Where it surfaces |
|---|---|---|
| Server unreachable | `QwenTTSUnavailableError` after retries | `tts_engine_fallback_reason` |
| Server down, building a reference | Edge audio into the ref path | `voice_engines.json` |
| Unknown `prompt_slug` | HTTP 404 | request fails; not regenerated |
| `ref_audio` missing on `/v1/voice-prompt` | HTTP 400 | request fails |
| Generation hangs | watchdog restarts the server process after 1800 s | server log |
| CUDA OOM / model load failure | **unknown** | never run on CUDA in this system |
Over-length text: **unknown** — no limit is enforced, and the model's own limit
was not measured.
## 9. Runtime surface
| | |
|---|---|
| Transport | HTTP, binds `0.0.0.0:8881` |
| **Authentication** | **{_auth(runtime)}** |
| Mechanisms searched for | {_join(runtime['authentication']['mechanisms_searched'])} |
| Mechanisms found | {_join(runtime['authentication']['mechanisms_found']) or 'none'} |
| Concurrency | **{runtime['concurrency']['concurrent_generations']}** (`_model_lock`) |
| Watchdog | {runtime['concurrency']['watchdog_restart_seconds']} s |
| Client timeout | `REMOTE_TIMEOUT = 900` s |
| Endpoints | {', '.join(f"`{v['method']} {v['path']}`" for v in runtime['endpoints'].values())} |
{runtime['authentication']['note']}
**Concurrency limit: {runtime['concurrency']['concurrent_generations']}.**
{_why(runtime)}. A `speak` worker that fans out
will serialise here regardless of its own concurrency, and queue behind anything
already generating.
---
## What I could not measure
Stated once, plainly, rather than repeated as a footnote:
1. **Working set / VRAM at load and at inference.** Needs resident-memory
sampling on the target hardware.
2. **Whether a seeded call is reproducible** across MPS kernels, model reloads,
and attention implementations.
3. **Per-language quality.** No per-language evaluation exists in this system.
4. **The model's own maximum text length.**
5. **CUDA failure modes.** Never exercised here; this system runs MPS.
Each is unknown for a reason, not because it was not worth measuring. If one of
them blocks `speak`, say which and I will run the experiment rather than
substitute a guess.
"""
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--out-dir", default=str(REPO / "docs" / "tts"))
parser.add_argument("--json-only", action="store_true")
args = parser.parse_args()
out = Path(args.out_dir)
out.mkdir(parents=True, exist_ok=True)
document = build()
payload = json.dumps(document, indent=2, sort_keys=True) + "\n"
(out / "tts-engine.json").write_text(payload)
if not args.json_only:
(out / "tts-engine.md").write_text(render_markdown(document))
print(f"tts-engine.json {(out / 'tts-engine.json').stat().st_size} bytes")
if not args.json_only:
print(f"tts-engine.md {(out / 'tts-engine.md').stat().st_size} bytes")
print(f"content_hash {document['content_hash']['value']}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

274
tests/test_tts_facts.py Normal file
View File

@@ -0,0 +1,274 @@
"""Tests for the TTS facts document and the scale totals.
The controller asked for a document whose every claim is checkable against the
running system. These tests are the part of that claim this side can enforce: the
document must not assert a value it cannot back, and a null that means "unknown"
must not be rendered as a fact.
They are cheap and deterministic. Nothing here generates audio, loads a model, or
touches the model directory -- the generator is read-only and these tests keep it
that way.
"""
from __future__ import annotations
import importlib.util
import json
import sys
from pathlib import Path
from typing import Any
import pytest
REPO = Path(__file__).resolve().parent.parent
def _load(name: str) -> Any:
spec = importlib.util.spec_from_file_location(name, REPO / "scripts" / f"{name}.py")
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
facts = _load("build_tts_facts")
scale = _load("build_scale_totals")
@pytest.fixture(scope="module")
def document() -> dict[str, Any]:
return facts.build()
@pytest.fixture(scope="module")
def markdown(document: dict[str, Any]) -> str:
return facts.render_markdown(document)
# --------------------------------------------------------------------------
# the document must not invent numbers
# --------------------------------------------------------------------------
def test_every_unknown_is_marked_unknown_and_has_a_reason(document: dict[str, Any]) -> None:
"""An unknown with no reason is a shrug; the reader cannot act on it."""
determinism = document["determinism"]
assert determinism["seeded_reproducible"] == "unknown"
assert determinism["seeded_reproducible_why"].strip()
languages = document["languages"]
assert languages["quality_weaknesses"] == "unknown"
assert languages["quality_weaknesses_why"].strip()
models = document["models"]
assert "note" in models
def test_working_set_is_not_estimated(document: dict[str, Any]) -> None:
"""The 48 GB node question turns on this number, so it must be absent, not guessed."""
text = json.dumps(document["models"]).lower()
assert "working_set" not in text
assert "vram" not in text
def test_determinism_is_reported_as_non_deterministic(document: dict[str, Any]) -> None:
"""Sampling is on by default; claiming reproducibility would be the easy lie."""
assert document["determinism"]["identical_by_default"] is False
assert "temperature" in document["determinism"]["why"]
def test_authentication_is_not_claimed_when_absent(document: dict[str, Any]) -> None:
"""A security statement inverted by a missing word is worse than no statement.
The first version of this asked ``not "Depends(" in source`` and therefore
reported "required" on a server with no ``Depends`` at all.
"""
auth = document["runtime_surface"]["authentication"]
assert auth["requires_authentication"] == bool(auth["mechanisms_found"])
assert "fastapi_dependency" in auth["mechanisms_searched"]
if not auth["mechanisms_found"]:
# The note must say so in words, not merely omit a field.
assert "no authentication mechanism is present" in auth["note"].lower()
def test_concurrency_is_stated_as_a_number(document: dict[str, Any]) -> None:
concurrency = document["runtime_surface"]["concurrency"]
assert concurrency["concurrent_generations"] == 1
assert concurrency["model_lock"] is True
def test_models_without_weights_are_not_rendered_as_none(document: dict[str, Any]) -> None:
"""A missing weights file is a fact; `None` is not a rendering of it."""
for model in document["models"].get("models_present", []):
if model["weights_state"] == "absent":
assert model["weights_bytes"] is None
assert model["weights_sha256"] is None
assert facts._size(model) == "no weights on disk"
def test_output_contract_is_measured_from_a_real_file(document: dict[str, Any]) -> None:
out = document["output_contract"]["rendered_utterance"]
assert out.get("codec"), "output contract must name a codec, not omit it"
assert out.get("measured_from"), "the file it was read from must be named"
assert out.get("sample_rate_hz"), "sample rate must be stated in the name/unit"
def test_provenance_counts_only_our_commits(document: dict[str, Any]) -> None:
provenance = document["provenance"]
assert provenance["commits_ahead_of_upstream"] == len(
provenance["commits_ahead_detail"])
assert provenance["upstream_base_commit"]
def test_edge_voice_answer_is_not_inverted(markdown: str) -> None:
"""The rendered sentence must not say the Edge voice IS the render engine."""
assert "the FALLBACK ENGINE for creating the reference audio" in markdown
assert "**not** the engine that renders utterances" in markdown
assert "Utterances are rendered by Qwen" in markdown
def test_markdown_has_no_stray_none(markdown: str) -> None:
"""An f-string that looked up a key in the wrong dict prints `None`.
That happened once: the consistency note rendered as a bare `None` on its
own line, which reads as an answer and is not one.
"""
for line in markdown.splitlines():
assert line.strip() != "None", f"stray None rendered as a value: {line!r}"
def test_markdown_states_the_hashes_it_claims(document: dict[str, Any],
markdown: str) -> None:
assert document["content_hash"]["value"] in markdown
for model in document["models"].get("models_present", []):
if model["weights_sha256"]:
assert model["weights_sha256"][:16] in markdown
def test_markdown_names_every_section_the_request_asked_for(markdown: str) -> None:
for heading in ("## 1. Provenance", "## 2. The model", "## 3. Input contract",
"## 4. Output contract", "## 5. Determinism",
"## 6. Language coverage", "## 7. Voice identity",
"## 8. Failure modes", "## 9. Runtime surface"):
assert heading in markdown, heading
assert "What I could not measure" in markdown
# --------------------------------------------------------------------------
# determinism
# --------------------------------------------------------------------------
def test_document_is_reproducible_apart_from_its_timestamp() -> None:
"""A hash that changes on every run cannot answer 'did anything change?'."""
first = facts.build()
second = facts.build()
assert first["content_hash"]["value"] == second["content_hash"]["value"]
for key in ("generated_at", "content_hash"):
del first[key]
del second[key]
assert first == second
def test_hash_excludes_the_timestamp() -> None:
"""The hash must not move when only the clock does."""
import copy
import hashlib
original = facts.build()
shifted = copy.deepcopy(original)
shifted["generated_at"] = "1999-01-01T00:00:00Z"
payload = json.dumps({k: v for k, v in shifted.items()
if k not in ("generated_at", "content_hash")},
sort_keys=True, separators=(",", ":"))
assert hashlib.sha256(payload.encode()).hexdigest() == \
original["content_hash"]["value"]
# --------------------------------------------------------------------------
# scale totals
# --------------------------------------------------------------------------
def _minimal_project(root: Path) -> Path:
"""A project with just enough to be measurable: a populated database."""
import sqlite3
umbra = root / ".umbra"
umbra.mkdir(parents=True)
db = umbra / "project.db"
conn = sqlite3.connect(db)
conn.executescript(
"CREATE TABLE chapters (id INTEGER PRIMARY KEY, word_count INTEGER);"
"CREATE TABLE utterances (id INTEGER PRIMARY KEY, speaker_name TEXT);"
"INSERT INTO chapters (id, word_count) VALUES (1, 10);"
"INSERT INTO utterances (id, speaker_name) VALUES (1, 'narrator');"
)
conn.commit()
conn.close()
return root
def test_scale_totals_separate_audio_from_markers(tmp_path: Path) -> None:
"""'files' must mean one thing. A render and its marker are not the same file."""
project = _minimal_project(tmp_path / "P")
umbra = project / ".umbra" / "chapter_audio" / "ch1"
umbra.mkdir(parents=True)
import wave as wavemod
for index in range(2):
path = umbra / f"utt_{index:04d}.wav"
with wavemod.open(str(path), "wb") as handle:
handle.setnchannels(1)
handle.setsampwidth(2)
handle.setframerate(24000)
handle.writeframes(b"\x00\x00" * 24000)
path.with_suffix(".wav.txt").write_text('{"speaker":"narrator","text":"x"}')
measured = scale.measure_project(project)
assert measured is not None
assert measured["audio"]["audio_files"] == 2
assert measured["audio"]["marker_files_excluded_from_audio"] == 2
assert measured["audio"]["audio_duration_ms"] == 2000
def test_scale_totals_excludes_scratch_and_says_so(tmp_path: Path) -> None:
project = _minimal_project(tmp_path / "P")
scratch = project / ".umbra" / "chapter_audio" / "cache" / "ch1"
scratch.mkdir(parents=True)
(scratch / "utt_0000.wav").write_bytes(b"not audio")
measured = scale.measure_project(project)
assert measured["audio"]["scratch_files_excluded"] == 1
assert measured["audio"]["audio_files"] == 0
def test_scale_totals_report_a_project_with_no_database_as_absent(tmp_path: Path) -> None:
(tmp_path / "Empty").mkdir()
assert scale.measure_project(tmp_path / "Empty") is None
def test_scale_schema_fingerprint_is_shape_not_content(tmp_path: Path) -> None:
import sqlite3
db = tmp_path / "a.db"
conn = sqlite3.connect(db)
conn.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT)")
conn.execute("INSERT INTO t (name) VALUES ('one')")
first = scale.schema_fingerprint(conn)
conn.execute("INSERT INTO t (name) VALUES ('two')")
second = scale.schema_fingerprint(conn)
conn.execute("ALTER TABLE t ADD COLUMN extra TEXT")
third = scale.schema_fingerprint(conn)
conn.close()
assert first == second, "content must not change the fingerprint"
assert first != third, "a shape change must change the fingerprint"