UmbraAudio/tts_server

UmbraAudio Qwen3-TTS Server package

Self-contained Qwen3-TTS server (our /v1/tts + /v1/voice-prompt API) for running the audiobook generation on a CUDA/GPU machine instead of the 16 GB M4 Mac (where MPS inference is prohibitively slow).

All the pipeline code (QwenTTSEngine, ChapterBuilder) talks to this server unchanged — it exposes the exact same endpoints as the local deployment.

The vmatrix (voice identity)

A character's voice = a reference WAV (what you listen to) + a vmatrix (the set of numbers inside the TTS that is the voice, used for consistency). The vmatrix is captured by POST /v1/voice-prompt and cached as <slug>.pt in QWEN_PROMPT_DIR:

• ref_code — the ICL "8k matrix": a (T, 16) tensor of codec tokens produced by re-speaking the reference WAV through the model's codec encoder
• ref_spk_embedding — the 2048-dim extracted speaker x-vector

x_vector_only=0 (the default) captures the FULL vmatrix (ref_code + embedding, icl_mode=True) — faithful and perfectly consistent. x_vector_only=1 captures only the sparse embedding (ref_code=None) and is legacy/experimental; the pipeline client (QwenTTSEngine.build_voice_prompt) defaults to the full vmatrix and passes it on every utterance as prompt_slug, so a character's voice can never drift between chapters.

Contents

• qwen_tts_server.py — the FastAPI server (port 8881, /health, /load, /unload, /v1/tts, /v1/voice-prompt)
• install_server.sh — one-command installer (venv + CUDA torch + deps + models)
• run_server.sh — launcher (sets CUDA device, model & prompt dirs)
• requirements.txt — pip dependencies
• sync_voice_data.sh — bundles voice refs + cached prompts to ship to the GPU box
• models/ — (created by install/download)

Install on a Linux/CUDA GPU machine

# 1. Copy this whole directory to the GPU machine
scp -r tts_server user@gpu:/path/to/

# 2. Install (venv, CUDA torch, qwen-tts, flash-attn)
cd tts_server
./install_server.sh

# 3. Download the 1.7B models (or point QWEN_MODELS_DIR at existing ones)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir models/Qwen3-TTS-12Hz-1.7B-Base
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --local-dir models/Qwen3-TTS-12Hz-1.7B-VoiceDesign

# 4. Ship the voice references + cached ICL prompts (run ON the Mac):
#    ./sync_voice_data.sh  or  ./sync_voice_data.sh user@gpu:/path/to/tts_server/
#    (this creates voice_prompts/ on the GPU box with the speaker .pt files)

# 5. Run the server
./run_server.sh

Verify

curl http://<gpu-ip>:8881/health          # {"status":"ok", ...}
curl -X POST http://<gpu-ip>:8881/v1/voice-prompt ...   # build a prompt
curl -X POST http://<gpu-ip>:8881/v1/tts ...            # generate (prompt_slug)

POST /v1/voice-prompt — capture the vmatrix

form field	default	meaning
slug	required	cache key; stored as {QWEN_PROMPT_DIR}/{slug}.pt (same key the client passes as prompt_slug)
ref_audio	required (file)	the approved reference WAV for the speaker
ref_text	Rainbow Passage	reference transcript used to rebuild the ref_code "8k matrix"
x_vector_only	0	0 = FULL vmatrix (ref_code + ref_spk_embedding, icl_mode=True); 1 = embedding-only (ref_code=None)
force	0	1 = rebuild even if <slug>.pt already cached

Idempotent: if <slug>.pt exists and force is unset, it returns the cached prompt without re-extracting. The client's build_voice_prompt defaults to the full vmatrix (x_vector_only=False) and calls this once per distinct speaker per generation run.

Point the pipeline at it

The engine already reads the TTS URL from config (qwen_tts_url, default http://localhost:8881). Set it to the GPU box, e.g.:

QWEN_DEVICE=cuda ./run_server.sh            # on GPU host, binds 0.0.0.0:8881
# on the Mac, point config qwen_tts_url at http://<gpu-ip>:8881

Env vars

var	default	purpose
QWEN_DEVICE	cuda	cuda/mps/cpu
QWEN_BASE_MODEL	models/...-1.7B-Base	voice-clone / ICL model
QWEN_VOICEDESIGN_MODEL	models/...-1.7B-VoiceDesign	voice-design model
QWEN_PROMPT_DIR	./voice_prompts	cached vmatrix .pt prompts
QWEN_ATTN	sdpa	sdpa (fast, built-in) / eager (slow) / flash_attention_2
QWEN_MAX_GEN_S	1800	watchdog: if a generation hangs this long, the server auto-restarts itself
QWEN_PORT	8881	listen port
QWEN_HOST	0.0.0.0	bind address

Deployment note: in the live deployment (192.168.123.14:8881) QWEN_PROMPT_DIR points at the project's .umbra/voice_prompts on the shared volume, and sync_voice_data.sh ships both voices/*_ref.wav and voice_prompts/*.pt to the GPU box. That shared location is why Approve/build_voice_prompt immediately makes the vmatrix available to generation without extra steps.

Flash-attn is recommended but not required. With no flash-attn the server sets attn_implementation="sdpa" (the default) for generation, which is fast and light on VRAM and requires no compilation. eager is pathologically slow for Qwen3-TTS generation (~20x) — but note that on CUDA the create_voice_clone_prompt extraction pass is NOT reliable under sdpa (it can hang the model). So use the two-phase pattern below rather than leaving QWEN_ATTN as a fixed env value.

Runtime attention switching (no restart)

attn_implementation is fixed at model-load time, but you can switch it live without restarting the process via the /attn endpoint — this unloads the model(s) and reloads base with the new implementation:

curl http://HOST:8881/attn                          # -> {"impl":"sdpa"}
curl -X POST http://HOST:8881/attn -H 'Content-Type: application/json' \
     -d '{"impl":"eager"}'                           # reload model with eager

Recommended two-phase build (avoids the sdpa extraction hang):

1. Phase 1 — extract prompts under eager (fast + stable): switch /attn to eager, then call POST /v1/voice-prompt once per speaker to build each <slug>.pt. /v1/voice-prompt is idempotent — it returns the cached prompt without re-extracting if <slug>.pt already exists.
2. Phase 2 — generate under sdpa: switch /attn back to sdpa, then run the chapter batch. build_voice_prompt calls now hit the idempotency short circuit (no extraction, no wedge) and /v1/tts?prompt_slug=... generates from the cached prompt quickly and reliably.

Crash / hang recovery

A CUDA kernel error (e.g. OOM) can poison the GPU context: /health keeps returning OK but every generation request hangs forever. The server ships with a hang watchdog: if a generation (or prompt extraction) runs past QWEN_MAX_GEN_S (default 1800s = 30min), it relaunches itself via os.execv, clearing the poisoned context. No SSH needed — it self-heals. If the watchdog fires mid-batch, the client's own retry logic (health polling) reconnects once the process is back up.