UmbraAudio/tts_server/README.md

UmbraAudio Qwen3-TTS Server package

Self-contained Qwen3-TTS server (our /v1/tts + /v1/voice-prompt API) for running the audiobook generation on a CUDA/GPU machine instead of the 16 GB M4 Mac (where MPS inference is prohibitively slow).

All the pipeline code (QwenTTSEngine, ChapterBuilder) talks to this server unchanged — it exposes the exact same endpoints as the local deployment.

Contents

• qwen_tts_server.py — the FastAPI server (port 8881, /health, /load, /unload, /v1/tts, /v1/voice-prompt)
• install_server.sh — one-command installer (venv + CUDA torch + deps + models)
• run_server.sh — launcher (sets CUDA device, model & prompt dirs)
• requirements.txt — pip dependencies
• sync_voice_data.sh — bundles voice refs + cached prompts to ship to the GPU box
• models/ — (created by install/download)

Install on a Linux/CUDA GPU machine

# 1. Copy this whole directory to the GPU machine
scp -r tts_server user@gpu:/path/to/

# 2. Install (venv, CUDA torch, qwen-tts, flash-attn)
cd tts_server
./install_server.sh

# 3. Download the 1.7B models (or point QWEN_MODELS_DIR at existing ones)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base --local-dir models/Qwen3-TTS-12Hz-1.7B-Base
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --local-dir models/Qwen3-TTS-12Hz-1.7B-VoiceDesign

# 4. Ship the voice references + cached ICL prompts (run ON the Mac):
#    ./sync_voice_data.sh  or  ./sync_voice_data.sh user@gpu:/path/to/tts_server/
#    (this creates voice_prompts/ on the GPU box with the speaker .pt files)

# 5. Run the server
./run_server.sh

Verify

curl http://<gpu-ip>:8881/health          # {"status":"ok", ...}
curl -X POST http://<gpu-ip>:8881/v1/voice-prompt ...   # build a prompt
curl -X POST http://<gpu-ip>:8881/v1/tts ...            # generate (prompt_slug)

Point the pipeline at it

The engine already reads the TTS URL from config (qwen_tts_url, default http://localhost:8881). Set it to the GPU box, e.g.:

QWEN_DEVICE=cuda ./run_server.sh            # on GPU host, binds 0.0.0.0:8881
# on the Mac, point config qwen_tts_url at http://<gpu-ip>:8881

Env vars

var	default	purpose
QWEN_DEVICE	cuda	cuda/mps/cpu
QWEN_BASE_MODEL	models/...-1.7B-Base	voice-clone / ICL model
QWEN_VOICEDESIGN_MODEL	models/...-1.7B-VoiceDesign	voice-design model
QWEN_PROMPT_DIR	./voice_prompts	cached ICL .pt prompts
QWEN_ATTN	sdpa	sdpa (fast, built-in) / eager (slow) / flash_attention_2
QWEN_MAX_GEN_S	1800	watchdog: if a generation hangs this long, the server auto-restarts itself
QWEN_PORT	8881	listen port
QWEN_HOST	0.0.0.0	bind address
QWEN_PROMPT_DIR	./voice_prompts	cached prompts

Flash-attn is recommended but not required. With no flash-attn the server sets attn_implementation="sdpa" (the default) for generation, which is fast and light on VRAM and requires no compilation. eager is pathologically slow for Qwen3-TTS generation (~20x) — but note that on CUDA the create_voice_clone_prompt extraction pass is NOT reliable under sdpa (it can hang the model). So use the two-phase pattern below rather than leaving QWEN_ATTN as a fixed env value.

Runtime attention switching (no restart)

attn_implementation is fixed at model-load time, but you can switch it live without restarting the process via the /attn endpoint — this unloads the model(s) and reloads base with the new implementation:

curl http://HOST:8881/attn                          # -> {"impl":"sdpa"}
curl -X POST http://HOST:8881/attn -H 'Content-Type: application/json' \
     -d '{"impl":"eager"}'                           # reload model with eager

Recommended two-phase build (avoids the sdpa extraction hang):

1. Phase 1 — extract prompts under eager (fast + stable): switch /attn to eager, then call POST /v1/voice-prompt once per speaker to build each <slug>.pt. /v1/voice-prompt is idempotent — it returns the cached prompt without re-extracting if <slug>.pt already exists.
2. Phase 2 — generate under sdpa: switch /attn back to sdpa, then run the chapter batch. build_voice_prompt calls now hit the idempotency short circuit (no extraction, no wedge) and /v1/tts?prompt_slug=... generates from the cached prompt quickly and reliably.

Crash / hang recovery

A CUDA kernel error (e.g. OOM) can poison the GPU context: /health keeps returning OK but every generation request hangs forever. The server ships with a hang watchdog: if a generation (or prompt extraction) runs past QWEN_MAX_GEN_S (default 1800s = 30min), it relaunches itself via os.execv, clearing the poisoned context. No SSH needed — it self-heals. If the watchdog fires mid-batch, the client's own retry logic (health polling) reconnects once the process is back up.