UmbraAudio/specs/00-project-overview.md

UmbraAudio — Project Overview & Architecture

Purpose

UmbraAudio is a quality-first, multi-pass audiobook generation system that converts English manuscripts (EPUB, PDF, TXT, MD) into fully produced audiobooks using neural TTS. Each pass has quality gates. Quality is the primary concern; generation speed is secondary.

Two acoustic-quality invariants run through every pass and are non-negotiable for release:

1. Narrator consistency — the narrator's voice stays calm, even, and low-emphasis for the entire book, independent of how dramatic the underlying scene text is.
2. Cast distinctiveness — every character who shares dialogue proximity with another character must be audibly distinguishable, not just textually different.

A third principle governs delivery: punctuation and typography are prosody instructions, not decoration. A question mark, an ellipsis, an em-dash interruption, or a shouted ALL-CAPS word must change how a line is spoken, not just how it reads.

A book is a project. Its source, analysis, intermitent files, and outputs are kept seperate from other projects.

High-Level Pipeline

EPUB/PDF/TXT/MD
    │
    ▼
┌─────────────────────────────────────────────────┐
│  PASS 1a: PARSE                                 │
│  Parse manuscript into chapters and scense "<<<>>>"│
│  Preserves emphasis markup + punctuation verbatim│
│  Output: chapters table (source_text)           │
└─────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────┐
│  PASS 1b: CHARACTER DISCOVERY                   │
│  LLM chapter-by-chapter character extraction    │
│  Merge across chapters, deduplicate aliases     │
│  Narrator identified (not a character)          │
| identify characteristics/voice of each character
| Each voice must be unique for the listener.
│  Voice reference generation via Qwen3-TTS      │
│  Verification samples generated                 │
│  Output: speakers.json, voice_assignments.json  │
└─────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────┐
│  PASS 2: TWO-PHASE SEGMENTATION                 │
│  Phase 1: Mechanical segmenter splits text      │
│    - Sentence splitting at .!?                  │
│    - Dialogue detection via quotes              │
│    - Tag-based attribution (forward/backward)   │
│    - Scene break (<<<>>>) handling               │
│  Phase 2: LLM speaker attribution               │
│    - Attributes ALL dialogue (overrides guesses)│
│    - Resolves pronouns and context              │
│    - Uses character profiles from speakers.json │
| Uterances are narrator "Bob said," and then Bob speaking.
│  Output: utterances table                       │
└─────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────┐
│  PASS 3: AUDIO GENERATION                       │
│  Qwen3-TTS Base → voice cloning from references │
|  Each character has a unique voice the listener can decern
│  Voice assignment from voice_assignments.json   │
│  Retry logic for rate limiting                  │
│  Output: chapter_audio/ch{id}/*.wav             │
└─────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────┐
│  PASS 4: REVIEW                                 │
│  Web UI for human review of:                    │
│    - Character profiles and voice assignments   │
│    - Voice test samples                         │
│    - Chapter audio with per-utterance playback  │
│    - Speaker attribution accuracy               │
│  Output: approved chapters                      │
└─────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────┐
│  EXPORT                                         │
│  Stitch chapter audio → full audiobook          │
│  Output: export/*.m4b, export/*.mp3             │
└─────────────────────────────────────────────────┘

Runtime Data Layout

.umbra/                          # Runtime data directory
├── project.db                   # SQLite database
├── speakers.json                # Character profiles (61 speakers)
├── voice_assignments.json       # Voice assignments (60 mappings)
├── voice_tests/                 # Verification audio samples
│   ├── {speaker_name}.wav
│   └── ...
├── chapter_audio/               # Generated chapter audio
│   ├── ch5/
│   │   ├── utt_0000.wav
│   │   └── ...
│   ├── ch6/
│   └── ch7/
├── analysis/
│   └── input.txt                # Extracted manuscript text
└── defects.json                 # Quality defect log

Technology Stack

Layer	Technology	Version
Language	Python	3.13+
CLI	Click + Rich	8.1+ / 13.0+
Database	SQLite (aiosqlite)	0.19+
LLM	OpenAI-compatible (LM Studio)	1.10+
TTS	Qwen3-TTS 1.7B (local GPU)	0.1+
LLM Model	gemma-4-12b-it-qat (remote LM Studio, context_length=8192)	remote
Web	FastAPI + HTMX + Jinja2	0.104+
Parsing	ebooklib, beautifulsoup4, PyMuPDF	0.18+ / 4.0+ / 1.23+

GPU Requirements

• Qwen3-TTS 1.7B Base: ~3.9GB VRAM (voice cloning)
• Qwen3-TTS 1.7B VoiceDesign: ~3.9GB VRAM (voice reference generation)
• LM Studio (gemma-4-12b-it-qat): remote server; load with context_length: 8192 (speaker attribution)
• Note: Only one Qwen3-TTS model can be loaded at a time (auto-swaps). LM Studio must be unloaded before TTS generation if VRAM is tight.

Module Structure

umbraaudio/
├── cli.py              # Click CLI (all commands)
├── config.py           # Pydantic settings (AppConfig tree)
├── database.py         # SQLite schema + async CRUD
├── orchestrator.py     # Full pipeline coordinator
├── parsers/            # Manuscript parsing (EPUB, PDF, TXT, MD)
├── analysis/           # LLM character discovery, profiling
│   ├── character_discovery.py    # Chapter-by-chapter LLM extraction
│   ├── llm_attributor.py         # Two-phase speaker attribution
│   ├── llm_segmenter.py          # LLM-based segmentation (legacy)
│   └── CHARACTER_DISCOVERY_PROMPT.md
├── mapping/            # Mechanical segmentation, speaker attribution
│   └── segmenter.py              # Text splitting + tag-based attribution
├── voices/             # TTS engine and voice management
│   ├── qwen_tts.py               # Qwen3-TTS engine (voice design + cloning)
│   ├── lmstudio_lifecycle.py     # LM Studio VRAM management
│   ├── edge_tts.py               # Edge-TTS wrapper (legacy)
│   ├── voice_designer.py         # Voice reference generation pipeline
│   └── embeddings.py             # Speaker embedding extraction
├── generation/         # Audio generation
├── web/                # FastAPI review UI
└── scripts/            # Standalone scripts
    ├── resegment_all.py          # Batch segmentation + attribution
    ├── generate_voice_references.py  # Voice ref generation
    ├── generate_chapter_audio.py     # Per-chapter audio
    └── generate_all_audio.py         # Background full generation

Key Design Decisions

1. Quality over speed: Each pass has quality gates; the pipeline can stop and wait for human review
2. Two-phase segmentation: Mechanical segmenter handles text splitting (narration by sentence; quoted dialogue kept whole), LLM handles speaker attribution — this division of labor produces a 0.05% dialogue-as-narrator error rate (1/1,882)
3. LLM attributes all dialogue: The LLM attributes every dialogue utterance, using the mechanical current_speaker as a hint and overriding it when wrong — ambiguous or incorrectly guessed lines get corrected
4. Narrator is not a character: Dedicated soft-spoken male voice (en-US-GuyNeural:-10), never participates in character discovery
5. 59 unique voices via Qwen3-TTS: VoiceDesign generates unique voice profiles from character descriptions; VoiceClone maintains consistency across utterances using reference audio
6. Character profiles in JSON: speakers.json stores all character data; voice_assignments.json maps characters to voice references
7. Scene break handling: <<<>>> markers preserved as special utterances, LLM maintains context across scene breaks
8. Web review at checkpoints: HTMX-powered UI for human approval of voices, speaker assignments, and chapter audio
9. Punctuation as prosody: Question marks, ellipses, em-dashes, and ALL-CAPS words change how lines are spoken
10. Cast-wide distinctiveness gate: Characters must be audibly distinguishable, not just textually different
11. Verification phrase: Rainbow Passage (phonetically balanced paragraph covering all English phonemes) used for voice testing
12. Author watermark: "Author B.D. Murphy" embedded in final audiobook