UmbraAudio/AGENTS.md

UmbraAudio — Development Conventions

Project Overview

Quality-first multi-pass audiobook generation from English manuscripts. Each pass has quality gates. Quality is the primary concern; generation speed is secondary.

Architecture

• 5-pass pipeline: Analysis → Voice Design → Mapping → Test → Generate
• Multi-engine TTS: Edge-TTS (primary for voice variety), Chatterbox (secondary for cloning)
• LLM-enhanced: gemma-4-12b-it-qat for character discovery and profiling
• Web review: FastAPI + HTMX at checkpoints for human approval

Voice System

• Narrator: Dedicated soft-spoken male voice (en-US-GuyNeural:-10), deep, radio speaker style
• Characters: 59 unique voices from Edge-TTS with pitch shifting
• Voice Assignment: Stored in .umbra/voice_assignments.json
• Test Samples: Verification phrase uses the Rainbow Passage (phonetically balanced paragraph covering all English phonemes)

Tech Stack

• Python 3.11+, asyncio throughout
• SQLite via aiosqlite for project state
• FastAPI + HTMX for web review UI
• Click + Rich for CLI
• Edge-TTS for voice generation
• LM Studio (google/gemma-4-e4b) for character discovery and speaker attribution

Commands

• Lint: ruff check umbraaudio/ scripts/
• Format: ruff format umbraaudio/ scripts/
• Type check: mypy umbraaudio/
• Test: pytest tests/ -v
• Run: python -m umbraaudio.cli
• Web: python -c "from pathlib import Path; from umbraaudio.web import create_app; from umbraaudio.config import load_config; import uvicorn; config = load_config(Path('.')); app = create_app(config); uvicorn.run(app, host='0.0.0.0', port=8420)"
• Mechanical re-segment: python scripts/resegment_all.py --mechanical-only
• LLM speaker attribution: python scripts/reattribute_all.py --model google/gemma-4-e4b
• LLM character discovery: 4-pass pipeline via /api/analyze endpoint (see specs/02-analysis.md)

Code Conventions

• Type hints on all function signatures
• Docstrings on public classes/methods
• from __future__ import annotations in every file
• No comments unless explaining why, not what
• Use logging module, never print() for debug output
• Async for all I/O-bound operations
• Dataclasses or Pydantic models for structured data
• Paths via pathlib.Path, never string paths

Testing

• Tests mirror source structure: umbraaudio/parsers/ → tests/test_parsers.py
• Use pytest fixtures for database, config, mock TTS engine
• Mock external services (LLM APIs, TTS engines) in tests
• Integration tests for full pipeline with sample manuscript

File Naming

• Modules: snake_case.py
• Classes: PascalCase
• Functions: snake_case
• Constants: UPPER_SNAKE_CASE
• Private: _leading_underscore

Dependencies

• Check pyproject.toml before adding new dependencies
• Never add a dependency if stdlib suffices
• Prefer: stdlib → third-party → custom implementation

Current Session Summary

Goal

Fix speaker-attribution errors so the user can review/accept voices with confidence. Reported failures in chapter "Salvage" (ch6): utt35 "We have strict orders…" (after "she asks the lead guard… He replies") was tagged Kabir instead of the lead guard, and utt156–157 "The voice calls again" / "Help me, please." was tagged Deal Hunter instead of the female space-survivor (Kainda).

Root Cause Analysis

1. No rule for unnamed role tags: narration "the lead guard", "the robot" → LLM guessed a named character present in the scene (Kabir) instead of the role.
2. Old prompt rule 3e over-applied: any "voice"/comm → Deal Hunter, so a human voice ("Help me, please.") became the ship AI.
3. Context window too narrow: _build_position_context only looked 8 utterances back → "It sounds female," (13 lines before) fell outside, so a female survivor's voice went to Deal Hunter.
4. No verification pass: attribution was trust-on-faith; wrong speakers were never caught mechanically.
5. ModelCoordinator.load_lm was not idempotent: re-entering CHAPTER_ANALYSIS always re-POSTed /v1/models/load → a second copy of a 9 GB model failed to allocate on the 16 GB Mac.

Fixes Implemented (this session)

• umbraaudio/analysis/llm_attributor.py prompt: new rule 3e (Deal Hunter/AI-1 ONLY when narration explicitly names a computer/AI/comm system; a bare "a voice is heard"/"the voice calls again" over radio is HUMAN — use gender cues like "it sounds female"), 3f (pronoun coreference; never resolve to an absent character or a named char when the antecedent is an unnamed role), 3g (unnamed roles → the role speaker's EXACT primary_name e.g. "Chief of the Guard", else a Title Case role name; NEVER a random named character), 3h (repeated lines — "The voice calls again, 'Help me, please.'" — keep the first occurrence's speaker). Context widened: _extract_scene_narration 12→20 lines; _build_position_context 8→40 utterances back, labelled dialogue, 4200-char / 45-part caps.
• New umbraaudio/analysis/attribution_qa.py (mechanical cross-check): check_attribution() flags speech_tag_mismatch (conf 0.9 resolved / 0.7 unresolved-name), narrator_dialogue (0.95), role_tag_guessed (0.5). Tag applied only from the IMMEDIATELY preceding narration (never across a dialogue line — prevents shared-tag bleed like "Kainda asks," + reply). Pronoun subjects skipped (coreference is the LLM's job). _names_match() falls back to word-overlap for unresolved titles ("Wakam" vs "Trader Wakam", "Admiral Wake" vs "Admiral Wilbur Wake"). fix_high_confidence() writes back only resolved exact-name mismatches. Role regex capped at 2 role words and rejects trailing conjunctions ("the robot and tells").
• New scripts/qa_attribution.py CLI: --chapters N, --fix, --type. _build_alias_map includes primary, aliases, and first-word mappings.
• umbraaudio/voices/model_coordinator.py: load_lm (sync + async) now no-ops when the requested model already has a loaded instance — re-entering CHAPTER_ANALYSIS no longer tries to allocate a second copy.
• .umbra/speakers.json aliases added: Admiral Wake/The Admiral→Admiral Wilbur Wake, Captain Antlion/Antlion→Captain Wolfgang Antlion, Wakam/Wakam the Trader→Trader Wakam, Vicky→Duchess Victoria, Pirate Leader/The Pirate Leader→Pirate Clan Leader, Niñera/Nini/The Nanny→Nanny (Niñera = the nanny robot, Support Robot 74532). This session: Lead Guard→Chief of the Guard, Lead Trainer→Training Officer, Bill→Worker Robot (Bill), Captain Alec Weston→Captain Weston.

Completed Work

• Full 35-chapter LLM re-attribute with the new prompt (google/gemma-4-e4b, ~21 tok/s at ~1.3 min/batch; total ~2 h): 1878 dialogue utterances, 0 left as narrator. Chapters 0–4 and 33–35 are front/back matter with no dialogue (skipped as "no utterances" or 0 dialogue).
• QA + auto-fix loop: book-wide QA fell from 38 → 10 issues. 21 auto-fixes applied (resolved tag mismatches) plus 5 evidence-driven role fixes.
• Fixed speakers: ch6 utt157→Kainda Walters Foster Murphy ("Help me, please." = female survivor); ch6 utt44/49/52/56/211→Nanny (the "robot"/Niñera who rushes in and explains the attack is the nanny bot); ch8 utt18/33→Trader Wakam; ch9 utt22→Duchess Victoria (tag "Vicky says assertively"); ch16 utt120→Pirate Clan Leader (the "modified voice" on the call); ch17 utt94→Deal Hunter; ch18 utt154→Xu Ming, utt168→Kainda; ch20 utt230/utt345→Warbots / Pirate Clan Leader; ch23 utt63→Deal Hunter; ch24 utt15→Kainda; ch25 utt131/utt224→Kainda, utt133→Warbots; ch30 utt82/115→Admiral Wilbur Wake, utt113/133→Duchess Victoria, utt162/170→Kabir, utt194/241→Captain Wolfgang Antlion; ch32 utt68/87→Captain Wolfgang Antlion.
• Ghost-speaker sweep (post-review, driven by "Kabir isn't in ch6"): script cross-checked every assigned speaker against all name/alias tokens present in the chapter text (stopwords filtered, alias-aware — Willamina's "Captain" alias counts). Found and fixed:
  • ch6 utt38/40→Nanny (were Kabir; the "service robot … named Niñera" who enters with the suit says these; Kabir appears only once in ch6, in narration as the off-screen brother).
  • ch9 utt2/8/11/13→Training Officer (were Admiral Wilbur Wake; narration: "She is talking to the lead trainer for the young royals" / "the argument with the training officer"; profile: [9]).
  • ch16 utt122/123/124→Pirate Clan Leader (were Admiral Wilbur Wake; the "modified voice" scolding the failing pirate, same voice as utt120).
  • ch24 utt16→Pilot (Freighter) (were Alec; same freighter pilot as utt18/20; Alec is a ch8-only Wakam associate).
  • ch26 utt128/139→Frigate Captain (were Admiral Wilbur Wake; "the pirate captain … yells" on the frigate bridge), ch26 utt135→Frigate Captain (were Willamina; "Seal the bridge" order after the captain sees the video). Profile: [24,26]. ch26 utt194→Kainda Walters Foster Murphy (were Admiral Wilbur Wake; she's the one who made it to 12.5G in the pilot chair — Warbots reply "Yes, Captain").
• Remaining QA flags are deliberate assignments the checker can't see intent behind (Nanny=robot, Kainda="voice", Warbots, Pirate Clan Leader) — treat as clean.
• pytest: 181 passed, 3 warnings (17 in tests/test_attribution_qa.py covering tag mismatch, alias resolution, tag isolation, pronouns, word-overlap fallback, conjunction rejection, auto-fix gating). ruff check + mypy clean on attribution_qa.py, qa_attribution.py, model_coordinator.py, tests.
• Hardware reality (16 GB Mac): gemma-4-12b needs 16.88 GB → cannot fit; qwen3.5-9b fits but generates ~1 tok/s under swap (~15 h/book) → not viable; gemma-4-e4b is the practical best (~21 tok/s). LLM and Qwen TTS must not be resident simultaneously (coordinator enforces).

Flagged for User Review (ambiguous, left as-is)

Loc	Line	Currently	Question
ch16 utt121/125	"The ship landed with a different designation…" / "Yes."	Adonai Maul	The unnamed pirate subordinate failing the Clan Leader (Adonai Maul is a ch27 female bounty — wrong). No role profile; suggest a new "Pirate Commander"
ch20 utt223	"You are taking out multiple air pressure areas…"	Deal Hunter	Ship AI warning — likely already correct
ch24 utt108	"We don't know. Checking."	Deal Hunter	Probably a bridge/crew voice, not Deal Hunter
ch26 utt129	"Captain, it is a real alarm. Warbots just boarded…"	Willamina	Frigate bridge crew reporting to the pirate captain (not Willamina, she's on Deal Hunter); no crew role profile
ch26 utt155	"Four minutes and fifty seconds."	Warbots	Who is "the speaker"? A leader/announcer?
ch26 utt158	"Captain, they have taken the bridge…"	Willamina	Narration: "Status info is radioed to Deal Hunter, who tells Kainda" — could be Deal Hunter the AI, not Willamina

Known Pre-existing Bugs (reported, not fixed)

• scripts/generate_chapter_audio.py:132 references undefined phonetic (NameError on first utterance; superseded by generate_all_audio.py).
• start_server.sh launches the Gradio qwen-tts-demo on port 8000 — NOT the API server scripts/qwen_tts_server.py (port 8881) the app actually uses. Use qwen_tts_server.py for restarts.
• Orphaned Qwen generations: a client that times out does NOT cancel the server-side thread; the model keeps generating under _model_lock for minutes, blocking subsequent requests. Recovery = kill -9 the qwen server PID and restart scripts/qwen_tts_server.py.
• dictionaryapi.dev free tier rate-limits; residual bad records no longer matter for audio (dictionaryapi records are not applied to speech), would need bulk re-lookup to clean review panel.

Audio-Generation Prep (voice resolution + staleness)

• umbraaudio/generation/staleness.py: markers are now JSON {"speaker", "text"} instead of plain text. is_current(text, wav, speaker=...) only returns True when BOTH match. Legacy plain-text markers are ALWAYS stale → any existing ch1–7 wavs regenerate with corrected attribution on the next run. All call sites updated (generate_all_audio.py, routes.py ×3), speaker default normalized to "narrator".
• scripts/generate_all_audio.py: load_voice_references now uses rglob (finds nested refs like human_staff_person/messenger_ref.wav); resolve_reference now does case-insensitive canonical alias lookup, unique first-word match, and longest-substring fallback (was exact/casefold only → silent narrator fallback). 50/50 distinct speakers now resolve to a voice ref (0 fallback).
• New speakers added (speakers.json + voice_profiles.json + generated refs): Crew Member (ch24 bridge crew, male, clipped urgency; crew_member_ref.wav) and Supervisor (ch10 female production supervisor, stressed/assertive; supervisor_ref.wav). Both generated via scripts/generate_voice_references.py.
• All 184 pytest tests pass; ruff clean on staleness.py, generate_all_audio.py, tests.
• Next: run scripts/generate_all_audio.py (nohup) — Qwen TTS server up (port 8881); all 4406 utterances will regenerate since legacy markers are stale (~6.4 s/utt, multi-hour run).

Remaining Edge Cases

Chapter-5 audio verified end-to-end (files, markers, stitch, ASR). A/B samples: /tmp/author_note_before.wav, /tmp/author_note_natural.wav, /tmp/ch5_combined.wav. Optional future tooling: a per-utterance Whisper-based QA transcription pass over ch*_full.wav.

Previous Session (chapter-5 audio fix)

Complete — ch5 audio (markers, staleness, pronunciation, curly punctuation) verified. History below this section documents the model-coordinator session and speaker-attribution completion.