umbragit/UmbraAudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
UmbraAudio/MULTI_PROJECT_PLAN.md

8.5 KiB
Raw Permalink Blame History

Multi-Project Readiness — Audit & Action Plan

Status: HOLD. Do NOT implement these changes until the Deal Hunter audiobook is fully built (including the stale-voice rebuild batch and m4b publish). These are code-only changes that touch shared pipeline code and must not risk the in-flight Deal Hunter build.

Created: 2026-09-02 Related spec: specs/15-multi-project.md (target/design). This doc is the current-state audit plus a staged execution plan to reconcile reality with that spec. Where the two differ, this audit reflects what is actually deployed today.

1. Why this matters (user requirement)

Starting a new book after Deal Hunter finishes. Requirements:

  • Separate in the UI — add projects and switch between them easily.
  • Separate in sync/backup NAS — each book stays isolated in backup.
  • Each project processed correctly — no cross-contamination of data/voices/audio.
  • Answer the DB question — does the database need to become a multi-user server?

2. Current-State Audit (verified 2026-09-02)

What is ALREADY correct

  1. Per-project disk isolation. Each book lives in its own directory ProjectBase/<slug>/ with its own .umbra/ containing project.db, voices/, chapter_audio/, speakers.json, manifests/, voice_tests/, export/, etc. (config.py list_projects() only counts dirs with .umbra/project.db).

  2. Per-project SQLite DB. Each project has its own project.db (10 tables; all project tables carry project_id FKs). Chapters/utterances/audio are structurally isolated by project directory.

    • Global shared voice registry at ~/.umbra/voice_registry.db (multi-project, tracks source_project) is shared by design — the cross-book voice library.

  3. Project switching (Web). /projects page; clicking a project sets an active_project cookie; _get_active_config() in web/routes.py:223 routes all requests to that project dir.

  4. Project switching (CLI). Every CLI command accepts -p <slug>.

  5. NAS sync is multi-project by nature. scripts/nas_lifecycle.py uses rsync -a --delete mirroring the whole tree, so each project folder syncs independently and stays separate on /Volumes/Umbra/UmbraAudio. (NAS is a file mirror, not a git clone.)

Issues found (must fix before 2nd book)

# Severity Issue Where
1 CRITICAL Hard-coded project_id=1 in pipeline code — works only because a fresh project DB has a single row id=1; breaks if two sources ever land in one DB cli.py, orchestrator.py, scripts/run_llm_attribution.py
2 HIGH No Web UI to create a new project. POST /api/upload-manuscript writes into the currently active project's existing DB (reuses id=1). Must use CLI init to bootstrap a new book web/routes.py:259
3 MEDIUM Stale legacy root project .umbra/project.db (named ...docx) shows in /projects, duplicates the real project name, pollutes switching top-level .umbra/
4 MEDIUM No SQLite concurrency protection (no WAL, no locks); web opens fresh sqlite3.connect() per request while background TTS threads also write → SQLITE_BUSY if UI + CLI run on same project simultaneously database.py, web/routes.py
5 LOW Hardcoded project paths in sync/NAS helper scripts (single-project assumption) tts_server/sync_voice_data.sh, ~/.umbra/backup_project_to_nas.sh

3. Does the database need to become a multi-user server?

No. Current DB is SQLite, local, single-user (no postgres/mysql anywhere). For a single operator working on one book at a time this is correct and appropriate — keep SQLite.

Recommended minimal hardening (no server migration needed):

  1. Enable WAL mode on project + registry DB connections (cheap, prevents SQLITE_BUSY when web UI and a CLI batch touch the same project).
  2. Keep one project per DB file (already the case) — this is the real isolation guarantee.
  3. Only revisit the "server DB" question if you ever need multiple concurrent editors on the same book (not your workflow).

4. Recommended Fixes (staged plan)

Phase A — Correctness (do not block Deal Hunter; apply on next branch)

  • A1. Kill hard-coded project_id=1. In cli.py, orchestrator.py, scripts/run_llm_attribution.py, replace get_project(1)/get_chapters(1)/ project_id=1 with a dynamic lookup of the active project's id from config.data_dir. Add a small helper active_project_id(db).
  • A2. Enable SQLite WAL for project DB and ~/.umbra/voice_registry.db connections (PRAGMA journal_mode=WAL).
  • A3. Parameterize hardcoded sync/NAS scripts by project slug (tts_server/sync_voice_data.sh, ~/.umbra/backup_project_to_nas.sh, ~/.umbra/sync_to_nas.sh).

Phase B — Web project creation (UI-native new books)

  • B1. Add POST /api/projects — create a fresh isolated project directory + DB from an uploaded manuscript (title/author/slug). Reuse umbraaudio init logic as a library call.
  • B2. Add /projects/new page with upload form; on success activate the new project (set cookie) and redirect to dashboard.
  • B3. Add "New Project" button to project_list.html + navbar.

Phase C — Cleanup & reconciliation with specs/15-multi-project.md

  • C1. Remove/migrate legacy root .umbra so /projects list is clean (use existing migrate CLI or a one-off cleanup). Keep the spec's intended patterns in mind; decide whether to adopt the projects/ registry layout from the spec or keep the flat sibling-directory layout. Recommendation: keep the current flat per-project-directory layout (already works) and only add a lightweight global projects index if project metadata (title/author/slug) is needed in the list without opening each DB.
  • C2. Add tests tests/test_multi_project.py (isolation, dynamic id, web create flow).

Phase D — Verification scripts (idempotent, safe to run anytime)

  • D1. verify_projects.py — list every project, its DB path, project id, chapters, speakers, voice count; assert each project.db has exactly one project row; flag cross-project references. (Model after the existing voice-consistency verifier.)
  • D2. verify_nas.py — after a sync, assert each project dir exists on the NAS mirror and matches local (rsync --dry-run -i).

5. How to add a NEW book (today, before the fixes land)

Workflow that works right now and keeps projects isolated:

# 1. Create a new isolated project from a manuscript
cd /Volumes/1TB_APL_Storage/UmbraAudio
.venv/bin/python -m umbraaudio.cli init /path/to/manuscript.docx    # -> ProjectBase/<slug>/

# 2. Run discovery/analysis on it
.venv/bin/python -m umbraaudio.cli -p <slug> discover

# 3. Full pipeline
.venv/bin/python -m umbraaudio.cli -p <slug> pipeline -c all

# 4. Open in web UI and activate
.venv/bin/umbraaudio review   # then use /projects

CAUTION: never point a CLI -p A batch and the web UI at a different project simultaneously on the same project (SQLite contention) until A2 (WAL) lands.

6. Deal Hunter completion checklist (gate before implementing plan above)

  • Stale-voice rebuild batch finishes (PID 3600): chapters 7,9-14,16-32 regenerated clean.
  • Re-run verify_voices.py → stale count = 0 (all voices current).
  • m4b metadata entered (title/author/narrator/synopsis/cover) in m4b Meta page.
  • m4b mux/publish step built and run → .m4b produced.
  • NAS sync runs with the final Deal Hunter audio.
  • THEN implement Phases A–D above.

7. Related files (for when we return)

  • umbraaudio/config.py — list_projects() (line ~20), load_config().
  • umbraaudio/web/routes.py — _get_active_config() (~223), /api/upload-manuscript (259), /api/projects/activate/{slug} (~244), /projects (~232).
  • umbraaudio/web/templates/project_list.html, base.html.
  • umbraaudio/cli.py — init (~25), projects (~969), migrate (~988); hard-coded ids.
  • umbraaudio/orchestrator.py — hard-coded ids (~188,584).
  • scripts/run_llm_attribution.py — hard-coded id (~30).
  • umbraaudio/database.py, umbraaudio/registry.py, umbraaudio/library.py — DB layer.
  • scripts/nas_lifecycle.py — NAS sync/prune; ~/.umbra/*.sh, tts_server/sync_voice_data.sh.
  • specs/15-multi-project.md — target design spec.