diff --git a/MULTI_PROJECT_PLAN.md b/MULTI_PROJECT_PLAN.md new file mode 100644 index 0000000..b0b528c --- /dev/null +++ b/MULTI_PROJECT_PLAN.md @@ -0,0 +1,174 @@ +# Multi-Project Readiness — Audit & Action Plan + +> **Status: HOLD.** Do **NOT** implement these changes until the **Deal Hunter** +> audiobook is **fully built** (including the stale-voice rebuild batch and m4b +> publish). These are code-only changes that touch shared pipeline code and must +> not risk the in-flight Deal Hunter build. +> +> **Created:** 2026-09-02 +> **Related spec:** `specs/15-multi-project.md` (target/design). This doc is the +> **current-state audit** plus a **staged execution plan** to reconcile reality +> with that spec. Where the two differ, this audit reflects what is actually +> deployed today. + +--- + +## 1. Why this matters (user requirement) + +Starting a new book after Deal Hunter finishes. Requirements: + +- **Separate in the UI** — add projects and switch between them easily. +- **Separate in sync/backup NAS** — each book stays isolated in backup. +- **Each project processed correctly** — no cross-contamination of data/voices/audio. +- **Answer the DB question** — does the database need to become a multi-user server? + +--- + +## 2. Current-State Audit (verified 2026-09-02) + +### What is ALREADY correct + +1. **Per-project disk isolation.** Each book lives in its own directory + `ProjectBase//` with its own `.umbra/` containing `project.db`, + `voices/`, `chapter_audio/`, `speakers.json`, `manifests/`, `voice_tests/`, + `export/`, etc. (config.py `list_projects()` only counts dirs with `.umbra/project.db`). + +2. **Per-project SQLite DB.** Each project has its own `project.db` (10 tables; + all project tables carry `project_id` FKs). Chapters/utterances/audio are + structurally isolated by project directory. + - Global shared **voice registry** at `~/.umbra/voice_registry.db` (multi-project, + tracks `source_project`) is shared by design — the cross-book voice library. + +3. **Project switching (Web).** `/projects` page; clicking a project sets an + `active_project` cookie; `_get_active_config()` in `web/routes.py:223` routes + all requests to that project dir. + +4. **Project switching (CLI).** Every CLI command accepts `-p `. + +5. **NAS sync is multi-project by nature.** `scripts/nas_lifecycle.py` uses + `rsync -a --delete` mirroring the **whole tree**, so each project folder + syncs independently and stays separate on `/Volumes/Umbra/UmbraAudio`. + (NAS is a file mirror, **not** a git clone.) + +### Issues found (must fix before 2nd book) + +| # | Severity | Issue | Where | +|---|----------|-------|-------| +| 1 | **CRITICAL** | Hard-coded `project_id=1` in pipeline code — works only because a fresh project DB has a single row `id=1`; breaks if two sources ever land in one DB | `cli.py`, `orchestrator.py`, `scripts/run_llm_attribution.py` | +| 2 | **HIGH** | No Web UI to **create a new project**. `POST /api/upload-manuscript` writes into the *currently active* project's existing DB (reuses `id=1`). Must use CLI `init` to bootstrap a new book | `web/routes.py:259` | +| 3 | **MEDIUM** | Stale legacy root project `.umbra/project.db` (named `...docx`) shows in `/projects`, duplicates the real project name, pollutes switching | top-level `.umbra/` | +| 4 | **MEDIUM** | No SQLite concurrency protection (no WAL, no locks); web opens fresh `sqlite3.connect()` per request while background TTS threads also write → `SQLITE_BUSY` if UI + CLI run on same project simultaneously | `database.py`, `web/routes.py` | +| 5 | **LOW** | Hardcoded project paths in sync/NAS helper scripts (single-project assumption) | `tts_server/sync_voice_data.sh`, `~/.umbra/backup_project_to_nas.sh` | + +--- + +## 3. Does the database need to become a multi-user server? + +**No.** Current DB is **SQLite, local, single-user** (no postgres/mysql anywhere). +For a single operator working on one book at a time this is correct and +appropriate — keep SQLite. + +Recommended minimal hardening (no server migration needed): + +1. **Enable WAL mode** on project + registry DB connections (cheap, prevents + `SQLITE_BUSY` when web UI and a CLI batch touch the same project). +2. Keep **one project per DB file** (already the case) — this is the real + isolation guarantee. +3. Only revisit the "server DB" question if you ever need **multiple concurrent + editors** on the *same* book (not your workflow). + +--- + +## 4. Recommended Fixes (staged plan) + +### Phase A — Correctness (do not block Deal Hunter; apply on next branch) + +- [ ] **A1. Kill hard-coded `project_id=1`.** In `cli.py`, `orchestrator.py`, + `scripts/run_llm_attribution.py`, replace `get_project(1)`/`get_chapters(1)`/ + `project_id=1` with a dynamic lookup of the active project's id from + `config.data_dir`. Add a small helper `active_project_id(db)`. +- [ ] **A2. Enable SQLite WAL** for project DB and `~/.umbra/voice_registry.db` + connections (`PRAGMA journal_mode=WAL`). +- [ ] **A3. Parameterize hardcoded sync/NAS scripts** by project slug + (`tts_server/sync_voice_data.sh`, `~/.umbra/backup_project_to_nas.sh`, + `~/.umbra/sync_to_nas.sh`). + +### Phase B — Web project creation (UI-native new books) + +- [ ] **B1. Add `POST /api/projects`** — create a fresh isolated project directory + + DB from an uploaded manuscript (title/author/slug). Reuse `umbraaudio init` + logic as a library call. +- [ ] **B2. Add `/projects/new` page** with upload form; on success activate the + new project (set cookie) and redirect to dashboard. +- [ ] **B3. Add "New Project" button** to `project_list.html` + navbar. + +### Phase C — Cleanup & reconciliation with `specs/15-multi-project.md` + +- [ ] **C1. Remove/migrate legacy root `.umbra`** so `/projects` list is clean + (use existing `migrate` CLI or a one-off cleanup). Keep the spec's intended + patterns in mind; decide whether to adopt the `projects/` registry layout + from the spec or keep the flat sibling-directory layout. **Recommendation: + keep the current flat per-project-directory layout** (already works) and + only add a lightweight global `projects` index if project metadata + (title/author/slug) is needed in the list without opening each DB. +- [ ] **C2. Add tests** `tests/test_multi_project.py` (isolation, dynamic id, + web create flow). + +### Phase D — Verification scripts (idempotent, safe to run anytime) + +- [ ] **D1. `verify_projects.py`** — list every project, its DB path, project id, + chapters, speakers, voice count; assert each `project.db` has exactly one + `project` row; flag cross-project references. (Model after the existing + voice-consistency verifier.) +- [ ] **D2. `verify_nas.py`** — after a sync, assert each project dir exists on the + NAS mirror and matches local (rsync `--dry-run -i`). + +--- + +## 5. How to add a NEW book (today, before the fixes land) + +Workflow that works **right now** and keeps projects isolated: + +```bash +# 1. Create a new isolated project from a manuscript +cd /Volumes/1TB_APL_Storage/UmbraAudio +.venv/bin/python -m umbraaudio.cli init /path/to/manuscript.docx # -> ProjectBase// + +# 2. Run discovery/analysis on it +.venv/bin/python -m umbraaudio.cli -p discover + +# 3. Full pipeline +.venv/bin/python -m umbraaudio.cli -p pipeline -c all + +# 4. Open in web UI and activate +.venv/bin/umbraaudio review # then use /projects +``` + +**CAUTION:** never point a CLI `-p A` batch and the web UI at a different project +*simultaneously on the same project* (SQLite contention) until A2 (WAL) lands. + +--- + +## 6. Deal Hunter completion checklist (gate before implementing plan above) + +- [ ] Stale-voice rebuild batch finishes (PID 3600): chapters 7,9-14,16-32 regenerated clean. +- [ ] Re-run `verify_voices.py` → stale count = 0 (all voices current). +- [ ] m4b metadata entered (title/author/narrator/synopsis/cover) in m4b Meta page. +- [ ] m4b mux/publish step built and run → `.m4b` produced. +- [ ] NAS sync runs with the final Deal Hunter audio. +- [ ] THEN implement Phases A–D above. + +--- + +## 7. Related files (for when we return) + +- `umbraaudio/config.py` — `list_projects()` (line ~20), `load_config()`. +- `umbraaudio/web/routes.py` — `_get_active_config()` (~223), `/api/upload-manuscript` (259), + `/api/projects/activate/{slug}` (~244), `/projects` (~232). +- `umbraaudio/web/templates/project_list.html`, `base.html`. +- `umbraaudio/cli.py` — `init` (~25), `projects` (~969), `migrate` (~988); hard-coded ids. +- `umbraaudio/orchestrator.py` — hard-coded ids (~188,584). +- `scripts/run_llm_attribution.py` — hard-coded id (~30). +- `umbraaudio/database.py`, `umbraaudio/registry.py`, `umbraaudio/library.py` — DB layer. +- `scripts/nas_lifecycle.py` — NAS sync/prune; `~/.umbra/*.sh`, `tts_server/sync_voice_data.sh`. +- `specs/15-multi-project.md` — target design spec.