Add multi-project readiness audit and action plan (HOLD until Deal Hunter done)
This commit is contained in:
B.D. Murphy
174
MULTI_PROJECT_PLAN.md
Normal file
174
MULTI_PROJECT_PLAN.md
Normal file
@@ -0,0 +1,174 @@
|
|||||||
|
# Multi-Project Readiness — Audit & Action Plan
|
||||||
|
|
||||||
|
> **Status: HOLD.** Do **NOT** implement these changes until the **Deal Hunter**
|
||||||
|
> audiobook is **fully built** (including the stale-voice rebuild batch and m4b
|
||||||
|
> publish). These are code-only changes that touch shared pipeline code and must
|
||||||
|
> not risk the in-flight Deal Hunter build.
|
||||||
|
>
|
||||||
|
> **Created:** 2026-09-02
|
||||||
|
> **Related spec:** `specs/15-multi-project.md` (target/design). This doc is the
|
||||||
|
> **current-state audit** plus a **staged execution plan** to reconcile reality
|
||||||
|
> with that spec. Where the two differ, this audit reflects what is actually
|
||||||
|
> deployed today.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why this matters (user requirement)
|
||||||
|
|
||||||
|
Starting a new book after Deal Hunter finishes. Requirements:
|
||||||
|
|
||||||
|
- **Separate in the UI** — add projects and switch between them easily.
|
||||||
|
- **Separate in sync/backup NAS** — each book stays isolated in backup.
|
||||||
|
- **Each project processed correctly** — no cross-contamination of data/voices/audio.
|
||||||
|
- **Answer the DB question** — does the database need to become a multi-user server?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Current-State Audit (verified 2026-09-02)
|
||||||
|
|
||||||
|
### What is ALREADY correct
|
||||||
|
|
||||||
|
1. **Per-project disk isolation.** Each book lives in its own directory
|
||||||
|
`ProjectBase/<slug>/` with its own `.umbra/` containing `project.db`,
|
||||||
|
`voices/`, `chapter_audio/`, `speakers.json`, `manifests/`, `voice_tests/`,
|
||||||
|
`export/`, etc. (config.py `list_projects()` only counts dirs with `.umbra/project.db`).
|
||||||
|
|
||||||
|
2. **Per-project SQLite DB.** Each project has its own `project.db` (10 tables;
|
||||||
|
all project tables carry `project_id` FKs). Chapters/utterances/audio are
|
||||||
|
structurally isolated by project directory.
|
||||||
|
- Global shared **voice registry** at `~/.umbra/voice_registry.db` (multi-project,
|
||||||
|
tracks `source_project`) is shared by design — the cross-book voice library.
|
||||||
|
|
||||||
|
3. **Project switching (Web).** `/projects` page; clicking a project sets an
|
||||||
|
`active_project` cookie; `_get_active_config()` in `web/routes.py:223` routes
|
||||||
|
all requests to that project dir.
|
||||||
|
|
||||||
|
4. **Project switching (CLI).** Every CLI command accepts `-p <slug>`.
|
||||||
|
|
||||||
|
5. **NAS sync is multi-project by nature.** `scripts/nas_lifecycle.py` uses
|
||||||
|
`rsync -a --delete` mirroring the **whole tree**, so each project folder
|
||||||
|
syncs independently and stays separate on `/Volumes/Umbra/UmbraAudio`.
|
||||||
|
(NAS is a file mirror, **not** a git clone.)
|
||||||
|
|
||||||
|
### Issues found (must fix before 2nd book)
|
||||||
|
|
||||||
|
| # | Severity | Issue | Where |
|
||||||
|
|---|----------|-------|-------|
|
||||||
|
| 1 | **CRITICAL** | Hard-coded `project_id=1` in pipeline code — works only because a fresh project DB has a single row `id=1`; breaks if two sources ever land in one DB | `cli.py`, `orchestrator.py`, `scripts/run_llm_attribution.py` |
|
||||||
|
| 2 | **HIGH** | No Web UI to **create a new project**. `POST /api/upload-manuscript` writes into the *currently active* project's existing DB (reuses `id=1`). Must use CLI `init` to bootstrap a new book | `web/routes.py:259` |
|
||||||
|
| 3 | **MEDIUM** | Stale legacy root project `.umbra/project.db` (named `...docx`) shows in `/projects`, duplicates the real project name, pollutes switching | top-level `.umbra/` |
|
||||||
|
| 4 | **MEDIUM** | No SQLite concurrency protection (no WAL, no locks); web opens fresh `sqlite3.connect()` per request while background TTS threads also write → `SQLITE_BUSY` if UI + CLI run on same project simultaneously | `database.py`, `web/routes.py` |
|
||||||
|
| 5 | **LOW** | Hardcoded project paths in sync/NAS helper scripts (single-project assumption) | `tts_server/sync_voice_data.sh`, `~/.umbra/backup_project_to_nas.sh` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Does the database need to become a multi-user server?
|
||||||
|
|
||||||
|
**No.** Current DB is **SQLite, local, single-user** (no postgres/mysql anywhere).
|
||||||
|
For a single operator working on one book at a time this is correct and
|
||||||
|
appropriate — keep SQLite.
|
||||||
|
|
||||||
|
Recommended minimal hardening (no server migration needed):
|
||||||
|
|
||||||
|
1. **Enable WAL mode** on project + registry DB connections (cheap, prevents
|
||||||
|
`SQLITE_BUSY` when web UI and a CLI batch touch the same project).
|
||||||
|
2. Keep **one project per DB file** (already the case) — this is the real
|
||||||
|
isolation guarantee.
|
||||||
|
3. Only revisit the "server DB" question if you ever need **multiple concurrent
|
||||||
|
editors** on the *same* book (not your workflow).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Recommended Fixes (staged plan)
|
||||||
|
|
||||||
|
### Phase A — Correctness (do not block Deal Hunter; apply on next branch)
|
||||||
|
|
||||||
|
- [ ] **A1. Kill hard-coded `project_id=1`.** In `cli.py`, `orchestrator.py`,
|
||||||
|
`scripts/run_llm_attribution.py`, replace `get_project(1)`/`get_chapters(1)`/
|
||||||
|
`project_id=1` with a dynamic lookup of the active project's id from
|
||||||
|
`config.data_dir`. Add a small helper `active_project_id(db)`.
|
||||||
|
- [ ] **A2. Enable SQLite WAL** for project DB and `~/.umbra/voice_registry.db`
|
||||||
|
connections (`PRAGMA journal_mode=WAL`).
|
||||||
|
- [ ] **A3. Parameterize hardcoded sync/NAS scripts** by project slug
|
||||||
|
(`tts_server/sync_voice_data.sh`, `~/.umbra/backup_project_to_nas.sh`,
|
||||||
|
`~/.umbra/sync_to_nas.sh`).
|
||||||
|
|
||||||
|
### Phase B — Web project creation (UI-native new books)
|
||||||
|
|
||||||
|
- [ ] **B1. Add `POST /api/projects`** — create a fresh isolated project directory +
|
||||||
|
DB from an uploaded manuscript (title/author/slug). Reuse `umbraaudio init`
|
||||||
|
logic as a library call.
|
||||||
|
- [ ] **B2. Add `/projects/new` page** with upload form; on success activate the
|
||||||
|
new project (set cookie) and redirect to dashboard.
|
||||||
|
- [ ] **B3. Add "New Project" button** to `project_list.html` + navbar.
|
||||||
|
|
||||||
|
### Phase C — Cleanup & reconciliation with `specs/15-multi-project.md`
|
||||||
|
|
||||||
|
- [ ] **C1. Remove/migrate legacy root `.umbra`** so `/projects` list is clean
|
||||||
|
(use existing `migrate` CLI or a one-off cleanup). Keep the spec's intended
|
||||||
|
patterns in mind; decide whether to adopt the `projects/` registry layout
|
||||||
|
from the spec or keep the flat sibling-directory layout. **Recommendation:
|
||||||
|
keep the current flat per-project-directory layout** (already works) and
|
||||||
|
only add a lightweight global `projects` index if project metadata
|
||||||
|
(title/author/slug) is needed in the list without opening each DB.
|
||||||
|
- [ ] **C2. Add tests** `tests/test_multi_project.py` (isolation, dynamic id,
|
||||||
|
web create flow).
|
||||||
|
|
||||||
|
### Phase D — Verification scripts (idempotent, safe to run anytime)
|
||||||
|
|
||||||
|
- [ ] **D1. `verify_projects.py`** — list every project, its DB path, project id,
|
||||||
|
chapters, speakers, voice count; assert each `project.db` has exactly one
|
||||||
|
`project` row; flag cross-project references. (Model after the existing
|
||||||
|
voice-consistency verifier.)
|
||||||
|
- [ ] **D2. `verify_nas.py`** — after a sync, assert each project dir exists on the
|
||||||
|
NAS mirror and matches local (rsync `--dry-run -i`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. How to add a NEW book (today, before the fixes land)
|
||||||
|
|
||||||
|
Workflow that works **right now** and keeps projects isolated:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Create a new isolated project from a manuscript
|
||||||
|
cd /Volumes/1TB_APL_Storage/UmbraAudio
|
||||||
|
.venv/bin/python -m umbraaudio.cli init /path/to/manuscript.docx # -> ProjectBase/<slug>/
|
||||||
|
|
||||||
|
# 2. Run discovery/analysis on it
|
||||||
|
.venv/bin/python -m umbraaudio.cli -p <slug> discover
|
||||||
|
|
||||||
|
# 3. Full pipeline
|
||||||
|
.venv/bin/python -m umbraaudio.cli -p <slug> pipeline -c all
|
||||||
|
|
||||||
|
# 4. Open in web UI and activate
|
||||||
|
.venv/bin/umbraaudio review # then use /projects
|
||||||
|
```
|
||||||
|
|
||||||
|
**CAUTION:** never point a CLI `-p A` batch and the web UI at a different project
|
||||||
|
*simultaneously on the same project* (SQLite contention) until A2 (WAL) lands.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Deal Hunter completion checklist (gate before implementing plan above)
|
||||||
|
|
||||||
|
- [ ] Stale-voice rebuild batch finishes (PID 3600): chapters 7,9-14,16-32 regenerated clean.
|
||||||
|
- [ ] Re-run `verify_voices.py` → stale count = 0 (all voices current).
|
||||||
|
- [ ] m4b metadata entered (title/author/narrator/synopsis/cover) in m4b Meta page.
|
||||||
|
- [ ] m4b mux/publish step built and run → `.m4b` produced.
|
||||||
|
- [ ] NAS sync runs with the final Deal Hunter audio.
|
||||||
|
- [ ] THEN implement Phases A–D above.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Related files (for when we return)
|
||||||
|
|
||||||
|
- `umbraaudio/config.py` — `list_projects()` (line ~20), `load_config()`.
|
||||||
|
- `umbraaudio/web/routes.py` — `_get_active_config()` (~223), `/api/upload-manuscript` (259),
|
||||||
|
`/api/projects/activate/{slug}` (~244), `/projects` (~232).
|
||||||
|
- `umbraaudio/web/templates/project_list.html`, `base.html`.
|
||||||
|
- `umbraaudio/cli.py` — `init` (~25), `projects` (~969), `migrate` (~988); hard-coded ids.
|
||||||
|
- `umbraaudio/orchestrator.py` — hard-coded ids (~188,584).
|
||||||
|
- `scripts/run_llm_attribution.py` — hard-coded id (~30).
|
||||||
|
- `umbraaudio/database.py`, `umbraaudio/registry.py`, `umbraaudio/library.py` — DB layer.
|
||||||
|
- `scripts/nas_lifecycle.py` — NAS sync/prune; `~/.umbra/*.sh`, `tts_server/sync_voice_data.sh`.
|
||||||
|
- `specs/15-multi-project.md` — target design spec.
|
||||||
Reference in New Issue
Block a user