175 lines
8.5 KiB
Markdown
175 lines
8.5 KiB
Markdown
# Multi-Project Readiness — Audit & Action Plan
|
||
|
||
> **Status: HOLD.** Do **NOT** implement these changes until the **Deal Hunter**
|
||
> audiobook is **fully built** (including the stale-voice rebuild batch and m4b
|
||
> publish). These are code-only changes that touch shared pipeline code and must
|
||
> not risk the in-flight Deal Hunter build.
|
||
>
|
||
> **Created:** 2026-09-02
|
||
> **Related spec:** `specs/15-multi-project.md` (target/design). This doc is the
|
||
> **current-state audit** plus a **staged execution plan** to reconcile reality
|
||
> with that spec. Where the two differ, this audit reflects what is actually
|
||
> deployed today.
|
||
|
||
---
|
||
|
||
## 1. Why this matters (user requirement)
|
||
|
||
Starting a new book after Deal Hunter finishes. Requirements:
|
||
|
||
- **Separate in the UI** — add projects and switch between them easily.
|
||
- **Separate in sync/backup NAS** — each book stays isolated in backup.
|
||
- **Each project processed correctly** — no cross-contamination of data/voices/audio.
|
||
- **Answer the DB question** — does the database need to become a multi-user server?
|
||
|
||
---
|
||
|
||
## 2. Current-State Audit (verified 2026-09-02)
|
||
|
||
### What is ALREADY correct
|
||
|
||
1. **Per-project disk isolation.** Each book lives in its own directory
|
||
`ProjectBase/<slug>/` with its own `.umbra/` containing `project.db`,
|
||
`voices/`, `chapter_audio/`, `speakers.json`, `manifests/`, `voice_tests/`,
|
||
`export/`, etc. (config.py `list_projects()` only counts dirs with `.umbra/project.db`).
|
||
|
||
2. **Per-project SQLite DB.** Each project has its own `project.db` (10 tables;
|
||
all project tables carry `project_id` FKs). Chapters/utterances/audio are
|
||
structurally isolated by project directory.
|
||
- Global shared **voice registry** at `~/.umbra/voice_registry.db` (multi-project,
|
||
tracks `source_project`) is shared by design — the cross-book voice library.
|
||
|
||
3. **Project switching (Web).** `/projects` page; clicking a project sets an
|
||
`active_project` cookie; `_get_active_config()` in `web/routes.py:223` routes
|
||
all requests to that project dir.
|
||
|
||
4. **Project switching (CLI).** Every CLI command accepts `-p <slug>`.
|
||
|
||
5. **NAS sync is multi-project by nature.** `scripts/nas_lifecycle.py` uses
|
||
`rsync -a --delete` mirroring the **whole tree**, so each project folder
|
||
syncs independently and stays separate on `/Volumes/Umbra/UmbraAudio`.
|
||
(NAS is a file mirror, **not** a git clone.)
|
||
|
||
### Issues found (must fix before 2nd book)
|
||
|
||
| # | Severity | Issue | Where |
|
||
|---|----------|-------|-------|
|
||
| 1 | **CRITICAL** | Hard-coded `project_id=1` in pipeline code — works only because a fresh project DB has a single row `id=1`; breaks if two sources ever land in one DB | `cli.py`, `orchestrator.py`, `scripts/run_llm_attribution.py` |
|
||
| 2 | **HIGH** | No Web UI to **create a new project**. `POST /api/upload-manuscript` writes into the *currently active* project's existing DB (reuses `id=1`). Must use CLI `init` to bootstrap a new book | `web/routes.py:259` |
|
||
| 3 | **MEDIUM** | Stale legacy root project `.umbra/project.db` (named `...docx`) shows in `/projects`, duplicates the real project name, pollutes switching | top-level `.umbra/` |
|
||
| 4 | **MEDIUM** | No SQLite concurrency protection (no WAL, no locks); web opens fresh `sqlite3.connect()` per request while background TTS threads also write → `SQLITE_BUSY` if UI + CLI run on same project simultaneously | `database.py`, `web/routes.py` |
|
||
| 5 | **LOW** | Hardcoded project paths in sync/NAS helper scripts (single-project assumption) | `tts_server/sync_voice_data.sh`, `~/.umbra/backup_project_to_nas.sh` |
|
||
|
||
---
|
||
|
||
## 3. Does the database need to become a multi-user server?
|
||
|
||
**No.** Current DB is **SQLite, local, single-user** (no postgres/mysql anywhere).
|
||
For a single operator working on one book at a time this is correct and
|
||
appropriate — keep SQLite.
|
||
|
||
Recommended minimal hardening (no server migration needed):
|
||
|
||
1. **Enable WAL mode** on project + registry DB connections (cheap, prevents
|
||
`SQLITE_BUSY` when web UI and a CLI batch touch the same project).
|
||
2. Keep **one project per DB file** (already the case) — this is the real
|
||
isolation guarantee.
|
||
3. Only revisit the "server DB" question if you ever need **multiple concurrent
|
||
editors** on the *same* book (not your workflow).
|
||
|
||
---
|
||
|
||
## 4. Recommended Fixes (staged plan)
|
||
|
||
### Phase A — Correctness (do not block Deal Hunter; apply on next branch)
|
||
|
||
- [ ] **A1. Kill hard-coded `project_id=1`.** In `cli.py`, `orchestrator.py`,
|
||
`scripts/run_llm_attribution.py`, replace `get_project(1)`/`get_chapters(1)`/
|
||
`project_id=1` with a dynamic lookup of the active project's id from
|
||
`config.data_dir`. Add a small helper `active_project_id(db)`.
|
||
- [ ] **A2. Enable SQLite WAL** for project DB and `~/.umbra/voice_registry.db`
|
||
connections (`PRAGMA journal_mode=WAL`).
|
||
- [ ] **A3. Parameterize hardcoded sync/NAS scripts** by project slug
|
||
(`tts_server/sync_voice_data.sh`, `~/.umbra/backup_project_to_nas.sh`,
|
||
`~/.umbra/sync_to_nas.sh`).
|
||
|
||
### Phase B — Web project creation (UI-native new books)
|
||
|
||
- [ ] **B1. Add `POST /api/projects`** — create a fresh isolated project directory +
|
||
DB from an uploaded manuscript (title/author/slug). Reuse `umbraaudio init`
|
||
logic as a library call.
|
||
- [ ] **B2. Add `/projects/new` page** with upload form; on success activate the
|
||
new project (set cookie) and redirect to dashboard.
|
||
- [ ] **B3. Add "New Project" button** to `project_list.html` + navbar.
|
||
|
||
### Phase C — Cleanup & reconciliation with `specs/15-multi-project.md`
|
||
|
||
- [ ] **C1. Remove/migrate legacy root `.umbra`** so `/projects` list is clean
|
||
(use existing `migrate` CLI or a one-off cleanup). Keep the spec's intended
|
||
patterns in mind; decide whether to adopt the `projects/` registry layout
|
||
from the spec or keep the flat sibling-directory layout. **Recommendation:
|
||
keep the current flat per-project-directory layout** (already works) and
|
||
only add a lightweight global `projects` index if project metadata
|
||
(title/author/slug) is needed in the list without opening each DB.
|
||
- [ ] **C2. Add tests** `tests/test_multi_project.py` (isolation, dynamic id,
|
||
web create flow).
|
||
|
||
### Phase D — Verification scripts (idempotent, safe to run anytime)
|
||
|
||
- [ ] **D1. `verify_projects.py`** — list every project, its DB path, project id,
|
||
chapters, speakers, voice count; assert each `project.db` has exactly one
|
||
`project` row; flag cross-project references. (Model after the existing
|
||
voice-consistency verifier.)
|
||
- [ ] **D2. `verify_nas.py`** — after a sync, assert each project dir exists on the
|
||
NAS mirror and matches local (rsync `--dry-run -i`).
|
||
|
||
---
|
||
|
||
## 5. How to add a NEW book (today, before the fixes land)
|
||
|
||
Workflow that works **right now** and keeps projects isolated:
|
||
|
||
```bash
|
||
# 1. Create a new isolated project from a manuscript
|
||
cd /Volumes/1TB_APL_Storage/UmbraAudio
|
||
.venv/bin/python -m umbraaudio.cli init /path/to/manuscript.docx # -> ProjectBase/<slug>/
|
||
|
||
# 2. Run discovery/analysis on it
|
||
.venv/bin/python -m umbraaudio.cli -p <slug> discover
|
||
|
||
# 3. Full pipeline
|
||
.venv/bin/python -m umbraaudio.cli -p <slug> pipeline -c all
|
||
|
||
# 4. Open in web UI and activate
|
||
.venv/bin/umbraaudio review # then use /projects
|
||
```
|
||
|
||
**CAUTION:** never point a CLI `-p A` batch and the web UI at a different project
|
||
*simultaneously on the same project* (SQLite contention) until A2 (WAL) lands.
|
||
|
||
---
|
||
|
||
## 6. Deal Hunter completion checklist (gate before implementing plan above)
|
||
|
||
- [ ] Stale-voice rebuild batch finishes (PID 3600): chapters 7,9-14,16-32 regenerated clean.
|
||
- [ ] Re-run `verify_voices.py` → stale count = 0 (all voices current).
|
||
- [ ] m4b metadata entered (title/author/narrator/synopsis/cover) in m4b Meta page.
|
||
- [ ] m4b mux/publish step built and run → `.m4b` produced.
|
||
- [ ] NAS sync runs with the final Deal Hunter audio.
|
||
- [ ] THEN implement Phases A–D above.
|
||
|
||
---
|
||
|
||
## 7. Related files (for when we return)
|
||
|
||
- `umbraaudio/config.py` — `list_projects()` (line ~20), `load_config()`.
|
||
- `umbraaudio/web/routes.py` — `_get_active_config()` (~223), `/api/upload-manuscript` (259),
|
||
`/api/projects/activate/{slug}` (~244), `/projects` (~232).
|
||
- `umbraaudio/web/templates/project_list.html`, `base.html`.
|
||
- `umbraaudio/cli.py` — `init` (~25), `projects` (~969), `migrate` (~988); hard-coded ids.
|
||
- `umbraaudio/orchestrator.py` — hard-coded ids (~188,584).
|
||
- `scripts/run_llm_attribution.py` — hard-coded id (~30).
|
||
- `umbraaudio/database.py`, `umbraaudio/registry.py`, `umbraaudio/library.py` — DB layer.
|
||
- `scripts/nas_lifecycle.py` — NAS sync/prune; `~/.umbra/*.sh`, `tts_server/sync_voice_data.sh`.
|
||
- `specs/15-multi-project.md` — target design spec.
|