diff --git a/specs/20-enterprise-database-integration.md b/specs/20-enterprise-database-integration.md new file mode 100644 index 0000000..3b75fc2 --- /dev/null +++ b/specs/20-enterprise-database-integration.md @@ -0,0 +1,1856 @@ +# Spec 20 — Umbra Enterprise Database Integration: Continuity, Access, and Schema Contract + +- **Status:** DISCOVERY AND DESIGN ONLY. No production migration SQL. No connection to the + production database. No DDL/DML executed. +- **Date:** 2026-09-25 +- **Scope:** Align UmbraAudio (the English audiobook engine) and the future UmbraTrans + translation/review UI onto one shared enterprise PostgreSQL database without interrupting + audiobook production. +- **Authoritative inputs read:** `infra/postgres/001_control_plane.sql` (584 lines, verified + byte-identical to `/Volumes/Dock/worstauthor/oldbox/postgres/001_control_plane.sql`), + `infra/bootstrap_umbracontrol_database.sh`, `specs/18-UmbraTrans.md`, `specs/19-enterprise-control-plane.md`, + and the full current UmbraAudio source tree. +- **No secrets, credentials, connection strings, tokens, or key material appear in this document.** + +--- + +## A. Executive summary + +### A.1 What was actually verified + +The shared control-plane schema at version `001_control_plane` **exists as a file** and has a +bootstrap script. It contains **13 tables**, all tenancy/identity/audit: + +`schema_migrations`, `organizations`, `users`, `projects`, `roles`, `permissions`, +`role_permissions`, `user_roles`, `invitations`, `sessions`, `mfa_methods`, +`bootstrap_tokens`, `audit_events`. + +**Verified absent** (I am not claiming these exist): there is **no** source book, edition, +work, locale, analysis, speaker, voice, license, translation, audio run, artifact, job, lease, +or idempotency table. There is **no** `project_members` table (membership is encoded in +`user_roles`). There is **no** password column on `users`. There is **no** service-identity +table. Every ebook, voice, translation and audiobook domain concept must be created. + +### A.2 The three findings that decide the plan + +1. **There is no local account system to migrate.** The prompt asks me to "map any existing + local accounts to canonical enterprise user IDs." Verified: the audio system has **zero** + authentication, **zero** users, **zero** sessions, and no user model anywhere. The only + identity traces are the literal string `'user'` in `voice_review_history.reviewed_by` and + `exception_reviews.resolved_by`. There is exactly one human operator. Mapping therefore + means: create one canonical enterprise user, and attribute those legacy `'user'` strings to + that user id during backfill. No per-project users may be invented. + +2. **The audio system has no durable utterance identity, and it names its artifacts by a + mutable index.** On-disk audio is `chapter_audio/ch{N}/utt_{utterance_index:04d}.wav` + (`generation/chapter_builder.py:512`). `utterance_index` is renumbered by re-segmentation + and by `DELETE /api/utterance/{id}`, which reindexes the whole chapter and renames files on + disk (`web/routes.py:3405-3524`). `specs/18-UmbraTrans.md` §3.3 explicitly forbids mutable + indices as durable cross-language IDs. **The enterprise model must introduce a + content-addressed unit key and record it at generation time, while leaving the on-disk + filename unchanged** so that continuity and the existing staleness markers keep working. + +3. **The `umbracontrol_worker` role cannot read any project and cannot attribute an audit + event.** RLS is enabled on `projects` and every policy requires `app.current_user_id()`. + The worker sets no user GUC, so it matches zero rows. `audit_events_insert` requires + `actor_user_id = app.current_user_id()`, which is `NULL` for the worker, so worker actions + are anonymous. **This is a BLOCKER** (see §M). Spec 19 §5.5 anticipated it ("Worker service + accounts receive only the permissions required for their queue and artifact scope") but + `001_control_plane` never implemented a mechanism. + +### A.3 Direction change that needs ratification + +`specs/19-enterprise-control-plane.md` §2.3 and §16 M4 specify that the control plane +**imports a read-only snapshot** of UmbraAudio and that the English path keeps running in +UmbraAudio unchanged. This document **reverses that**: the enterprise database becomes +authoritative for project state, source analysis, speakers, voices, translation references, +audio runs, artifacts, lineage and audit, and UmbraAudio reads and writes it as a client. +See §M DECISION-1. Spec 19 §1's narrower prohibition — never put control-plane tables inside +the local per-project SQLite database — remains in force and is honoured. + +### A.4 Production continuity verdict + +Audiobook production can and must continue throughout. The design keeps the existing +filesystem + SQLite engine running untouched and introduces a **repository/adapter boundary** +so that "where state lives" becomes a configuration decision rather than a code rewrite. The +enterprise database is authoritative for **identity, tenancy, metadata and release**; the audio +worker holds a **bounded signed lease** and can keep rendering audio if the enterprise +database is briefly unreachable, which is the only way to satisfy "production remains +available" and "the enterprise database is authoritative" simultaneously. + +--- + +## B. Audio-production continuity assessment + +### B.1 How an ebook enters the system today + +1. `POST /api/projects` (`web/routes.py:533`) or `POST /api/upload-manuscript` + (`web/routes.py:620`) receives a `.docx/.epub/.txt/.md/.pdf` upload. +2. `permanent_path = permanent_dir / filename` (`routes.py:647`) — the raw multipart + filename is used, unsanitised. See §J.6. +3. `project_bootstrap.create_project_from_manuscript_async` → `load_config(name)` → + `config.py:165` `ensure_dirs()` creates five directories as a **side effect of resolving + the project**. +4. `detect_format` + `parse_manuscript` populate `chapters` (`source_text` holds the full + copyrighted text) and `utterances`. +5. `UPDATE project SET source_path` (`routes.py:574`, `665`). +6. With `mode=new`, the previous run is copied to `archives//` and then + `shutil.rmtree` is applied to `voices/`, `voice_tests/`, `chapter_audio/`, `analysis/` + (`routes.py:712-742`). **This is the most destructive operation in the system.** + +### B.2 How analysis creates speakers and voices + +- **Discovery** (pass 1-3, LLM via LM Studio, `analysis/character_discovery.py`) writes + `discovery/discovery_pass{1,1b,2,3}.json`, ending with `speakers.json`. +- **Profiling** writes `voice_profiles.json` (`voices/voice_profile_manager.py:753`). +- **Casting** is a **fifth** parallel store: `voice_assignments.json` + (`{character_name: "en-US-JennyNeural"}`, `scripts/assign_voices.py:161`) plus the global + `~/.umbra/voice_registry.db` (`voice_registry`, `registry_aliases`) and + `library_speakers` (`library.py:21`). +- **Voice reference WAVs** are written to `.umbra/voices/{slug}_ref.wav` + (`voices/voice_designer.py:248`), with `~/.umbra/library/voices/` as a global set, + `voice_embeddings/{slug}.npy`, and server-side `voice_prompts/{slug}.pt`. +- **Attribution** (`analysis/llm_attributor.py`) writes `utterances.speaker_name` — a + **free-text denormalised label, not a foreign key to `characters`**. + +Verified consequence: **one speaker exists in five stores under four different keys** +(`characters.id` integer, `speakers.json.primary_name`, `voice_profiles.json` name key, +`voice_registry.character_name`, `library_speakers.primary_name`). This is the single largest +identity defect in the current system and is the reason §E proposes a canonical `speakers` +table with `speaker_aliases`. + +### B.3 How audiobook jobs are queued, retried and completed + +**There is no queue and no job table.** Verified across `umbraaudio/` and `scripts/`: zero +occurrences of `lease`, `heartbeat`, `backoff`, or `exponential`. + +- Orchestration primitive: `threading.Thread(target=..., daemon=True)`, at 15 call sites + (`routes.py:827, 1925, 3139, 3372, 3474, 3657, 3893, 3950, 4039, 4179, 4253, 4565, 5019, 5109, 5380`), + plus `PipelineManager.start()` as an asyncio task (`pipeline_manager.py:121`). +- Work is a `for` loop over `SELECT ... FROM utterances WHERE chapter_id = ? ORDER BY utterance_index`. +- The only mutual exclusion is `GenerationManager._lock` (`voices/generation_manager.py:25`), + which is **re-instantiated per HTTP request** (`routes.py:475-479`), so the real guard is + the on-disk `active` boolean in `generation_status.json` — a TOCTOU check, not a mutex. + `ChapterBuilder` has **no lock at all** (`routes.py:5097-5110`). +- Retry: linear `time.sleep(5 * (attempt + 1))` on `QwenTTSUnavailableError` + (`chapter_builder.py:615`); **the batch driver `scripts/generate_all_audio.py` has no retry + loop at all** (`:202-241`). +- `umbraaudio/generation/retry.py` defines a `RetryHandler` that **nothing imports** — dead code. +- **Job identity is the output filename**, derived from `(chapter_id, utterance_index)`. + +### B.4 How artifacts are named, stored and retrieved + +| Artifact | Path pattern | Versioned? | Checksum? | +|---|---|---|---| +| Per-utterance WAV | `.umbra/chapter_audio/ch{N}/utt_{idx:04d}.wav` | No — overwritten in place | **No** | +| Staleness marker | `.../utt_{idx:04d}.wav.txt` | n/a | 16-hex truncated SHA-256 of the **voice ref** only | +| Chapter master | `.umbra/chapter_audio/ch{N}_full.wav` | No | **No** | +| Chapter MP3 | `.umbra/chapter_audio/ch{N}_full.mp3` | No | **No** | +| Full book | `.umbra/export/{m4b_meta.output_name}` e.g. `Deal_Hunter.m4b` | No | **No** | +| AR package | `.umbra/export/ar_package/{Base}_{NN}_{Title}.mp3` | No | **No** (only `size_bytes`) | +| AR manifest | `.umbra/export/ar_package.json` | Overwritten wholesale | **No** | + +**Verified: no SHA-256 or MD5 of any produced audio file is computed or stored anywhere.** Every +hash in the repository is a cache key, a voice fingerprint, a watermark bit-vector seed, or a +deterministic TTS parameter seed. Retrieval is via a `StaticFiles` mount plus 12 +`FileResponse` endpoints, of which **zero perform a user authorization check**. + +### B.5 How the API and UI identify a project + +A single browser cookie, `active_project`, set at `routes.py:529` and `routes.py:616` with +**only `max_age=86400*365`** — no `httponly`, no `samesite`, no `secure`. The complete gate for +all 129 project-scoped routes is: + +```python +# web/routes.py:482-491 +def _get_active_config(request: Request): + active_slug = request.cookies.get("active_project") + if not active_slug: + raise NoActiveProjectError("No active project selected.") + config = load_config(active_slug) + if not (config.data_dir / "project.db").exists(): + raise NoActiveProjectError(f"Project '{active_slug}' no longer exists.") + return config +``` + +The cookie is **never** checked against `list_projects()` or any allow-list. `load_config` +accepts **absolute paths verbatim** (`config.py:156-158`) with no containment check. And the +`if not cfg.data_dir.exists(): return 404` guard at `routes.py:525` is **unreachable dead +code**, because `load_config` already created the directory at `config.py:165`. + +`list_projects()` (`config.py:20-47`) is a **directory scan of the entire repository root** — +the same directory holding `umbraaudio/`, `scripts/`, `tests/`, `models/`, `.venv/` — and it +opens a SQLite connection to every candidate on every page render. + +### B.6 Operations that must continue during migration + +Uninterrupted: upload/parse, discovery, profiling, voice-reference generation, LLM +attribution, speaker/alias/merge editing, name-token resolution, pronunciation editing, +utterance audio generation, chapter build + stitch, full-book generation, m4b and AR MP3 +export, QA checks. + +Explicitly **not** a prerequisite: the translation/review UI. The audio worker must reach a +usable state with the UI absent or offline. + +### B.7 Authority classification of local data + +| Data | Class | Note | +|---|---|---| +| `chapters.source_text`, `utterances.text` | **Authoritative** | Full copyrighted manuscript text | +| `speakers.json`, `voice_profiles.json`, `voice_assignments.json`, `characters` | **Authoritative, 4-way conflict** | Must be reconciled in Phase P3 | +| `voices/*_ref.wav`, `~/.umbra/library/voices/`, `voice_embeddings/*.npy`, `voice_prompts/*.pt` | **Authoritative + biometric** | See §C.2 | +| `*.wav`, `*.mp3`, `*.m4b` | **Authoritative artifacts, unversioned** | No checksums | +| `*.wav.txt` markers | **Authoritative idempotency state** | Must be preserved verbatim across cutover | +| `manifests/ch{N}.json` | **Derived** from SQLite | | +| `pipeline_status.json`, `pipeline_checkpoint.json`, `analysis_status.json`, `ar_build_status.json`, `scan_results.json` | **Cache/progress** | 4 of 5 written non-atomically | +| `generation_status.json` | **Cache/progress, atomic** | The one correct pattern (`generation_manager.py:52-55`) | +| `ch{N}_build_status.json` | **Cache/progress, non-atomic** | Rewritten ~2× per utterance (`chapter_builder.py:503`) | +| `defects.json` (1.3 MB), `debug_*.json` | **Derived/debug** | Contains LLM reasoning | +| `discovery_pass*.json` | **Temporary** | Inputs to `speakers.json` | +| `cache/cache_index.json`, `voice_analysis.json` | **Cache** | | + +--- + +## C. Current-state persistence and identity inventory + +### C.1 Legacy objects (full field detail in §F rationale; summary here) + +| # | Logical entity | Storage | Owner | Primary identifier | Lifecycle | Class | Sensitivity | +|---|---|---|---|---|---|---|---| +| 1 | Project | `//` + SQLite `project` | audio app | filesystem dir name (slug); `project.id` INTEGER | `initialized` | Authoritative | Copyrighted | +| 2 | Source manuscript | `uploads/`, `last_upload/`, `project.source_path` | audio app | filesystem path | replaced in place | Authoritative | **Copyrighted + licensed** | +| 3 | Chapter | SQLite `chapters` | audio app | `chapters.id` INTEGER (per-DB) | `parsed` | Authoritative | **Copyrighted** | +| 4 | Utterance | SQLite `utterances` (26 cols) | audio app | `utterances.id` INTEGER; **no UNIQUE on `(chapter_id, utterance_index)`** | `pending` | Authoritative | **Copyrighted derivative** | +| 5 | Speaker | 5 stores (see §B.2) | audio app | 4 different keys | `identified` / `unidentified` | Authoritative, conflicting | Personal (voice) | +| 6 | Voice review | SQLite `voice_review_history` (**no `project_id`**) | audio app | `id` INTEGER | append | Authoritative | Personal | +| 7 | Pronunciation term | 3 stores (`pronunciation_guide`, `phonetic_names.json`, `library_names`) | audio app | term TEXT; UNIQUE `(project_id, term)` on one only | `confirmed` | Authoritative | Low | +| 8 | Generated audio | SQLite `generated_audio` + filesystem | audio app | `id` INTEGER | `generated` | Table **not written by the pipeline**; filesystem is truth | — | +| 9 | Quality report | SQLite `quality_reports` | audio app | `id` INTEGER | — | **Table exists, no row is ever written** (`database.py:549` is the only caller) | — | +| 10 | Review decision | SQLite `review_decisions` (polymorphic `entity_type`/`entity_id`, no FK) | audio app | `id` INTEGER | — | Authoritative | Low | +| 11 | Attribution exception | SQLite `exception_reviews` (**web-only table**, absent from `database.py`) | audio web | `id` INTEGER | `pending`/`resolved` | Authoritative | Low | +| 12 | Orphan name token | SQLite `name_tokens`; UNIQUE `(project_id, token)` | audio app | `token` TEXT | `unresolved` | Authoritative | Low | +| 13 | Voice reference | `voices/{slug}_ref.wav`, `~/.umbra/library/voices/` | audio app | filename slug | exists/absent | Authoritative | **Biometric + consent** | +| 14 | Voice embedding | `voice_embeddings/{slug}.npy` | audio app | filename slug | — | Cache | **Biometric** | +| 15 | Qwen voice prompt | `voice_prompts/{slug}.pt` (server-side) | Qwen TTS server | slug | — | Cache | **Biometric** | +| 16 | Pipeline checkpoint | `pipeline_status.json`, `pipeline_checkpoint.json` | audio app | none | step state | Cache | Low | +| 17 | Generation run | `generation_status.json`, `ch{N}_build_status.json` | audio app | none | `active` bool / phase | Cache | Low | +| 18 | Artifact manifest | `export/ar_package.json` | audio app | — | rebuilt wholesale | Authoritative for shipped package | Low | +| 19 | Job | **DOES NOT EXIST** | — | — | — | — | — | +| 20 | User / session / role | **DOES NOT EXIST** | — | — | — | — | — | + +### C.2 Data classification — the two sensitive classes + +**Class R (Restricted — biometric + consent).** `voices/*_ref.wav`, +`voice_embeddings/*.npy`, `voice_prompts/*.pt`, `~/.umbra/library/voices/`. These encode a +human or cloned vocal identity. Where a real person's voice was cloned, EU/UK AI and biometric +rules and the provider's own consent terms apply. **No retention period, no consent-evidence +record, no territory or media restriction, and no takedown mechanism exists today.** These must +never appear in PostgreSQL; only keys, checksums and rights metadata do. + +**Class C (Confidential — copyrighted).** `chapters.source_text`, `utterances.text`, +`utterances.context_before/after`, `uploads/`, all produced audio and exports, `defects.json`. +Underlying the author's licence. Translation derivatives inherit this class and must not +outlive the source entitlement. + +**Class P (Personal).** `users.email`, `sessions.ip_address`, `sessions.user_agent`, +`audit_events.ip_address`, plus the legacy `'user'` actor strings. + +### C.3 Verified schema divergence (a migration hazard in its own right) + +There are **four independent DDL authorities** for the same project database: + +1. `umbraaudio/database.py:9-166` — 10 tables (`SCHEMA`). +2. `umbraaudio/web/routes.py:102-275` — 11 tables (`SCHEMA_SQL`); adds `exception_reviews` + + 3 indexes + 3 `qa_*` columns on `utterances`. +3. `umbraaudio/analysis/name_tokens.py:504-520` — `name_tokens` only, **no indexes**. +4. `scripts/reattribute_all.py:30-42` — `ALTER TABLE` only. + +Both live project databases have been built by `ALTER TABLE` under older schemas, so column +*ordering* differs from both schema files; `DealHunterShip` lacks `name_tokens` entirely; and +**neither database has `PRAGMA foreign_keys=ON`**, so every `REFERENCES` clause in the legacy +schema is declarative only. **There are zero `CHECK` constraints in either legacy schema file.** + +--- + +## D. Enterprise identity, session, and project-access contract + +### D.1 Canonical model + +The enterprise `users`, `organizations`, `projects`, `roles`, `permissions` and `user_roles` +records **are** the canonical account and project model. No second password store is created +(verified: `users` has no password column, and `users_auth_subject_idx` is a partial unique +index on `(auth_provider, auth_subject)`). No second project namespace is created. + +**Membership is not a new table.** `user_roles` with `scope='project'` **is** the membership +record. "Active membership" ≡ `user_roles.scope='project' AND user_roles.status='active' AND +users.status='active'`. This is deliberate: a separate `project_members` table would create two +sources of truth for the same fact. I **reject** the `project_members` and +`project_access_grants` models from spec 19 §8.1 for phase one. + +### D.2 Login flow + +1. Unauthenticated request → `302` to the enterprise IdP (OIDC authorization-code + PKCE). +2. IdP callback → the API **verifies the ID token signature, issuer, audience, `nonce`, + `exp` and `azp`** server-side, and maps `iss` + `sub` to `users.auth_provider` + + `users.auth_subject`. A `users` row is created on first login only if a matching + `invitations` row exists and is unexpired, unrevoked and unaccepted. +3. If `users.mfa_required` is true, the IdP must assert an MFAacr/amr value satisfying the + policy; the API records `sessions.mfa_verified_at`. **Today no such column exists (§M B4).** +4. The API issues a session: 32 random bytes, stored **only as SHA-256** in + `sessions.token_hash`; the plaintext goes to the browser in an `HttpOnly; Secure; + SameSite=Lax` cookie. +5. `sessions.ip_address` and `user_agent` recorded. +6. Bootstrap path (first run only): `bootstrap_tokens.token_hash` → set password/MFA at the + IdP → `consumed_at`. The bootstrap token is single-use and 24 h (bootstrap script:301-302). + +The **single authentication entry point** is one `SECURITY DEFINER` function, +`app.resolve_session(p_token_hash text)`, returning the user id **only if** all of: hash match, +`revoked_at IS NULL`, `expires_at > now()`, `users.status='active'`, `organizations.status` is +`active` for the requested org, and MFA satisfied when required. Every other code path +authorises from the returned principal, never from a GUC or a request value. + +### D.3 Session lifetime and revocation + +- Idle timeout 8 h, absolute 7 days, both stored in `sessions.expires_at` (absolute) plus an + application-side idle clock. +- Logout: `UPDATE sessions SET revoked_at = now()` for the presented token. +- "Sign out everywhere": **requires new capability** — `sessions_self` is the only policy + (`001:526-529`) and is `USING (user_id = app.current_user_id())`, so a user cannot revoke + their own other sessions and an admin cannot revoke anyone's. Fix: `app.revoke_user_sessions(uuid)` + as `SECURITY DEFINER`, executable by the owning user and by an org admin, writing + `audit_events`. +- **Suspension is enforced in `app.resolve_session`, not by RLS**, because no RLS policy reads + `users.status`. A suspended user's existing session stops resolving on the next request; the + row remains for forensics. This is the accepted design, with the caveat recorded in §M B5. + +### D.4 Project selection + +Selection is **server-derived, never client-declared**. The request path carries an opaque +`project_id` (uuid). The API resolves `organization_id` from `projects` and validates +`app.can_access_project(project_id)` **on every request** before any domain read. The browser +may hold a *selected project* hint; it is a signed, server-issued value and is **advisory only** +— a stale or forged value cannot widen access, because access is recomputed from the +authorised-project set each time. The existing unsigned `active_project` cookie is **retired**. + +### D.5 Authorisation checks by action + +Every action maps to a `permissions.key`. Existing keys are listed in `001:378-391`; the keys +**missing** and required by the prompt are enumerated in §H as ADD TABLE (data) / BACKFILL. + +| Action | Required permission | Role(s) granted | Exists in 001? | +|---|---|---|---| +| View project metadata | `project.access.view` (via membership) | any project role | **No** — implicit only | +| Read manuscripts/utterances | `project.source.read` | all project roles | **No** | +| Edit speakers/aliases/merges | `project.speakers.edit` | `project_manager`, `audio_director` | **No** — **destructive today, unpermissioned** | +| Run analysis / attribution | `project.analysis.run` | `project_manager`, `audio_director` | **No** | +| Review translations | `project.translation.review` | `proofreader` | Yes | +| Edit translations | `project.translation.edit` | `translator`, `project_manager` | Yes | +| **Generate audio** | `project.audio.generate` | `project_manager`, `audio_director` | **No** — **no generate permission exists** | +| Approve audio | `project.audio.approve` | `audio_director`, `project_manager` | Yes | +| Cast voices | `project.voice.cast` | `audio_director` | **No** | +| Manage voice library/licences | `voice.catalog.manage` | `voice_manager` | Yes (but see §J.3 — cannot see a project) | +| Download approved artifact | `artifact.download` | `member`, `viewer` | Yes | +| Download **unapproved/unreleased** | `artifact.download.unreleased` | `audio_director`, `release_manager` | **No** — and today *all* downloads are ungated | +| Approve release | `project.release.approve` | `release_manager`, `project_owner` | Yes | +| Manage project access | `project.access.manage` | `project_owner`, `organization_admin` | Yes | +| Delete/reset project state | `project.state.destroy` | `project_owner` | **No** — `rmtree` is unpermissioned today | +| Manage own sessions/MFA | `user.self.manage` | self | **No** | + +### D.6 Suspended / deactivated behaviour + +`users.status` ∈ `invited | active | suspended | deactivated | locked`. In every non-`active` +state: existing sessions stop resolving, new logins are refused, background jobs owned by that +user are cancelled, and **historical attribution and audit rows are preserved** — suspension +never rewrites history. `organizations.status` ∈ `active | suspended | archived` gates all +project access for members of that org. + +### D.7 Service-to-service authentication for the audio worker + +The worker is **never** a `users` row and **never** impersonates one. It authenticates as the +`umbracontrol_worker` PostgreSQL role, and presents a **service identity** (`service_identities`, +§E) plus a **lease** (`job_leases`). RLS policy branches for the worker are keyed on +`app.current_service_identity_id()` **and** the existence of a live lease joining that service +identity to the row's `project_id`. A worker can therefore read and write exactly the projects +it is currently engaged on, and nothing else. A worker with no lease sees zero rows. This +satisfies "a worker maps to a project without impersonating a user" and "a worker cannot read a +project outside its lease" (spec 19 §5.5). + +### D.8 Account and project segmentation rules + +1. `organization_id` is the tenant boundary; present on every tenant-owned row. +2. `project_id` scopes work; a user reaches a project only through an active `user_roles` + project-scope row, or org-admin, or platform-admin. +3. A user in multiple organizations holds **separate** `user_roles` rows per org; there is no + cross-org inheritance and no "default org" that grants implicit access. +4. Sessions are **user-global**, not org-scoped. A session token is therefore valid for every + org that user belongs to — correct for a human, and the reason project/org context must be + re-derived and re-authorised on **every** request rather than cached in the session. +5. Voice assets, licences and entitlements are **org-scoped** and reusable across that org's + projects only, and only when language, accent, consent, territory, media and term allow. + +--- + +## E. Proposed canonical entity model + +Accept / reject decisions on the models the prompt asked me to evaluate. + +| # | Proposed model | Decision | Rationale | +|---|---|---|---| +| 1 | **Source books / editions / source documents** → `works` + `source_documents` | **ACCEPT** | A public-domain edition and an author manuscript are different assets of one bibliographic work (spec 19 §6.1). UmbraAudio's `project.source_path` becomes a `source_documents` row. | +| 2 | **Immutable analysis runs + snapshots** → `analysis_runs` + `analysis_snapshots` | **ACCEPT** | Mandatory for invariant "analysis has immutable versions". Snapshots are content-addressed, never updated. | +| 3 | **Speakers / characters** → `speakers` + `speaker_aliases` | **ACCEPT** | Resolves the 5-store/4-key defect (§B.2). One row per logical speaking entity per project. | +| 4 | **Canonical voice profiles** → `voice_profiles` + `voice_profile_revisions` | **ACCEPT** | Spec 19 §11.1: logical identity ≠ provider asset ≠ profile. | +| 5 | **Voice casting and reuse** → `voice_casts` | **ACCEPT** | Casting is a fact about (project, locale, speaker, revision) and must be snapshot-able. | +| 6 | **Voice licensing / consent / usage** → `voice_licenses` | **ACCEPT (phase 1: 1 table)** | Rights are the gating fact for reuse. `voice_entitlements`, `voice_reservations`, `voice_rate_cards`, `voice_invoices` are **DEFERRED** to a later phase — no commercial billing is in scope for this transition. | +| 7 | **Translation runs + language targets** → `project_locales` + `translation_runs` | **ACCEPT** | `project_locales` is the locale lane; **no** `translations_de`-style tables. | +| 8 | **Translation segments / units** → `translation_units` | **ACCEPT** | Grain = one ordered target render span. | +| 9 | **Audio generation runs + voice-cast snapshot** → `audio_runs` + `audio_run_casts` | **ACCEPT** | The cast snapshot must be frozen per run so a release can be reproduced after casting changes. | +| 10 | **Generated ebook / audiobook artifacts** → `artifacts` | **ACCEPT** | Metadata only; bytes in object storage. | +| 11 | **Pipeline jobs, retries, idempotency** → `jobs` + `job_attempts` + `job_leases` | **ACCEPT** | Closes the "no queue, filename is identity" defect. | +| 12 | **Provenance, lineage, checksums, audit** → `artifact_links` + reuse `audit_events` | **ACCEPT** | `audit_events` already exists and is append-only by trigger (`001:356-359`). | +| 13 | `project_members` (separate table) | **REJECT** | Membership is `user_roles(scope='project')`. A second table is a second source of truth. | +| 14 | `project_access_grants` (spec 19 §8.1) | **REJECT for phase 1** | No non-user principals exist yet. Revisit when a translation-vendor principal type is needed. | +| 15 | `works` ↔ `projects` 1:1 | **REJECT** — use 1:N | A work may be produced in several projects (re-translation, new edition). `projects.work_id` FK. | +| 16 | `semantic_packages`, `identity_registries`, `glossary_revisions`, `memory_revisions`, `alignment_links`, `target_render_spans` (spec 19 §8.2) | **DEFER** | These are UmbraTrans translation-design objects. They are required for the translation UI, not for the audio system to keep producing books. Adding them now would exceed the continuity objective. Listed in §M as a phase-2 scope item. | +| 17 | Storing WAV/MP3/M4B/EPUB bytes in PostgreSQL | **REJECT — hard** | Object storage only (§G invariant 8). | +| 18 | Storing voice reference audio or embeddings in PostgreSQL | **REJECT — hard** | Class R biometric data; object storage with rights metadata. | + +### E.1 The stable utterance identity (the load-bearing new concept) + +Because artifacts are named by a mutable index, the enterprise model introduces a +**content-addressed unit key** and records it **at generation time**, without changing any +on-disk filename: + +``` +source_unit_key = sha256( + project_id ‖ source_revision_id ‖ + chapter_ordinal ‖ occurrence_index ‖ + text_sha256 +) +``` + +`source_units` is keyed by `source_unit_key`. `artifacts` reference `(source_revision_id, +unit_ordinal, text_sha256)` **and** record the legacy path in `legacy_object_key`. Consequences: + +- A re-segmentation is **detectable**: the same ordinal now has a different `text_sha256`, so + the enterprise record and the on-disk file diverge visibly instead of silently. +- The existing `*.wav.txt` marker logic (`generation/staleness.py`) keeps working unchanged, + so continuity is preserved and the audio path is not blocked on this migration. +- UmbraTrans can reference English units with a durable key, satisfying spec 18 §3.3, which + forbids using a mutable `utterance_index` or a database row id as a cross-language ID. + +--- + +## F. Table-by-table contract + +Common conventions for every table below: + +- **PK**: `id uuid PRIMARY KEY DEFAULT gen_random_uuid()` (v4, matching `001` exactly — no + version dependency on the server's PostgreSQL version). +- **Tenancy**: `organization_id uuid NOT NULL`, `project_id uuid NOT NULL` (nullable only for + org-scoped tables), enforced by + `FOREIGN KEY (project_id, organization_id) REFERENCES projects(id, organization_id)`. + This composite FK is possible because `001:51` already declares `UNIQUE (id, organization_id)` + on `projects` — the schema anticipated this pattern. +- **Concurrency**: `row_version integer NOT NULL DEFAULT 1`, incremented by trigger, with + optimistic `UPDATE ... WHERE id = ? AND row_version = ?`. Job claiming additionally uses + `FOR UPDATE SKIP LOCKED`. +- **RLS**: `ENABLE ROW LEVEL SECURITY` (never `FORCE` on the mutually-referencing tables — see + §J.2), plus a `tenant_read` / `tenant_write` policy pair using the composite FK columns. +- **Idempotency**: natural unique constraints stated per table; all writes are `INSERT … + ON CONFLICT` or guarded by a uniqueness violation, never blind overwrites. +- **New / extension / not-in-PostgreSQL** stated per table. + +--- + +### F.1 `works` +- **Purpose:** bibliographic identity of a book, independent of any production project. +- **Grain:** one row per distinct work. +- **Authoritative writer:** ingestion service (human via API). +- **Columns:** `id uuid PK`; `organization_id uuid NOT NULL`; `title text NOT NULL`; + `author text`; `series text`; `isbn13 text`; `source_language text`; `rights_status text + NOT NULL DEFAULT 'unknown'`; `rights_evidence_uri text`; `created_at`, `updated_at timestamptz + NOT NULL DEFAULT now()`. +- **FKs:** `organization_id → organizations(id) ON DELETE RESTRICT`. +- **Unique:** `UNIQUE (organization_id, isbn13)` where `isbn13 IS NOT NULL`. +- **Indexes:** `works_org_idx (organization_id)`; `works_title_trgm` (pg_trgm) later. +- **State machine:** `active → archived`. No separate status column needed in phase 1; reuse + `archived_at timestamptz NULL`. +- **RLS:** org-scoped read for members; write requires `organization.projects.manage`. +- **Classification:** C (title/author are bibliographic, not licensed text). +- **Idempotency/concurrency:** `UNIQUE (organization_id, isbn13)`; `row_version`. +- **Verdict:** NEW. + +### F.2 `source_documents` +- **Purpose:** an immutable ingested file (DOCX/EPUB/TXT/PDF) of a work. +- **Grain:** one row per distinct ingested file version. +- **Authoritative writer:** ingestion service. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `work_id uuid NOT NULL + → works(id) ON DELETE RESTRICT`; `media_type text NOT NULL`; `byte_size bigint NOT NULL + CHECK (byte_size > 0)`; `sha256 char(64) NOT NULL`; `object_key text NOT NULL`; `original + _filename text`; `ingested_by_user_id uuid → users(id) ON DELETE SET NULL`; `ingested_at + timestamptz NOT NULL DEFAULT now()`; `superseded_at timestamptz`; `row_version integer NOT + NULL DEFAULT 1`. +- **Unique:** `UNIQUE (project_id, sha256)` — re-ingesting the same bytes is a no-op. +- **Indexes:** `source_documents_project_idx (project_id, ingested_at DESC)`; + `source_documents_work_idx (work_id)`. +- **State machine:** `active → superseded → withdrawn`. +- **RLS:** project tenant policy; `WITH CHECK` also requires the actor to hold + `project.source.edit`. +- **Classification:** C. +- **Idempotency:** content-addressed by `sha256`; re-upload returns the existing row. +- **Verdict:** NEW. Backfilled from `project.source_path` and `uploads/`. + +### F.3 `source_revisions` +- **Purpose:** a frozen, numbered revision of the source text — the anchor for all analysis, + translation and audio lineage. +- **Grain:** one row per frozen source-text revision. +- **Authoritative writer:** source-editor workflow. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `work_id uuid NOT NULL`; + `source_document_id uuid NOT NULL → source_documents(id) ON DELETE RESTRICT`; `revision_number + integer NOT NULL`; `label text`; `sha256 char(64) NOT NULL`; `object_key text NOT NULL`; + `word_count integer`; `locked boolean NOT NULL DEFAULT false`; `locked_by_user_id uuid`; + `locked_at timestamptz`; `approved_by_user_id uuid`; `approved_at timestamptz`; `created_at + timestamptz NOT NULL DEFAULT now()`; `row_version integer NOT NULL DEFAULT 1`. +- **FKs:** `locked_by_user_id`/`approved_by_user_id → users(id) ON DELETE SET NULL` + (history must survive account deletion). +- **Unique:** `UNIQUE (project_id, revision_number)`; `UNIQUE (id, project_id, + organization_id)` to enable child composite FKs. +- **Indexes:** `source_revisions_project_idx (project_id, revision_number DESC)`. +- **State machine:** `draft → locked → approved` (immutable after `locked`; enforced by a + `BEFORE UPDATE` trigger raising when `OLD.locked`), or `draft → abandoned`. +- **RLS:** tenant policy; write requires `project.source.edit`. +- **Classification:** C. +- **Idempotency:** `UNIQUE (project_id, revision_number)`; content `sha256` recorded. +- **Verdict:** NEW. + +### F.4 `chapters` (enterprise) +- **Purpose:** chapter boundaries of a source revision. +- **Grain:** one row per chapter per source revision. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `source_revision_id uuid + NOT NULL`; `ordinal integer NOT NULL`; `title text`; `text_sha256 char(64) NOT NULL`; + `word_count integer NOT NULL DEFAULT 0`; `created_at`; `row_version`. +- **Unique:** `UNIQUE (source_revision_id, ordinal)`. +- **Indexes:** `chapters_project_idx (project_id, ordinal)`. +- **State machine:** none; immutable with its source revision. +- **RLS:** tenant read; write requires `project.source.edit`. +- **Idempotency:** `ON CONFLICT (source_revision_id, ordinal) DO UPDATE`. +- **Verdict:** NEW (distinct from legacy SQLite `chapters`; name is the same by intent, but + the enterprise table is uuid-keyed and revision-scoped). **To avoid confusion the enterprise + table is named `source_chapters`.** + +### F.5 `source_units` +- **Purpose:** the atomic addressable source element — the durable unit that audio clips, + translation units and review items all reference. +- **Grain:** one row per addressable source text unit within a source revision. +- **Authoritative writer:** analysis/segmentation pipeline. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `source_revision_id uuid + NOT NULL`; `source_chapter_id uuid NOT NULL`; `ordinal integer NOT NULL`; `occurrence + integer NOT NULL DEFAULT 0`; `text text NOT NULL`; `text_sha256 char(64) NOT NULL`; + `unit_kind text NOT NULL DEFAULT 'utterance' CHECK (unit_kind IN ('narration','utterance', + 'heading','scene_break','footnote'))`; `speaker_name text`; `speaker_id uuid → speakers(id) + ON DELETE SET NULL`; `attribution_confidence numeric(4,3)`; `attribution_evidence_kind text`; + `attribution_tag_name text`; `attribution_uncertain boolean NOT NULL DEFAULT false`; + `needs_review boolean NOT NULL DEFAULT false`; `review_reason text`; `user_edited boolean NOT + NULL DEFAULT false`; `created_at`; `row_version`. +- **Unique:** `UNIQUE (source_revision_id, ordinal, occurrence)`; **plus a generated + `source_unit_key char(64)`** = `sha256(source_revision_id || ordinal || occurrence || + text_sha256)`, `UNIQUE`. See §E.1. +- **Indexes:** `source_units_chapter_idx (source_chapter_id, ordinal)`; + `source_units_speaker_idx (project_id, speaker_id)`; + `source_units_review_idx (project_id, needs_review) WHERE needs_review`; + `source_units_sha_idx (source_revision_id, text_sha256)`. +- **State machine:** `proposed → attributed → reviewed`; `needs_review` is an overlay flag, not + a state. +- **RLS:** tenant read; write requires `project.analysis.run` or `project.speakers.edit`. +- **Classification:** C. +- **Idempotency:** upsert on `(source_revision_id, ordinal, occurrence)`; text change bumps + `row_version` and flips `needs_review`. +- **Verdict:** NEW. This is the enterprise replacement for legacy SQLite `utterances`. + +### F.6 `speakers` +- **Purpose:** the logical speaking entity — the stable cross-language character identity. +- **Grain:** one row per logical speaker per project. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `canonical_name text NOT + NULL`; `kind text NOT NULL DEFAULT 'character' CHECK (kind IN ('character','narrator', + 'group','system','unknown'))`; `gender text`; `age_range text`; `description text`; + `communication_style text`; `is_group boolean NOT NULL DEFAULT false`; + `is_unidentified boolean NOT NULL DEFAULT false`; `unidentified_reason text`; `designation + text NOT NULL DEFAULT 'needs_review' CHECK (designation IN ('clear','needs_review'))`; `sample + _dialogue text`; `source text NOT NULL DEFAULT 'analysis'` CHECK (source IN ('analysis', + 'manual','imported','legacy')); `created_at`; `updated_at`; `row_version integer NOT NULL + DEFAULT 1`. +- **Unique:** `UNIQUE (project_id, canonical_name)`. +- **Indexes:** `speakers_project_idx (project_id, designation)`; + `speakers_name_trgm (project_id, canonical_name gin_trgm_ops)` (pg_trgm, phase 2). +- **State machine:** `discovered → profiled → confirmed`; merge is a `superseded_by` pointer, + never a delete. +- **RLS:** tenant read; write requires `project.speakers.edit`. +- **Classification:** P (name + gender + description). +- **Idempotency:** `ON CONFLICT (project_id, canonical_name) DO UPDATE`; merges are + `UPDATE speakers SET superseded_by = target` + alias fold, both in one transaction. +- **Verdict:** NEW. Replaces `characters` + `speakers.json` + `voice_profiles.json` as the + single speaker authority. + +### F.7 `speaker_aliases` +- **Purpose:** alternate surface forms and the evidence for each, preserving today's + `aliases` / `alias_evidence` model. +- **Grain:** one row per (speaker, alias form). +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `speaker_id uuid NOT NULL + → speakers(id) ON DELETE CASCADE`; `alias citext NOT NULL`; `evidence jsonb NOT NULL DEFAULT + '{}'`; `confidence numeric(4,3)`; `is_confirmed boolean NOT NULL DEFAULT false`; + `created_at`; `updated_at`. +- **Unique:** `UNIQUE (project_id, alias)`. +- **Indexes:** `speaker_aliases_alias_idx (project_id, alias)`; + `speaker_aliases_speaker_idx (speaker_id)`. +- **State machine:** none; `is_confirmed` is a flag. +- **RLS:** tenant; write requires `project.speakers.edit`. +- **Idempotency:** `ON CONFLICT (project_id, alias) DO NOTHING` then merge evidence. +- **Verdict:** NEW. + +### F.8 `analysis_runs` +- **Purpose:** an immutable record of one analysis execution (which model, which prompt + version, which inputs, which parameters). +- **Grain:** one row per analysis execution. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `source_revision_id uuid + NOT NULL`; `run_kind text NOT NULL` CHECK (`discovery`, `segmentation`, `attribution`, + `emotion`, `voice_analysis`); `engine text NOT NULL`; `model_id text NOT NULL`; + `prompt_version text NOT NULL`; `input_sha256 char(64) NOT NULL`; `params jsonb NOT NULL + DEFAULT '{}'`; `status text NOT NULL DEFAULT 'running' CHECK (status IN ('running', + 'succeeded','failed','cancelled','superseded'))`; `started_by_service_id uuid`; `started_at + timestamptz NOT NULL DEFAULT now()`; `finished_at timestamptz`; `metrics jsonb NOT NULL + DEFAULT '{}'`; `error text`; `superseded_by_run_id uuid → analysis_runs(id) ON DELETE SET + NULL`. +- **Unique:** `UNIQUE (project_id, run_kind, source_revision_id, input_sha256, prompt_version, + model_id)` — re-running an identical analysis is a no-op, which is the idempotency + requirement. +- **Indexes:** `analysis_runs_project_idx (project_id, run_kind, started_at DESC)`. +- **State machine:** `running → succeeded | failed | cancelled`; `succeeded → superseded`. +- **RLS:** tenant read; write: worker lease-scoped, human `project.analysis.run`. +- **Classification:** C (params may embed text). +- **Idempotency/concurrency:** the wide UNIQUE; `FOR UPDATE` on transition. +- **Verdict:** NEW. + +### F.9 `analysis_snapshots` +- **Purpose:** the immutable result payload of an analysis run, content-addressed. +- **Grain:** one row per produced snapshot. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `analysis_run_id uuid NOT + NULL → analysis_runs(id) ON DELETE RESTRICT`; `snapshot_kind text NOT NULL`; + `sha256 char(64) NOT NULL`; `object_key text`; `inline_payload jsonb`; `byte_size bigint`; + `created_at`. +- **Unique:** `UNIQUE (project_id, snapshot_kind, sha256)`; `UNIQUE (analysis_run_id, + snapshot_kind)`. +- **Indexes:** `analysis_snapshots_run_idx (analysis_run_id)`. +- **State machine:** immutable, no updates. +- **RLS:** tenant read; write only by the producing worker under lease. +- **Classification:** C. +- **Idempotency:** content-addressed. +- **Verdict:** NEW. Satisfies "analysis has immutable versions". + +### F.10 `voice_profiles` +- **Purpose:** a reusable voice identity profile (spec 19 §11.1) — the *logical* reusable + character/performance identity, not a provider asset. +- **Grain:** one row per voice profile per organization. +- **Columns:** `id uuid PK`; `organization_id uuid NOT NULL`; `project_id uuid NULL` (null = + org-shared); `name text NOT NULL`; `locale_tag text`; `gender text`; `accent text`; + `pitch_register`, `pace_baseline`, `energy`, `timbre`, `texture`, `formality`, `emotional + _baseline`, `accent_marker`, `distinguishing_trait` (all `text`); `voice_description text`; + `reference_object_key text`; `reference_sha256 char(64)`; `engine text NOT NULL DEFAULT + 'qwen3-tts'`; `engine_voice_id text`; `is_narrator boolean NOT NULL DEFAULT false`; + `status text NOT NULL DEFAULT 'draft' CHECK (status IN ('draft','in_review','approved', + 'rejected','retired'))`; `created_at`; `updated_at`; `row_version integer NOT NULL DEFAULT 1`. +- **Unique:** `UNIQUE (organization_id, name)`. +- **Indexes:** `voice_profiles_org_idx (organization_id, status)`. +- **State machine:** `draft → in_review → approved | rejected`; `approved → retired`. +- **RLS:** org policy — any org member may read metadata; `reference_object_key` access + requires `voice.catalog.manage` or an active cast. +- **Classification:** P; the reference **audio** is Class R and never in PostgreSQL. +- **Idempotency:** upsert on `(organization_id, name)`; profile *changes* create + `voice_profile_revisions`. +- **Verdict:** NEW. Replaces `voice_registry` + `library_speakers` + `voice_profiles.json`. + +### F.11 `voice_profile_revisions` +- **Purpose:** immutable history of a voice profile so a past render stays reproducible. +- **Grain:** one row per profile change. +- **Columns:** `id uuid PK`; `organization_id uuid NOT NULL`; `voice_profile_id uuid NOT NULL → + voice_profiles(id) ON DELETE CASCADE`; `revision_number integer NOT NULL`; `payload jsonb NOT + NULL`; `sha256 char(64) NOT NULL`; `changed_by_user_id uuid`; `change_reason text`; + `created_at timestamptz NOT NULL DEFAULT now()`. +- **Unique:** `UNIQUE (voice_profile_id, revision_number)`. +- **Indexes:** `voice_profile_revisions_profile_idx (voice_profile_id, revision_number DESC)`. +- **State machine:** immutable. +- **RLS:** org read; write requires `voice.catalog.manage`. +- **Idempotency:** `UNIQUE (voice_profile_id, revision_number)`; `UNIQUE (voice_profile_id, + sha256)`. +- **Verdict:** NEW. + +### F.12 `voice_licenses` +- **Purpose:** the rights record that gates reuse of a voice asset across projects, locales, + media and time. +- **Grain:** one row per licence grant for a voice profile. +- **Columns:** `id uuid PK`; `organization_id uuid NOT NULL`; `voice_profile_id uuid NOT NULL → + voice_profiles(id) ON DELETE RESTRICT`; `license_type text NOT NULL`; `licensor text`; + `territories text[] NOT NULL DEFAULT '{}'`; `media text[] NOT NULL DEFAULT '{}'`; `languages + text[] NOT NULL DEFAULT '{}'`; `exclusive boolean NOT NULL DEFAULT false`; `term_start date + NOT NULL`; `term_end date`; `max_concurrency integer`; `derivative_rights boolean NOT NULL + DEFAULT false`; `consent_evidence_uri text`; `consent_recorded_at timestamptz`; `status text + NOT NULL DEFAULT 'pending' CHECK (status IN ('pending','active','expired','revoked', + 'takedown'))`; `revoked_at`, `revoked_reason text`; `created_at`, `updated_at`; `row_version`. +- **Unique:** `UNIQUE (voice_profile_id, term_start)`. +- **Indexes:** `voice_licenses_profile_idx (voice_profile_id, status)`; + `voice_licenses_expiry_idx (organization_id, term_end) WHERE status = 'active'` — supports the + "assets expiring before release" dashboard. +- **State machine:** `pending → active → expired | revoked | takedown`. +- **RLS:** org read for `voice_manager`; write requires `voice.catalog.manage`. A cast may + **read** only `status='active'` rows covering the requested locale/territory/media/date — + enforced by `app.voice_licence_covers(voice_profile_id, locale, territory, media, at_date)`. +- **Classification:** P + legal. `consent_evidence_uri` points to object storage, never inline. +- **Idempotency:** upsert on `(voice_profile_id, term_start)`. +- **Verdict:** NEW. Backfilled as `status='pending'` with `rights_status='unknown'` — UmbraAudio + holds no licence evidence today, so every legacy voice is **non-reusable for paid/public + release** until a voice manager approves it. This is a deliberate, conservative default. + +### F.13 `voice_casts` +- **Purpose:** assignment of a speaker to a voice profile for a project/locale/revision, and + the snapshot input to an audio run. +- **Grain:** one row per (project, locale, speaker, source revision, revision_number). +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `project_locale_id uuid`; + `speaker_id uuid NOT NULL → speakers(id) ON DELETE RESTRICT`; `voice_profile_id uuid NOT NULL + → voice_profiles(id) ON DELETE RESTRICT`; `source_revision_id uuid NOT NULL`; `revision_number + integer NOT NULL DEFAULT 1`; `voice_licence_id uuid → voice_licenses(id) ON DELETE RESTRICT`; + `cast_by_user_id uuid`; `cast_at timestamptz NOT NULL DEFAULT now()`; `row_version`. +- **Unique:** `UNIQUE (project_id, speaker_id, source_revision_id, revision_number)`. +- **Indexes:** `voice_casts_locale_idx (project_locale_id)`; + `voice_casts_profile_idx (voice_profile_id)` — reverse-lookup "where is this voice used". +- **State machine:** `proposed → approved → superseded`. +- **RLS:** tenant read; write requires `project.voice.cast`. **Deny-by-default trigger:** a + cast whose `voice_licence_id` is not `active` and covering the locale/date is rejected. +- **Classification:** P. +- **Idempotency:** upsert on the 4-column unique; a change increments `revision_number` and + supersedes the prior row (never updates in place). +- **Verdict:** NEW. Replaces `voice_assignments.json` and `characters.voice_description`. + +### F.14 `project_locales` +- **Purpose:** the target-language lane for a project. **No `translations_de` tables.** +- **Grain:** one row per (project, BCP-47 locale). +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `locale_tag text NOT + NULL`; `is_source boolean NOT NULL DEFAULT false`; `translation_status text NOT NULL DEFAULT + 'not_started'`; `proofread_status text NOT NULL DEFAULT 'not_started'`; `cast_status text NOT + NULL DEFAULT 'not_started'`; `audio_status text NOT NULL DEFAULT 'not_started'`; `qa_status + text NOT NULL DEFAULT 'pending'`; `release_state text NOT NULL DEFAULT 'draft'`; `current_ + translation_run_id uuid`; `target_voice_profile_set_hash char(64)`; `row_version integer NOT + NULL DEFAULT 1`; `created_at`, `updated_at`. +- **Unique:** `UNIQUE (project_id, locale_tag)`. +- **Indexes:** `project_locales_project_idx (project_id)`. +- **State machine:** per-column monotone ladders (`not_started → in_progress → in_review → + approved`), `release_state`: `draft → approved → released | withdrawn`. +- **RLS:** tenant read; write requires the matching role for the column + (`project.translation.edit`, `project.translation.review`, `project.voice.cast`, + `project.audio.generate`, `project.release.approve`). +- **Idempotency:** upsert on `(project_id, locale_tag)`. +- **Verdict:** NEW. This is the one row per language that replaces all per-language tables. + +### F.15 `translation_runs` +- **Purpose:** an immutable execution record for translating a source revision into a locale. +- **Grain:** one row per translation execution. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `project_locale_id uuid + NOT NULL`; `source_revision_id uuid NOT NULL`; `run_kind text NOT NULL DEFAULT 'draft'` + (`draft`, `revise`, `verify`); `engine text NOT NULL`; `model_id text NOT NULL`; + `prompt_version text NOT NULL`; `input_sha256 char(64) NOT NULL`; `status text NOT NULL + DEFAULT 'running'` (same enum as `analysis_runs`); `started_by_service_id uuid`; + `started_at`, `finished_at`; `metrics jsonb NOT NULL DEFAULT '{}'`; `error text`; + `superseded_by_run_id uuid`. +- **Unique:** `UNIQUE (project_locale_id, run_kind, source_revision_id, input_sha256, + prompt_version, model_id)`. +- **Indexes:** `translation_runs_locale_idx (project_locale_id, started_at DESC)`. +- **State machine:** `running → succeeded | failed | cancelled`; `succeeded → superseded`. +- **RLS:** tenant read; write: `project.translation.edit` (human) or worker under lease. +- **Idempotency:** the wide UNIQUE; a retry with identical inputs returns the existing row. +- **Verdict:** NEW. + +### F.16 `translation_units` +- **Purpose:** an ordered target render span for one source unit in one locale. +- **Grain:** one row per (translation run, source unit, ordinal). +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `translation_run_id uuid + NOT NULL → translation_runs(id) ON DELETE CASCADE`; `source_unit_id uuid NOT NULL → + source_units(id) ON DELETE RESTRICT`; `source_unit_key char(64) NOT NULL` (denormalised so + cross-language joins never need a chain); `ordinal integer NOT NULL`; `target_text text NOT + NULL`; `target_sha256 char(64) NOT NULL`; `status text NOT NULL DEFAULT 'draft' CHECK + (status IN ('draft','submitted','proofread','approved','rejected'))`; `proofreader_user_id uuid`; + `reviewed_at timestamptz`; `review_notes text`; `row_version integer NOT NULL DEFAULT 1`; + `created_at`, `updated_at`. +- **Unique:** `UNIQUE (translation_run_id, ordinal)`; `UNIQUE (translation_run_id, + source_unit_id)`. +- **Indexes:** `translation_units_run_idx (translation_run_id, ordinal)`; + `translation_units_source_idx (project_id, source_unit_key)` — the cross-language join path. +- **State machine:** `draft → submitted → proofread → approved`; `→ rejected` from + `submitted`/`proofread`. Enforced by a trigger on `status` transitions. +- **RLS:** tenant read; `draft→submitted` requires `project.translation.edit`; + `→ approved` requires `project.translation.review` **and** `reviewer ≠ author` (four-eyes, + implemented as a trigger comparing `row_version`-tracked author field). +- **Classification:** C. +- **Idempotency:** upsert; `row_version` gives two reviewers a conflict instead of a silent + overwrite. +- **Verdict:** NEW. + +### F.17 `audio_runs` +- **Purpose:** one generation run over a project (or a bounded chapter range within it). +- **Grain:** one row per generation execution. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `project_locale_id uuid + NULL` (null = source language); `source_revision_id uuid NOT NULL`; `translation_run_id uuid + NULL`; `run_scope text NOT NULL DEFAULT 'full' CHECK (run_scope IN ('full','chapter','unit', + 'range'))`; `scope_from integer`, `scope_to integer`; `cast_snapshot_sha256 char(64) NOT + NULL`; `engine text NOT NULL`; `model_id text NOT NULL`; `params jsonb NOT NULL DEFAULT '{}'`; + `status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued','leased','running', + 'succeeded','partial','failed','cancelled'))`; `units_total integer NOT NULL DEFAULT 0`; + `units_generated integer NOT NULL DEFAULT 0`; `units_skipped integer NOT NULL DEFAULT 0`; + `units_failed integer NOT NULL DEFAULT 0`; `started_by_service_id uuid`; `started_at`, + `finished_at`; `error text`; `row_version integer NOT NULL DEFAULT 1`. +- **Unique:** `UNIQUE (project_id, source_revision_id, cast_snapshot_sha256, input_sha256, + engine, model_id, run_scope, scope_from, scope_to)` where `input_sha256` is a new column + holding the sha256 of the ordered `(unit_key, text_sha256, voice_profile_id, instruct, + fx_version)` tuple list. **This is the enterprise form of the marker-based idempotency.** +- **Indexes:** `audio_runs_project_idx (project_id, started_at DESC)`; + `audio_runs_status_idx (status) WHERE status IN ('queued','leased','running')`. +- **State machine:** `queued → leased → running → succeeded | partial | failed | cancelled`; + `leased → queued` on lease expiry (reaper). +- **RLS:** tenant read; write: worker under lease, or human with `project.audio.generate`. +- **Classification:** C. +- **Idempotency:** the wide UNIQUE — a retried identical run is a no-op, matching today's + marker-skip behaviour exactly. Concurrency: `FOR UPDATE SKIP LOCKED` on claim. +- **Verdict:** NEW. + +### F.18 `audio_run_casts` +- **Purpose:** the frozen cast snapshot an audio run actually used, so a release is + reproducible after casting changes. +- **Grain:** one row per (audio run, speaker). +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `audio_run_id uuid NOT + NULL → audio_runs(id) ON DELETE CASCADE`; `speaker_id uuid NOT NULL`; `voice_profile_id uuid + NOT NULL`; `voice_licence_id uuid`; `voice_reference_sha256 char(64)`; `cast_revision_number + integer NOT NULL`; `created_at`. +- **Unique:** `UNIQUE (audio_run_id, speaker_id)`. +- **Indexes:** `audio_run_casts_run_idx (audio_run_id)`; + `audio_run_casts_profile_idx (organization_id, voice_profile_id)`. +- **State machine:** immutable. +- **RLS:** tenant read; write only by the run's lease holder. +- **Idempotency:** `ON CONFLICT DO NOTHING`; the run-level `cast_snapshot_sha256` is the + digest of the ordered rows. +- **Verdict:** NEW. + +### F.19 `artifacts` +- **Purpose:** the registry of every produced file. **Metadata only.** +- **Grain:** one row per immutable produced artifact version. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `project_locale_id uuid`; + `artifact_kind text NOT NULL` (`source_clip`, `chapter_master`, `book_master`, `m4b`, + `mp3_track`, `wav_master`, `cover`, `epub`, `pdf`, `qa_report`, `manifest`, `voice_reference`, + `pronunciation_clip`); `logical_key text NOT NULL` (e.g. `ch11/utt_0042`, + `ar_package/Deal_Hunter_01_Salvage.mp3`); `object_key text NOT NULL`; `sha256 char(64) NOT + NULL`; `byte_size bigint NOT NULL CHECK (byte_size >= 0)`; `media_type text NOT NULL`; + `duration_ms integer`; `sample_rate integer`; `channels integer`; `loudness_lufs numeric(6,2)`; + `peak_dbfs numeric(6,2)`; `source_revision_id uuid`; `translation_run_id uuid`; `audio_run_id + uuid`; `supersedes_artifact_id uuid → artifacts(id) ON DELETE SET NULL`; `superseded_by_ + artifact_id uuid`; `retention_class text NOT NULL DEFAULT 'standard'`; `legal_hold boolean + NOT NULL DEFAULT false`; `availability text NOT NULL DEFAULT 'available' CHECK (availability + IN ('pending','quarantined','available','withdrawn'))`; `produced_by_service_id uuid`; + `created_at`; `row_version integer NOT NULL DEFAULT 1`; `legacy_object_key text`; + `legacy_marker jsonb`. +- **Unique:** `UNIQUE (project_id, object_key)`; `UNIQUE (project_id, sha256, artifact_kind)`. +- **Indexes:** `artifacts_project_idx (project_id, artifact_kind, created_at DESC)`; + `artifacts_run_idx (audio_run_id)`; `artifacts_source_unit_lookups via artifact_links`. +- **State machine:** `pending → quarantined | available`; `available → withdrawn`. **Never + deleted while `legal_hold` or while referenced by a release manifest.** +- **RLS:** tenant read; `availability='available'` additionally requires membership; + `withdrawn` requires `project.release.approve`; **write via the artifact service only**, and + download via short-lived signed URLs audited in `audit_events`. +- **Classification:** metadata C/P; **bytes never in PostgreSQL**. +- **Idempotency:** content-addressed by `(project_id, sha256, artifact_kind)`; re-producing + identical bytes returns the existing row and increments a `reproduced_count`. +- **Verdict:** NEW. `legacy_object_key` and `legacy_marker` are how the existing on-disk files + and their `*.wav.txt` markers are registered without moving a byte (§I). + +### F.20 `artifact_links` +- **Purpose:** the lineage DAG edge table. +- **Grain:** one row per (parent artifact, child artifact, relation). +- **Columns:** `id bigint GENERATED ALWAYS AS IDENTITY PK`; `organization_id uuid NOT NULL`; + `project_id uuid NOT NULL`; `parent_artifact_id uuid NOT NULL ON DELETE RESTRICT`; `child_ + artifact_id uuid NOT NULL ON DELETE RESTRICT`; `relation text NOT NULL` (`derived_from`, + `stitched_from`, `encoded_from`, `supersedes`, `voice_reference_for`, `translation_of`); + `created_at`. +- **Unique:** `UNIQUE (parent_artifact_id, child_artifact_id, relation)`; + `CHECK (parent_artifact_id <> child_artifact_id)`. +- **Indexes:** `artifact_links_child_idx (child_artifact_id)`; + `artifact_links_parent_idx (parent_artifact_id)`. +- **State machine:** append-only. +- **RLS:** tenant read; write by the producing service. +- **Idempotency:** the 3-column UNIQUE. +- **Verdict:** NEW. + +### F.21 `jobs` +- **Purpose:** the durable work queue that does not exist today. +- **Grain:** one row per unit of schedulable work. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `project_locale_id uuid`; + `job_type text NOT NULL` (`source_import`, `parse`, `semantic_analysis`, `identity_review`, + `translation_draft`, `translation_verify`, `pronunciation_resolve`, `voice_cast`, + `voice_sample_generate`, `audio_render`, `audio_qa`, `ebook_export`, `pdf_export`, + `audio_assembly`, `release_package`, `artifact_derivative`); `payload jsonb NOT NULL`; + `idempotency_key text NOT NULL`; `priority smallint NOT NULL DEFAULT 100`; + `deadline_at timestamptz`; `required_pool text`; `status text NOT NULL DEFAULT 'queued'` + (`queued`,`leased`,`running`,`succeeded`,`failed`,`cancelled`,`dead_letter`); `attempt_count + integer NOT NULL DEFAULT 0`; `max_attempts integer NOT NULL DEFAULT 3`; `next_attempt_at + timestamptz NOT NULL DEFAULT now()`; `last_error text`; `progress jsonb NOT NULL DEFAULT + '{}'`; `output_job_id uuid`; `created_at`, `updated_at`; `row_version integer NOT NULL + DEFAULT 1`; `created_by_user_id uuid`; `created_by_service_id uuid`. +- **Unique:** `UNIQUE (project_id, job_type, idempotency_key)`. **The idempotency key is + content-derived, never a filename** — this is the direct answer to spec 19 §12.2. +- **Indexes:** `jobs_claim_idx (status, next_attempt_at, priority) WHERE status IN ('queued', + 'leased')`; `jobs_project_idx (project_id, created_at DESC)`. +- **State machine:** `queued → leased → running → succeeded | failed`; `failed → queued` while + `attempt_count < max_attempts`; `failed → dead_letter` when exhausted; any → `cancelled`. +- **RLS:** tenant read; human write requires the permission mapped to `job_type`; + **worker write/update permitted only for a row whose `job_leases` entry names the caller's + service identity and is unexpired.** +- **Classification:** payload may contain text → C. +- **Idempotency/concurrency:** claim with + `UPDATE jobs SET status='leased' WHERE id = (SELECT id FROM jobs WHERE ... FOR UPDATE SKIP + LOCKED LIMIT 1) RETURNING *`. +- **Verdict:** NEW. + +### F.22 `job_attempts` +- **Purpose:** per-attempt history and the structured log pointer. +- **Grain:** one row per attempt. +- **Columns:** `id bigint GENERATED ALWAYS AS IDENTITY PK`; `organization_id`, `project_id` + NOT NULL; `job_id uuid NOT NULL ON DELETE CASCADE`; `attempt_number integer NOT NULL`; + `service_identity_id uuid NOT NULL`; `lease_id uuid`; `started_at`, `finished_at`; + `outcome text NOT NULL CHECK (outcome IN ('succeeded','failed','cancelled','expired'))`; + `error_class text`; `error_message text`; `log_uri text`; `metrics jsonb NOT NULL DEFAULT + '{}'`. +- **Unique:** `UNIQUE (job_id, attempt_number)`. +- **Indexes:** `job_attempts_job_idx (job_id, attempt_number DESC)`. +- **State machine:** append-only. +- **RLS:** tenant read; insert only by the lease holder. +- **Idempotency:** the UNIQUE; a retried attempt increments the number. +- **Verdict:** NEW. + +### F.23 `job_leases` +- **Purpose:** worker ownership with heartbeat and expiry — the mechanism that lets RLS scope a + worker to a project without impersonation. +- **Grain:** one row per active lease. +- **Columns:** `id uuid PK`; `organization_id`, `project_id` NOT NULL; `job_id uuid NOT NULL ON + DELETE CASCADE`; `service_identity_id uuid NOT NULL`; `lease_token_hash char(64) NOT NULL`; + `acquired_at timestamptz NOT NULL DEFAULT now()`; `heartbeat_at timestamptz NOT NULL DEFAULT + now()`; `expires_at timestamptz NOT NULL`; `released_at timestamptz`; `worker_fingerprint + text`. +- **Unique:** `UNIQUE (job_id) WHERE released_at IS NULL` — one live lease per job; + `UNIQUE (lease_token_hash)`. +- **Indexes:** `job_leases_expiry_idx (expires_at) WHERE released_at IS NULL` (reaper); + `job_leases_service_idx (service_identity_id, expires_at)`. +- **State machine:** `active → released | expired`. A reaper moves expired leases back to + `queued` and increments `attempt_count`. +- **RLS:** insert/renew by the worker for its own service identity; a worker may read only its + own rows. +- **Idempotency:** `ON CONFLICT (job_id) WHERE released_at IS NULL DO UPDATE SET heartbeat_at, + expires_at` — heartbeat is idempotent and repeatable. +- **Verdict:** NEW. **This table plus `service_identities` is the fix for BLOCKER B1.** + +### F.24 `service_identities` +- **Purpose:** a non-human principal. A worker or service is **never** a `users` row. +- **Grain:** one row per service principal. +- **Columns:** `id uuid PK`; `organization_id uuid NULL` (null = platform-wide); + `name citext NOT NULL`; `kind text NOT NULL CHECK (kind IN ('audio_worker','analysis_worker', + 'export_worker','migration_tool','readonly_report'))`; `status text NOT NULL DEFAULT 'active' + CHECK (status IN ('active','disabled'))`; `description text`; `created_at`, `updated_at`. +- **Unique:** `UNIQUE (name)`. +- **Indexes:** `service_identities_org_idx (organization_id, status)`. +- **State machine:** `active ↔ disabled`. +- **RLS:** readable by `platform_admin` and `organization_admin`; **not** readable by ordinary + users. Writes by `platform.manage` only. +- **Classification:** P (operational identity). +- **Idempotency:** `ON CONFLICT (name) DO NOTHING`. +- **Verdict:** NEW. **Required by BLOCKER B1.** + +### F.25 `legacy_id_map` +- **Purpose:** the permanent, auditable bridge from every legacy identifier to its canonical + enterprise id. This is the mechanism that makes rollback and re-sync safe forever. +- **Grain:** one row per (legacy system, entity kind, legacy key). +- **Columns:** `id uuid PK`; `organization_id uuid NOT NULL`; `project_id uuid NULL`; + `legacy_system text NOT NULL` (`umbra_sqlite`, `umbra_json`, `umbra_registry`, `umbra_file`); + `legacy_source_ref text NOT NULL` (e.g. `project.db:utterances:4711`, `ch11/utt_0042`); + `entity_kind text NOT NULL` (`project`, `chapter`, `source_unit`, `speaker`, `voice_profile`, + `artifact`, `job`); `entity_id uuid NULL`; `legacy_key text NOT NULL`; + `source_sha256 char(64)`; `mapping_state text NOT NULL DEFAULT 'mapped' CHECK (mapping_state + IN ('mapped','unmapped','ambiguous','rejected')); `confidence numeric(4,3)`; `note text`; + `mapped_at timestamptz NOT NULL DEFAULT now()`; `mapped_by_service_id uuid`. +- **Unique:** `UNIQUE (legacy_system, legacy_source_ref, legacy_key)`. +- **Indexes:** `legacy_id_map_entity_idx (entity_kind, entity_id)`; + `legacy_id_map_unmapped_idx (project_id) WHERE mapping_state <> 'mapped'`. +- **State machine:** `unmapped → mapped | ambiguous | rejected`. +- **RLS:** tenant read; write by migration tooling only. +- **Classification:** C. +- **Idempotency:** the UNIQUE; re-running the mapper is a no-op. +- **Verdict:** NEW. **This is where unmappable historical records go** — `mapping_state` + = `unmapped`/`ambiguous`/`rejected` is the answer to "historical local records that cannot + be mapped" (§I.7). Nothing is dropped. + +### F.26 Extensions to existing `001_control_plane` tables + +| Table | Change | Why | +|---|---|---| +| `projects` | ADD `work_id uuid → works(id) ON DELETE RESTRICT` | §E item 1; a project produces one work | +| `projects` | ADD `source_locale text NOT NULL DEFAULT 'en-US'` | `project_locales.is_source` needs an anchor | +| `projects` | ADD `row_version integer NOT NULL DEFAULT 1` | optimistic concurrency | +| `projects` | ADD `migration_state text NOT NULL DEFAULT 'legacy' CHECK (migration_state IN ('legacy','mirroring','dual_write','enterprise_authoritative','retired'))` | the per-project cutover flag (§I.5) | +| `projects` | ADD `legacy_project_path text` | the old `/Volumes/.../` directory, so both systems can find the same book | +| `users` | ADD `password_disabled_at timestamptz` | defence-in-depth: makes "no second password store" explicit and auditable | +| `sessions` | ADD `mfa_verified_at timestamptz`, ADD `idle_deadline_at timestamptz` | B4 fix; idle timeout | +| `audit_events` | ADD `actor_service_identity_id uuid → service_identities(id) ON DELETE SET NULL` | **B1 fix — worker attribution** | +| `audit_events` | ADD `principal_kind text NOT NULL DEFAULT 'user' CHECK (principal_kind IN ('user','service','system'))` | explicit principal type | +| `audit_events` | ADD `actor_subject_snapshot text` | immutability of identity behind an audit row when `users.email` later changes (§J.5) | +| `audit_events` | ADD `session_id uuid → sessions(id) ON DELETE SET NULL` | ties an authorization decision to a session | +| `permissions` | ADD rows (§D.5) | B3 fix | +| `roles` | ADD `audio_engineer` (project), `producer` (project) if required | optional; decide in M | + +**Deliberately NOT changed:** `audit_events.project_id` and `audit_events.organization_id` get +**no foreign keys**. Audit rows must survive the deletion of the project and organization they +describe; an FK would either block deletion or cascade the audit trail away. The absence of +those FKs is a feature and should be documented as intentional so a future reviewer does not +"fix" it. + +--- + +## G. Shared-record and cross-language invariants + +Each is stated as a testable invariant with its enforcement point. + +1. **One source ebook has one stable source identity.** *Invariant:* for a given + `(organization_id, sha256)` there is at most one `source_documents` row. + *Enforced:* `UNIQUE (project_id, sha256)` + content-addressed ingest. +2. **One canonical project contains the audiobook and the translation work.** + *Invariant:* every `source_units`, `speakers`, `artifacts`, `jobs` row for a book resolves to + exactly one `projects.id`, and that id is shared by UmbraAudio and UmbraTrans. + *Enforced:* NOT NULL `project_id` + composite FK + composite RLS. +3. **Analysis has immutable versions.** *Invariant:* no `UPDATE` on a `source_revisions` row + with `locked = true`, and no `UPDATE`/`DELETE` on `analysis_runs`/`analysis_snapshots` in + state `succeeded`. + *Enforced:* `BEFORE UPDATE` triggers; append-only trigger on `audit_events` already exists as + a precedent (`001:356-359`). +4. **Speaker identity is stable across languages.** *Invariant:* `speakers.canonical_name` and + its uuid are the only cross-language speaker reference; `translation_units.source_unit_key` + and `speaker_id` carry that identity, never a translated display name. + *Enforced:* no translated name is ever written to `speakers`. +5. **A voice profile is reusable across translations only when language, accent, consent and + licensing allow.** *Invariant:* a row in `voice_casts` may only be created when + `voice_licences.status='active'` and the licence covers the target `locale_tag`, the + requested territory, the media, and the release date, and the profile's `locale_tag`/`accent` + are compatible. + *Enforced:* deny-by-default trigger on `voice_casts` + `app.voice_licence_covers(...)`. +6. **Every translation references a source book, a project and a specific analysis version.** + *Invariant:* `translation_runs.source_revision_id IS NOT NULL` and its + `input_sha256` matches the `analysis_runs` input for the same revision. + *Enforced:* NOT NULL FK + a reconciliation check in the nightly validator. +7. **Every audiobook references a project, a translation when applicable, a voice-cast + snapshot and a generation run.** *Invariant:* every `artifacts` row with + `artifact_kind IN ('m4b','mp3_track','book_master','chapter_master')` has + `audio_run_id IS NOT NULL` and `cast_snapshot_sha256` resolvable in `audio_run_casts`. + *Enforced:* NOT NULL FK + `CHECK` + the nightly lineage validator. +8. **Translated ebooks and audiobooks never become independent duplicate source records.** + *Invariant:* a translated artifact's lineage path to a `source_documents` row always exists + and is acyclic; no translated artifact is ever registered as a `source_documents` row. + *Enforced:* `artifact_links` acyclicity check (recursive CTE, nightly) + application rule. +9. **The audio system keeps creating audiobooks before, during and after cutover.** + *Invariant:* the audio worker's critical path never requires a successful enterprise write. + *Enforced:* degraded-mode lease (§I.6); PG write failure is logged and reconciled, never + fatal. **This is the invariant that makes the whole plan safe, and it must be tested by + stopping PostgreSQL mid-run (§L, test 14).** +10. **Large media live in object storage; PostgreSQL holds only object keys, checksums, media + metadata and lifecycle state.** *Invariant:* no table in this design has a `bytea`/`blob` + column. + *Enforced:* schema review + a test asserting no `bytea` column exists in schema `public`. +11. **Every write is idempotent and safe to retry.** *Invariant:* every table has a natural + unique constraint; re-executing any write with identical inputs changes nothing. + *Enforced:* the UNIQUE constraints in §F; replay tests in §L. +12. **Concurrent workers cannot silently overwrite one another.** *Invariant:* a job has at most + one live lease; metadata edits use `row_version`. + *Enforced:* partial UNIQUE on `job_leases`, `FOR UPDATE SKIP LOCKED` claims, `row_version` + on every mutable table. +13. **Service identities are not represented by arbitrary client-supplied user ids.** + *Invariant:* `audit_events.principal_kind='service'` rows have + `actor_service_identity_id IS NOT NULL` and `actor_user_id IS NULL`. + *Enforced:* `CHECK` constraint + the GUC-derivation rule in §J.1 (the service identity comes + from the connection, never from a payload). +14. **The future translation UI can discover authorized projects and audiobooks without a + separate account system.** *Invariant:* the UI's project list is exactly + `SELECT id FROM projects WHERE app.can_access_project(id)` — the same predicate the audio + system uses. *Enforced:* both clients use one `app.authorized_projects()` function. +15. **The existing staleness rule survives.** *Invariant:* an artifact whose + `legacy_marker.fx_version` is set is stale for a run that requests no `fx_version` + (`generation/staleness.py:98`). Any enterprise re-implementation of staleness must + preserve this, because the two current generation drivers differ on it + (`ChapterBuilder` does not pass `instruct`/`fx_version`; `scripts/generate_all_audio.py` + does). *Enforced:* documented and covered by a regression test. + +--- + +## H. Schema delta and migration order + +No SQL is emitted. Each row is a reviewed change with its own rollback and verification. + +| # | Class | Change | Reason | Depends on | Rollout | Rollback | Volume | Lock risk | Verification | Human approval | +|---|---|---|---|---|---|---|---|---|---|---| +| H1 | NEW SERVICE ROLE | `service_identities` principal + `app.current_service_identity_id()` | B1 blocker | — | additive | drop function | 0 | none | §L-6 | **Yes** | +| H2 | ADD TABLE | `service_identities` | B1 | H1 | additive | drop table | ~5 rows | none | §L-6 | **Yes** | +| H3 | ADD TABLE | `works`, `source_documents`, `source_revisions`, `source_chapters` | §E 1 | — | additive | drop | 10s–100s | none | §L-8 | **Yes** | +| H4 | ADD TABLE | `legacy_id_map` | §I stable-ID mapping | H3 | additive | drop | 10⁵ rows | none | §L-8 | **Yes** | +| H5 | ALTER TABLE | `projects`: `work_id`, `source_locale`, `row_version`, `migration_state`, `legacy_project_path` | cutover flag + work link | H3 | nullable columns, then backfill, then `NOT NULL` on `source_locale` | drop columns | ~10 rows | brief ACCESS EXCLUSIVE; use `NOT VALID` patterns, `lock_timeout` | §L-5 | **Yes** | +| H6 | ADD TABLE | `speakers`, `speaker_aliases` | 5-store identity fix | H3 | additive | drop | ~10² | none | §L-9 | **Yes** | +| H7 | ADD TABLE | `source_units` (+ generated `source_unit_key`) | §E.1 stable identity | H3, H6 | additive | drop | 10⁴–10⁶ | none | §L-9 | **Yes** | +| H8 | ADD TABLE | `analysis_runs`, `analysis_snapshots` | immutable analysis | H3 | additive | drop | 10² | none | §L-10 | **Yes** | +| H9 | ADD TABLE | `voice_profiles`, `voice_profile_revisions`, `voice_licenses`, `voice_casts` | voice reuse gating | H6 | additive | drop | 10² | none | §L-11 | **Yes** | +| H10 | ADD TABLE | `project_locales`, `translation_runs`, `translation_units` | locale lanes | H3 | additive | drop | 0 initially | none | §L-12 | **Yes** | +| H11 | ADD TABLE | `audio_runs`, `audio_run_casts` | generation runs | H7, H9 | additive | drop | 10²–10³ | none | §L-13 | **Yes** | +| H12 | ADD TABLE | `artifacts`, `artifact_links` | artifact registry + lineage | H11 | additive | drop | 10⁴–10⁶ | none | §L-13 | **Yes** | +| H13 | ADD TABLE | `jobs`, `job_attempts`, `job_leases` | durable queue | H1 | additive | drop | 10³–10⁵ | none | §L-14 | **Yes** | +| H14 | ADD CONSTRAINT | `artifacts`: `CHECK` requiring `audio_run_id` for audio kinds; `artifact_links` self-edge `CHECK` | G7 | H12 | `NOT VALID` then `VALIDATE` | drop constraint | large | none if `NOT VALID` | §L-13 | **Yes** | +| H15 | ALTER TABLE | `audit_events`: `actor_service_identity_id`, `principal_kind`, `actor_subject_snapshot`, `session_id` | worker attribution | H1, H2 | nullable add | drop columns | large table; add nullable only | brief | §L-7 | **Yes** | +| H16 | ENABLE/PROVE RLS | RLS policies on all 20 new tables, tenant read/write, worker-lease branch | §J | H2–H13 | create policies, then `ENABLE`, then prove | drop policies | — | none | §L-1,2,3 | **Yes** | +| H17 | ALTER TABLE | `sessions`: `mfa_verified_at`, `idle_deadline_at` | B4 | — | nullable add | drop columns | small | brief | §L-4 | **Yes** | +| H18 | ADD CONSTRAINT | `users.password_disabled_at` | no second password store, explicit | — | nullable add | drop column | small | brief | §L-4 | **Yes** | +| H19 | BACKFILL | Insert `permissions` rows + `role_permissions` grants from §D.5 | B3 | — | idempotent upserts | delete the added rows | ~15 rows | none | §L-5 | **Yes** | +| H20 | ADD CONSTRAINT | `users.status='active'` + `organizations.status='active'` enforcement inside `app.resolve_session` | B5 | H17 | replace function | revert function | — | none | §L-4 | **Yes** | +| H21 | ADD FUNCTION | `app.revoke_user_sessions(uuid)`, `app.authorized_projects()`, `app.can(action, project_id)` | D.3, D.5, G14 | H1 | create | drop | — | none | §L-2,4 | **Yes** | +| H22 | ENABLE/PROVE RLS | Replace `projects_select`, `organizations_select`, `user_roles_select` policies to (a) admit project-only members to their org, (b) admit `voice_manager` to project metadata, (c) scope the worker | §J.3, §J.4 | H1, H21 | `CREATE OR REPLACE` in one transaction | restore prior policy text | small | brief | §L-2,3 | **Yes** | +| H23 | GRANT/REVOKE | `ALTER DEFAULT PRIVILEGES` — revoke `EXECUTE ON FUNCTIONS` from `PUBLIC` in schema `app`; grant table DML to `umbracontrol_api`; grant lease-scoped DML to `umbracontrol_worker` | **B7 — new `app` functions are world-executable by default today** | H1 | per-migrator `ALTER DEFAULT PRIVILEGES` | revert | — | none | §L-15 | **Yes** | +| H24 | GRANT/REVOKE | Re-own `app` schema + all tables from `postgres` to `umbracontrol_owner` (NOLOGIN); migrations connect as `umbracontrol_migrator` then `SET ROLE umbracontrol_owner`; **never `FORCE ROW LEVEL SECURITY`** | §J.2 superuser dependency | H16 | maintenance window; `REASSIGN OWNED` in one transaction | `REASSIGN OWNED ... TO postgres` | all | brief; blocks writes | §L-16,17 | **Yes — highest risk item** | +| H25 | BACKFILL | Project → `projects` (with `legacy_project_path`); manuscript → `source_documents`; speakers → `speakers`/`speaker_aliases`; utterances → `source_units`; voice refs → `voice_profiles` (+`voice_licenses` `pending`); artifacts → `artifacts` (+`legacy_object_key`, `legacy_marker`) | §I.4 | H4–H12 | resumable, batched, idempotent | delete by `legacy_id_map` batch | 10⁵–10⁶ | none (batched) | §L-8,9,11,13 | **Yes** | +| H26 | ADD INDEX | All indexes in §F, created `CONCURRENTLY` | query paths | tables exist | `CREATE INDEX CONCURRENTLY`, one at a time | `DROP INDEX` | large | none | `EXPLAIN` checks | No | +| H27 | ADD CONSTRAINT | Triggers: immutable revisions, status state machines, four-eyes approval, cast licence gate, `row_version` bump | G3, G5, G12 | tables exist | create, then backfill-violation audit | drop triggers | — | none | §L-13,18 | **Yes** | +| H28 | NEW SERVICE ROLE | Optional `audio_engineer` / `producer` roles | §M open question | H19 | seed upsert | delete | 0 | none | §L-5 | **Yes** | +| H29 | NO CHANGE | `organizations`, `roles`, `role_permissions`, `invitations`, `mfa_methods`, `bootstrap_tokens`, `schema_migrations` | no delta required | — | — | — | — | — | — | No | +| H30 | NO CHANGE | Every legacy SQLite table, JSON file, WAV, MP3, M4B, marker | §G invariant 9 | — | — | — | — | — | — | No | + +**Dependency chain:** H1 → H2 → H3 → H4 → {H5, H6, H7, H8} → {H9, H10, H11} → H12 → H13; +H16 after all tables; H22–H24 last (they change the enforcement layer and must be proven +against a fully-populated schema). + +--- + +## I. Gradual cutover, reconciliation, and rollback plan + +### I.1 The repository/adapter boundary (Phase P1 — no behaviour change) + +Introduce a single seam in the audio codebase, behind which all state access lives: + +``` +umbraaudio/persistence/ + ProjectRepository (protocol) + SqliteProjectRepository -> today's project.db + JSON + files [DEFAULT] + ControlPlaneProjectRepository -> enterprise PostgreSQL + ArtifactStore + LocalArtifactStore -> today's filesystem tree [DEFAULT] + ObjectArtifactStore -> enterprise object storage +``` + +Rule: **the default implementation is today's code, byte-for-byte.** Phase P1 ships with the +enterprise implementation present but unused behind a per-project flag. Nothing about audiobook +production changes. This is what makes the rest of the plan reversible. + +### I.2 Read-only mirror / shadow-read phase (P2) + +- H25 backfill populates the enterprise DB from SQLite/files, recording every mapping in + `legacy_id_map`. +- The audio system keeps **reading and writing only** SQLite/files. Zero risk. +- A nightly **shadow-read** job runs the *same queries* through both repositories and records + discrepancies in a reconciliation table. Discrepancies are **logged, never fatal**. + +### I.3 Stable-ID mapping and reconciliation rules + +| Legacy identity | Canonical id | Rule | +|---|---|---| +| `//` directory | `projects.id` | `legacy_project_path`; slug collisions resolved by suffix, never overwritten | +| SQLite `project.id` | `projects.id` | via `legacy_id_map` | +| `chapters.id` | `source_chapters.id` | within a `source_revision` | +| `utterances.id` / `utterance_index` | `source_units.source_unit_key` | §E.1; **`utterance_index` alone is never used as an id** | +| `characters.id`, `speakers.json` `primary_name`, `voice_profiles.json` key, `voice_registry.character_name`, `library_speakers.primary_name` | `speakers.id` | **five sources merged**; conflicts recorded, ambiguous ones left `mapping_state='ambiguous'` | +| `voices/{slug}_ref.wav` | `voice_profiles.id` | `reference_object_key`; `voice_licenses` seeded `pending` | +| `chapter_audio/ch{N}/utt_{I}.wav` | `artifacts.id` | `legacy_object_key`; `legacy_marker` = parsed `*.wav.txt` | +| `ch{N}_full.wav` / `.mp3` / `export/*.m4b` / `ar_package/*.mp3` | `artifacts.id` | `legacy_object_key` | +| `generation_status.json`, `ch{N}_build_status.json` | `audio_runs` + `jobs` | seeded as `succeeded`/`partial` with `legacy_marker` | +| `'user'` in `reviewed_by` / `resolved_by` | the single canonical `users.id` | `legacy_system='umbra_json'`, note `legacy_actor='user'` | + +**Reconciliation rules (nightly):** row counts per entity per project; `sha256` of every +registered artifact re-verified against the file; status-transition legality (no +`succeeded → running`); lineage completeness (§G7); `mapping_state <> 'mapped'` counts; and +speaker-name consistency between `speakers.canonical_name` and `speaker_aliases`. + +### I.4 Backfill plan + +Order: projects → works/source documents/revisions → speakers + aliases → source units → +analysis runs/snapshots → voice profiles + licences + casts → artifacts + links → audio runs → +jobs. Batched at 1000 rows, resumable, idempotent, each batch one transaction, and each batch +only advances `migration_state` after its reconciliation passes. **`peak memory` and runtime are +bounded by batching; nothing is held in memory.** + +### I.5 Feature flag / per-project cutover + +`projects.migration_state` ∈ `legacy | mirroring | dual_write | enterprise_authoritative | +retired`. The audio worker reads it per request; a per-project override in local config allows +an immediate per-project rollback without a database change. Cutover is **per project, never +global**, and is reversible in one flag flip. + +### I.6 Dual-write / write-through period + +For a project in `dual_write`: + +- **Primary path: unchanged.** Audio generation writes the WAV, then the marker, then the + SQLite row — exactly as today. It never waits on PostgreSQL. +- **Secondary path: best-effort mirror** of the small, high-value records: `artifacts` + (object key + `sha256` + `legacy_marker`), `audio_runs` progress, `jobs`, `speakers`, + `source_units`, `audit_events`. +- **On PostgreSQL failure:** log, increment a `mirror_deficit` counter, and **continue**. The + audio work is never blocked, never rolled back, never retried inline. A reconciler replays the + deficit from the local state. This is the concrete meaning of G9. +- **Degraded mode:** if PostgreSQL is unreachable when a worker starts a run, the worker + continues under a **previously issued lease** with a bounded validity (6 h), after which it + must re-check in. Exceeding that window halts *new* runs, not in-flight rendering, and + raises an operator alert. Production is degraded, never stopped, and never silently + un-reconciled. +- **TTS saturation note:** the Mac serves one TTS model and must not co-reside with the LLM + (AGENTS.md). The `resource_pool` column on `jobs` (`tts_local`, `llm_local`, `gpu_remote`, + `cpu`) is what lets the queue express that constraint, replacing today's in-process lock. + +### I.7 Historical local records that cannot be mapped + +They are **never dropped**. They are written to `legacy_id_map` with +`mapping_state ∈ ('unmapped','ambiguous','rejected')`, a `note`, and the original +`legacy_source_ref`. A monthly report lists them. Resolution is a human decision +(merge / confirm / retire). The five-way speaker conflict (§I.3) is the largest expected +population. This directly satisfies "a plan for historical local records that cannot be mapped". + +### I.8 The point at which the enterprise database becomes authoritative + +Per project, when all of the following hold for **three consecutive nightly reconciliations**: +counts match; every registered artifact's `sha256` matches; zero `mapping_state='unmapped'` +rows for that project; a full generation run has completed **with PostgreSQL as the primary +store** (the `enterprise_authoritative` dry-run sub-phase); and a restore drill (§K) has passed. +Then `migration_state = 'enterprise_authoritative'`: UmbraAudio reads project state, +speakers, units and artifacts from PostgreSQL; SQLite becomes a read-only mirror; **the +filesystem remains the audio working area and the `*.wav.txt` markers remain the idempotency +record**, so the TTS path is unchanged. Historical local records are retained read-only +indefinitely. + +### I.9 Rollback plan + +1. Flip `migration_state` back to `dual_write` (or `mirroring`). One statement. +2. The audio worker reverts to SQLite + files as primary. **The worker is never stopped.** +3. Enterprise rows written during the dual-write window are retained, not deleted; they are + marked `superseded_by_local` so reconciliation knows they are not authoritative. +4. No source data is deleted at any point in this plan. Legacy SQLite, JSON, WAV, MP3, M4B and + markers are all read-only after cutover and are never pruned until a separate, explicit + retention decision (§M). +5. Rollback is safe **at any point**, because Phase P1's default repository is still the + current code. + +--- + +## J. Security / RLS review + +### J.1 Can RLS be bypassed by client-controlled settings? — **Yes. Documented residual risk.** + +`app.current_user_id()` is: + +```sql +-- 001_control_plane.sql:218-224 +SELECT NULLIF(current_setting('app.user_id', true), '')::uuid +``` + +`app.user_id` is a **custom placeholder GUC**, and any role may `SET` it. There is no +`REVOKE SET ON PARAMETER` for a GUC that does not exist in `pg_settings`, so this **cannot be +prevented in PostgreSQL**. Consequences: + +- Any holder of the `umbracontrol_api` or `umbracontrol_worker` credential can impersonate + **any** user, including a `platform_admin`, and read everything that user's RLS permits. +- A non-UUID value raises `22P02`, a minor robustness/DoS issue. + +This is the standard pattern used by Supabase/PostgREST and it is **acceptable only under +compensating controls**, all of which are mandatory: + +1. The API/worker credential lives in a root-owned `0640` file readable only by the + `umbracontrol` service group (the bootstrap script already creates + `/etc/umbracontrol/db-credentials.env` with exactly these permissions — preserve that). +2. `app.user_id` / `app.service_identity_id` are set with `SET LOCAL` inside an explicit + transaction, by trusted framework code only, **derived from a verified session token**. +3. **No code path may ever read these GUCs from a request body, query parameter, cookie, + header, or file.** This is a code-review invariant with a lint rule. +4. The database is never exposed directly; only the API connects. +5. `ALTER ROLE umbracontrol_api SET log_statement = 'none'` is already applied by the bootstrap + script's `PGOPTIONS`; statement logging of GUC changes is available for incident response. +6. A dedicated non-superuser owner (§H24) means a superuser is not in the data path. + +**Residual risk, stated plainly: leakage of the `umbracontrol_api` password is equivalent to +full tenant bypass.** That is why §L-16 tests credential handling and why the API must never +be internet-exposed. + +### J.2 Superuser ownership of the RLS machinery — **HIGH finding** + +Verified: `infra/bootstrap_umbracontrol_database.sh:243` streams the schema through +`psql_as_postgres`, i.e. as the `postgres` **superuser**. Therefore **all 13 tables and all +`app.*` functions are owned by a superuser**, and the database itself is owned by +`umbracontrol_owner` (NOLOGIN, `createdb --owner` at line 226) which owns nothing else. + +Two consequences: + +1. Superusers bypass RLS unconditionally. This is **load-bearing**, not incidental: the policy + `projects_select USING (app.can_access_project(id))` calls a function that itself does + `SELECT 1 FROM projects …` (`001:298-320`). That is a self-referencing policy. It does **not** + raise `42P17 infinite recursion` **only because** the `SECURITY DEFINER` function runs as a + superuser and therefore bypasses RLS on the inner query. +2. Therefore **`ALTER TABLE projects FORCE ROW LEVEL SECURITY` would break the entire access + model with `42P17`.** Equally, re-owning the `app` functions to a role that does **not** own + the tables would break it. The only working configuration is: *the `app.*` SECURITY DEFINER + functions and the tables they read must share one owner, and RLS must not be FORCE'd on the + mutually-referencing tables (`organizations`, `projects`, `users`, `user_roles`).* + +**Required (H24):** re-own the `app` schema and all tables to the dedicated NOLOGIN role +`umbracontrol_owner`; run migrations as `umbracontrol_migrator` then `SET ROLE +umbracontrol_owner`. This preserves the required semantics, removes the superuser from the data +path, and makes the invariant explicit. Add the regression assertion in §L-17 so nobody later +"improves" it with `FORCE`. + +### J.3 `voice_manager` cannot see any project — **authorization gap** + +`voice_manager` is seeded at `scope='organization'` (`001:373`). `projects_select` requires +`can_access_project`, which admits only platform admins, **organization admins**, and holders +of a `scope='project'` role. An org-scoped `voice_manager` therefore matches **zero** projects +and cannot perform their job. Fix (H22): a policy branch granting org-scoped non-admin roles +visibility of project **metadata only** (`id, name, slug, status`) — implemented as a +column-restricting view or a `column_privilege` arrangement, not a blanket row grant. **This +requires a human decision** (§M Q3): least privilege vs. usability. + +### J.4 Project-only members cannot read their own organization — **availability gap** + +`organizations_select` requires `is_organization_member`, which admits only org-scope roles +named `organization_admin` or `member` (`001:257-261`). A user holding only `translator` on one +project cannot `SELECT` their organization row, so the UI cannot render an org name. Under- +permissive, not a hole. Fix (H22): also admit any active `user_roles` row for that organization +at any scope. + +### J.5 Human session vs worker identity + +- **Human:** `sessions` row; `app.resolve_session(token_hash)` → `users.id`; all authorisation + derives from `user_roles`. `sessions_self` (`001:526-529`) confines a user to their own rows. +- **Worker:** `service_identities` row + `job_leases` row; `app.current_service_identity_id()`; + authorisation derives from an **unexpired lease**, never from a user. A worker with no lease + sees no project rows. `audit_events.principal_kind` distinguishes the two permanently. +- **No worker path may set `app.user_id`.** Enforced by code review + the fact that + `principal_kind='service'` audit rows require `actor_service_identity_id IS NOT NULL AND + actor_user_id IS NULL`. +- **Audit integrity note:** `users.email` is mutable and `audit_events.actor_user_id` has a FK + with no `ON DELETE` action, so (a) a user with audit history **cannot be deleted** (FK + violation) and (b) changing a user's email silently changes the identity behind historical + audit rows. Fix: H15 `actor_subject_snapshot` freezes the subject at write time; and + document that account deletion is intentionally blocked once audit history exists. + +### J.6 Pre-existing application-level defects (preconditions, not migration scope) + +These must be fixed **before** the audio web UI is placed behind the enterprise identity +layer, because the identity layer is only as strong as the endpoint it guards. Reported, not +fixed here: + +1. **No authentication at all.** Zero `Depends`, zero middleware, zero CSRF, zero security + headers, zero `TrustedHost` (`umbraaudio/web/`). 129 routes, 65 mutating, **0 + authorization checks**. The only gate is the unsigned `active_project` cookie + (`routes.py:482-491`). +2. **The server binds `0.0.0.0:8420` over plain HTTP** with the macOS firewall disabled and a + LAN Caddy proxy in front. Today this is reachable by anyone on the LAN. **Until auth lands, + the audio UI must remain LAN-only behind an authenticating reverse proxy.** +3. **`load_config` accepts absolute paths verbatim** (`config.py:156-158`), so the cookie is a + path-selection primitive. `active_project=/any/dir` with `/any/dir/.umbra/project.db` grants + full read/write. Verified. +4. **The `activate` existence check is dead code** (`routes.py:525`), because + `load_config` pre-creates the directory (`config.py:165`). +5. **Download traversal:** `routes.py:5049` uses `str(file_path).startswith(str(export_dir))` — + a **string prefix, not path-component containment**. A sibling directory such as + `export_secret/` satisfies the prefix and is served. `Path.is_relative_to()` is the correct + primitive. The existence check at `:5047` also runs **before** the containment check, giving a + 404-vs-403 existence oracle. +6. **Upload path traversal:** `routes.py:633, 640, 647, 652` use `upload.filename` raw, with no + `Path(...).name`. A multipart part named `../../../evil.txt` passes the suffix allow-list + (`.txt`) and writes outside `data_dir/uploads`. Same pattern at `routes.py:558, 572`. + (`routes.py:6627` does it correctly — the fix pattern already exists in the codebase.) +7. **Unauthenticated resource control:** `POST /api/models/lm/unload` (`routes.py:5814`) unloads + every LM Studio instance; `POST /api/models/switch/{stage}` unloads/loads GB-scale models. +8. **Unauthenticated destructive operations:** `POST /api/pipeline/reset?step=discover` + (`routes.py:4690`) `shutil.rmtree`s the voices directory; `POST /api/pipeline/run-all` + (`routes.py:4711`) wipes `speakers.json`, all utterances and all generated audio. +9. **Zero download authorization.** Of 12 file-serving endpoints, none checks a user; the two + slug comparisons (`routes.py:2604, 2991`) are cache-buster guards, not authorization. +10. **A second, CWD-relative project-resolution path** exists in the `speaker_designations` + template global (`web/__init__.py:51`): `Path(slug) / ".umbra"`, with no `load_config`, no + containment check and no `project.db` existence test. +11. **Concurrent `ChapterBuilder` runs are unprotected** (`routes.py:5097-5110`), so two clicks + write the same `utt_*.wav` concurrently. +12. **Unauthenticated outbound-request proxy:** `POST /api/pronunciations/{name}/lookup` + (`routes.py:1847`) fetches attacker-supplied strings from `dictionaryapi.dev`. +13. **Destructive manuscript upload:** `POST /api/upload-manuscript` with `mode=new` rmtree's + voice and audio directories (`routes.py:712-742`) with no confirmation and no permission. + +### J.7 Service-to-service write authorisation + +Writes are authorised by **lease**, not by role. A worker may `UPDATE` a `jobs` row only while +it holds the unexpired `job_leases` row for that job, and the policy derives that from +`app.current_service_identity_id()` — never from a parameter. Renewals are +`ON CONFLICT (job_id) WHERE released_at IS NULL DO UPDATE`, so a heartbeat is idempotent. Audit +inserts by a service require `actor_service_identity_id` to match the current service identity. + +### J.8 Can login state be forged or reused across organizations? + +- **Forged:** no. A session requires a 256-bit token whose SHA-256 matches `sessions.token_hash`; + there is no forgeable signed cookie. +- **Reused across organizations:** **yes, by design, and this is correct for a human** — a + session is user-global, so one token is valid for every organization that user belongs to. The + protection is that *every* request re-derives and re-authorises the organization/project + context from the session's user id, and RLS admits only organizations for which that user has + an active `user_roles` row. There is no "default organization" and no cross-org inheritance. + The practical consequence to enforce: **never cache an authorised project/organization in the + session payload.** + +### J.9 Secrets + +`001_control_plane.sql` contains no secret material and neither does this document. +`bootstrap_tokens.token_hash`, `sessions.token_hash`, `mfa_methods.secret_reference` and +`invitations.token_hash` are all **hash-or-reference columns**, never plaintext. +`lease_token_hash char(64)` (F.23) follows the same pattern. `voice_licenses.consent_evidence_ +uri` points at object storage. The bootstrap admin's one-time token is root-only at +`/root/umbra-enterprise-bootstrap.token` and, per the infra README, must never be pasted into +chat or logs. **Add to the contract: the API must never log a connection string, a token, a GUC +value carrying a user id, or a query parameter that could contain one.** + +--- + +## K. Backup and restore requirements + +1. **PostgreSQL PITR** with WAL archiving; target RPO ≤ 5 min, RTO ≤ 60 min. Daily full + backup retained 35 days, monthly retained 13 months. +2. **Restore drill before cutover (G-K1):** restore a full backup to a scratch instance, then + verify: row counts per table match the source; every `artifacts.sha256` matches its object in + object storage; every project is reachable by an authorised user and **only** by authorised + users; `audit_events` is intact and append-only; a `source_revision`, a `speakers` row, a + `voice_casts` row and an `artifacts` lineage path survive and resolve end-to-end. +3. **Object storage versioning enabled**, with server-side checksum validation and + immutability (or object-lock) on released artifacts. A `DELETE` must never be a filesystem + unlink: deletion is a `retention_class`/tombstone transition. +4. **Class R (voice reference audio, embeddings, voice prompts) backed up separately** from + project data, with its own access control and its own retention/consent policy, because it is + biometric data. +5. **Class C (manuscripts, translations, produced audio) retention tied to the underlying + licence.** When an author's licence expires, the derived audiobook and translation + entitlements are withdrawn (`artifacts.availability='withdrawn'`) — **not silently deleted**, + so the audit trail and any accounting survive. +6. **Legacy sources protected:** the local project tree, `~/.umbra/voice_registry.db` and all + markers are included in the backup set for the whole transition and are **read-only** after + cutover. A full UmbraAudio project export/re-import must preserve lineage (spec 19 §15.3). +7. **The backup role is `umbracontrol_backup` (LOGIN, `SELECT` on all tables, no schema + `CREATE`)** — note it is granted `SELECT ON ALL TABLES` at `001:576`, which is a + point-in-time grant; new tables get no backup access until H23 sets default privileges. + **Backup coverage of new tables is a release gate.** + +--- + +## L. Acceptance tests + +| # | Test | Pass condition | +|---|---|---| +| L-1 | Two-organization isolation | A user in org A cannot `SELECT` **any** row of org B's `projects`, `source_units`, `speakers`, `artifacts`, `jobs` via any API route or direct query. | +| L-2 | Authorized project discovery | `app.authorized_projects()` returns exactly the projects with an active membership; the UI project list equals that set; forging a project id in a path yields 404/403, never data. | +| L-3 | Policy-only membership | A `translator` on project X can read project X and **cannot** read project Y in the same org; an org `voice_manager` sees project **metadata** but no manuscript text (per the J.3 decision). | +| L-4 | Session lifecycle | Expiry, logout, "sign out everywhere", suspension and `deactivated` all stop resolution on the next request; a user with `mfa_required=true` cannot obtain a session without MFA asserted; revoked tokens never resolve. | +| L-5 | Permission enforcement | Each action in §D.5 is denied for a role lacking the permission and allowed for a role holding it — including `project.audio.generate` and `project.speakers.edit`, which have **no** equivalent today. | +| L-6 | Worker lease scoping | A worker holding a lease on project X can read/write X; the same worker on project Y (no lease) sees **zero** rows and every write raises `42501`. A worker cannot read `sessions`, `mfa_methods`, or `users.password`-like columns. | +| L-7 | Worker audit attribution | Every worker-initiated mutation produces an `audit_events` row with `principal_kind='service'`, `actor_service_identity_id` set, `actor_user_id` NULL, and the correct `project_id`. | +| L-8 | Backfill completeness | For every project: legacy project count = `projects` count; every legacy chapter/utterance has a `legacy_id_map` row; re-running the backfill changes **zero** rows. | +| L-9 | Identity merge correctness | All five legacy speaker stores map to one `speakers.id` per canonical name; `source_units.speaker_name` values are all present in `speakers.canonical_name ∪ speaker_aliases.alias`; no utterance references a missing speaker. | +| L-10 | Analysis immutability | An `UPDATE` on a locked `source_revisions` row, or on a `succeeded` `analysis_runs` row, raises. | +| L-11 | Voice licence gating | A cast against a `pending`/`expired`/`takedown` licence, or one not covering the target locale/territory/media/date, is rejected; an existing render is **not** deleted and a project issue is raised instead. | +| L-12 | Locale independence | Two locales on one project have fully independent translation/cast/audio/release state; no table is named for a language. | +| L-13 | Lineage + idempotency | Every audio artifact resolves a complete acyclic path to a `source_documents` row through `artifact_links`; re-running an identical `audio_runs` payload creates no new row; two concurrent claimers produce exactly one lease. | +| L-14 | **Continuity under database outage** | With PostgreSQL **stopped** mid-generation, audio production continues to completion; on restart, the mirror reconciler converges with zero lost artifacts; the 6 h lease window then halts *new* runs while in-flight rendering completes. | +| L-15 | Default-privilege safety | A newly created function in schema `app` is **not** executable by `PUBLIC`; a newly created table is readable/writable by `umbracontrol_api` and scoped for the worker. | +| L-16 | Credential handling | The database credential file is root-owned `0640`; no secret appears in any log line, environment dump, report, or migration file; the API is not reachable without a session. | +| L-17 | RLS invariant | `relforcerowsecurity` is **false** for `organizations`, `projects`, `users`, `user_roles`; `app.*` functions and the tables they read share one owner; no superuser owns any object in schema `public` or `app`. | +| L-18 | Concurrency | Two reviewers editing the same `translation_units` row: the second receives a conflict (row_version), never a silent overwrite. | +| L-19 | **Unmapped history** | Records that cannot be mapped appear in `legacy_id_map` with `mapping_state` set and are **never** dropped; the monthly report is non-empty and actionable. | +| L-20 | Application regression | The three verified defects are fixed and tested: (a) `GET /api/export/download/../export_secret/x.mp3` returns 403; (b) an upload named `../../../evil.txt` cannot escape `data_dir/uploads`; (c) `active_project=/tmp/evil` is rejected. | + +--- + +## M. Open questions and blockers + +### M.1 BLOCKERS — must be resolved before any implementation + +| ID | Blocker | Evidence | Resolution | +|---|---|---|---| +| **B1** | **The worker role cannot read any project and cannot attribute an audit event.** All RLS policies require `app.current_user_id()`; the worker sets none, so it matches zero rows. `audit_events_insert` requires `actor_user_id = app.current_user_id()`, so worker actions are anonymous. | `001:471-473, 548-552` | H1/H2: `service_identities` + `app.current_service_identity_id()` + lease-scoped policy branch + `audit_events.actor_service_identity_id`. **Do not work around it by letting the worker set `app.user_id`.** | +| **B2** | **No permission exists for generating audio, editing speakers, casting voices, running analysis, or destroying project state.** The prompt requires authorization checks for exactly these. | `001:378-391` | H19. Without it, the contract is unimplementable as written. | +| **B3** | **`mfa_required` is stored but never enforced**, and `sessions` has no `mfa_verified_at`. A user flagged `mfa_required=true` can hold a fully valid session. | `001:30, 120-129` | H17 + `app.resolve_session`. | +| **B4** | **New functions in schema `app` are `EXECUTE`-able by `PUBLIC` by default.** `REVOKE ALL ON ALL FUNCTIONS` at `001:580` was a one-time statement with no `ALTER DEFAULT PRIVILEGES` behind it. Every future migration that adds a SECURITY DEFINER helper silently publishes it. | `001:580` | H23, applied **before** any further migration. | +| **B5** | **`users.status` and `organizations.status` are not enforced by any policy.** Suspension does not deny access at the database layer. | `001:28-29, 15-17` | H20: enforce inside `app.resolve_session`, plus a documented decision that RLS is not the suspension gate. | + +### M.2 HIGH findings (not blockers, must be scheduled) + +- **H-1:** superuser ownership of all RLS machinery (§J.2) — H24. +- **H-2:** `voice_manager` cannot see any project (§J.3) — H22. +- **H-3:** project-only members cannot read their organization (§J.4) — H22. +- **H-4:** `app.user_id` impersonation by any credential holder (§J.1) — accepted residual + risk with six mandatory compensating controls. +- **H-5:** no admin session revocation (§D.3) — H21. +- **H-6:** `users.email` mutability undermines historical audit identity (§J.5) — H15. +- **H-7:** `umbracontrol_backup` gets no `SELECT` on tables created after `001` until H23. +- **H-8:** the four divergent legacy DDL authorities (§C.3) mean the legacy schema cannot be + trusted as a backfill source without per-database introspection. + +### M.3 DECISIONS requiring human adjudication + +- **DECISION-1 (highest).** This document **reverses** `specs/19-enterprise-control-plane.md` + §2.3 and §16 M4, which specify a one-way read-only import with UmbraAudio unchanged. The new + instruction makes the enterprise DB authoritative for the audio system. Confirm the reversal; + spec 19 should then be amended so the two documents do not contradict each other. + *Recommendation:* adopt the new model, and amend spec 19 §2.3/§16 M4 to "the enterprise + database is authoritative; UmbraAudio is a client, with the filesystem retained as the audio + working area." Spec 19 §1's prohibition on control-plane tables inside local SQLite stands. +- **DECISION-2.** `specs/18-UmbraTrans.md` §4.2 forbids UmbraTrans from modifying UmbraAudio + and specifies an online-backup snapshot protocol (§4.2.1). Under the new model, UmbraTrans + reads the same authoritative database instead. Confirm that §4.2.1 is superseded — this + *simplifies* spec 18 and removes the snapshot-consistency problem entirely. +- **DECISION-3.** Org-scoped `voice_manager` project visibility: metadata-only (recommended) or + full membership required? +- **DECISION-4.** Should `audio_engineer` and `producer` roles be added, or is + `project_manager` + `audio_director` sufficient? +- **DECISION-5.** Retention: how long are Class C artifacts and legacy SQLite kept after + cutover? The plan assumes **indefinite read-only retention**. +- **DECISION-6.** Which volume actually hosts object storage, and is it on the same host as + PostgreSQL? Object storage is a **hard prerequisite** for §G invariant 8; until it exists, + artifacts stay on the local filesystem and the enterprise DB holds only `object_key` values + that resolve locally. +- **DECISION-7.** Is the identity provider external OIDC, or is the `bootstrap` provider + promoted to first-class? `001` defaults `auth_provider='bootstrap'`, which is not a + production identity mechanism. +- **DECISION-8.** Does the audio UI get retired in favour of the shared review UI, or do both + ship? This determines whether the J.6 fixes are a prerequisite or parallel work. +- **DECISION-9.** Scope of phase 2 (deferred in §E): the UmbraTrans translation-design tables + (`semantic_packages`, `identity_registries`, `glossary_revisions`, `memory_revisions`, + `alignment_links`, `target_render_spans`) plus the commercial voice subsystem + (`voice_entitlements`, `voice_reservations`, `voice_rate_cards`, `voice_invoices`). + +### M.4 Explicitly out of scope for this phase + +Connecting to any database; any DDL, DML, migration, grant or role change; moving or rewriting +any UmbraAudio file; stopping or pausing the audio worker; producing migration SQL; requesting +or using any credential, token, key or connection string. + +--- + +## N. Machine-readable contract + +```yaml +contract: + name: umbra-enterprise-database-integration + version: 1.0.0-draft + date: 2026-09-25 + phase: discovery_and_design + status: awaiting_human_approval + production_changes_made: false + database_contacted: false + migration_sql_emitted: false + +baseline: + schema_reference: infra/postgres/001_control_plane.sql + schema_version: 001_control_plane + sha256_verified_identical_copy_at: /Volumes/Dock/worstauthor/oldbox/postgres/001_control_plane.sql + existing_tables: + - schema_migrations + - organizations + - users + - projects + - roles + - permissions + - role_permissions + - user_roles + - invitations + - sessions + - mfa_methods + - bootstrap_tokens + - audit_events + verified_absent: + - no source/work/edition tables + - no locale tables + - no analysis tables + - no speaker or voice tables + - no license or entitlement tables + - no translation tables + - no audio run or artifact tables + - no job, lease, attempt or idempotency tables + - no service identity table + - no project_members table + - no password column on users + existing_roles: [platform_admin, organization_admin, member, project_owner, + project_manager, translator, proofreader, audio_director, + voice_manager, release_manager, viewer] + existing_permissions: [platform.manage, organization.manage, organization.users.manage, + organization.projects.manage, project.access.manage, + project.source.edit, project.translation.edit, + project.translation.review, project.audio.approve, + project.release.approve, voice.catalog.manage, artifact.download] + ownership: + tables_and_app_functions_owner: postgres # SUPERUSER - see finding H-1 + database_owner: umbracontrol_owner # NOLOGIN, owns nothing else + migrations_applied_as: postgres # bootstrap_umbracontrol_database.sh:243 + rls: + enabled_on: [organizations, projects, users, roles, permissions, role_permissions, + user_roles, invitations, sessions, mfa_methods, bootstrap_tokens, + audit_events] + forced_on: [] + invariant: >- + app.* SECURITY DEFINER functions and the tables they read MUST share one owner, + and RLS MUST NOT be FORCE'd on organizations, projects, users, user_roles. + Violating either causes 42P17 infinite recursion in policy for relation "projects". + +current_state: + authentication: none + authorization_checks: 0 + routes_total: 129 + mutating_endpoints: 65 + project_identifier: unsigned browser cookie "active_project", max_age 1 year + job_queue: none + job_identity: output filename derived from (chapter_id, utterance_index) + leases_or_heartbeats: none + retry: linear sleep, innermost loop only; batch driver has none + durable_utterance_identity: none + artifact_checksums: none + checksummed_content: voice reference fingerprint only (sha256 truncated to 16 hex) + local_accounts_to_migrate: 0 + legacy_identity_strings: ["'user' in voice_review_history.reviewed_by", + "'user' in exception_reviews.resolved_by"] + legacy_speaker_stores: 5 + legacy_ddl_authorities: 4 + enforced_foreign_keys: false + check_constraints: 0 + project_identifier_path_pattern: //.umbra/project.db + artifact_path_patterns: + utterance: .umbra/chapter_audio/ch{N}/utt_{utterance_index:04d}.wav + marker: .umbra/chapter_audio/ch{N}/utt_{utterance_index:04d}.wav.txt + chapter_master: .umbra/chapter_audio/ch{N}_full.wav + chapter_mp3: .umbra/chapter_audio/ch{N}_full.mp3 + book_master: .umbra/export/{m4b_meta.output_name} + ar_track: .umbra/export/ar_package/{Base}_{NN}_{Title}.mp3 + ar_manifest: .umbra/export/ar_package.json + voice_reference: .umbra/voices/{name_slug}_ref.wav + +proposed_tables: + - {name: works, grain: one bibliographic work, tenancy: organization, writer: human_api, pk: uuid, delta: ADD_TABLE} + - {name: source_documents, grain: one ingested file version, tenancy: organization+project, writer: ingestion, pk: uuid, delta: ADD_TABLE} + - {name: source_revisions, grain: one frozen source revision, tenancy: organization+project, writer: source_editor, pk: uuid, delta: ADD_TABLE} + - {name: source_chapters, grain: one chapter per revision, tenancy: organization+project, writer: analysis, pk: uuid, delta: ADD_TABLE} + - {name: source_units, grain: one addressable source unit, tenancy: organization+project, writer: analysis, pk: uuid, delta: ADD_TABLE, note: generated source_unit_key = sha256(source_revision_id||ordinal||occurrence||text_sha256)} + - {name: speakers, grain: one logical speaker per project, tenancy: organization+project, writer: analysis+human, pk: uuid, delta: ADD_TABLE} + - {name: speaker_aliases, grain: one alias per speaker, tenancy: organization+project, writer: analysis+human, pk: uuid, delta: ADD_TABLE} + - {name: analysis_runs, grain: one analysis execution, tenancy: organization+project, writer: worker, pk: uuid, delta: ADD_TABLE} + - {name: analysis_snapshots, grain: one immutable result payload, tenancy: organization+project, writer: worker, pk: uuid, delta: ADD_TABLE} + - {name: voice_profiles, grain: one voice profile per org, tenancy: organization, writer: voice_manager, pk: uuid, delta: ADD_TABLE} + - {name: voice_profile_revisions, grain: one profile change, tenancy: organization, writer: voice_manager, pk: uuid, delta: ADD_TABLE} + - {name: voice_licenses, grain: one licence grant, tenancy: organization, writer: voice_manager, pk: uuid, delta: ADD_TABLE} + - {name: voice_casts, grain: one speaker cast per revision, tenancy: organization+project, writer: audio_director, pk: uuid, delta: ADD_TABLE} + - {name: project_locales, grain: one locale lane per project, tenancy: organization+project, writer: translation, pk: uuid, delta: ADD_TABLE} + - {name: translation_runs, grain: one translation execution, tenancy: organization+project, writer: translation_worker, pk: uuid, delta: ADD_TABLE} + - {name: translation_units, grain: one target render span, tenancy: organization+project, writer: translator, pk: uuid, delta: ADD_TABLE} + - {name: audio_runs, grain: one generation run, tenancy: organization+project, writer: audio_worker, pk: uuid, delta: ADD_TABLE} + - {name: audio_run_casts, grain: one cast row per run+speaker, tenancy: organization+project, writer: audio_worker, pk: uuid, delta: ADD_TABLE} + - {name: artifacts, grain: one immutable artifact version, tenancy: organization+project, writer: producer, pk: uuid, delta: ADD_TABLE, bytes_in_postgres: false} + - {name: artifact_links, grain: one lineage edge, tenancy: organization+project, writer: producer, pk: bigint_identity, delta: ADD_TABLE} + - {name: jobs, grain: one schedulable work unit, tenancy: organization+project, writer: api+worker, pk: uuid, delta: ADD_TABLE, idempotency: "UNIQUE(project_id, job_type, idempotency_key)", key_is_never_a_filename: true} + - {name: job_attempts, grain: one attempt per job, tenancy: organization+project, writer: worker, pk: bigint_identity, delta: ADD_TABLE} + - {name: job_leases, grain: one live lease per job, tenancy: organization+project, writer: worker, pk: uuid, delta: ADD_TABLE, unique: "UNIQUE(job_id) WHERE released_at IS NULL"} + - {name: service_identities, grain: one non-human principal, tenancy: organization_or_platform, writer: platform_admin, pk: uuid, delta: ADD_TABLE} + - {name: legacy_id_map, grain: one legacy key mapping, tenancy: organization+project, writer: migration_tool, pk: uuid, delta: ADD_TABLE} + +alters_existing: + - {table: projects, change: "ADD work_id, source_locale, row_version, migration_state, legacy_project_path", delta: ALTER_TABLE} + - {table: users, change: "ADD password_disabled_at", delta: ALTER_TABLE} + - {table: sessions, change: "ADD mfa_verified_at, idle_deadline_at", delta: ALTER_TABLE} + - {table: audit_events, change: "ADD actor_service_identity_id, principal_kind, actor_subject_snapshot, session_id", delta: ALTER_TABLE} + - {table: permissions, change: "ADD missing audio/speaker/cast/analysis/destroy/self-manage keys", delta: BACKFILL} + - {table: audit_events, change: "NO foreign keys on organization_id/project_id - audit must survive deletion", delta: NO_CHANGE} + +invariants: + - id: G1 + rule: one source ebook has one stable source identity + enforced_by: "UNIQUE (project_id, sha256) on source_documents" + - id: G2 + rule: one canonical project contains audiobook and translation work + enforced_by: "NOT NULL project_id + composite FK (project_id, organization_id)" + - id: G3 + rule: analysis has immutable versions + enforced_by: "BEFORE UPDATE triggers on source_revisions/analysis_runs/analysis_snapshots" + - id: G4 + rule: speaker identity is stable across languages + enforced_by: "speakers.id + source_unit_key; translated names never written to speakers" + - id: G5 + rule: voice reuse only when language/accent/consent/licence allow + enforced_by: "deny-by-default trigger on voice_casts + app.voice_licence_covers()" + - id: G6 + rule: every translation references source book, project and analysis version + enforced_by: "NOT NULL source_revision_id + nightly reconciliation" + - id: G7 + rule: every audiobook references project, translation when applicable, cast snapshot, generation run + enforced_by: "NOT NULL audio_run_id + CHECK + nightly lineage validator" + - id: G8 + rule: translated ebooks and audiobooks never become duplicate source records + enforced_by: "artifact_links acyclicity check + application rule" + - id: G9 + rule: audio production continues before, during and after cutover + enforced_by: "degraded-mode lease; PG failure logged, never fatal" + - id: G10 + rule: large media in object storage only + enforced_by: "no bytea/blob column in any new table; test asserts absence" + - id: G11 + rule: every write is idempotent and safe to retry + enforced_by: "natural unique constraints on every table + replay tests" + - id: G12 + rule: concurrent workers cannot silently overwrite one another + enforced_by: "partial UNIQUE on job_leases + FOR UPDATE SKIP LOCKED + row_version" + - id: G13 + rule: service identities are not arbitrary client-supplied user ids + enforced_by: "CHECK (principal_kind='service' => actor_service_identity_id NOT NULL AND actor_user_id IS NULL)" + - id: G14 + rule: translation UI discovers authorized projects without a separate account system + enforced_by: "single app.authorized_projects() used by both clients" + - id: G15 + rule: existing staleness semantics preserved, including the fx_version rule + enforced_by: "regression test; generation/staleness.py:98" + +identity_contract: + canonical_users_table: users + canonical_projects_table: projects + membership_model: user_roles where scope='project' and status='active' + project_members_table: REJECTED + second_password_store: FORBIDDEN + authentication_entry_point: "app.resolve_session(p_token_hash text) SECURITY DEFINER" + project_context: "derived server-side per request; app.can_access_project(project_id) recomputed every request; never trusted from request" + retired_mechanism: "unsigned active_project browser cookie" + session_absolute_lifetime: 7d + session_idle_lifetime: 8h + session_storage: "sha256(token) only" + session_revoke_all: "app.revoke_user_sessions(uuid) SECURITY DEFINER - NEW" + suspension_enforcement: app.resolve_session + mfa_enforcement: app.resolve_session, requires sessions.mfa_verified_at - NEW + human_principal: users row via sessions + worker_principal: service_identities row + job_leases row + worker_impersonation: PROHIBITED + session_scope: user-global; organisation context re-derived and re-authorised per request + +security: + guc_impersonation: + guc: app.user_id + settable_by_any_role: true + revocable: false + rationale: "custom placeholder GUCs are not in pg_settings, so REVOKE SET ON PARAMETER is unavailable" + classification: ACCEPTED_RESIDUAL_RISK + compensating_controls: + - credential in root-owned 0640 file, service-group readable only + - "SET LOCAL inside an explicit transaction by trusted framework code only" + - never derived from request body, query, cookie, header or file (lint-enforced) + - database reachable only by the API + - non-superuser owner role (H24) + - incident-response logging available + superuser_in_data_path: true + force_rls_permitted: false + new_app_functions_public_executable_by_default: true + default_privileges_configured: false + cookie_flags_present: [max_age] + cookie_flags_missing: [httponly, samesite, secure] + server_binding: "0.0.0.0:8420 plain HTTP behind LAN reverse proxy" + application_auth: none + csrf_protection: none + security_headers: none + trusted_host: none + download_authorization: none + verified_application_defects: + - {id: A1, severity: critical, finding: "load_config accepts absolute paths verbatim; cookie is a path-selection primitive", evidence: "umbraaudio/config.py:156-158"} + - {id: A2, severity: critical, finding: "download traversal via string-prefix containment instead of path components", evidence: "umbraaudio/web/routes.py:5049"} + - {id: A3, severity: critical, finding: "upload filename used raw, no Path().name; writes outside uploads/", evidence: "umbraaudio/web/routes.py:633,640,647,652"} + - {id: A4, severity: high, finding: "activate-project existence check is unreachable dead code", evidence: "umbraaudio/web/routes.py:525 with config.py:165"} + - {id: A5, severity: critical, finding: "65 mutating endpoints with zero authorization, including rmtree of voices/ and DELETE FROM utterances", evidence: "umbraaudio/web/routes.py:4674,4711,5097"} + - {id: A6, severity: high, finding: "unauthenticated model load/unload of GB-scale models", evidence: "umbraaudio/web/routes.py:5744,5775,5814"} + - {id: A7, severity: high, finding: "second CWD-relative project resolution path bypasses load_config and its checks", evidence: "umbraaudio/web/__init__.py:51"} + - {id: A8, severity: medium, finding: "unauthenticated outbound-request proxy to dictionaryapi.dev", evidence: "umbraaudio/web/routes.py:1847"} + - {id: A9, severity: medium, finding: "404-vs-403 existence oracle: existence checked before containment", evidence: "umbraaudio/web/routes.py:5047"} + - {id: A10, severity: medium, finding: "ChapterBuilder has no lock; concurrent builds write the same files", evidence: "umbraaudio/web/routes.py:5097-5110"} + +cutover: + strategy: per_project_flagged_dual_write + default_repository: SqliteProjectRepository + phase_order: [P0_inventory, P1_adapter_boundary, P2_mirror_backfill_shadow_read, + P3_enterprise_schema, P4_dual_write, P5_dry_run_primary, + P6_enterprise_authoritative, P7_retention] + flag: projects.migration_state + flag_values: [legacy, mirroring, dual_write, enterprise_authoritative, retired] + global_switch: FORBIDDEN + dual_write_failure_policy: log_and_continue + degraded_mode: + lease_validity_hours: 6 + on_expiry: halt new runs, complete in-flight rendering, alert operator + authoritative_when: + - three consecutive nightly reconciliations pass + - artifact sha256 all match + - zero mapping_state=unmapped for the project + - one full generation run completed with enterprise store primary + - restore drill passed + rollback: + method: single flag flip + stops_worker: false + deletes_source_data: false + retains_enterprise_rows: true + marks_rows: superseded_by_local + unmappable_records: + table: legacy_id_map + states: [unmapped, ambiguous, rejected] + dropped: false + +schema_delta_counts: + add_table: 24 + alter_table: 4 + add_column: 12 + add_constraint: 6 + add_index: 30 + enable_prove_rls: 21 + backfill: 3 + grant_revoke: 2 + new_service_role: 2 + no_change: 2 + +blockers: + - {id: B1, severity: BLOCKER, title: "worker role can read no project and cannot attribute audit events", evidence: "001:471-473,548-552", fix: [H1, H2, H15, H22], workaround_forbidden: "setting app.user_id for the worker"} + - {id: B2, severity: BLOCKER, title: "no permission exists for audio generation, speaker editing, voice casting, analysis, or state destruction", evidence: "001:378-391", fix: [H19]} + - {id: B3, severity: BLOCKER, title: "mfa_required stored but never enforced; sessions lack mfa_verified_at", evidence: "001:30,120-129", fix: [H17, H20]} + - {id: B4, severity: BLOCKER, title: "new functions in schema app are EXECUTE-able by PUBLIC by default", evidence: "001:580", fix: [H23]} + - {id: B5, severity: BLOCKER, title: "users.status and organizations.status not enforced by any policy", evidence: "001:28-29,15-17", fix: [H20]} + +open_decisions: + - {id: DECISION-1, priority: highest, question: "confirm reversal of spec 19 section 2.3 and 16 M4 (one-way import) in favour of an authoritative enterprise database", recommendation: "adopt, and amend spec 19"} + - {id: DECISION-2, question: "confirm spec 18 section 4.2.1 online-backup snapshot protocol is superseded by shared-database reads", recommendation: "adopt; removes snapshot-consistency problem"} + - {id: DECISION-3, question: "org-scoped voice_manager project visibility: metadata-only or full membership", recommendation: "metadata-only"} + - {id: DECISION-4, question: "add audio_engineer and producer roles", recommendation: "defer; project_manager plus audio_director is sufficient"} + - {id: DECISION-5, question: "retention period for artifacts and legacy SQLite after cutover", recommendation: "indefinite read-only"} + - {id: DECISION-6, question: "object storage host and availability", recommendation: "must exist before artifact registration; until then object_key resolves locally"} + - {id: DECISION-7, question: "identity provider: external OIDC or promoted bootstrap provider", recommendation: "external OIDC"} + - {id: DECISION-8, question: "is the audio UI retired in favour of the shared review UI", recommendation: "decide early; determines whether defect fixes are prerequisite or parallel"} + - {id: DECISION-9, question: "phase 2 scope: translation-design tables and commercial voice subsystem", recommendation: "defer to a separate approved phase"} + +acceptance_tests: 20 +backup_requirements: 7 +production_actions_taken: none +next_step: "human review and approval of this contract; no connection, migration, or application data write until approved" +``` + +--- + +**End of contract. No database was contacted, no migration was executed, no production data was +written, and no secret was used or requested. Awaiting human approval.**