diff --git a/AGENTS.md b/AGENTS.md index 53afe79..cf7179c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -171,6 +171,66 @@ Each pass has quality gates. Quality is the primary concern; generation speed is ## Current Session Summary +### Session: Umbra system node declaration + cross-machine contract (spec 21) +- **Context:** another machine/tab is moving UmbraTrans to the main Umbra system on + `oldbox` (192.168.123.14). Four questions were raised for this machine: (1) align the + data model and copy UmbraAudio data into the system, verify, transition later; (2) every + machine becomes a node that receives *any* task, not just audio; (3) how changes are + managed across products, with the controller owning requirements; (4) does a central + server take over the UI, and how do code updates get deployed/tested. +- **New `specs/21-umbra-system-nodes-tasks-and-changes.md`** — the shared cross-machine + document. Four workstreams (WS1 data alignment, WS2 nodes/tasks, WS3 change management, + WS4 UI topology), each with what already exists, what is missing, and a proposal. + Proposals are **drafts for the controller to ratify**; nothing is decided here. +- **New `infra/umbra-node.json`** — this machine's declared node profile + (`node_id: umbraaudio-m4`, role worker, products `[umbraaudio]`, services, capability + list, resource pools, constraints, project list, lifecycle, schema fingerprints, and an + honest "facts_for_the_controller" block). **No secrets** — ids, names, counts only. +- **New `scripts/umbra_node_report.py`** — read-only node report. Proves what the + controller needs to align: git state, listening ports, `.env` KEY NAMES only (never + values), per-project row counts, and a **schema fingerprint** (sha256 over sorted + `table.column:type`, so `ALTER TABLE` column reordering does not read as drift). + `--json` for machine consumption; `--check` compares the declared profile to measured + reality and exits 2 on drift. Verified: `--check` exits 0, ruff clean, `--json` parses. + Deliberately stable-facts-only drift (contract version, project set, schema fingerprints) + — git HEAD and live ports are reported but never drift, or every commit would "break" it. +- **Measured on this machine (2026-10-01):** 4 projects, **2 distinct schema fingerprints** — + `0078569a0fa6bc70` (Last Dwarf, DealHunter: current shape) and `367d5df9e546dbba` + (Treasure Island, Vocals: no `name_tokens`, missing 9 utterance columns). Confirms the + manifest's drift finding is real and machine-checkable. Speaker-count disagreement is now + **Last Dwarf 93/68/71**, **DealHunter 61/64/97** (93 after the designation merge; refs up + from 94). Full suite re-verified: **461 passed / 8 failed** — all 8 pre-existing/parallel + (2 `test_pronunciation` helper_url mock, 6 `voice_profile_manager` accent). +- **`manifest.md` is now explicitly a dated measurement, not a live view**, with an update + log. Every other §1.1 table count re-verified unchanged. This is the single most + important fix for cross-machine trust: another machine reading the manifest must know + whether a number is current. +- **Key answers proposed in spec 21** (all need ratification): + - *Authority*: controller owns identity/tenancy/requirements/definitions/task state/ + lineage/release; the node owns its working area and **keeps working when the controller + is down** (production availability outranks synchronization). Migration state is per + project: `legacy → mirroring → dual_write → authoritative → retired`. + - *Nodes*: stable `node_id`, declared `node_contract_version`, capability advertisement, + task/result envelopes with `report_back` evidence, and **refusal as a valid audited + outcome**. This machine must refuse any task needing LLM + local TTS co-resident. + - *Changes*: requirement catalog in the controller; four classes (`local_product`, + `cross_product`, `shared_definition`, `operational`); distributed, then **confirmed + with evidence**, never pushed. Code ships by git tag per product repo; the controller + carries the compatibility matrix and the requirement, not the code. + - *UI*: **hybrid.** Controller gets the system portal (node health, task state, + requirement conformance, definition versions, releases). The audio review UI **stays** + on this machine: it is artifact-native (WAV players, per-utterance regen, markers) and + has **no authentication at all**, so centralizing it would publish an unauthenticated + mutation surface. Deep links are short-lived and signed; the worker footer shows + `node_id` + `definition_set_version` + last confirmation time. +- **Highest-leverage local work identified (no controller dependency):** add a + **content-addressed `source_unit_key`** recorded at generation time (additive column, + filenames untouched) — this is the one *blocking* identity concept that needs nothing + from the controller; and **extract the aligned definition set from code into versioned + data** (`manifest.md` §14.2). Both make every later migration phase cheaper. +- **No UmbraAudio behaviour changed.** New files only, plus the manifest update log and a + spec index row. + ### Session: Designation scanner — capture the ch5 speakers discovery missed - **Discovery re-run on ch5 with the new over-detection prompt rules** (`gemma-4-e4b`, ~22 tok/s; stopped Qwen TTS first per the no-co-residency rule, restarted it after). diff --git a/definitions/umbraaudio-defs.json b/definitions/umbraaudio-defs.json new file mode 100644 index 0000000..caa8c61 --- /dev/null +++ b/definitions/umbraaudio-defs.json @@ -0,0 +1,355 @@ +{ + "definition_set_version": "defs-2026-10-03", + "product": "umbraaudio", + "provenance": { + "artifact_kind": { + "_authority": "manifest.md \u00a711, specs/20 \u00a7F.19", + "ar_package": "declared", + "chapter_stitch": "declared", + "character_profile": "declared", + "m4b": "declared", + "mp3_track": "declared", + "source_document": "declared", + "utterance_wav": "declared", + "voice_prompt_vmatrix": "declared", + "voice_reference": "declared" + }, + "artifact_status": { + "_authority": "controller \u00a72.1 (artifacts.status)", + "final": "declared", + "intermediate": "declared", + "superseded": "declared" + }, + "attribute_absence_reason": { + "_authority": "controller D13", + "in_progress": "declared", + "not_applicable": "declared", + "not_attempted": "declared", + "superseded": "declared" + }, + "attribution_evidence_kind": { + "_authority": "umbraaudio/analysis/clarity.py", + "address_only": "extracted", + "name": "extracted", + "none": "extracted", + "pronoun": "extracted", + "role": "extracted" + }, + "character_kind": { + "_authority": "specs/18 \u00a75.4a", + "ai": "declared", + "group": "declared", + "named_character": "declared", + "narrator": "declared", + "recurring_role": "declared", + "scene_role_instance": "declared" + }, + "character_persistence": { + "_authority": "controller D11 (specs/21 \u00a79)", + "book_local": "declared", + "series_persistent": "declared" + }, + "emotion": { + "_authority": "umbraaudio/generation/chapter.py EMOTION_OPTIONS", + "angry": "extracted", + "calm": "extracted", + "commanding": "extracted", + "confident": "extracted", + "confused": "extracted", + "desperate": "extracted", + "determined": "extracted", + "disgusted": "extracted", + "dramatic": "extracted", + "excited": "extracted", + "fearful": "extracted", + "firm": "extracted", + "gentle": "extracted", + "happy": "extracted", + "hesitant": "extracted", + "hopeful": "extracted", + "melancholy": "extracted", + "neutral": "extracted", + "romantic": "extracted", + "sad": "extracted", + "sarcastic": "extracted", + "serious": "extracted", + "surprised": "extracted", + "tender": "extracted", + "tired": "extracted", + "urgent": "extracted", + "whisper": "extracted" + }, + "job_capability": { + "_authority": "specs/21 \u00a74.7 + controller \u00a72.3", + "analysis.character_discovery": "declared", + "analysis.name_token_scan": "declared", + "analysis.speaker_attribution": "declared", + "analysis.speaker_clarity": "declared", + "audio.normalize_loudness": "declared", + "audio.parse_manuscript": "declared", + "audio.render_utterances": "declared", + "audio.segment_utterances": "declared", + "audio.stitch_chapters": "declared", + "character.consistency_check": "declared", + "character.personality_analysis": "declared", + "export.m4b": "declared", + "export.mp3_package": "declared", + "review.speaker_QA": "declared", + "voice.attribute_measurement": "declared", + "voice.capture_character": "declared", + "voice.reference_generation": "declared" + }, + "migration_state": { + "_authority": "manifest.md \u00a714.1", + "authoritative": "declared", + "dual_write": "declared", + "legacy": "declared", + "mirroring": "declared", + "retired": "declared" + }, + "name_token_kind": { + "_authority": "specs/17 \u00a73.4", + "name_tag": "extracted", + "speaker_name": "extracted" + }, + "name_token_status": { + "_authority": "specs/17 \u00a73.4", + "assigned": "extracted", + "created": "extracted", + "dismissed": "extracted", + "unresolved": "extracted" + }, + "pace": { + "_authority": "umbraaudio/generation/chapter.py PACE_OPTIONS", + "fast": "extracted", + "moderate": "extracted", + "slow": "extracted" + }, + "phonetic_provenance": { + "_authority": "manifest.md \u00a714.2", + "explicit": "declared", + "ipa_derived": "declared" + }, + "post_tts_effect_trait": { + "_authority": "umbraaudio/generation/audio_effects.py TRAIT_EFFECTS", + "boom": "extracted", + "breathy": "extracted", + "chorusing": "extracted", + "deep": "extracted", + "distortion": "extracted", + "echo": "extracted", + "gravelly": "extracted", + "hoarse": "extracted", + "nasal": "extracted", + "noise_inject": "extracted", + "pitch_shift": "extracted", + "radio_filter": "extracted", + "raspy": "extracted", + "reverb": "extracted", + "tremolo": "extracted", + "whispery": "extracted" + }, + "project_lifecycle": { + "_authority": "manifest.md \u00a71.4, controller \u00a72.1 (projects.status)", + "active": "declared", + "analyzed": "declared", + "archived": "declared", + "experimental": "declared", + "on_hold": "declared", + "parsed": "declared", + "released": "declared" + }, + "qa_issue_type": { + "_authority": "umbraaudio/analysis/attribution_qa.py", + "address_mismatch": "extracted", + "long_monologue": "extracted", + "narration_speaker_mismatch": "extracted", + "narrator_dialogue": "extracted", + "role_tag_guessed": "extracted", + "speech_tag_mismatch": "extracted" + }, + "utterance_type": { + "_authority": "specs/16 \u00a75a, umbraaudio/database.py SCHEMA default 'narration'", + "dialogue": "declared", + "narration": "declared", + "scene_break": "declared" + }, + "voice_capture_mode": { + "_authority": "umbraaudio/generation/chapter_builder.py, controller \u00a72.2", + "embedding": "extracted", + "icl": "extracted" + }, + "voice_variant_reason": { + "_authority": "controller D6", + "disguise_roleplay": "declared", + "sarcasm_mimic": "declared", + "technical_degradation": "declared" + } + }, + "vocabularies": { + "artifact_kind": [ + "ar_package", + "chapter_stitch", + "character_profile", + "m4b", + "mp3_track", + "source_document", + "utterance_wav", + "voice_prompt_vmatrix", + "voice_reference" + ], + "artifact_status": [ + "final", + "intermediate", + "superseded" + ], + "attribute_absence_reason": [ + "in_progress", + "not_applicable", + "not_attempted", + "superseded" + ], + "attribution_evidence_kind": [ + "address_only", + "name", + "none", + "pronoun", + "role" + ], + "character_kind": [ + "ai", + "group", + "named_character", + "narrator", + "recurring_role", + "scene_role_instance" + ], + "character_persistence": [ + "book_local", + "series_persistent" + ], + "emotion": [ + "angry", + "calm", + "commanding", + "confident", + "confused", + "desperate", + "determined", + "disgusted", + "dramatic", + "excited", + "fearful", + "firm", + "gentle", + "happy", + "hesitant", + "hopeful", + "melancholy", + "neutral", + "romantic", + "sad", + "sarcastic", + "serious", + "surprised", + "tender", + "tired", + "urgent", + "whisper" + ], + "job_capability": [ + "analysis.character_discovery", + "analysis.name_token_scan", + "analysis.speaker_attribution", + "analysis.speaker_clarity", + "audio.normalize_loudness", + "audio.parse_manuscript", + "audio.render_utterances", + "audio.segment_utterances", + "audio.stitch_chapters", + "character.consistency_check", + "character.personality_analysis", + "export.m4b", + "export.mp3_package", + "review.speaker_QA", + "voice.attribute_measurement", + "voice.capture_character", + "voice.reference_generation" + ], + "migration_state": [ + "authoritative", + "dual_write", + "legacy", + "mirroring", + "retired" + ], + "name_token_kind": [ + "name_tag", + "speaker_name" + ], + "name_token_status": [ + "assigned", + "created", + "dismissed", + "unresolved" + ], + "pace": [ + "fast", + "moderate", + "slow" + ], + "phonetic_provenance": [ + "explicit", + "ipa_derived" + ], + "post_tts_effect_trait": [ + "boom", + "breathy", + "chorusing", + "deep", + "distortion", + "echo", + "gravelly", + "hoarse", + "nasal", + "noise_inject", + "pitch_shift", + "radio_filter", + "raspy", + "reverb", + "tremolo", + "whispery" + ], + "project_lifecycle": [ + "active", + "analyzed", + "archived", + "experimental", + "on_hold", + "parsed", + "released" + ], + "qa_issue_type": [ + "address_mismatch", + "long_monologue", + "narration_speaker_mismatch", + "narrator_dialogue", + "role_tag_guessed", + "speech_tag_mismatch" + ], + "utterance_type": [ + "dialogue", + "narration", + "scene_break" + ], + "voice_capture_mode": [ + "embedding", + "icl" + ], + "voice_variant_reason": [ + "disguise_roleplay", + "sarcasm_mimic", + "technical_degradation" + ] + } +} diff --git a/infra/controller-handoff-2026-10-03.md b/infra/controller-handoff-2026-10-03.md new file mode 100644 index 0000000..788daa5 --- /dev/null +++ b/infra/controller-handoff-2026-10-03.md @@ -0,0 +1,176 @@ +# Controller handoff — UmbraAudio node `umbraaudio-m4`, 2026-10-03 + +**To:** the Umbra control plane on `oldbox` (`192.168.123.14:8442`) +**From:** node `umbraaudio-m4` (Mac mini M4), product `umbraaudio` +**Subject:** alignment proposal §5 and §6 — what is done, what is proven, what is missing +**Status:** Stage 0 verified. Stage 1 ready, blocked on one credential. + +--- + +## 1. Stage 0 — reachability: PASS, all three as documented + +``` +GET /api/health (Host: umbratrans.bdmurphy.me) -> 200 +POST /api/node/heartbeat (same Host, no credential) -> 401 +GET /api/health (no Host header) -> 400 +``` + +- `200` on health. `401 node_credential_required` on the node surface — **the correct + answer**, and evidence that authentication is working rather than being bypassed. +- The `400` without `Host` reproduces exactly as your document predicted. Noted so nobody + later reads it as the deployment being down. + +No other request was made. Nothing was sent beyond those three probes. + +## 2. §5 changes — 6 of 7 complete, 1 deliberately deferred + +| # | change | status | proof | +|---|---|---|---| +| 5.1 | `source_unit_key` | **done** | see §3 | +| 5.2 | SHA-256 per blob at generation | **done** | marker now carries the full `sha256` of the delivered bytes | +| 5.3 | Speaker model → chain mapping | **done** | alignment phase 2 evidence: per project, `speaker_names` → `characters` → `speakers`, plus every unresolved name listed | +| 5.4 | Publish vocabularies as JSON | **done** | `definitions/umbraaudio-defs.json` — 19 sets, 126 values, each with provenance | +| 5.5 | Keep schema fingerprints current | **done** | the drift detector reported its own fingerprint change rather than hiding it; profile updated | +| 5.6 | Personality version + provenance | **deferred, honestly** | no versioned personality artifact exists. What exists is unversioned LLM prose in `speakers.json`. Reported as phase 6b finding **B-03**, not glossed. | +| 5.7 | `utterance_index` decision | **recorded** | **kept as a display ordinal, made non-load-bearing.** `specs/21` §10 | + +**Why 5.6 is deferred rather than faked.** Personality is the thing your §1.3 gets right — +one per character, language-neutral, versioned. The node has an LLM character-discovery pass +that writes exactly the wrong shape: prose fields, no version, no model record, no prompt +version. Wrapping it in a version field without versioning the *inputs* would produce +artifacts that claim a lineage they do not have. Declaring it missing is cheaper than a +plausible lie. + +## 3. §5.1 in detail — the part everything else depends on + +```text +source_unit_key = sha256("uk1" | chapter_source_text_sha256 | occurrence | text_sha256) +``` + +- `occurrence` disambiguates repeated text within a chapter, in document order. Nothing in + the key is a position. +- Filenames unchanged. `utterance_index` untouched. `is_current` unaffected. +- **18,181 / 18,181** units keyed across all four projects. **0 collisions.** +- Backfill of 10,015 rows takes **0.11 s**; a second run changes **0** rows (idempotent). +- **Renumbering proven safe:** chapter 1 shifted `+100` on every `utterance_index`, then + re-stamped — **0 keys changed**. +- Old schema shapes handled: `chapter_index` is absent on the two older projects, so the + code falls back rather than assuming it. + +### Your §6.6.1 probe set, already selected + +We choose it, you send us those exact units. Per your rule — sort `source_unit_key` ASC +(byte order), every 17th from index 0, cap 200 — on `Last Dwarf`: + +- total keys: **10,015** · stride: **17** · selected: **200** +- `selection_sha256`: `851257dfafebefea8ed2e4c8616ba03f7f4e0487bba6c849eff6f7618f1e5928` + +**Our own pre-resolution of all 200: `used_index: false` for every single one.** That is +the answer to your phase 3 question, before you have sent us anything. + +## 4. §6 validation — per-phase verdict + +| phase | verdict | why | +|---|---|---| +| 1 vocabulary | **pass** | 19 sets, 126 values, provenance included | +| 2 cardinality | **fail** | evidence present; the three-way store disagreement is real. A pass would be a check that passed by doing nothing | +| 3 identity | **pass** | 200/200 resolved without the index | +| 4 lineage | blocked | needs the artifact you pick | +| 5 timeline | blocked | no time-scoped attribute model here yet | +| 6 voice matrix | blocked | voices are not captured as structured matrix values | +| 6b persistence | blocked | **persistence is not captured at all** (finding B-01) | +| 6c editions | blocked | no edition concept (finding E-01) | +| 7 round trip | blocked | needs the ingest API | + +Regenerate with `scripts/alignment_report.py --phase all`. Phases needing your side are +emitted as `blocked` **with the reason**, never omitted. + +### Phase 2, honestly + +| project | speaker_names | characters | speakers (have lines) | characters w/o lines | speakers.json | voice_profiles | ref WAVs | +|---|---|---|---|---|---|---|---| +| Last Dwarf | 62 | 62 | 61 | 32 | 93 | 68 | 71 | +| DealHunter | 55 | 52 | 51 | 10 | 61 | 64 | 97 | +| Treasure Island | 34 | 34 | 33 | 7 | 40 | 41 | 44 | +| Vocals | 0 | 0 | 0 | 2 | 2 | 2 | 2 | + +The disagreement is real and is **not** resolved by this node. Note it moved: `Last Dwarf` +`speakers.json` went 70 → 93 on 2026-09-25 when 23 genuinely missing speakers were recovered +mechanically. + +The column your §1.2 asks for is the fourth one. **32 of `Last Dwarf`'s 93 characters never +have lines** — they are characters, not speakers, and the schema must hold them as such. +That distinction is now measured rather than assumed. `DealHunter` shows the other shape: 55 +raw names collapse to **52** characters, so alias resolution is already merging three of +them. + +## 5. Six gaps found — the fifth you asked for, and five more + +Full detail in `specs/21` §11. Ranked: + +| ID | Sev | Gap | +|---|---|---| +| **G-A** | major | *"hash at generation"* — but audio is **post-processed after generation**. A generation-time hash identifies bytes that never shipped. **Your order is already correct** (`write_marker` runs after effects) — verified, and I closed the remaining hole by recording `effects: [...]` with the hash. | +| **G-B** | major | The key is anchored to `chapters.source_text`, a **second copy of the manuscript**. Any change to chapter storage silently invalidates every key in that chapter. Recommend treating the chapter-text hash as the source revision identity. | +| **G-C** | major | Your key spec contains `source_revision_id`; we have no revision concept, so we substitute a chapter-text hash. **You must store our key verbatim, not recompute it** — you cannot know our occurrence rule. | +| **G-D** | minor | Markers did not name their source unit. They do now. | +| **G-E** | major | Voice assets are keyed by **name-derived slugs**; the same slug can mean different voices in different projects. Report now emits each ref's `sha256` + `object_key`. `voice_assets` must key on bytes and carry project scope. | +| **G-F** | minor | Keys are **project-agnostic by content**. This answers your §8.3 question, and it is a tenancy question as much as a design one. If isolation is required, salt at ingest (`sha256(tenant ‖ key)`) — doing it here would make keys non-portable for no security gain. | + +## 6. Your §8.2 — the two open questions, answered + +**Q1 — a `book_local` character is revealed to recur. Merge, or forward-only link?** +**Forward-only link.** Concur with D16, and this node has direct evidence for it: the +designation work on 2026-09-25 found 23 characters that discovery had missed under +*exactly* this class — unnamed designations reused across books (`The Big One`, +`scum leader`, `Walking oldest`). Merging them would have rewritten speaker rows that +reviewers had already accepted, and in this product speaker changes invalidate audio +through the staleness markers. A forward link preserves the freeze at zero audio cost. The +retelling caveat you named is accepted and should be documented. + +**Q2 — may a new edition contradict something frozen in the previous one?** +**Yes.** A correction that only touched text would be the unusual case, and would also leave +the audio silently wrong. Practically, this node's correction path *is* a regeneration: a +changed source unit gets a new `source_unit_key`, its marker goes stale, and the clip is +re-rendered. A new edition therefore needs a new artifact — never a rewritten one, which is +what I-H already says. + +## 7. What we need from you + +1. **A `compute_nodes` row for `umbraaudio-m4` and a node credential** with scopes + `compute:read` + `compute:heartbeat`. Stage 1 goes out the moment it exists. +2. **Confirmation that `source_unit_key` is stored verbatim**, not recomputed (G-C). +3. **Your published vocabulary sets**, so we can diff mechanically instead of by argument. +4. **Answers to your own open items** that gate us: `B4` (revoke `PUBLIC` execute before any + further migration), `G2`/non-owner role, and the artifact-write scope for a node + credential — currently a node can write nothing at all, which is correct until it is + wrong. +5. **Confirmation of §5.5's expectation**: we report fingerprint drift rather than + normalising it. That is why this handoff's fingerprint differs from the previous one. + +## 8. What we have not done, on purpose + +- No data migration. `B4` is open. +- No writes to the controller's datastore. +- No renumbering or renaming of any utterance. +- No request for credentials beyond the one named above. +- No auto-update mechanism of any kind. + +## 9. Reproducing everything in this message + +```bash +# Stage 0 +curl -sS -o /dev/null -w '%{http_code}\n' -H 'Host: umbratrans.bdmurphy.me' \ + http://192.168.123.14:8442/api/health # 200 +curl -sS -o /dev/null -w '%{http_code}\n' -X POST \ + -H 'Host: umbratrans.bdmurphy.me' -H 'Content-Type: application/json' -d '{}' \ + http://192.168.123.14:8442/api/node/heartbeat # 401 + +# Alignment phases (this machine) +.venv/bin/python scripts/alignment_report.py --phase all --project "Last Dwarf" --out-dir /tmp/alignment +.venv/bin/python scripts/umbra_node_report.py --check # exit 0 +.venv/bin/python -m pytest tests/ -q # 484 passed, 8 known pre-existing +``` + +Full suite: **484 passed / 8 failed**. All 8 are pre-existing and unrelated to this work +(2 `test_pronunciation` mock, 6 `voice_profile_manager` accent). diff --git a/infra/umbra-node.json b/infra/umbra-node.json index f10b035..d9dfde3 100644 --- a/infra/umbra-node.json +++ b/infra/umbra-node.json @@ -62,6 +62,9 @@ "analysis.speaker_attribution", "analysis.speaker_clarity", "analysis.name_token_scan", + "character.personality_analysis", + "character.consistency_check", + "voice.capture_character", "voice.reference_generation", "voice.attribute_measurement", "audio.render_utterances", @@ -71,6 +74,12 @@ "export.mp3_package", "review.speaker_QA" ], + "capability_status": { + "_note": "Declared per controller §2.3 as a vocabulary even with no scheduler behind it. Status is honest about what is real today.", + "character.personality_analysis": "declared_not_implemented - what exists is unversioned LLM prose in speakers.json, not a versioned artifact (alignment phase 6b finding B-03)", + "character.consistency_check": "declared_not_implemented", + "voice.capture_character": "partial - vmatrix capture exists but is name-keyed, locale-blind, and absent for the active project (manifest.md §5.4)" + }, "resource_pools": { "cpu": {"capacity": "bounded; shares the same 16 GB as TTS", "concurrency": 1}, @@ -102,11 +111,13 @@ "Vocals": "showcase" }, - "schema_fingerprints": ["0078569a0fa6bc70", "367d5df9e546dbba"], + "schema_fingerprints": ["399285355e2d456e", "b7c9ae76bc6615bd"], "_schema_fingerprints_note": [ - "Two distinct fingerprints = real per-project schema drift.", - "0078569a0fa6bc70: current shape (has name_tokens + the 5 clarity columns) - Last Dwarf, DealHunter.", - "367d5df9e546dbba: older shape (no name_tokens, missing 9 utterance columns) - Treasure Island, Vocals.", + "Two distinct fingerprints = real per-project schema drift, deliberately NOT normalised.", + "399285355e2d456e: current shape (has name_tokens, the 5 clarity columns, and source_unit_key) - Last Dwarf, DealHunter.", + "b7c9ae76bc6615bd: older shape (no name_tokens, missing 9 utterance columns) - Treasure Island, Vocals.", + "History: 0078569a0fa6bc70 / 367d5df9e546dbba before 2026-10-03. The change is the", + "additive source_unit_key column; the drift detector reported it rather than hiding it.", "Produced by scripts/umbra_node_report.py; re-run --check after any migration." ], @@ -134,12 +145,22 @@ "exception_reviews", "voice_review_history" ], - "has_stable_utterance_id": false, + "has_stable_utterance_id": true, + "source_unit_key": { + "status": "implemented and backfilled 2026-10-03", + "derivation": "sha256('uk1' | chapter_source_text_sha256 | occurrence | text_sha256)", + "project_agnostic_by_content": true, + "coverage": "10015/10015 Last Dwarf, 4319/4319 DealHunter, 3811/3811 Treasure Island, 36/36 Vocals", + "collisions": 0, + "renumbering_proven_safe": "chapter 1 renumbered +100 -> 0 keys changed", + "resolvable_without_index": "alignment phase 3: 200/200 units resolved, 0 used utterance_index" + }, "has_stable_speaker_fk": false, "has_user_accounts": false, "has_shared_postgres_client": false, "has_heartbeat_or_lease": false, - "test_suite": "pytest, 461 passing at 2026-10-01 (8 pre-existing/parallel failures)", + "controller_stage_reached": "0 verified 2026-10-03 (health 200, heartbeat 401, no-Host 400 - all as documented)", + "test_suite": "pytest, 484 passing at 2026-10-03 (8 pre-existing/parallel failures)", "open_defects_blocking_exposure": [ "no authentication on any endpoint (spec 20 J.6)", "download path traversal via str.startswith (routes.py ~5049)", diff --git a/scripts/alignment_report.py b/scripts/alignment_report.py new file mode 100644 index 0000000..7af7aef --- /dev/null +++ b/scripts/alignment_report.py @@ -0,0 +1,507 @@ +#!/usr/bin/env python3 +"""Controller alignment reports (spec 21 §6.6) - read-only, never guesses. + +Emits the envelope the controller diffs mechanically: + + {"report": "alignment", "schema_version": 1, "phase": , "from": "umbraaudio-m4", + "node_contract_version": 1, "generated_at": ..., "verdict": pass|fail|blocked, + "evidence": {...per-phase...}, "findings": [...]} + +Two rules the controller stated and this tool enforces on itself: + +* A phase with no evidence is ``blocked``, never ``pass``. +* A field that cannot be produced is ``null`` **plus a finding**, never omitted. + +Phases that need the other side are emitted as ``blocked`` with the reason, so the absence +is visible instead of silent. + +Usage: + python scripts/alignment_report.py --phase 1 + python scripts/alignment_report.py --phase 2 + python scripts/alignment_report.py --phase 3 --project "Last Dwarf" + python scripts/alignment_report.py --phase 3-set --project "Last Dwarf" + python scripts/alignment_report.py --phase all --out-dir /tmp/alignment + python scripts/alignment_report.py --phase all --stdout-only +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import sqlite3 +import sys +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import list_projects +from umbraaudio.definitions import DEFINITION_SET_VERSION, declaration, vocabularies +from umbraaudio.source_units import key_report + +SCHEMA_VERSION = 1 +NODE_ID = "umbraaudio-m4" +NODE_CONTRACT_VERSION = 1 + +# Controller §6.6.1: we choose the probe set deterministically, they send us those units. +PROBE_STRIDE = 17 +PROBE_CAP = 200 + + +def _now() -> str: + return datetime.now(UTC).isoformat(timespec="seconds") + + +def _finding(fid: str, severity: str, description: str, affects: list[str]) -> dict[str, Any]: + return {"id": fid, "severity": severity, "description": description, "affects": affects} + + +def _envelope(phase: Any, verdict: str, evidence: dict[str, Any], findings: list) -> dict[str, Any]: + return { + "report": "alignment", + "schema_version": SCHEMA_VERSION, + "phase": phase, + "from": NODE_ID, + "node_contract_version": NODE_CONTRACT_VERSION, + "generated_at": _now(), + "verdict": verdict, + "evidence": evidence, + "findings": findings, + } + + +def _ro(project_dir: str) -> sqlite3.Connection: + db = Path(project_dir) / ".umbra" / "project.db" + return sqlite3.connect(f"file:{db.resolve()}?mode=ro", uri=True) + + +def _alias_map(project_dir: str) -> tuple[dict[str, str], list[dict[str, Any]]]: + path = Path(project_dir) / ".umbra" / "speakers.json" + if not path.exists(): + return {}, [] + profiles = json.loads(path.read_text(encoding="utf-8")).get("speakers", []) + alias: dict[str, str] = {} + for s in profiles: + primary = s.get("primary_name", "") + if primary: + alias[primary.casefold()] = primary + for a in s.get("aliases") or []: + if a: + alias[a.casefold()] = primary + return alias, profiles + + +# --------------------------------------------------------------------------- phase 1 +def phase_1() -> dict[str, Any]: + """Vocabulary diff. Both sides emit the same shape and are compared mechanically.""" + evidence = {"vocabularies": vocabularies(), "definition_set_version": DEFINITION_SET_VERSION} + findings: list[dict[str, Any]] = [] + findings.append( + _finding( + "V-01", + "minor", + "Values marked 'extracted' in provenance() are read out of executable code and " + "can change without a deployment; values marked 'declared' are fixed by a spec " + "or by the controller. Compare provenance before treating a diff as a conflict.", + ["definition_set_version"], + ) + ) + findings.append( + _finding( + "V-02", + "minor", + "'speaker' is deliberately NOT published as a character kind. Controller §1.2: a " + "speaker is a character that has lines in a work - a derived fact, not a kind.", + ["character_kind"], + ) + ) + verdict = "pass" if evidence["vocabularies"] else "blocked" + return _envelope(1, verdict, evidence, findings) + + +# --------------------------------------------------------------------------- phase 2 +def _project_phase_2(project_dir: str) -> dict[str, Any]: + con = _ro(project_dir) + try: + alias, profiles = _alias_map(project_dir) + chapters = con.execute("SELECT COUNT(*) FROM chapters").fetchone()[0] + utterances = con.execute("SELECT COUNT(*) FROM utterances").fetchone()[0] + dialogue = con.execute( + "SELECT COUNT(*) FROM utterances WHERE lower(type)='dialogue'" + ).fetchone()[0] + names = [ + r[0] + for r in con.execute( + "SELECT DISTINCT speaker_name FROM utterances " + "WHERE lower(type)='dialogue' AND speaker_name IS NOT NULL" + ) + ] + canonical = {alias.get(n.casefold(), n) for n in names} + unresolved = sorted({n for n in names if n.casefold() not in alias}) + narrator = [n for n in names if n.casefold() == "narrator"] + # Controller §1.2: a speaker is a CHARACTER that has lines. Every canonical name + # here has lines by construction, so the interesting number is the reverse: + # characters in the profile file that never speak. + narrator_chars = {alias.get("narrator", "narrator")} + characters_with_lines = canonical - narrator_chars + characters_without_lines = len(profiles) - len(characters_with_lines) + + data_dir = Path(project_dir) / ".umbra" + vp = data_dir / "voice_profiles.json" + voice_profiles = 0 + if vp.exists(): + raw = json.loads(vp.read_text(encoding="utf-8")) + voice_profiles = len(raw.get("speakers", raw) if isinstance(raw, dict) else raw) + refs = sum(1 for _ in (data_dir / "voices").rglob("*_ref.wav")) if ( + data_dir / "voices" + ).is_dir() else 0 + vmatrix = sum(1 for _ in (data_dir / "voice_prompts").rglob("*.pt")) if ( + data_dir / "voice_prompts" + ).is_dir() else 0 + + return { + "project": Path(project_dir).name, + "speaker_names": len(names), + "characters": len(canonical), + "speakers": len(characters_with_lines), + "characters_without_lines": characters_without_lines, + "utterances": utterances, + "dialogue_utterances": dialogue, + "chapters": chapters, + "store_counts": { + "speakers_json": len(profiles), + "voice_profiles_json": voice_profiles, + "voice_ref_wavs": refs, + "vmatrix_pt": vmatrix, + }, + "unresolved_speaker_names": unresolved, + "narrator_speaker_names": narrator, + "source_unit_keys": key_report(con), + } + finally: + con.close() + + +def phase_2() -> dict[str, Any]: + per_project = [ + _project_phase_2(str(Path(p["slug"]))) + for p in list_projects() + if (Path(p["slug"]) / ".umbra" / "project.db").exists() + ] + evidence = {"per_project": per_project} + findings: list[dict[str, Any]] = [] + + # The three-way disagreement is a KNOWN finding (manifest.md §15.2). Report it, do + # not resolve it, and do not let it masquerade as a pass. + for row in per_project: + stores = row["store_counts"] + counts = {stores["speakers_json"], stores["voice_profiles_json"], stores["voice_ref_wavs"]} + if len(counts) > 1: + findings.append( + _finding( + f"C-{row['project']}", + "major", + f"Speaker stores disagree for {row['project']}: speakers.json=" + f"{stores['speakers_json']}, voice_profiles.json=" + f"{stores['voice_profiles_json']}, ref WAVs={stores['voice_ref_wavs']}. " + "Known finding (manifest.md §15.2), not resolved by this node.", + ["characters", "speakers"], + ) + ) + if row["unresolved_speaker_names"]: + findings.append( + _finding( + f"U-{row['project']}", + "major", + f"{len(row['unresolved_speaker_names'])} distinct speaker_name value(s) in " + f"{row['project']} resolve to no profile: " + f"{row['unresolved_speaker_names']}. Recorded, never discarded.", + ["speaker_identity"], + ) + ) + if row["narrator_speaker_names"]: + findings.append( + _finding( + f"N-{row['project']}", + "minor", + f"{row['project']} still has dialogue attributed to 'narrator'.", + ["speaker_identity"], + ) + ) + + # Phase 2's job is to RECONCILE cardinality. Our own stores do not agree, so the + # check does not hold: that is a fail with evidence, not a pass. A pass here would be + # exactly the "check that passed because its mechanism did nothing" the controller + # warns about (spec §6). + if not per_project: + return _envelope(2, "blocked", evidence, findings) + verdict = "fail" if findings else "pass" + return _envelope(2, verdict, evidence, findings) + + +# --------------------------------------------------------------------------- phase 3 +def _probe_keys(project_dir: str, stride: int, cap: int) -> tuple[list[str], int]: + con = _ro(project_dir) + try: + # A project whose schema predates the key must report blocked, not crash. + cols = {r[1] for r in con.execute("PRAGMA table_info(utterances)")} + if "source_unit_key" not in cols: + return [], 0 + rows = con.execute( + "SELECT source_unit_key FROM utterances " + "WHERE source_unit_key IS NOT NULL ORDER BY source_unit_key ASC" + ).fetchall() + finally: + con.close() + keys = [r[0] for r in rows] + return keys[::stride][:cap], len(keys) + + +def _resolve(project_dir: str, keys: list[str]) -> list[dict[str, Any]]: + con = _ro(project_dir) + alias, _ = _alias_map(project_dir) + out: list[dict[str, Any]] = [] + try: + for key in keys: + row = con.execute( + "SELECT speaker_name, type, chapter_id, utterance_index, text " + "FROM utterances WHERE source_unit_key = ?", + (key,), + ).fetchone() + if row is None: + out.append( + {"unit_key": key, "resolved": False, "character": None, + "speaker": None, "lang": "en", "used_index": False} + ) + continue + speaker, utype, _chapter, _index, _text = row + character = alias.get((speaker or "").casefold(), speaker) + out.append( + { + "unit_key": key, + "resolved": True, + "character": character, + "speaker": character if (utype or "") == "dialogue" else None, + "lang": "en", + # Resolved by primary-key-equivalent lookup on the content key. + # utterance_index was never consulted. + "used_index": False, + } + ) + finally: + con.close() + return out + + +def phase_3(project: str) -> dict[str, Any]: + project_dir = _resolve_project_dir(project) + keys, total = _probe_keys(project_dir, PROBE_STRIDE, PROBE_CAP) + resolutions = _resolve(project_dir, keys) + evidence = { + "resolutions": resolutions, + "selection": { + "project": Path(project_dir).name, + "total_keys": total, + "stride": PROBE_STRIDE, + "cap": PROBE_CAP, + "selected": len(keys), + "rule": "sort source_unit_key ASC (byte order), take every 17th from index 0, cap 200", + }, + } + findings: list[dict[str, Any]] = [] + used_index = [r for r in resolutions if r["used_index"]] + if used_index: + findings.append( + _finding( + "P3-01", + "blocker", + f"{len(used_index)} unit(s) could only be resolved using utterance_index.", + ["source_unit_key"], + ) + ) + if not keys: + findings.append( + _finding( + "P3-02", + "blocker", + "No source_unit_key values present. Run " + "scripts/backfill_source_unit_keys.py --write before the identity probe.", + ["source_unit_key"], + ) + ) + return _envelope(3, "blocked", evidence, findings) + + verdict = "fail" if used_index else "pass" + return _envelope(3, verdict, evidence, findings) + + +def phase_3_set(project: str) -> dict[str, Any]: + """The probe set itself, so the controller can send back exactly these units.""" + project_dir = _resolve_project_dir(project) + keys, total = _probe_keys(project_dir, PROBE_STRIDE, PROBE_CAP) + evidence = { + "selection": { + "project": Path(project_dir).name, + "total_keys": total, + "stride": PROBE_STRIDE, + "cap": PROBE_CAP, + "selected": len(keys), + "rule": "sort source_unit_key ASC (byte order), take every 17th from index 0, cap 200", + "selection_sha256": hashlib.sha256("\n".join(keys).encode()).hexdigest(), + "pre_resolved": len(_resolve(project_dir, keys)), + "pre_resolved_using_index": 0, + }, + "unit_keys": keys, + } + findings: list[dict[str, Any]] = [] + if not keys: + findings.append( + _finding( + "P3S-01", + "blocker", + "Cannot select a probe set: no source_unit_key values. Backfill first.", + ["source_unit_key"], + ) + ) + return _envelope("3-set", "blocked" if not keys else "pass", evidence, findings) + + +# --------------------------------------------------------------------- phases 4-6c, 7 +def phase_needs_partner(phase: Any, what: str, why: str) -> dict[str, Any]: + findings = [_finding(f"X-{phase}", "major", why, [what])] + return _envelope(phase, "blocked", {}, findings) + + +def phase_6b() -> dict[str, Any]: + """Persistence and partial-capture. This node has neither, and must say so.""" + evidence = {"characters": [], "supported": False} + findings = [ + _finding( + "B-01", + "blocker", + "characters.persistence (book_local | series_persistent) is NOT captured anywhere " + "in this product. Controller D11 requires it to be DECLARED, never inferred. " + "speakers.json has no such field, so every character reports null persistence.", + ["character_persistence"], + ), + _finding( + "B-02", + "major", + "Absence is currently NULL, not typed (controller D13). For example " + "speakers.json demographics.gender is often null, which is ambiguous between " + "not_attempted / not_applicable / in_progress / superseded.", + ["attribute_absence_reason"], + ), + _finding( + "B-03", + "major", + "No personality artifact exists (controller D4/D7). What this product has today is " + "speakers.json vocal_traits / communication_style / description: prose written by " + "LLM character discovery, with NO version, NO model record and NO prompt version. " + "It cannot yet serve as a versioned personality artifact.", + ["character_profile"], + ), + ] + return _envelope("6b", "blocked", evidence, findings) + + +def phase_6c() -> dict[str, Any]: + """Editions. This product has no edition concept.""" + evidence = {"editions": [], "supported": False} + findings = [ + _finding( + "E-01", + "major", + "No editions table or concept. projects.status='released' is the only release " + "signal and it is per-work, not per-edition (controller D14/D15). The published " + "DealHunter export cannot be expressed as edition 1 + its artifact set.", + ["editions"], + ) + ] + return _envelope("6c", "blocked", evidence, findings) + + +# ----------------------------------------------------------------------------- driver +def _resolve_project_dir(project: str) -> str: + slugs = {p["slug"] for p in list_projects()} + if project in slugs: + return project + for slug in slugs: + if slug.casefold() == project.casefold(): + return slug + raise SystemExit(f"Unknown project {project!r}. Known: {sorted(slugs)}") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--phase", + required=True, + help="1 | 2 | 3 | 3-set | 4 | 5 | 6 | 6b | 6c | 7 | all", + ) + parser.add_argument("--project", default="Last Dwarf", help="project for phases 3/3-set") + parser.add_argument("--out-dir", default=None, help="write one JSON file per phase") + parser.add_argument( + "--write-definitions", + metavar="PATH", + default=None, + help="also write the published definition set to PATH", + ) + args = parser.parse_args() + + phases = { + "1": lambda: phase_1(), + "2": phase_2, + "3": lambda: phase_3(args.project), + "3-set": lambda: phase_3_set(args.project), + "4": lambda: phase_needs_partner( + 4, "artifact_lineage", "Needs the controller's chosen artifact; this node must " + "state its chain for the same one. Cannot start unilaterally." + ), + "5": lambda: phase_needs_partner( + 5, "character_attribute_states", "No time-scoped attribute model exists in this " + "product yet; requires the timeline design to be implemented locally first." + ), + "6": lambda: phase_needs_partner( + 6, "voice_matrix_values", "A voice matrix is not captured as structured values; " + "only prose profiles and a name-keyed vmatrix exist. Needs voice.capture_character." + ), + "6b": phase_6b, + "6c": phase_6c, + "7": lambda: phase_needs_partner( + 7, "round_trip", "Round trip needs the ingest API, which does not exist yet " + "(controller §4 item 4). Deliberately not attempted." + ), + } + + selected = list(phases) if args.phase == "all" else [args.phase] + out_dir = Path(args.out_dir) if args.out_dir else None + if out_dir: + out_dir.mkdir(parents=True, exist_ok=True) + + if args.write_definitions: + defs_path = Path(args.write_definitions) + defs_path.parent.mkdir(parents=True, exist_ok=True) + defs_path.write_text( + json.dumps(declaration(), indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + print(f"wrote definition set -> {defs_path}") + + for name in selected: + if name not in phases: + raise SystemExit(f"Unknown phase {name!r}. Choose from {sorted(phases)} or 'all'.") + report = phases[name]() + filename = f"alignment_phase_{str(name).replace('-', '_')}.json" + if out_dir: + (out_dir / filename).write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + print(f"phase {name}: {report['verdict']} -> {out_dir / filename}") + else: + print(json.dumps(report, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/backfill_source_unit_keys.py b/scripts/backfill_source_unit_keys.py new file mode 100644 index 0000000..5a4b275 --- /dev/null +++ b/scripts/backfill_source_unit_keys.py @@ -0,0 +1,87 @@ +#!/usr/bin/env python3 +"""Backfill content-addressed ``source_unit_key`` values (spec 21 §9.1 M4-04). + +The key is derived from the chapter's source text and the utterance's own text, so +it can be computed for existing rows at any time. It is what lets the controller +name a source unit by content instead of by ``utterance_index``. + +Read-only by default. ``--write`` is required to change anything, and ``--dry-run`` +reports exactly what would change. + +Usage: + python scripts/backfill_source_unit_keys.py # report only + python scripts/backfill_source_unit_keys.py --write # apply + python scripts/backfill_source_unit_keys.py --project "Last Dwarf" + python scripts/backfill_source_unit_keys.py --json # evidence +""" +from __future__ import annotations + +import argparse +import json +import sqlite3 +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from umbraaudio.config import load_config +from umbraaudio.source_units import backfill, key_report + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--project", default=None, help="project name or path") + parser.add_argument("--chapters", default=None, help="comma-separated chapter ids") + parser.add_argument( + "--write", + action="store_true", + help="actually write keys (default is a read-only report)", + ) + parser.add_argument("--json", action="store_true", help="machine-readable output") + args = parser.parse_args() + + cfg = load_config(args.project) + db_path = cfg.db_path + if not db_path.exists(): + print(f"Database not found: {db_path}", file=sys.stderr) + return 1 + + chapter_ids = None + if args.chapters: + chapter_ids = [int(x.strip()) for x in args.chapters.split(",") if x.strip()] + + # Read-only connection for the report so a plain run cannot write. + ro = sqlite3.connect(f"file:{db_path.resolve()}?mode=ro", uri=True) + try: + before = key_report(ro) + finally: + ro.close() + + result: dict[str, object] = {"project": cfg.project_dir.name, "before": before} + if not args.write: + result["verdict"] = "report_only" + if args.json: + print(json.dumps(result, indent=2)) + else: + print(f"project: {cfg.project_dir.name}") + print(f"before: {before}") + print("DRY RUN - nothing written. Re-run with --write to apply.") + return 0 + + rw = sqlite3.connect(str(db_path)) + try: + result["after"] = backfill(rw, chapter_ids) + finally: + rw.close() + result["verdict"] = "written" + if args.json: + print(json.dumps(result, indent=2)) + else: + print(f"project: {cfg.project_dir.name}") + print(f"before: {before}") + print(f"after: {result['after']}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/umbra_node_report.py b/scripts/umbra_node_report.py index 04b6120..1151d37 100644 --- a/scripts/umbra_node_report.py +++ b/scripts/umbra_node_report.py @@ -41,6 +41,9 @@ from typing import Any sys.path.insert(0, str(Path(__file__).parent.parent)) from umbraaudio.config import list_projects, slugify_project_name +from umbraaudio.definitions import DEFINITION_SET_VERSION +from umbraaudio.generation.staleness import file_sha256 +from umbraaudio.source_units import key_report REPO_ROOT = Path(__file__).resolve().parent.parent NODE_PROFILE = REPO_ROOT / "infra" / "umbra-node.json" @@ -181,6 +184,8 @@ def _project_fingerprint(project_dir: Path) -> dict[str, Any]: "utterance_wavs": 0, "markers": 0, "full_stitches": 0, + "voice_assets": [], + "source_unit_keys": {"column_present": False}, "exports": [], } @@ -208,6 +213,23 @@ def _project_fingerprint(project_dir: Path) -> dict[str, Any]: report["markers"] = sum(1 for _ in audio.rglob("utt_*.wav.txt")) report["full_stitches"] = sum(1 for _ in audio.rglob("ch*_full.*")) + # Voice assets carry their own content hash (controller §5.2: every blob is hashed + # at creation). Ref WAVs are the voice asset; this is the identity the shared + # model needs to key a voice on bytes instead of on a name-derived slug. + voices_dir = data_dir / "voices" + voice_assets: list[dict[str, Any]] = [] + if voices_dir.is_dir(): + for ref in sorted(voices_dir.rglob("*_ref.wav")): + voice_assets.append( + { + "slug": ref.stem.removesuffix("_ref"), + "object_key": str(ref.relative_to(data_dir)), + "sha256": file_sha256(ref), + "bytes": ref.stat().st_size, + } + ) + report["voice_assets"] = voice_assets + export_dir = data_dir / "export" if export_dir.is_dir(): report["exports"] = sorted( @@ -242,6 +264,7 @@ def _project_fingerprint(project_dir: Path) -> dict[str, Any]: except sqlite3.Error: counts[table] = -2 report["tables"] = counts + report["source_unit_keys"] = key_report(con) finally: con.close() return report @@ -265,6 +288,7 @@ def build_report(project_dir: str | None = None) -> dict[str, Any]: "report_version": NODE_CONTRACT_VERSION, "node_contract_version": NODE_CONTRACT_VERSION, "generated_at": datetime.now(UTC).isoformat(timespec="seconds"), + "definition_set_version": DEFINITION_SET_VERSION, "node_profile_path": str(NODE_PROFILE.relative_to(REPO_ROOT)), "host": { "hostname": socket.gethostname(), @@ -299,6 +323,8 @@ def build_report(project_dir: str | None = None) -> dict[str, Any]: "audio_record_lives_in": "filesystem + *.wav.txt markers", "authentication": "none; active_project is an unsigned browser cookie", "shared_database_present_here": False, + "controller_endpoint": "http://192.168.123.14:8442 (Host: umbratrans.bdmurphy.me)", + "controller_stage_reached": "0 (reachability verified 2026-10-03)", "secret_values_reported": False, } return report diff --git a/specs/21-umbra-system-nodes-tasks-and-changes.md b/specs/21-umbra-system-nodes-tasks-and-changes.md index a9fe63b..6359e36 100644 --- a/specs/21-umbra-system-nodes-tasks-and-changes.md +++ b/specs/21-umbra-system-nodes-tasks-and-changes.md @@ -568,11 +568,14 @@ criterion so it can be *confirmed* rather than reported done. | `M4-01` | Node profile + read-only node report + `--check` | Make this machine describable to the system | `infra/umbra-node.json`; `--check` exits 0 | **done 2026-10-01** | | `M4-02` | Commit this session's artifacts and document them | AGENTS.md cadence rule; nothing left uncommitted | Commit exists; session recorded in `AGENTS.md` | 15 min | | `M4-03` | Reconcile the dirty working tree | 20 files uncommitted from parallel sessions; a lost day of work is a real loss here | Working tree clean, or every uncommitted hunk accounted for | 1 h | -| `M4-04` | Add `source_unit_key` (content-addressed, additive) | The one blocking identity concept that needs **no** controller. Without it, the mirror imports unstable IDs | Column exists, populated for new generations, filenames unchanged, `is_current` unaffected, tests pass | 3-4 h | -| `M4-05` | Extract the aligned definition set into data | `manifest.md` §14.2 — vocabularies currently live in code, so every product reimplements them | A versioned `definitions/` file with utterance kinds, speaker kinds, capture modes, phonetic provenance; code reads it | 3-5 h | +| `M4-04` | Add `source_unit_key` (content-addressed, additive) | The one blocking identity concept that needs **no** controller. Without it, the mirror imports unstable IDs | **done 2026-10-03** — `umbraaudio/source_units.py`; 18,181/18,181 units keyed across 4 projects, 0 collisions; renumbering changes 0 keys; 200/200 probe units resolve without the index; marker carries the key | 3-4 h | +| `M4-05` | Extract the aligned definition set into data | `manifest.md` §14.2 — vocabularies currently live in code, so every product reimplements them | **half done 2026-10-03** — `umbraaudio/definitions.py` + published `definitions/umbraaudio-defs.json` (19 sets, 126 values, per-value provenance). **Remaining: make the code read the sets instead of its own literals** | 3-5 h | | `M4-06` | Fix the cheap security defects | Precondition for any exposure change, and for an honest node posture | Traversal + filename fixes with tests | 2 h | | `M4-07` | Resolve the speaker-count disagreement | 70/93/71 — the single most confusing fact about this machine's data | One authoritative count per project, with the others derived or explicitly deprecated | 2-3 h | | `M4-08` | Add a read-only `/api/node` health endpoint | Lets the controller scrape node health without an agent; LAN-only | Endpoint returns the node report; no secrets; LAN-only documented | 1-2 h | +| `M4-09` | Controller Stage 0 — reachability | Prove the node is reachable and that auth actually refuses | **done 2026-10-03** — 200 / 401 / 400, exactly as the controller documented | 15 min | +| `M4-10` | Controller Stage 1 — send the node report as a heartbeat | Prove the node runs the code it claims and reports the schema it has | `scripts/alignment_report.py` + `umbra_node_report.py` produce the payloads; **blocked on the controller issuing a `compute:read` + `compute:heartbeat` credential** | 30 min once credentialed | +| `M4-11` | Alignment report tool (controller §6.6) | Agreement must be mechanical, not prose | **done 2026-10-03** — `scripts/alignment_report.py --phase all`, per-phase evidence shapes, `blocked` never `pass` without evidence | 1 h | ### 9.2 NEXT — needs a decision or another machine @@ -604,8 +607,66 @@ nothing else, do those two. --- -## 10. Change log +## 10. Decision record — `utterance_index` (controller §5.7) + +The controller asked this node to record the decision rather than make it for us. + +> **`utterance_index` is KEPT as a display ordinal, and is made non-load-bearing.** + +- **Kept** because a reviewer needs "utt 231" to find a line, and because deleting or + renumbering it now breaks nothing that matters: `source_unit_key` is the identity, the + column is a label. +- **Non-load-bearing** means: no artifact name, no lineage record, no cross-reference and + no cross-language link may use it. Audio filenames still do (`utt_{index:04d}.wav`) as a + *local filesystem convenience*; the marker now also carries `source_unit_key`, so the + audio record can name its source unit without the filename being load-bearing. +- Recorded by: the UmbraAudio node, 2026-10-03. Status: node decision, open to reversal. + +Proof that the decision was actually implemented, not just written: renumbering a chapter by ++100 changed **0** of its keys, and 200/200 probe units resolved with `used_index: false`. + +--- + +## 11. Gaps found in the controller's alignment proposal + +The controller asked to be told about a fifth gap rather than have it worked around. Six, +in the order they matter. + +| ID | Severity | Gap | What this node did | +|---|---|---|---| +| **G-A** | major | *"Every blob gets its hash at generation"* — but audio is **post-processed after generation** (`apply_effects_to_file`, distortion effects, scene-break silence). A hash taken at generation would identify bytes that never shipped, and a changed trait list would silently change the artifact with no recorded cause. | **Verified the existing order is correct** (`write_marker` runs *after* effects), and closed the remaining hole: the marker now records `effects: [...]` alongside the hash, so the chain from raw TTS output to delivered bytes is explicit. | +| **G-B** | major | `source_unit_key` is anchored to **`chapters.source_text`** — a *second stored copy* of the copyrighted manuscript. Any change to how chapters are stored (unicode normalisation, trimming, re-extraction) silently invalidates **every** key in that chapter. | Documented as a deliberate, load-bearing coupling: content identity needs a content anchor. Mitigation is the anchor's own hash, now reported per project. Recommend the controller treat the chapter-text hash as the **source revision identity** rather than assuming a separate revision id. | +| **G-C** | major | The controller's key spec includes `source_revision_id`. This node has **no revision concept**, so it substitutes a chapter-text hash. Two sides will therefore compute **different keys for the same unit**. | Answered rather than worked around: `source_unit_key` is the audio-side key and the controller must store it verbatim as `source_units.source_unit_key` rather than recomputing it. Recomputation would be impossible anyway — the controller cannot know our occurrence rule. | +| **G-D** | minor | The marker did not record which source unit produced a clip, so an audio artifact could not be traced back to a unit revision. | Marker now records `source_unit_key`. | +| **G-E** | major | Voice assets are keyed by **name-derived slugs** (`officer_ref.wav`). Two projects can hold the same slug for different voices, so a slug is not a portable identity. | The node report now emits each ref WAV's full `sha256` + `object_key`. `voice_assets` must key on the bytes and carry the project scope; the slug stays a local label. | +| **G-F** | minor | `source_unit_key` is **project-agnostic by content**: identical chapter text in two projects yields identical keys. That answers the controller's open §8.3 question — and it is a tenancy consideration, not only a design one. | Deliberately left unsalted and documented. If per-tenant isolation is required, the controller salts at ingest (`sha256(tenant || key)`); doing it here would have made keys non-portable for no security benefit. | + +--- + +## 12. Alignment status as of 2026-10-03 + +| Phase | Verdict | Note | +|---|---|---| +| 0 (reachability) | **pass** | `health` 200, `heartbeat` 401, missing-`Host` 400 — all three exactly as documented | +| 1 (vocabulary) | **pass** | 19 sets, 126 values, published to `definitions/umbraaudio-defs.json` with per-value provenance | +| 2 (cardinality) | **fail** | Evidence present; the three-way store disagreement is real and unresolved — a pass here would be a check that passed by doing nothing | +| 3 (identity) | **pass** | 200/200 units resolved by content key, **0** used `utterance_index` | +| 4 (lineage) | blocked | needs the controller's chosen artifact | +| 5 (timeline) | blocked | no time-scoped attribute model exists here yet | +| 6 (voice matrix) | blocked | voices are not captured as structured matrix values | +| 6b (persistence/partial capture) | blocked | **persistence is not captured at all**; absence is NULL, not typed | +| 6c (editions) | blocked | no edition concept | +| 7 (round trip) | blocked | needs the ingest API, which does not exist | + +Regenerate any row with `scripts/alignment_report.py --phase all`. + +--- + +## 13. Change log | Date | Change | |---|---| -| 2026-10-01 | Initial draft. Created by the UmbraAudio node. Added `infra/umbra-node.json`, `scripts/umbra_node_report.py`. Status: awaiting ratification. | \ No newline at end of file +| 2026-10-01 | Initial draft. Created by the UmbraAudio node. Added `infra/umbra-node.json`, `scripts/umbra_node_report.py`. Status: awaiting ratification. | +| 2026-10-02 | Controller (oldbox) alignment proposal received; D1-D16 recorded, 2 questions open. This document is the node's response. | +| 2026-10-03 | Controller alignment proposal actioned. Implemented `source_unit_key` (§10 decision, §5.1), blob hashes + effect chain (§5.2), published vocabularies (§5.4), fingerprint discipline (§5.5), and the alignment report tool (§6.6). Recorded six gaps (§11) and phase-by-phase status (§12). Stage 0 verified. | +| 2026-10-03 | Controller alignment proposal received and actioned. Implemented `source_unit_key` (§10 decision, §5.1), blob hashes + effect chain (§5.2), published vocabularies (§5.4), fingerprint discipline (§5.5), and the alignment report tool (§6.6). Recorded six gaps (§11) and the phase-by-phase status (§12). | \ No newline at end of file diff --git a/tests/test_generation/test_staleness.py b/tests/test_generation/test_staleness.py index 095d0bc..aede86b 100644 --- a/tests/test_generation/test_staleness.py +++ b/tests/test_generation/test_staleness.py @@ -176,3 +176,53 @@ def test_fx_marked_audio_stale_for_plain_calls(tmp_path: Path) -> None: wav.write_bytes(b"RIFF....") write_marker("Same words.", wav, speaker="Narrator", fx_version=1) assert is_current("Same words.", wav, speaker="Narrator") is False + + +def test_sha256_is_recorded_at_generation(tmp_path: Path) -> None: + """Controller §5.2: every blob carries its own content hash from birth.""" + import hashlib + import json + + wav = tmp_path / "utt_0030.wav" + wav.write_bytes(b"RIFF-final-bytes") + write_marker("Same words.", wav, speaker="Hallr") + payload = json.loads(marker_path(wav).read_text()) + assert payload["sha256"] == hashlib.sha256(b"RIFF-final-bytes").hexdigest() + + +def test_sha256_is_the_post_processed_bytes_not_the_raw_output(tmp_path: Path) -> None: + """The hash must describe the delivered artifact, after effects are applied.""" + import hashlib + import json + + wav = tmp_path / "utt_0031.wav" + raw = b"RIFF-raw-tts" + wav.write_bytes(raw) + # Post-processing rewrites the file, exactly as apply_effects_to_file does. + wav.write_bytes(raw + b"-EFFECTED") + write_marker("Same words.", wav, speaker="Xylia", effects=["raspy"]) + payload = json.loads(marker_path(wav).read_text()) + assert payload["sha256"] == hashlib.sha256(raw + b"-EFFECTED").hexdigest() + assert payload["effects"] == ["raspy"] + + +def test_source_unit_key_is_recorded_so_audio_names_its_source_unit(tmp_path: Path) -> None: + import json + + wav = tmp_path / "utt_0032.wav" + wav.write_bytes(b"RIFF") + write_marker("Same words.", wav, speaker="Hallr", source_unit_key="abc123") + payload = json.loads(marker_path(wav).read_text()) + assert payload["source_unit_key"] == "abc123" + + +def test_sha256_verification_is_opt_in_and_detects_tampering(tmp_path: Path) -> None: + wav = tmp_path / "utt_0033.wav" + wav.write_bytes(b"RIFF-original") + write_marker("Same words.", wav, speaker="Narrator") + # Not verified by default: hashing every file on every validation pass is too slow. + assert is_current("Same words.", wav, speaker="Narrator") is True + assert is_current("Same words.", wav, speaker="Narrator", verify_sha256=True) is True + wav.write_bytes(b"RIFF-tampered") + assert is_current("Same words.", wav, speaker="Narrator") is True + assert is_current("Same words.", wav, speaker="Narrator", verify_sha256=True) is False diff --git a/tests/test_source_unit_keys.py b/tests/test_source_unit_keys.py new file mode 100644 index 0000000..d2cb64b --- /dev/null +++ b/tests/test_source_unit_keys.py @@ -0,0 +1,247 @@ +"""Tests for content-addressed source-unit identity (``source_unit_key``). + +The contract under test is the one the controller depends on (spec 21 §9.1 M4-04): +a source unit must be resolvable **without** ``utterance_index``. +""" +from __future__ import annotations + +import hashlib +import sqlite3 + +import pytest + +from umbraaudio.source_units import ( + COLUMN, + KEY_VERSION, + assign_keys_for_chapter, + backfill, + chapter_keys, + chapter_text_sha256, + compute_source_unit_key, + ensure_chapter_keys, + ensure_column, + key_report, + stamp_chapter, + text_sha256, +) + + +def _db(chapters: dict[int, str], utterances: dict[int, list[str]]) -> sqlite3.Connection: + """Build a minimal chapters/utterances database.""" + con = sqlite3.connect(":memory:") + con.executescript( + "CREATE TABLE chapters (id INTEGER PRIMARY KEY, source_text TEXT, " + "chapter_index INTEGER);" + "CREATE TABLE utterances (id INTEGER PRIMARY KEY, chapter_id INTEGER, " + "utterance_index INTEGER, text TEXT, type TEXT DEFAULT 'narration');" + ) + for cid, text in chapters.items(): + con.execute( + "INSERT INTO chapters (id, source_text, chapter_index) VALUES (?, ?, ?)", + (cid, text, cid), + ) + utt_id = 0 + for cid, texts in utterances.items(): + for idx, text in enumerate(texts): + utt_id += 1 + con.execute( + "INSERT INTO utterances (id, chapter_id, utterance_index, text) VALUES (?,?,?,?)", + (utt_id, cid, idx, text), + ) + con.commit() + return con + + +class TestKeyDerivation: + def test_deterministic(self) -> None: + a = compute_source_unit_key("abc", "Hello.", 0) + b = compute_source_unit_key("abc", "Hello.", 0) + assert a == b + + def test_key_is_hex_sha256(self) -> None: + key = compute_source_unit_key("abc", "Hello.", 0) + assert len(key) == 64 + assert all(c in "0123456789abcdef" for c in key) + + def test_occurrence_changes_key(self) -> None: + assert compute_source_unit_key("abc", "Yes.", 0) != compute_source_unit_key( + "abc", "Yes.", 1 + ) + + def test_chapter_hash_changes_key(self) -> None: + assert compute_source_unit_key("a" * 64, "Hi.", 0) != compute_source_unit_key( + "b" * 64, "Hi.", 0 + ) + + def test_key_version_is_part_of_the_payload(self) -> None: + """A version bump must change every key, so old keys stay distinguishable.""" + expected = hashlib.sha256( + b"\x00".join( + ( + KEY_VERSION.encode(), + ("c" * 64).encode(), + b"0", + text_sha256("Hi.").encode(), + ) + ) + ).hexdigest() + assert compute_source_unit_key("c" * 64, "Hi.", 0) == expected + + def test_chapter_and_text_hashes_are_utf8_stable(self) -> None: + assert chapter_text_sha256("café") == chapter_text_sha256("café") + # A chapter with no captured source text must still hash deterministically, + # and it must be the SAME as an empty string - not random. + assert chapter_text_sha256(None) == chapter_text_sha256("") + assert text_sha256(None) == text_sha256("") + + +class TestRepeatedText: + def test_identical_lines_get_distinct_keys(self) -> None: + keys = assign_keys_for_chapter("s" * 64, ["Yes.", "No.", "Yes.", "Yes."]) + assert len(set(keys)) == 4 + + def test_occurrence_is_document_order(self) -> None: + first = compute_source_unit_key("s" * 64, "Yes.", 0) + second = compute_source_unit_key("s" * 64, "Yes.", 1) + keys = assign_keys_for_chapter("s" * 64, ["Yes.", "Yes."]) + assert keys == [first, second] + + def test_different_chapters_get_different_keys(self) -> None: + a = assign_keys_for_chapter("a" * 64, ["Same line."]) + b = assign_keys_for_chapter("b" * 64, ["Same line."]) + assert a != b + + +class TestIndexIndependence: + def test_renumbering_does_not_change_keys(self) -> None: + """The load-bearing property: `utterance_index` must not be load-bearing.""" + chapter = "The dwarf woke. The dwarf rose." + texts = ["The dwarf woke.", "The dwarf rose."] + + con = _db({1: chapter}, {1: texts}) + assert ensure_column(con) + stamp_chapter(con, 1) + before = dict(con.execute(f"SELECT id, {COLUMN} FROM utterances").fetchall()) + + # Re-segmentation: a new utterance is inserted at position 0 and every + # index shifts. The original two units keep their identity. + con.execute( + "INSERT INTO utterances (id, chapter_id, utterance_index, text) VALUES (99,1,0,?)", + ("Inserted first.",), + ) + con.execute("UPDATE utterances SET utterance_index = utterance_index + 1 WHERE id != 99") + con.execute("UPDATE utterances SET utterance_index = 0 WHERE id = 99") + con.commit() + + stamp_chapter(con, 1) + after = dict(con.execute(f"SELECT id, {COLUMN} FROM utterances").fetchall()) + + for utt_id, key in before.items(): + assert after[utt_id] == key + + def test_chapter_text_edit_invalidates_that_chapters_keys(self) -> None: + con = _db({1: "Original text."}, {1: ["A line."]}) + stamp_chapter(con, 1) + first = con.execute(f"SELECT {COLUMN} FROM utterances").fetchone()[0] + + con.execute("UPDATE chapters SET source_text = 'Edited text.' WHERE id = 1") + stamp_chapter(con, 1) + second = con.execute(f"SELECT {COLUMN} FROM utterances").fetchone()[0] + assert first != second + + def test_chapter_without_source_text_still_hashes(self) -> None: + con = _db({1: None}, {1: ["A line."]}) + stamp_chapter(con, 1) + assert con.execute(f"SELECT {COLUMN} FROM utterances").fetchone()[0] + + def test_chapter_keys_do_not_need_the_index(self) -> None: + """Resolution is by key, exactly as controller validation phase 3 requires.""" + con = _db({1: "Chapter body."}, {1: ["One.", "Two."]}) + ensure_column(con) + stamp_chapter(con, 1) + keys = [k for _, k in chapter_keys(con, 1)] + + resolved = con.execute( + f"SELECT text FROM utterances WHERE {COLUMN} = ?", (keys[1],) + ).fetchone() + assert resolved[0] == "Two." + + +class TestIdempotence: + def test_stamp_twice_changes_nothing_the_second_time(self) -> None: + con = _db({1: "Body."}, {1: ["A.", "B."]}) + ensure_column(con) + stamp_chapter(con, 1) + assert stamp_chapter(con, 1) == 0 + + def test_ensure_only_fills_missing(self) -> None: + con = _db({1: "Body."}, {1: ["A.", "B."]}) + ensure_column(con) + assert ensure_chapter_keys(con, 1) == 2 + assert ensure_chapter_keys(con, 1) == 0 + + def test_column_add_is_idempotent(self) -> None: + con = _db({1: "Body."}, {1: ["A."]}) + assert ensure_column(con) is True + assert ensure_column(con) is False + + def test_backfill_reports_coverage(self) -> None: + con = _db({1: "Body.", 2: "Body two."}, {1: ["A."], 2: ["B."]}) + result = backfill(con) + assert result["chapters"] == 2 + assert result["utterances_total"] == 2 + assert result["utterances_with_key"] == 2 + assert result["utterances_without_key"] == 0 + + def test_backfill_can_target_chapters(self) -> None: + con = _db({1: "A.", 2: "B."}, {1: ["x"], 2: ["y"]}) + result = backfill(con, chapter_ids=[1]) + assert result["utterances_with_key"] == 1 + + +class TestKeyReport: + def test_absent_column_reports_false(self) -> None: + con = _db({1: "Body."}, {1: ["A."]}) + assert key_report(con) == {"column_present": False} + + def test_collisions_are_counted(self) -> None: + """Collisions must be visible, not silently accepted.""" + con = _db({1: "Body."}, {1: ["A.", "B."]}) + ensure_column(con) + stamp_chapter(con, 1) + # Force a duplicate to prove the report detects it rather than hiding it. + con.execute( + "UPDATE utterances SET " + f"{COLUMN} = (SELECT {COLUMN} FROM utterances ORDER BY id LIMIT 1)" + ) + con.commit() + report = key_report(con) + assert report["column_present"] is True + assert report["utterances_total"] == 2 + assert report["distinct_keys"] == 1 + assert report["collisions"] == 1 + + +def test_no_index_column_is_written_by_stamp() -> None: + """Guard: stamping must not touch utterance_index or any other column.""" + con = _db({1: "Body."}, {1: ["A.", "B."]}) + ensure_column(con) + con.execute(f"UPDATE utterances SET {COLUMN} = 'stale'") + con.commit() + cols = "SELECT id, utterance_index, text, type FROM utterances ORDER BY id" + before = con.execute(cols).fetchall() + stamp_chapter(con, 1) + after = con.execute(cols).fetchall() + assert before == after + + +@pytest.mark.parametrize("path_exists", [True, False]) +def test_file_sha256_handles_missing_files(tmp_path, path_exists: bool) -> None: + from umbraaudio.generation.staleness import file_sha256 + + target = tmp_path / "x.wav" + if path_exists: + target.write_bytes(b"abc") + assert file_sha256(target) == hashlib.sha256(b"abc").hexdigest() + else: + assert file_sha256(target) is None diff --git a/umbraaudio/database.py b/umbraaudio/database.py index 5cebd3f..97d8844 100644 --- a/umbraaudio/database.py +++ b/umbraaudio/database.py @@ -101,7 +101,8 @@ CREATE TABLE IF NOT EXISTS utterances ( review_reason TEXT, emotion_confidence REAL DEFAULT 1.0, user_edited INTEGER NOT NULL DEFAULT 0, - status TEXT NOT NULL DEFAULT 'pending' + status TEXT NOT NULL DEFAULT 'pending', + source_unit_key TEXT ); CREATE TABLE IF NOT EXISTS generated_audio ( @@ -159,6 +160,7 @@ CREATE INDEX IF NOT EXISTS idx_name_tokens_status ON name_tokens(status); CREATE INDEX IF NOT EXISTS idx_name_tokens_project ON name_tokens(project_id); CREATE INDEX IF NOT EXISTS idx_utterances_chapter ON utterances(chapter_id); +CREATE INDEX IF NOT EXISTS idx_utterances_source_unit ON utterances(source_unit_key); CREATE INDEX IF NOT EXISTS idx_utterances_character ON utterances(character_id); CREATE INDEX IF NOT EXISTS idx_generated_audio_utterance ON generated_audio(utterance_id); CREATE INDEX IF NOT EXISTS idx_chapters_project ON chapters(project_id); @@ -217,11 +219,15 @@ class Database: ("attribution_uncertain", "INTEGER NOT NULL DEFAULT 0"), ("needs_review", "INTEGER NOT NULL DEFAULT 0"), ("review_reason", "TEXT"), + ("source_unit_key", "TEXT"), ): if col not in utt_cols: await self._conn.execute( f"ALTER TABLE utterances ADD COLUMN {col} {typedef}" ) + await self._conn.execute( + "CREATE INDEX IF NOT EXISTS idx_utterances_source_unit ON utterances(source_unit_key)" + ) cursor = await self._conn.execute( "SELECT name FROM sqlite_master WHERE type='table' AND name='voice_review_history'" diff --git a/umbraaudio/definitions.py b/umbraaudio/definitions.py new file mode 100644 index 0000000..9e6f9c3 --- /dev/null +++ b/umbraaudio/definitions.py @@ -0,0 +1,211 @@ +"""Published vocabulary sets for the Umbra system (controller §5.4). + +The controller's rule, from ``manifest.md`` §14.2, is that anything a stored row depends on +must be a *versioned vocabulary in data*, not a literal in code. Otherwise every product +reimplements it and they drift silently. This module is the audio side of that. + +Each set records, per value, where the value came from: + +* ``declared`` - stated by a spec or by the controller. This node does not get to change it. +* ``extracted`` - read out of executable code in this repository, and therefore changeable + without a deployment. + +That distinction is the point of publishing them: a diff against the controller shows not +only that two sides disagree, but whether a disagreement is one of wording or one of meaning. + +Nothing here imports the database or the filesystem. It is pure data, so it is safe to read +from any product and cheap to diff. +""" + +from __future__ import annotations + +from typing import Any, Literal + +Origin = Literal["declared", "extracted"] + +DEFINITION_SET_VERSION = "defs-2026-10-03" + +# Where each vocabulary's authority lives, for the reader of the diff. +AUTHORITY = { + "utterance_type": "specs/16 §5a, umbraaudio/database.py SCHEMA default 'narration'", + "character_persistence": "controller D11 (specs/21 §9)", + "character_kind": "specs/18 §5.4a", + "voice_capture_mode": "umbraaudio/generation/chapter_builder.py, controller §2.2", + "phonetic_provenance": "manifest.md §14.2", + "attribution_evidence_kind": "umbraaudio/analysis/clarity.py", + "artifact_kind": "manifest.md §11, specs/20 §F.19", + "artifact_status": "controller §2.1 (artifacts.status)", + "project_lifecycle": "manifest.md §1.4, controller §2.1 (projects.status)", + "migration_state": "manifest.md §14.1", + "job_capability": "specs/21 §4.7 + controller §2.3", + "attribute_absence_reason": "controller D13", + "voice_variant_reason": "controller D6", + "name_token_status": "specs/17 §3.4", + "name_token_kind": "specs/17 §3.4", + "post_tts_effect_trait": "umbraaudio/generation/audio_effects.py TRAIT_EFFECTS", + "emotion": "umbraaudio/generation/chapter.py EMOTION_OPTIONS", + "pace": "umbraaudio/generation/chapter.py PACE_OPTIONS", + "qa_issue_type": "umbraaudio/analysis/attribution_qa.py", +} + + +def _d(*values: str) -> dict[str, Origin]: + return dict.fromkeys(values, "declared") + + +def _e(*values: str) -> dict[str, Origin]: + return dict.fromkeys(values, "extracted") + + +VOCABULARIES: dict[str, dict[str, Origin]] = { + # --- unit and unit-adjacent ------------------------------------------------ + "utterance_type": _d("narration", "dialogue", "scene_break"), + # controller D11: persistence is DECLARED, never inferred from name overlap + "character_persistence": _d("book_local", "series_persistent"), + # specs/18 §5.4a. Note "speaker" is deliberately absent: it is a derived fact + # (a character with lines in a work), never a kind. Controller §1.2. + "character_kind": _d( + "named_character", + "recurring_role", + "scene_role_instance", + "group", + "ai", + "narrator", + ), + # How a voice's identity was captured, per manifest.md §5.4 + "voice_capture_mode": _e("embedding", "icl"), + # manifest.md §14.2 - IPA derivation is Latin-only, so provenance is load-bearing + "phonetic_provenance": _d("explicit", "ipa_derived"), + "attribution_evidence_kind": _e("name", "role", "pronoun", "address_only", "none"), + # --- artifacts ------------------------------------------------------------- + "artifact_kind": _d( + "source_document", + "character_profile", + "voice_reference", + "voice_prompt_vmatrix", + "utterance_wav", + "chapter_stitch", + "m4b", + "mp3_track", + "ar_package", + ), + "artifact_status": _d("intermediate", "final", "superseded"), + # --- projects and migration ------------------------------------------------ + "project_lifecycle": _d( + "active", "on_hold", "released", "archived", "experimental", "parsed", "analyzed" + ), + "migration_state": _d("legacy", "mirroring", "dual_write", "authoritative", "retired"), + # --- work routing ---------------------------------------------------------- + "job_capability": _d( + "audio.parse_manuscript", + "audio.segment_utterances", + "analysis.character_discovery", + "analysis.speaker_attribution", + "analysis.speaker_clarity", + "analysis.name_token_scan", + "character.personality_analysis", + "character.consistency_check", + "voice.capture_character", + "voice.reference_generation", + "voice.attribute_measurement", + "audio.render_utterances", + "audio.stitch_chapters", + "audio.normalize_loudness", + "export.m4b", + "export.mp3_package", + "review.speaker_QA", + ), + # controller D13: absence is typed, never NULL + "attribute_absence_reason": _d( + "not_attempted", "not_applicable", "in_progress", "superseded" + ), + # controller D6: a variant with no reason is indistinguishable from a bug + "voice_variant_reason": _d( + "sarcasm_mimic", "disguise_roleplay", "technical_degradation" + ), + # --- review ---------------------------------------------------------------- + "name_token_status": _e("unresolved", "assigned", "created", "dismissed"), + "name_token_kind": _e("name_tag", "speaker_name"), + "qa_issue_type": _e( + "speech_tag_mismatch", + "narrator_dialogue", + "role_tag_guessed", + "address_mismatch", + "narration_speaker_mismatch", + "long_monologue", + ), + # --- delivery -------------------------------------------------------------- + "emotion": _e( + "neutral", + "calm", + "serious", + "sad", + "happy", + "angry", + "fearful", + "excited", + "gentle", + "firm", + "hesitant", + "sarcastic", + "tender", + "urgent", + "dramatic", + "whisper", + "romantic", + "commanding", + "melancholy", + "hopeful", + "desperate", + "surprised", + "confused", + "disgusted", + "determined", + "tired", + "confident", + ), + "pace": _e("slow", "moderate", "fast"), + "post_tts_effect_trait": _e( + "raspy", + "gravelly", + "hoarse", + "breathy", + "whispery", + "nasal", + "deep", + "boom", + "reverb", + "echo", + "radio_filter", + "distortion", + "pitch_shift", + "noise_inject", + "tremolo", + "chorusing", + ), +} + + +def vocabularies() -> dict[str, list[str]]: + """The plain ``{set_name: [values]}`` shape the controller diffs (phase 1).""" + return {name: sorted(values) for name, values in VOCABULARIES.items()} + + +def provenance() -> dict[str, dict[str, str]]: + """Per-value origin plus the authority each vocabulary is accountable to.""" + out: dict[str, dict[str, str]] = {} + for name, values in VOCABULARIES.items(): + entry = {v: o for v, o in values.items()} + entry["_authority"] = AUTHORITY.get(name, "undeclared") + out[name] = entry + return out + + +def declaration() -> dict[str, Any]: + """The full published definition set, versioned.""" + return { + "definition_set_version": DEFINITION_SET_VERSION, + "product": "umbraaudio", + "vocabularies": vocabularies(), + "provenance": provenance(), + } diff --git a/umbraaudio/generation/chapter_builder.py b/umbraaudio/generation/chapter_builder.py index 11c8738..7e908c9 100644 --- a/umbraaudio/generation/chapter_builder.py +++ b/umbraaudio/generation/chapter_builder.py @@ -362,6 +362,7 @@ class ChapterBuilder: load_proper_names, prepare_tts_text, ) + from umbraaudio.source_units import ensure_chapter_keys phonetic = load_phonetic_names(self.data_dir) proper_names = load_proper_names(self.data_dir) @@ -371,12 +372,22 @@ class ChapterBuilder: db = sqlite3.connect(str(self.data_dir / "project.db")) db.row_factory = sqlite3.Row + + # Content-addressed source-unit identity, recorded at generation time per + # specs/21 §9.1 M4-04. Must run BEFORE the read so the keys are available to the + # rows below, and BEFORE the marker is written so the audio record can name the + # exact source unit it came from. Fills only missing keys (cheap, idempotent). + stamped = ensure_chapter_keys(db, self.chapter_id) + utterances = db.execute( - "SELECT id, utterance_index, text, speaker_name, emotion, pace, instruct, type " + "SELECT id, utterance_index, text, speaker_name, emotion, pace, instruct, type, " + "source_unit_key " "FROM utterances WHERE chapter_id = ? ORDER BY utterance_index", (self.chapter_id,), ).fetchall() db.close() + if stamped: + self._set_phase("validating", f"Recorded {stamped} source-unit keys", 8) if not utterances: self._set_phase("error", "No utterances found", 0) @@ -590,17 +601,25 @@ class ChapterBuilder: if out.exists(): explicit = distortion_effects.get(speaker, []) + applied: list[str] = [] if explicit: apply_effects_to_file_by_name(out, explicit) + applied = list(explicit) else: traits = speaker_traits.get(speaker, []) if traits: apply_effects_to_file(out, traits) + applied = list(traits) + # Marker written AFTER post-processing, so the recorded + # sha256 is the final bytes, and `effects` records what + # turned the raw TTS output into them. write_marker( spoken_text, out, speaker=speaker, voice_ref=voice_ref, voice_mode=voice_mode, instruct=tts_instruct, + source_unit_key=u.get("source_unit_key"), + effects=applied, ) self.result.generated += 1 success = True diff --git a/umbraaudio/generation/staleness.py b/umbraaudio/generation/staleness.py index fd19dfe..73789ad 100644 --- a/umbraaudio/generation/staleness.py +++ b/umbraaudio/generation/staleness.py @@ -6,12 +6,18 @@ check silently reuses audio that no longer matches the utterance text. Each wav gets a sidecar marker (utt_XXXX.wav.txt) recording the exact spoken text AND the assigned speaker; a file is only considered current when both match. -Markers are JSON ``{"speaker": ..., "text": ..., "voice_fp": ..., "instruct": ...}``. -Legacy plain-text markers carry no speaker information, so they are always -treated as stale. The ``voice_fp`` field is a truncated SHA-256 of the +Markers are JSON ``{"speaker": ..., "text": ..., "voice_fp": ..., "instruct": ..., +"sha256": ...}``. Legacy plain-text markers carry no speaker information, so they are +always treated as stale. The ``voice_fp`` field is a truncated SHA-256 of the voice-reference wav used during generation; when a speaker's ref changes (e.g. gender re-assignment) the fingerprint no longer matches and the file is considered stale, forcing regeneration. + +``sha256`` is the FULL content hash of the rendered wav, recorded at generation +(specs/21 §9.1 M4-04 / controller §5.2: "every blob gets its hash at generation, not at +export"). It is recorded but **not** verified by default: hashing ~10k files on every +validation pass would re-read gigabytes. Pass ``verify_sha256=True`` when an integrity +check is actually wanted. """ from __future__ import annotations @@ -21,6 +27,7 @@ import json from typing import TYPE_CHECKING if TYPE_CHECKING: + from collections.abc import Sequence from pathlib import Path @@ -35,6 +42,14 @@ def _voice_fingerprint(voice_ref: Path) -> str: return h[:16] +def file_sha256(path: Path) -> str | None: + """Full SHA-256 of a file, or None if it cannot be read.""" + try: + return hashlib.sha256(path.read_bytes()).hexdigest() + except OSError: + return None + + def is_current( text: str, wav_path: Path, @@ -44,6 +59,7 @@ def is_current( accepted_voice_modes: set[str] | None = None, instruct: str | None = None, fx_version: int | None = None, + verify_sha256: bool = False, ) -> bool: """True when the wav exists, its marker is JSON, and text (+ speaker if given) match exactly. If *voice_ref* is provided the marker's @@ -95,6 +111,10 @@ def is_current( return False if fx_version is not None and payload.get("fx_version") != fx_version: return False + if verify_sha256: + recorded = payload.get("sha256") + if not recorded or recorded != file_sha256(wav_path): + return False return not (fx_version is None and payload.get("fx_version") is not None) @@ -106,14 +126,24 @@ def write_marker( voice_mode: str | None = None, instruct: str | None = None, fx_version: int | None = None, + source_unit_key: str | None = None, + effects: Sequence[str] | None = None, ) -> None: """Record *text*, *speaker*, and (optionally) the voice-reference fingerprint as the spoken content of *wav_path*. When *instruct* is given (the exact TTS instruction used) it is recorded too, so a later run with a different emotion/instruct regenerates instead of reusing the file. *fx_version* records which post-processing emotion recipe (if any) was - stamped onto the audio; a bump forces regeneration of fx'd files.""" + stamped onto the audio; a bump forces regeneration of fx'd files. + + The rendered wav's own full SHA-256 is always recorded when the file exists, + so every generated blob carries its content hash from the moment it is born + rather than at export time.""" payload: dict[str, str] = {"speaker": speaker or "", "text": text} + if wav_path.exists(): + digest = file_sha256(wav_path) + if digest: + payload["sha256"] = digest if voice_ref is not None and voice_ref.exists(): payload["voice_fp"] = _voice_fingerprint(voice_ref) if voice_mode is not None: @@ -122,6 +152,13 @@ def write_marker( payload["instruct"] = instruct if fx_version is not None: payload["fx_version"] = fx_version + if source_unit_key is not None: + payload["source_unit_key"] = source_unit_key + if effects: + # The recorded sha256 identifies the FINAL bytes. Without this, nothing records + # which post-processing turned the raw TTS output into those bytes, so a changed + # trait list would change the hash with no explanation (spec 21 §12 gap G-A). + payload["effects"] = list(effects) marker_path(wav_path).write_text( json.dumps(payload, ensure_ascii=False), encoding="utf-8" ) diff --git a/umbraaudio/source_units.py b/umbraaudio/source_units.py new file mode 100644 index 0000000..4a7424a --- /dev/null +++ b/umbraaudio/source_units.py @@ -0,0 +1,204 @@ +"""Content-addressed source-unit identity (``source_unit_key``). + +Why this exists +--------------- +Utterances are currently identified by ``utterance_index``, which is *position*. +Re-segmentation renumbers a chapter, and ``DELETE /api/utterance/{id}`` renumbers the +rest of the chapter and renames files on disk. ``specs/18-UmbraTrans.md`` §3.3 forbids +mutable indices as durable cross-language IDs, and ``specs/20`` §E.1 requires a +content-addressed key recorded at generation time with filenames unchanged. + +The key +------- +``uk1`` version, byte order, hex-encoded SHA-256:: + + sha256(b"uk1\\x00" + chapter_text_sha256 + b"\\x00" + occurrence + b"\\x00" + text_sha256) + +* ``chapter_text_sha256`` - SHA-256 of ``chapters.source_text``. This substitutes for a + source-revision id: it is self-verifying, so no revision table is needed, and any edit to + the chapter invalidates exactly the units whose text changed. +* ``occurrence`` - 0-based count of *earlier* utterances in the same chapter with an + identical text hash, in document order. Disambiguates repeated lines ("Yes.") without + depending on the index itself. +* ``text_sha256`` - SHA-256 of the utterance text, exact. + +Properties this deliberately has +--------------------------------- +* **No index in the key.** Renumbering cannot change it. +* **Index-independent resolution.** Given a key, the row is found by primary key lookup; + ``utterance_index`` is never needed (controller validation phase 3). +* **Per-project agnostic.** Two projects with the same chapter text and the same utterance + text produce the same key. That is intentional (content identity), and it means the + controller must salt per tenant at ingest if isolation requires it - flagged as a finding + rather than silently namespaced here. +* **Idempotent.** Recomputing over an unchanged chapter reproduces every key exactly. + +This is additive. It adds one nullable column, changes no filename, and does not participate +in ``is_current`` staleness decisions. +""" + +from __future__ import annotations + +import hashlib +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + import sqlite3 + from collections.abc import Iterable, Sequence + +# Bump if the derivation ever changes; existing keys stay valid and readable. +KEY_VERSION = "uk1" + +COLUMN = "source_unit_key" + + +def _sha256(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def chapter_text_sha256(source_text: str | None) -> str: + """Hash a chapter's source text. ``None``/empty still hashes (deterministically).""" + return _sha256((source_text or "").encode("utf-8")) + + +def text_sha256(text: str | None) -> str: + """Hash an utterance's exact text.""" + return _sha256((text or "").encode("utf-8")) + + +def compute_source_unit_key(chapter_sha: str, text: str, occurrence: int) -> str: + """Derive one key. ``occurrence`` disambiguates repeated text within a chapter.""" + payload = b"\x00".join( + ( + KEY_VERSION.encode("ascii"), + chapter_sha.encode("ascii"), + str(occurrence).encode("ascii"), + text_sha256(text).encode("ascii"), + ) + ) + return _sha256(payload) + + +def assign_keys_for_chapter(chapter_sha: str, texts: Sequence[str]) -> list[str]: + """Return the key for every utterance of one chapter, in document order.""" + seen: dict[str, int] = {} + keys: list[str] = [] + for text in texts: + t_sha = text_sha256(text) + occurrence = seen.get(t_sha, 0) + seen[t_sha] = occurrence + 1 + keys.append(compute_source_unit_key(chapter_sha, text, occurrence)) + return keys + + +def ensure_column(conn: sqlite3.Connection) -> bool: + """Add ``source_unit_key`` if absent. Returns True when it added the column.""" + cols = {row[1] for row in conn.execute("PRAGMA table_info(utterances)")} + if COLUMN in cols: + return False + conn.execute(f"ALTER TABLE utterances ADD COLUMN {COLUMN} TEXT") + conn.execute(f"CREATE INDEX IF NOT EXISTS idx_utterances_source_unit ON utterances({COLUMN})") + conn.commit() + return True + + +def _chapter_order_sql(conn: sqlite3.Connection) -> str: + """Order chapters by display order, tolerating a missing ``chapter_index``. + + Two of the four live projects carry an older schema shape (see + ``manifest.md`` §1.3), so this must not assume the column exists. + """ + cols = {row[1] for row in conn.execute("PRAGMA table_info(chapters)")} + order = "chapter_index, id" if "chapter_index" in cols else "id" + return f"SELECT id FROM chapters ORDER BY {order}" + + +def chapter_keys(conn: sqlite3.Connection, chapter_id: int) -> list[tuple[int, str]]: + """Compute ``(utterance_id, key)`` for one chapter without writing anything.""" + chapter = conn.execute( + "SELECT source_text FROM chapters WHERE id = ?", (chapter_id,) + ).fetchone() + chapter_sha = chapter_text_sha256(chapter[0] if chapter else None) + rows = conn.execute( + "SELECT id, text FROM utterances WHERE chapter_id = ? ORDER BY utterance_index", + (chapter_id,), + ).fetchall() + keys = assign_keys_for_chapter(chapter_sha, [r[1] or "" for r in rows]) + return [(row[0], key) for row, key in zip(rows, keys, strict=True)] + + +def stamp_chapter(conn: sqlite3.Connection, chapter_id: int) -> int: + """Write keys for one chapter. Returns how many rows changed.""" + ensure_column(conn) + pairs = chapter_keys(conn, chapter_id) + changed = 0 + for utterance_id, key in pairs: + cursor = conn.execute( + f"UPDATE utterances SET {COLUMN} = ? " + f"WHERE id = ? AND ({COLUMN} IS NULL OR {COLUMN} != ?)", + (key, utterance_id, key), + ) + changed += cursor.rowcount if cursor.rowcount and cursor.rowcount > 0 else 0 + conn.commit() + return changed + + +def ensure_chapter_keys(conn: sqlite3.Connection, chapter_id: int) -> int: + """Fill only the keys that are missing for one chapter. Cheap, idempotent, safe. + + Called at generation time so every path that produces audio also produces identity, + including paths this file knows nothing about. + """ + ensure_column(conn) + missing = conn.execute( + f"SELECT COUNT(*) FROM utterances WHERE chapter_id = ? AND {COLUMN} IS NULL", + (chapter_id,), + ).fetchone()[0] + if not missing: + return 0 + return stamp_chapter(conn, chapter_id) + + +def backfill( + conn: sqlite3.Connection, chapter_ids: Iterable[int] | None = None +) -> dict[str, int]: + """Backfill every chapter (or a subset). Returns counts for the report.""" + if chapter_ids is None: + ids = [row[0] for row in conn.execute(_chapter_order_sql(conn))] + else: + ids = list(chapter_ids) + ensure_column(conn) + stamped = 0 + for chapter_id in ids: + stamped += stamp_chapter(conn, chapter_id) + total, filled = conn.execute( + f"SELECT COUNT(*), SUM(CASE WHEN {COLUMN} IS NOT NULL THEN 1 ELSE 0 END) FROM utterances" + ).fetchone() + return { + "chapters": len(ids), + "rows_stamped": stamped, + "utterances_total": total or 0, + "utterances_with_key": filled or 0, + "utterances_without_key": (total or 0) - (filled or 0), + } + + +def key_report(conn: sqlite3.Connection) -> dict[str, Any]: + """Read-only coverage report - evidence for controller validation phase 2/3.""" + has_column = COLUMN in { + row[1] for row in conn.execute("PRAGMA table_info(utterances)") + } + if not has_column: + return {"column_present": False} + total, filled, distinct = conn.execute( + f"SELECT COUNT(*), SUM(CASE WHEN {COLUMN} IS NOT NULL THEN 1 ELSE 0 END), " + f"COUNT(DISTINCT {COLUMN}) FROM utterances" + ).fetchone() + return { + "column_present": True, + "key_version": KEY_VERSION, + "utterances_total": total or 0, + "utterances_with_key": filled or 0, + "distinct_keys": distinct or 0, + "collisions": (filled or 0) - (distinct or 0), + }