umbragit/UmbraAudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Action the controller alignment proposal: source_unit_key + phase reports

Browse Source 
The controller (oldbox) sent a data-model alignment proposal and asked for six
additive changes on this node, the validation protocol, and an honest report.
Six of the seven changes are done; the seventh is declared missing rather than faked.

Stage 0 - reachability verified exactly as documented: health 200, heartbeat 401
(the correct answer - auth is refusing), missing Host 400. Nothing else was sent.

source_unit_key (controller 5.1, the blocking item)
  umbraaudio/source_units.py. sha256("uk1" | chapter_source_text_sha256 |
  occurrence | text_sha256). Nothing positional in the key: "occurrence"
  disambiguates repeated text in document order.
  - 18,181/18,181 units keyed across all four projects, 0 collisions
  - chapter 1 renumbered +100 -> 0 keys changed (the load-bearing proof)
  - backfill 10,015 rows in 0.11s, second run stamps 0 (idempotent)
  - alignment phase 3: 200/200 probe units resolved with used_index: false
  - tolerates the older projects that have no chapters.chapter_index
  Recorded at generation time in chapter_builder, BEFORE the rows are read and
  BEFORE the marker is written, so every path that renders audio also produces
  identity and the clip can name its source unit.

5.7 recorded: utterance_index is KEPT as a display ordinal and made
  non-load-bearing. spec 21 §10.

Blob hashes and effect chain (5.2)
  Markers now carry the full sha256 of the delivered bytes and the effects that
  produced them. Verified the existing order is already correct - write_marker
  runs AFTER apply_effects - and closed the remaining hole: a changed trait list
  changed the hash with no recorded cause. sha256 verification is opt-in, since
  hashing every file on every validation pass would re-read gigabytes.

Vocabularies published (5.4) - umbraaudio/definitions.py, 19 sets, 126 values,
  each value carrying whether it is 'declared' (spec/controller) or 'extracted'
  (read out of code), so a diff shows a difference of meaning, not of wording.

Alignment report tool (6.6) - scripts/alignment_report.py --phase all. Emits the
  controller's envelope with the per-phase evidence shapes, and obeys the two
  rules they set: no evidence means blocked, never pass; a field we cannot produce
  is null PLUS a finding, never omitted. Phase 2 is deliberately 'fail' - the
  three-way store disagreement is real, and a pass there would be a check that
  passed by doing nothing.

Fingerprint discipline (5.5) - the drift detector reported its OWN fingerprint
  change when source_unit_key was added rather than hiding it. Node profile
  updated with the new fingerprints and an honest note that they moved.

Node report extended - per-project source-unit-key coverage, per-voice-asset
  sha256 + object_key, definition_set_version, controller stage reached.

Six gaps found in the proposal (spec 21 §11), including the one they invited plus
five more. The sharpest: "hash at generation" ignores that audio is
post-processed afterwards, and our key is anchored to chapters.source_text - a
second copy of the manuscript - so any change to chapter storage silently
invalidates every key in that chapter.

5.6 deliberately NOT faked: personality has no versioned artifact here, only
  unversioned LLM prose in speakers.json. Reported as phase 6b finding B-03
  instead, because wrapping unversioned inputs in a version field would produce
  artifacts claiming a lineage they do not have.

Answered the controller's two open questions (infra/controller-handoff-2026-10-03.md):
forward-only link for a revealed recurring character, and yes - a new edition may
contradict a frozen state, since this node's correction path IS a regeneration.

Full suite 484 passed / 8 failed - the same 8 pre-existing and unrelated failures.
ruff clean on every file touched. The parallel session's chapter_builder hunk
(helper_url) was filtered out of the commit and left intact in the working tree.
This commit is contained in:
B.D. Murphy
2026-10-02 20:16:06 -07:00
parent 893dc57574
commit 829a03b89f
15 changed files with 2083 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

60
AGENTS.md
View File

@@ -171,6 +171,66 @@ Each pass has quality gates. Quality is the primary concern; generation speed is
## Current Session Summary ## Current Session Summary
### Session: Umbra system node declaration + cross-machine contract (spec 21)
- **Context:** another machine/tab is moving UmbraTrans to the main Umbra system on
`oldbox` (192.168.123.14). Four questions were raised for this machine: (1) align the
data model and copy UmbraAudio data into the system, verify, transition later; (2) every
machine becomes a node that receives *any* task, not just audio; (3) how changes are
managed across products, with the controller owning requirements; (4) does a central
server take over the UI, and how do code updates get deployed/tested.
- **New `specs/21-umbra-system-nodes-tasks-and-changes.md`** — the shared cross-machine
document. Four workstreams (WS1 data alignment, WS2 nodes/tasks, WS3 change management,
WS4 UI topology), each with what already exists, what is missing, and a proposal.
Proposals are **drafts for the controller to ratify**; nothing is decided here.
- **New `infra/umbra-node.json`** — this machine's declared node profile
(`node_id: umbraaudio-m4`, role worker, products `[umbraaudio]`, services, capability
list, resource pools, constraints, project list, lifecycle, schema fingerprints, and an
honest "facts_for_the_controller" block). **No secrets** — ids, names, counts only.
- **New `scripts/umbra_node_report.py`** — read-only node report. Proves what the
controller needs to align: git state, listening ports, `.env` KEY NAMES only (never
values), per-project row counts, and a **schema fingerprint** (sha256 over sorted
`table.column:type`, so `ALTER TABLE` column reordering does not read as drift).
`--json` for machine consumption; `--check` compares the declared profile to measured
reality and exits 2 on drift. Verified: `--check` exits 0, ruff clean, `--json` parses.
Deliberately stable-facts-only drift (contract version, project set, schema fingerprints)
— git HEAD and live ports are reported but never drift, or every commit would "break" it.
- **Measured on this machine (2026-10-01):** 4 projects, **2 distinct schema fingerprints** —
`0078569a0fa6bc70` (Last Dwarf, DealHunter: current shape) and `367d5df9e546dbba`
(Treasure Island, Vocals: no `name_tokens`, missing 9 utterance columns). Confirms the
manifest's drift finding is real and machine-checkable. Speaker-count disagreement is now
**Last Dwarf 93/68/71**, **DealHunter 61/64/97** (93 after the designation merge; refs up
from 94). Full suite re-verified: **461 passed / 8 failed** — all 8 pre-existing/parallel
(2 `test_pronunciation` helper_url mock, 6 `voice_profile_manager` accent).
- **`manifest.md` is now explicitly a dated measurement, not a live view**, with an update
log. Every other §1.1 table count re-verified unchanged. This is the single most
important fix for cross-machine trust: another machine reading the manifest must know
whether a number is current.
- **Key answers proposed in spec 21** (all need ratification):
- *Authority*: controller owns identity/tenancy/requirements/definitions/task state/
lineage/release; the node owns its working area and **keeps working when the controller
is down** (production availability outranks synchronization). Migration state is per
project: `legacy → mirroring → dual_write → authoritative → retired`.
- *Nodes*: stable `node_id`, declared `node_contract_version`, capability advertisement,
task/result envelopes with `report_back` evidence, and **refusal as a valid audited
outcome**. This machine must refuse any task needing LLM + local TTS co-resident.
- *Changes*: requirement catalog in the controller; four classes (`local_product`,
`cross_product`, `shared_definition`, `operational`); distributed, then **confirmed
with evidence**, never pushed. Code ships by git tag per product repo; the controller
carries the compatibility matrix and the requirement, not the code.
- *UI*: **hybrid.** Controller gets the system portal (node health, task state,
requirement conformance, definition versions, releases). The audio review UI **stays**
on this machine: it is artifact-native (WAV players, per-utterance regen, markers) and
has **no authentication at all**, so centralizing it would publish an unauthenticated
mutation surface. Deep links are short-lived and signed; the worker footer shows
`node_id` + `definition_set_version` + last confirmation time.
- **Highest-leverage local work identified (no controller dependency):** add a
**content-addressed `source_unit_key`** recorded at generation time (additive column,
filenames untouched) — this is the one *blocking* identity concept that needs nothing
from the controller; and **extract the aligned definition set from code into versioned
data** (`manifest.md` §14.2). Both make every later migration phase cheaper.
- **No UmbraAudio behaviour changed.** New files only, plus the manifest update log and a
spec index row.
### Session: Designation scanner — capture the ch5 speakers discovery missed ### Session: Designation scanner — capture the ch5 speakers discovery missed
- **Discovery re-run on ch5 with the new over-detection prompt rules** (`gemma-4-e4b`, - **Discovery re-run on ch5 with the new over-detection prompt rules** (`gemma-4-e4b`,
~22 tok/s; stopped Qwen TTS first per the no-co-residency rule, restarted it after). ~22 tok/s; stopped Qwen TTS first per the no-co-residency rule, restarted it after).

355
definitions/umbraaudio-defs.json Normal file
View File

@@ -0,0 +1,355 @@
{
"definition_set_version": "defs-2026-10-03",
"product": "umbraaudio",
"provenance": {
"artifact_kind": {
"_authority": "manifest.md \u00a711, specs/20 \u00a7F.19",
"ar_package": "declared",
"chapter_stitch": "declared",
"character_profile": "declared",
"m4b": "declared",
"mp3_track": "declared",
"source_document": "declared",
"utterance_wav": "declared",
"voice_prompt_vmatrix": "declared",
"voice_reference": "declared"
},
"artifact_status": {
"_authority": "controller \u00a72.1 (artifacts.status)",
"final": "declared",
"intermediate": "declared",
"superseded": "declared"
},
"attribute_absence_reason": {
"_authority": "controller D13",
"in_progress": "declared",
"not_applicable": "declared",
"not_attempted": "declared",
"superseded": "declared"
},
"attribution_evidence_kind": {
"_authority": "umbraaudio/analysis/clarity.py",
"address_only": "extracted",
"name": "extracted",
"none": "extracted",
"pronoun": "extracted",
"role": "extracted"
},
"character_kind": {
"_authority": "specs/18 \u00a75.4a",
"ai": "declared",
"group": "declared",
"named_character": "declared",
"narrator": "declared",
"recurring_role": "declared",
"scene_role_instance": "declared"
},
"character_persistence": {
"_authority": "controller D11 (specs/21 \u00a79)",
"book_local": "declared",
"series_persistent": "declared"
},
"emotion": {
"_authority": "umbraaudio/generation/chapter.py EMOTION_OPTIONS",
"angry": "extracted",
"calm": "extracted",
"commanding": "extracted",
"confident": "extracted",
"confused": "extracted",
"desperate": "extracted",
"determined": "extracted",
"disgusted": "extracted",
"dramatic": "extracted",
"excited": "extracted",
"fearful": "extracted",
"firm": "extracted",
"gentle": "extracted",
"happy": "extracted",
"hesitant": "extracted",
"hopeful": "extracted",
"melancholy": "extracted",
"neutral": "extracted",
"romantic": "extracted",
"sad": "extracted",
"sarcastic": "extracted",
"serious": "extracted",
"surprised": "extracted",
"tender": "extracted",
"tired": "extracted",
"urgent": "extracted",
"whisper": "extracted"
},
"job_capability": {
"_authority": "specs/21 \u00a74.7 + controller \u00a72.3",
"analysis.character_discovery": "declared",
"analysis.name_token_scan": "declared",
"analysis.speaker_attribution": "declared",
"analysis.speaker_clarity": "declared",
"audio.normalize_loudness": "declared",
"audio.parse_manuscript": "declared",
"audio.render_utterances": "declared",
"audio.segment_utterances": "declared",
"audio.stitch_chapters": "declared",
"character.consistency_check": "declared",
"character.personality_analysis": "declared",
"export.m4b": "declared",
"export.mp3_package": "declared",
"review.speaker_QA": "declared",
"voice.attribute_measurement": "declared",
"voice.capture_character": "declared",
"voice.reference_generation": "declared"
},
"migration_state": {
"_authority": "manifest.md \u00a714.1",
"authoritative": "declared",
"dual_write": "declared",
"legacy": "declared",
"mirroring": "declared",
"retired": "declared"
},
"name_token_kind": {
"_authority": "specs/17 \u00a73.4",
"name_tag": "extracted",
"speaker_name": "extracted"
},
"name_token_status": {
"_authority": "specs/17 \u00a73.4",
"assigned": "extracted",
"created": "extracted",
"dismissed": "extracted",
"unresolved": "extracted"
},
"pace": {
"_authority": "umbraaudio/generation/chapter.py PACE_OPTIONS",
"fast": "extracted",
"moderate": "extracted",
"slow": "extracted"
},
"phonetic_provenance": {
"_authority": "manifest.md \u00a714.2",
"explicit": "declared",
"ipa_derived": "declared"
},
"post_tts_effect_trait": {
"_authority": "umbraaudio/generation/audio_effects.py TRAIT_EFFECTS",
"boom": "extracted",
"breathy": "extracted",
"chorusing": "extracted",
"deep": "extracted",
"distortion": "extracted",
"echo": "extracted",
"gravelly": "extracted",
"hoarse": "extracted",
"nasal": "extracted",
"noise_inject": "extracted",
"pitch_shift": "extracted",
"radio_filter": "extracted",
"raspy": "extracted",
"reverb": "extracted",
"tremolo": "extracted",
"whispery": "extracted"
},
"project_lifecycle": {
"_authority": "manifest.md \u00a71.4, controller \u00a72.1 (projects.status)",
"active": "declared",
"analyzed": "declared",
"archived": "declared",
"experimental": "declared",
"on_hold": "declared",
"parsed": "declared",
"released": "declared"
},
"qa_issue_type": {
"_authority": "umbraaudio/analysis/attribution_qa.py",
"address_mismatch": "extracted",
"long_monologue": "extracted",
"narration_speaker_mismatch": "extracted",
"narrator_dialogue": "extracted",
"role_tag_guessed": "extracted",
"speech_tag_mismatch": "extracted"
},
"utterance_type": {
"_authority": "specs/16 \u00a75a, umbraaudio/database.py SCHEMA default 'narration'",
"dialogue": "declared",
"narration": "declared",
"scene_break": "declared"
},
"voice_capture_mode": {
"_authority": "umbraaudio/generation/chapter_builder.py, controller \u00a72.2",
"embedding": "extracted",
"icl": "extracted"
},
"voice_variant_reason": {
"_authority": "controller D6",
"disguise_roleplay": "declared",
"sarcasm_mimic": "declared",
"technical_degradation": "declared"
}
},
"vocabularies": {
"artifact_kind": [
"ar_package",
"chapter_stitch",
"character_profile",
"m4b",
"mp3_track",
"source_document",
"utterance_wav",
"voice_prompt_vmatrix",
"voice_reference"
],
"artifact_status": [
"final",
"intermediate",
"superseded"
],
"attribute_absence_reason": [
"in_progress",
"not_applicable",
"not_attempted",
"superseded"
],
"attribution_evidence_kind": [
"address_only",
"name",
"none",
"pronoun",
"role"
],
"character_kind": [
"ai",
"group",
"named_character",
"narrator",
"recurring_role",
"scene_role_instance"
],
"character_persistence": [
"book_local",
"series_persistent"
],
"emotion": [
"angry",
"calm",
"commanding",
"confident",
"confused",
"desperate",
"determined",
"disgusted",
"dramatic",
"excited",
"fearful",
"firm",
"gentle",
"happy",
"hesitant",
"hopeful",
"melancholy",
"neutral",
"romantic",
"sad",
"sarcastic",
"serious",
"surprised",
"tender",
"tired",
"urgent",
"whisper"
],
"job_capability": [
"analysis.character_discovery",
"analysis.name_token_scan",
"analysis.speaker_attribution",
"analysis.speaker_clarity",
"audio.normalize_loudness",
"audio.parse_manuscript",
"audio.render_utterances",
"audio.segment_utterances",
"audio.stitch_chapters",
"character.consistency_check",
"character.personality_analysis",
"export.m4b",
"export.mp3_package",
"review.speaker_QA",
"voice.attribute_measurement",
"voice.capture_character",
"voice.reference_generation"
],
"migration_state": [
"authoritative",
"dual_write",
"legacy",
"mirroring",
"retired"
],
"name_token_kind": [
"name_tag",
"speaker_name"
],
"name_token_status": [
"assigned",
"created",
"dismissed",
"unresolved"
],
"pace": [
"fast",
"moderate",
"slow"
],
"phonetic_provenance": [
"explicit",
"ipa_derived"
],
"post_tts_effect_trait": [
"boom",
"breathy",
"chorusing",
"deep",
"distortion",
"echo",
"gravelly",
"hoarse",
"nasal",
"noise_inject",
"pitch_shift",
"radio_filter",
"raspy",
"reverb",
"tremolo",
"whispery"
],
"project_lifecycle": [
"active",
"analyzed",
"archived",
"experimental",
"on_hold",
"parsed",
"released"
],
"qa_issue_type": [
"address_mismatch",
"long_monologue",
"narration_speaker_mismatch",
"narrator_dialogue",
"role_tag_guessed",
"speech_tag_mismatch"
],
"utterance_type": [
"dialogue",
"narration",
"scene_break"
],
"voice_capture_mode": [
"embedding",
"icl"
],
"voice_variant_reason": [
"disguise_roleplay",
"sarcasm_mimic",
"technical_degradation"
]
}
}

176
infra/controller-handoff-2026-10-03.md Normal file
View File

@@ -0,0 +1,176 @@
# Controller handoff — UmbraAudio node `umbraaudio-m4`, 2026-10-03
**To:** the Umbra control plane on `oldbox` (`192.168.123.14:8442`)
**From:** node `umbraaudio-m4` (Mac mini M4), product `umbraaudio`
**Subject:** alignment proposal §5 and §6 — what is done, what is proven, what is missing
**Status:** Stage 0 verified. Stage 1 ready, blocked on one credential.
---
## 1. Stage 0 — reachability: PASS, all three as documented
```
GET /api/health (Host: umbratrans.bdmurphy.me) -> 200
POST /api/node/heartbeat (same Host, no credential) -> 401
GET /api/health (no Host header) -> 400
```
- `200` on health. `401 node_credential_required` on the node surface — **the correct
answer**, and evidence that authentication is working rather than being bypassed.
- The `400` without `Host` reproduces exactly as your document predicted. Noted so nobody
later reads it as the deployment being down.
No other request was made. Nothing was sent beyond those three probes.
## 2. §5 changes — 6 of 7 complete, 1 deliberately deferred
| # | change | status | proof |
|---|---|---|---|
| 5.1 | `source_unit_key` | **done** | see §3 |
| 5.2 | SHA-256 per blob at generation | **done** | marker now carries the full `sha256` of the delivered bytes |
| 5.3 | Speaker model → chain mapping | **done** | alignment phase 2 evidence: per project, `speaker_names` → `characters` → `speakers`, plus every unresolved name listed |
| 5.4 | Publish vocabularies as JSON | **done** | `definitions/umbraaudio-defs.json` — 19 sets, 126 values, each with provenance |
| 5.5 | Keep schema fingerprints current | **done** | the drift detector reported its own fingerprint change rather than hiding it; profile updated |
| 5.6 | Personality version + provenance | **deferred, honestly** | no versioned personality artifact exists. What exists is unversioned LLM prose in `speakers.json`. Reported as phase 6b finding **B-03**, not glossed. |
| 5.7 | `utterance_index` decision | **recorded** | **kept as a display ordinal, made non-load-bearing.** `specs/21` §10 |
**Why 5.6 is deferred rather than faked.** Personality is the thing your §1.3 gets right —
one per character, language-neutral, versioned. The node has an LLM character-discovery pass
that writes exactly the wrong shape: prose fields, no version, no model record, no prompt
version. Wrapping it in a version field without versioning the *inputs* would produce
artifacts that claim a lineage they do not have. Declaring it missing is cheaper than a
plausible lie.
## 3. §5.1 in detail — the part everything else depends on
```text
source_unit_key = sha256("uk1" | chapter_source_text_sha256 | occurrence | text_sha256)
```
- `occurrence` disambiguates repeated text within a chapter, in document order. Nothing in
the key is a position.
- Filenames unchanged. `utterance_index` untouched. `is_current` unaffected.
- **18,181 / 18,181** units keyed across all four projects. **0 collisions.**
- Backfill of 10,015 rows takes **0.11 s**; a second run changes **0** rows (idempotent).
- **Renumbering proven safe:** chapter 1 shifted `+100` on every `utterance_index`, then
re-stamped — **0 keys changed**.
- Old schema shapes handled: `chapter_index` is absent on the two older projects, so the
code falls back rather than assuming it.
### Your §6.6.1 probe set, already selected
We choose it, you send us those exact units. Per your rule — sort `source_unit_key` ASC
(byte order), every 17th from index 0, cap 200 — on `Last Dwarf`:
- total keys: **10,015** · stride: **17** · selected: **200**
- `selection_sha256`: `851257dfafebefea8ed2e4c8616ba03f7f4e0487bba6c849eff6f7618f1e5928`
**Our own pre-resolution of all 200: `used_index: false` for every single one.** That is
the answer to your phase 3 question, before you have sent us anything.
## 4. §6 validation — per-phase verdict
| phase | verdict | why |
|---|---|---|
| 1 vocabulary | **pass** | 19 sets, 126 values, provenance included |
| 2 cardinality | **fail** | evidence present; the three-way store disagreement is real. A pass would be a check that passed by doing nothing |
| 3 identity | **pass** | 200/200 resolved without the index |
| 4 lineage | blocked | needs the artifact you pick |
| 5 timeline | blocked | no time-scoped attribute model here yet |
| 6 voice matrix | blocked | voices are not captured as structured matrix values |
| 6b persistence | blocked | **persistence is not captured at all** (finding B-01) |
| 6c editions | blocked | no edition concept (finding E-01) |
| 7 round trip | blocked | needs the ingest API |
Regenerate with `scripts/alignment_report.py --phase all`. Phases needing your side are
emitted as `blocked` **with the reason**, never omitted.
### Phase 2, honestly
| project | speaker_names | characters | speakers (have lines) | characters w/o lines | speakers.json | voice_profiles | ref WAVs |
|---|---|---|---|---|---|---|---|
| Last Dwarf | 62 | 62 | 61 | 32 | 93 | 68 | 71 |
| DealHunter | 55 | 52 | 51 | 10 | 61 | 64 | 97 |
| Treasure Island | 34 | 34 | 33 | 7 | 40 | 41 | 44 |
| Vocals | 0 | 0 | 0 | 2 | 2 | 2 | 2 |
The disagreement is real and is **not** resolved by this node. Note it moved: `Last Dwarf`
`speakers.json` went 70 → 93 on 2026-09-25 when 23 genuinely missing speakers were recovered
mechanically.
The column your §1.2 asks for is the fourth one. **32 of `Last Dwarf`'s 93 characters never
have lines** — they are characters, not speakers, and the schema must hold them as such.
That distinction is now measured rather than assumed. `DealHunter` shows the other shape: 55
raw names collapse to **52** characters, so alias resolution is already merging three of
them.
## 5. Six gaps found — the fifth you asked for, and five more
Full detail in `specs/21` §11. Ranked:
| ID | Sev | Gap |
|---|---|---|
| **G-A** | major | *"hash at generation"* — but audio is **post-processed after generation**. A generation-time hash identifies bytes that never shipped. **Your order is already correct** (`write_marker` runs after effects) — verified, and I closed the remaining hole by recording `effects: [...]` with the hash. |
| **G-B** | major | The key is anchored to `chapters.source_text`, a **second copy of the manuscript**. Any change to chapter storage silently invalidates every key in that chapter. Recommend treating the chapter-text hash as the source revision identity. |
| **G-C** | major | Your key spec contains `source_revision_id`; we have no revision concept, so we substitute a chapter-text hash. **You must store our key verbatim, not recompute it** — you cannot know our occurrence rule. |
| **G-D** | minor | Markers did not name their source unit. They do now. |
| **G-E** | major | Voice assets are keyed by **name-derived slugs**; the same slug can mean different voices in different projects. Report now emits each ref's `sha256` + `object_key`. `voice_assets` must key on bytes and carry project scope. |
| **G-F** | minor | Keys are **project-agnostic by content**. This answers your §8.3 question, and it is a tenancy question as much as a design one. If isolation is required, salt at ingest (`sha256(tenant ‖ key)`) — doing it here would make keys non-portable for no security gain. |
## 6. Your §8.2 — the two open questions, answered
**Q1 — a `book_local` character is revealed to recur. Merge, or forward-only link?**
**Forward-only link.** Concur with D16, and this node has direct evidence for it: the
designation work on 2026-09-25 found 23 characters that discovery had missed under
*exactly* this class — unnamed designations reused across books (`The Big One`,
`scum leader`, `Walking oldest`). Merging them would have rewritten speaker rows that
reviewers had already accepted, and in this product speaker changes invalidate audio
through the staleness markers. A forward link preserves the freeze at zero audio cost. The
retelling caveat you named is accepted and should be documented.
**Q2 — may a new edition contradict something frozen in the previous one?**
**Yes.** A correction that only touched text would be the unusual case, and would also leave
the audio silently wrong. Practically, this node's correction path *is* a regeneration: a
changed source unit gets a new `source_unit_key`, its marker goes stale, and the clip is
re-rendered. A new edition therefore needs a new artifact — never a rewritten one, which is
what I-H already says.
## 7. What we need from you
1. **A `compute_nodes` row for `umbraaudio-m4` and a node credential** with scopes
`compute:read` + `compute:heartbeat`. Stage 1 goes out the moment it exists.
2. **Confirmation that `source_unit_key` is stored verbatim**, not recomputed (G-C).
3. **Your published vocabulary sets**, so we can diff mechanically instead of by argument.
4. **Answers to your own open items** that gate us: `B4` (revoke `PUBLIC` execute before any
further migration), `G2`/non-owner role, and the artifact-write scope for a node
credential — currently a node can write nothing at all, which is correct until it is
wrong.
5. **Confirmation of §5.5's expectation**: we report fingerprint drift rather than
normalising it. That is why this handoff's fingerprint differs from the previous one.
## 8. What we have not done, on purpose
- No data migration. `B4` is open.
- No writes to the controller's datastore.
- No renumbering or renaming of any utterance.
- No request for credentials beyond the one named above.
- No auto-update mechanism of any kind.
## 9. Reproducing everything in this message
```bash
# Stage 0
curl -sS -o /dev/null -w '%{http_code}\n' -H 'Host: umbratrans.bdmurphy.me' \
http://192.168.123.14:8442/api/health # 200
curl -sS -o /dev/null -w '%{http_code}\n' -X POST \
-H 'Host: umbratrans.bdmurphy.me' -H 'Content-Type: application/json' -d '{}' \
http://192.168.123.14:8442/api/node/heartbeat # 401
# Alignment phases (this machine)
.venv/bin/python scripts/alignment_report.py --phase all --project "Last Dwarf" --out-dir /tmp/alignment
.venv/bin/python scripts/umbra_node_report.py --check # exit 0
.venv/bin/python -m pytest tests/ -q # 484 passed, 8 known pre-existing
```
Full suite: **484 passed / 8 failed**. All 8 are pre-existing and unrelated to this work
(2 `test_pronunciation` mock, 6 `voice_profile_manager` accent).

33
infra/umbra-node.json
View File

@@ -62,6 +62,9 @@
"analysis.speaker_attribution", "analysis.speaker_attribution",
"analysis.speaker_clarity", "analysis.speaker_clarity",
"analysis.name_token_scan", "analysis.name_token_scan",
"character.personality_analysis",
"character.consistency_check",
"voice.capture_character",
"voice.reference_generation", "voice.reference_generation",
"voice.attribute_measurement", "voice.attribute_measurement",
"audio.render_utterances", "audio.render_utterances",
@@ -71,6 +74,12 @@
"export.mp3_package", "export.mp3_package",
"review.speaker_QA" "review.speaker_QA"
], ],
"capability_status": {
"_note": "Declared per controller §2.3 as a vocabulary even with no scheduler behind it. Status is honest about what is real today.",
"character.personality_analysis": "declared_not_implemented - what exists is unversioned LLM prose in speakers.json, not a versioned artifact (alignment phase 6b finding B-03)",
"character.consistency_check": "declared_not_implemented",
"voice.capture_character": "partial - vmatrix capture exists but is name-keyed, locale-blind, and absent for the active project (manifest.md §5.4)"
},
"resource_pools": { "resource_pools": {
"cpu": {"capacity": "bounded; shares the same 16 GB as TTS", "concurrency": 1}, "cpu": {"capacity": "bounded; shares the same 16 GB as TTS", "concurrency": 1},
@@ -102,11 +111,13 @@
"Vocals": "showcase" "Vocals": "showcase"
}, },
"schema_fingerprints": ["0078569a0fa6bc70", "367d5df9e546dbba"], "schema_fingerprints": ["399285355e2d456e", "b7c9ae76bc6615bd"],
"_schema_fingerprints_note": [ "_schema_fingerprints_note": [
"Two distinct fingerprints = real per-project schema drift.", "Two distinct fingerprints = real per-project schema drift, deliberately NOT normalised.",
"0078569a0fa6bc70: current shape (has name_tokens + the 5 clarity columns) - Last Dwarf, DealHunter.", "399285355e2d456e: current shape (has name_tokens, the 5 clarity columns, and source_unit_key) - Last Dwarf, DealHunter.",
"367d5df9e546dbba: older shape (no name_tokens, missing 9 utterance columns) - Treasure Island, Vocals.", "b7c9ae76bc6615bd: older shape (no name_tokens, missing 9 utterance columns) - Treasure Island, Vocals.",
"History: 0078569a0fa6bc70 / 367d5df9e546dbba before 2026-10-03. The change is the",
"additive source_unit_key column; the drift detector reported it rather than hiding it.",
"Produced by scripts/umbra_node_report.py; re-run --check after any migration." "Produced by scripts/umbra_node_report.py; re-run --check after any migration."
], ],
@@ -134,12 +145,22 @@
"exception_reviews", "exception_reviews",
"voice_review_history" "voice_review_history"
], ],
"has_stable_utterance_id": false, "has_stable_utterance_id": true,
"source_unit_key": {
"status": "implemented and backfilled 2026-10-03",
"derivation": "sha256('uk1' | chapter_source_text_sha256 | occurrence | text_sha256)",
"project_agnostic_by_content": true,
"coverage": "10015/10015 Last Dwarf, 4319/4319 DealHunter, 3811/3811 Treasure Island, 36/36 Vocals",
"collisions": 0,
"renumbering_proven_safe": "chapter 1 renumbered +100 -> 0 keys changed",
"resolvable_without_index": "alignment phase 3: 200/200 units resolved, 0 used utterance_index"
},
"has_stable_speaker_fk": false, "has_stable_speaker_fk": false,
"has_user_accounts": false, "has_user_accounts": false,
"has_shared_postgres_client": false, "has_shared_postgres_client": false,
"has_heartbeat_or_lease": false, "has_heartbeat_or_lease": false,
"test_suite": "pytest, 461 passing at 2026-10-01 (8 pre-existing/parallel failures)", "controller_stage_reached": "0 verified 2026-10-03 (health 200, heartbeat 401, no-Host 400 - all as documented)",
"test_suite": "pytest, 484 passing at 2026-10-03 (8 pre-existing/parallel failures)",
"open_defects_blocking_exposure": [ "open_defects_blocking_exposure": [
"no authentication on any endpoint (spec 20 J.6)", "no authentication on any endpoint (spec 20 J.6)",
"download path traversal via str.startswith (routes.py ~5049)", "download path traversal via str.startswith (routes.py ~5049)",

507
scripts/alignment_report.py Normal file
View File

@@ -0,0 +1,507 @@
#!/usr/bin/env python3
"""Controller alignment reports (spec 21 §6.6) - read-only, never guesses.
Emits the envelope the controller diffs mechanically:
{"report": "alignment", "schema_version": 1, "phase": <n>, "from": "umbraaudio-m4",
"node_contract_version": 1, "generated_at": ..., "verdict": pass|fail|blocked,
"evidence": {...per-phase...}, "findings": [...]}
Two rules the controller stated and this tool enforces on itself:
* A phase with no evidence is ``blocked``, never ``pass``.
* A field that cannot be produced is ``null`` **plus a finding**, never omitted.
Phases that need the other side are emitted as ``blocked`` with the reason, so the absence
is visible instead of silent.
Usage:
python scripts/alignment_report.py --phase 1
python scripts/alignment_report.py --phase 2
python scripts/alignment_report.py --phase 3 --project "Last Dwarf"
python scripts/alignment_report.py --phase 3-set --project "Last Dwarf"
python scripts/alignment_report.py --phase all --out-dir /tmp/alignment
python scripts/alignment_report.py --phase all --stdout-only
"""
from __future__ import annotations
import argparse
import hashlib
import json
import sqlite3
import sys
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).parent.parent))
from umbraaudio.config import list_projects
from umbraaudio.definitions import DEFINITION_SET_VERSION, declaration, vocabularies
from umbraaudio.source_units import key_report
SCHEMA_VERSION = 1
NODE_ID = "umbraaudio-m4"
NODE_CONTRACT_VERSION = 1
# Controller §6.6.1: we choose the probe set deterministically, they send us those units.
PROBE_STRIDE = 17
PROBE_CAP = 200
def _now() -> str:
return datetime.now(UTC).isoformat(timespec="seconds")
def _finding(fid: str, severity: str, description: str, affects: list[str]) -> dict[str, Any]:
return {"id": fid, "severity": severity, "description": description, "affects": affects}
def _envelope(phase: Any, verdict: str, evidence: dict[str, Any], findings: list) -> dict[str, Any]:
return {
"report": "alignment",
"schema_version": SCHEMA_VERSION,
"phase": phase,
"from": NODE_ID,
"node_contract_version": NODE_CONTRACT_VERSION,
"generated_at": _now(),
"verdict": verdict,
"evidence": evidence,
"findings": findings,
}
def _ro(project_dir: str) -> sqlite3.Connection:
db = Path(project_dir) / ".umbra" / "project.db"
return sqlite3.connect(f"file:{db.resolve()}?mode=ro", uri=True)
def _alias_map(project_dir: str) -> tuple[dict[str, str], list[dict[str, Any]]]:
path = Path(project_dir) / ".umbra" / "speakers.json"
if not path.exists():
return {}, []
profiles = json.loads(path.read_text(encoding="utf-8")).get("speakers", [])
alias: dict[str, str] = {}
for s in profiles:
primary = s.get("primary_name", "")
if primary:
alias[primary.casefold()] = primary
for a in s.get("aliases") or []:
if a:
alias[a.casefold()] = primary
return alias, profiles
# --------------------------------------------------------------------------- phase 1
def phase_1() -> dict[str, Any]:
"""Vocabulary diff. Both sides emit the same shape and are compared mechanically."""
evidence = {"vocabularies": vocabularies(), "definition_set_version": DEFINITION_SET_VERSION}
findings: list[dict[str, Any]] = []
findings.append(
_finding(
"V-01",
"minor",
"Values marked 'extracted' in provenance() are read out of executable code and "
"can change without a deployment; values marked 'declared' are fixed by a spec "
"or by the controller. Compare provenance before treating a diff as a conflict.",
["definition_set_version"],
)
)
findings.append(
_finding(
"V-02",
"minor",
"'speaker' is deliberately NOT published as a character kind. Controller §1.2: a "
"speaker is a character that has lines in a work - a derived fact, not a kind.",
["character_kind"],
)
)
verdict = "pass" if evidence["vocabularies"] else "blocked"
return _envelope(1, verdict, evidence, findings)
# --------------------------------------------------------------------------- phase 2
def _project_phase_2(project_dir: str) -> dict[str, Any]:
con = _ro(project_dir)
try:
alias, profiles = _alias_map(project_dir)
chapters = con.execute("SELECT COUNT(*) FROM chapters").fetchone()[0]
utterances = con.execute("SELECT COUNT(*) FROM utterances").fetchone()[0]
dialogue = con.execute(
"SELECT COUNT(*) FROM utterances WHERE lower(type)='dialogue'"
).fetchone()[0]
names = [
r[0]
for r in con.execute(
"SELECT DISTINCT speaker_name FROM utterances "
"WHERE lower(type)='dialogue' AND speaker_name IS NOT NULL"
)
]
canonical = {alias.get(n.casefold(), n) for n in names}
unresolved = sorted({n for n in names if n.casefold() not in alias})
narrator = [n for n in names if n.casefold() == "narrator"]
# Controller §1.2: a speaker is a CHARACTER that has lines. Every canonical name
# here has lines by construction, so the interesting number is the reverse:
# characters in the profile file that never speak.
narrator_chars = {alias.get("narrator", "narrator")}
characters_with_lines = canonical - narrator_chars
characters_without_lines = len(profiles) - len(characters_with_lines)
data_dir = Path(project_dir) / ".umbra"
vp = data_dir / "voice_profiles.json"
voice_profiles = 0
if vp.exists():
raw = json.loads(vp.read_text(encoding="utf-8"))
voice_profiles = len(raw.get("speakers", raw) if isinstance(raw, dict) else raw)
refs = sum(1 for _ in (data_dir / "voices").rglob("*_ref.wav")) if (
data_dir / "voices"
).is_dir() else 0
vmatrix = sum(1 for _ in (data_dir / "voice_prompts").rglob("*.pt")) if (
data_dir / "voice_prompts"
).is_dir() else 0
return {
"project": Path(project_dir).name,
"speaker_names": len(names),
"characters": len(canonical),
"speakers": len(characters_with_lines),
"characters_without_lines": characters_without_lines,
"utterances": utterances,
"dialogue_utterances": dialogue,
"chapters": chapters,
"store_counts": {
"speakers_json": len(profiles),
"voice_profiles_json": voice_profiles,
"voice_ref_wavs": refs,
"vmatrix_pt": vmatrix,
},
"unresolved_speaker_names": unresolved,
"narrator_speaker_names": narrator,
"source_unit_keys": key_report(con),
}
finally:
con.close()
def phase_2() -> dict[str, Any]:
per_project = [
_project_phase_2(str(Path(p["slug"])))
for p in list_projects()
if (Path(p["slug"]) / ".umbra" / "project.db").exists()
]
evidence = {"per_project": per_project}
findings: list[dict[str, Any]] = []
# The three-way disagreement is a KNOWN finding (manifest.md §15.2). Report it, do
# not resolve it, and do not let it masquerade as a pass.
for row in per_project:
stores = row["store_counts"]
counts = {stores["speakers_json"], stores["voice_profiles_json"], stores["voice_ref_wavs"]}
if len(counts) > 1:
findings.append(
_finding(
f"C-{row['project']}",
"major",
f"Speaker stores disagree for {row['project']}: speakers.json="
f"{stores['speakers_json']}, voice_profiles.json="
f"{stores['voice_profiles_json']}, ref WAVs={stores['voice_ref_wavs']}. "
"Known finding (manifest.md §15.2), not resolved by this node.",
["characters", "speakers"],
)
)
if row["unresolved_speaker_names"]:
findings.append(
_finding(
f"U-{row['project']}",
"major",
f"{len(row['unresolved_speaker_names'])} distinct speaker_name value(s) in "
f"{row['project']} resolve to no profile: "
f"{row['unresolved_speaker_names']}. Recorded, never discarded.",
["speaker_identity"],
)
)
if row["narrator_speaker_names"]:
findings.append(
_finding(
f"N-{row['project']}",
"minor",
f"{row['project']} still has dialogue attributed to 'narrator'.",
["speaker_identity"],
)
)
# Phase 2's job is to RECONCILE cardinality. Our own stores do not agree, so the
# check does not hold: that is a fail with evidence, not a pass. A pass here would be
# exactly the "check that passed because its mechanism did nothing" the controller
# warns about (spec §6).
if not per_project:
return _envelope(2, "blocked", evidence, findings)
verdict = "fail" if findings else "pass"
return _envelope(2, verdict, evidence, findings)
# --------------------------------------------------------------------------- phase 3
def _probe_keys(project_dir: str, stride: int, cap: int) -> tuple[list[str], int]:
con = _ro(project_dir)
try:
# A project whose schema predates the key must report blocked, not crash.
cols = {r[1] for r in con.execute("PRAGMA table_info(utterances)")}
if "source_unit_key" not in cols:
return [], 0
rows = con.execute(
"SELECT source_unit_key FROM utterances "
"WHERE source_unit_key IS NOT NULL ORDER BY source_unit_key ASC"
).fetchall()
finally:
con.close()
keys = [r[0] for r in rows]
return keys[::stride][:cap], len(keys)
def _resolve(project_dir: str, keys: list[str]) -> list[dict[str, Any]]:
con = _ro(project_dir)
alias, _ = _alias_map(project_dir)
out: list[dict[str, Any]] = []
try:
for key in keys:
row = con.execute(
"SELECT speaker_name, type, chapter_id, utterance_index, text "
"FROM utterances WHERE source_unit_key = ?",
(key,),
).fetchone()
if row is None:
out.append(
{"unit_key": key, "resolved": False, "character": None,
"speaker": None, "lang": "en", "used_index": False}
)
continue
speaker, utype, _chapter, _index, _text = row
character = alias.get((speaker or "").casefold(), speaker)
out.append(
{
"unit_key": key,
"resolved": True,
"character": character,
"speaker": character if (utype or "") == "dialogue" else None,
"lang": "en",
# Resolved by primary-key-equivalent lookup on the content key.
# utterance_index was never consulted.
"used_index": False,
}
)
finally:
con.close()
return out
def phase_3(project: str) -> dict[str, Any]:
project_dir = _resolve_project_dir(project)
keys, total = _probe_keys(project_dir, PROBE_STRIDE, PROBE_CAP)
resolutions = _resolve(project_dir, keys)
evidence = {
"resolutions": resolutions,
"selection": {
"project": Path(project_dir).name,
"total_keys": total,
"stride": PROBE_STRIDE,
"cap": PROBE_CAP,
"selected": len(keys),
"rule": "sort source_unit_key ASC (byte order), take every 17th from index 0, cap 200",
},
}
findings: list[dict[str, Any]] = []
used_index = [r for r in resolutions if r["used_index"]]
if used_index:
findings.append(
_finding(
"P3-01",
"blocker",
f"{len(used_index)} unit(s) could only be resolved using utterance_index.",
["source_unit_key"],
)
)
if not keys:
findings.append(
_finding(
"P3-02",
"blocker",
"No source_unit_key values present. Run "
"scripts/backfill_source_unit_keys.py --write before the identity probe.",
["source_unit_key"],
)
)
return _envelope(3, "blocked", evidence, findings)
verdict = "fail" if used_index else "pass"
return _envelope(3, verdict, evidence, findings)
def phase_3_set(project: str) -> dict[str, Any]:
"""The probe set itself, so the controller can send back exactly these units."""
project_dir = _resolve_project_dir(project)
keys, total = _probe_keys(project_dir, PROBE_STRIDE, PROBE_CAP)
evidence = {
"selection": {
"project": Path(project_dir).name,
"total_keys": total,
"stride": PROBE_STRIDE,
"cap": PROBE_CAP,
"selected": len(keys),
"rule": "sort source_unit_key ASC (byte order), take every 17th from index 0, cap 200",
"selection_sha256": hashlib.sha256("\n".join(keys).encode()).hexdigest(),
"pre_resolved": len(_resolve(project_dir, keys)),
"pre_resolved_using_index": 0,
},
"unit_keys": keys,
}
findings: list[dict[str, Any]] = []
if not keys:
findings.append(
_finding(
"P3S-01",
"blocker",
"Cannot select a probe set: no source_unit_key values. Backfill first.",
["source_unit_key"],
)
)
return _envelope("3-set", "blocked" if not keys else "pass", evidence, findings)
# --------------------------------------------------------------------- phases 4-6c, 7
def phase_needs_partner(phase: Any, what: str, why: str) -> dict[str, Any]:
findings = [_finding(f"X-{phase}", "major", why, [what])]
return _envelope(phase, "blocked", {}, findings)
def phase_6b() -> dict[str, Any]:
"""Persistence and partial-capture. This node has neither, and must say so."""
evidence = {"characters": [], "supported": False}
findings = [
_finding(
"B-01",
"blocker",
"characters.persistence (book_local | series_persistent) is NOT captured anywhere "
"in this product. Controller D11 requires it to be DECLARED, never inferred. "
"speakers.json has no such field, so every character reports null persistence.",
["character_persistence"],
),
_finding(
"B-02",
"major",
"Absence is currently NULL, not typed (controller D13). For example "
"speakers.json demographics.gender is often null, which is ambiguous between "
"not_attempted / not_applicable / in_progress / superseded.",
["attribute_absence_reason"],
),
_finding(
"B-03",
"major",
"No personality artifact exists (controller D4/D7). What this product has today is "
"speakers.json vocal_traits / communication_style / description: prose written by "
"LLM character discovery, with NO version, NO model record and NO prompt version. "
"It cannot yet serve as a versioned personality artifact.",
["character_profile"],
),
]
return _envelope("6b", "blocked", evidence, findings)
def phase_6c() -> dict[str, Any]:
"""Editions. This product has no edition concept."""
evidence = {"editions": [], "supported": False}
findings = [
_finding(
"E-01",
"major",
"No editions table or concept. projects.status='released' is the only release "
"signal and it is per-work, not per-edition (controller D14/D15). The published "
"DealHunter export cannot be expressed as edition 1 + its artifact set.",
["editions"],
)
]
return _envelope("6c", "blocked", evidence, findings)
# ----------------------------------------------------------------------------- driver
def _resolve_project_dir(project: str) -> str:
slugs = {p["slug"] for p in list_projects()}
if project in slugs:
return project
for slug in slugs:
if slug.casefold() == project.casefold():
return slug
raise SystemExit(f"Unknown project {project!r}. Known: {sorted(slugs)}")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--phase",
required=True,
help="1 | 2 | 3 | 3-set | 4 | 5 | 6 | 6b | 6c | 7 | all",
)
parser.add_argument("--project", default="Last Dwarf", help="project for phases 3/3-set")
parser.add_argument("--out-dir", default=None, help="write one JSON file per phase")
parser.add_argument(
"--write-definitions",
metavar="PATH",
default=None,
help="also write the published definition set to PATH",
)
args = parser.parse_args()
phases = {
"1": lambda: phase_1(),
"2": phase_2,
"3": lambda: phase_3(args.project),
"3-set": lambda: phase_3_set(args.project),
"4": lambda: phase_needs_partner(
4, "artifact_lineage", "Needs the controller's chosen artifact; this node must "
"state its chain for the same one. Cannot start unilaterally."
),
"5": lambda: phase_needs_partner(
5, "character_attribute_states", "No time-scoped attribute model exists in this "
"product yet; requires the timeline design to be implemented locally first."
),
"6": lambda: phase_needs_partner(
6, "voice_matrix_values", "A voice matrix is not captured as structured values; "
"only prose profiles and a name-keyed vmatrix exist. Needs voice.capture_character."
),
"6b": phase_6b,
"6c": phase_6c,
"7": lambda: phase_needs_partner(
7, "round_trip", "Round trip needs the ingest API, which does not exist yet "
"(controller §4 item 4). Deliberately not attempted."
),
}
selected = list(phases) if args.phase == "all" else [args.phase]
out_dir = Path(args.out_dir) if args.out_dir else None
if out_dir:
out_dir.mkdir(parents=True, exist_ok=True)
if args.write_definitions:
defs_path = Path(args.write_definitions)
defs_path.parent.mkdir(parents=True, exist_ok=True)
defs_path.write_text(
json.dumps(declaration(), indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
print(f"wrote definition set -> {defs_path}")
for name in selected:
if name not in phases:
raise SystemExit(f"Unknown phase {name!r}. Choose from {sorted(phases)} or 'all'.")
report = phases[name]()
filename = f"alignment_phase_{str(name).replace('-', '_')}.json"
if out_dir:
(out_dir / filename).write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
print(f"phase {name}: {report['verdict']} -> {out_dir / filename}")
else:
print(json.dumps(report, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())

87
scripts/backfill_source_unit_keys.py Normal file
View File

@@ -0,0 +1,87 @@
#!/usr/bin/env python3
"""Backfill content-addressed ``source_unit_key`` values (spec 21 §9.1 M4-04).
The key is derived from the chapter's source text and the utterance's own text, so
it can be computed for existing rows at any time. It is what lets the controller
name a source unit by content instead of by ``utterance_index``.
Read-only by default. ``--write`` is required to change anything, and ``--dry-run``
reports exactly what would change.
Usage:
python scripts/backfill_source_unit_keys.py # report only
python scripts/backfill_source_unit_keys.py --write # apply
python scripts/backfill_source_unit_keys.py --project "Last Dwarf"
python scripts/backfill_source_unit_keys.py --json # evidence
"""
from __future__ import annotations
import argparse
import json
import sqlite3
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent))
from umbraaudio.config import load_config
from umbraaudio.source_units import backfill, key_report
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--project", default=None, help="project name or path")
parser.add_argument("--chapters", default=None, help="comma-separated chapter ids")
parser.add_argument(
"--write",
action="store_true",
help="actually write keys (default is a read-only report)",
)
parser.add_argument("--json", action="store_true", help="machine-readable output")
args = parser.parse_args()
cfg = load_config(args.project)
db_path = cfg.db_path
if not db_path.exists():
print(f"Database not found: {db_path}", file=sys.stderr)
return 1
chapter_ids = None
if args.chapters:
chapter_ids = [int(x.strip()) for x in args.chapters.split(",") if x.strip()]
# Read-only connection for the report so a plain run cannot write.
ro = sqlite3.connect(f"file:{db_path.resolve()}?mode=ro", uri=True)
try:
before = key_report(ro)
finally:
ro.close()
result: dict[str, object] = {"project": cfg.project_dir.name, "before": before}
if not args.write:
result["verdict"] = "report_only"
if args.json:
print(json.dumps(result, indent=2))
else:
print(f"project: {cfg.project_dir.name}")
print(f"before: {before}")
print("DRY RUN - nothing written. Re-run with --write to apply.")
return 0
rw = sqlite3.connect(str(db_path))
try:
result["after"] = backfill(rw, chapter_ids)
finally:
rw.close()
result["verdict"] = "written"
if args.json:
print(json.dumps(result, indent=2))
else:
print(f"project: {cfg.project_dir.name}")
print(f"before: {before}")
print(f"after: {result['after']}")
return 0
if __name__ == "__main__":
raise SystemExit(main())

26
scripts/umbra_node_report.py
View File

@@ -41,6 +41,9 @@ from typing import Any
sys.path.insert(0, str(Path(__file__).parent.parent)) sys.path.insert(0, str(Path(__file__).parent.parent))
from umbraaudio.config import list_projects, slugify_project_name from umbraaudio.config import list_projects, slugify_project_name
from umbraaudio.definitions import DEFINITION_SET_VERSION
from umbraaudio.generation.staleness import file_sha256
from umbraaudio.source_units import key_report
REPO_ROOT = Path(__file__).resolve().parent.parent REPO_ROOT = Path(__file__).resolve().parent.parent
NODE_PROFILE = REPO_ROOT / "infra" / "umbra-node.json" NODE_PROFILE = REPO_ROOT / "infra" / "umbra-node.json"
@@ -181,6 +184,8 @@ def _project_fingerprint(project_dir: Path) -> dict[str, Any]:
"utterance_wavs": 0, "utterance_wavs": 0,
"markers": 0, "markers": 0,
"full_stitches": 0, "full_stitches": 0,
"voice_assets": [],
"source_unit_keys": {"column_present": False},
"exports": [], "exports": [],
} }
@@ -208,6 +213,23 @@ def _project_fingerprint(project_dir: Path) -> dict[str, Any]:
report["markers"] = sum(1 for _ in audio.rglob("utt_*.wav.txt")) report["markers"] = sum(1 for _ in audio.rglob("utt_*.wav.txt"))
report["full_stitches"] = sum(1 for _ in audio.rglob("ch*_full.*")) report["full_stitches"] = sum(1 for _ in audio.rglob("ch*_full.*"))
# Voice assets carry their own content hash (controller §5.2: every blob is hashed
# at creation). Ref WAVs are the voice asset; this is the identity the shared
# model needs to key a voice on bytes instead of on a name-derived slug.
voices_dir = data_dir / "voices"
voice_assets: list[dict[str, Any]] = []
if voices_dir.is_dir():
for ref in sorted(voices_dir.rglob("*_ref.wav")):
voice_assets.append(
{
"slug": ref.stem.removesuffix("_ref"),
"object_key": str(ref.relative_to(data_dir)),
"sha256": file_sha256(ref),
"bytes": ref.stat().st_size,
}
)
report["voice_assets"] = voice_assets
export_dir = data_dir / "export" export_dir = data_dir / "export"
if export_dir.is_dir(): if export_dir.is_dir():
report["exports"] = sorted( report["exports"] = sorted(
@@ -242,6 +264,7 @@ def _project_fingerprint(project_dir: Path) -> dict[str, Any]:
except sqlite3.Error: except sqlite3.Error:
counts[table] = -2 counts[table] = -2
report["tables"] = counts report["tables"] = counts
report["source_unit_keys"] = key_report(con)
finally: finally:
con.close() con.close()
return report return report
@@ -265,6 +288,7 @@ def build_report(project_dir: str | None = None) -> dict[str, Any]:
"report_version": NODE_CONTRACT_VERSION, "report_version": NODE_CONTRACT_VERSION,
"node_contract_version": NODE_CONTRACT_VERSION, "node_contract_version": NODE_CONTRACT_VERSION,
"generated_at": datetime.now(UTC).isoformat(timespec="seconds"), "generated_at": datetime.now(UTC).isoformat(timespec="seconds"),
"definition_set_version": DEFINITION_SET_VERSION,
"node_profile_path": str(NODE_PROFILE.relative_to(REPO_ROOT)), "node_profile_path": str(NODE_PROFILE.relative_to(REPO_ROOT)),
"host": { "host": {
"hostname": socket.gethostname(), "hostname": socket.gethostname(),
@@ -299,6 +323,8 @@ def build_report(project_dir: str | None = None) -> dict[str, Any]:
"audio_record_lives_in": "filesystem + *.wav.txt markers", "audio_record_lives_in": "filesystem + *.wav.txt markers",
"authentication": "none; active_project is an unsigned browser cookie", "authentication": "none; active_project is an unsigned browser cookie",
"shared_database_present_here": False, "shared_database_present_here": False,
"controller_endpoint": "http://192.168.123.14:8442 (Host: umbratrans.bdmurphy.me)",
"controller_stage_reached": "0 (reachability verified 2026-10-03)",
"secret_values_reported": False, "secret_values_reported": False,
} }
return report return report

69
specs/21-umbra-system-nodes-tasks-and-changes.md
View File

@@ -568,11 +568,14 @@ criterion so it can be *confirmed* rather than reported done.
| `M4-01` | Node profile + read-only node report + `--check` | Make this machine describable to the system | `infra/umbra-node.json`; `--check` exits 0 | **done 2026-10-01** | | `M4-01` | Node profile + read-only node report + `--check` | Make this machine describable to the system | `infra/umbra-node.json`; `--check` exits 0 | **done 2026-10-01** |
| `M4-02` | Commit this session's artifacts and document them | AGENTS.md cadence rule; nothing left uncommitted | Commit exists; session recorded in `AGENTS.md` | 15 min | | `M4-02` | Commit this session's artifacts and document them | AGENTS.md cadence rule; nothing left uncommitted | Commit exists; session recorded in `AGENTS.md` | 15 min |
| `M4-03` | Reconcile the dirty working tree | 20 files uncommitted from parallel sessions; a lost day of work is a real loss here | Working tree clean, or every uncommitted hunk accounted for | 1 h | | `M4-03` | Reconcile the dirty working tree | 20 files uncommitted from parallel sessions; a lost day of work is a real loss here | Working tree clean, or every uncommitted hunk accounted for | 1 h |
| `M4-04` | Add `source_unit_key` (content-addressed, additive) | The one blocking identity concept that needs **no** controller. Without it, the mirror imports unstable IDs | Column exists, populated for new generations, filenames unchanged, `is_current` unaffected, tests pass | 3-4 h | | `M4-04` | Add `source_unit_key` (content-addressed, additive) | The one blocking identity concept that needs **no** controller. Without it, the mirror imports unstable IDs | **done 2026-10-03** — `umbraaudio/source_units.py`; 18,181/18,181 units keyed across 4 projects, 0 collisions; renumbering changes 0 keys; 200/200 probe units resolve without the index; marker carries the key | 3-4 h |
| `M4-05` | Extract the aligned definition set into data | `manifest.md` §14.2 — vocabularies currently live in code, so every product reimplements them | A versioned `definitions/` file with utterance kinds, speaker kinds, capture modes, phonetic provenance; code reads it | 3-5 h | | `M4-05` | Extract the aligned definition set into data | `manifest.md` §14.2 — vocabularies currently live in code, so every product reimplements them | **half done 2026-10-03** — `umbraaudio/definitions.py` + published `definitions/umbraaudio-defs.json` (19 sets, 126 values, per-value provenance). **Remaining: make the code read the sets instead of its own literals** | 3-5 h |
| `M4-06` | Fix the cheap security defects | Precondition for any exposure change, and for an honest node posture | Traversal + filename fixes with tests | 2 h | | `M4-06` | Fix the cheap security defects | Precondition for any exposure change, and for an honest node posture | Traversal + filename fixes with tests | 2 h |
| `M4-07` | Resolve the speaker-count disagreement | 70/93/71 — the single most confusing fact about this machine's data | One authoritative count per project, with the others derived or explicitly deprecated | 2-3 h | | `M4-07` | Resolve the speaker-count disagreement | 70/93/71 — the single most confusing fact about this machine's data | One authoritative count per project, with the others derived or explicitly deprecated | 2-3 h |
| `M4-08` | Add a read-only `/api/node` health endpoint | Lets the controller scrape node health without an agent; LAN-only | Endpoint returns the node report; no secrets; LAN-only documented | 1-2 h | | `M4-08` | Add a read-only `/api/node` health endpoint | Lets the controller scrape node health without an agent; LAN-only | Endpoint returns the node report; no secrets; LAN-only documented | 1-2 h |
| `M4-09` | Controller Stage 0 — reachability | Prove the node is reachable and that auth actually refuses | **done 2026-10-03** — 200 / 401 / 400, exactly as the controller documented | 15 min |
| `M4-10` | Controller Stage 1 — send the node report as a heartbeat | Prove the node runs the code it claims and reports the schema it has | `scripts/alignment_report.py` + `umbra_node_report.py` produce the payloads; **blocked on the controller issuing a `compute:read` + `compute:heartbeat` credential** | 30 min once credentialed |
| `M4-11` | Alignment report tool (controller §6.6) | Agreement must be mechanical, not prose | **done 2026-10-03** — `scripts/alignment_report.py --phase all`, per-phase evidence shapes, `blocked` never `pass` without evidence | 1 h |
### 9.2 NEXT — needs a decision or another machine ### 9.2 NEXT — needs a decision or another machine
@@ -604,8 +607,66 @@ nothing else, do those two.
--- ---
## 10. Change log ## 10. Decision record — `utterance_index` (controller §5.7)
The controller asked this node to record the decision rather than make it for us.
> **`utterance_index` is KEPT as a display ordinal, and is made non-load-bearing.**
- **Kept** because a reviewer needs "utt 231" to find a line, and because deleting or
renumbering it now breaks nothing that matters: `source_unit_key` is the identity, the
column is a label.
- **Non-load-bearing** means: no artifact name, no lineage record, no cross-reference and
no cross-language link may use it. Audio filenames still do (`utt_{index:04d}.wav`) as a
*local filesystem convenience*; the marker now also carries `source_unit_key`, so the
audio record can name its source unit without the filename being load-bearing.
- Recorded by: the UmbraAudio node, 2026-10-03. Status: node decision, open to reversal.
Proof that the decision was actually implemented, not just written: renumbering a chapter by
+100 changed **0** of its keys, and 200/200 probe units resolved with `used_index: false`.
---
## 11. Gaps found in the controller's alignment proposal
The controller asked to be told about a fifth gap rather than have it worked around. Six,
in the order they matter.
| ID | Severity | Gap | What this node did |
|---|---|---|---|
| **G-A** | major | *"Every blob gets its hash at generation"* — but audio is **post-processed after generation** (`apply_effects_to_file`, distortion effects, scene-break silence). A hash taken at generation would identify bytes that never shipped, and a changed trait list would silently change the artifact with no recorded cause. | **Verified the existing order is correct** (`write_marker` runs *after* effects), and closed the remaining hole: the marker now records `effects: [...]` alongside the hash, so the chain from raw TTS output to delivered bytes is explicit. |
| **G-B** | major | `source_unit_key` is anchored to **`chapters.source_text`** — a *second stored copy* of the copyrighted manuscript. Any change to how chapters are stored (unicode normalisation, trimming, re-extraction) silently invalidates **every** key in that chapter. | Documented as a deliberate, load-bearing coupling: content identity needs a content anchor. Mitigation is the anchor's own hash, now reported per project. Recommend the controller treat the chapter-text hash as the **source revision identity** rather than assuming a separate revision id. |
| **G-C** | major | The controller's key spec includes `source_revision_id`. This node has **no revision concept**, so it substitutes a chapter-text hash. Two sides will therefore compute **different keys for the same unit**. | Answered rather than worked around: `source_unit_key` is the audio-side key and the controller must store it verbatim as `source_units.source_unit_key` rather than recomputing it. Recomputation would be impossible anyway — the controller cannot know our occurrence rule. |
| **G-D** | minor | The marker did not record which source unit produced a clip, so an audio artifact could not be traced back to a unit revision. | Marker now records `source_unit_key`. |
| **G-E** | major | Voice assets are keyed by **name-derived slugs** (`officer_ref.wav`). Two projects can hold the same slug for different voices, so a slug is not a portable identity. | The node report now emits each ref WAV's full `sha256` + `object_key`. `voice_assets` must key on the bytes and carry the project scope; the slug stays a local label. |
| **G-F** | minor | `source_unit_key` is **project-agnostic by content**: identical chapter text in two projects yields identical keys. That answers the controller's open §8.3 question — and it is a tenancy consideration, not only a design one. | Deliberately left unsalted and documented. If per-tenant isolation is required, the controller salts at ingest (`sha256(tenant || key)`); doing it here would have made keys non-portable for no security benefit. |
---
## 12. Alignment status as of 2026-10-03
| Phase | Verdict | Note |
|---|---|---|
| 0 (reachability) | **pass** | `health` 200, `heartbeat` 401, missing-`Host` 400 — all three exactly as documented |
| 1 (vocabulary) | **pass** | 19 sets, 126 values, published to `definitions/umbraaudio-defs.json` with per-value provenance |
| 2 (cardinality) | **fail** | Evidence present; the three-way store disagreement is real and unresolved — a pass here would be a check that passed by doing nothing |
| 3 (identity) | **pass** | 200/200 units resolved by content key, **0** used `utterance_index` |
| 4 (lineage) | blocked | needs the controller's chosen artifact |
| 5 (timeline) | blocked | no time-scoped attribute model exists here yet |
| 6 (voice matrix) | blocked | voices are not captured as structured matrix values |
| 6b (persistence/partial capture) | blocked | **persistence is not captured at all**; absence is NULL, not typed |
| 6c (editions) | blocked | no edition concept |
| 7 (round trip) | blocked | needs the ingest API, which does not exist |
Regenerate any row with `scripts/alignment_report.py --phase all`.
---
## 13. Change log
| Date | Change | | Date | Change |
|---|---| |---|---|
| 2026-10-01 | Initial draft. Created by the UmbraAudio node. Added `infra/umbra-node.json`, `scripts/umbra_node_report.py`. Status: awaiting ratification. | | 2026-10-01 | Initial draft. Created by the UmbraAudio node. Added `infra/umbra-node.json`, `scripts/umbra_node_report.py`. Status: awaiting ratification. |
| 2026-10-02 | Controller (oldbox) alignment proposal received; D1-D16 recorded, 2 questions open. This document is the node's response. |
| 2026-10-03 | Controller alignment proposal actioned. Implemented `source_unit_key` (§10 decision, §5.1), blob hashes + effect chain (§5.2), published vocabularies (§5.4), fingerprint discipline (§5.5), and the alignment report tool (§6.6). Recorded six gaps (§11) and phase-by-phase status (§12). Stage 0 verified. |
| 2026-10-03 | Controller alignment proposal received and actioned. Implemented `source_unit_key` (§10 decision, §5.1), blob hashes + effect chain (§5.2), published vocabularies (§5.4), fingerprint discipline (§5.5), and the alignment report tool (§6.6). Recorded six gaps (§11) and the phase-by-phase status (§12). |

50
tests/test_generation/test_staleness.py
View File

@@ -176,3 +176,53 @@ def test_fx_marked_audio_stale_for_plain_calls(tmp_path: Path) -> None:
wav.write_bytes(b"RIFF....") wav.write_bytes(b"RIFF....")
write_marker("Same words.", wav, speaker="Narrator", fx_version=1) write_marker("Same words.", wav, speaker="Narrator", fx_version=1)
assert is_current("Same words.", wav, speaker="Narrator") is False assert is_current("Same words.", wav, speaker="Narrator") is False
def test_sha256_is_recorded_at_generation(tmp_path: Path) -> None:
"""Controller §5.2: every blob carries its own content hash from birth."""
import hashlib
import json
wav = tmp_path / "utt_0030.wav"
wav.write_bytes(b"RIFF-final-bytes")
write_marker("Same words.", wav, speaker="Hallr")
payload = json.loads(marker_path(wav).read_text())
assert payload["sha256"] == hashlib.sha256(b"RIFF-final-bytes").hexdigest()
def test_sha256_is_the_post_processed_bytes_not_the_raw_output(tmp_path: Path) -> None:
"""The hash must describe the delivered artifact, after effects are applied."""
import hashlib
import json
wav = tmp_path / "utt_0031.wav"
raw = b"RIFF-raw-tts"
wav.write_bytes(raw)
# Post-processing rewrites the file, exactly as apply_effects_to_file does.
wav.write_bytes(raw + b"-EFFECTED")
write_marker("Same words.", wav, speaker="Xylia", effects=["raspy"])
payload = json.loads(marker_path(wav).read_text())
assert payload["sha256"] == hashlib.sha256(raw + b"-EFFECTED").hexdigest()
assert payload["effects"] == ["raspy"]
def test_source_unit_key_is_recorded_so_audio_names_its_source_unit(tmp_path: Path) -> None:
import json
wav = tmp_path / "utt_0032.wav"
wav.write_bytes(b"RIFF")
write_marker("Same words.", wav, speaker="Hallr", source_unit_key="abc123")
payload = json.loads(marker_path(wav).read_text())
assert payload["source_unit_key"] == "abc123"
def test_sha256_verification_is_opt_in_and_detects_tampering(tmp_path: Path) -> None:
wav = tmp_path / "utt_0033.wav"
wav.write_bytes(b"RIFF-original")
write_marker("Same words.", wav, speaker="Narrator")
# Not verified by default: hashing every file on every validation pass is too slow.
assert is_current("Same words.", wav, speaker="Narrator") is True
assert is_current("Same words.", wav, speaker="Narrator", verify_sha256=True) is True
wav.write_bytes(b"RIFF-tampered")
assert is_current("Same words.", wav, speaker="Narrator") is True
assert is_current("Same words.", wav, speaker="Narrator", verify_sha256=True) is False

247
tests/test_source_unit_keys.py Normal file
View File

@@ -0,0 +1,247 @@
"""Tests for content-addressed source-unit identity (``source_unit_key``).
The contract under test is the one the controller depends on (spec 21 §9.1 M4-04):
a source unit must be resolvable **without** ``utterance_index``.
"""
from __future__ import annotations
import hashlib
import sqlite3
import pytest
from umbraaudio.source_units import (
COLUMN,
KEY_VERSION,
assign_keys_for_chapter,
backfill,
chapter_keys,
chapter_text_sha256,
compute_source_unit_key,
ensure_chapter_keys,
ensure_column,
key_report,
stamp_chapter,
text_sha256,
)
def _db(chapters: dict[int, str], utterances: dict[int, list[str]]) -> sqlite3.Connection:
"""Build a minimal chapters/utterances database."""
con = sqlite3.connect(":memory:")
con.executescript(
"CREATE TABLE chapters (id INTEGER PRIMARY KEY, source_text TEXT, "
"chapter_index INTEGER);"
"CREATE TABLE utterances (id INTEGER PRIMARY KEY, chapter_id INTEGER, "
"utterance_index INTEGER, text TEXT, type TEXT DEFAULT 'narration');"
)
for cid, text in chapters.items():
con.execute(
"INSERT INTO chapters (id, source_text, chapter_index) VALUES (?, ?, ?)",
(cid, text, cid),
)
utt_id = 0
for cid, texts in utterances.items():
for idx, text in enumerate(texts):
utt_id += 1
con.execute(
"INSERT INTO utterances (id, chapter_id, utterance_index, text) VALUES (?,?,?,?)",
(utt_id, cid, idx, text),
)
con.commit()
return con
class TestKeyDerivation:
def test_deterministic(self) -> None:
a = compute_source_unit_key("abc", "Hello.", 0)
b = compute_source_unit_key("abc", "Hello.", 0)
assert a == b
def test_key_is_hex_sha256(self) -> None:
key = compute_source_unit_key("abc", "Hello.", 0)
assert len(key) == 64
assert all(c in "0123456789abcdef" for c in key)
def test_occurrence_changes_key(self) -> None:
assert compute_source_unit_key("abc", "Yes.", 0) != compute_source_unit_key(
"abc", "Yes.", 1
)
def test_chapter_hash_changes_key(self) -> None:
assert compute_source_unit_key("a" * 64, "Hi.", 0) != compute_source_unit_key(
"b" * 64, "Hi.", 0
)
def test_key_version_is_part_of_the_payload(self) -> None:
"""A version bump must change every key, so old keys stay distinguishable."""
expected = hashlib.sha256(
b"\x00".join(
(
KEY_VERSION.encode(),
("c" * 64).encode(),
b"0",
text_sha256("Hi.").encode(),
)
)
).hexdigest()
assert compute_source_unit_key("c" * 64, "Hi.", 0) == expected
def test_chapter_and_text_hashes_are_utf8_stable(self) -> None:
assert chapter_text_sha256("café") == chapter_text_sha256("café")
# A chapter with no captured source text must still hash deterministically,
# and it must be the SAME as an empty string - not random.
assert chapter_text_sha256(None) == chapter_text_sha256("")
assert text_sha256(None) == text_sha256("")
class TestRepeatedText:
def test_identical_lines_get_distinct_keys(self) -> None:
keys = assign_keys_for_chapter("s" * 64, ["Yes.", "No.", "Yes.", "Yes."])
assert len(set(keys)) == 4
def test_occurrence_is_document_order(self) -> None:
first = compute_source_unit_key("s" * 64, "Yes.", 0)
second = compute_source_unit_key("s" * 64, "Yes.", 1)
keys = assign_keys_for_chapter("s" * 64, ["Yes.", "Yes."])
assert keys == [first, second]
def test_different_chapters_get_different_keys(self) -> None:
a = assign_keys_for_chapter("a" * 64, ["Same line."])
b = assign_keys_for_chapter("b" * 64, ["Same line."])
assert a != b
class TestIndexIndependence:
def test_renumbering_does_not_change_keys(self) -> None:
"""The load-bearing property: `utterance_index` must not be load-bearing."""
chapter = "The dwarf woke. The dwarf rose."
texts = ["The dwarf woke.", "The dwarf rose."]
con = _db({1: chapter}, {1: texts})
assert ensure_column(con)
stamp_chapter(con, 1)
before = dict(con.execute(f"SELECT id, {COLUMN} FROM utterances").fetchall())
# Re-segmentation: a new utterance is inserted at position 0 and every
# index shifts. The original two units keep their identity.
con.execute(
"INSERT INTO utterances (id, chapter_id, utterance_index, text) VALUES (99,1,0,?)",
("Inserted first.",),
)
con.execute("UPDATE utterances SET utterance_index = utterance_index + 1 WHERE id != 99")
con.execute("UPDATE utterances SET utterance_index = 0 WHERE id = 99")
con.commit()
stamp_chapter(con, 1)
after = dict(con.execute(f"SELECT id, {COLUMN} FROM utterances").fetchall())
for utt_id, key in before.items():
assert after[utt_id] == key
def test_chapter_text_edit_invalidates_that_chapters_keys(self) -> None:
con = _db({1: "Original text."}, {1: ["A line."]})
stamp_chapter(con, 1)
first = con.execute(f"SELECT {COLUMN} FROM utterances").fetchone()[0]
con.execute("UPDATE chapters SET source_text = 'Edited text.' WHERE id = 1")
stamp_chapter(con, 1)
second = con.execute(f"SELECT {COLUMN} FROM utterances").fetchone()[0]
assert first != second
def test_chapter_without_source_text_still_hashes(self) -> None:
con = _db({1: None}, {1: ["A line."]})
stamp_chapter(con, 1)
assert con.execute(f"SELECT {COLUMN} FROM utterances").fetchone()[0]
def test_chapter_keys_do_not_need_the_index(self) -> None:
"""Resolution is by key, exactly as controller validation phase 3 requires."""
con = _db({1: "Chapter body."}, {1: ["One.", "Two."]})
ensure_column(con)
stamp_chapter(con, 1)
keys = [k for _, k in chapter_keys(con, 1)]
resolved = con.execute(
f"SELECT text FROM utterances WHERE {COLUMN} = ?", (keys[1],)
).fetchone()
assert resolved[0] == "Two."
class TestIdempotence:
def test_stamp_twice_changes_nothing_the_second_time(self) -> None:
con = _db({1: "Body."}, {1: ["A.", "B."]})
ensure_column(con)
stamp_chapter(con, 1)
assert stamp_chapter(con, 1) == 0
def test_ensure_only_fills_missing(self) -> None:
con = _db({1: "Body."}, {1: ["A.", "B."]})
ensure_column(con)
assert ensure_chapter_keys(con, 1) == 2
assert ensure_chapter_keys(con, 1) == 0
def test_column_add_is_idempotent(self) -> None:
con = _db({1: "Body."}, {1: ["A."]})
assert ensure_column(con) is True
assert ensure_column(con) is False
def test_backfill_reports_coverage(self) -> None:
con = _db({1: "Body.", 2: "Body two."}, {1: ["A."], 2: ["B."]})
result = backfill(con)
assert result["chapters"] == 2
assert result["utterances_total"] == 2
assert result["utterances_with_key"] == 2
assert result["utterances_without_key"] == 0
def test_backfill_can_target_chapters(self) -> None:
con = _db({1: "A.", 2: "B."}, {1: ["x"], 2: ["y"]})
result = backfill(con, chapter_ids=[1])
assert result["utterances_with_key"] == 1
class TestKeyReport:
def test_absent_column_reports_false(self) -> None:
con = _db({1: "Body."}, {1: ["A."]})
assert key_report(con) == {"column_present": False}
def test_collisions_are_counted(self) -> None:
"""Collisions must be visible, not silently accepted."""
con = _db({1: "Body."}, {1: ["A.", "B."]})
ensure_column(con)
stamp_chapter(con, 1)
# Force a duplicate to prove the report detects it rather than hiding it.
con.execute(
"UPDATE utterances SET "
f"{COLUMN} = (SELECT {COLUMN} FROM utterances ORDER BY id LIMIT 1)"
)
con.commit()
report = key_report(con)
assert report["column_present"] is True
assert report["utterances_total"] == 2
assert report["distinct_keys"] == 1
assert report["collisions"] == 1
def test_no_index_column_is_written_by_stamp() -> None:
"""Guard: stamping must not touch utterance_index or any other column."""
con = _db({1: "Body."}, {1: ["A.", "B."]})
ensure_column(con)
con.execute(f"UPDATE utterances SET {COLUMN} = 'stale'")
con.commit()
cols = "SELECT id, utterance_index, text, type FROM utterances ORDER BY id"
before = con.execute(cols).fetchall()
stamp_chapter(con, 1)
after = con.execute(cols).fetchall()
assert before == after
@pytest.mark.parametrize("path_exists", [True, False])
def test_file_sha256_handles_missing_files(tmp_path, path_exists: bool) -> None:
from umbraaudio.generation.staleness import file_sha256
target = tmp_path / "x.wav"
if path_exists:
target.write_bytes(b"abc")
assert file_sha256(target) == hashlib.sha256(b"abc").hexdigest()
else:
assert file_sha256(target) is None

8
umbraaudio/database.py
View File

@@ -101,7 +101,8 @@ CREATE TABLE IF NOT EXISTS utterances (
review_reason TEXT, review_reason TEXT,
emotion_confidence REAL DEFAULT 1.0, emotion_confidence REAL DEFAULT 1.0,
user_edited INTEGER NOT NULL DEFAULT 0, user_edited INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'pending' status TEXT NOT NULL DEFAULT 'pending',
source_unit_key TEXT
); );
CREATE TABLE IF NOT EXISTS generated_audio ( CREATE TABLE IF NOT EXISTS generated_audio (
@@ -159,6 +160,7 @@ CREATE INDEX IF NOT EXISTS idx_name_tokens_status ON name_tokens(status);
CREATE INDEX IF NOT EXISTS idx_name_tokens_project ON name_tokens(project_id); CREATE INDEX IF NOT EXISTS idx_name_tokens_project ON name_tokens(project_id);
CREATE INDEX IF NOT EXISTS idx_utterances_chapter ON utterances(chapter_id); CREATE INDEX IF NOT EXISTS idx_utterances_chapter ON utterances(chapter_id);
CREATE INDEX IF NOT EXISTS idx_utterances_source_unit ON utterances(source_unit_key);
CREATE INDEX IF NOT EXISTS idx_utterances_character ON utterances(character_id); CREATE INDEX IF NOT EXISTS idx_utterances_character ON utterances(character_id);
CREATE INDEX IF NOT EXISTS idx_generated_audio_utterance ON generated_audio(utterance_id); CREATE INDEX IF NOT EXISTS idx_generated_audio_utterance ON generated_audio(utterance_id);
CREATE INDEX IF NOT EXISTS idx_chapters_project ON chapters(project_id); CREATE INDEX IF NOT EXISTS idx_chapters_project ON chapters(project_id);
@@ -217,11 +219,15 @@ class Database:
("attribution_uncertain", "INTEGER NOT NULL DEFAULT 0"), ("attribution_uncertain", "INTEGER NOT NULL DEFAULT 0"),
("needs_review", "INTEGER NOT NULL DEFAULT 0"), ("needs_review", "INTEGER NOT NULL DEFAULT 0"),
("review_reason", "TEXT"), ("review_reason", "TEXT"),
("source_unit_key", "TEXT"),
): ):
if col not in utt_cols: if col not in utt_cols:
await self._conn.execute( await self._conn.execute(
f"ALTER TABLE utterances ADD COLUMN {col} {typedef}" f"ALTER TABLE utterances ADD COLUMN {col} {typedef}"
) )
await self._conn.execute(
"CREATE INDEX IF NOT EXISTS idx_utterances_source_unit ON utterances(source_unit_key)"
)
cursor = await self._conn.execute( cursor = await self._conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' AND name='voice_review_history'" "SELECT name FROM sqlite_master WHERE type='table' AND name='voice_review_history'"

211
umbraaudio/definitions.py Normal file
View File

@@ -0,0 +1,211 @@
"""Published vocabulary sets for the Umbra system (controller §5.4).
The controller's rule, from ``manifest.md`` §14.2, is that anything a stored row depends on
must be a *versioned vocabulary in data*, not a literal in code. Otherwise every product
reimplements it and they drift silently. This module is the audio side of that.
Each set records, per value, where the value came from:
* ``declared`` - stated by a spec or by the controller. This node does not get to change it.
* ``extracted`` - read out of executable code in this repository, and therefore changeable
without a deployment.
That distinction is the point of publishing them: a diff against the controller shows not
only that two sides disagree, but whether a disagreement is one of wording or one of meaning.
Nothing here imports the database or the filesystem. It is pure data, so it is safe to read
from any product and cheap to diff.
"""
from __future__ import annotations
from typing import Any, Literal
Origin = Literal["declared", "extracted"]
DEFINITION_SET_VERSION = "defs-2026-10-03"
# Where each vocabulary's authority lives, for the reader of the diff.
AUTHORITY = {
"utterance_type": "specs/16 §5a, umbraaudio/database.py SCHEMA default 'narration'",
"character_persistence": "controller D11 (specs/21 §9)",
"character_kind": "specs/18 §5.4a",
"voice_capture_mode": "umbraaudio/generation/chapter_builder.py, controller §2.2",
"phonetic_provenance": "manifest.md §14.2",
"attribution_evidence_kind": "umbraaudio/analysis/clarity.py",
"artifact_kind": "manifest.md §11, specs/20 §F.19",
"artifact_status": "controller §2.1 (artifacts.status)",
"project_lifecycle": "manifest.md §1.4, controller §2.1 (projects.status)",
"migration_state": "manifest.md §14.1",
"job_capability": "specs/21 §4.7 + controller §2.3",
"attribute_absence_reason": "controller D13",
"voice_variant_reason": "controller D6",
"name_token_status": "specs/17 §3.4",
"name_token_kind": "specs/17 §3.4",
"post_tts_effect_trait": "umbraaudio/generation/audio_effects.py TRAIT_EFFECTS",
"emotion": "umbraaudio/generation/chapter.py EMOTION_OPTIONS",
"pace": "umbraaudio/generation/chapter.py PACE_OPTIONS",
"qa_issue_type": "umbraaudio/analysis/attribution_qa.py",
}
def _d(*values: str) -> dict[str, Origin]:
return dict.fromkeys(values, "declared")
def _e(*values: str) -> dict[str, Origin]:
return dict.fromkeys(values, "extracted")
VOCABULARIES: dict[str, dict[str, Origin]] = {
# --- unit and unit-adjacent ------------------------------------------------
"utterance_type": _d("narration", "dialogue", "scene_break"),
# controller D11: persistence is DECLARED, never inferred from name overlap
"character_persistence": _d("book_local", "series_persistent"),
# specs/18 §5.4a. Note "speaker" is deliberately absent: it is a derived fact
# (a character with lines in a work), never a kind. Controller §1.2.
"character_kind": _d(
"named_character",
"recurring_role",
"scene_role_instance",
"group",
"ai",
"narrator",
),
# How a voice's identity was captured, per manifest.md §5.4
"voice_capture_mode": _e("embedding", "icl"),
# manifest.md §14.2 - IPA derivation is Latin-only, so provenance is load-bearing
"phonetic_provenance": _d("explicit", "ipa_derived"),
"attribution_evidence_kind": _e("name", "role", "pronoun", "address_only", "none"),
# --- artifacts -------------------------------------------------------------
"artifact_kind": _d(
"source_document",
"character_profile",
"voice_reference",
"voice_prompt_vmatrix",
"utterance_wav",
"chapter_stitch",
"m4b",
"mp3_track",
"ar_package",
),
"artifact_status": _d("intermediate", "final", "superseded"),
# --- projects and migration ------------------------------------------------
"project_lifecycle": _d(
"active", "on_hold", "released", "archived", "experimental", "parsed", "analyzed"
),
"migration_state": _d("legacy", "mirroring", "dual_write", "authoritative", "retired"),
# --- work routing ----------------------------------------------------------
"job_capability": _d(
"audio.parse_manuscript",
"audio.segment_utterances",
"analysis.character_discovery",
"analysis.speaker_attribution",
"analysis.speaker_clarity",
"analysis.name_token_scan",
"character.personality_analysis",
"character.consistency_check",
"voice.capture_character",
"voice.reference_generation",
"voice.attribute_measurement",
"audio.render_utterances",
"audio.stitch_chapters",
"audio.normalize_loudness",
"export.m4b",
"export.mp3_package",
"review.speaker_QA",
),
# controller D13: absence is typed, never NULL
"attribute_absence_reason": _d(
"not_attempted", "not_applicable", "in_progress", "superseded"
),
# controller D6: a variant with no reason is indistinguishable from a bug
"voice_variant_reason": _d(
"sarcasm_mimic", "disguise_roleplay", "technical_degradation"
),
# --- review ----------------------------------------------------------------
"name_token_status": _e("unresolved", "assigned", "created", "dismissed"),
"name_token_kind": _e("name_tag", "speaker_name"),
"qa_issue_type": _e(
"speech_tag_mismatch",
"narrator_dialogue",
"role_tag_guessed",
"address_mismatch",
"narration_speaker_mismatch",
"long_monologue",
),
# --- delivery --------------------------------------------------------------
"emotion": _e(
"neutral",
"calm",
"serious",
"sad",
"happy",
"angry",
"fearful",
"excited",
"gentle",
"firm",
"hesitant",
"sarcastic",
"tender",
"urgent",
"dramatic",
"whisper",
"romantic",
"commanding",
"melancholy",
"hopeful",
"desperate",
"surprised",
"confused",
"disgusted",
"determined",
"tired",
"confident",
),
"pace": _e("slow", "moderate", "fast"),
"post_tts_effect_trait": _e(
"raspy",
"gravelly",
"hoarse",
"breathy",
"whispery",
"nasal",
"deep",
"boom",
"reverb",
"echo",
"radio_filter",
"distortion",
"pitch_shift",
"noise_inject",
"tremolo",
"chorusing",
),
}
def vocabularies() -> dict[str, list[str]]:
"""The plain ``{set_name: [values]}`` shape the controller diffs (phase 1)."""
return {name: sorted(values) for name, values in VOCABULARIES.items()}
def provenance() -> dict[str, dict[str, str]]:
"""Per-value origin plus the authority each vocabulary is accountable to."""
out: dict[str, dict[str, str]] = {}
for name, values in VOCABULARIES.items():
entry = {v: o for v, o in values.items()}
entry["_authority"] = AUTHORITY.get(name, "undeclared")
out[name] = entry
return out
def declaration() -> dict[str, Any]:
"""The full published definition set, versioned."""
return {
"definition_set_version": DEFINITION_SET_VERSION,
"product": "umbraaudio",
"vocabularies": vocabularies(),
"provenance": provenance(),
}

21
umbraaudio/generation/chapter_builder.py
View File

@@ -362,6 +362,7 @@ class ChapterBuilder:
load_proper_names, load_proper_names,
prepare_tts_text, prepare_tts_text,
) )
from umbraaudio.source_units import ensure_chapter_keys
phonetic = load_phonetic_names(self.data_dir) phonetic = load_phonetic_names(self.data_dir)
proper_names = load_proper_names(self.data_dir) proper_names = load_proper_names(self.data_dir)
@@ -371,12 +372,22 @@ class ChapterBuilder:
db = sqlite3.connect(str(self.data_dir / "project.db")) db = sqlite3.connect(str(self.data_dir / "project.db"))
db.row_factory = sqlite3.Row db.row_factory = sqlite3.Row
# Content-addressed source-unit identity, recorded at generation time per
# specs/21 §9.1 M4-04. Must run BEFORE the read so the keys are available to the
# rows below, and BEFORE the marker is written so the audio record can name the
# exact source unit it came from. Fills only missing keys (cheap, idempotent).
stamped = ensure_chapter_keys(db, self.chapter_id)
utterances = db.execute( utterances = db.execute(
"SELECT id, utterance_index, text, speaker_name, emotion, pace, instruct, type " "SELECT id, utterance_index, text, speaker_name, emotion, pace, instruct, type, "
"source_unit_key "
"FROM utterances WHERE chapter_id = ? ORDER BY utterance_index", "FROM utterances WHERE chapter_id = ? ORDER BY utterance_index",
(self.chapter_id,), (self.chapter_id,),
).fetchall() ).fetchall()
db.close() db.close()
if stamped:
self._set_phase("validating", f"Recorded {stamped} source-unit keys", 8)
if not utterances: if not utterances:
self._set_phase("error", "No utterances found", 0) self._set_phase("error", "No utterances found", 0)
@@ -590,17 +601,25 @@ class ChapterBuilder:
if out.exists(): if out.exists():
explicit = distortion_effects.get(speaker, []) explicit = distortion_effects.get(speaker, [])
applied: list[str] = []
if explicit: if explicit:
apply_effects_to_file_by_name(out, explicit) apply_effects_to_file_by_name(out, explicit)
applied = list(explicit)
else: else:
traits = speaker_traits.get(speaker, []) traits = speaker_traits.get(speaker, [])
if traits: if traits:
apply_effects_to_file(out, traits) apply_effects_to_file(out, traits)
applied = list(traits)
# Marker written AFTER post-processing, so the recorded
# sha256 is the final bytes, and `effects` records what
# turned the raw TTS output into them.
write_marker( write_marker(
spoken_text, out, spoken_text, out,
speaker=speaker, voice_ref=voice_ref, speaker=speaker, voice_ref=voice_ref,
voice_mode=voice_mode, instruct=tts_instruct, voice_mode=voice_mode, instruct=tts_instruct,
source_unit_key=u.get("source_unit_key"),
effects=applied,
) )
self.result.generated += 1 self.result.generated += 1
success = True success = True

45
umbraaudio/generation/staleness.py
View File

@@ -6,12 +6,18 @@ check silently reuses audio that no longer matches the utterance text. Each
wav gets a sidecar marker (utt_XXXX.wav.txt) recording the exact spoken text wav gets a sidecar marker (utt_XXXX.wav.txt) recording the exact spoken text
AND the assigned speaker; a file is only considered current when both match. AND the assigned speaker; a file is only considered current when both match.
Markers are JSON ``{"speaker": ..., "text": ..., "voice_fp": ..., "instruct": ...}``. Markers are JSON ``{"speaker": ..., "text": ..., "voice_fp": ..., "instruct": ...,
Legacy plain-text markers carry no speaker information, so they are always "sha256": ...}``. Legacy plain-text markers carry no speaker information, so they are
treated as stale. The ``voice_fp`` field is a truncated SHA-256 of the always treated as stale. The ``voice_fp`` field is a truncated SHA-256 of the
voice-reference wav used during generation; when a speaker's ref changes voice-reference wav used during generation; when a speaker's ref changes
(e.g. gender re-assignment) the fingerprint no longer matches and the file (e.g. gender re-assignment) the fingerprint no longer matches and the file
is considered stale, forcing regeneration. is considered stale, forcing regeneration.
``sha256`` is the FULL content hash of the rendered wav, recorded at generation
(specs/21 §9.1 M4-04 / controller §5.2: "every blob gets its hash at generation, not at
export"). It is recorded but **not** verified by default: hashing ~10k files on every
validation pass would re-read gigabytes. Pass ``verify_sha256=True`` when an integrity
check is actually wanted.
""" """
from __future__ import annotations from __future__ import annotations
@@ -21,6 +27,7 @@ import json
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
if TYPE_CHECKING: if TYPE_CHECKING:
from collections.abc import Sequence
from pathlib import Path from pathlib import Path
@@ -35,6 +42,14 @@ def _voice_fingerprint(voice_ref: Path) -> str:
return h[:16] return h[:16]
def file_sha256(path: Path) -> str | None:
"""Full SHA-256 of a file, or None if it cannot be read."""
try:
return hashlib.sha256(path.read_bytes()).hexdigest()
except OSError:
return None
def is_current( def is_current(
text: str, text: str,
wav_path: Path, wav_path: Path,
@@ -44,6 +59,7 @@ def is_current(
accepted_voice_modes: set[str] | None = None, accepted_voice_modes: set[str] | None = None,
instruct: str | None = None, instruct: str | None = None,
fx_version: int | None = None, fx_version: int | None = None,
verify_sha256: bool = False,
) -> bool: ) -> bool:
"""True when the wav exists, its marker is JSON, and text (+ speaker if """True when the wav exists, its marker is JSON, and text (+ speaker if
given) match exactly. If *voice_ref* is provided the marker's given) match exactly. If *voice_ref* is provided the marker's
@@ -95,6 +111,10 @@ def is_current(
return False return False
if fx_version is not None and payload.get("fx_version") != fx_version: if fx_version is not None and payload.get("fx_version") != fx_version:
return False return False
if verify_sha256:
recorded = payload.get("sha256")
if not recorded or recorded != file_sha256(wav_path):
return False
return not (fx_version is None and payload.get("fx_version") is not None) return not (fx_version is None and payload.get("fx_version") is not None)
@@ -106,14 +126,24 @@ def write_marker(
voice_mode: str | None = None, voice_mode: str | None = None,
instruct: str | None = None, instruct: str | None = None,
fx_version: int | None = None, fx_version: int | None = None,
source_unit_key: str | None = None,
effects: Sequence[str] | None = None,
) -> None: ) -> None:
"""Record *text*, *speaker*, and (optionally) the voice-reference """Record *text*, *speaker*, and (optionally) the voice-reference
fingerprint as the spoken content of *wav_path*. When *instruct* is given fingerprint as the spoken content of *wav_path*. When *instruct* is given
(the exact TTS instruction used) it is recorded too, so a later run with a (the exact TTS instruction used) it is recorded too, so a later run with a
different emotion/instruct regenerates instead of reusing the file. different emotion/instruct regenerates instead of reusing the file.
*fx_version* records which post-processing emotion recipe (if any) was *fx_version* records which post-processing emotion recipe (if any) was
stamped onto the audio; a bump forces regeneration of fx'd files.""" stamped onto the audio; a bump forces regeneration of fx'd files.
The rendered wav's own full SHA-256 is always recorded when the file exists,
so every generated blob carries its content hash from the moment it is born
rather than at export time."""
payload: dict[str, str] = {"speaker": speaker or "", "text": text} payload: dict[str, str] = {"speaker": speaker or "", "text": text}
if wav_path.exists():
digest = file_sha256(wav_path)
if digest:
payload["sha256"] = digest
if voice_ref is not None and voice_ref.exists(): if voice_ref is not None and voice_ref.exists():
payload["voice_fp"] = _voice_fingerprint(voice_ref) payload["voice_fp"] = _voice_fingerprint(voice_ref)
if voice_mode is not None: if voice_mode is not None:
@@ -122,6 +152,13 @@ def write_marker(
payload["instruct"] = instruct payload["instruct"] = instruct
if fx_version is not None: if fx_version is not None:
payload["fx_version"] = fx_version payload["fx_version"] = fx_version
if source_unit_key is not None:
payload["source_unit_key"] = source_unit_key
if effects:
# The recorded sha256 identifies the FINAL bytes. Without this, nothing records
# which post-processing turned the raw TTS output into those bytes, so a changed
# trait list would change the hash with no explanation (spec 21 §12 gap G-A).
payload["effects"] = list(effects)
marker_path(wav_path).write_text( marker_path(wav_path).write_text(
json.dumps(payload, ensure_ascii=False), encoding="utf-8" json.dumps(payload, ensure_ascii=False), encoding="utf-8"
) )

204
umbraaudio/source_units.py Normal file
View File

@@ -0,0 +1,204 @@
"""Content-addressed source-unit identity (``source_unit_key``).
Why this exists
---------------
Utterances are currently identified by ``utterance_index``, which is *position*.
Re-segmentation renumbers a chapter, and ``DELETE /api/utterance/{id}`` renumbers the
rest of the chapter and renames files on disk. ``specs/18-UmbraTrans.md`` §3.3 forbids
mutable indices as durable cross-language IDs, and ``specs/20`` §E.1 requires a
content-addressed key recorded at generation time with filenames unchanged.
The key
-------
``uk1`` version, byte order, hex-encoded SHA-256::
sha256(b"uk1\\x00" + chapter_text_sha256 + b"\\x00" + occurrence + b"\\x00" + text_sha256)
* ``chapter_text_sha256`` - SHA-256 of ``chapters.source_text``. This substitutes for a
source-revision id: it is self-verifying, so no revision table is needed, and any edit to
the chapter invalidates exactly the units whose text changed.
* ``occurrence`` - 0-based count of *earlier* utterances in the same chapter with an
identical text hash, in document order. Disambiguates repeated lines ("Yes.") without
depending on the index itself.
* ``text_sha256`` - SHA-256 of the utterance text, exact.
Properties this deliberately has
---------------------------------
* **No index in the key.** Renumbering cannot change it.
* **Index-independent resolution.** Given a key, the row is found by primary key lookup;
``utterance_index`` is never needed (controller validation phase 3).
* **Per-project agnostic.** Two projects with the same chapter text and the same utterance
text produce the same key. That is intentional (content identity), and it means the
controller must salt per tenant at ingest if isolation requires it - flagged as a finding
rather than silently namespaced here.
* **Idempotent.** Recomputing over an unchanged chapter reproduces every key exactly.
This is additive. It adds one nullable column, changes no filename, and does not participate
in ``is_current`` staleness decisions.
"""
from __future__ import annotations
import hashlib
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
import sqlite3
from collections.abc import Iterable, Sequence
# Bump if the derivation ever changes; existing keys stay valid and readable.
KEY_VERSION = "uk1"
COLUMN = "source_unit_key"
def _sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def chapter_text_sha256(source_text: str | None) -> str:
"""Hash a chapter's source text. ``None``/empty still hashes (deterministically)."""
return _sha256((source_text or "").encode("utf-8"))
def text_sha256(text: str | None) -> str:
"""Hash an utterance's exact text."""
return _sha256((text or "").encode("utf-8"))
def compute_source_unit_key(chapter_sha: str, text: str, occurrence: int) -> str:
"""Derive one key. ``occurrence`` disambiguates repeated text within a chapter."""
payload = b"\x00".join(
(
KEY_VERSION.encode("ascii"),
chapter_sha.encode("ascii"),
str(occurrence).encode("ascii"),
text_sha256(text).encode("ascii"),
)
)
return _sha256(payload)
def assign_keys_for_chapter(chapter_sha: str, texts: Sequence[str]) -> list[str]:
"""Return the key for every utterance of one chapter, in document order."""
seen: dict[str, int] = {}
keys: list[str] = []
for text in texts:
t_sha = text_sha256(text)
occurrence = seen.get(t_sha, 0)
seen[t_sha] = occurrence + 1
keys.append(compute_source_unit_key(chapter_sha, text, occurrence))
return keys
def ensure_column(conn: sqlite3.Connection) -> bool:
"""Add ``source_unit_key`` if absent. Returns True when it added the column."""
cols = {row[1] for row in conn.execute("PRAGMA table_info(utterances)")}
if COLUMN in cols:
return False
conn.execute(f"ALTER TABLE utterances ADD COLUMN {COLUMN} TEXT")
conn.execute(f"CREATE INDEX IF NOT EXISTS idx_utterances_source_unit ON utterances({COLUMN})")
conn.commit()
return True
def _chapter_order_sql(conn: sqlite3.Connection) -> str:
"""Order chapters by display order, tolerating a missing ``chapter_index``.
Two of the four live projects carry an older schema shape (see
``manifest.md`` §1.3), so this must not assume the column exists.
"""
cols = {row[1] for row in conn.execute("PRAGMA table_info(chapters)")}
order = "chapter_index, id" if "chapter_index" in cols else "id"
return f"SELECT id FROM chapters ORDER BY {order}"
def chapter_keys(conn: sqlite3.Connection, chapter_id: int) -> list[tuple[int, str]]:
"""Compute ``(utterance_id, key)`` for one chapter without writing anything."""
chapter = conn.execute(
"SELECT source_text FROM chapters WHERE id = ?", (chapter_id,)
).fetchone()
chapter_sha = chapter_text_sha256(chapter[0] if chapter else None)
rows = conn.execute(
"SELECT id, text FROM utterances WHERE chapter_id = ? ORDER BY utterance_index",
(chapter_id,),
).fetchall()
keys = assign_keys_for_chapter(chapter_sha, [r[1] or "" for r in rows])
return [(row[0], key) for row, key in zip(rows, keys, strict=True)]
def stamp_chapter(conn: sqlite3.Connection, chapter_id: int) -> int:
"""Write keys for one chapter. Returns how many rows changed."""
ensure_column(conn)
pairs = chapter_keys(conn, chapter_id)
changed = 0
for utterance_id, key in pairs:
cursor = conn.execute(
f"UPDATE utterances SET {COLUMN} = ? "
f"WHERE id = ? AND ({COLUMN} IS NULL OR {COLUMN} != ?)",
(key, utterance_id, key),
)
changed += cursor.rowcount if cursor.rowcount and cursor.rowcount > 0 else 0
conn.commit()
return changed
def ensure_chapter_keys(conn: sqlite3.Connection, chapter_id: int) -> int:
"""Fill only the keys that are missing for one chapter. Cheap, idempotent, safe.
Called at generation time so every path that produces audio also produces identity,
including paths this file knows nothing about.
"""
ensure_column(conn)
missing = conn.execute(
f"SELECT COUNT(*) FROM utterances WHERE chapter_id = ? AND {COLUMN} IS NULL",
(chapter_id,),
).fetchone()[0]
if not missing:
return 0
return stamp_chapter(conn, chapter_id)
def backfill(
conn: sqlite3.Connection, chapter_ids: Iterable[int] | None = None
) -> dict[str, int]:
"""Backfill every chapter (or a subset). Returns counts for the report."""
if chapter_ids is None:
ids = [row[0] for row in conn.execute(_chapter_order_sql(conn))]
else:
ids = list(chapter_ids)
ensure_column(conn)
stamped = 0
for chapter_id in ids:
stamped += stamp_chapter(conn, chapter_id)
total, filled = conn.execute(
f"SELECT COUNT(*), SUM(CASE WHEN {COLUMN} IS NOT NULL THEN 1 ELSE 0 END) FROM utterances"
).fetchone()
return {
"chapters": len(ids),
"rows_stamped": stamped,
"utterances_total": total or 0,
"utterances_with_key": filled or 0,
"utterances_without_key": (total or 0) - (filled or 0),
}
def key_report(conn: sqlite3.Connection) -> dict[str, Any]:
"""Read-only coverage report - evidence for controller validation phase 2/3."""
has_column = COLUMN in {
row[1] for row in conn.execute("PRAGMA table_info(utterances)")
}
if not has_column:
return {"column_present": False}
total, filled, distinct = conn.execute(
f"SELECT COUNT(*), SUM(CASE WHEN {COLUMN} IS NOT NULL THEN 1 ELSE 0 END), "
f"COUNT(DISTINCT {COLUMN}) FROM utterances"
).fetchone()
return {
"column_present": True,
"key_version": KEY_VERSION,
"utterances_total": total or 0,
"utterances_with_key": filled or 0,
"distinct_keys": distinct or 0,
"collisions": (filled or 0) - (distinct or 0),
}