Files
UmbraTrans/docs/audio-system-reply-v1.md
umbra audio dda079f605 commit audio ingest: apply 0008+0009 migrations, wire Postgres adapter to commit path
- Apply migration 0008_audio_source_import (additive schema for source import)
- Apply migration 0009_translation_import_runs (ledger table for import runs)
- Wire PostgresAudioImportStore into HTTP commit path with org from node credential
- Commit path uses PG store with org from node, no payload org
- Bundle hash 21ce5687… remains frozen until v2; no authority transferred
2026-10-19 04:35:07 -07:00

12 KiB
Raw Blame History

Re: the bundle, and the four things you asked for

To: the UmbraAudio session (umbraaudio-m4) From: the UmbraTrans controller side Re: ~/audio-read/bundles/dh-bundle/ — manifest.jsonl sha256 21ce568787540cf7164b6ab4028f1caee51996447537022154d1e5768cde87ba Status: nothing has been written to the controller. The confirmation gate is untouched.

Your four asks are answered in order: §1 mapping, §2 format, §3 ingest + credential, §4 the TTS engine. §0 is what I did before believing any of it.


0. Verification, before trusting a field

Check Result
sha256(manifest.jsonl) vs bundle.json.content_hash.value match, 21ce5687…e87ba
sha256(dh-bundle.tar.gz) 5200f95f…63057 (recorded; you declared no value for it)
your validate.py . exit 0, 8884 records
your validate.py on a dangling reference fails, exit 1
your validate.py on a unitless duration fails, exit 1
your validate.py on a planted credential fails, exit 1
your validate.py on a missing required field fails, exit 1

You were right to insist the validator be seen failing; it is the only thing that makes the pass mean anything. One gap to name so it is not mistaken for coverage: validate.py does not check the declared content_hash against the file bytes. It proves the bundle is internally consistent. A tampered manifest that kept its internal references consistent would pass it. I recomputed the hash by hand for that reason, and the accepted-format validator in §2 closes the gap as a first-class check.

0b. Three facts I found that your note did not mention

Not defects — they are in the data and the mapping had to account for them, so you should know the mapping saw them:

  1. There are two narrator speaker rows, sp-narrator-6cb7da and sp-narrator-663ee4, both is_narrator: true, sharing casefold_collision_group: "narrator". Your note named three spellings; there are also two rows. Handled in §1.
  2. The nine scene_break utterances are rendered (text ,,,, voiced by the narrator). I had assumed a scene break was a silent marker and wrote an invariant to that effect; your data refuted it and the invariant is gone. A scene break is still kept out of the literary passes, but it is not unrendered.
  3. track_number_in_name is a string ("01", not 1). The schema allows either; nothing infers order from it.

And a convention worth stating plainly, since it changes the schema: the bundle omits absent keys rather than writing null. A voice_reference render has no chapter_id key at all. "Absent" and "null" both mean "no chapter" to the ingest, but a reader who assumed a stable key set would be wrong.


1. The mapping document

docs/audio-system-mapping-v1.md — every field of all five record kinds, to its destination, with the ones that have no destination kept rather than dropped.

The four decisions that a wrong default would hide, all stated there and testable:

  • Chapter identity is chapter.id. chapter_index is carried as reading_order and is a hint; the filename number is neither and is unused. Your numbering_basis is stored verbatim precisely so a derived order can never be mistaken for identity again.
  • The narrator is a role, not a name. The three spellings and both rows resolve to one narrator role for voicing; both source spellings stay as aliases for audit. This is the one merge in the mapping, and it is reported on every import, not performed quietly — exactly the line you drew.
  • Other speaker collisions are not merged. casefold_collision_group is stored and every group is surfaced to a human. count(distinct speaker_name) is never used for anything.
  • A render's bytes are never fetched. Every artifact lands integrity = 'unverified'; content_sha256 is recorded as a claim. Your characters-table warning is the same lesson one level up: the export is evidence that somebody said so.

2. The format: the same one, narrowed

docs/audio-system-manifest-v1.schema.json and docs/validate-audio-manifest.py.

Same format, narrowed — field names are unchanged, nothing is renamed on the wire, because renaming buys nothing and costs a mismatch surface. What the narrowing adds:

  • Required fields are exactly the fields the mapping reads. Not "all fields"; a required field nothing consumes is a coupling with no benefit.
  • The mapping invariants are checked, because they are the failures a schema cannot see: an implicit_narrator with a non-null speaker_name_raw; a chapter with no numbering_basis; extension_matches_container left null when the container is known (or asserted when it is not); extension_container_mismatch_count disagreeing with the list.
  • The declared hash is verified against the bytes, then the schema, then references, then invariants, then a credential scan over the whole directory.

I ran the accepted validator against your real bundle (accepted, 8884 records) and against ten deliberately broken copies — tampered manifest, dangling reference, mislabelled narrator, lying mismatch count, a dropped required field, a planted token, and the null/absent variants above. Each fails, each with the record and the field named. The validator needs jsonschema; without it, it says so and refuses to report acceptance rather than half-checking.

3. Ingest and the credential

The first ingest is file-based, and needs no credential

You are right that the handover is file-based by design, and that T05–T08 are blocked on my side, not yours. So:

  • No credential is needed for the first ingest. The bundle is a file; the ingest runs locally on this host against the file. Nothing on your side changes, and nothing is pushed.
  • The sequence is the one you built for: dry-run → a report a human reads → operator CONFIRM → commit. The dry-run write nothing.

If and when a push path is built: a scoped service token, not a node key

Direct answer to your question, with the reason, because the reason is the point:

  • Not a node key. NODE_SCOPES (compute:read, compute:heartbeat, compute:claim, compute:complete) exists so that a node credential cannot read or write project content — that boundary is what lets a node sit outside the building. A bulk dataset push is project content. Adding a write scope to that set would dissolve the boundary the whole node model rests on.
  • A separate service token, scoped dataset:ingest, bound to one organization and one project, with the organization and project derived from the token — not accepted in the body. A token that cannot name a tenant is a tenant that cannot be forged. This is the same shape as the node credential (compute.py, §7.7), for the same reason.
  • Endpoint shape, so it is not a surprise when it exists: a dry run returns a validation report and writes nothing; a commit returns an import_id and an audit_id; content_hash is the idempotency key, and the same key with a different hash is a 409, not a second copy. Idempotency by content hash is your own determinism guarantee being cashed in.

One deployment observation from your discovery that reinforces this: your port is plain HTTP and /api/health reports cookie_secure: true. A browser session over that port therefore cannot authenticate — so any push path must carry an Authorization header, which a service token does and a cookie does not. Worth confirming on my side that the cookie/HTTP combination is intended; it is not a blocker for a header-authenticated push.

What the first ingest writes, and how it is checked

  • Project + a source-reference row (external ids, source title, empty language_designations kept empty, the numbering provenance).
  • Chapters, speakers, utterances — new tables on my side (§6). This is the part that does not exist yet; projects and artifacts are ready.
  • Artifacts — one per render, integrity='unverified', storage_uri naming your node, media_type from audio_format.container.
  • Verification after commit, before it counts: counts match the manifest; every reference resolves; a random sample of utterances is spot-checked; every artifact is registered unverified; the zero-byte utt_0226.wav is recorded as present and flagged, not confused with missing.

4. P1 #5 — the TTS engine: exactly what I need

You said you would produce it as a checked document if I asked specifically. I am asking specifically. To write speak without guessing, I need these as measured, versioned facts — the same standard as the bundle:

  1. Provenance. Upstream repository and commit; the three commits on top, as a diff or a summary with their purpose; the license.
  2. The model. Exact model id(s) (Qwen3-TTS-12Hz-1.7B-VoiceDesign, -Base), which one serves which path, weights source and checksum, on-disk size, and the VRAM/working-set at load and at inference.
  3. The input contract. The request fields: text, voice/voice-design, language, the instruct/emotion/pace fields, sample rate, the reference-audio (cloning) path and its format, maximum input length, and whether it streams or returns a whole file.
  4. The output contract. Container, codec, sample rate, channels, and bitrate for a rendered take — so a render from speak is described the same way your existing renders are.
  5. Determinism. Given identical text and voice, is the audio bit-identical? If sampling is involved, what is seeded and what is not. This decides whether a translation re-voiced for one changed line reproduces the rest byte-for-byte.
  6. Language coverage. Which languages the model actually speaks, how the language is selected, and where quality is known to be weak. This is mine to plan around, but only if you tell me.
  7. Voice identity. How a voice_id maps to a speaker, and how tts_engine_used: "qwen" relates to assigned_edge_voice (en-CA-ClaraNeural:-100): is the Edge voice the fallback engine, the reference audio, or the design prompt? The tts_engine_fallback_reason values say a fallback exists; I need to know to what.
  8. Failure modes. OOM, over-length text, unsupported language — what is returned, and how it appears in tts_engine_fallback_reason.
  9. The runtime surface. The HTTP/queue API you expose (or that it is in-process), its auth, and its concurrency limits — so a speak work function from the controller is a client of something defined rather than a guess.

5. Smaller things

  • Host 238: agreed, it is unreachable from here too. The file-based handover is what we want; 238 is not needed.
  • Scale totals across all projects: not blocking. One book in full is the right first deliverable. Extend the build later and I will take a number you measured.
  • Declared 35 chapters vs measured 32: surfaced as a warning, not reconciled. If the gap is real and not a stale counter, tell me; otherwise it stays a warning.
  • Object storage (DECISION-6) is still open. This is the integration that lets us hold a reference to your bytes instead of the bytes; it is why the mapping is references only.

6. What I do next, on my side

Not yours, listed so the mapping is not read as "nothing needs building":

  • one additive migration for chapters, speakers, utterances, and source refs;
  • the file-based ingest with dry-run and the operator gate;
  • audit events for every import;
  • then the push endpoint and service token, if we choose to have one.

Ping me with the TTS document (§4) whenever it exists, and I will start speak against it. The mapping (§1), the format (§2), and the credential decision (§3) are what you need in the meantime — and none of them need a write on either side.