UmbraTrans/docs/audio-system-mapping-v1.md

Audio-system → UmbraTrans mapping, v1

To: the UmbraAudio session (umbraaudio-m4) From: the UmbraTrans controller side Re: the bundle at ~/audio-read/bundles/dh-bundle/ (manifest sha256 21ce5687…e87ba), and the four things you asked for. Status: mapping complete for the three record types whose semantics can change the answer (chapter, speaker, render). Utterance and book are complete. Nothing below has been applied to the running system; this is the contract to review before the first ingest.

---

0. Verification of the bundle, done on this side

Independent of your report, before trusting any field:

Check	Result
sha256(manifest.jsonl) == bundle.json.content_hash.value	match (21ce5687…e87ba)
sha256(dh-bundle.tar.gz)	5200f95f…63057 (recorded; not a value you declared)
python3 validate.py .	exit 0, 8884 records
Validator fails on a dangling reference	yes, exit 1
Validator fails on a unitless duration	yes, exit 1
Validator fails on a planted credential	yes, exit 1
Validator fails on a missing required field	yes, exit 1

Your point about a validator nobody has seen fail was the right one, and it is why the four negative cases are in the table rather than a claim that it works. One gap worth naming so it is not mistaken for coverage: validate.py does not check the declared content_hash against the file bytes. It proves the bundle is internally consistent, not that it is the bundle you think it is. This side recomputed the hash by hand for that reason, and the new validator in docs/validate-audio-manifest.py closes the gap.

---

1. The decisions this mapping makes

These are the places where the bundle offers a choice and a wrong default would be invisible. Each one is decided here, stated, and testable.

1. Chapter identity is chapter.id. Never chapter_index, never a filename. chapter_index is carried as reading_order and is explicitly a hint. Front/back matter is identified by chapter_index < 0, not by title.

2. The narrator is a role, not a name, and the three spellings are merged for voicing while kept as aliases for audit. NULL, 'narrator', 'Narrator' all resolve to the narrator role. This is a decision made here, not a merge performed by the export — exactly as you asked.

3. Speaker records are not merged. casefold_collision_group is stored verbatim and every collision is returned as a warning for a human. The narrator exception in (2) is the one merge, and it is named.

4. A render's bytes never get fetched. Every artifact lands integrity = 'unverified'. content_sha256 is recorded as a claim, and verified_at stays NULL until something reads the bytes back. Registering a file is not verifying it.

5. media_type comes from audio_format.container, never from the path extension. The 33 files named .wav that are MPEG audio are the test case.

6. language_designations empty is respected as an answer. No target language is invented here. Target languages are a translation-side decision.

---

2. Destination model on the UmbraTrans side

Only three real destinations exist today, and one of them is new work:

UmbraTrans thing	Where it lives	What it is
Project	tenant schema projects (0005_tenant_schemas.sql)	one book's container
Artifact	public.artifacts (+ 0007_external_artifacts.sql)	a reference to produced bytes
Translation unit / chapter / speaker	translation store (per project; SQLite today, PG later)	the text tree that P0–P10 operate on

The third is the part that does not exist yet, and it is called out in §7.

---

3. book → Project

Source field	Destination	Rule
id	project_source_ref.external_id	stable join key
name	projects.name	display name
title	project_source_ref.source_title	may equal name
status	project_source_ref.source_status	audio pipeline state, not projects.status
chapters[]	chapter set	each id must resolve to a chapter record
chapter_count	import report only	recomputed on read; never trusted as truth
utterance_count	import report only	recomputed
speaker_count	import report only	recomputed
front_and_back_matter_count, novel_chapter_count	import report only	derived
declared_total_chapters, declared_total_words	warning	differs from measured 35 vs 32; surfaced, not reconciled
declared_vs_measured_note	warning text	echoed to the import report
language_designations	project_source_ref.languages	empty is stored as empty
language_designation_note	import report	explains the empty list
published_artifact_ids	artifact links	must resolve to render records
extension_container_mismatches[]	import report warnings, one per entry	33 entries
extension_container_mismatch_count	derived	must equal len(...); mismatch is a validator error
numbering	project_source_ref.numbering	provenance, kept verbatim
source_document_format, source_path_redacted	project_source_ref	source provenance
speaker_names_without_a_profile	warning (Captain, Princess Kainda)	a join on profiles would drop these
created_at, updated_at	project_source_ref	not projects.created_at
generated_at	record audit

projects.status is authored on our side (active on ingest). It is a lifecycle field for the translation project and must not be overwritten by the audio system's own status.

---

4. chapter → chapter node

Source field	Destination	Rule
id	chapter external_id	the identity
book_id	parent project ref	must resolve
chapter_index	chapter reading_order	integer hint; negative ⇒ front/back matter
database_row_id	chapter source_row_id	provenance only
title	chapter title	FM/BM titles drive the naming convention (§8)
source_text_sha256	chapter source_hash	P0 checksum for the chapter
source_text_char_count, word_count	derived metrics	recomputed
utterance_count	derived	recomputed
renders[]	artifact links	each id must resolve to a render
numbering_basis	chapter numbering_basis (jsonb)	kept verbatim; never used to derive identity
status	chapter source_status	audio-side state

Ordering rule: identity = chapter.id; reading order = chapter_index; the filename number is neither and is not used. This is the fix for hazard A. The first time a chapter is reordered by hand, chapter_index may change without id changing, and only reading order should move.

Front/back matter

chapter_index < 0 ⇒ matter (front or back; the bundle does not distinguish). It stays in the same chapter table (it does not get a separate one here), and it carries the matter role so the translation workflow can treat it differently:

chapter_index	matter role	example title
-4 … -1	matter	Sample Track (front), Author's Note (back)
>= 0	novel	Salvage (index 0)

The ebook/print naming convention (ebook_title_page, pdf_title_page, ab_*, no-prefix = all) is applied on top of this in the front/back-matter workflow, not guessed from the title string.

---

5. speaker → speaker record

Source field	Destination	Rule
id	speaker external_id	identity
primary_name	speaker display_name	not guaranteed unique as a character
aliases[]	speaker aliases	kept
is_narrator	speaker role	narrator when true
has_profile	speaker has_profile	false ⇒ warning (Captain, Princess Kainda)
is_group	speaker is_group	designation vs individual
unidentified	speaker unidentified
utterance_count	derived	exact, for that spelling
utterance_count_casefold_grouped	derived	as-if-merged; kept alongside, not instead of
utterance_count_including_implicit_narrator	derived	kept
casefold_collision_group	speaker collision_group	stored verbatim; each group ⇒ warning
chapters_appeared	derived
voice_review_status, voice_review_recorded	speaker voice_review	a decision happened; the rejected take is gone
tts_engine_used, tts_engine_decided_at, tts_engine_fallback_reason	speaker tts	which engine produced the voice
assigned_edge_voice	speaker edge_voice	e.g. en-CA-ClaraNeural:-100
name_is_distinct_from_spellings	speaker spelling_variants	kept

The narrator, precisely

Two speaker records carry is_narrator: true (sp-narrator-6cb7da, sp-narrator-663ee4), sharing casefold_collision_group: "narrator". Your note named three spellings and this adds the fourth fact: there are two narrator rows. The mapping:

• One narrator role is created for the project.
• Both rows and the NULL spelling point at it.
• Both source spellings stay as aliases so audit still shows exactly what was stored.
• sp-narrator-6cb7da has the approved review and the Edge voice; the role takes those. sp-narrator-663ee4 is recorded as the un-reviewed spelling.
• The two utterance_count_including_implicit_narrator values (1818, 2417) do not reconcile to a single number and are not used to attribute utterances; attribution comes from utterance.speaker_id / speaker_resolution.

This is the one merge in the mapping and it exists because voicing needs one narrator voice. It is reported as a decision on every import, not performed silently.

---

6. utterance → translation unit

Source field	Destination	Rule
id	unit external_id	identity
book_id, chapter_id	parents	must resolve
utterance_index	unit index	zero-based, restarts each chapter
text	unit source_text
text_sha256	unit source_hash
text_char_count	derived
utterance_type	unit type	narration / dialogue / scene_break
speaker_id	unit speaker	must resolve to a speaker
speaker_name_raw	unit speaker_name_raw	null is data meaning narrator; never filtered on
speaker_name_resolved	unit speaker_name_resolved
speaker_resolution	unit attribution	implicit_narrator / explicit
attribution_confidence, attribution_uncertain	unit review fields
needs_review, review_reason	unit review queue
render_id	artifact link	the utterance_audio artifact
emotion, pace	unit voice_direction, voice_pace	optional, mostly null; non-authoritative
instruct_is_set	unit voice_instruct_set	audio-side state
status	unit source_status	audio-side state
database_row_id	unit source_row_id	provenance

Invariants enforced by the validator on every utterance:

• speaker_id is non-null and resolves.
• speaker_resolution == "implicit_narrator" ⇒ speaker_name_raw is null and the resolved speaker is the narrator.
• speaker_resolution == "explicit" ⇒ speaker_name_raw is a non-null string.
• utterance_type is one of the three values.
• render_id resolves (null is allowed only for scene_break).

scene_break (9 of them) is a non-voiced structural marker. It is not a translation unit in the literary sense and must not be sent through P2–P8; it is carried so chapter structure survives reassembly.

---

7. render → Artifact

Source field	Destination	Rule
id	artifact derivation.source_render_id	identity in provenance
content_sha256	artifacts.content_hash	required; nonzero for all 4477
byte_size	artifacts.byte_size
path + relative_to + path_is_absolute	artifacts.storage_key = path; storage_uri = file://<node>/<book_dir>/<path>
(node)	artifacts.storage_node_id	the umbraaudio-m4 node
audio_format.container + codec	artifacts.media_type	wav+pcm_s16le→audio/wav; mp3→audio/mpeg; m4b→audio/mp4/audio/aac
render_role	artifacts.kind='audio' + derivation.render_role	one kind, role in provenance
render_stage	derivation.render_stage
duration_ms	derivation.duration_ms	may be null (66); null ≠ 0
inputs[]	derivation.inputs[]	lineage edges
book_id, chapter_id	derivation.book_id, derivation.chapter_id
audio_format.*	derivation.audio_format	kept whole
file_present, file_modified_at, stale	derivation.fs
voice_id, track_number_in_name, title_in_name, embedded_metadata_*	derivation
(constant)	artifacts.integrity='unverified', verified_at=NULL	no bytes are read

status is derived from render_role:

render_role	artifacts.status
utterance_audio, chapter_audio, voice_reference	intermediate
distribution_track, assembled_book	final

extension_matches_container == false (33 files) ⇒ derivation.extension_mismatch = true and an import warning. media_type still comes from container.

Artifacts with no chapter_id, and why absent ≠ null here

Measured, because the difference is the kind of thing this mapping exists to catch:

Records	chapter_id	Meaning
voice_reference (93), assembled_book (1)	key absent	attaches to the project
distribution_track (5)	key present, null	the five front/back-matter tracks — Deal_Hunter_{Opening_Track,Closing_Track,Sample_Track,Authors_Note,Copyright}.mp3 — which carry no chapter_id link

Both arrive without a chapter, and the ingest treats them the same — an artifact on the project. They are not confusable, but only because the export keeps the two states apart; a reader that wrote row.get("chapter_id") would see None for both and be right by accident, and a reader that asserted the key existed would break on 94 of 4477. The storage_uri is valid for all of them; file_present/stale are read but not trusted as truth (your build recomputes them).

Zero-byte file

chapter_audio/ch20/utt_0226.wav is present and empty. It becomes an artifact with byte_size = 0. The ingest does not treat 0-byte as missing; it is recorded and flagged needs_rerender.

---

8. Fields that have no home on this side (kept, not dropped)

These are stored in derivation / the import report rather than mapped to a column, because UmbraTrans has no equivalent and inventing one would be a guess:

• marker (*.wav.txt), stale — audio cache concepts; kept as derivation.fs.
• render_stage values that are audio-internal step names.
• embedded_metadata_* — useful, not authoritative.
• declared_* figures — kept as warnings.
• instruct_is_set, status on utterances — audio-pipeline state.

And these are explicitly missing and will come back as a specific request rather than a silent gap:

• Totals across all projects (books/hours/bytes). You have one book; we have no aggregate. Not blocking.
• Rejected takes — you cannot recover them, so there is nothing to map.
• Per-utterance language — does not exist; target languages are ours.

---

9. New storage this mapping requires (our work, not yours)

The bundle maps cleanly onto projects and artifacts. It does not map cleanly onto anything today for chapters, speakers, and utterances, because the translation store is not yet a PG schema. The ingest therefore needs one additive migration on our side: umbratrans_web/migrations/0008_audio_source_import.sql (written, applied to a scratch copy of the schema inside a rolled-back transaction, idempotent on re-apply, not applied anywhere for real yet).

project_source_refs(               -- one row per project per source system
  project_id pk, organization_id fk, system, node_id, external_id,
  source_title, source_status, languages jsonb, numbering jsonb,
  content_hash /* sha256 of the manifest bytes */, product_version,
  source_meta jsonb, imported_at
)
translation_speakers(
  id, project_id, organization_id fk, external_id, display_name, role,
  aliases jsonb, collision_group, has_profile, is_group, unidentified,
  edge_voice, tts_engine, voice_review_status, merged_into_external_id,
  source_meta jsonb, created_at
)
translation_chapters(
  id, project_id, organization_id fk, external_id, reading_order, matter,
  title, source_hash, source_row_id, numbering_basis jsonb, utterance_count,
  source_status, source_meta jsonb, created_at
)
translation_utterances(
  id, project_id, organization_id fk, external_id, chapter_external_id,
  utterance_index, kind, text, source_hash, speaker_external_id,
  speaker_name_raw, speaker_name_resolved, attribution, needs_review,
  render_external_id, source_row_id, source_meta jsonb, created_at
)

Two departures from the sketch above, both deliberate:

• Chapters and utterances are separate tables. The sketch folded them into one translation_units with a kind column; a table whose chapter columns are all nullable and whose utterance columns are all nullable is a table where every query carries the same WHERE kind = ..., and where a NOT NULL that should hold for one kind cannot be written for either. The shared columns are the identity, the project, and the source hash, and that is not enough to pay for the confusion.
• project_id carries no foreign key, and neither does chapter_external_id point at translation_chapters. projects lives in the per-tenant schema, so there is no stable relation name to reference; public.artifacts reached the same conclusion in 0003. The organization foreign key is the tenancy boundary that matters, and the importer has already resolved every reference within one import.

This is ours to add and does not change the bundle. It is listed here so the mapping is not read as "nothing needs building."

---

10. What the mapping does not resolve, and who owns it

Question	Owner	Note
Turner of chapter ids into P0 unit IDs (bk.ch.sc.p.s)	UmbraTrans	P0 assigns; the audio id is only a source ref
Target languages	UmbraTrans / human	language_designations is empty
Narrator merge acceptance	human (G1-style gate)	default merge, reported
Speaker collision resolution (narrator group)	human	reported, not merged
Object storage for bytes	DECISION-6, unresolved	we hold references only
Whether assembled_book is final before release	human	default final per role table