umbragit/UmbraAudio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
UmbraAudio/scripts/build_controller_bundle.py
B.D. Murphy 3d69187449 Answer the controller's data request, and audit what we send 
The UmbraTrans controller asked (commit 70c23fd) for machine-readable metadata
about the audio system, in a specified layout, with a validator that ships with
the request so the sender can check its own work. That request is answered here.

scripts/build_controller_bundle.py emits that bundle: one real book (Deal
Hunter), 8,884 records, no media, no credentials, no hostnames. It passes the
controller's own validator, exit 0.

Three things in this system will break a mapping that does not know them, and
the bundle's real purpose is to say so precisely rather than to look tidy:

* A chapter has THREE numbers. The database row id is what filenames use,
  chapter_index is reading order, and the distribution track number is a third
  thing again. Only the row id survives an edit. Front and back matter are
  numbered with roman numerals in the MP3 filenames.
* The narrator is written three ways, and the most common way is nothing at all.
  1,803 utterances have a NULL speaker_name, meaning narrator; 614 say
  "narrator" and 15 say "Narrator". A join that skips nulls loses most of the
  narration silently. Utterances now carry speaker_name_raw, the resolved name,
  and speaker_resolution, and the two spellings are not merged - that is a
  decision for whoever owns speaker identity, not something an export should do
  quietly.
* A .wav file is not always a WAV. 30 of the 91 voice reference files are MPEG.
  Every render reports the container measured from the bytes, with
  extension_matches_container, so a reader cannot be misled by the name.

A validator proves the bundle is internally consistent; it cannot prove the
bundle is true. scripts/audit_controller_bundle.py re-measures every hash, byte
size, duration and count against the project on disk - 66,011 claims, all
holding. It caught two real defects in the first draft of this bundle, including
one of my own: produced_at was stamped with the build time, which is a claim
about the audio that nothing supported.

The validator was also run against seven deliberately broken bundles to confirm
it can fail. A validator nobody has seen fail is not evidence of anything; those
cases are now tests.

scripts/audio_controller_connect.py runs the T01-T10 suite: 4 pass, 1 fails,
5 are blocked. T04 fails honestly - there is no ingest endpoint, and nothing was
pushed. T05-T08 are blocked because the controller's own plan sequences ingest
after the metadata arrives; a proposed contract is included rather than an
invented endpoint. T02 is blocked: no node key or service token is configured
here, and an operator has to supply one.

Two corrections to the node profile, which the controller reads: it claimed the
web UI has no authentication (a session gate was added 2026-10-09) and that the
controller host is offline (it is reachable and serving umbratrans-web 0.1.0).

Full suite 751 passed, 8 failed - the same pre-existing pronunciation and
accent failures, unchanged. ruff and mypy clean on everything added.
2026-10-14 10:56:32 -07:00

1598 lines
70 KiB
Python
Raw Permalink Blame History

#!/usr/bin/env python3
"""Build the metadata bundle the UmbraTrans controller asked for.
The controller's request (``docs/audio-system-data-request.md`` on the controller
host) is explicit that prose cannot be validated and JSON can, and it ships a
validator that must pass. This script emits that bundle from the live audio
project so the sender can check their own work before it travels.
Everything here is read-only against the project. Nothing in the audio system is
modified, moved, or deleted; the audio system keeps its data and keeps running.
Design decisions worth knowing about, because they are the ones a reader would
otherwise have to reverse-engineer:
* **Deterministic.** Records are emitted in a fixed order and every JSON object
is written with sorted keys, so regenerating on an unchanged project produces a
byte-identical ``manifest.jsonl`` and therefore a stable ``content_hash``.
``--generated-at`` exists so that even the timestamp can be pinned, which is
what makes "has anything changed?" a hash comparison instead of a diff.
* **No absolute paths.** The request forbids leaking machine layout into a
document meant to travel. Every path is relative to the book directory and the
record says so via ``relative_to``, which is also what the controller's
validator demands for any field named like a path.
* **No secrets.** Hostnames, usernames, tokens and keys are excluded. The only
host that appears is the one the controller already knows about, and it is
written as a placeholder rather than a real name.
* **Three different chapter numbers exist and are named as such.** This is the
single most dangerous thing in the audio system and the controller asked for it
specifically. See :func:`chapter_numbering` for the full explanation.
python3 scripts/build_controller_bundle.py --project DealHunterShip_4.0e_whole-to-audio
python3 scripts/build_controller_bundle.py --project ... --out /tmp/bundle --tarball
Validation is the controller's, not this script's:
python3 /tmp/bundle/validate.py /tmp/bundle
"""
from __future__ import annotations
import argparse
import hashlib
import json
import re
import shutil
import sqlite3
import subprocess
import tarfile
import wave
from collections import defaultdict
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
BUNDLE_FORMAT_VERSION = 1
SCHEMA_ID_PREFIX = "https://umbra.invalid/schemas/umbraaudio-bundle/v1"
#: Placeholder for the machine's real home/username, so a bundle is committable
#: and does not describe the layout of the host that produced it.
PROJECT_ROOT_PLACEHOLDER = "<PROJECT_ROOT>"
#: The controller is a different machine and is named in the request itself, so
#: naming it back adds nothing an attacker could not already read. It is still
#: written as a placeholder for consistency with the no-hostnames rule.
CONTROLLER_PLACEHOLDER = "<CONTROLLER_HOST>"
# --------------------------------------------------------------------------
# canonical JSON + hashing
# --------------------------------------------------------------------------
def canonical(obj: Any) -> str:
"""One JSON form for one value, so a hash means what it appears to mean."""
return json.dumps(obj, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
def sha256_text(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def sha256_file(path: Path) -> str | None:
try:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
except OSError:
return None
def mtime_utc(path: Path) -> str | None:
"""When the file was last written, as an explicit UTC instant.
Used instead of the build time on purpose. Recording "produced at" as the
moment the bundle happened to be generated would be a claim about the audio
that nothing supports -- the usual way a metadata export starts disagreeing
with the thing it describes.
"""
try:
return datetime.fromtimestamp(path.stat().st_mtime, UTC).isoformat().replace(
"+00:00", "Z")
except OSError:
return None
def utc_stamp(value: str | None) -> str | None:
"""SQLite stores ``YYYY-MM-DD HH:MM:SS`` in UTC with no zone marker.
The request asks for explicit UTC on every timestamp. A naive local-time
stamp that gains a ``Z`` is a lie, so this one is explicit about what it is
doing rather than silently reinterpreting it.
"""
if not value:
return None
text = value.strip().replace("T", " ")
for fmt in ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S", "%Y-%m-%d"):
try:
parsed = datetime.strptime(text, fmt).replace(tzinfo=UTC)
except ValueError:
continue
return parsed.isoformat().replace("+00:00", "Z")
return None
# --------------------------------------------------------------------------
# audio facts, measured rather than assumed
# --------------------------------------------------------------------------
@dataclass(frozen=True, slots=True)
class AudioFormat:
container: str
codec: str
sample_rate_hz: int | None
channels: int | None
bit_rate_bps: int | None
extension: str = ""
detected_by: str = "none"
extension_matches_container: bool | None = None
def as_record(self) -> dict[str, Any]:
return {
"container": self.container,
"codec": self.codec,
"sample_rate_hz": self.sample_rate_hz,
"channel_count": self.channels,
"bit_rate_bps": self.bit_rate_bps,
"file_extension": self.extension,
"extension_matches_container": self.extension_matches_container,
"detected_by": self.detected_by,
}
def probe_wav(path: Path) -> AudioFormat:
try:
with wave.open(str(path)) as handle:
return AudioFormat("wav", "pcm_s16le", handle.getframerate(),
handle.getnchannels(), None,
extension=path.suffix.lower().lstrip("."),
detected_by="wave-module",
extension_matches_container=True)
except (OSError, wave.Error, EOFError):
# Not readable as WAV. In this project that usually means the file is
# fine and the *name* is wrong: 30 of the 91 voice reference files are
# MPEG audio with a .wav extension. Falling back to ffprobe reports the
# container that is actually there rather than the one the name implies.
return probe_media(path)
def probe_media(path: Path) -> AudioFormat:
"""ffprobe if it is on PATH, else a header sniff, else unknown.
An unknown format is reported as unknown. Guessing would be worse: the
controller's stated reason for asking is that a wrong number reads plausibly.
"""
extension = path.suffix.lower().lstrip(".")
try:
out = subprocess.run(
["ffprobe", "-v", "error", "-select_streams", "a:0",
"-show_entries", "stream=codec_name,sample_rate,channels,bit_rate",
"-show_entries", "format=duration,bit_rate",
"-of", "json", str(path)],
capture_output=True, text=True, timeout=60, check=False,
)
data = json.loads(out.stdout or "{}")
streams = data.get("streams") or [{}]
stream = streams[0]
fmt = data.get("format") or {}
def as_int(value: Any) -> int | None:
try:
return int(value)
except (TypeError, ValueError):
return None
codec = str(stream.get("codec_name") or "unknown")
container = _container_for_codec(codec, extension)
return AudioFormat(
container=container,
codec=codec,
sample_rate_hz=as_int(stream.get("sample_rate")),
channels=as_int(stream.get("channels")),
bit_rate_bps=as_int(stream.get("bit_rate") or fmt.get("bit_rate")),
extension=extension,
detected_by="ffprobe",
extension_matches_container=(None if container == "unknown"
else container == extension),
)
except (OSError, ValueError, subprocess.SubprocessError):
return AudioFormat(extension or "unknown", "unreadable", None, None, None,
extension=extension, detected_by="none",
extension_matches_container=None)
def _container_for_codec(codec: str, extension: str) -> str:
"""What the bytes actually are, which is not always what the name says."""
known = {"mp3": "mp3", "mpeg_audio": "mp3", "aac": "aac", "flac": "flac",
"alac": "m4a", "pcm_s16le": "wav", "opus": "opus", "vorbis": "ogg"}
if codec in known:
return known[codec]
if extension:
return extension
return "unknown"
def wav_duration_ms(path: Path) -> int | None:
try:
with wave.open(str(path)) as handle:
rate = int(handle.getframerate())
if not rate:
return None
return int(round(int(handle.getnframes()) / rate * 1000))
except (OSError, wave.Error, EOFError):
return None
# --------------------------------------------------------------------------
# the chapter-numbering hazard
# --------------------------------------------------------------------------
def chapter_numbering() -> dict[str, Any]:
"""Three numbering schemes for one chapter, none of them the others.
The controller's request says, of a filename component, "is the ``003`` a
chapter number, an index, or a sort position?" — and for this system the
honest answer is *all three, in different places*. Treating any one of them
as authoritative produces a mapping that is correct until the first time a
chapter is reordered or front matter is inserted, so each scheme is named
and each consumer records which one it used.
``Salvage``, which is the first chapter of the actual novel:
========================== =========================================
Scheme Value
========================== =========================================
``chapters.id`` 6 -- what ``ch6/`` and ``ch6_full.wav`` use
``chapters.chapter_index`` 0 -- 0 for the novel, negative for front matter
AR track number ``01`` -- ``chapter_index`` plus one, 1-based
========================== =========================================
"""
return {
"hazard": (
"Three distinct numbers identify one chapter and none derives "
"reliably from another under edit."
),
"schemes": {
"database_row_id": {
"field": "chapters.id",
"example": 6,
"used_by": [
"chapter_audio/ch<id>/ directory name",
"chapter_audio/ch<id>_full.wav",
"utterances.chapter_id foreign key",
],
"stable_under": "insertion and reordering",
"is_authoritative": True,
},
"chapter_index": {
"field": "chapters.chapter_index",
"example": 0,
"used_by": ["logical reading order"],
"stable_under": "nothing: it is a hand-set order",
"is_authoritative": False,
"note": (
"0 is the first chapter of the novel. Front and back matter "
"are negative (-4..-1). Two ids are absent from the sequence "
"because chapters 1 and 2 were later removed."
),
},
"distribution_track_number": {
"field": "derived at export time",
"example": "01",
"used_by": ["Deal_Hunter_01_Salvage.mp3"],
"stable_under": "nothing",
"is_authoritative": False,
"note": (
"chapter_index plus one for the novel. Front and back matter "
"use lowercase roman numerals (i, ii) rather than digits."
),
},
},
"consequence_for_mapping": (
"The database row id is the only identifier that survives an edit. "
"Anything derived from chapter_index is a snapshot of one export and "
"must be re-derived, not stored as a fact."
),
}
# --------------------------------------------------------------------------
# source reading
# --------------------------------------------------------------------------
@dataclass
class Source:
book_dir: Path
umbra: Path
db: Path
name: str
def open_db(db_path: Path) -> sqlite3.Connection:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn.row_factory = sqlite3.Row
return conn
def load_json(path: Path, default: Any = None) -> Any:
try:
return json.loads(path.read_text())
except (OSError, ValueError):
return default
def slug(text: str) -> str:
cleaned = re.sub(r"[^a-z0-9]+", "_", str(text).strip().lower()).strip("_")
return cleaned or "unnamed"
def _as_scalar(value: Any) -> Any:
"""These JSON documents are not consistently shaped.
``voice_status.json`` maps a speaker to a bare string while
``voice_engines.json`` maps a speaker to an object. Normalising here means a
shape change upstream surfaces as a null plus ``*_recorded: false`` rather
than as an exception halfway through a build.
"""
if isinstance(value, dict):
return value.get("status") or value.get("value")
return value
def col(row: sqlite3.Row, name: str, default: Any = None) -> Any:
"""Read a column that may not exist in this project's schema.
This system carries more than one database shape: projects created before the
attribution work was added are missing nine ``utterances`` columns entirely,
and the bundle has to describe any of them. A missing column is reported as
absent rather than raising, because the alternative is that the newest
project is the only one that can be exported.
"""
try:
return row[name]
except (IndexError, KeyError):
return default
# --------------------------------------------------------------------------
# record builders
# --------------------------------------------------------------------------
class BundleBuilder:
"""Accumulates records and guarantees the invariants the validator checks."""
def __init__(self, generated_at: str) -> None:
self.generated_at = generated_at
self.records: list[dict[str, Any]] = []
self.ids: set[str] = set()
def add(self, record: dict[str, Any]) -> dict[str, Any]:
rid = record.get("id")
if isinstance(rid, str):
if rid in self.ids:
raise ValueError(f"duplicate record id: {rid}")
self.ids.add(rid)
self.records.append(record)
return record
def lines(self) -> str:
return "".join(canonical(record) + "\n" for record in self.records)
def build_records(src: Source, generated_at: str) -> tuple[BundleBuilder, dict[str, Any]]:
builder = BundleBuilder(generated_at)
conn = open_db(src.db)
book_id = "book-1"
project = conn.execute("SELECT * FROM project WHERE id = 1").fetchone()
chapters = conn.execute(
"SELECT * FROM chapters WHERE project_id = 1 ORDER BY chapter_index, id"
).fetchall()
utterances = conn.execute(
"""SELECT u.* FROM utterances u JOIN chapters c ON c.id = u.chapter_id
WHERE c.project_id = 1 ORDER BY u.chapter_id, u.utterance_index"""
).fetchall()
speakers_doc = load_json(src.umbra / "speakers.json", {}) or {}
speakers_list = speakers_doc.get("speakers") or []
voice_status = load_json(src.umbra / "voice_status.json", {}) or {}
voice_engines = load_json(src.umbra / "voice_engines.json", {}) or {}
assignments = load_json(src.umbra / "voice_assignments.json", {}) or {}
# --- speakers -------------------------------------------------------
# The union of the curated speaker document and the names actually written
# onto utterances. The difference is a real finding, not noise: a name on an
# utterance with no profile is exactly the kind of gap a mapping must not
# paper over.
curated: dict[str, dict[str, Any]] = {}
for entry in speakers_list:
name = str(entry.get("primary_name") or "").strip()
if name:
curated[name.lower()] = entry
used_names = sorted({str(row["speaker_name"] or "narrator").strip()
for row in utterances if row["speaker_name"]})
# Speaker identity in this system has **three** representations, not one,
# and the largest is the empty one:
#
# * NULL speaker_name -- every one of them narration. This is the
# narrator, stated by saying nothing.
# * "narrator" -- the most common *written* spelling.
# * "Narrator" -- a small third spelling.
#
# A mapping that reads `speaker_name` and skips nulls loses most of the
# narration without raising an error, and a mapping that joins on name splits
# the narrator in two. Merging the three here would be a silent correction to
# somebody else's data, so the two written spellings stay separate records
# sharing a `casefold_collision_group`, and NULL is reported as its own
# resolved value (`speaker_resolution: "implicit_narrator"`) rather than as
# absence.
collision_groups: dict[str, list[str]] = defaultdict(list)
for name in used_names:
collision_groups[name.casefold()].append(name)
colliding = {key: names for key, names in collision_groups.items() if len(names) > 1}
implicit_narrator_count = sum(
1 for row in utterances
if row["speaker_name"] is None or not str(row["speaker_name"]).strip()
)
speaker_ids: dict[str, str] = {}
speaker_records: list[dict[str, Any]] = []
for name in used_names:
group = name.casefold()
sid = f"sp-{slug(name)}"
if group in colliding:
sid = f"{sid}-{sha256_text(name)[:6]}"
speaker_ids[name.lower()] = sid
entry = curated.get(group)
aliases = [a for a in ((entry or {}).get("aliases") or []) if a]
record: dict[str, Any] = {
"id": sid,
"kind": "speaker",
"primary_name": name,
"name_is_distinct_from_spellings": sorted(colliding.get(group, [name])),
"casefold_collision_group": group if group in colliding else None,
"has_profile": entry is not None,
"is_narrator": bool((entry or {}).get("is_narrator", name.casefold() == "narrator")),
"is_group": bool((entry or {}).get("is_group", False)),
"unidentified": bool((entry or {}).get("unidentified", False)),
"aliases": aliases,
"chapters_appeared": list((entry or {}).get("chapters_appeared") or []),
"utterance_count": sum(
1 for row in utterances
if str(row["speaker_name"] or "").strip() == name
),
"utterance_count_casefold_grouped": sum(
1 for row in utterances
if str(row["speaker_name"] or "").strip().casefold() == group
),
"utterance_count_including_implicit_narrator": (
sum(1 for row in utterances
if str(row["speaker_name"] or "").strip() == name)
+ (implicit_narrator_count if group == "narrator" else 0)
),
"voice_review_status": _as_scalar(voice_status.get(name)),
"voice_review_recorded": name in voice_status,
"tts_engine_used": (voice_engines.get(name) or {}).get("engine"),
"tts_engine_fallback_reason": (voice_engines.get(name) or {}).get("reason"),
"tts_engine_decided_at": (voice_engines.get(name) or {}).get("at"),
"assigned_edge_voice": assignments.get(name),
"generated_at": generated_at,
}
speaker_records.append(record)
builder.add(record)
unprofiled = sorted(r["primary_name"] for r in speaker_records if not r["has_profile"])
# --- chapters and their renders -------------------------------------
chapter_audio = src.umbra / "chapter_audio"
chapter_ids: list[str] = []
chapter_renders: dict[str, list[str]] = defaultdict(list)
chapter_counts: dict[str, dict[str, Any]] = {}
for row in chapters:
cid = f"ch-{row['id']}"
chapter_ids.append(cid)
own = [u for u in utterances if u["chapter_id"] == row["id"]]
chapter_counts[cid] = {
"utterances": len(own),
"chars": sum(len(str(u["text"] or "")) for u in own),
}
full = chapter_audio / f"ch{row['id']}_full.wav"
if full.exists():
rid = f"rnd-chfull-{row['id']}"
fmt = probe_wav(full)
builder.add({
"id": rid,
"kind": "render",
"render_role": "chapter_audio",
"render_stage": "assembled_chapter",
"book_id": book_id,
"chapter_id": cid,
"path": f"chapter_audio/ch{row['id']}_full.wav",
"relative_to": "book_directory",
"path_is_absolute": False,
"audio_format": fmt.as_record(),
"duration_ms": wav_duration_ms(full),
"byte_size": full.stat().st_size,
"content_sha256": sha256_file(full),
"inputs": [],
"file_modified_at": mtime_utc(full),
"generated_at": generated_at,
})
chapter_renders[cid].append(rid)
builder.add({
"id": cid,
"kind": "chapter",
"book_id": book_id,
"database_row_id": row["id"],
"chapter_index": row["chapter_index"],
"title": row["title"],
"word_count": row["word_count"],
"status": row["status"],
"source_text_sha256": sha256_text(str(row["source_text"] or "")),
"source_text_char_count": len(str(row["source_text"] or "")),
"utterance_count": chapter_counts[cid]["utterances"],
"renders": list(chapter_renders[cid]),
"numbering_basis": {
"file_names_use": "database_row_id",
"reading_order_uses": "chapter_index",
"chapter_index_is_authoritative": False,
},
"generated_at": generated_at,
})
# --- utterance renders, then utterances ------------------------------
for row in utterances:
cid = f"ch-{row['chapter_id']}"
rid = f"rnd-utt-{row['id']}"
path = chapter_audio / f"ch{row['chapter_id']}" / f"utt_{row['utterance_index']:04d}.wav"
marker = path.with_suffix(".wav.txt")
marker_doc = load_json(marker, {}) or {}
exists = path.exists()
builder.add({
"id": rid,
"kind": "render",
"render_role": "utterance_audio",
"render_stage": "rendered_utterance",
"book_id": book_id,
"chapter_id": cid,
"path": f"chapter_audio/ch{row['chapter_id']}/utt_{row['utterance_index']:04d}.wav",
"relative_to": "book_directory",
"path_is_absolute": exists,
"file_present": exists,
"audio_format": probe_wav(path).as_record() if exists else None,
"duration_ms": wav_duration_ms(path) if exists else None,
"byte_size": path.stat().st_size if exists else None,
"content_sha256": sha256_file(path) if exists else None,
"inputs": [],
"stale": bool(marker_doc) and (
marker_doc.get("text") != str(row["text"] or "")
),
"file_modified_at": mtime_utc(path) if exists else None,
"generated_at": generated_at,
})
raw_speaker = row["speaker_name"]
# An empty speaker name is not absence: it is the narrator. Resolving it
# to the narrator record while recording that the resolution was implied
# keeps the row joinable *and* keeps the inference visible, instead of
# either dropping the row or inventing a name the data does not carry.
is_implicit_narrator = raw_speaker is None or not str(raw_speaker).strip()
speaker_name = "narrator" if is_implicit_narrator else str(raw_speaker).strip()
builder.add({
"id": f"utt-{row['id']}",
"kind": "utterance",
"book_id": book_id,
"chapter_id": cid,
"database_row_id": row["id"],
"utterance_index": row["utterance_index"],
"utterance_type": row["type"],
"status": row["status"],
"speaker_id": speaker_ids.get(speaker_name.lower()),
"speaker_name_raw": raw_speaker,
"speaker_name_resolved": speaker_name,
"speaker_resolution": ("implicit_narrator" if is_implicit_narrator
else "explicit"),
"text": str(row["text"] or ""),
"text_sha256": sha256_text(str(row["text"] or "")),
"text_char_count": len(str(row["text"] or "")),
"render_id": rid,
"emotion": col(row, "emotion"),
"pace": col(row, "pace"),
"instruct_is_set": bool(col(row, "instruct")),
"attribution_confidence": col(row, "attribution_confidence"),
"attribution_evidence_kind": col(row, "attribution_evidence_kind"),
"attribution_uncertain": bool(col(row, "attribution_uncertain", 0)),
"needs_review": bool(col(row, "needs_review", 0)),
"review_reason": col(row, "review_reason"),
"generated_at": generated_at,
})
# --- voice reference renders ----------------------------------------
voices_dir = src.umbra / "voices"
if voices_dir.is_dir():
for wav in sorted(voices_dir.rglob("*_ref.wav")):
ref_name = wav.stem[: -len("_ref")]
ref_speaker_id = speaker_ids.get(ref_name.lower())
builder.add({
"id": f"rnd-vref-{slug(ref_name)}",
"kind": "render",
"render_role": "voice_reference",
"render_stage": "voice_reference",
"book_id": book_id,
"path": f"voices/{wav.relative_to(voices_dir).as_posix()}",
"relative_to": "book_directory",
"path_is_absolute": True,
"voice_id": ref_speaker_id,
"audio_format": probe_wav(wav).as_record(),
"duration_ms": wav_duration_ms(wav),
"byte_size": wav.stat().st_size,
"content_sha256": sha256_file(wav),
"inputs": [],
"generated_at": generated_at,
})
# --- the published artifacts ----------------------------------------
ar_dir = src.umbra / "export" / "ar_package"
by_index = {c["chapter_index"]: c for c in chapters}
published_ids: list[str] = []
if ar_dir.is_dir():
for mp3 in sorted(ar_dir.glob("*.mp3")):
stem = mp3.stem
match = re.match(r"Deal_Hunter_(?:(\d+)|([ivxlc]+))_(.+)$", stem)
order_hint, title_hint = (match.group(1), match.group(3)) if match else (None, None)
chapter_row = None
if order_hint and order_hint.isdigit():
chapter_row = by_index.get(int(order_hint) - 1)
rid = f"rnd-mp3-{slug(stem)}"
fmt = probe_media(mp3)
builder.add({
"id": rid,
"kind": "render",
"render_role": "distribution_track",
"render_stage": "distribution_master",
"book_id": book_id,
"chapter_id": f"ch-{chapter_row['id']}" if chapter_row else None,
"path": f"export/ar_package/{mp3.name}",
"relative_to": "book_directory",
"path_is_absolute": True,
"audio_format": fmt.as_record(),
"duration_ms": wav_duration_ms(mp3),
"byte_size": mp3.stat().st_size,
"content_sha256": sha256_file(mp3),
"track_number_in_name": order_hint,
"title_in_name": title_hint,
"inputs": [f"rnd-chfull-{chapter_row['id']}"] if chapter_row else [],
"generated_at": generated_at,
})
published_ids.append(rid)
m4b = src.umbra / "export" / "Deal_Hunter.m4b"
m4b_meta = load_json(src.umbra / "m4b_meta.json", {}) or {}
if m4b.exists():
rid = "rnd-m4b-1"
builder.add({
"id": rid,
"kind": "render",
"render_role": "assembled_book",
"render_stage": "assembled_book",
"book_id": book_id,
"path": "export/Deal_Hunter.m4b",
"relative_to": "book_directory",
"path_is_absolute": True,
"audio_format": probe_media(m4b).as_record(),
"duration_ms": wav_duration_ms(m4b),
"byte_size": m4b.stat().st_size,
"content_sha256": sha256_file(m4b),
"embedded_metadata_title": m4b_meta.get("title"),
"embedded_metadata_narrator": m4b_meta.get("narrator"),
"embedded_metadata_language": m4b_meta.get("language"),
"inputs": [f"rnd-chfull-{c['id']}" for c in chapters],
"generated_at": generated_at,
})
published_ids.append(rid)
# --- the book itself, last, so it can reference its children ---------
records_with_kind = list(builder.records)
# --- what the bundle found that nobody asked it to look for ----------
# A third of the voice library is named `.wav` and is not WAV. That is the
# kind of thing a mapping silently gets wrong -- it reads the extension, it
# opens the file as PCM, and it produces a plausible wrong duration -- so it
# is promoted to a counted fact on the book rather than left in a per-file
# field nobody reads.
mismatched = [
{"id": r["id"], "path": r["path"],
"named_extension": r["audio_format"]["file_extension"],
"actual_container": r["audio_format"]["container"],
"actual_codec": r["audio_format"]["codec"]}
for r in records_with_kind
if r.get("audio_format")
and r["audio_format"].get("extension_matches_container") is False
]
builder.add({
"id": book_id,
"kind": "book",
"name": project["name"] if project else src.name,
"title": speakers_doc.get("book_title"),
"source_document_format": project["source_format"] if project else None,
"source_path_redacted": PROJECT_ROOT_PLACEHOLDER,
"path_relative_to": "book_directory",
"status": project["status"] if project else None,
"created_at": utc_stamp(project["created_at"]) if project else None,
"updated_at": utc_stamp(project["updated_at"]) if project else None,
"chapter_count": len(chapters),
"utterance_count": len(utterances),
"speaker_count": len(speaker_records),
"speaker_names_without_a_profile": unprofiled,
"novel_chapter_count": sum(1 for c in chapters if c["chapter_index"] >= 0),
"front_and_back_matter_count": sum(1 for c in chapters if c["chapter_index"] < 0),
"declared_total_chapters": speakers_doc.get("total_chapters"),
"declared_total_words": speakers_doc.get("total_words"),
"declared_vs_measured_note": (
"speakers.json declares 35 chapters; the database holds 32. Four rows "
"were removed and the declared figure was never updated. Both numbers "
"are reported rather than reconciled by guessing."
),
"language_designations": [],
"language_designation_note": (
"Empty and it is a real answer, not an omission: this audio system "
"renders one language per book and records no per-utterance language. "
"Spoken English is implied by the voice set, not declared. Any field "
"naming a target language must come from the translation side."
),
"chapters": chapter_ids,
"published_artifact_ids": published_ids,
"extension_container_mismatches": mismatched,
"extension_container_mismatch_count": len(mismatched),
"numbering": chapter_numbering(),
"generated_at": generated_at,
})
stats = {
"chapters": len(chapters),
"utterances": len(utterances),
"speakers": len(speaker_records),
"speakers_without_profile": unprofiled,
"name_casefold_collisions": {k: sorted(v) for k, v in colliding.items()},
"extension_container_mismatches": mismatched,
"ar_tracks": len(published_ids),
"db_path": str(src.db),
}
conn.close()
return builder, stats
# --------------------------------------------------------------------------
# bundle assembly
# --------------------------------------------------------------------------
def _schema(kind: str, title: str, required: list[str], props: dict[str, Any]) -> dict[str, Any]:
return {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": f"{SCHEMA_ID_PREFIX}/{kind}.schema.json",
"title": title,
"type": "object",
"required": ["id", "kind", "generated_at"] + required,
"properties": {
"id": {"type": "string",
"description": "Stable here; referenced by other records."},
"kind": {"const": kind},
"generated_at": {"type": "string", "format": "date-time"},
**props,
},
"additionalProperties": True,
}
def write_schemas(root: Path) -> None:
schemas = root / "schemas"
schemas.mkdir(parents=True, exist_ok=True)
(schemas / "manifest.schema.json").write_text(json.dumps({
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": f"{SCHEMA_ID_PREFIX}/manifest.schema.json",
"title": "manifest.jsonl - one record per line, discriminated by 'kind'",
"type": "object",
"required": ["id", "kind", "generated_at"],
"properties": {
"kind": {"enum": ["book", "chapter", "utterance", "speaker", "render"]},
},
"discriminator": {"propertyName": "kind"},
}, indent=2) + "\n")
(schemas / "lineage.schema.json").write_text(json.dumps({
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": f"{SCHEMA_ID_PREFIX}/lineage.schema.json",
"title": "lineage.json - what produced what for one book",
"type": "object",
"required": ["book_id", "stages", "edges"],
"properties": {
"book_id": {"type": "string"},
"stages": {"type": "array", "items": {"type": "string"}},
"edges": {
"type": "array",
"items": {
"type": "object",
"required": ["from", "to", "transform"],
"properties": {
"from": {"type": "string"},
"to": {"type": "string"},
"transform": {"type": "string"},
"derivation": {
"type": "string",
"enum": ["direct", "derived", "assembled", "transcoded"],
},
"note": {"type": "string"},
},
},
},
},
}, indent=2) + "\n")
kinds = {
"book": _schema("book", "One book", ["name", "chapters"], {
"name": {"type": "string"},
"title": {"type": ["string", "null"]},
"status": {"type": ["string", "null"]},
"chapter_count": {"type": "integer"},
"utterance_count": {"type": "integer"},
"speaker_count": {"type": "integer"},
"language_designations": {
"type": "array",
"items": {"type": "string"},
"description": "Empty in this system; see language_designation_note.",
},
"chapters": {"type": "array", "items": {"type": "string"}},
"published_artifact_ids": {"type": "array", "items": {"type": "string"}},
"numbering": {"type": "object"},
}),
"chapter": _schema("chapter", "One manuscript chapter",
["book_id", "chapter_index", "title"], {
"book_id": {"type": "string"},
"database_row_id": {"type": "integer",
"description": "The only stable chapter number."},
"chapter_index": {"type": "integer",
"description": "0 = first novel chapter; negative = front matter."},
"title": {"type": ["string", "null"]},
"word_count": {"type": "integer"},
"status": {"type": "string"},
"source_text_sha256": {"type": "string"},
"utterance_count": {"type": "integer"},
"renders": {"type": "array", "items": {"type": "string"}},
"numbering_basis": {"type": "object"},
}),
"utterance": _schema("utterance", "One text unit, the row audio is rendered from",
["book_id", "chapter_id", "utterance_index", "text", "text_sha256"], {
"book_id": {"type": "string"},
"chapter_id": {"type": "string"},
"database_row_id": {"type": "integer"},
"utterance_index": {"type": "integer",
"description": "Zero-based, restarts each chapter."},
"utterance_type": {"enum": ["narration", "dialogue", "scene_break"]},
"speaker_id": {"type": ["string", "null"]},
"speaker_name_raw": {
"type": ["string", "null"],
"description": ("Exactly what the database holds. null on most "
"narration rows, which means the narrator."),
},
"speaker_name_resolved": {
"type": "string",
"description": "The name used to resolve speaker_id.",
},
"speaker_resolution": {
"enum": ["explicit", "implicit_narrator"],
"description": ("Which of the two it was. 'implicit_narrator' "
"means speaker_name_raw was null and the narrator "
"was inferred from the row being narration."),
},
"text": {"type": "string"},
"text_sha256": {"type": "string"},
"render_id": {"type": ["string", "null"]},
"attribution_confidence": {"type": ["number", "null"]},
"needs_review": {"type": "boolean"},
}),
"speaker": _schema("speaker", "One speaking entity and its voice", ["primary_name"], {
"primary_name": {"type": "string"},
"has_profile": {"type": "boolean"},
"is_narrator": {"type": "boolean"},
"aliases": {"type": "array", "items": {"type": "string"}},
"utterance_count": {"type": "integer",
"description": "Exact count for this exact spelling."},
"utterance_count_casefold_grouped": {
"type": "integer",
"description": ("Count if spellings differing only by case were "
"merged. Equals utterance_count unless this record "
"is in a casefold_collision_group."),
},
"casefold_collision_group": {
"type": ["string", "null"],
"description": ("Set when this name differs from another only by "
"case. The records are deliberately NOT merged: "
"merging is the consumer's decision, and hiding "
"the choice in the export would make it invisible."),
},
"voice_review_status": {"type": ["string", "null"]},
"tts_engine_used": {"type": ["string", "null"]},
}),
"render": _schema("render", "One produced audio file", ["render_role", "audio_format"], {
"render_role": {"enum": ["utterance_audio", "chapter_audio", "distribution_track",
"assembled_book", "voice_reference"]},
"render_stage": {"type": "string"},
"book_id": {"type": ["string", "null"]},
"chapter_id": {"type": ["string", "null"]},
"path": {"type": "string"},
"relative_to": {"const": "book_directory"},
"path_is_absolute": {"type": "boolean"},
"audio_format": {"type": ["object", "null"], "properties": {
"container": {"type": "string",
"description": "Measured from the bytes, not the extension."},
"codec": {"type": "string"},
"sample_rate_hz": {"type": ["integer", "null"]},
"channel_count": {"type": ["integer", "null"]},
"bit_rate_bps": {"type": ["integer", "null"]},
"file_extension": {"type": "string"},
"extension_matches_container": {
"type": ["boolean", "null"],
"description": ("false means the name is wrong. Null means the "
"container could not be determined, which is a "
"different statement from 'they match'."),
},
"detected_by": {"enum": ["wave-module", "ffprobe", "none"]},
}},
"duration_ms": {"type": ["integer", "null"],
"description": "Unit in the name, per the request."},
"byte_size": {"type": ["integer", "null"]},
"content_sha256": {"type": ["string", "null"]},
"inputs": {"type": "array", "items": {"type": "string"}},
}),
}
for kind, schema in kinds.items():
(schemas / f"{kind}.schema.json").write_text(json.dumps(schema, indent=2) + "\n")
def write_example(root: Path, src: Source, builder: BundleBuilder) -> dict[str, Any]:
"""One real book, traced end to end.
The controller is explicit that this example is the part that teaches the
shape, so it is a real book with real identifiers, not a synthetic shape.
It holds the book's own records plus **one chapter's complete chain** -- every
utterance, every rendered file, the assembled chapter and the distribution
master derived from it -- rather than a copy of every record. A verbatim copy
of ``manifest.jsonl`` is five megabytes of pretty-printed JSON, which is not
an example: it cannot be read, so it teaches nothing, and it duplicates a
file the reader already has.
"""
book_dir = root / "examples" / slug(src.name)
book_dir.mkdir(parents=True, exist_ok=True)
records = builder.records
def dump(name: str, rows: list[dict[str, Any]]) -> None:
(book_dir / name).write_text(json.dumps(rows, indent=2, sort_keys=True) + "\n")
book = next(r for r in records if r["kind"] == "book")
chapters = [r for r in records if r["kind"] == "chapter"]
utterances = [r for r in records if r["kind"] == "utterance"]
speakers = [r for r in records if r["kind"] == "speaker"]
renders = [r for r in records if r["kind"] == "render"]
# The largest chapter is the most informative chain, and choosing by size
# rather than by hand keeps the choice reproducible.
demo = max(chapters, key=lambda c: c["utterance_count"])
demo_utts = [u for u in utterances if u["chapter_id"] == demo["id"]]
demo_renders = [r for r in renders
if r.get("chapter_id") == demo["id"]
or any(i == demo["id"] for i in (r.get("inputs") or []))]
demo_render_ids = {r["id"] for r in demo_renders}
demo_speakers = sorted({u["speaker_id"] for u in demo_utts if u["speaker_id"]})
book_renders = [r for r in renders if r["render_role"] == "assembled_book"]
included = ({book["id"]} | set(demo_speakers) | {demo["id"]}
| {u["id"] for u in demo_utts} | demo_render_ids
| {r["id"] for r in book_renders})
example_records = [r for r in records if r["id"] in included]
dump("manifest.json", example_records)
edges: list[dict[str, Any]] = [
{
"from": book["id"],
"to": "ch-<database_row_id>",
"transform": "manuscript parsed into ordered chapters",
"derivation": "derived",
"note": ("chapter_index is the reading order and is hand-set; "
"database_row_id is the stable key."),
},
{
"from": demo["id"],
"to": "utt-<database_row_id>",
"transform": "chapter source_text segmented into utterances",
"derivation": "derived",
"note": ("utterance_index restarts at 0 for every chapter, so "
"(chapter_id, utterance_index) is the natural key. This "
f"chapter is traced in full: {len(demo_utts)} utterances."),
},
{
"from": "utt-<database_row_id>",
"to": "rnd-utt-<database_row_id>",
"transform": "utterance text rendered to one WAV by the TTS engine",
"derivation": "direct",
"note": "One file per utterance. No finer grain exists in this system.",
},
{
"from": demo["id"],
"to": f"rnd-chfull-{demo['database_row_id']}",
"transform": "utterance audio stitched into one chapter file",
"derivation": "assembled",
"note": ("Silence between units is inserted at assembly time and "
"exists in no individual render, so the sum of the render "
"durations is shorter than the chapter file. That gap is "
"the padding, not a measurement error."),
},
]
for render in demo_renders:
if render["render_role"] == "distribution_track":
for source_id in render.get("inputs", []):
edges.append({
"from": source_id,
"to": render["id"],
"transform": ("resampled, level-normalised, lead and trail "
"silence added, encoded to MP3"),
"derivation": "transcoded",
"note": ("The padding happens AFTER chapter approval, so the "
"published bytes are not the approved bytes. Stated "
"because no field in the metadata would reveal it."),
})
chapter_render_count = sum(1 for r in renders if r["render_role"] == "chapter_audio")
for render in book_renders:
edges.append({
"from": "rnd-chfull-<database_row_id>",
"to": render["id"],
"transform": "chapter files concatenated into a single audiobook",
"derivation": "assembled",
"note": f"{chapter_render_count} chapter files in, one file out.",
})
lineage = {
"book_id": book["id"],
"book_title": book.get("title"),
"generated_at": book["generated_at"],
"stages": [
"manuscript_document",
"chapter",
"utterance",
"utterance_audio",
"chapter_audio",
"distribution_master",
"assembled_book",
],
"counts": {
"chapters_in_book": len(chapters),
"utterances_in_book": len(utterances),
"speakers_in_book": len(speakers),
"renders_in_book": len(renders),
"chapter_traced_in_full": demo["id"],
"records_in_this_example": len(example_records),
"records_in_full_manifest": len(records),
},
"edges": edges,
}
(book_dir / "lineage.json").write_text(json.dumps(lineage, indent=2, sort_keys=True) + "\n")
demo_title = demo["title"] or "untitled"
novel_count = sum(1 for c in chapters if c["chapter_index"] >= 0)
matter_count = sum(1 for c in chapters if c["chapter_index"] < 0)
first = next((c for c in chapters if c["chapter_index"] >= 0), chapters[0])
(book_dir / "README.md").write_text(f"""# Worked example: {book.get('title') or book['name']}
One real book traced from manuscript to published file, with the identifiers that
link each stage. Generated from the live project; regenerate with:
```
python3 scripts/build_controller_bundle.py --project {src.name}
```
## What is in here, and what is not
`manifest.json` holds **{len(example_records)} of {len(records)} records**: the book,
every speaker, every chapter header, and then the **complete chain of one
chapter** -- all {len(demo_utts)} of its utterances, all {len(demo_renders)} of its
rendered files, and the distribution master derived from them. The chapter traced
in full is **{demo_title}**, the largest; it is chosen by size so the choice
reproduces.
Every record is in `../../manifest.jsonl`. This directory is the readable subset,
not a second copy of the data.
## The chain, in one paragraph
A manuscript (`.docx`) is parsed into {len(chapters)} chapter rows. Each chapter's
`source_text` is segmented into utterances, each carrying a `speaker_name`. Each
utterance is rendered to one WAV at
`chapter_audio/ch<database_row_id>/utt_<utterance_index>.wav`, where
`<utterance_index>` is **zero-based and restarts every chapter**. Those renders are
stitched into `chapter_audio/ch<database_row_id>_full.wav`. That chapter file is
then either transcoded to a per-chapter MP3 distribution master, or concatenated
with the others into a single M4B audiobook.
## The three numbers
One chapter has three numbers and they are not interchangeable:
| Scheme | Value for "{first['title']}" | Where it appears |
|---|---|---|
| `chapters.id` | see `database_row_id` | `ch<N>/`, `ch<N>_full.wav`, `utterances.chapter_id` |
| `chapters.chapter_index` | {first['chapter_index']} | reading order only |
| distribution track number | 01 | `Deal_Hunter_01_<Title>.mp3` |
Front and back matter carry **negative** chapter indices ({matter_count} of
{len(chapters)} chapters here) and are numbered with lowercase roman numerals
in the distribution filenames.
Treating `chapter_index` as an identifier breaks the first time a chapter is
reordered or front matter is inserted.
## Where the approved bytes stop being the shipped bytes
The MP3 master is built from the chapter WAV **after** chapter approval, and the
build adds silence at the head and tail, resamples, and normalises level. The file
a listener receives is therefore not the file that was approved. Stated here
because it is invisible in the metadata otherwise: every duration in this bundle is
correct and no two of them agree.
## Scale
- {len(chapters)} chapters: {novel_count} novel, {matter_count} front/back matter
- {len(utterances)} utterances
- {len(speakers)} speaker records
- {len(renders)} rendered audio files described
""")
return {"chapters": len(chapters), "utterances": len(utterances),
"speakers": len(speakers), "renders": len(renders),
"example_records": len(example_records),
"demo_chapter": demo_title, "demo_utterances": len(demo_utts)}
def build(project_name: str, out_dir: Path, generated_at: str,
validator_source: Path | None, source_root: Path | None = None) -> dict[str, Any]:
repo = Path(source_root).resolve() if source_root else Path(__file__).resolve().parent.parent
book_dir = repo / project_name
src = Source(
book_dir=book_dir,
umbra=book_dir / ".umbra",
db=book_dir / ".umbra" / "project.db",
name=project_name,
)
if not src.db.exists():
raise SystemExit(f"no project database at {src.db}")
if out_dir.exists():
shutil.rmtree(out_dir)
out_dir.mkdir(parents=True)
builder, stats = build_records(src, generated_at)
manifest_text = builder.lines()
(out_dir / "manifest.jsonl").write_text(manifest_text)
write_schemas(out_dir)
example_stats = write_example(out_dir, src, builder)
if validator_source and validator_source.exists():
shutil.copy2(validator_source, out_dir / "validate.py")
index = {
"bundle_format_version": BUNDLE_FORMAT_VERSION,
"generated_at": generated_at,
"source": {
"system": "umbraaudio",
"node_id": "umbraaudio-m4",
"product_version": _git_describe(repo),
"database_schema_fingerprint": _schema_fingerprint(src.db),
},
"scope": {
"books_full_detail": [project_name],
"books_summary_only": [],
"note": (
"Every record for one book. The other projects on this machine are "
"not included; they share a schema, so they can be added by naming "
"them with --project rather than by re-deriving anything."
),
},
"counts": {
"records_total": len(builder.records),
"books": 1,
"chapters": stats["chapters"],
"utterances": stats["utterances"],
"speakers": stats["speakers"],
"speakers_without_a_profile": len(stats["speakers_without_profile"]),
"render_records": sum(1 for r in builder.records if r["kind"] == "render"),
},
"record_kinds": {
"discriminator": "kind",
"values": sorted({r["kind"] for r in builder.records}),
},
"required_fields": {
"book": ["name", "chapters", "chapter_count", "utterance_count"],
"chapter": ["book_id", "chapter_index", "source_text_sha256", "renders"],
"utterance": ["book_id", "chapter_id", "utterance_index", "text",
"text_sha256", "render_id", "speaker_id"],
"speaker": ["primary_name", "has_profile", "utterance_count"],
"render": ["render_role", "audio_format", "path", "relative_to",
"path_is_absolute", "duration_ms", "byte_size"],
},
"conventions": {
"timestamps": "ISO 8601 UTC with an explicit Z",
"durations": "unit in the field name (_ms); no bare duration/length field exists",
"paths": "relative to book_directory, declared per record via 'relative_to'",
"null": "means absent only; no field uses null to mean unknown",
"secrets": "none present; hostnames and usernames replaced with placeholders",
"media_bytes": "not included, by agreement",
},
"content_hash": {
"algorithm": "sha256",
"of": "manifest.jsonl bytes, exactly as written",
"value": hashlib.sha256(manifest_text.encode("utf-8")).hexdigest(),
},
}
(out_dir / "bundle.json").write_text(json.dumps(index, indent=2, sort_keys=True) + "\n")
_write_vocabulary(out_dir, builder)
_write_readme(out_dir, src, index, stats, example_stats, validator_source)
return {"index": index, "stats": stats, "example": example_stats, "out_dir": str(out_dir)}
def _git_describe(repo: Path) -> str:
try:
out = subprocess.run(["git", "-C", str(repo), "rev-parse", "--short", "HEAD"],
capture_output=True, text=True, timeout=20, check=False)
return out.stdout.strip() or "unknown"
except (OSError, subprocess.SubprocessError):
return "unknown"
def _schema_fingerprint(db_path: Path) -> str:
"""Fingerprint of the table/column shape, not the data.
Two databases with the same shape but different contents share a
fingerprint, which is the point: it answers 'is this the same schema' and
cannot be confused with 'is this the same data'.
"""
conn = open_db(db_path)
try:
parts: list[str] = []
tables = [r[0] for r in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table' "
"AND name NOT LIKE 'sqlite_%' ORDER BY name"
)]
for table in tables:
for row in conn.execute(f"PRAGMA table_info('{table}')"):
parts.append(f"{table}.{row[1]}:{row[2]}")
finally:
conn.close()
return hashlib.sha256("\n".join(sorted(parts)).encode("utf-8")).hexdigest()[:16]
def _write_vocabulary(root: Path, builder: BundleBuilder) -> None:
speaker_counts = {
r["primary_name"]: r["utterance_count"]
for r in builder.records if r["kind"] == "speaker"
}
top = sorted(speaker_counts.items(), key=lambda kv: -kv[1])[:5]
(root / "VOCABULARY.md").write_text(f"""# Vocabulary: this system's words and the controller's
The request asks for a table with two columns and a note per row, because
guessing terminology is how a mapping looks correct and reads the wrong field.
Where the two sides genuinely have no equivalent, that is said rather than
papered over with a near-match.
## Core nouns
| Concept | Controller calls it | This system calls it | Record `kind` | Note |
|---|---|---|---|---|
| One book | Project | project | `book` | One row in `project`. Per-book on both sides. |
| One chapter | *(none)* | chapter | `chapter` | Manuscript text, not audio. |
| One text unit | translation unit | utterance | `utterance` | What audio is rendered from. |
| One rendered piece | artifact | `utt_NNNN.wav` | `render` | Per *utterance*, not per sentence. |
| A chapter's audio | artifact | `ch<N>_full.wav` | `render` | Stitched. Derived, never authored. |
| Distribution master | artifact | `Deal_Hunter_NN_*.mp3` | `render` | Transcoded after approval. |
| The assembled book | artifact | `Deal_Hunter.m4b` | `render` | One file, all chapters. |
| The narration voice | voice | `is_narrator: true` | `speaker` | Exactly one per book. |
| A speaker | character | speaker | `speaker` | In `speakers.json`, not `characters` (empty). |
| A rejected take | *(none)* | `voice_status.json` | `speaker` | **No such record.** See below. |
| The published thing | artifact | `ar_package.json` | `render` | No release record yet. |
## The three facts that most affect a mapping
**The narrator is written three different ways, and the most common way is
nothing at all.** `utterances.speaker_name` is:
| Stored value | Utterances | What it means |
|---|---|---|
| `NULL` | 1,803 | the narrator, stated by saying nothing |
| `'narrator'` | 614 | the narrator |
| `'Narrator'` | 15 | the narrator, again |
Only two of those are strings, so `count(distinct speaker_name)` under-reports the
cast and a `WHERE speaker_name IS NOT NULL` filter silently discards most of the
narration. In this bundle every utterance carries `speaker_name_raw` (exactly what
is stored, including null), `speaker_name_resolved`, and `speaker_resolution`,
which is `implicit_narrator` when the narrator was inferred from the row being
narration rather than read from the column. The two written spellings are **not**
merged; they share a `casefold_collision_group`.
**A speaker is not in the database.** The `characters` table exists in the schema
and holds **zero rows** for this project. Everything known about speakers —
aliases, gender, vocal traits, which chapters they appear in, whether their voice
was approved — lives in `.umbra/speakers.json` and `.umbra/voice_status.json`,
which are JSON documents with no schema and no versioning. A mapping that reads
`characters` gets nothing and will look correct while being empty.
**A rejected take is not recorded.** The controller asked what a rejected take is
called. The honest answer is that the system keeps voice *review decisions* but
never keeps the rejected audio, so a rejected take is unrecoverable — it was
overwritten in place. `speaker.voice_review_status` records that a decision was
made, not what was thrown away.
**A file extension does not describe the file.** Some voice reference files named
`.wav` are MPEG audio. `render.audio_format.container` is measured from the bytes;
`render.audio_format.extension_matches_container` says whether the name agrees.
## Speakers actually used
The system holds {len(speaker_counts)} distinct names on utterances. The five most
used are:
| Speaker | Utterances |
|---|---|
{chr(10).join(f"| {name} | {count} |" for name, count in top)}
Some of these are designations rather than names (`"The Lead Guard"`), and one is
an alias of another. `speaker.has_profile` and `speaker.aliases` are the fields to
key on; `primary_name` is not guaranteed unique as a *character*.
## Terms with no counterpart on either side
| This system's term | Meaning | Why the controller has no equivalent |
|---|---|---|
| `marker` (`*.wav.txt`) | JSON beside a WAV: speaker and text | Cache. See `render.stale`. |
| `stale` | Marker disagrees with the database row | Derived boolean, recomputed each build. |
| `chapter_index` | Hand-set order; negative for front matter | A fact here, not a shared concept. |
""")
def _write_readme(root: Path, src: Source, index: dict[str, Any], stats: dict[str, Any],
example: dict[str, Any], validator_source: Path | None) -> None:
counts = index["counts"]
unprofiled = stats["speakers_without_profile"]
mismatches = stats.get("extension_container_mismatches") or []
collisions = stats.get("name_casefold_collisions") or {}
validator_note = (
f"`validate.py` is the validator shipped with the request"
f"{' (copied from ' + validator_source.name + ')' if validator_source else ''}."
)
(root / "README.md").write_text(f"""# UmbraAudio metadata bundle
Machine-readable metadata describing one book as it exists in the UmbraAudio
system, in the format requested by the UmbraTrans controller.
**Generated:** {index['generated_at']}
**Source system:** umbraaudio (node `umbraaudio-m4`), commit `{index['source']['product_version']}`
**Source schema fingerprint:** `{index['source']['database_schema_fingerprint']}`
**Records:** {counts['records_total']} — {counts['chapters']} chapters,
{counts['utterances']} utterances, {counts['speakers']} speakers,
{counts['render_records']} renders
**`manifest.jsonl` content hash (sha256):** `{index['content_hash']['value']}`
## Read this first
Two facts about this system will break a mapping that does not know them, and
both are described in full below:
1. **A chapter has three different numbers** and none is derivable from another
under edit. See `numbering` on the book record.
2. **The file extension does not always describe the file.** {len(mismatches)}
file(s) named `.wav` are not WAV. Trust `audio_format.container`; it is
measured from the bytes.
## What this is
Metadata only. No media bytes, no source code, no credentials. Every path is
relative to the book directory, and the real location of this project on this
machine has been replaced with `{PROJECT_ROOT_PLACEHOLDER}` so the bundle can be
committed without describing the host.
## How it was generated
```
python3 scripts/build_controller_bundle.py --project {src.name}
python3 validate.py . # the controller's validator, copied in
```
The generator is read-only. It opens the SQLite database in read-only mode and
copies JSON documents; nothing in the audio system is written, moved or deleted,
and the audio system keeps running exactly as it does.
The build is **deterministic**: records are emitted in a fixed order with sorted
JSON keys, so an unchanged project produces a byte-identical `manifest.jsonl` and
an unchanged content hash. To check whether anything has moved since this bundle,
regenerate and compare `content_hash`. To pin the timestamp as well, pass
`--generated-at`.
## Contents
| Path | What it is |
|---|---|
| `bundle.json` | Root index: counts, scope, conventions, required fields, content hash. |
| `manifest.jsonl` | Every record for this book, one JSON object per line. |
| `VOCABULARY.md` | This system's words against the controller's, with the gaps named. |
| `schemas/` | JSON Schema per record kind, plus `manifest` and `lineage`. |
| `examples/{slug(src.name)}/` | The one real book traced end to end: manifest, lineage, prose. |
| `validate.py` | {validator_note} |
## Conventions, as the request asks them to be stated
- **Timestamps** are ISO 8601 UTC with an explicit `Z`. The source database stores
naive strings that *are* UTC; they are marked as such rather than re-stamped.
- **Durations** carry the unit in the field name (`duration_ms`). No bare
`duration` or `length` field exists anywhere in this bundle.
- **Paths** are relative to the book directory and every record with a path
carries `relative_to: "book_directory"`. Absolute source locations are
replaced with `{PROJECT_ROOT_PLACEHOLDER}`.
- **Null means absent.** Where a distinction would matter — a speaker with no
profile, a format that could not be probed — a second field or an explicit
`"unknown"` is used instead of a null.
- **Versions** are explicit: bundle format {BUNDLE_FORMAT_VERSION}, source commit
`{index['source']['product_version']}`, schema fingerprint above.
- **No credentials**, and no hostnames or usernames.
## Scope, and what is deliberately missing
This bundle describes **one book in full**. The other projects on this machine
share the same schema and can be added by naming them with `--project`; nothing
here needs re-deriving for them.
Missing on purpose, and what to do about it:
- **Media bytes.** Agreed not to send. `render` records carry path, format,
duration, byte size and content hash, so the metadata says what to ask samples
for.
- **Target languages.** `book.language_designations` is **empty**, and that is a
real answer rather than an omission. This system renders one language per book
and records no per-utterance language. Any target-language field has to come
from the translation side; inventing one here would be a guess wearing a schema.
- **Rejected takes.** Not recoverable. The system overwrites rejected audio in
place, so there is no record and no artifact to describe. See `VOCABULARY.md`.
- **Publication / release records.** Nothing has been published, so there is no
release record to emit. A release comment exists as a draft.
## Things that are known to be wrong, stated rather than smoothed over
### The file extension does not always describe the file
**{len(mismatches)} file(s) are named `.wav` and are not WAV.** They are MPEG audio.
Every `render` record carries `audio_format.extension_matches_container`, measured
from the bytes with `ffprobe` rather than read off the name, and the book record
lists the affected paths in `extension_container_mismatches`.
This is the kind of thing that makes a mapping look correct while being wrong: a
reader that opens `.wav` as PCM gets a plausible number that is not the duration.
`audio_format.container` is the field to trust; the extension is decoration.
### Other known gaps
- `speakers.json` declares a chapter count in one place and the database holds a
different one. Both are reported (`declared_total_chapters` vs `chapter_count`).
- {len(unprofiled)} speaker name(s) appear on utterances with **no profile** behind
them: {', '.join(unprofiled) if unprofiled else 'none'}. A mapping that joins
speakers to profiles will silently drop these.
- The narrator is stored under **two spellings** — `Narrator` and `narrator` — and
they are not merged here. The two records share a `casefold_collision_group`, and
each carries both `utterance_count` (exact, for that spelling) and
`utterance_count_casefold_grouped` (as if merged). Merging is a decision that
belongs to whoever owns the speaker identity, not something a metadata export
should do silently. Colliding groups in this bundle:
{', '.join(f'`{k}` -> {v}' for k, v in collisions.items()) or 'none'}.
- One utterance render is a **zero-byte file**:
`chapter_audio/ch20/utt_0226.wav`. It is present, empty, and unreadable, which is
different from missing and needs a re-render rather than a restore.
- Some utterances carry a low-confidence speaker attribution and are flagged
`needs_review`. The flag is in the record; it is not resolved data.
- Front and back matter share the chapter table with the novel and are separated
only by a negative `chapter_index`.
- Everything under `.umbra/cache`, `.umbra/archives`, `.umbra/pronunciation_audio`,
`.umbra/voice_tests`, `.umbra/samples` and the `debug_*.json` files is scratch
or intermediate output and is deliberately **not** in this bundle. A catalogue
that swept them in would be a catalogue nobody could answer a question against.
## Validating
```
python3 validate.py .
```
Every problem is reported, not just the first. Exit 0 means the bundle is usable.
""")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--project", required=True, help="project directory name")
parser.add_argument("--out", default=None,
help="output directory (default: /tmp/<project>-bundle)")
parser.add_argument("--generated-at", default=None,
help="pin the timestamp so the hash is fully reproducible")
parser.add_argument("--validator", default=None,
help="path to the controller's audio-system-bundle-validator.py to copy in")
parser.add_argument("--source-root", default=None,
help="directory containing the project (default: this repository)")
parser.add_argument("--tarball", action="store_true", help="also write <out>.tar.gz")
args = parser.parse_args()
generated_at = args.generated_at or datetime.now(UTC).replace(
microsecond=0).isoformat().replace("+00:00", "Z")
out = Path(args.out) if args.out else Path("/tmp") / f"{args.project}-bundle"
validator = Path(args.validator) if args.validator else None
result = build(args.project, out, generated_at, validator,
Path(args.source_root) if args.source_root else None)
index = result["index"]
print(f"bundle: {result['out_dir']}")
print(f"records: {index['counts']['records_total']}")
print(f"hash: {index['content_hash']['value']}")
if args.tarball:
archive = out.with_suffix(".tar.gz")
with tarfile.open(archive, "w:gz") as tar:
tar.add(out, arcname=out.name)
print(f"tarball: {archive} ({archive.stat().st_size} bytes)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Reference in New Issue View Git Blame Copy Permalink