Answer the controller's data request, and audit what we send
The UmbraTrans controller asked (commit 70c23fd) for machine-readable metadata about the audio system, in a specified layout, with a validator that ships with the request so the sender can check its own work. That request is answered here. scripts/build_controller_bundle.py emits that bundle: one real book (Deal Hunter), 8,884 records, no media, no credentials, no hostnames. It passes the controller's own validator, exit 0. Three things in this system will break a mapping that does not know them, and the bundle's real purpose is to say so precisely rather than to look tidy: * A chapter has THREE numbers. The database row id is what filenames use, chapter_index is reading order, and the distribution track number is a third thing again. Only the row id survives an edit. Front and back matter are numbered with roman numerals in the MP3 filenames. * The narrator is written three ways, and the most common way is nothing at all. 1,803 utterances have a NULL speaker_name, meaning narrator; 614 say "narrator" and 15 say "Narrator". A join that skips nulls loses most of the narration silently. Utterances now carry speaker_name_raw, the resolved name, and speaker_resolution, and the two spellings are not merged - that is a decision for whoever owns speaker identity, not something an export should do quietly. * A .wav file is not always a WAV. 30 of the 91 voice reference files are MPEG. Every render reports the container measured from the bytes, with extension_matches_container, so a reader cannot be misled by the name. A validator proves the bundle is internally consistent; it cannot prove the bundle is true. scripts/audit_controller_bundle.py re-measures every hash, byte size, duration and count against the project on disk - 66,011 claims, all holding. It caught two real defects in the first draft of this bundle, including one of my own: produced_at was stamped with the build time, which is a claim about the audio that nothing supported. The validator was also run against seven deliberately broken bundles to confirm it can fail. A validator nobody has seen fail is not evidence of anything; those cases are now tests. scripts/audio_controller_connect.py runs the T01-T10 suite: 4 pass, 1 fails, 5 are blocked. T04 fails honestly - there is no ingest endpoint, and nothing was pushed. T05-T08 are blocked because the controller's own plan sequences ingest after the metadata arrives; a proposed contract is included rather than an invented endpoint. T02 is blocked: no node key or service token is configured here, and an operator has to supply one. Two corrections to the node profile, which the controller reads: it claimed the web UI has no authentication (a session gate was added 2026-10-09) and that the controller host is offline (it is reachable and serving umbratrans-web 0.1.0). Full suite 751 passed, 8 failed - the same pre-existing pronunciation and accent failures, unchanged. ruff and mypy clean on everything added.
This commit is contained in:
B.D. Murphy
@@ -36,17 +36,17 @@
|
||||
"stack": "FastAPI + HTMX",
|
||||
"public_name": "umbraaudio.bdmurphy.me",
|
||||
"exposure": "Caddy reverse proxy on ANOTHER LAN machine -> this port",
|
||||
"auth": "NONE (unsigned active_project cookie). Do not expose beyond the LAN.",
|
||||
"autostart": "launchd LaunchAgent com.umbraaudio.web (login-scanned, KeepAlive)"
|
||||
"auth": "HTTP Basic session auth (umbraaudio/auth.py, PBKDF2-HMAC-SHA256) enforced by middleware in umbraaudio/web/__init__.py, fail-closed. added 2026-10-09.",
|
||||
"autostart": "launchd LaunchAgent com.umbraaudio.web at /Library/LaunchAgents (system-wide, KeepAlive)"
|
||||
},
|
||||
"tts_local": {
|
||||
"url": "http://localhost:8881",
|
||||
"engine": "Qwen3-TTS (transformers, MPS)",
|
||||
"autostart": "NONE - manual nohup .venv/bin/python scripts/qwen_tts_server.py"
|
||||
"autostart": "launchd LaunchAgent com.umbraaudio.qwentts (KeepAlive). LM Studio is MANUAL by design: an LLM model and the TTS model cannot be co-resident in 16 GB."
|
||||
},
|
||||
"tts_relay_listener": {
|
||||
"url": "http://127.0.0.1:18881",
|
||||
"note": "root LaunchDaemon on this host forwards to a LAN GPU box; has no upstream fallback"
|
||||
"status": "RETIRED 2026-10-08. The GPU box 192.168.123.14 was taken offline and the root relay removed; port 18881 is closed. Local :8881 is the only TTS backend.",
|
||||
"note": "Do not re-add qwen_tts_helper_url or :18881 speculatively; re-add a TTS backend only when the controller exposes one."
|
||||
},
|
||||
"llm_local": {
|
||||
"url": "http://localhost:1234/v1",
|
||||
@@ -160,9 +160,9 @@
|
||||
"has_shared_postgres_client": false,
|
||||
"has_heartbeat_or_lease": false,
|
||||
"controller_stage_reached": "0 verified 2026-10-03 (health 200, heartbeat 401, no-Host 400 - all as documented)",
|
||||
"test_suite": "pytest, 484 passing at 2026-10-03 (8 pre-existing/parallel failures)",
|
||||
"test_suite": "pytest, 751 passing at 2026-10-14 (8 pre-existing failures: 2 test_pronunciation helper_url mock, 6 test_voice_profile_manager accent)",
|
||||
"open_defects_blocking_exposure": [
|
||||
"no authentication on any endpoint (spec 20 J.6)",
|
||||
"no authentication on any endpoint (spec 20 J.6) -- RESOLVED 2026-10-09, auth gate added in umbraaudio/web/__init__.py",
|
||||
"download path traversal via str.startswith (routes.py ~5049)",
|
||||
"upload filename not sanitised with Path(...).name",
|
||||
"no CSRF, no security headers, no TrustedHost"
|
||||
|
||||
564
scripts/audio_controller_connect.py
Normal file
564
scripts/audio_controller_connect.py
Normal file
@@ -0,0 +1,564 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Test this audio system's readiness to talk to the UmbraTrans controller.
|
||||
|
||||
Writes two artefacts and prints a summary:
|
||||
|
||||
* ``audio_controller_test_report.json`` -- the machine-readable result
|
||||
* ``audio_controller_sync.log`` -- one JSON object per line, per step
|
||||
|
||||
Every request is logged with its endpoint, status and duration, and **no secret
|
||||
is ever logged** -- not in the report, not in the log, not in an error message.
|
||||
Header values are reduced to a presence flag before they reach a log line.
|
||||
|
||||
What this deliberately does not do: invent an endpoint. The brief anticipated
|
||||
``/api/audio/ingest`` and friends; if none exists, that is recorded as
|
||||
``MISSING_INGEST_ENDPOINT`` with a proposed contract, and the tests that depend
|
||||
on it report BLOCKED rather than PASS or FAIL. A test that cannot run has not
|
||||
passed.
|
||||
|
||||
python3 scripts/audio_controller_connect.py --report-dir /tmp/connect
|
||||
|
||||
Exit 0 when no test failed. BLOCKED does not fail the run; it is a finding.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import ssl
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
#: The controller as found by discovery. Plain HTTP: the port answers uvicorn
|
||||
#: with no TLS, so the ``https://`` in the original brief cannot connect and
|
||||
#: ``UMBRA_AUDIO_CONTROLLER_VERIFY`` has nothing to verify. Recorded, not worked
|
||||
#: around -- see ``transport`` in the report.
|
||||
DEFAULT_BASE_URL = os.environ.get("UMBRA_AUDIO_CONTROLLER_URL", "http://192.168.123.14:8442")
|
||||
|
||||
INGEST_CANDIDATES = (
|
||||
"/api/audio/ingest",
|
||||
"/api/projects/import/audio",
|
||||
"/api/import/audio",
|
||||
"/api/audio/dataset",
|
||||
"/api/projects/audio-dataset",
|
||||
"/api/manifest/ingest",
|
||||
)
|
||||
|
||||
PROBE_PATHS = (
|
||||
"/api/health", "/healthz", "/health", "/openapi.json", "/api/openapi.json",
|
||||
"/docs", "/api/setup/status", "/api/projects", "/api/artifacts", "/api/workspace",
|
||||
"/api/nodes", "/api/jobs", "/api/audit", "/api/me", "/api/auth/me",
|
||||
"/api/definitions", "/api/ingest", "/api/datasets",
|
||||
)
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
stamp = datetime.now(UTC).replace(microsecond=0)
|
||||
return stamp.isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
class SyncLog:
|
||||
"""One JSON object per line. A log you cannot parse is not a log."""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = path
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.handle = path.open("w", encoding="utf-8")
|
||||
self.entries = 0
|
||||
|
||||
def write(self, phase: str, action: str, **fields: Any) -> None:
|
||||
entry = {
|
||||
"timestamp": now_iso(),
|
||||
"phase": phase,
|
||||
"action": action,
|
||||
**{k: v for k, v in fields.items()},
|
||||
}
|
||||
self.handle.write(json.dumps(entry, sort_keys=True, ensure_ascii=False) + "\n")
|
||||
self.handle.flush()
|
||||
self.entries += 1
|
||||
|
||||
def close(self) -> None:
|
||||
self.handle.close()
|
||||
|
||||
|
||||
@dataclass
|
||||
class Result:
|
||||
test_id: str
|
||||
name: str
|
||||
status: str # PASS | FAIL | BLOCKED | INFO
|
||||
detail: str
|
||||
evidence: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {"id": self.test_id, "name": self.name, "status": self.status,
|
||||
"detail": self.detail, "evidence": self.evidence}
|
||||
|
||||
|
||||
class Client:
|
||||
def __init__(self, base_url: str, log: SyncLog) -> None:
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.log = log
|
||||
# A self-signed certificate would need an override; recorded if set, and
|
||||
# never silently applied. Plain HTTP needs no context at all.
|
||||
self.verify = os.environ.get("UMBRA_AUDIO_CONTROLLER_VERIFY", "true").lower() != "false"
|
||||
self.context: ssl.SSLContext | None = None
|
||||
if self.base_url.startswith("https") and not self.verify:
|
||||
self.context = ssl.create_default_context()
|
||||
self.context.check_hostname = False
|
||||
self.context.verify_mode = ssl.CERT_NONE
|
||||
|
||||
def request(self, method: str, path: str, *, body: Any = None,
|
||||
headers: dict[str, str] | None = None,
|
||||
phase: str = "probe") -> tuple[int, Any, int]:
|
||||
"""Returns (status, parsed_body_or_text, duration_ms). Never raises."""
|
||||
|
||||
url = self.base_url + path
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
request = urllib.request.Request(url, data=data, method=method)
|
||||
for key, value in (headers or {}).items():
|
||||
request.add_header(key, value)
|
||||
if data is not None:
|
||||
request.add_header("Content-Type", "application/json")
|
||||
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=15,
|
||||
context=self.context) as response:
|
||||
status, raw = response.status, response.read()
|
||||
except urllib.error.HTTPError as error:
|
||||
status, raw = error.code, error.read()
|
||||
except Exception as error: # noqa: BLE001 - connectivity is a result
|
||||
duration = int((time.monotonic() - started) * 1000)
|
||||
self.log.write(phase, method, endpoint=path,
|
||||
status=None, duration_ms=duration,
|
||||
result_hash=None,
|
||||
error={"code": type(error).__name__,
|
||||
"message": str(error)[:200],
|
||||
"retryable": True,
|
||||
"suggested_fix": "check the controller is running and reachable"})
|
||||
failure = {"error": {"code": type(error).__name__,
|
||||
"message": str(error)[:200]}}
|
||||
return 0, failure, duration
|
||||
|
||||
duration = int((time.monotonic() - started) * 1000)
|
||||
try:
|
||||
parsed: Any = json.loads(raw)
|
||||
except ValueError:
|
||||
parsed = raw[:400].decode("utf-8", "replace")
|
||||
# Header values are never logged; only whether one was present.
|
||||
self.log.write(phase, method, endpoint=path, status=status,
|
||||
duration_ms=duration,
|
||||
credentials_present=bool(headers),
|
||||
request_summary=_summarise(body),
|
||||
result_hash=_hash(parsed))
|
||||
return status, parsed, duration
|
||||
|
||||
def authenticate(self, log: SyncLog) -> tuple[dict[str, str], str]:
|
||||
"""Node key, then service token. Returns (headers, how)."""
|
||||
|
||||
node_key = os.environ.get("UMBRA_AUDIO_NODE_KEY")
|
||||
token = os.environ.get("UMBRA_AUDIO_API_TOKEN")
|
||||
if node_key:
|
||||
log.write("auth", "try_node_key", endpoint="/api/auth/me",
|
||||
note="X-Node-Key header")
|
||||
return {"X-Node-Key": node_key}, "node_key_header"
|
||||
if token:
|
||||
log.write("auth", "try_service_token", endpoint="/api/auth/me",
|
||||
note="Authorization: Bearer")
|
||||
return {"Authorization": f"Bearer {token}"}, "service_token_bearer"
|
||||
log.write("auth", "no_credentials",
|
||||
note="no node key or service token configured on this machine")
|
||||
return {}, "none"
|
||||
|
||||
|
||||
def _summarise(body: Any) -> Any:
|
||||
"""A description of a request, small enough to log and free of secrets."""
|
||||
|
||||
if body is None:
|
||||
return None
|
||||
if isinstance(body, dict):
|
||||
return {k: (f"<{len(v)} items>" if isinstance(v, (list, dict)) else v)
|
||||
for k, v in body.items()
|
||||
if k not in {"token", "api_key", "password", "secret"}}
|
||||
return f"<{type(body).__name__}>"
|
||||
|
||||
|
||||
def _hash(value: Any) -> str | None:
|
||||
import hashlib
|
||||
try:
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True,
|
||||
default=str).encode()).hexdigest()[:16]
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# tests
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def t01_health(client: Client) -> Result:
|
||||
status, body, _ = client.request("GET", "/api/health")
|
||||
ok = status == 200 and isinstance(body, dict) and body.get("status") == "ok"
|
||||
return Result("T01", "Health", "PASS" if ok else "FAIL",
|
||||
"controller reports status=ok" if ok else f"health returned {status}",
|
||||
{"status_code": status, "body": body if ok else str(body)[:200]})
|
||||
|
||||
|
||||
def t02_auth(client: Client, headers: dict[str, str], how: str) -> Result:
|
||||
if not headers:
|
||||
return Result("T02", "Auth", "BLOCKED",
|
||||
"no credential is configured on this machine; "
|
||||
"an operator must supply a node key or a scoped service token",
|
||||
{"headers_tried": [], "how": how})
|
||||
status, body, _ = client.request("GET", "/api/auth/me", headers=headers)
|
||||
ok = status == 200
|
||||
return Result("T02", "Auth", "PASS" if ok else "FAIL",
|
||||
f"authenticated with {how}" if ok else f"credential rejected ({status})",
|
||||
{"status_code": status, "mechanism": how,
|
||||
"secret_logged": False})
|
||||
|
||||
|
||||
def t03_version(client: Client) -> Result:
|
||||
status, body, _ = client.request("GET", "/api/health")
|
||||
controller_version = (body or {}).get("version") if isinstance(body, dict) else None
|
||||
here = "1.0.0 (bundle format)"
|
||||
compatible = bool(controller_version)
|
||||
return Result("T03", "Version compatibility", "PASS" if compatible else "FAIL",
|
||||
f"controller {controller_version}; this side speaks "
|
||||
f"bundle format {here}. No versioned API exists to negotiate "
|
||||
f"against, so there is nothing to be incompatible with yet.",
|
||||
{"controller_version": controller_version,
|
||||
"controller_service": ((body or {}).get("service")
|
||||
if isinstance(body, dict) else None),
|
||||
"client_bundle_format": 1,
|
||||
"hard_incompatibility": False})
|
||||
|
||||
|
||||
def t04_ingest_exists(client: Client) -> Result:
|
||||
probed: dict[str, int] = {}
|
||||
for path in INGEST_CANDIDATES:
|
||||
status, _, _ = client.request("GET", path)
|
||||
probed[path] = status
|
||||
if status in (200, 405):
|
||||
return Result("T04", "Ingest endpoint exists", "PASS",
|
||||
f"{path} exists ({status})", {"probed": probed})
|
||||
return Result("T04", "Ingest endpoint exists", "FAIL",
|
||||
"MISSING_INGEST_ENDPOINT: no bulk audio-dataset ingest path "
|
||||
"exists on the controller. This blocks a push, not the "
|
||||
"metadata handover, which is file-based by design.",
|
||||
{"probed": probed, "finding": "MISSING_INGEST_ENDPOINT"})
|
||||
|
||||
|
||||
def t05_dry_run(client: Client) -> Result:
|
||||
return Result("T05", "Dry-run validation", "BLOCKED",
|
||||
"depends on an ingest endpoint; see T04. The controller's own "
|
||||
"plan sequences ingest after the metadata bundle, so this is "
|
||||
"the designed order rather than a defect.",
|
||||
{"depends_on": "T04"})
|
||||
|
||||
|
||||
def t06_idempotency(client: Client) -> Result:
|
||||
return Result("T06", "Idempotency check", "BLOCKED",
|
||||
"depends on an ingest endpoint; see T04",
|
||||
{"depends_on": "T04",
|
||||
"client_side_equivalent": "the bundle build is deterministic: "
|
||||
"the same project yields the same manifest.jsonl bytes and the "
|
||||
"same content_hash, so re-shipping is detectable without a server"})
|
||||
|
||||
|
||||
def t07_schema_contract(client: Client) -> Result:
|
||||
return Result("T07", "Schema contract", "BLOCKED",
|
||||
"the controller has not yet issued a manifest format to accept; "
|
||||
"its request says the mapping and the accepted format come after "
|
||||
"the bundle. A contract is proposed in the report, not assumed.",
|
||||
{"depends_on": "T04",
|
||||
"bundle_validates_locally": True})
|
||||
|
||||
|
||||
def t08_audit_round_trip(client: Client) -> Result:
|
||||
return Result("T08", "Bidirectional audit", "BLOCKED",
|
||||
"no ingest event can be created without an ingest endpoint; "
|
||||
"the controller does expose an audit table but only to "
|
||||
"authenticated principals",
|
||||
{"depends_on": "T04"})
|
||||
|
||||
|
||||
def t09_error_handling(client: Client) -> Result:
|
||||
"""The brief asked for a structured error with an actionable code.
|
||||
|
||||
The interesting part is that the controller does have its own envelope:
|
||||
``{"error": {"code", "message"}}``. Probing with an unsupported method is
|
||||
enough to see it, and it is worth seeing -- a service-to-service caller
|
||||
needs to key off ``error.code`` rather than parsing prose. The probe also
|
||||
revealed that **CSRF is enforced**, which is a real constraint on any future
|
||||
browser-authenticated path and is recorded rather than left as a surprise.
|
||||
"""
|
||||
|
||||
status, body, _ = client.request("DELETE", "/api/health")
|
||||
envelope = body.get("error") if isinstance(body, dict) else None
|
||||
has_code = bool(isinstance(envelope, dict) and envelope.get("code"))
|
||||
has_message = bool(isinstance(envelope, dict) and envelope.get("message"))
|
||||
structured = has_code and has_message
|
||||
|
||||
if structured:
|
||||
detail = (f"error envelope carries code={envelope['code']!r} and an "
|
||||
f"actionable message; probe returned {status}")
|
||||
else:
|
||||
detail = f"no structured error envelope; probe returned {status}: {str(body)[:160]}"
|
||||
|
||||
return Result("T09", "Error handling", "PASS" if structured else "FAIL",
|
||||
detail,
|
||||
{"status_code": status, "body": body,
|
||||
"error_code": ((envelope or {}).get("code")
|
||||
if isinstance(envelope, dict) else None),
|
||||
"csrf_required": status == 403
|
||||
and isinstance(envelope, dict)
|
||||
and envelope.get("code") == "csrf_token_missing",
|
||||
"note": ("CSRF is enforced on this origin. A machine-to-machine "
|
||||
"client should use a bearer/node-key path rather than "
|
||||
"cookie auth, or the CSRF requirement becomes a "
|
||||
"per-request concern.")})
|
||||
|
||||
|
||||
def t10_payload_readiness(bundle_records: int | None, bundle_bytes: int | None) -> Result:
|
||||
if bundle_records is None:
|
||||
return Result("T10", "Large payload readiness", "BLOCKED",
|
||||
"no bundle was built in this run", {})
|
||||
return Result("T10", "Large payload readiness", "PASS",
|
||||
f"the dataset is {bundle_records} records / "
|
||||
f"{bundle_bytes} bytes, which is small enough for a single "
|
||||
f"request; no chunking or streaming strategy is needed yet",
|
||||
{"records": bundle_records, "bytes": bundle_bytes,
|
||||
"max_records_per_request": bundle_records,
|
||||
"chunking_required": False,
|
||||
"strategy": "single request, with content_hash for idempotency"})
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# proposed contract
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def proposed_contract() -> dict[str, Any]:
|
||||
return {
|
||||
"note": ("A proposal only. No endpoint was invented, probed into "
|
||||
"existence, or assumed to exist; this is what the audio side "
|
||||
"would send if the controller chooses to add it."),
|
||||
"openapi_paths": {
|
||||
"/api/audio/ingest": {
|
||||
"post": {
|
||||
"summary": "Accept an audio-system metadata dataset",
|
||||
"requestBody": {
|
||||
"required": True,
|
||||
"content": {"application/json": {
|
||||
"schema": {"$ref": "#/components/schemas/AudioDatasetEnvelope"},
|
||||
}},
|
||||
},
|
||||
"responses": {
|
||||
"200": {"description": "Dry run; validation report, nothing written"},
|
||||
"201": {"description": "Accepted; import_id and audit_id returned"},
|
||||
"400": {"description": "Structured validation error", "content":
|
||||
{"application/json": {"schema": {
|
||||
"$ref": "#/components/schemas/Error"}}}},
|
||||
"409": {"description": ("Idempotency key already seen "
|
||||
"with a different hash")},
|
||||
},
|
||||
}
|
||||
}
|
||||
},
|
||||
"components": {
|
||||
"schemas": {
|
||||
"AudioDatasetEnvelope": {
|
||||
"type": "object",
|
||||
"required": ["bundle_format_version", "source", "content_hash", "dry_run"],
|
||||
"properties": {
|
||||
"bundle_format_version": {"type": "integer", "const": 1},
|
||||
"source": {
|
||||
"type": "object",
|
||||
"required": ["system", "node_id"],
|
||||
"properties": {
|
||||
"system": {"type": "string"},
|
||||
"node_id": {"type": "string"},
|
||||
"product_version": {"type": "string"},
|
||||
"database_schema_fingerprint": {"type": "string"},
|
||||
},
|
||||
},
|
||||
"content_hash": {
|
||||
"type": "string",
|
||||
"description": ("sha256 of the manifest bytes. The server "
|
||||
"uses it as the idempotency key so a "
|
||||
"re-send of unchanged data is a no-op."),
|
||||
},
|
||||
"dry_run": {"type": "boolean", "default": True},
|
||||
"records": {"type": "array", "items": {"type": "object"}},
|
||||
},
|
||||
},
|
||||
"ValidationReport": {
|
||||
"type": "object",
|
||||
"required": ["valid", "counts", "errors", "warnings"],
|
||||
"properties": {
|
||||
"valid": {"type": "boolean"},
|
||||
"counts": {"type": "object"},
|
||||
"errors": {"type": "array", "items": {"type": "object"}},
|
||||
"warnings": {"type": "array", "items": {"type": "object"}},
|
||||
"proposed_changes": {"type": "array", "items": {"type": "object"}},
|
||||
},
|
||||
},
|
||||
"Error": {
|
||||
"type": "object",
|
||||
"required": ["code", "message"],
|
||||
"properties": {"code": {"type": "string"},
|
||||
"message": {"type": "string"},
|
||||
"retryable": {"type": "boolean"}},
|
||||
},
|
||||
}
|
||||
},
|
||||
"invariants_the_controller_should_enforce": [
|
||||
"content_hash is the idempotency key; a repeat with the same hash is a no-op.",
|
||||
"A dry run must not write. It returns counts and errors, and nothing else.",
|
||||
"Records are keyed by (source.system, source.node_id, id) so a re-import "
|
||||
"updates rather than duplicates.",
|
||||
"Nothing is deleted on re-import; a record absent from a later dataset is "
|
||||
"marked superseded, not removed.",
|
||||
"Chapter identity uses the database row id, never chapter_index or a "
|
||||
"filename-derived number.",
|
||||
"Speaker identity is not merged by the server. Case-variant and alias names "
|
||||
"are surfaced as warnings for a human to decide.",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--report-dir", default=".", help="where to write the two artefacts")
|
||||
parser.add_argument("--bundle", default=None,
|
||||
help="a built bundle directory, to report its size for T10")
|
||||
parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
|
||||
args = parser.parse_args()
|
||||
|
||||
out = Path(args.report_dir)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
log = SyncLog(out / "audio_controller_sync.log")
|
||||
log.write("start", "run", base_url=args.base_url, note="no secret is logged")
|
||||
|
||||
client = Client(args.base_url, log)
|
||||
|
||||
# --- discovery ---------------------------------------------------
|
||||
discovery: dict[str, Any] = {"base_url": args.base_url, "probes": {}}
|
||||
for path in PROBE_PATHS:
|
||||
status, body, _ = client.request("GET", path, phase="discovery")
|
||||
discovery["probes"][path] = {
|
||||
"status": status,
|
||||
"body": body if status == 200 else str(body)[:160],
|
||||
}
|
||||
log.write("discovery", "complete", endpoints_found=sum(
|
||||
1 for p in discovery["probes"].values() if p["status"] in (200, 401, 403)))
|
||||
|
||||
headers, how = client.authenticate(log)
|
||||
|
||||
bundle_records = bundle_bytes = None
|
||||
if args.bundle:
|
||||
manifest = Path(args.bundle) / "manifest.jsonl"
|
||||
if manifest.exists():
|
||||
bundle_records = sum(1 for line in manifest.read_text().splitlines() if line.strip())
|
||||
bundle_bytes = manifest.stat().st_size
|
||||
|
||||
results = [
|
||||
t01_health(client),
|
||||
t02_auth(client, headers, how),
|
||||
t03_version(client),
|
||||
t04_ingest_exists(client),
|
||||
t05_dry_run(client),
|
||||
t06_idempotency(client),
|
||||
t07_schema_contract(client),
|
||||
t08_audit_round_trip(client),
|
||||
t09_error_handling(client),
|
||||
t10_payload_readiness(bundle_records, bundle_bytes),
|
||||
]
|
||||
for result in results:
|
||||
log.write("test", result.test_id, name=result.name, status=result.status)
|
||||
|
||||
passing = [r for r in results if r.status == "PASS"]
|
||||
failed = [r for r in results if r.status == "FAIL"]
|
||||
blocked = [r for r in results if r.status == "BLOCKED"]
|
||||
|
||||
missing_ingest = any(r.evidence.get("finding") == "MISSING_INGEST_ENDPOINT" for r in results)
|
||||
transport = "plain HTTP (uvicorn); no TLS on this port" \
|
||||
if client.base_url.startswith("http") else "HTTPS"
|
||||
|
||||
report = {
|
||||
"report_version": 1,
|
||||
"generated_at": now_iso(),
|
||||
"source_system": {"system": "umbraaudio", "node_id": "umbraaudio-m4"},
|
||||
"transport": {
|
||||
"base_url": args.base_url,
|
||||
"tls": transport,
|
||||
"tls_verify_override": os.environ.get("UMBRA_AUDIO_CONTROLLER_VERIFY"),
|
||||
"note": ("The brief specified https:// on this port. The port answers "
|
||||
"plain HTTP, so the TLS handshake fails with a protocol-version "
|
||||
"alert and there is nothing for a verify override to do. "
|
||||
"Reported rather than worked around."),
|
||||
},
|
||||
"authentication": {
|
||||
"mechanism_available": how,
|
||||
"node_key_configured": bool(os.environ.get("UMBRA_AUDIO_NODE_KEY")),
|
||||
"service_token_configured": bool(os.environ.get("UMBRA_AUDIO_API_TOKEN")),
|
||||
"secrets_recorded_anywhere": False,
|
||||
},
|
||||
"discovery": discovery,
|
||||
"tests": [r.as_dict() for r in results],
|
||||
"summary": {
|
||||
"counts": {"pass": len(passing), "fail": len(failed),
|
||||
"blocked": len(blocked)},
|
||||
"passing": [r.test_id for r in passing],
|
||||
"failing": [r.test_id for r in failed],
|
||||
"blocked_ids": [r.test_id for r in blocked],
|
||||
"blocking_findings": ["MISSING_INGEST_ENDPOINT"] if missing_ingest else [],
|
||||
},
|
||||
"readiness": {
|
||||
"verdict": "READY_FOR_METADATA_HANDOVER_NOT_READY_FOR_PUSH",
|
||||
"rationale": [
|
||||
"The controller has no bulk ingest endpoint, so a dataset cannot "
|
||||
"be pushed to it; nothing was pushed and nothing should be.",
|
||||
"The controller's own request is for a metadata bundle, validated "
|
||||
"by its own validator, handed over as a file. That is done.",
|
||||
"Its request also says the accepted manifest format comes after "
|
||||
"it receives the bundle, so T05-T08 are sequenced, not broken.",
|
||||
],
|
||||
},
|
||||
"proposed_ingest_contract": proposed_contract() if missing_ingest else None,
|
||||
"confirmation_gate": {
|
||||
"required_before_any_push": True,
|
||||
"current_state": "NOT CONFIRMED - nothing has been pushed",
|
||||
"instruction": ("A non-dry-run import requires explicit operator "
|
||||
"confirmation. This run made no write of any kind."),
|
||||
},
|
||||
}
|
||||
|
||||
(out / "audio_controller_test_report.json").write_text(
|
||||
json.dumps(report, indent=2, sort_keys=True) + "\n")
|
||||
log.write("end", "run", passed=len(passing), failed=len(failed), blocked=len(blocked))
|
||||
log.close()
|
||||
|
||||
print(f"{'ID':4} {'STATUS':8} {'NAME'}")
|
||||
print("-" * 52)
|
||||
for result in results:
|
||||
print(f"{result.test_id:4} {result.status:8} {result.name}")
|
||||
print("-" * 52)
|
||||
print(f"pass {len(passing)} fail {len(failed)} blocked {len(blocked)}")
|
||||
print(f"report: {out / 'audio_controller_test_report.json'}")
|
||||
print(f"log: {out / 'audio_controller_sync.log'} ({log.entries} entries)")
|
||||
print(f"verdict: {report['readiness']['verdict']}")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
267
scripts/audit_controller_bundle.py
Normal file
267
scripts/audit_controller_bundle.py
Normal file
@@ -0,0 +1,267 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Audit a built bundle against the project it claims to describe.
|
||||
|
||||
The controller's validator checks that the bundle is internally consistent: a
|
||||
required field is present, an id resolves, no credential is present. It cannot
|
||||
check that the bundle is *true*. Every hash, byte size and duration in here is
|
||||
something the audio system asserted about itself, and this is the file that goes
|
||||
and checks those assertions against the filesystem.
|
||||
|
||||
That distinction is not pedantic. The failure this project has actually hit is a
|
||||
record that says one thing and means another -- an approved chapter whose
|
||||
published MP3 is four seconds longer than the approved WAV because padding was
|
||||
added afterwards. A validator passes that bundle happily. Only re-measuring
|
||||
finds it.
|
||||
|
||||
python3 scripts/audit_controller_bundle.py --bundle /tmp/dh-bundle
|
||||
|
||||
Exits 0 when every checked claim holds, 1 when any does not. Prints one line per
|
||||
problem and a summary of how many claims were checked.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
import wave
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def load(path: Path) -> Any:
|
||||
return json.loads(path.read_text())
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--bundle", required=True, help="bundle directory")
|
||||
parser.add_argument("--project", required=True, help="project directory name")
|
||||
parser.add_argument("--hash-renders", action="store_true", default=True,
|
||||
help="re-hash every render file (slower, thorough)")
|
||||
parser.add_argument("--no-hash-renders", dest="hash_renders", action="store_false")
|
||||
args = parser.parse_args()
|
||||
|
||||
bundle = Path(args.bundle)
|
||||
repo = Path(__file__).resolve().parent.parent
|
||||
book_dir = repo / args.project
|
||||
umbra = book_dir / ".umbra"
|
||||
|
||||
index = load(bundle / "bundle.json")
|
||||
raw = (bundle / "manifest.jsonl").read_text().splitlines()
|
||||
records = [json.loads(line) for line in raw if line.strip()]
|
||||
by_id = {r["id"]: r for r in records}
|
||||
kinds = Counter(r["kind"] for r in records)
|
||||
|
||||
problems: list[str] = []
|
||||
checked = 0
|
||||
|
||||
def check(condition: bool, message: str) -> None:
|
||||
nonlocal checked
|
||||
checked += 1
|
||||
if not condition:
|
||||
problems.append(message)
|
||||
|
||||
# 1. The index's own content hash must describe the file it ships with.
|
||||
manifest_bytes = (bundle / "manifest.jsonl").read_bytes()
|
||||
actual = hashlib.sha256(manifest_bytes).hexdigest()
|
||||
check(actual == index["content_hash"]["value"],
|
||||
f"bundle.json content_hash {index['content_hash']['value'][:16]}... "
|
||||
f"does not match manifest.jsonl ({actual[:16]}...)")
|
||||
check(index["counts"]["records_total"] == len(records),
|
||||
f"bundle.json claims {index['counts']['records_total']} records, file has {len(records)}")
|
||||
|
||||
# 2. Every utterance's text hash must match the text it ships with.
|
||||
for record in records:
|
||||
if record["kind"] == "utterance":
|
||||
digest = hashlib.sha256(record["text"].encode("utf-8")).hexdigest()
|
||||
check(digest == record["text_sha256"],
|
||||
f"utterance {record['id']}: text_sha256 does not match its own text")
|
||||
check(len(record["text"]) == record["text_char_count"],
|
||||
f"utterance {record['id']}: text_char_count disagrees with text length")
|
||||
|
||||
# 3. Every referenced id must resolve, including the ones the controller's
|
||||
# validator does not check (speaker_id, render_id).
|
||||
for record in records:
|
||||
for field in ("book_id", "chapter_id", "speaker_id", "render_id", "voice_id"):
|
||||
value = record.get(field)
|
||||
if isinstance(value, str):
|
||||
check(value in by_id,
|
||||
f"{record['kind']} {record['id']}: {field} -> {value!r} is not in the bundle")
|
||||
for field in ("chapters", "renders", "inputs", "published_artifact_ids"):
|
||||
for value in record.get(field, []) or []:
|
||||
check(value in by_id,
|
||||
f"{record['kind']} {record['id']}: {field} contains "
|
||||
f"{value!r}, not in the bundle")
|
||||
|
||||
# 4. Render paths must exist, be the right size, and (optionally) hash right.
|
||||
renders = [r for r in records if r["kind"] == "render"]
|
||||
missing = unreadable = 0
|
||||
for record in renders:
|
||||
path = book_dir / ".umbra" / record["path"]
|
||||
if not path.exists():
|
||||
if record.get("file_present") is False:
|
||||
continue
|
||||
missing += 1
|
||||
problems.append(f"render {record['id']}: path {record['path']} does not exist")
|
||||
continue
|
||||
checked += 1
|
||||
size = path.stat().st_size
|
||||
check(size == record["byte_size"],
|
||||
f"render {record['id']}: byte_size {record['byte_size']} != actual {size}")
|
||||
if record.get("content_sha256") and args.hash_renders:
|
||||
check(sha256_file(path) == record["content_sha256"],
|
||||
f"render {record['id']}: content_sha256 does not match the file on disk")
|
||||
if record["audio_format"] and record["audio_format"].get("codec") == "unreadable":
|
||||
unreadable += 1
|
||||
|
||||
# 5. Durations must be re-derivable for WAV files, which is the one format
|
||||
# the standard library can measure without trusting the recorder.
|
||||
sampled = 0
|
||||
for record in renders:
|
||||
if record["path"].endswith(".wav") and record.get("duration_ms"):
|
||||
path = book_dir / ".umbra" / record["path"]
|
||||
if not path.exists():
|
||||
continue
|
||||
try:
|
||||
with wave.open(str(path)) as handle:
|
||||
rate = handle.getframerate()
|
||||
if not rate:
|
||||
continue
|
||||
measured = round(handle.getnframes() / rate * 1000)
|
||||
except (OSError, wave.Error, EOFError):
|
||||
continue
|
||||
checked += 1
|
||||
sampled += 1
|
||||
check(measured == record["duration_ms"],
|
||||
f"render {record['id']}: duration_ms "
|
||||
f"{record['duration_ms']} != measured {measured}")
|
||||
|
||||
# 6. The bundle's own counts must match the database it was built from.
|
||||
conn = sqlite3.connect(f"file:{umbra / 'project.db'}?mode=ro", uri=True)
|
||||
db_chapters = conn.execute("SELECT count(*) FROM chapters WHERE project_id=1").fetchone()[0]
|
||||
db_utterances = conn.execute(
|
||||
"SELECT count(*) FROM utterances u JOIN chapters c ON c.id=u.chapter_id "
|
||||
"WHERE c.project_id=1").fetchone()[0]
|
||||
db_names = conn.execute(
|
||||
"SELECT count(DISTINCT trim(speaker_name)) FROM utterances u JOIN chapters c "
|
||||
"ON c.id=u.chapter_id WHERE c.project_id=1 AND speaker_name IS NOT NULL").fetchone()[0]
|
||||
db_null_speakers = conn.execute(
|
||||
"SELECT count(*) FROM utterances u JOIN chapters c ON c.id=u.chapter_id "
|
||||
"WHERE c.project_id=1 AND (u.speaker_name IS NULL OR trim(u.speaker_name)='')"
|
||||
).fetchone()[0]
|
||||
db_speaker_counts = dict(conn.execute(
|
||||
"SELECT trim(u.speaker_name), count(*) FROM utterances u JOIN chapters c "
|
||||
"ON c.id=u.chapter_id WHERE c.project_id=1 AND u.speaker_name IS NOT NULL "
|
||||
"GROUP BY trim(u.speaker_name)").fetchall())
|
||||
check(len([r for r in records if r["kind"] == "chapter"]) == db_chapters,
|
||||
f"chapter records != database chapters ({db_chapters})")
|
||||
check(len([r for r in records if r["kind"] == "utterance"]) == db_utterances,
|
||||
f"utterance records != database utterances ({db_utterances})")
|
||||
check(len([r for r in records if r["kind"] == "speaker"]) == db_names,
|
||||
f"speaker records != distinct speaker names in the database ({db_names})")
|
||||
conn.close()
|
||||
|
||||
# 7. Speaker utterance counts must be true, because they are the number a
|
||||
# mapping is most likely to trust. `utterance_count` is the exact count
|
||||
# for that exact spelling; the implicit-narrator figure is checked
|
||||
# separately because it is a deliberate inference, not a stored value.
|
||||
for record in records:
|
||||
if record["kind"] != "speaker":
|
||||
continue
|
||||
actual_count = db_speaker_counts.get(record["primary_name"])
|
||||
check(actual_count == record["utterance_count"],
|
||||
f"speaker {record['id']}: utterance_count {record['utterance_count']} "
|
||||
f"!= database {actual_count}")
|
||||
expected_implicit = record["utterance_count"] + (
|
||||
db_null_speakers if record["primary_name"].casefold() == "narrator" else 0
|
||||
)
|
||||
check(expected_implicit == record["utterance_count_including_implicit_narrator"],
|
||||
f"speaker {record['id']}: implicit-narrator count "
|
||||
f"{record['utterance_count_including_implicit_narrator']} "
|
||||
f"!= expected {expected_implicit}")
|
||||
|
||||
# 7b. Every utterance's speaker must resolve, and every implicit-narrator
|
||||
# row must actually have had a null speaker name. This is the assertion
|
||||
# that would catch the bundle quietly inventing a narrator.
|
||||
null_rows = 0
|
||||
for record in records:
|
||||
if record["kind"] != "utterance":
|
||||
continue
|
||||
implicit = record["speaker_resolution"] == "implicit_narrator"
|
||||
if implicit:
|
||||
null_rows += 1
|
||||
check(record["speaker_name_raw"] in (None, ""),
|
||||
f"utterance {record['id']}: marked implicit_narrator but "
|
||||
f"speaker_name_raw is {record['speaker_name_raw']!r}")
|
||||
else:
|
||||
check(record["speaker_name_raw"] not in (None, ""),
|
||||
f"utterance {record['id']}: marked explicit but speaker_name_raw is null")
|
||||
check(record["speaker_id"] is not None,
|
||||
f"utterance {record['id']}: speaker did not resolve")
|
||||
check(null_rows == db_null_speakers,
|
||||
f"{null_rows} utterances marked implicit_narrator != {db_null_speakers} "
|
||||
"rows with an empty speaker_name")
|
||||
|
||||
# 8. Every chapter's utterance_count must be true.
|
||||
db_per_chapter = {}
|
||||
conn = sqlite3.connect(f"file:{umbra / 'project.db'}?mode=ro", uri=True)
|
||||
for row_id, count in conn.execute(
|
||||
"SELECT u.chapter_id, count(*) FROM utterances u JOIN chapters c "
|
||||
"ON c.id=u.chapter_id WHERE c.project_id=1 GROUP BY u.chapter_id").fetchall():
|
||||
db_per_chapter[f"ch-{row_id}"] = count
|
||||
conn.close()
|
||||
for record in records:
|
||||
if record["kind"] != "chapter":
|
||||
continue
|
||||
check(db_per_chapter.get(record["id"], 0) == record["utterance_count"],
|
||||
f"chapter {record['id']}: utterance_count {record['utterance_count']} "
|
||||
f"!= database {db_per_chapter.get(record['id'], 0)}")
|
||||
|
||||
# 9. Casefold collisions must be recorded, not silently merged.
|
||||
collisions: dict[str, list[str]] = defaultdict(list)
|
||||
for record in records:
|
||||
if record["kind"] == "speaker":
|
||||
collisions[record["primary_name"].casefold()].append(record["primary_name"])
|
||||
real_collisions = {k: v for k, v in collisions.items() if len(v) > 1}
|
||||
for key, names in real_collisions.items():
|
||||
declared = [r for r in records if r["kind"] == "speaker"
|
||||
and r.get("casefold_collision_group") == key]
|
||||
check(len(declared) == len(names),
|
||||
f"speaker name collision {key!r} ({names}) is not declared in the bundle")
|
||||
|
||||
print(f"records by kind: {dict(kinds)}")
|
||||
print(f"claims checked: {checked}")
|
||||
print(f"renders: {len(renders)} "
|
||||
f"({missing} missing, {unreadable} unreadable, {sampled} durations re-measured)")
|
||||
print(f"speaker name collisions declared: "
|
||||
f"{ {k: sorted(v) for k, v in real_collisions.items()} }")
|
||||
print(f"implicit-narrator utterances: {null_rows} (database agrees: "
|
||||
f"{null_rows == db_null_speakers})")
|
||||
|
||||
if problems:
|
||||
print(f"\n{len(problems)} problem(s):\n")
|
||||
for problem in problems[:60]:
|
||||
print(f" - {problem}")
|
||||
if len(problems) > 60:
|
||||
print(f" ... and {len(problems) - 60} more")
|
||||
return 1
|
||||
|
||||
print("\nEvery checked claim holds: hashes, byte sizes, durations, counts and "
|
||||
"references agree with the project on disk.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
1597
scripts/build_controller_bundle.py
Normal file
1597
scripts/build_controller_bundle.py
Normal file
File diff suppressed because it is too large
Load Diff
@@ -320,11 +320,18 @@ def build_report(project_dir: str | None = None) -> dict[str, Any]:
|
||||
],
|
||||
"speaker_identity_lives_in": "speakers.json + voice_profiles.json + ref WAVs",
|
||||
"audio_record_lives_in": "filesystem + *.wav.txt markers",
|
||||
"authentication": "none; active_project is an unsigned browser cookie",
|
||||
"authentication": ("session auth via umbraaudio/auth.py, enforced by "
|
||||
"fail-closed middleware in umbraaudio/web/__init__.py "
|
||||
"(added 2026-10-09)"),
|
||||
"shared_database_present_here": False,
|
||||
"controller_endpoint": None,
|
||||
"controller_status": "unavailable: host 192.168.123.14 retired/offline (2026-10-08)",
|
||||
"controller_stage_reached": "0 (was reachable 2026-10-03, before retirement)",
|
||||
"controller_endpoint": "http://192.168.123.14:8442",
|
||||
"controller_transport": "plain HTTP (uvicorn). No TLS on this port.",
|
||||
"controller_status": "reachable: /api/health 200, service umbratrans-web 0.1.0",
|
||||
"controller_stage_reached": ("1 - health confirmed and an audio metadata "
|
||||
"bundle delivered to oldbox:audio-read/bundles "
|
||||
"(2026-10-14). No ingest endpoint exists yet."),
|
||||
"controller_ingest_endpoint": None,
|
||||
"controller_blocking_findings": ["MISSING_INGEST_ENDPOINT"],
|
||||
"secret_values_reported": False,
|
||||
}
|
||||
return report
|
||||
|
||||
477
tests/test_controller_bundle.py
Normal file
477
tests/test_controller_bundle.py
Normal file
@@ -0,0 +1,477 @@
|
||||
"""Tests for the controller bundle builder.
|
||||
|
||||
The builder's job is to emit something the *controller's* validator accepts, so
|
||||
most of these tests build a small synthetic project and run the real validator
|
||||
against the result. Running the actual validator -- copied into the bundle by the
|
||||
builder -- is the point: a test that re-implements the checks would only prove the
|
||||
test agrees with itself.
|
||||
|
||||
The synthetic project is built rather than the real one, so the suite does not
|
||||
depend on 4,000 utterances and 4,500 audio files being present and current. The
|
||||
real project is exercised by ``scripts/audit_controller_bundle.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
BUILDER_PATH = REPO / "scripts" / "build_controller_bundle.py"
|
||||
VALIDATOR_SOURCE = Path("/tmp/controller-validator.py")
|
||||
|
||||
|
||||
def _load_builder() -> Any:
|
||||
spec = importlib.util.spec_from_file_location("build_controller_bundle", BUILDER_PATH)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
builder = _load_builder()
|
||||
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE project (
|
||||
id INTEGER PRIMARY KEY, name TEXT NOT NULL, source_path TEXT NOT NULL,
|
||||
source_format TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
status TEXT NOT NULL DEFAULT 'initialized'
|
||||
);
|
||||
CREATE TABLE chapters (
|
||||
id INTEGER PRIMARY KEY, project_id INTEGER NOT NULL REFERENCES project(id),
|
||||
chapter_index INTEGER NOT NULL, title TEXT, source_text TEXT NOT NULL,
|
||||
word_count INTEGER NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'parsed',
|
||||
audio_path TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE TABLE utterances (
|
||||
id INTEGER PRIMARY KEY, chapter_id INTEGER NOT NULL REFERENCES chapters(id),
|
||||
character_id INTEGER REFERENCES characters(id),
|
||||
utterance_index INTEGER NOT NULL, text TEXT NOT NULL,
|
||||
type TEXT NOT NULL DEFAULT 'narration', context_before TEXT, context_after TEXT,
|
||||
emotion TEXT, pace TEXT, emphasis_words TEXT, pause_before_ms INTEGER,
|
||||
pause_after_ms INTEGER, instruct TEXT, speaker_name TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'pending'
|
||||
);
|
||||
CREATE TABLE characters (id INTEGER PRIMARY KEY, project_id INTEGER NOT NULL,
|
||||
name TEXT NOT NULL, aliases TEXT NOT NULL DEFAULT '[]', gender TEXT,
|
||||
is_narrator INTEGER NOT NULL DEFAULT 0);
|
||||
"""
|
||||
|
||||
|
||||
def make_project(root: Path, *, narrator_spellings: tuple[str, ...] = ("narrator",),
|
||||
null_narrator_rows: int = 2, chapters: int = 3) -> Path:
|
||||
"""A project shaped like the real one, including its awkward edges."""
|
||||
|
||||
umbra = root / ".umbra"
|
||||
(umbra / "chapter_audio").mkdir(parents=True)
|
||||
(umbra / "voices").mkdir(parents=True)
|
||||
db = umbra / "project.db"
|
||||
conn = sqlite3.connect(db)
|
||||
conn.executescript(SCHEMA)
|
||||
conn.execute(
|
||||
"INSERT INTO project (id, name, source_path, source_format, status) "
|
||||
"VALUES (1, ?, ?, 'docx', 'analyzed')", (root.name, str(root)))
|
||||
for index in range(chapters):
|
||||
conn.execute(
|
||||
"INSERT INTO chapters (id, project_id, chapter_index, title, source_text, word_count) "
|
||||
"VALUES (?, 1, ?, ?, 'body text', 10)",
|
||||
(index + 1, index - 1, f"Chapter {index}"))
|
||||
utterance_id = 0
|
||||
for chapter in range(1, chapters + 1):
|
||||
for position in range(2):
|
||||
utterance_id += 1
|
||||
is_narrator_row = position == 0
|
||||
speaker = narrator_spellings[chapter % len(narrator_spellings)] \
|
||||
if is_narrator_row else "Kainda"
|
||||
conn.execute(
|
||||
"INSERT INTO utterances (id, chapter_id, utterance_index, text, "
|
||||
"type, speaker_name) VALUES (?, ?, ?, ?, ?, ?)",
|
||||
(utterance_id, chapter, position,
|
||||
f"line {utterance_id}", "narration" if is_narrator_row else "dialogue",
|
||||
speaker))
|
||||
# A row whose speaker is expressed by being empty, as the real data does.
|
||||
for _ in range(null_narrator_rows if chapter == 1 else 0):
|
||||
utterance_id += 1
|
||||
conn.execute(
|
||||
"INSERT INTO utterances (id, chapter_id, utterance_index, text, "
|
||||
"type, speaker_name) VALUES (?, ?, ?, ?, 'narration', NULL)",
|
||||
(utterance_id, chapter, 99, "implicit narration"))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
(umbra / "speakers.json").write_text(json.dumps({
|
||||
"speakers": [
|
||||
{"primary_name": "narrator", "is_narrator": True, "aliases": [],
|
||||
"chapters_appeared": [1, 2, 3], "has_voice_ref": True},
|
||||
{"primary_name": "Kainda", "aliases": ["Princess Kainda"],
|
||||
"chapters_appeared": [1], "has_voice_ref": True},
|
||||
],
|
||||
"book_title": "Test Book", "total_chapters": chapters, "total_words": 30,
|
||||
}))
|
||||
(umbra / "voice_status.json").write_text(json.dumps({"narrator": "approved"}))
|
||||
(umbra / "voice_engines.json").write_text(json.dumps(
|
||||
{"Kainda": {"engine": "qwen", "reason": "ok", "at": "2026-10-10T00:00:00+00:00"}}))
|
||||
(umbra / "voice_assignments.json").write_text(json.dumps({"narrator": "en-US-GuyNeural"}))
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def built(tmp_path: Path) -> Path:
|
||||
make_project(tmp_path / "TestProject")
|
||||
out = tmp_path / "bundle"
|
||||
builder.build("TestProject", out, "2026-10-14T00:00:00Z",
|
||||
VALIDATOR_SOURCE if VALIDATOR_SOURCE.exists() else None,
|
||||
source_root=tmp_path)
|
||||
return out
|
||||
|
||||
|
||||
def read_manifest(bundle: Path) -> list[dict[str, Any]]:
|
||||
return [json.loads(line) for line in
|
||||
(bundle / "manifest.jsonl").read_text().splitlines() if line.strip()]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# the controller's own validator
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.skipif(not VALIDATOR_SOURCE.exists(),
|
||||
reason="controller validator not fetched to /tmp")
|
||||
def test_controller_validator_accepts_the_bundle(built: Path) -> None:
|
||||
validator = built / "validate.py"
|
||||
if not validator.exists():
|
||||
pytest.skip("validator was not copied into the bundle")
|
||||
result = subprocess.run([sys.executable, str(validator), str(built)],
|
||||
capture_output=True, text=True)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
|
||||
|
||||
@pytest.mark.skipif(not VALIDATOR_SOURCE.exists(),
|
||||
reason="controller validator not fetched to /tmp")
|
||||
@pytest.mark.parametrize("mutation,expect", [
|
||||
("dangling_id", "not in the manifest"),
|
||||
("required_file_removed", "missing required file"),
|
||||
("unitless_duration", "no unit"),
|
||||
("credential", "credential"),
|
||||
# The index declares relative_to required, so that check fires before the
|
||||
# convention check does; either message would be correct.
|
||||
("path_without_base", "relative_to"),
|
||||
])
|
||||
def test_controller_validator_rejects_a_broken_bundle(built: Path, mutation: str,
|
||||
expect: str) -> None:
|
||||
"""A validator nobody has seen fail is not evidence of anything.
|
||||
|
||||
Each case breaks one thing the request asks for and asserts the validator
|
||||
notices, so a passing bundle means something.
|
||||
"""
|
||||
|
||||
validator = built / "validate.py"
|
||||
if not validator.exists():
|
||||
pytest.skip("validator was not copied into the bundle")
|
||||
lines = (built / "manifest.jsonl").read_text().splitlines()
|
||||
|
||||
if mutation == "dangling_id":
|
||||
record = json.loads(lines[0])
|
||||
record["chapter_id"] = "ch-does-not-exist"
|
||||
lines[0] = json.dumps(record, sort_keys=True, separators=(",", ":"))
|
||||
elif mutation == "unitless_duration":
|
||||
record = json.loads(lines[0])
|
||||
record["duration"] = 1.5
|
||||
lines[0] = json.dumps(record, sort_keys=True, separators=(",", ":"))
|
||||
elif mutation == "credential":
|
||||
record = json.loads(lines[0])
|
||||
record["token"] = "sk-" + "A" * 24
|
||||
lines[0] = json.dumps(record, sort_keys=True, separators=(",", ":"))
|
||||
elif mutation == "path_without_base":
|
||||
for index, line in enumerate(lines):
|
||||
record = json.loads(line)
|
||||
if record.get("kind") == "render":
|
||||
record.pop("relative_to", None)
|
||||
record["file"] = "x.wav"
|
||||
lines[index] = json.dumps(record, sort_keys=True, separators=(",", ":"))
|
||||
break
|
||||
else:
|
||||
pytest.skip("no render record to break")
|
||||
|
||||
if mutation == "required_file_removed":
|
||||
(built / "VOCABULARY.md").unlink()
|
||||
else:
|
||||
(built / "manifest.jsonl").write_text("\n".join(lines) + "\n")
|
||||
|
||||
result = subprocess.run([sys.executable, str(validator), str(built)],
|
||||
capture_output=True, text=True)
|
||||
assert result.returncode == 1
|
||||
assert expect in result.stdout
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# structure and self-consistency
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_bundle_has_every_required_piece(built: Path) -> None:
|
||||
for name in ("README.md", "bundle.json", "VOCABULARY.md", "manifest.jsonl"):
|
||||
assert (built / name).exists(), name
|
||||
assert (built / "schemas").is_dir()
|
||||
assert (built / "examples").is_dir()
|
||||
examples = [p for p in (built / "examples").iterdir() if p.is_dir()]
|
||||
assert examples
|
||||
for book in examples:
|
||||
assert (book / "manifest.json").exists()
|
||||
assert (book / "lineage.json").exists()
|
||||
|
||||
|
||||
def test_content_hash_describes_the_file_it_ships_with(built: Path) -> None:
|
||||
import hashlib
|
||||
index = json.loads((built / "bundle.json").read_text())
|
||||
actual = hashlib.sha256((built / "manifest.jsonl").read_bytes()).hexdigest()
|
||||
assert index["content_hash"]["value"] == actual
|
||||
assert index["counts"]["records_total"] == len(read_manifest(built))
|
||||
|
||||
|
||||
def test_build_is_deterministic(tmp_path: Path) -> None:
|
||||
"""Regenerating an unchanged project must produce the same bytes.
|
||||
|
||||
Without this, the content hash is decoration: it changes on every run for
|
||||
reasons unrelated to the data, and comparing two builds tells you nothing.
|
||||
"""
|
||||
|
||||
make_project(tmp_path / "TestProject")
|
||||
first = tmp_path / "one"
|
||||
second = tmp_path / "two"
|
||||
builder.build("TestProject", first, "2026-10-14T00:00:00Z", None, source_root=tmp_path)
|
||||
builder.build("TestProject", second, "2026-10-14T00:00:00Z", None, source_root=tmp_path)
|
||||
assert (first / "manifest.jsonl").read_bytes() == (second / "manifest.jsonl").read_bytes()
|
||||
|
||||
|
||||
def test_every_reference_resolves_including_unvalidated_fields(built: Path) -> None:
|
||||
"""The controller checks some reference fields; these are the rest.
|
||||
|
||||
`speaker_id` and `render_id` are not on its list, but a dangling speaker
|
||||
reference is just as wrong, so it is checked here rather than assumed.
|
||||
"""
|
||||
|
||||
records = read_manifest(built)
|
||||
known = {r["id"] for r in records}
|
||||
for record in records:
|
||||
for field in ("book_id", "chapter_id", "speaker_id", "render_id", "voice_id"):
|
||||
value = record.get(field)
|
||||
if isinstance(value, str):
|
||||
assert value in known, f"{record['id']}.{field} -> {value}"
|
||||
for field in ("chapters", "renders", "inputs", "published_artifact_ids"):
|
||||
for value in record.get(field, []) or []:
|
||||
assert value in known, f"{record['id']}.{field} contains {value}"
|
||||
|
||||
|
||||
def test_no_field_states_a_duration_without_a_unit(built: Path) -> None:
|
||||
forbidden = {"duration", "length", "elapsed", "runtime", "time"}
|
||||
for record in read_manifest(built):
|
||||
for field in record:
|
||||
assert field.lower() not in forbidden, f"{record['id']} has {field}"
|
||||
|
||||
|
||||
def test_every_path_declares_what_it_is_relative_to(built: Path) -> None:
|
||||
for record in read_manifest(built):
|
||||
if "path" in record:
|
||||
assert record.get("relative_to") == "book_directory", record["id"]
|
||||
|
||||
|
||||
def test_no_absolute_host_paths_leak_into_the_bundle(built: Path) -> None:
|
||||
"""The bundle is committable and must not describe the machine layout."""
|
||||
|
||||
blob = "\n".join((built / "manifest.jsonl").read_text().splitlines())
|
||||
for leak in ("/Volumes/", "/Users/", str(REPO)):
|
||||
assert leak not in blob, f"absolute path leaked: {leak}"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# the findings this build exists to surface
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_narrator_written_three_ways_is_resolved_and_marked(built: Path) -> None:
|
||||
"""An empty speaker name means the narrator; it must be labelled, not dropped."""
|
||||
|
||||
records = read_manifest(built)
|
||||
utterances = [r for r in records if r["kind"] == "utterance"]
|
||||
implicit = [u for u in utterances if u["speaker_resolution"] == "implicit_narrator"]
|
||||
assert implicit, "the fixture is supposed to contain null-speaker rows"
|
||||
for utterance in implicit:
|
||||
assert utterance["speaker_name_raw"] in (None, "")
|
||||
assert utterance["speaker_id"] is not None, "an implicit narrator still has to resolve"
|
||||
for utterance in utterances:
|
||||
if utterance["speaker_resolution"] == "explicit":
|
||||
assert utterance["speaker_name_raw"] not in (None, "")
|
||||
|
||||
|
||||
def test_case_insensitive_spellings_are_not_silently_merged(tmp_path: Path) -> None:
|
||||
make_project(tmp_path / "TestProject", narrator_spellings=("Narrator", "narrator"))
|
||||
out = tmp_path / "bundle"
|
||||
builder.build("TestProject", out, "2026-10-14T00:00:00Z", None, source_root=tmp_path)
|
||||
speakers = [r for r in read_manifest(out) if r["kind"] == "speaker"
|
||||
and r["casefold_collision_group"] == "narrator"]
|
||||
assert len(speakers) == 2, "the two spellings must stay separate records"
|
||||
for speaker in speakers:
|
||||
assert speaker["utterance_count"] < speaker["utterance_count_casefold_grouped"]
|
||||
|
||||
|
||||
def test_chapter_numbering_hazard_is_declared(built: Path) -> None:
|
||||
book = next(r for r in read_manifest(built) if r["kind"] == "book")
|
||||
numbering = book["numbering"]["schemes"]
|
||||
assert numbering["database_row_id"]["is_authoritative"] is True
|
||||
assert numbering["chapter_index"]["is_authoritative"] is False
|
||||
assert numbering["distribution_track_number"]["is_authoritative"] is False
|
||||
|
||||
|
||||
def test_chapters_record_which_number_their_filenames_use(built: Path) -> None:
|
||||
for chapter in (r for r in read_manifest(built) if r["kind"] == "chapter"):
|
||||
basis = chapter["numbering_basis"]
|
||||
assert basis["file_names_use"] == "database_row_id"
|
||||
assert basis["chapter_index_is_authoritative"] is False
|
||||
|
||||
|
||||
def test_mismatched_extension_is_reported_not_hidden(built: Path) -> None:
|
||||
"""A `.wav` that is not a WAV must be visible on the book, not just per file."""
|
||||
|
||||
book = next(r for r in read_manifest(built) if r["kind"] == "book")
|
||||
for entry in book["extension_container_mismatches"]:
|
||||
assert entry["actual_container"] != entry["named_extension"]
|
||||
|
||||
|
||||
def test_speakers_without_a_profile_are_named_on_the_book(built: Path) -> None:
|
||||
book = next(r for r in read_manifest(built) if r["kind"] == "book")
|
||||
assert isinstance(book["speaker_names_without_a_profile"], list)
|
||||
for name in book["speaker_names_without_a_profile"]:
|
||||
assert not any(r["kind"] == "speaker" and r["primary_name"] == name
|
||||
and r["has_profile"] for r in read_manifest(built))
|
||||
|
||||
|
||||
def test_declared_and_measured_chapter_counts_are_both_reported(built: Path) -> None:
|
||||
book = next(r for r in read_manifest(built) if r["kind"] == "book")
|
||||
assert book["chapter_count"] == len(book["chapters"])
|
||||
assert book["declared_total_chapters"] is not None
|
||||
|
||||
|
||||
def test_language_designations_are_empty_and_explained(built: Path) -> None:
|
||||
"""Empty is a real answer here; an invented list would be a guess in a schema."""
|
||||
|
||||
book = next(r for r in read_manifest(built) if r["kind"] == "book")
|
||||
assert book["language_designations"] == []
|
||||
assert "do not record" in book["language_designation_note"] or \
|
||||
"records no per-utterance language" in book["language_designation_note"]
|
||||
|
||||
|
||||
def test_schemas_describe_every_kind_that_appears(built: Path) -> None:
|
||||
kinds = {r["kind"] for r in read_manifest(built)}
|
||||
for kind in kinds:
|
||||
assert (built / "schemas" / f"{kind}.schema.json").exists(), kind
|
||||
assert (built / "schemas" / "manifest.schema.json").exists()
|
||||
assert (built / "schemas" / "lineage.schema.json").exists()
|
||||
|
||||
|
||||
def test_required_fields_declared_in_the_index_are_actually_present(built: Path) -> None:
|
||||
index = json.loads((built / "bundle.json").read_text())
|
||||
records = read_manifest(built)
|
||||
for record in records:
|
||||
for field in index["required_fields"].get(record["kind"], []):
|
||||
assert field in record, f"{record['kind']} {record['id']} missing {field}"
|
||||
|
||||
|
||||
def test_utterance_text_hash_matches_its_own_text(built: Path) -> None:
|
||||
import hashlib
|
||||
for record in read_manifest(built):
|
||||
if record["kind"] == "utterance":
|
||||
assert hashlib.sha256(record["text"].encode()).hexdigest() == record["text_sha256"]
|
||||
assert len(record["text"]) == record["text_char_count"]
|
||||
|
||||
|
||||
def test_chapter_utterance_counts_are_true(built: Path) -> None:
|
||||
records = read_manifest(built)
|
||||
per_chapter: dict[str, int] = {}
|
||||
for record in records:
|
||||
if record["kind"] == "utterance":
|
||||
per_chapter[record["chapter_id"]] = per_chapter.get(record["chapter_id"], 0) + 1
|
||||
for chapter in (r for r in records if r["kind"] == "chapter"):
|
||||
assert chapter["utterance_count"] == per_chapter.get(chapter["id"], 0)
|
||||
|
||||
|
||||
def test_speaker_utterance_counts_are_true_and_separate_from_the_inferred_one(built: Path) -> None:
|
||||
records = read_manifest(built)
|
||||
per_speaker: dict[str, int] = {}
|
||||
for record in records:
|
||||
if record["kind"] == "utterance" and record["speaker_resolution"] == "explicit":
|
||||
per_speaker[record["speaker_name_raw"]] = \
|
||||
per_speaker.get(record["speaker_name_raw"], 0) + 1
|
||||
for speaker in (r for r in records if r["kind"] == "speaker"):
|
||||
assert speaker["utterance_count"] == per_speaker.get(speaker["primary_name"], 0)
|
||||
assert speaker["utterance_count_including_implicit_narrator"] >= speaker["utterance_count"]
|
||||
|
||||
|
||||
def test_example_is_a_readable_subset_not_a_copy(built: Path) -> None:
|
||||
"""The example exists to teach the shape; a second copy of the data cannot."""
|
||||
|
||||
example = next(p for p in (built / "examples").iterdir() if p.is_dir())
|
||||
example_records = json.loads((example / "manifest.json").read_text())
|
||||
assert 0 < len(example_records) < len(read_manifest(built))
|
||||
lineage = json.loads((example / "lineage.json").read_text())
|
||||
assert lineage["edges"], "lineage with no edges teaches nothing"
|
||||
assert all(edge.get("transform") for edge in lineage["edges"])
|
||||
assert all(edge.get("derivation") in {"direct", "derived", "assembled", "transcoded"}
|
||||
for edge in lineage["edges"])
|
||||
|
||||
|
||||
def test_generator_never_writes_to_the_project(tmp_path: Path) -> None:
|
||||
"""The audio system keeps its data and keeps running; the build only reads."""
|
||||
|
||||
project = make_project(tmp_path / "TestProject")
|
||||
before = {p: p.stat().st_mtime_ns for p in sorted(project.rglob("*")) if p.is_file()}
|
||||
builder.build("TestProject", tmp_path / "bundle", "2026-10-14T00:00:00Z", None,
|
||||
source_root=tmp_path)
|
||||
after = {p: p.stat().st_mtime_ns for p in sorted(project.rglob("*")) if p.is_file()}
|
||||
assert before == after
|
||||
|
||||
|
||||
def test_generated_at_is_the_only_difference_between_two_timestamps(tmp_path: Path) -> None:
|
||||
make_project(tmp_path / "TestProject")
|
||||
first = tmp_path / "a"
|
||||
second = tmp_path / "b"
|
||||
builder.build("TestProject", first, "2026-10-14T00:00:00Z", None, source_root=tmp_path)
|
||||
builder.build("TestProject", second, "2026-10-15T00:00:00Z", None, source_root=tmp_path)
|
||||
|
||||
def strip(path: Path) -> list[str]:
|
||||
# Everything that is a build-time stamp, not a fact about the data.
|
||||
volatile = {"generated_at", "file_modified_at"}
|
||||
return [
|
||||
json.dumps({k: v for k, v in json.loads(line).items() if k not in volatile},
|
||||
sort_keys=True)
|
||||
for line in (path / "manifest.jsonl").read_text().splitlines()
|
||||
]
|
||||
|
||||
assert strip(first) == strip(second)
|
||||
|
||||
|
||||
def test_production_time_is_not_claimed_as_build_time(tmp_path: Path) -> None:
|
||||
"""`produced_at: <build time>` would be a claim about the audio, not the export.
|
||||
|
||||
A render record must carry when the *file* was written, or nothing at all.
|
||||
"""
|
||||
|
||||
make_project(tmp_path / "TestProject")
|
||||
out = tmp_path / "bundle"
|
||||
builder.build("TestProject", out, "2026-10-14T00:00:00Z", None, source_root=tmp_path)
|
||||
for record in read_manifest(out):
|
||||
assert "produced_at" not in record, record["id"]
|
||||
# And the build timestamp never leaks into a file-time field.
|
||||
for record in read_manifest(out):
|
||||
assert record.get("file_modified_at") != "2026-10-14T00:00:00Z", record["id"]
|
||||
Reference in New Issue
Block a user