main
39 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Umbra Tools
|
ff73c99f8d |
Show the reviewer why a translation lane has a gap, not just that it has one
A delivered gap is not a missing translation. The delivery arrives and says this target language does not voice this utterance, with a reason - the translation renders a different paragraph, say. That is a different problem with a different fix, and the reviewer needs to be able to tell them apart. The reason was being thrown away. The row is stored with an empty text and a review_reason; the resolver skipped every row with empty text, so the gap was dropped and the review lane said "Chinese has not been translated yet". That is a false statement about a translation that was delivered and rejected. So the resolver now keeps a not-voiced row as a gap with its reason, the chapter page renders it as a gap rather than as missing text, and the language page lists the delivered gaps under "Needs attention" with their reasons. A gap row is also no longer counted as translated text, so the coverage the reviewer sees is the coverage the book has. 4 new tests. Suite: 913 passed, 8 failed (the same 8 pre-existing failures). Also adds scripts/probe_cross_language_tts.py: the open prerequisite for any target-language audio lane is whether Qwen3-TTS can speak Chinese with an English voice reference. The probe answers it by transcription rather than by listening - Whisper transcribes each clip and reports what language it heard, and the English-reference clip reading Chinese is compared against the same clip reading Chinese with English phonology, so the only variable is the language parameter. |
||
|
Umbra Tools
|
3db6ac3b8b |
Refuse a delivery whose text is not written in the language it claims
The production directory oldbox calls "deal-hunter-es" declares target_language "es-ES", its glossary says Spanish, and its 32-chapter run finished - and all 3,038 translated paragraphs are in Chinese. It is a second Chinese draft filed under a Spanish name. The cause is in the pipeline: the drafter's instructions are hard-coded to Simplified Chinese in three places (SYSTEM_DRAFTER, draft_prompt, SYSTEM_CRITIC), so the glossary's target language was ignored. No non-Chinese translation has ever been produced by that pipeline. The danger at this end is that the rows would be stored under "es", the review lane would look populated, and a reviewer would approve Chinese as Spanish. So the ingest now checks the script of the delivered text against the declared language, and refuses the delivery before writing a single row. The check is about the script, not the language: it cannot tell Spanish from Portuguese, but it can tell Chinese from Spanish, which is the mistake that actually happens. Japanese needs its kana and Korean its Hangul, and a Chinese delivery carrying kana is a Japanese translation filed in the wrong lane. An unknown language is not checked - there is no expectation to check it against. Proven on the real data: the es delivery is refused with "the text is mostly Han characters, so it is not Spanish", and the 4,300-row Chinese delivery still imports unchanged. 19 new tests. Suite: 909 passed, 8 failed (the same 8 pre-existing failures). |
||
|
Umbra Tools
|
301ff17f3e |
Replace a translation delivery instead of merging into it
A re-delivery could leave a stale row behind for an utterance the new delivery does not cover: the upsert only touches rows the delivery mentions. --replace clears the language's draft rows first, so what the reviewer sees is what the delivery actually says. Confirmed rows are never deleted. A confirmed translation is an approval, and the import skips confirmed rows anyway, so the approved text survives a re-delivery. 2 new tests. |
||
|
Umbra Tools
|
c476cfea1d |
Ingest the translation delivery, and stop mis-reporting locale codes
UmbraTrans delivers one row per utterance per target language, in the shape translations.import_translations already accepts. There was no script to run that ingest, so the delivery side of the contract existed only as a function. scripts/import_translations.py runs it: report-only by default, --write to store, --confirm to store as confirmed rather than draft. It prints the coverage before and after, so a delivery can be checked without writing it. Two bugs in the ingest, both found by running a real delivery: * import_translations folded the target language with lang.normalize, which does not know script subtags. A delivery labelled "zh-Hans" - which is what UmbraTrans writes - matched nothing, so every row was dropped and reported as a problem. It now folds with canonical_code, and an unknown language is still stored: the text is reviewable even when the render will be refused. * coverage and not_voiced_rows compared the stored code to the requested code as strings, so a row stored as "zh-Hans" read as "nothing delivered" when the reviewer asked about "zh". The coverage report was wrong in the direction that makes a reviewer distrust it. Measured on the real Deal Hunter Chinese delivery: 4,300 rows ingested, the review UI's resolver then renders Chinese for 12/12 utterances of chapter 1 and 415/416 of chapter 6 - the missing one being the spoken chapter announcement, which is deliberately not part of the reading edition. spec 27 records what the shipped samples actually are: single-pass drafts made against a hand-cleaned paraphrase of the English, matching the speaker-verified database text in 0 of 12 utterances. The production Chinese edition on oldbox does use the intent-based process; what it lacked was the alignment back onto the utterances, which is now implemented in UmbraTrans. 7 new tests. Suite: 890 passed, 8 failed (the same 8 pre-existing failures). |
||
|
B.D. Murphy
|
bf83bb35cc |
Merge minieq/main: Apache-2.0 LICENSE, NOTICE and SPDX headers
The remote had one commit this branch did not (
|
||
|
B.D. Murphy
|
2b294d7fa4 |
Per-utterance language flag + a working language selector (spec 31)
The review UI's language dropdown did nothing: every selection kept rendering the English text. Two independent causes, both fixed. 1. The translations were never being read. The pipeline's `utterance_translations` table holds one row in Deal Hunter, and that row stores the *English* text under a `zh-Hans` code. The real translations live in the on-disk sample files under `samples/translations/`. `umbraaudio/translations.py` now resolves both sources and merges them by (chapter_id, utterance_index): the table wins where it holds a real translation, the sample files fill the gaps. A candidate whose text is identical to the source is a placeholder, not a translation, so it is ignored - otherwise the page would show English under a Chinese heading and claim it was translated. 2. The query that read them was broken. `language_routes.py` selected `u.index` and `u.speaker`; the columns are `utterance_index` and `speaker_name`, so every read raised `sqlite3.OperationalError: near "index": syntax error` and `/language-mockup` returned 500. It also opened `<project>/project.db` - a 0-byte decoy - instead of `<project>/.umbra/project.db`, the file that actually holds the data. Also in this commit: * The chapter review page honours `?lang=`, remembers the choice in a cookie (set by a middleware, so it survives on every page), and shows the selected language's *name* in the "has not been translated yet" message. * The page now shows the translation, not the source, and the edit box is bound to the source text so saving cannot overwrite the English with the Chinese. * The stray language selector that sat after `</html>` in base.html is gone; the chapters-list page's dropdown had zero options because the route never passed the language list. * The language tag is wired into generation: `utterance.language_code` (NULL = project source language) reaches `engine.generate()` and the staleness marker, so a retagged line is re-rendered instead of keeping English phonology behind an unchanged text. The marker records the language; a marker written before the tag existed means English, so the ~4,300 existing clips are not invalidated. * `project.source_language` (spec 31 section 13.1) records the book's own language, with a migration for existing project DBs. * A scene-break marker deliberately records no language: it is silence, identical in every language, and tagging it would claim a phonology the audio does not have. Tests: 883 passed / 8 failed - the same 8 pre-existing failures (2 test_pronunciation, 6 voice_profile_manager accent). 26 new tests cover the resolver and the generation wiring. ruff introduces no new findings. |
||
|
umbra audio
|
03f8a858dd | Add Apache-2.0 LICENSE, NOTICE, and SPDX headers | ||
|
B.D. Murphy
|
9dfa7915ee |
Probe suite: T06/T07 now test, T08 names its real blocker
The dry run passes since routes_compute.py:1792 was aligned with read_bundle (separators + ensure_ascii=False), so three tests had to catch up. All three carried narratives written when the bundle was rejected, and each kept asserting a reason that had stopped being true — a BLOCKED test is a finding, so a stale BLOCKED reason is a finding about the wrong world. T06 re-runs the dry run and compares responses, proving validation is deterministic. It says plainly that re-import suppressing stored rows is NOT covered: that path needs rows that persist, and the import still builds InMemoryAudioImportStore. T07 reconciles the controller's counts against the bytes sent. 8884 records in, 8883 accounted for, plus the book record it does not count — so a controller that silently dropped a record kind now fails here instead of passing quietly. T08 keeps BLOCKED, but for the reason that is actually true: no Postgres AudioImportStore adapter exists, so even a successful commit would return a uuid4 project_id while translation_utterances stayed empty. Its text names InMemoryAudioImportStore on purpose, so wiring an adapter fails a test and forces the verdict to be re-decided. Suite: 9 PASS / 0 FAIL / 1 BLOCKED (T08). Tests 30 -> 40. |
||
|
B.D. Murphy
|
0909d050d9 |
Probe the route correctly: credentials, CSRF ordering, and a real dry run
The controller's ingest endpoint came up between runs and this tool misreported it three more times, each in the same direction — naming the state it was in when the check was written rather than the state it was reading. * POST was probed without a credential, so a live route kept answering 401 and the verdict stayed AUTH_REQUIRED after auth was configured and T02 passed. The probe now sends whatever headers exist, because an unauthenticated probe answers "is there a door", never "does it open". * The rule for reading a status was wrong twice. First it accepted only 200/201/405 as routed, so a 401 on a live route read as absent. Then it was widened to "anything but 404", which would have read a CSRF 403 as a route: the controller checks CSRF in middleware before routing (app.py:246-248) and exempts only /api/node (app.py:89), so POST /api/compute/audio/import answers 403 whether or not a route exists — and none does. Now: on a safe method, 404 means unrouted and anything else means routed; on an unsafe method a 403 off an exempt prefix is discarded and recorded as csrf_masked_probe. * T04 dropped csrf_masked_probe from its FAIL evidence, which is exactly the case where the next reader needs it. Found by the new tests, not by reading. * T05 returned BLOCKED unconditionally, so the one test that mattered was the least willing to run. It now executes a real dry run against --bundle. dry_run short-circuits before store.commit(), so running it cannot write. * authenticate() sent X-Node-Key, but require_node parses only "bearer " (routes_compute.py:477) — a valid key reported as rejected. It also probed /api/auth/me, which resolves a browser session, so no node credential could ever pass it. Now Bearer on /api/node/agent, and 403 there counts as authenticated-but-unscoped, since the scope check runs after resolution. * _summarise reduced a plain-text body to "<str>", discarding "Internal Server Error" and every handler message — the detail worth reading. * Four verdict names, including INGEST_ROUTED_NEEDS_RESTART, kept describing a restart that had already happened. _readiness now derives the name from evidence held by this run, and a real dry run outranks every transport observation: the current verdict is DRY_RUN_REJECTED. Measured against the controller: migration 0008 applied (27 -> 31 tables, guard proven to fire, idempotent on re-run), service restarted to PID 392511, endpoint routed and parsing. The dry run executes and is rejected on content_hash, because the HTTP path re-serializes records (json.dumps default separators, ensure_ascii=True) while their own read_bundle() hashes the file bytes — the same bundle yields d0011ae6 over HTTP and 21ce5687 on disk, and their validator reports 0 problems on disk. Reported to the controller; bundle hash unchanged. Also reported: routes_compute.py:1809 constructs InMemoryAudioImportStore, whose commit() appends to a list, and no PostgreSQL adapter exists — so a successful commit would return a uuid4() project_id while all four translation_* tables stayed empty. Read from source, not reproduced. 30 new tests pin the routing rule, the CSRF ordering, the credential header, and every verdict branch. Full suite 799 passed / 8 failed (pre-existing voice_profile_manager + test_pronunciation). ruff and mypy clean. |
||
|
B.D. Murphy
|
4c1e6ebc98 |
The TTS document, measured, with five honest unknowns
The controller will not write a `speak` work function against a description, so
this measures instead of describing. scripts/build_tts_facts.py emits
docs/tts/tts-engine.{md,json} from the running system and the repository at run
time; anything unmeasurable says "unknown" with a reason. The content hash
excludes the timestamp, so it is stable across runs and answers "did anything
change?" rather than "was it run again?". 18 tests enforce that it does not
estimate -- including one that fails if a missing weights file is rendered as
"None", and one that fails if the Edge-voice answer is inverted, which it was, in
the first draft.
Four things that change the controller's plans:
* The fork is ONE commit ahead of upstream and it is cosmetic -- a logo and the
web demo. The contract being written against is not the fork, it is
umbraaudio/scripts/qwen_tts_server.py, where the HTTP surface and the four
generation paths were written.
* Synthesis is not deterministic. temperature 0.9, do_sample true, no seed. A
translation re-voiced because one line changed will not reproduce the other
lines byte-for-byte, so comparing takes needs an embedding measure, not a hash.
* The TTS server has NO authentication and binds 0.0.0.0:8881, and concurrency
is 1 behind a single lock. The generator names the seven auth mechanisms it
searched for so the claim can be checked rather than believed.
* assigned_edge_voice is the fallback engine for creating reference audio, not
the engine that renders utterances.
Not measured, and said so: working set / VRAM (the one that decides the 48 GB
CPU node question -- 9.5 GB of weights on disk is not the memory the process
holds, and on CPU that gap is the whole question); whether a seeded call is
reproducible; per-language quality; the model's own maximum text length; CUDA
failure modes.
scripts/build_scale_totals.py measures across all projects rather than handing
over a typed number: 4 projects, 131 chapters, 18,181 utterances, 14,526 audio
files, 6.98 GB, 38.58 hours, and 2 distinct schema fingerprints -- matching the
two already recorded in manifest.md. Markers are counted separately from audio,
because "files" that means both is a number nobody can use. Treasure Island
contributes 49 chapters and zero audio: that is a fact about it, not a failed
measurement.
Also delivered to the controller: their accepted-format validator accepts the
bundle, exit 0, 8884 records, and I confirmed it fails on a broken copy before
believing the pass.
|
||
|
B.D. Murphy
|
3d69187449 |
Answer the controller's data request, and audit what we send
The UmbraTrans controller asked (commit 70c23fd) for machine-readable metadata about the audio system, in a specified layout, with a validator that ships with the request so the sender can check its own work. That request is answered here. scripts/build_controller_bundle.py emits that bundle: one real book (Deal Hunter), 8,884 records, no media, no credentials, no hostnames. It passes the controller's own validator, exit 0. Three things in this system will break a mapping that does not know them, and the bundle's real purpose is to say so precisely rather than to look tidy: * A chapter has THREE numbers. The database row id is what filenames use, chapter_index is reading order, and the distribution track number is a third thing again. Only the row id survives an edit. Front and back matter are numbered with roman numerals in the MP3 filenames. * The narrator is written three ways, and the most common way is nothing at all. 1,803 utterances have a NULL speaker_name, meaning narrator; 614 say "narrator" and 15 say "Narrator". A join that skips nulls loses most of the narration silently. Utterances now carry speaker_name_raw, the resolved name, and speaker_resolution, and the two spellings are not merged - that is a decision for whoever owns speaker identity, not something an export should do quietly. * A .wav file is not always a WAV. 30 of the 91 voice reference files are MPEG. Every render reports the container measured from the bytes, with extension_matches_container, so a reader cannot be misled by the name. A validator proves the bundle is internally consistent; it cannot prove the bundle is true. scripts/audit_controller_bundle.py re-measures every hash, byte size, duration and count against the project on disk - 66,011 claims, all holding. It caught two real defects in the first draft of this bundle, including one of my own: produced_at was stamped with the build time, which is a claim about the audio that nothing supported. The validator was also run against seven deliberately broken bundles to confirm it can fail. A validator nobody has seen fail is not evidence of anything; those cases are now tests. scripts/audio_controller_connect.py runs the T01-T10 suite: 4 pass, 1 fails, 5 are blocked. T04 fails honestly - there is no ingest endpoint, and nothing was pushed. T05-T08 are blocked because the controller's own plan sequences ingest after the metadata arrives; a proposed contract is included rather than an invented endpoint. T02 is blocked: no node key or service token is configured here, and an operator has to supply one. Two corrections to the node profile, which the controller reads: it claimed the web UI has no authentication (a session gate was added 2026-10-09) and that the controller host is offline (it is reachable and serving umbratrans-web 0.1.0). Full suite 751 passed, 8 failed - the same pre-existing pronunciation and accent failures, unchanged. ruff and mypy clean on everything added. |
||
|
B.D. Murphy
|
9bda553311 |
Add a speaker-review audit: is the attribution data fit for translation?
Runs after a reviewer finishes a book and answers the two questions that
gate the next stage:
1. Are there utterances with no speaker?
2. Now that speakers have been defined, renamed, merged or deleted, are
there utterances that should change?
Read-only - it never edits the database, and a test asserts the files are
byte-identical afterwards. Output is a markdown table and a CSV, one row per
finding with chapter, utterance number and a text preview, so each item is
locatable rather than a count.
Deal Hunter, after the completed review (56 speakers, 0 still needing
review, 4319 utterances):
- Dialogue with no speaker: 0. The 1803 speakerless rows are narration
and scene breaks, which is correct and is reported as a number rather
than silently dropped.
- Utterances attributed to a deleted speaker: 0.
- One speaker stored under two spellings: 6 rows across 3 speakers
(narrator/Narrator, Kainda/Princess Kainda, Willamina Wake/Captain).
All resolve to one voice so the audio is correct, but a per-speaker
export would split them into six - caught before the translation stage,
not after.
- 6 speech-tag mismatches where the narration immediately before the line
names a different speaker.
- 10 dialogue rows voiced by the narrator: 9 are quoted labels ("Reactor.",
"operations") that are correctly read by the narrator, 1 (ch28 utt46) is
a real segmentation error. Classified advisory so the one real defect is
not buried under nine false positives.
build_alias_map moved from scripts/qa_attribution.py into attribution_qa.py
so the CLI and the audit cannot disagree about what a speech tag resolves to;
qa_attribution output verified byte-identical.
13 new tests. Suite: 721 passed, 8 pre-existing.
|
||
|
B.D. Murphy
|
970e6e884a |
Implement DELETE /api/speakers/{index}; the Delete button did nothing
The speakers page has carried a Delete button that called
DELETE /api/speakers/{index}. That endpoint did not exist - the server
answered 405 Method Not Allowed and the button silently did nothing.
Deleting is more than dropping an entry, so the endpoint refuses to orphan
dialogue: a speaker with utterances returns 409 with the count and the list
of possible targets, and the UI asks where that dialogue should go.
reassign_to then moves every utterance recorded under the speaker's name OR
any of its aliases - utterances are frequently stored under an alias - and
keeps the old name as an alias on the target so existing references still
resolve. A speaker with no dialogue is simply removed.
Also removes the speaker's own reference audio (never the shared narrator or
general ref) and its voice_status approval entry, so a speaker re-created
later cannot inherit an approval nobody gave for it.
The button is no longer hx-delete: htmx does not swap a 409, and deleting
renumbers every speaker after it, so the handler re-renders the whole list.
Verified live on Deal Hunter, reversibly: create throwaway -> delete (200,
gone) -> recreate with an utterance attached -> delete (409, count 1, 61
targets offered) -> reassign_to=Warbots (200, utterance restored, name kept
as alias) -> compared against a pre-test backup, speaker count and utterance
count identical.
10 new tests. Suite: 708 passed, 8 pre-existing.
|
||
|
B.D. Murphy
|
44d77cdc9b |
Resolve a queued voice to its speaker by alias, not just exact name
Every item in Deal Hunter's voice_queue.json had failed with
error: "'primary_name'". _speaker_info matched only an exact primary_name,
so any other spelling returned {} and build_profile_from_speaker subscripts
that key directly - the job died before synthesising a single sample and the
reviewer saw only an error tag.
That is a live trap for Save & Re-voice specifically: renaming a speaker and
then re-voicing it leaves the queued name matching nothing.
- Case-insensitive, whitespace-tolerant match, then an alias pass.
- An unknown name now raises LookupError naming the speaker, instead of a
KeyError about a dictionary key three frames away.
Fixed in the queue layer rather than voice_profile_manager.py, which has 6
pre-existing failures in that area and should not be disturbed by this.
8 new tests. Suite: 698 passed, 8 pre-existing.
|
||
|
B.D. Murphy
|
9328159ec1 |
Fix re-voice never firing, by driving it from the event that carries the form
Reported: Save & Re-voice still gave no response after the previous fix.
That fix bound hx-on::after-request to the form. It never fires. htmx
reports a different detail.elt for each phase of this request - measured in
a real browser:
beforeRequest FORM
beforeSwap TR
afterRequest TBODY
Row swaps get re-parented into a table context, so afterRequest is
dispatched on the enclosing TBODY. A listener on the form never sees it,
because the event does not bubble downwards. An earlier probe suggested the
attribute worked, but that probe used a form outside a table, where elt
stays the FORM - it did not reproduce the real page.
So the profile was being saved (the PUT fired and swapped) while the
re-voice never ran and nothing on screen confirmed anything. That is
exactly "no response".
- Capture the row index at beforeRequest, where detail.elt is still the
form, and match it to the completing request by path at afterRequest so
a different response can never trigger a re-voice for the wrong speaker.
- Saving now shows a confirmation toast. The button previously looked dead
even when the edit did save.
- Tests now guard the mechanism, including that hx-on::after-request is not
reintroduced on this form.
Verified by clicking the button in a real browser on Deal Hunter's Dealer:
POST /api/voices/queue with {"speaker_name":"Dealer","row_index":14} and
the confirmation shown. Suite: 690 passed, 8 pre-existing.
|
||
|
B.D. Murphy
|
305f6a3912 |
Fix Save & Re-voice silently discarding every speaker edit
Reported: editing a speaker, then "Save & Re-voice", did nothing.
This is a regression from
|
||
|
B.D. Murphy
|
2fda85f9d3 |
Make approval visible, and fix a real key-truncation bug in the auth gate
Follow-up to the badge fix: approving a speaker removed it from the needs-check panel with no confirmation, which reads exactly like the speaker being deleted. Reported as "the speaker advisor isn't shown now". - A person-approved speaker now renders "✓ reviewed" with a tooltip saying so. It previously rendered "✓ clear" whose tooltip claimed "auto-accepted" - a false claim about a decision a machine never made. - Approving confirms itself with a toast, drops the entry from the worklist, and updates the header counts, all without a reload. Separately, a pre-existing flake in tests/test_auth.py turned out to be a real defect rather than test noise: _session_secret() read the key file with .strip(), but bytes.strip() removes ASCII whitespace from both ends. A secret beginning or ending with one of those six bytes (4.35% of generated keys, measured) came back one byte shorter on every read after the write - a different HMAC key, silently, so a token signed before the flip could never be verified again. Now read verbatim, with a length floor to reject a corrupt file. That test also lacked the isolated_store fixture, so it read and rewrote the real .umbra/session_secret - running the suite mutated production state. This install's key is 32 bytes with no whitespace at either end, so no existing session is invalidated by the fix. Also verified: Advisor remains at display position 1 of 91 with its utterance (utt 3668, ch15) intact - only the warning badge changed. auth.py test flake 4/25 runs -> 0/30. Suite: 684 passed, 8 pre-existing. |
||
|
B.D. Murphy
|
a4c88f8bde |
Make speaker approval actually clear the badge, without navigating
Reviewer report: pressing Approve left the "needs check" badge in place and
appeared to navigate to a new page.
Root cause was a split brain, not a single bug. Approve writes the
voice_status.json ledger, but speaker_confidence read the voice_status key
embedded in speakers.json. That inline copy is stale - the web layer
overwrites it from the ledger on read (_load_speakers), so the page showed
the right value while the assessment computed from the old one. In Last
Dwarf, "Advisor" was 'pending' in speakers.json and 'approved' in the
ledger, so the reviewer's click had saved and still left it flagged.
- assess_speakers() takes the ledger as an explicit argument, authoritative
over the inline key; load_speaker_assessments() reads voice_status.json.
- New SpeakerAssessment.reviewed flag, surfaced in the serialized view.
- Approval clears the badge but keeps the reasons visible: the underlying
facts (role-only name, one utterance) stay inspectable on hover rather
than being silently discarded.
- Corrupt/ non-dict ledgers degrade to {} - this module never raises.
- Approve swapped #speakers-list with innerHTML, replacing the whole
reviewer list with a single <tr>; now swaps only its own row via
outerHTML on #speaker-row-N.
- The review panel explained only rename/merge and never mentioned Approve,
so the process was unclear; it now lists the three outcomes.
16 new tests. Suite: 681 passed, 8 pre-existing failures unchanged.
|
||
|
B.D. Murphy
|
e810b49d2a |
Add a voice-regeneration queue with inline per-speaker state
Reviewing speaker exceptions meant clicking Regenerate, watching a modal that
blocked the page, then navigating back to find out what happened. Worse, the
route refused a second request with a 409 while one was running, and
/api/generation/status exposed only a single global `active` flag - so there
was nowhere to put a second speaker and nothing to render per row.
Now: POST returns as soon as the task is captured, the row shows a status tag
(queued / working / done / failed), and further speakers can be queued while one
generates. No navigation, no modal.
umbraaudio/voices/voice_queue.py
- Ordered, serial queue. TTS is strictly serial here, so items run one at a
time in enqueue order; the second speaker waits rather than contending.
- Per-item state, persisted to voice_queue.json. An item left `running` by a
crash is marked `error` on load - the honest outcome, since silently
re-running could write audio nobody asked for twice.
- Enqueueing a speaker that is already pending is a no-op returning the
existing state, so a double-click cannot queue the same work twice.
- `cancel` withdraws a queued item but refuses one already running.
- Status vocabulary (queued/running/done/error) is deliberately small and
stable: it is what the controller will map onto node task states.
umbraaudio/web/voice_queue_routes.py - POST /api/voices/queue (capture the task),
GET /api/voices/queue (report per-item state), POST .../{speaker}/cancel. Kept
out of the hot routes.py, following name_token_routes.
/api/speakers/regenerate-voice is now a thin shim onto the queue so a
bookmarked call still works and the two paths cannot race. Net -24 lines in
routes.py; the blocking implementation is gone, not commented out.
The UI's modal, its 409 handler and the page-load resume handshake were removed
(67 lines) rather than left dormant - queue state is re-read from the snapshot
on load and after each HTMX row swap, so it survives navigation.
Unknown speaker names are rejected with a 404 at enqueue time. Previously they
reached the profile builder and surfaced as a bare KeyError('primary_name') in
the failed tag.
Verified live: two speakers enqueued back to back, first ran while the second
stayed queued, then the second started automatically on completion; both wrote
real ref WAVs. 16 new tests. Full suite 665 passed, 8 pre-existing failures.
|
||
|
B.D. Murphy
|
2749c5249f |
Add single-user login gate to the review UI
The UI had no authentication at all: `active_project` is an unsigned cookie and the server binds 0.0.0.0, so every page - including the mutation endpoints and the audio downloads - was reachable by anything on the LAN and, through the Caddy reverse proxy, from the internet. Spec 21/22 recorded this as an unauthenticated mutation surface; this closes it locally until the controller on oldbox takes over identity. Gate is middleware, not per-route dependencies, so a newly added route is protected by default and forgetting a decorator cannot expose the UI. It is fail-CLOSED: an exception while checking a session denies the request, because falling through to "no session required" would turn a bug into an open door. - umbraaudio/auth.py: PBKDF2-HMAC-SHA256 (stdlib, per the project's stdlib-before-third-party rule) with a random per-user salt. Sessions are HMAC-signed and carry no secret, just username and expiry. All comparisons use hmac.compare_digest. Credential writes are atomic (tmp + os.replace) so an interrupted save cannot lock the user out, and both the store and the session secret are chmod 0600 under .umbra/, which is gitignored. - First use: seeds username `bdumbra` with the one-time password `First321`, stored hashed and marked must_change, so the bootstrap password never lands on disk in plaintext and is dead after the first login. - Password rule: more than six characters, at least one letter and one special. - Sessions last 14 days, sliding. - Login failures are deliberately vague: a wrong username and a wrong password produce the same message, so the form cannot be used to enumerate accounts. Verified live: unauthenticated requests to /, /speakers, /name-tokens, the export endpoints and the ZIP download all 303 to /login; the first-use password forces /change-password before any other page; weak, mismatched and reused passwords are rejected; the old password stops working; tampered, forged and expired session tokens are all rejected. Static assets remain reachable so the login page renders. 19 new tests. Full suite: 649 passed, 8 pre-existing failures. |
||
|
B.D. Murphy
|
79832700f5 |
Fix the m4b to match the MP3 masters: 44.1 kHz mono, loudness, peak
export_m4b encoded with `-c:a aac -b:a 128k` and no `-ar`, so the delivered file came out at 24 kHz / ~96 kbps mono while the AR MP3 masters were 44.1 kHz / 192 kbps CBR mono. Measured on the shipped Deal_Hunter.m4b: 24 kHz, ~96 kbps, and peak -2.0 dB - failing the 44.1 kHz and -3 dB requirements at Author's Republic and Spotify. AudiobookExporter had zero loudness or peak handling of any kind. - _encode_aac_m4b now passes -ar 44100 and -ac 1 explicitly, and takes the spec constants from ar_export so the two paths cannot drift apart again. - Added normalisation to the RMS target and peak enforcement on the delivered file, plus m4b_loudness_report() so the result can be verified. - AAC overshoot is NOT monotonic. Measured +0.8, +0.9 then +1.6 dB across three passes, so trimming by the exact excess oscillated and exhausted the pass budget still breaching. Trimming by excess + 1.5 dB margin converges: verified -2.20 -> -4.5 dB in two passes. - _concat_wavs_with_silence now targets 44.1 kHz rather than the TTS-native 24 kHz. - Closed spec 22 gap 1: _stitch_chapter hard-coded silence_between_chapters_ms=0 while ChapterBuilder's AudioStitcher defaults are 500/4000, so the packaged book had different silence than the audio a reviewer approved. Trade-off, flagged rather than hidden: the 1.5 dB overshoot margin costs ~1.3 dB of RMS, so a trimmed m4b measures about -22.8 dB against the -20.5 target. Still inside the -23..-18 band, but only ~0.2 dB from the floor; a chapter needing more than ~2.5 dB of trim would fall out of band and loudness_ok will flag it. Verified on real chapter audio: AAC 44100 Hz mono, RMS -22.8, peak -4.5. 6 new tests. Full suite: 630 passed, 8 pre-existing failures unchanged. |
||
|
B.D. Murphy
|
0962d821e6 |
Fix peak gate: measure and enforce the delivered file, not the intent
The compliance gate certified our intention rather than the artifact.
_encode_track measured the pre-encode float buffer, and _normalize
hard-clipped to exactly -3.0 dB, so all 32 shipped tracks recorded
peak_db = -3.0 and peak_ok = True. libmp3lame then overshot the ceiling, so
the MP3s a listener actually received peaked near -2.0 dB - above the -3 dB
limit that Author's Republic, INaudio and Spotify all require.
- _measure_file() measures an encoded file; the gate now reads the MP3.
- _encode_and_enforce_peak() re-encodes with the excess trimmed until the
FILE complies, so codec overshoot is corrected rather than assumed away.
Measured on real audio: -2.0 dB before, -3.1 dB after, and volumedetect,
ebur128 true peak and the gate all agree.
- RMS is still taken from the content before padding, because every
distributor defines RMS "across the entire program (not including room
tone)". Measuring the padded file read ~9 dB low (-29.4 dB on a title that
normalizes to -20.5). Peak comes from the encoded file, since silence
padding cannot raise a peak and only the file reveals the overshoot.
- Same fix applied to encode_chapter_mp3, which shared the defect.
- Closing credit is now emitted LAST. The roles loop emitted
("sample","opening","closing") before the chapters, so the end-of-book
credit sat at position 3, ahead of the story it was meant to conclude.
Tests: added coverage asserting the recorded peak equals an independently
measured peak of the delivered file, that no file exceeds the ceiling, and
that a clipping source is trimmed until it complies. Three existing
assertions had encoded the old behaviour and were corrected - including two
"-2.9" tolerances that existed only to accommodate the overshoot, and an
ordering assertion that pinned the closing track second. Those now assert the
real -3.0 limit and the correct order.
Full suite: 624 passed, 8 pre-existing failures unchanged.
|
||
|
B.D. Murphy
|
f19ada75fb |
Mechanical speech-evidence gate: catches the ch6 utt29 miss (spec 24 A)
The operator's chapter 6 review found a line attributed to Hallr that the text
gives to a scum boy, and no part of the system had flagged it. Spec 24 §2
diagnosed why: the duplicate scan only compares names that already exist, so it
cannot see a line given to the wrong EXISTING speaker. Spec 24 §4A proposed a
mechanical evidence gate. This is that gate, measured.
New umbraaudio/analysis/speech_evidence.py, model-free, plus
scripts/report_speech_evidence.py (read-only) and 44 tests.
For every dialogue line: extract the speaker the narration actually names, build
the candidate set from the cast rather than from the guess, and return
supported / contradicted / unsupported / unconstrained.
Last Dwarf, all 49 chapters, 3,918 dialogue lines:
supported 1,844 47.06%
unconstrained 2,032 51.86%
contradicted 34 0.87%
unsupported 8 0.20%
review queue 42 1.07%
It catches the chapter 6 defect exactly: "The leader on the right for Hallr
says," / "We want to be the ones to claim we killed the last dwarf" assigned to
Hallr, with the two leaders who could own it. attribution_qa reports 11 findings
book-wide and this one is not among them.
Two design decisions carried the precision, and both were forced by measurement.
First, "no tag" is not a finding: most dialogue has no tag because a tag
introduces only the first line of a conversation, so unconstrained and
unsupported are separate verdicts. The first working version queued 60% of the
book for this reason, which is the same as having no gate. Second, "ambiguous"
was removed as a verdict - it hid the case that matters, since a line assigned to
Hallr under "The leader on the right" is wrong no matter how many leaders exist.
Candidate count now decides whether a suggested fix is confident or a question.
Two corrections to spec 24, both from measurement. The 20 role-tag violations
reported in section 3 are really 11; my ad-hoc probe was worse than the checker
it was meant to replace. And identity coverage is already 100% - every dialogue
line resolves to a profile or alias - so the defect is never a missing speaker,
it is a line given to the wrong existing one.
Six extractor bugs the measurement exposed are now regression tests: a role
phrase with a spatial qualifier produced no tag at all; a coordinated predicate
spoke as its first subject instead of its last; a possessive voice resolved to a
profile called Voice; irregular plurals and superlatives never matched; and the
subject was read off trailing participial clauses. Two generalisable rules came
out of them - a speech tag's subject is a noun phrase and a prepositional
qualifier is still part of it, and a subordinating conjunction opens a clause of
its own ("The dragon swallows when Hallr says" is Hallr).
Reports carry chapter_id AND title because chapter_id and chapter_index differ
by one in every project here; a bare "ch8 utt264" sent the operator to the wrong
chapter, and the line they asked about is chapter 9 utt264.
Full suite 619 passed / 8 failed, the same 8 pre-existing failures. ruff clean.
|
||
|
B.D. Murphy
|
714b11367a |
Duplicate-speaker scan that refuses to eat an approved voice (spec 23)
Last Dwarf has accumulated duplicate speakers from successive discovery and
designation scans: The Girl / The Girl helping / One Girl Helping, Guard 1 / The
first guard, Walking oldest / The Oldest, plus three lowercase spoken names that
differ from their profiles by one letter. The operator wants them eliminated and
was explicit that created and approved voices must not be lost.
umbraaudio/analysis/speaker_dedup.py
Read-only fact gathering (profiles, ledger, voice refs, chapter spread) and a
matcher that explains itself: determiner and ordinal normalization, superlative
stemming, and three VETO rules, because string similarity alone would merge
people who are not the same person:
- distinguishing modifiers: The Oldest is not The Tallest; Guard 1 is not Guard 2
- role nouns: Soldier (Rider) is not One Rider
- proper names need ratio >= 0.88: Anfisa (0.77 from Tanisha) is a different
character; Rielan/Rieland at 0.93 is a misspelling
Verdicts auto / review / blocked. A group with two different voice references is
blocked, because merging would discard a generated voice. Survivor selection
prefers an approval, then a voice reference, then utterance volume.
Two rules were written and then removed, recorded in the spec so nobody re-adds
them: a "shared token plus role noun" rule proposed The Oldest = The Old Woman,
and a "single shared token" rule proposed Scum leader = The Leader. A missed
duplicate costs one manual merge; a wrong one silently rewrites a speaker.
Two real bugs, both found by running against a COPY of Last Dwarf rather than by
reading code:
1. orphan_resolution.merge_speakers never touches voice_status.json, so merging
an approved speaker into an unapproved target would have deleted the approval
record silently. merge_group() now carries the approval to the survivor,
drops the loser's entry, and reports approvals_carried.
2. The first re-point was unscoped and moved 5,981 rows instead of 3: `narrator`
is the labelling convention for ~5,978 NARRATION rows too, and renaming them
would invalidate thousands of clips because every marker records the speaker
exactly. The re-point is now dialogue-only. Narration casing is a separate,
deliberate decision and is flagged as such, not taken by a duplicate scan.
Verified on a copy: 93 -> 84 profiles, 23 utterances re-pointed, narration
untouched, 72 -> 72 voice ref files with zero removed, 13 -> 13 approvals with
zero lost.
Last Dwarf scan: 10 auto, 2 review, 4 blocked. Blocked groups are exactly the
ones the operator must decide - The Girl (two voices for one person), Officer,
the three-way Narrator family, and Border Soldier vs Soldier (Rider).
Nothing was applied to the live project. tests/test_speaker_dedup.py: 37 tests,
vetoes weighted as heavily as positives. Full suite 575 passed / 8 failed, the
same 8 pre-existing failures. ruff clean.
|
||
|
B.D. Murphy
|
9f3a0a4b18 |
Release comment as a release step; releasing grants implicit voice approval
Two corrections from the operator, and I had both wrong. 1. "I didn't explicitly approve all minor character voices. By releasing the audiobook I have given implicit approval those voices are ready and should be tagged." The previous session reported "24 of 61 speakers have an approval record; 38 do not" as if it were a gap. It is not a gap. Releasing IS the approval, and those voices were waiting to be tagged. chapter_approved and audiobook_approved now read true, with approved_by: "this publication record" - there is no separate audiobook approval in this system, and pretending otherwise understated the release. The boundary matters, so it is measured rather than assumed: implicit approval covers voices that HAVE LINES in the released chapters, computed from the database. A character who never speaks is not covered, because releasing a book does not approve a voice it never uses. On Deal Hunter edition 1 that is 24 explicitly approved, 29 implicitly approved, and 10 in-profile voices with no lines. The three categories are never summed - an explicitly approved voice can also be one that stopped speaking. grant_implicit_voice_approval() writes plain status strings, the shape the web UI already reads, and keeps the provenance in the immutable publication record rather than the mutable ledger, so it cannot be edited away. 2. The m4b inter-chapter gap should be the same 2 s as the lead and trail padding. export.py had gap_ms=4000 in one path and a default of 1000 in the other, while ar_export.SILENCE_MS = 2000 governed lead and trail: three drifting values for one concept. The gap now imports SILENCE_MS as CHAPTER_GAP_MS, so all three are one number that changes in one place. New umbraaudio/release_comments.py - the comment is a gate, not a caption. designator 2026-10-03 #2 14:32:05 slug release-20261003-143205-02 The sequence is per date, not global, which is the whole point: a problem found and corrected on the same day is unambiguous about which was last. Two comments authored in the same second still get distinct slugs, because the sequence is allocated by scanning the directory. First line is the summary, the rest is the body, like a commit message. An approved comment is FROZEN - changing an approved note is a different claim about a different state, so it needs a new comment and therefore a new publication. require_approved() blocks --publish: passing a draft exits 2 and says what to do next. Verified live. draft_comment_text() composes the comment from the measured build rather than from memory. Deal Hunter's draft is written and waiting: release-20261003-133508-01, 32 chapters, the 24/29/10 voice split, 32/32 loudness and peak gates green, and the post-approval padding note. New invariants R-8 through R-11 in spec 22. Two of my own test premises were wrong and the implementation was right - recorded in AGENTS.md because the tests were the weaker artefact. Full suite 538 passed / 8 failed, the same 8 pre-existing failures. ruff clean. Nothing was published: the Deal Hunter comment is a draft awaiting review, and voice_status.json is untouched because publishing never ran. |
||
|
B.D. Murphy
|
ed3e249d0b |
Add the publish flag: pinned release sets, immutability, the Deal Hunter lesson
The operator clarified the release model: the system keeps current and previous
state, there must be an explicit publish flag the user sets, there are approvals
for a speaker voice but none for the audiobook or chapters, the chapters that
make the m4b are the versions to flag, and a fix found after assembly causes a
change rather than editing what shipped.
Deal Hunter is the precedent, and it is measurable rather than theoretical.
Against the real package (built_at 2026-09-09T11:21:11Z), chapter 6:
ch6_full.wav 2227.26 s @ 24000 Hz what a reviewer listens to
Deal_Hunter_01_Salvage.mp3 2231.26 s @ 44100 Hz what a listener received
+4.00 s = 2 s lead + 2 s trail
plus a resample and a -20.5 dB RMS normalization. Found on 32 of 32 chapters.
The approved artifact and the published artifact were different files, and
nothing in the system recorded either fact. ar_package.json is rewritten by
every rebuild, so it is a build status, not a release record.
umbraaudio/release.py
DeliveryProfile.from_ar_package reads the profile the exporter actually used,
so the record follows the code instead of a declaration that can rot.
build_publication is read-only and pins every chapter wav, every mp3 and the
package by full SHA-256, plus a per-chapter source_unit_key_digest so a release
pins which source revision produced it, not merely which bytes.
verify_publication returns ok / drift / warnings and never silently accepts a
changed artifact. write_publication is atomic and refuses to overwrite an
existing publication id - immutability enforced by the writer, not by
convention. A later fix is edition n+1 with supersedes pointing at n.
The rule that prevents a repeat: you cannot approve a chapter and then pad it.
An approval must name the artifact a listener receives. Every record states
chapter_approved: false and audiobook_approved: false in its own words, because
that is what would have caught Deal Hunter.
The record also refuses to overstate approval. voice_status.json is an approval
LEDGER - it only holds speakers someone explicitly touched - so it has 24 entries
against 61 speakers in the profile. Reporting "24 approved" as if it meant "all
approved" would claim an approval nobody gave, so the counts are separate and the
38-speaker gap is named.
scripts/publish_release.py: dry run by default; --publish, --verify, --list,
--edition/--supersedes. Exits 2 while blockers exist.
Also found, reported and NOT fixed (specs/22 section 7 gap 1): export_m4b's
_stitch_chapter hard-codes 500 ms/0 ms silence while ChapterBuilder uses
AudioStitcher defaults and AudioConfig holds its own values, so the m4b path can
stitch a chapter with different silence than the approved one. Until that is
closed an m4b release record would pin the wrong bytes, so it is better not to
offer one yet.
And projects.status is never advanced to 'released' in any of the four live
projects - all sit at parsed/analyzed - so the publication record is currently the
only release signal in the system.
tests/test_release.py: 24 tests. Full suite 512 passed / 8 failed, the same 8
pre-existing failures. ruff clean. No export, TTS or project data was modified.
|
||
|
B.D. Murphy
|
829a03b89f |
Action the controller alignment proposal: source_unit_key + phase reports
The controller (oldbox) sent a data-model alignment proposal and asked for six
additive changes on this node, the validation protocol, and an honest report.
Six of the seven changes are done; the seventh is declared missing rather than faked.
Stage 0 - reachability verified exactly as documented: health 200, heartbeat 401
(the correct answer - auth is refusing), missing Host 400. Nothing else was sent.
source_unit_key (controller 5.1, the blocking item)
umbraaudio/source_units.py. sha256("uk1" | chapter_source_text_sha256 |
occurrence | text_sha256). Nothing positional in the key: "occurrence"
disambiguates repeated text in document order.
- 18,181/18,181 units keyed across all four projects, 0 collisions
- chapter 1 renumbered +100 -> 0 keys changed (the load-bearing proof)
- backfill 10,015 rows in 0.11s, second run stamps 0 (idempotent)
- alignment phase 3: 200/200 probe units resolved with used_index: false
- tolerates the older projects that have no chapters.chapter_index
Recorded at generation time in chapter_builder, BEFORE the rows are read and
BEFORE the marker is written, so every path that renders audio also produces
identity and the clip can name its source unit.
5.7 recorded: utterance_index is KEPT as a display ordinal and made
non-load-bearing. spec 21 §10.
Blob hashes and effect chain (5.2)
Markers now carry the full sha256 of the delivered bytes and the effects that
produced them. Verified the existing order is already correct - write_marker
runs AFTER apply_effects - and closed the remaining hole: a changed trait list
changed the hash with no recorded cause. sha256 verification is opt-in, since
hashing every file on every validation pass would re-read gigabytes.
Vocabularies published (5.4) - umbraaudio/definitions.py, 19 sets, 126 values,
each value carrying whether it is 'declared' (spec/controller) or 'extracted'
(read out of code), so a diff shows a difference of meaning, not of wording.
Alignment report tool (6.6) - scripts/alignment_report.py --phase all. Emits the
controller's envelope with the per-phase evidence shapes, and obeys the two
rules they set: no evidence means blocked, never pass; a field we cannot produce
is null PLUS a finding, never omitted. Phase 2 is deliberately 'fail' - the
three-way store disagreement is real, and a pass there would be a check that
passed by doing nothing.
Fingerprint discipline (5.5) - the drift detector reported its OWN fingerprint
change when source_unit_key was added rather than hiding it. Node profile
updated with the new fingerprints and an honest note that they moved.
Node report extended - per-project source-unit-key coverage, per-voice-asset
sha256 + object_key, definition_set_version, controller stage reached.
Six gaps found in the proposal (spec 21 §11), including the one they invited plus
five more. The sharpest: "hash at generation" ignores that audio is
post-processed afterwards, and our key is anchored to chapters.source_text - a
second copy of the manuscript - so any change to chapter storage silently
invalidates every key in that chapter.
5.6 deliberately NOT faked: personality has no versioned artifact here, only
unversioned LLM prose in speakers.json. Reported as phase 6b finding B-03
instead, because wrapping unversioned inputs in a version field would produce
artifacts claiming a lineage they do not have.
Answered the controller's two open questions (infra/controller-handoff-2026-10-03.md):
forward-only link for a revealed recurring character, and yes - a new edition may
contradict a frozen state, since this node's correction path IS a regeneration.
Full suite 484 passed / 8 failed - the same 8 pre-existing and unrelated failures.
ruff clean on every file touched. The parallel session's chapter_builder hunk
(helper_url) was filtered out of the commit and left intact in the working tree.
|
||
|
B.D. Murphy
|
4775e3b589 |
Add deterministic designation scanner; capture the ch5 speakers discovery missed
Ran discovery on Last Dwarf ch5 with the new over-detection prompt rules. It
improved (found 'The big one', 'The one on the left/right', 'The Leader',
'Girl helping' where it previously found none of them) but STILL dropped
'The oldest' and 'The tallest' from the same six-man crowd — the e4b model is
not deterministic enough to guarantee coverage.
So added a mechanical backstop that does not depend on the LLM at all:
- umbraaudio/analysis/designation_scan.py reads the three canonical
speech-attribution shapes ('The oldest says,' / '"I thought they died\n
off," says the tallest' / '"No," the old smith said.') and harvests every
designation a speaker is referred to by. Post-filters pronouns, narration
verbs ('Hallr Grips His Axe And'), finite-verb clauses, possessives
('Hallr's' -> 'Hallr', 'Sheyla's voice' -> 'Sheyla'), scene/internal labels,
indefinite collectives, and tags that fall inside a quotation (that is
someone being talked ABOUT, not speaking).
- scripts/merge_designations.py adds any designation the cast does not cover
as an 'unidentified' speaker with its evidence line. Never removes or
rewrites existing entries, atomic tmp+replace, idempotent (0 on re-run).
- scripts/run_discovery.py runs discovery to a SEPARATE output file so the
curated speakers.json is never overwritten by a discovery run.
Result on Last Dwarf: 56 designations, 33 already covered, 23 genuinely
missing — including the ch5 crowd (The Oldest, The Tallest, The Young One,
The One With The Biggest Sword, The Biggest One, The Leader, One Girl Helping)
plus Manfred, The Sergeant, One Witch, The Soldier and others. Cast 70 -> 93,
all new entries flagged 'needs check' at 0.00 confidence.
Also: scripts/run_discovery.py, and a 14-test suite covering the ch5 passage,
possessives, quoted narration, finite-verb clauses and post-quote lowercase
subjects. Full suite 461 passed (8 pre-existing/parallel failures unchanged).
|
||
|
B.D. Murphy
|
f7094e2b1f |
Discovery: over-detect speakers, and clean up their names
The reviewer found obvious speakers missing from Last Dwarf's cast that had to
be added by hand: the six men in ch5 identified only as 'the oldest', 'the
tallest', 'the young one', 'the one with the biggest sword' and 'the biggest
one', plus the scum group. The prompt had no rule requiring these to be
captured, and nothing stopping narration verbs from being baked into names
(the result: 'Walking tallest', 'Walking oldest', 'Walking Youngest').
- Rule 8 'Every Speaking Line Is a Speaker': a character who speaks any line is
a speaker. A designation ('the tallest', 'the guard', 'a voice', 'the three
scum people') is a valid name. If two designations might be the same person,
still list both and flag the unsure one — a wrong merge is permanent, a
duplicate costs one merge click. Includes the ch5 passage as a worked example
yielding six speakers.
- Rule 9 'Name Hygiene': never include narration verbs/adverbs in a name
('Walking tallest' -> 'the tallest'), never include '(Scene N)' internal
labels, keep capitalisation consistent, no empty names.
- Rule 6 extended: a designation is itself unidentified; never upgrade
'the tallest' into a guessed proper name.
- tests/test_discovery_prompt_rules.py guards all three so a future prompt
rewrite cannot silently drop them.
Re-running discovery is required for the new rules to affect speakers.json;
existing entries are untouched (the designation system flags the current mess
in the meantime). Full suite 447 passed (8 pre-existing/parallel failures).
|
||
|
B.D. Murphy
|
6ef66f6cd6 |
Add speaker identification confidence + clear/needs-check designation (spec 17 §3.1/§3.2)
Addresses two related gaps: discovery misses obvious role/comparative speakers
(the six men in ch5 identified only as 'the oldest', 'the tallest', 'the young
one', 'the one with the biggest sword'), and the speakers page gave no signal
about which speakers are trustworthy.
New umbraaudio/analysis/speaker_confidence.py (deterministic, no LLM) gives every
speaker an identification confidence and a designation of clear / needs_review:
- categorical (blocking) defects that always need a human decision: a narration
verb glued onto the name ('Walking tallest'), an internal '(Scene N)' label, an
uncapitalised name, a name made only of role/comparative words, a probable
unmerged duplicate, discovery 'unidentified', or zero dialogue anywhere
- graded penalties: unknown gender, no traits/description, thin utterance share,
high share of needs_review utterances, unevidenced aliases
- duplicate detection tuned to avoid false merges: 'Lieutenant' vs 'Lieutenant
Stonesplitter' and 'Walking oldest' vs 'Walking Tallest' are NOT duplicates,
while 'Guard 1'/'Guard 2', 'Rielan'/'Rieland' and 'The Girl'/'The Girl helping'
are.
UI: the speakers page now shows all 66 speakers each with a ✓ clear or
⚠ needs check badge; hovering a needs-check badge lists the machine-readable
reasons. A collapsible 'N speakers need checking' panel at the top lists every
flagged speaker with its reasons, plus a header count. Registered as a template
global in create_app so routes.py is untouched (it is being edited concurrently
by another session). Rows render in ~60ms; the assessment is computed once per
page, not per row.
Last Dwarf: 32 clear / 34 need checking, including every speaker the reviewer
had to add by hand (scum leader, scum country boy, Scum guy1, Walking oldest/
Tallest/Youngest/sword, The Girl) and the four real merge pairs.
tests/test_speaker_confidence.py: 17 tests. Full suite 442 passed (8 failures are
the pre-existing/parallel-session voice_profile_manager + test_pronunciation ones).
|
||
|
B.D. Murphy
|
abb99757bf |
Fix broken name-token assign buttons + scanner missed an unnamed speaker
Two real bugs found by the user clicking Assign on the Name Tokens tab:
1. The action buttons never fired. {{ token|tojson }} emits double quotes
inside onclick="...", so the HTML attribute terminated early
(onclick="ntAssign(8, " and the rest became stray attributes. The browser
threw a JS syntax error, no fetch was ever sent, and the token stayed
unresolved. Switched to data-token/data-idx attributes read in JS, which is
immune to spaces and quotes. Also added real error surfacing (.catch +
server error message) and made the result message explain the 0-utterance
case: a name-tag token like 'Tobias Teppo' comes from narration, not an
assigned speaker, so assigning it records an alias and re-points nothing.
2. The scanner missed a real unnamed speaker. _is_known('Witch') returned True
because single-word containment matched the profile 'The Witch Pair', so the
2 utterances attributed to an unnamed Witch never reached the queue. Expanded
_ROLE_NOUNS with identity/trade/supernatural/machine nouns and made the
single-generic-word guard use it, so generic identity words are never
auto-absorbed into a longer profile. Last Dwarf now reports 10 tokens
(was 9) with 'Witch' (ch40) included.
Verified: real assign round-trips (alias + evidence + utterance re-point),
error paths return 400, tokens with spaces route correctly. Added a regression
test for the Witch containment case. Full suite 425 passed (8 pre-existing /
parallel-session failures unchanged).
|
||
|
B.D. Murphy
|
4d900d77a6 |
Fix 3 Phase 2 bugs found by infra review
1. merge_speakers never committed its utterance re-normalization. The CLI action path also returned without closing the connection, so Python rolled the transaction back on exit — merges silently lost every utterance rename. Now commits explicitly, and the CLI closes via try/finally. Verified: merging Guard 1 -> Guard 2 moved 6 utterances; a fresh connection sees 0/6 instead of the old 6/0. 2. assign_token_to_speaker wrote a dict into alias_evidence, but the discovery schema is list[str] and CharacterDiscoverer._alias_has_evidence() calls .lower() on each item — the next character-discovery run would have raised AttributeError. Now writes a descriptive sentence that also contains the alias, so the evidence check returns True for it. 3. dismiss_token overwrote evidence_json with the reason, discarding the scan evidence. Now preserves the original list and appends 'dismissed: <reason>'. Added regression tests: cross-connection merge commit (file-backed DB), alias-evidence string-list schema + discovery compatibility, dismiss evidence preservation. 46 resolution/scanner tests green; 85 across Phase 1+2+discovery. |
||
|
B.D. Murphy
|
c338370f19 |
Add orphan name-token scanner + resolution actions (spec 17 Phase 2)
Surfaces proper-noun speaker identities in the manuscript that do not resolve to a known speaker profile, so each can be assigned/created/dismissed/merged by a reviewer instead of silently mis-attributing. New: - umbraaudio/analysis/name_tokens.py: scans two high-signal evidence positions (unresolved proper-noun speech tags + un-profiled speaker_name values) rather than every capitalized word (594-token noise -> 9 real orphans on Last Dwarf). Groups book-wide, offers fuzzy suggestions (word-overlap / difflib / soundex / 4-char common-prefix misspelling), and suppresses personal-name suggestions for role nouns. Persists to a new name_tokens table (idempotent, preserves human status across re-runs). - umbraaudio/analysis/orphan_resolution.py: assign-to-speaker (add alias + evidence, re-normalize utterances), create-speaker, dismiss, merge-speakers. Merging is idempotent; matching is case-insensitive. - scripts/scan_name_tokens.py: scan CLI (--write) + one-shot actions (--assign/--create/--dismiss/--merge). - DB: name_tokens table in database.py SCHEMA (web routes create it on demand via ensure_name_tokens_table, so no shared-file conflict). Last Dwarf: 9 orphan tokens (8 role speakers + 'Tobias Teppo' name-tag which suggests 'Tobias'). Merge candidates Guard 1/Guard 2 + Narrator/OrigNarrator are real speakers flagged for the merge action. Tests: tests/test_name_tokens.py (30) + tests/test_orphan_resolution.py (12) green. ruff/mypy clean (TC003 sqlite3/Path notes match the repo's existing pattern in database.py). Full suite: 420 passed; the 2 test_pronunciation failures and 6 voice_profile_manager accent failures are pre-existing/parallel-session (a concurrent 'qwen_tts_helper_url' change), unrelated to this work. NOT committed: umbraaudio/web/routes.py name_tokens schema hunk — that file is actively being edited by a parallel session (mixed with its helper_url changes); left in the working tree to avoid a merge hazard. Table is created on demand, so nothing is functionally blocked. |
||
|
B.D. Murphy
|
1bbbe5d92c |
Add utterance clarity signals + review queue (spec 17 Phase 1)
Persist per-utterance attribution transparency so the system can auto-accept high-confidence lines and surface only a small review queue: - DB: utterances gains attribution_evidence_kind, attribution_tag_name, attribution_uncertain, needs_review, review_reason; attribution_confidence default -> NULL. Idempotent migrations in database._migrate, routes _ensure_utterance_columns, and a new ensure in reattribute_all (raw-sqlite driver must add columns itself). - llm_attributor: accumulates scene-level mechanical evidence across batches and stamps evidence kind + tag name on attributed utterances. - New umbraaudio/analysis/clarity.py: deterministic evidence classification (name/role/pronoun/address_only/none), in-scene pronoun resolvability (gender-filtered candidate count), and composite needs_review rule: QA issues (long_monologue excluded), non-unique pronoun referent, role tag with no profile, none/address_only evidence < 0.75, unresolved uncertain flag, narrator-left dialogue, confidence < 0.7. Addressee counts only for known names/titles (guards 'But where...' false positives). - Drivers persist all fields: reattribute_all (per-chapter clarity pass), run_llm_attribution (via update_utterance_speaker), both web attribution paths. _populate_exceptions keys off needs_review (was confidence<0.7 + keyword sniff) and fixes a latent NameError (pid not in scope). - tests/test_clarity.py: 25 tests. Eval on Last Dwarf: 3918 dialogue lines -> 173 needs_review (4.4%), within the 1-5% target. Full suite 380 passed; 6 pre-existing voice_profile_manager accent failures. ruff/mypy clean on new code; routes.py baseline unchanged (73 pre-existing). |
||
|
B.D. Murphy
|
b09e324431 |
Multi-pass attribution + web voice-maintenance banner + qwen demo branding
Attribution (from prior session, now committed): - llm_attributor: mechanical narration evidence, unnamed-role rules, wider context (12->20 lines, 8->40 utterances), uncertain flag, verify pass with 20-item batches and invented-name gate - reattribute_all: --min-id/--max-id, --dry-run, --no-verify, --verify-rounds, qa_accepted skip guard, per-chapter elapsed times - test_multipass_attribution: 19 tests Web banner: - voice_maintenance module: pid-liveness-checked marker file so a crashed run self-heals the banner away - routes/base.html: amber banner on every page while the LLM re-attribution run has Qwen3-TTS offline - reattribute_all: sets banner before switching models, clears in finally (also on failure) - test_voice_maintenance: 8 tests qwen3-tts submodule: UmbraAudio Voice Studio branding in web demo (icon data-uri + favicon logo). |
||
|
B.D. Murphy
|
df36dad8aa |
Weekly commit: attribution QA, dramatic variations, GPU TTS, AR export
Squashes ~3 weeks of uncommitted work (2026-09-03 → 09-21). The repo is the
only durable copy; a single accidental `git checkout` on routes.py nearly lost
a month of work (recovered from an editor snapshot, not git). See AGENTS.md →
"Commit Cadence" — commit every Sunday.
Attribution & QA
- New umbraaudio/analysis/attribution_qa.py: mechanical check_attribution /
fix_high_confidence / verify_conversation_flow (speech-tag mismatch, narrator
dialogue, role-tag guessing) plus scripts/qa_attribution.py CLI.
- llm_attributor prompt rules 3e-3h (unnamed roles, pronoun coreference,
repeated lines) with wider scene/position context.
- Per-utterance QA box in chapter audio detail UI: hover issues, Recheck
(auto-fix loop) and Accept (persisted qa_accepted* + text/speaker signature,
auto re-surfaces when either changes). New POST /api/utterance/{id}/recheck
and /api/utterance/{id}/qa-accept. _ensure_utterance_columns migration fixes
legacy DBs that lacked the columns.
Generation
- Dramatic utterance variations: 26 emotions + pace + custom instruct,
composed into every TTS voice mode (clone/icl/ref/embedding) and all
generation sites; markers record the instruct so edits re-trigger.
- Staleness markers are JSON {speaker,text}; legacy plain-text markers always
stale.
- Scene-break segmentation (generation/scene_break.py) + resegment merge.
- AR package exporter (assembly/ar_export.py) with tests.
- Voice ref resolution: rglob refs, canonical alias / first-word fallback.
- Edge-TTS fallback visibility: durable voice_engines.json ledger + UI badge.
GPU / TTS server
- qwen_tts_server: instruct + clone/icl support (voice_description →
instruct_ids prepended to the talker input).
- qwen_tts.py remote path sends voice_description alongside voice_ref/slug.
Config / ops
- .env: UMBRA_TTS__QWEN_TTS_URL=http://localhost:8881.
- .gitignore: book projects, tmp_out, logs, one-off samples A/B demo.
- AGENTS.md + README.md: Commit Cadence (commit at least weekly, on Sunday).
New tests: test_qa_acceptance, test_variations, test_ar_export,
test_scene_break, test_resegment_merge, test_vocals_project. Suite: 327 pass
(6 pre-existing voice-profile accent-wording failures, unrelated).
|
||
|
B.D. Murphy
|
85b1137a2e |
Implement multi-project isolation: no hard-coded project_id, WAL, web New-Project flow, verifiers
Phase A: derive the active project id dynamically (active_project_id) instead of hard-coded project_id=1 in cli/orchestrator/scripts/web; enable SQLite WAL on the project + voice-registry DBs; parameterize tts_server/sync_voice_data.sh via UMBRA_DATA_DIR/PROJECT_SLUG. Phase B: reusable project_bootstrap.create_project_from_manuscript (+ async variant for web) boots an isolated project dir + DB from a manuscript; new POST /api/projects and /projects/new page + "New Project" button let a second book be created and activated (active_project cookie) without touching the current project. Fix async route (asyncio.run in running loop) and source-path persistence ordering bugs in the new endpoint. Phase C: list_projects() skips the legacy root .umbra so /projects is clean; load_config() now resolves a bare project name under PROJECT_BASE even when the dir doesn't exist yet (new-project creation no longer depends on CWD); add tests/test_multi_project.py (5 tests). Phase D: scripts/verify_projects.py + scripts/verify_nas.py idempotent integrity/mirror checkers. All 249 tests pass. |
||
|
B.D. Murphy
|
a172610b0e |
Add voice library, embedding-mode generation, and webUI improvements
- Add cross-project voice/name library (library.py + webUI library tabs) - chapter_builder: embedding-mode generation + auto-load TTS base model in preflight - qwen_tts: vmatrix capture, embedding/voice-prompt refinements - registry: vmatrix_path column + migration for approved voices - webUI: speakers list, chapter audio detail, proper names updates - docs: update voice generation and tts_server README |
||
|
B.D. Murphy
|
c86e323d7a |
Initial commit: UmbraAudio audiobook generation program
Quality-first multi-pass audiobook generation from English manuscripts. Includes: - umbraaudio/ core package (pipeline, config, voices, web review UI) - scripts/ CLI utilities + model download script - tts_server/ remote TTS server - specs/ design docs, tests/, pyproject.toml + uv.lock - Qwen3-TTS git submodule (pinned 022e286) with local demo.py UI patch saved at scripts/qwen3_tts_demo_umbra_patch.patch - .env with runtime config/secrets Excluded (see .gitignore): models/ (downloaded via scripts/download_models.sh), output/, logs/, .venv, .umbra runtime data, and book projects (DealHunterShip_*), which are committed separately. |