f0c2317ca01991001521f05e32cb4ac8fd7b3830
94 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Umbra Tools
|
f0c2317ca0 |
Create the alignment cache directory before the first model call
The client writes each answer as it arrives, so a cache directory that was never created turns every model call into an error - and the fallback would quietly turn the whole book into a mechanical split while the run still reported success. The directory is now created by the command line entry point, and the client is constructed with it rather than having it assigned afterwards. A test pins it. 45 tests, ruff clean. |
||
|
Umbra Tools
|
996fe37f4f |
Align the translated paragraphs back onto the verified source utterances
The translation pipeline works on paragraphs, because epub and pdf are paragraph media. The audiobook needs utterances: UmbraAudio has verified who speaks every one of them, and each clip is voiced separately by that speaker's voice. A paragraph-level translation cannot be spoken. This is the pass that was missing. tools/translation/README.md said so plainly: "It does not produce a 1:1 utterance alignment for TTS ... splitting or aligning back to the 4,319 utterances is a separate pass." The division is by meaning, not by position. An intent-based translation merges and splits sentences for the target language's sake, so the sentence counts do not line up on their own - that is the whole point of Passes 1-2. A model is given the source utterances and the translation and asked which part of the translation renders which source utterance. Two rules the model cannot break: * a source utterance may only be absorbed into another utterance by the same speaker - a line must not be handed to a different character; * the partition must reproduce the translated text exactly, modulo whitespace, so the alignment can never rewrite the translation. An answer that drops, duplicates or invents text is rejected and the paragraph falls back to a mechanical split, which is reported as needing review. A source utterance the target does not voice is delivered as voiced=false with the reason, which is what UmbraAudio's utterance_translations.voiced flag is for: the row is kept so the gap stays visible, and no clip is generated for it. The output is a delivery payload in the shape UmbraAudio's translations.import_translations already accepts, so neither side has to invent a second contract. Measured on the real book, 42 sampled multi-utterance paragraphs across seven chapters: 41 aligned by the model, 1 fell back to the mechanical split, 0 partition mismatches. The one fallback was a model answer that did not reproduce the translated text; the content check caught it. 44 tests, ruff clean. |
||
|
umbra audio
|
91a1c13307 | Fix JS syntax: balance braces in umbra-app.js (was off by 1 after renderFormat fix) | ||
|
umbra audio
|
89f134a182 |
Rebuild Format activity UI with Edit→Publish workflow
- Left sidebar simplified: EPUB, PDF, Publish (removed Metadata/ISBN/Cover sub-tabs) - Main panel: all metadata fields editable (Cover with file upload, ISBN, blurb, search terms, print profile, page count) - Publish button runs preflight checks: cover uploaded, ISBN present, blurb present - Output generation buttons (EPUB/PDF/Audio) remain with job queuing - Output downloads section for completed renders - CJK-optimized CSS added to styles.css - Spanish translation pipeline started (running in background, 4/32 chapters done) |
||
|
umbra audio
|
fbbe23d7f9 |
Add output generation, cover image management, and back-translation tool
- Add cover image CRUD endpoints: GET/POST/DELETE /api/projects/{id}/cover
- Add output generation endpoints: POST /api/projects/{id}/output/{format}
- Add cover image and output queue service methods
- Add CoverImageResponse and CoverImageUploadResponse schemas
- Add umbra-api.js request handling for FormData (cover uploads)
- Add output generation buttons (EPUB/PDF/Audio) to Format UI in umbra-app.js
- Add backtranslate.py tool for verifying translation quality via back-translation
- Add cover image support to render.py EPUB generation
- Copy cover image to deal-hunter-zh source directory
- Spanish translation pipeline started (running in background)
|
||
|
umbra audio
|
be04e19d04 |
Add UmbraAudio controller integration endpoints
- Add migration 0010: utterance_translations table with UNIQUE(utterance_id, language_code)
- Add schemas: OrganizationCreateRequest/Response, ProgramCreateRequest/Response,
UtteranceTranslationItem/Request/Response/ListResponse, SpeakerSyncRequest/Response
- Add service methods: create_organization, create_program, sync_speakers,
upsert_utterance_translations, get_utterance_translations
- Add routes_controller.py: POST /api/orgs, POST /api/programs,
POST /api/speakers/sync, POST /api/projects/{id}/translations,
GET /api/projects/{id}/translations/{language_code}
- Mount controller_router at /api in app.py
- Create bdumbra account (member, BDMurphy org) for integration testing
|
||
|
umbra audio
|
3e0568b9d1 |
Generate backend endpoints for language-project workflow
Add REST endpoints and frontend wiring for per-language project actions:
- POST /api/workspace/{project}/translation/{lang}/queue — queue translation
- POST /api/workspace/{project}/audio/{lang}/queue — queue audio generation
- GET/POST /api/workspace/{project}/format/{lang} — format metadata (EPUB/PDF)
- GET /api/workspace/{project}/review/{lang}/chapters/{chapter} — utterance review
- GET/POST /api/workspace/{project}/front-matter/{lang} — front matter editor
- GET /api/workspace/{project}/chapters/{lang} — chapter list
- GET /api/workspace/{project}/chapter-content/{lang}/{chapter} — chapter text
Frontend fixes:
- Review page shows per-language text (English source, Chinese translation, etc.)
- Non-translated languages show 'Queue translation' button only
- Speaker data from English source matched by paragraph ID
- UmbraAudio column layout: Speaker, Variation, Text, Actions
- Chapter tree in left sidebar: Front matter, Back matter, Body
- Audio Format label at top of sidebar
- Pandoc formatting toolbar for front matter text editing
|
||
|
umbra audio
|
bd8238f3be |
fix BundleError handler: map malformed bundles to 422 with structured envelope
- Added exception handler for BundleError in app.py that returns HTTP 422 - Response includes code='invalid_bundle' and message from error argument - Previously returned unstructured 500; now consistent with other validation errors |
||
|
umbra audio
|
dda079f605 |
commit audio ingest: apply 0008+0009 migrations, wire Postgres adapter to commit path
- Apply migration 0008_audio_source_import (additive schema for source import) - Apply migration 0009_translation_import_runs (ledger table for import runs) - Wire PostgresAudioImportStore into HTTP commit path with org from node credential - Commit path uses PG store with org from node, no payload org - Bundle hash 21ce5687… remains frozen until v2; no authority transferred |
||
|
umbra audio
|
4160ec49a0 | audio: Phase 2 — Postgres AudioImportStore adapter + 0009 ledger; use PG store on commit with org from node; 0008 header updated; tests | ||
|
umbra audio
|
e35d7a144f | Add Apache-2.0 LICENSE, NOTICE, and SPDX headers | ||
|
umbra audio
|
d7f84075db |
Prompt identity: knowing which prompt produced a result
docs/prompt-identity.md, and tests that assert the mechanism rather than describe it.
The question this came from: are UmbraAudio's LLM prompts code, or data that gets pulled
and sent? The answer is neither, exactly - they are markdown files read at runtime:
PROMPT_PATH = Path(__file__).parent / "CHARACTER_SEGMENTATION_PROMPT.md"
prompt = template.replace("{chapter_text}", text)
client.chat.completions.create(messages=[{"role": "user", "content": prompt}])
That is a good design for *using* a prompt, and it is worth keeping. Editing the file and
re-running needs no code change and nothing to redeploy. Three things are missing, and all
three bite the moment a prompt is iterated on - which is the normal way to improve one, and
certainly how the intent scan will be developed.
1. No identity on the output. `utterances` records text, speaker, emotion and pace, and
nothing about which prompt produced them. After an edit and a re-run, "which paragraphs
were scanned under which version" is not difficult to answer but unanswerable.
2. No caching, so no invalidation to reason about. `llm_segmenter` has no cache references
at all, so re-running `analyze` re-sends every chapter. Editing a prompt costs a full
book scan, which is the cost that makes people avoid iterating on prompts.
3. Re-analysis appends. `insert_utterance` is an unconditional INSERT with no upsert key, so
a second `analyze` adds duplicate utterances rather than superseding the old ones.
The precedent for fixing this already exists one layer too late. `generation/staleness.py`
solves exactly this for audio - a sidecar per wav recording spoken text, speaker, voice_fp
and sha256 - and its docstring explains why: a "file exists" check silently reuses audio
that no longer matches the utterance text. The reasoning is right and the implementation is
careful, and it applies to audio only. The expensive artefact is protected and the cheap
one is not. This design extends the existing mechanism one layer earlier rather than
inventing one.
Three fields travel with everything an LLM produced: producer, prompt_id, prompt_sha.
prompt_id is the label a human reasons with; prompt_sha is the mechanical guarantee, taken
over the *rendered* prompt rather than the template - because hashing the template would
report two runs over different chapters as the same call, which is the one distinction the
mechanism exists to make.
The load-bearing test is that a version and its content must agree. A prompt file edited
without a version bump makes "v1" name two different prompts, and every query written
against it silently returns half the answer while looking like provenance. That
combination is refused rather than recorded, because a label that can quietly become untrue
is worse than no label.
test_prompt_identity.py states the mechanism as behaviour: that any edit changes the hash,
that different chapters render differently, that an unbumped edit is refused, that a bump
makes it legitimate, and that v1 still resolves after v2 exists. One of those tests failed
on the way because the registry helper rewrote v1's hash - backwards for that case - which
is recorded rather than quietly fixed.
Mutation-verified: disabling the version/content guard fails 1 test, and hashing the
template placeholder instead of the rendered prompt fails 3.
Does not cache responses - that is a separate decision needing a separate answer about when
a cached response is wrong, and it comes after this, not before. Does not touch UmbraAudio's
own prompts, which need the same treatment for the same reasons; that is a change to their
repository.
716 passed, 4 skipped.
|
||
|
umbra audio
|
70c23fde1b |
A precise request for the audio system's data, plus a validator to check it against
Asks for what is needed to write a correct mapping, in a form that can be verified rather than read. Two documents: docs/audio-system-data-request.md - what to send, in what format, and why that format. docs/audio-system-bundle-validator.py - ships *with* the request rather than after it, so the sender can check their own work before it travels. Standard library only: whoever runs it runs it on the audio system, and adding a dependency to someone's working machine to validate a file is not a reasonable thing to ask. That means the schema checking is the subset possible without a library - a smoke alarm, not an inspector - and the file says so rather than implying more. The reasoning behind the format is the substance. Prose cannot be validated: every mismatch is something a human has to notice, and the ones that get missed are the interesting ones - derived chapter order, two fields that mean slightly different things, a path relative to something unspecified. A validator that names the record and the field turns a round trip into a checklist, and reports every problem rather than stopping at the first, because a validator that stops at the first failure makes a three-round-trip conversation. Also asks for the things that are cheapest to supply and most expensive to guess: the vocabulary table (guessing terminology is how a mapping looks correct and reads the wrong field), which fields are derived (a derived chapter order treated as authoritative will disagree the first time somebody reorders a chapter by hand), and what the TTS engine actually is - fork or patch, what changed, whether it loads a model and how big. That last one decides whether a 48 GB CPU-only node can run it at all, so the speak work function cannot be written until it is answered. Explicitly not asked for: media files, source code, screenshots, or anything the sender would have to reconstruct. "We do not record this" is a complete answer and far better than an estimate presented as fact. The validator was itself tested against a deliberately broken bundle, which is the only way to know a check can fail. It caught a dangling reference, a unitless duration, two unanchored paths and a planted credential, and passed a correct bundle. Two flaws in it were found that way and fixed: it reported the same path problem twice, and every credential twice, because a tool whose only job is to be trusted about specifics loses credibility the first time it says the same thing twice. 704 passed, 4 skipped - no product code touched. |
||
|
umbra audio
|
9432366183 |
The catalogue: declare files that already exist, without moving the bytes
The audio node is the master for the audiobooks. UmbraTrans holds the metadata needed to reason about a file and a reference to where it is; it does not hold the bytes. DECISION-6 (object storage) stays unresolved, and this is what lets the system be useful before it is resolved rather than after. POST /api/compute/artifacts takes a manifest against a project and declares each entry: content hash, size, media type, kind, and a storage_uri naming where the file actually is. GET /api/compute/artifacts lists the catalogue. There were no artifact routes at all before this - the tables existed since migration 0003 and nothing had ever written to them, so "the controller has the artifacts to show a job is complete" had nothing behind it. Every declared row lands integrity='unverified', and that is the point rather than a caveat. Ingesting is a claim: a node reporting the hash of a file is a statement about bytes nothing here has read, because there is no object storage to read them from. A design where submitting a manifest marks the catalogue verified would make the first ingest the most reassuring and least true event in the system's life. verified_at is set by reading the bytes back and hashing them, and nothing in this path does that. storage_node_id is part of the reference rather than an afterthought, because a bare path means nothing six months later: /audio/book.mp3 is on whichever machine happened to serve it that day. Idempotent on (project_id, content_hash), so the same manifest can be submitted twice, or by two operators who did not talk to each other, without the catalogue doubling - and a partly-present manifest reports what was new rather than just "done". An operator who submitted 300 entries and was told "done" would not know two were malformed and dropped, so malformed entries are named in the response instead. The listing is scoped by organization on every call. artifacts is platform-level, as every table a node serves from is, so a missing predicate there is a cross-tenant read rather than a style issue. A project in another tenant's schema is refused as *not found*, not as *not permitted*, because a refusal that says "this exists but is not yours" answers the question the caller was actually asking. Migration 0007 applied to the deployment. Mutation-verified: marking an ingest verified fails 1, dropping the idempotency check fails 2, and dropping the organization predicate from the listing fails 7. 704 passed, 4 skipped. |
||
|
umbra audio
|
e757630da2 |
A phone had no navigation at all
The 720px breakpoint contained three rules:
.admin-sidebar { display: none; }
.rail-link:not(.rail-link--active) { display: none; }
.topbar-admin-link { display: none; }
Every one hides navigation rather than relocating it. On a phone the console showed the
page you were on and no way to reach any other: the admin sidebar was gone, every rail
link except the current one was gone, and the route from the workspace to the control
panel was gone. There was no hamburger to find because nothing had ever been built for a
narrow screen - the breakpoint assumed the sidebar could simply go away, which is only true
when nothing else is left to navigate with.
A toggle appears below the breakpoint and reveals the navigation as a drawer. The drawer is
positioned rather than shown inline, because an inline nav pushes the page down by its own
height every time it opens, so on a short viewport the page the operator asked for ends up
below the fold. Escape closes it, a click on the page behind it closes it, following a link
closes it, and crossing the breakpoint settles the state - without which the drawer class
survives into the wide layout and the bug only reappears when the phone is rotated back.
Above the breakpoint the toggle is hidden, because a control that opens what is already
open misreports the state of the page.
nodes.html is deliberately not given a drawer. Its navigation is inline in <main> rather
than in a sidebar, so there is nothing to put in a drawer and a hamburger there would
toggle nothing; its nav was a single horizontal scroll strip instead, which hides links off
the right edge with no sign that there are more, so it now wraps.
test_navigation.py states the rule - navigation hidden at a width must still be reachable
at that width - and reads the breakpoint with comments stripped. Two things that check got
wrong on the way, both recorded in the file:
- It flagged the drawer's own "display: none" as a violation. Hiding a closed drawer is
correct, or it would push the page down on every load; what is wrong is hiding navigation
with no way to open it. The test now requires a "body.nav-open" rule alongside any hide.
- It only required that *some* page carry a toggle, so deleting it from the admin console
- the page that was reported broken - passed, because the workspace still had one. Now
per page.
Mutation-verified: restoring the original defect fails 1, removing the toggle from the
admin console fails 1.
695 passed, 4 skipped.
|
||
|
umbra audio
|
d6cebe4d55 |
The controller assigns work, and a job is done only when the artifacts say so
Replaces claiming with assignment. The controller hands a job to one or more nodes and a node asks what it has been given; jobs.lease_owner could name only one holder, so a job fanned out to several machines was inexpressible, and 'what was assigned and what is not complete' had no answer. job_assignments is one row per (job, node, attempt), so both are now readable. Nodes poll rather than being pushed, which is not a simplification: nodes sit behind NAT and the controller has no route to them. The only path that exists is a node calling out. The rule with teeth: a job is complete when the controller has the artifacts to show it. Reporting is not believing. A node that finishes and hands over nothing gets its assignment failed, with the reason naming what was missing, and the work stays available for another machine. The tempting implementation - mark complete, then verify afterwards - cannot fail, because the only thing it could compare against is the report it just trusted. artifacts_verified_at is written by verification alone, so 'the node says it is done' and 'the controller confirmed it' are different facts that cannot look alike. fan_out defaults to 1 and is never inferred. More than one is redundant execution: several machines do the same work and the first verified result wins, the rest are cancelled. Worth having for cheap idempotent work, wasteful for anything costing GPU-seconds, so a caller has to say so. The losers are cancelled rather than left running - otherwise every machine on the list burns its time on a result nobody will read. Four defects found while building it, all by tests written first: - Naming a node explicitly bypassed the drain check. An administrator could drain a machine, see it read as drained, and still hand it work: the control doing nothing while appearing to work. Status is now checked on both routes. - attempts were counted from jobs.attempt, which only claim_job incremented, so under assignments max_attempts was never reached and a failing job was re-offered forever. Counted from failed assignments now, so a fan-out to three costs one attempt, not three. - start_assignment reported 'not startable' when the real problem was that the assignment belonged to another node. Ownership is checked first, because an operator handed the first will look in the wrong place. - The artifact API accepted values the database rejects: role 'output' where job_artifacts_role_check allows only 'consumed' and 'produced', an unbounded status, a content_hash that was not 64 hex characters, and kind at 80 where the limit is 64. Every completion would have been a 500. The schema had fixed this vocabulary all along and the API had to learn it rather than invent it. Artifacts are recorded even when verification fails: a short run still produced evidence, and discarding it would destroy the record of why. storage_key is never treated as proof the bytes exist, because there is no object storage to ask yet (DECISION-6). Claim still exists. It is listed in the pinned node surface as pending removal and the next change takes it out - two paths handing out work is the state this project keeps finding and criticising. Mutation-verified: trusting the node fails 3 tests, dropping the drain check fails 1, leaving fan-out losers running fails 1. 674 passed, 4 skipped. |
||
|
umbra audio
|
02fa70d8ad |
Revoke a credential, and take a machine out of the fleet for good
node_credentials.revoked_at has existed since the first migration and
resolve_node_credential has always honoured it. Nothing could set it. The ability to cut a
node off was present in the data model and absent from every API, which is the shape of
defect worth naming: the column was the capability.
POST /nodes/{id}/revoke cuts a node off and leaves it - right for a compromised key or a
machine about to be reimaged. POST /nodes/{id}/remove takes the machine out: credentials
revoked, devices, heartbeats and model reports deleted, status 'removed'. 'removed' is
deliberately not settable through PATCH /nodes/{id}/status, because typing 'remove' as a
status change would skip the revocation and the release of held work, which are the two
things that make removal mean anything.
The row is kept and cannot be deleted. job_events.node_id and
node_pairing_codes.used_by_node_id both reference compute_nodes ON DELETE NO ACTION, so
the DELETE is refused; changing them to SET NULL does not help, because that is an UPDATE
of job_events and job_events_append_only raises on every update by design. So a node that
has ever done work cannot be removed without weakening the trail's immutability, and the
trail is worth more than a tidy DELETE. A trigger makes 'removed' terminal, and
set_node_status refuses it first so the caller gets a reason rather than a constraint
violation.
Work the node was holding is released at removal rather than left to its lease. An
operator removing dead hardware already knows the work did not finish, and making them
wait out a lease to find that out is the wrong way round - for a long lease, most of an
hour of queue stalled behind a machine nobody is going to switch on.
Also creates job_assignments, which is how the controller will hand work to a node:
one row per (job, node, attempt), so a job can be open on several machines at once and
the controller can say which of them finished it. state is deliberately not the job's
status, and artifacts_verified_at is written by verification alone - that column is the
whole of 'a job is complete when the controller has the artifacts to show it'.
Migration applied to the deployment inside a transaction that first normalised any
'online' node to 'provisioning', since the new terminal-state trigger refuses that
transition and those rows would otherwise have blocked it.
Mutation-verified: not releasing held work fails 6 tests, leaving the credential working
fails 1, and a revocation that sets revoked_at back to NULL fails 2.
652 passed, 4 skipped.
|
||
|
umbra audio
|
9eb419ffc8 |
Nodes get a unique name from their MAC, so two machines can share one
Several nodes legitimately want to be called VM238 - one per hypervisor, one per rack, one per testbed. The unique constraint on compute_nodes is (organization_id, name), so until now the second machine to pair was refused with node_name_taken for a name that looks free to the person typing it, and the fix of 'pick another name' pushed the bookkeeping onto the operator for something the controller can derive. A node's name is now its base name plus the last four hex digits of its MAC: VM238-AC4B. Four digits because the name appears in a UI column, a command line and a systemd unit. Sixteen bits narrows collisions rather than eliminating them, so a letter fallback sits behind it - VM238-AC4B-A - and the sequence continues past Z, because a sequence that silently runs out hands out a name that is already taken. The code stays bound to the *base* name. The controller cannot know a MAC before the machine speaks, so the final name is settled at redeem, where 'is this taken' can be asked of the database rather than guessed at by the script on the machine. The script reads the first non-loopback, non-virtual interface, sends the MAC, and then adopts the name the controller returns rather than the one it asked for - reporting one while registering the other is how a node ends up appearing under a name it does not have. Three things this deliberately does not do: - A MAC that is not six octets of hex is rejected rather than cleaned up. The permissive version turns 'not-a-mac' into AMAC, and VM238-AMAC is worse than a node admitting it does not know its own address. - Without a MAC the plain base name is used and a second node of that name is still refused. There is no evidence the machines differ, and inventing a letter would turn one ambiguous input into unbounded identities. - Re-running bootstrap on an already-onboarded machine now succeeds as VM238-AC4B-A rather than being refused. That follows the specified rule - on conflict, add a letter - and it is called out in the test that asserts it: a stale node cannot be assumed gone because its hardware came back. Mutation-verified: the octet bug fails 10 tests, removing the letter fallback fails 2, and the bootstrap script ceasing to send the MAC fails 1. 641 passed, 4 skipped. |
||
|
umbra audio
|
851d14ee30 |
The node write path was dead, and drain did not drain
Two defects, either of which alone would have stopped a node doing work, and both of which looked finished. Pairing issued a read-only credential. node_agent.py already calls claim, renew, complete, fail and decompose, and no credential pairing could ever issue was permitted any of them - so the agent looked ready, every endpoint existed, and not one of those requests could succeed. The same shape as the login form with no method attribute: everything present, nothing working. A paired node now receives the whole node scope set. The pairing code is the authorization: minted by an authenticated administrator, single-use, expiring, bound to one node name, rate-limited per source address. An administrator who mints a code has already decided to put that machine on the fleet. This does not weaken the boundary that matters - NODE_SCOPES contains nothing that reads project content, and a node can hold several credentials, so a monitoring-only machine is still issued a narrow one and this one revoked. Separately, claim_job never looked at the node. set_node_status documents that draining 'stops new claims and lets running work finish', and the add-node page offers drain, return and retire; none of them did anything to the claim path. A machine drained before a reboot kept being handed work by an agent that was still running. The check is inside the claim transaction and takes FOR UPDATE on the node row, so an administrator draining a node cannot land between the check and the claim. It reads effective_status, so a node whose agent has died stops being offered work rather than taking a lease it will never renew - and deliberately not is_schedulable, which also requires a free device, since claiming a job is what makes a device busy. That gate found four tests in test_node_tenancy.py claiming work from nodes that had never authenticated - the state I24 exists to prevent. They now bring their nodes online first. Confirmed the tenancy property is untouched: restoring the tenant filter still fails 5 tests, the same count as before. Every fix mutation-verified - removing the status gate fails 5, restoring read-only scopes fails 2 including a new end-to-end test that pairs a node over HTTP and has it claim and complete a real job. 605 passed, 4 skipped. |
||
|
umbra audio
|
8d1c6a6406 |
27 of 63 database tests were skipping, and the summary said nothing
The PostgreSQL adapter suite refuses to run against a database that already holds rows, because it truncates indiscriminately and cannot do so selectively - audit_events is append-only, with a trigger that raises on delete. The refusal is correct. Two separate faults meant it was firing when it should not have. The guard sampled the live tables at the moment it was first requested. pytest runs files alphabetically and several files before it create organizations and tenant schemas, so by the time it looked, this session's own rows read as pre-existing data. It skipped 27 of 63 tests on a scratch database that had been empty moments earlier, and '567 passed' carried no hint of it. The sample is now taken in pytest_sessionstart, before collection. A session-scoped autouse fixture was tried first and is not early enough - it is instantiated lazily and does not precede the other session fixtures that write. Nothing cleaned up after the session. Ten files open their own connection to the DSN outside the postgres fixture, each inserted organization provisions a tenant schema, and no test dropped it: 78 schemas per run, which after about fourteen runs exhausted max_locks_per_transaction and errored the integration tests with a message reading like a PostgreSQL defect. And the fix for that nearly caused the disaster the guard exists to prevent. The first version of the cleanup truncated unconditionally on teardown, so pointing the suite at a populated database destroyed it while every test that could have touched it skipped itself. Found by planting an organization, watching the suite refuse, and finding the organization gone afterwards. It now leaves an occupied database exactly as found, and says so - through pytest's terminal reporter, because a plain write to stderr is swallowed by file-descriptor capture and was seen by nobody. test_scratch_database_safety.py covers all three, and each was confirmed to fail when the protection is removed. Three mistakes in writing it are recorded in the file: a leading test file that cleaned up after itself and so reproduced nothing; assertions that passed because '-rs' was missing and pytest had printed no skip reasons; and a check that stopped at the warnings header, which pytest prints its summary line after. 598 passed, 4 skipped. The 4 are environmental: two pages with no password field, two needing passwordless sudo for device permissions. |
||
|
umbra audio
|
b0cdec63a1 |
The root-install instructions assumed an account the machine may not have
The hint said to run 'sudo visudo' as though the account running the command already had sudo. On a machine set up the way this one was - a separate administrator, then a daily-use account - the daily-use account is not in the sudo group, so that instruction could not be followed by the person who needed it. Replaced with the two steps that actually apply, in order: 1. The account running the command must be in the sudo group. Check with 'id -nG'; if it is not there, an administrator adds it with 'usermod -aG sudo <account>' and the account signs out and back in, because group membership is fixed at sign-in. This is the step that was missing, and it is the one that cannot be done by the person stuck. 2. Passwordless sudo, from an account that already has sudo: 'sudo visudo -f /etc/sudoers.d/umbra-node', add the NOPASSWD line, save. Optional - without it the command still works and simply prompts. Written after a run stopped at a sudo password prompt with nothing on screen but 'Sorry, try again.', which is sudo's own message and appears nowhere in this codebase, so the failure looked like the system refusing rather than the shell asking. Checked against 238: umbra is already in the sudo group, so only step 2 applies there. |
||
|
umbra audio
|
3419485aa7 |
Fix a syntax error I introduced, which put a password in a URL
The commit immediately before this one - the password-toggle fix - deleted a closing
brace and did not replace it. app.js did not parse. The browser reported it in a console
nobody was watching, and every script-driven behaviour on the sign-in page stopped.
The consequence was not a cosmetic one. The login form's submit handler, which is the
only thing calling preventDefault(), never ran, so the browser performed a native form
submission. Neither form declared a method, and a form with no method attribute submits
as GET. Both password fields went into the query string:
GET /?email=umbra%40bdmurphy.me&password=...
The server logged it in plain text, three times, before it was noticed. It would also
have sat in browser history and in any proxy access log between here and the public
internet.
Two fixes, the second of which is the one that matters:
- The toggle fix is reapplied correctly to the last known-good version of app.js, with
nothing else in the file touched - the diff is four lines against thirteen and two
against one.
- **Every form on every page now declares method=post.** The script still intercepts
and submits via fetch; this decides what happens when the script does not run, which is
precisely the case that failed. Five more forms carried the same latent defect,
including password reset and email verification.
test_form_safety.py checks both, and was confirmed to fail when app.js is broken the
same way. It uses esprima rather than a hand-written brace counter, which gave two false
readings during this work - one of them 'balanced' for a file that did not parse at all.
A check that can be wrong about a syntax error is worse than no check.
The account password should be treated as exposed and changed. So should any password
entered into that form while the script was broken.
|
||
|
umbra audio
|
0b898316ef |
The password toggle was inverted on every password field
The variable was called 'reveal' and held `input.type === 'password'` - which is true when the password is HIDDEN - and was then used to pick the label. So a masked field offered 'Hide', and clicking it revealed the password and relabelled itself 'Show'. The click handler inverted the same way, so the two errors cancelled and the visible result was a control that appeared to work while doing the opposite of what it said. Renamed to 'visible', which is what the value now means. A correct-looking name is not what prevents this; the value and the name agreeing is. Affects the login field and both fields on the change-password screen. Verified by running the two states rather than by reading the edit: masked says 'Show', plain says 'Hide', and two clicks return to the start. |
||
|
umbra audio
|
0a4c400022 |
The console links to the nodes page, and a lost session leaves it
Three navigation faults, all found from the service request log rather than by looking at the page: - The console had no link to the nodes page at all. It was a standalone file I created and never wired in, so leaving the nodes page meant there was no way back except the browser. It is in the nav now, and every href on the console was checked against the running service rather than assumed. - VIEWS omitted 'projects' although the section and its nav link both exist, so clicking Projects silently fell through to the dashboard. A link that looks live and does nothing is worse than no link. Flagged in an earlier session as not-my-problem and left alone; it was two characters. - A 401 left the nodes page sitting there saying the session had ended, with nothing on the page that could work without one. It now redirects to the console, which is what the console does itself. The empty organization dropdown was this too: with no session the organizations request is 401, the list is empty, and the form is disabled with the new banner - which is correct behaviour for a genuine no-organizations system and a misleading one for an expired session. Redirecting first is what tells those two apart at the point where it matters. 547 passed, 29 skipped. |
||
|
umbra audio
|
14eba11ffe |
An empty organization list is now said out loud, and the page stops pretending
The organization dropdown rendered empty. Three separate defects behind it, found from the service's own request log rather than by looking at the page: - /api/uadmin/organizations returned 200 - the data was arriving - while /api/compute/nodes returned 422 on every load. ORGANIZATION_ID was only set when the account could see exactly one organization, so for anyone who can see several the node list was requested with no organization and the compute routes refused, as they are designed to. The list is now fetched when a choice is made, not at load. - The page had no case for *no* organizations. A node belongs to exactly one, so there is nowhere to put it: the form is now disabled and a banner explains why, with a link to the organizations screen. A dropdown with nothing in it and no explanation is the same as a broken page. - A 401 - an expired session - was indistinguishable from an empty list. It now says the session ended, which is the one failure the operator can act on. Also removed eight CSS classes I had invented rather than looked up: card, card-body, card-note, field, form-actions, table-wrap, alert--warn and button--quiet. None exist in styles.css, so the page was rendering unstyled. Replaced with what the working console page uses - admin-card, form-field, modal-actions, admin-table-wrap, admin-banner-- warning, button--ghost. Every class on the page is now verified present in the stylesheet rather than assumed, and the markup is checked as balanced by a parser rather than by counting tags by eye. 0005 was applied to the deployment to unblock startup: the tenant-schema contract check runs at boot and correctly refused to start a service whose schema it could not find. That is the check working. Backup taken and checksum-verified first; three tenant schemas created, one per organization, with users=2 orgs=3 nodes=9 audit=46 unchanged. 547 passed, 29 skipped. |
||
|
umbra audio
|
25cd17c061 |
The as_root checkbox help says what it now does
It described the control as a hint whose effect was decided at run time, which was true and misleading: the paths are chosen here, at render time, and the command follows. |
||
|
umbra audio
|
0e29d863fb |
The as_root checkbox now changes the command it produces
as_root selects which paths the rendered script writes to: /opt/umbra and /etc/umbra/node.json, or the invoking user's own directories. The command did not follow it. Ticking the box produced a script that wanted to write to /etc while being run by an unprivileged shell, which fails at the first mkdir - after the pairing code has already been spent, so it also leaves the operator with a dead code and nothing to show for it. The command now carries sudo when the box is ticked. The script still checks for root at run time and takes whichever branch is actually available, so this is belt and braces rather than the mechanism; but a control that produces a command which cannot work is worse than no control at all. The checkbox's help text described it as a hint whose effect was decided at run time, which was true and misleading: the effect is decided here, at render time. |
||
|
umbra audio
|
359e566a1f |
Record the TTS engine decision and what it still depends on
Qwen3-TTS as modified on the audio node, kept. Quality over time is the principle that settles it, and the engine already in use is the one worth keeping. The modifications are on the audio node and unknown here, so the speak work function cannot be written blind. Listed what is needed: what changed and why, whether it is a fork or a carryable patch, the input contract, and whether it loads a model and how large - which decides whether node 238 can run it at all. Parallel synthesis is a requirement, not an optimisation. The mechanism exists already (submit_children and the decompose route), which means the work-function protocol has to be designed for decomposition rather than have parallelism retrofitted into it. |
||
|
umbra audio
|
4ce121470e |
A node may do work for any tenant, and the pipeline plan
`claim_job` took an `organization_id` and matched it against the job's. Read plainly that says: a machine may only do work for the tenant that owns it. Both paired nodes belong to BDMurphy and The Last Dwarf belongs to WorstAuthor, so a node could never do a word of the audio node's work however idle it was. The only way to have found that out was to queue the job and watch it sit there. `place()` had the same fault - it listed candidate nodes by organization - so the scheduler could not use the fleet across tenants either. Both are fixed. A node is a machine. It belongs to whoever administers it, not to a tenant, and a tenant's data reaches it inside one job's payload for the duration of that job, never by widening what the node itself can see. `kind` and the placement requirements decide what a node may do; ownership decides who may switch it off. What did not change: a node still returns one job's payload and nothing else, so it still cannot enumerate another tenant's work. test_node_tenancy.py asserts the behaviour, and was confirmed to detect the old one: five of its seven tests fail when the tenant filter is restored. Also adds docs/plan-translation-pipeline.md, which separates the critical path - the work-function protocol, TTS, assembly, lineage, and one finished episode of The Last Dwarf - from the work that can proceed beside it. Writing those tests cost more time than it should have and the reason is worth recording: four of them failed for reasons that had nothing to do with tenancy. A job left behind by one test is claimed by the next, which reads as the rule breaking. `TRUNCATE ... CASCADE` on a table shared with other fixtures destroyed `project_publishers` and surfaced two files away as "relation does not exist". The queue is now left alone and the claims filter on a job kind only this file creates, so a neighbour's leftovers cannot be returned and the assertions can be exact. 547 passed, 29 skipped. |
||
|
umbra audio
|
715b7bc69f |
Restore the guarantee migration 0005 had to drop
`projects` moved into a per-tenant schema and `jobs` stayed in `public`. A foreign key cannot cross that boundary without postgres_fdw, so 0005 dropped `jobs_project_id_fkey`, `artifacts_project_id_fkey` and a composite `user_roles` key. The migration recorded the consequence honestly - "the application still refuses it; the server no longer does" - but the application did not. So after the split nothing stopped a `public.jobs` row naming a project in a different tenant, which is precisely the leak the schema split exists to prevent. `submit_job` now verifies it. Two things that were wrong in the first attempt, both caught by running the tests: - The lookup was unqualified, relying on `search_path`. `ComputeRepository._tx` does not set it - only the account repository does - so the check ran against `public` and could never see any project, refusing every project-scoped job. The schema is now derived from the organization id and named explicitly: the check is self-contained, and does not depend on ambient state deciding whether a security check runs at all. - It filtered on `projects.organization_id`. That column no longer exists - the schema the row sits in *is* the tenant, which is the whole point of the split. There is no predicate to write. `test_a_nonexistent_project_is_refused` asserts that a caller cannot distinguish "no such project" from "another tenant's project". From inside a tenant schema the two are the same observation, and they should be. Confirmed the tests detect the gap: with the check removed, three of five fail, including `test_the_refused_job_is_not_created` - which exists because a refusal that still writes a row is not a refusal. 547 passed, 29 skipped. Deployment untouched: 30 tables, 0 tenant schemas. |
||
|
umbra audio
|
24d7ea6a41 |
Format the lines this change wrote, and leave the rest of the drift alone
ruff format disagrees with about two dozen files in this repository, most of them untouched by this work. Reformatting them would bury the tenant-schema change in noise, and reformatting only some of a file is its own kind of noise - so only the lines written here are brought into line, and the pre-existing drift stays where it is for whoever owns it. |
||
|
umbra audio
|
0179355fe7 |
Drop the tenant schemas one transaction at a time
The cleanup I added dropped every tenant schema in a single transaction, and a hundred runs' worth of them exhausts the lock table: the session teardown failed with 'out of shared memory ... max_locks_per_transaction', which reads like a PostgreSQL defect and is not one. It is the same constraint the 0005 rollback documents and I wrote there before hitting it here. Committing after each drop releases the locks, and the teardown now leaves zero schemas behind, so a run does not make the next one slower. |
||
|
umbra audio
|
9a84a9d227 |
The staleness fixture's project goes into the tenant schema, and comes out of it
Same shape as the append-only guards: the project row is created in the organization's own schema and the cleanup drops that schema. The old cleanup deleted from public.projects, which since 0005 has cleaned nothing up while looking like it did - the sort of teardown that reads as careful and is not. |
||
|
umbra audio
|
7b12964a9b |
Seed the guard fixture's project into the tenant schema it now lives in
public.projects does not exist since 0005, and this fixture was the last place still writing to it. The project row moves into the organization's own schema under an unqualified name, which is also the shape production writes it in - so the artifact and job rows guarded by these tests are being set up the way the application would set them up rather than the way the old shared table allowed. |
||
|
umbra audio
|
152735f145 |
Assert the tenant boundary where it now lives, not where it used to
Three changes, all of them consequences of the migration rather than new questions. test_rls_coverage.py drops project_authors and project_publishers from the policy list - they are per-tenant now and there is nothing for a policy to filter - and replaces that entry with checks that are stronger than what it replaces. A policy on a shared table proved the table was filtered; a per-tenant table cannot be filtered at all, so the properties worth asserting are that no shared copy exists in public, that no policy lingers on a tenant table, that every tenant schema carries all four tables, and that organization_id is gone from all of them. That last one matters because it was the column every removed policy filtered on: leaving it behind would leave an application filtering by a value nobody maintains. It also gains a check that POLICY_TABLES names tables that are actually in public. Without it, moving a table out of the policy list and out of public in the same change makes both existing checks pass by not looking at it - which is how a table can quietly stop being policed and still look policed. test_tenancy.py covers the search_path function directly, and the case that matters is the negative one: a request with no organization gets public alone. A wildcard, or leaving the connection's default in place, would put every tenant's tables in reach of a request that has not decided which tenant it is acting for. test_postgres_repository.py proves the new startup contract check holds against the real app function, and that it can fail. A check that cannot fail is the same shape as a policy nobody wrote: it reports success while enforcing nothing. The emptiness guard now also looks inside the tenant schemas and the run drops them at the end. Not tidiness: a database holding nothing in public can still hold somebody's books, and a guard that only looked at public would pass on a database it was about to empty. |
||
|
umbra audio
|
4781468181 |
Test the property that replaced the one a policy used to prove
The old file proved a policy refusing a row. With no shared table there is no policy and nothing to refuse, so the tested property is gone rather than rewritten - and Umbratools_2.md 5.0 says outright that this trade should be noticed rather than discovered later. This commit is the noticing. What is proved instead is stronger, not differently shaped. A statement with no organization in hand asks for `projects` unqualified and is told the relation does not exist: not zero rows, an error. That is a different claim from "the policy filtered it", because a policy that was silently not enforced returned zero rows and looked fine - which is the failure G2 was, and the reason the old test had to run as a non-owner role to mean anything. The new tests keep the non-owner discipline for the part that needs it. Naming another tenant's schema is refused outright; repointing the search path at it and using an ordinary name finds no table at all, because a role without USAGE on a schema does not see its contents when names are resolved. That second one is the outcome an application actually experiences, so both are asserted rather than the convenient one. Positive cases are here because a suite that only proves refusal passes just as happily against a database where every write is refused: a write that must be allowed is performed and then counted in both tenants. What the file cannot prove is written down in its docstring rather than left to be assumed. The application owns every tenant schema, so it can reach another tenant's tables by naming that schema, and schema-per-tenant cannot change that with one role serving every tenant. What it changes is that there is no shared relation to reach through, and reaching another tenant means writing down another organization's name - one call site in tenancy.py rather than a predicate in every query. |
||
|
umbra audio
|
79b3484083 |
Reach the tenant tables by name, not by a filter somebody remembers
Every project statement now says `projects` unqualified and takes its
organization through search_path. A shared table filtered by
`organization_id = %s` is a filter somebody has to remember, and there is always
a write path that would not have it; an unqualified name resolves into the
caller's own schema or it does not resolve at all. The strongest possible
mistake - forgetting to scope a query - now raises instead of returning a wider
result set.
Project.organization_id survives in the model and is filled in from the schema
the answer came from. Callers still see which tenant a row belongs to; it is
derived from where the row is rather than from what the row says about itself.
Two consequences that are worth more than the mechanical change:
- assign_user_to_project now takes the organization as an argument. It used to
read it off the project row and copy it into user_roles, and there is no
column to read now. user_roles is platform-level, so it cannot be written to
by looking inside a tenant schema - so the caller, which has already resolved
the project, has to say it. The protocol declares it, and the in-memory store
accepts and ignores it rather than being the one implementation with a
different signature.
- A platform administrator listing projects without naming an organization now
gets the answer assembled across tenant schemas, because there is no shared
table to ask. That is one round trip per organization and it is paid only by
that one caller; every per-tenant path, which is what an ordinary request
does, is still a single statement. The merge reproduces the ordering and the
total a shared table would have given, so paging across the boundary is the
same as paging inside one.
|
||
|
umbra audio
|
2264a6cc42 |
Publish the caller's schema onto every transaction
Two things now go onto a pooled connection before a statement runs: the caller's identity, which the surviving platform policies read, and the caller's schema, which decides which tenant tables the statement can even name. search_path is set on every transaction rather than only the ones that have an organization, so a platform context gets public alone instead of whatever the connection's own default happens to be. That is the difference between a design and a hope: a default naming a tenant schema is a silent leak with no line of code mentioning it. verify_tenant_schema_contract exists for the same reason verify_rls_contract does. The schema name is derived in two languages - this one and app's - and nothing joins them. A disagreement is silent and total: every tenant query resolves into a schema that is not there, every project request for every organization raises "relation does not exist", and the error names the statement rather than the cause. That presents as an empty database, which is the one diagnosis nobody reaches for. |
||
|
umbra audio
|
e1bbfdb604 |
The session-to-schema step, next to the session-to-organization step
tenant_search_path is the whole isolation model in one function: given the organization a request resolved to, return the schema its tables are in; given none, return public alone. That second case is the point. Returning a wildcard - or leaving the connection's own default in place, which may name a schema nobody chose - would put every tenant's tables in reach of a request that has not decided which tenant it is acting for. The name is derived from the organization id rather than stored, for the reason the migration derives it too: a recorded name that disagreed with the schema the rows are in would describe data that no computed name can reach, and that state would be indistinguishable from "this organization has no data". Deriving makes it unrepresentable. This lives in tenancy.py rather than in the adapter because it is the same decision as scope_for, not a second decision next to it. A module that picked the organization and another that picked the schema would be two places for the boundary to be got wrong, and the schema one is the silent one: an unqualified projects query simply finds nothing, and reads as an empty database. The value is quoted rather than interpolated because search_path parses its value as a *list*. A name carrying a comma would not name a schema, it would append one. The check before that is unreachable from a UUID and is there anyway. |
||
|
umbra audio
|
2cdebf5ac3 |
Schema per organization: the migration, and why these four tables
Umbratools_2.md 5.0 replaces shared-tables-plus-RLS with one schema per
organization, because RLS was measured to be unenforceable twice: the
application connects as the table owner so the policies did not apply to it
(gate G2), and one app function was PUBLIC-executable until it was fixed by
hand against one database (defect B4). A model whose correctness depends on a
grant somebody remembers to make is not a boundary.
The classification comes before the SQL because the SQL's blast radius depends
on it, and the obvious reading of "move the tenant tables" is wrong for eight of
the tables carrying an organization_id. Four move: projects, project_authors,
project_publishers, invitations. The other twenty-six stay, each for a reason
read out of the actual columns and foreign keys rather than the table names.
Two findings the plan did not price, both recorded in the classification:
- user_roles cannot move. It is the index that resolves a session to an
organization, so putting it in a tenant schema means finding the tenant by
searching every tenant schema, on the login path. It also holds
platform-scope rows, which have no organization to belong to. So one table
stays shared, and 5.0's "no table is shared between organizations" is not
fully achievable while the session-to-tenant resolution is a table.
- moving projects forces dropping three foreign keys that point at it from
artifacts, jobs and user_roles, because a foreign key cannot name a table
whose schema depends on the row. That is the one place the decision costs a
database-enforced guarantee, and it is written down rather than papered over
with a trigger that would have to rediscover which tenant is meant.
The migration names each tenant schema from the organization id rather than
storing it, so a name cannot disagree with the rows and the rollback can rebuild
the organization_id column from the schema. It is idempotent - verified by
applying it twice - and the rollback is a file that runs rather than a comment
block, because a rollback for a data migration that only exists as a comment is a
rollback nobody runs. Both were applied to the scratch database, rolled back, and
applied again.
|
||
|
umbra audio
|
62372a9c55 |
Wider code display, copy button beside the value
Feedback on the add-node page: the command was hard to read and hard to copy. Both the command and the six-digit code now use .otp-value-row / .otp-value - the wide monospace value with its copy button beside it that the provisioning dialog already uses. Reusing that rather than writing new CSS, because a second styling of the same idea is how the two drift apart. The command was a <textarea> carrying a class named `code-block` that exists nowhere in the stylesheet - I had invented it - so it rendered as a default textarea rather than the monospace block it was meant to be. It is now a <code>, which means it is written with setText rather than .value; setting .value on a <code> silently does nothing, which is the same class of silent failure as the missing toast element. The standalone "Copy the command" and "Copy the code" buttons at the bottom are gone: each value now has its own button next to it, so there is one obvious way to copy each thing and no need to match a button to a field. Done is now labelled to say what it does - "clear this from the page" - because the code is only in the page's memory and the controller cannot produce it again. |
||
|
umbra audio
|
a75959c2e9 |
Fix the add-node page: nothing happened because three bugs, all silent
Reported as "I entered info, clicked create, and nothing happened". Three separate
faults, each of which fails quietly, so the page looked like it simply did not work.
**1. The toast element had the wrong id.** api.js's showToast() looks for
`id="toast"` and returns silently when it is absent. The page had
`id="toast-region"`. Every message - success and failure alike - was appended to
nothing. This is the direct cause of "nothing happened": the request may well have
succeeded and the answer was thrown away. The working console page uses id="toast";
mine should have matched it.
**2. Seven `el()` calls passed a className where api.js wants a props object.**
`el(tag, props, children)` assigns `props.class`. `el('td', 'table-note')` therefore
set an attribute literally named "table-note" and no class at all, so the node list
would have rendered unstyled. Caught by reading the helper instead of assuming its
signature from the name.
**3. A platform administrator could not use the page at all.** The compute routes
refuse to infer an organization - they raise `organization_required` and ask for
`?organization_id=...`. The first version read an `organization_id` out of
/api/uadmin/summary, which has no such field, so it was always empty and every create
was a 422.
The page now loads organizations from /api/uadmin/organizations and presents a picker,
sending the chosen id as a **query parameter**. It went in the JSON body at first,
where the route declares it as a Query - a body field of the same name is silently
ignored, which looks exactly like it worked. An organization administrator has exactly
one and sees it preselected rather than being asked to choose.
Also added `defer` to both script tags, as the working console page has, so api.js is
defined before nodes.js boots.
Verified the corrected request path end to end: the same payload the browser now sends
produces a code, and that code pairs a node on 238 through the `curl | sh` flow.
525 passed, 27 skipped.
|
||
|
umbra audio
|
c3a3e506ec |
Add a node from the console, and make curl | sh actually work
The page an administrator uses to bring a node online: name it, get a one-line command and a six-digit code, hand both to whoever is at the machine. That is the last piece between "someone runs a command" and self-service onboarding. POST /api/compute/pairing-script now returns JSON - script, code, expiry and the command - rather than the bare script. There are two callers with two needs: a person at a terminal wants a script to pipe to sh, and the console wants all three values so it can show them once and then forget them. Returning text would have forced the console to parse a shell script to recover a six-digit number, which works until someone reformats the script. The code is shown once and there is no button to fetch it again, because the controller keeps only an Argon2id hash. That is the server's limit, not a UI choice, and offering a "show code" button would be a lie about what it can do. ## Three defects the template flow exposed Running `curl ... | sh` on node 238 - the flow the console hands out - found that the template could never pair, for three separate reasons: - It had no prompt and no code. `000000` is the placeholder that means "nobody filled this in", so the script now asks. A rendered script from the console has the code substituted and skips the prompt, so nobody is asked for a number they do not need. - It had no node name, and the name is what the code is bound to. It now asks too. The first version failed with "check the code has not expired" while the code was fine and the *name* was the placeholder - an error pointing at the wrong thing. - It hardcoded `controller.invalid` as its controller address, so every deployment but one got a script that could not resolve its own controller. The address now comes from the Host header the template was fetched with, and `UMBRA_CONTROLLER` overrides it. The error message itself was part of the problem: a DNS failure reported "check the code has not expired", which is a worse fault than no message because it sends an operator to check the one thing that was working. ## Verified `curl -fsSL http://192.168.123.14:8442/api/node/bootstrap | sh` on 238, code and name typed at the prompts: paired, agent installed, service registered, node `online`. `/api/node/bootstrap` is unauthenticated and returns only the inert template - a script that can check a machine and then stop. It is listed by name in the test that pins unauthenticated node routes, alongside /pair, with the reason each is safe. ## Also found, not fixed `VIEWS` in web/assets/admin.js is `['dashboard','accounts','organizations','audit']` but `view-projects` exists in the markup, so the projects view can never be activated. It predates this work and I have left it alone rather than change navigation behaviour as a side effect of adding a page. 525 passed, 27 skipped. |
||
|
umbra audio
|
bc031dd079 |
Bootstrap: one file, and it works with or without root
Enrolling a node meant a person reading instructions and running them, with a credential typed into a terminal. That does not scale, and it makes the controller a courier. The flow is now: an administrator renders a script, hands over one curl, and the node sets itself up. `GET /api/node/agent` serves the agent to a credentialed node. A node credential already implies "run what the controller sends" - that is what claiming a job is - so this adds no authority; it removes the hand-copying. `POST /api/compute/pairing-script` renders the bootstrap, minting a pairing code server-side so a code pasted into a request body cannot be one issued earlier for something else. ## The exclusive/shared distinction, encoded The script asks whether it has root and branches. With root: `/opt/umbra`, a system unit, and the agent running as a dedicated `umbra-node` account with NoNewPrivileges and ProtectSystem=strict. Without: the user's own directories and a user unit. Both layouts are in the output and the runtime decides, because whoever renders the script does not always know how the person at the machine will run it. The credential goes in a 0600 file referenced by systemd's EnvironmentFile. Three alternatives were rejected and the reasons are in the code: a unit file is world-readable, a config holding a secret ends up in a backup, and the agent deliberately refuses to read the key from its config when the environment can carry it. ## Verified on node 238, not just reasoned about Ran it as `umbra` with no root: paired, installed, registered, and the agent went `online`. Six defects found only by running it: - `$BOOTSTRAP_VERSION` was interpolated into the shell body but never declared as a shell variable. `set -u` caught it on the first line that used it. - The agent fetch sent no credential and got a 401 - correct behaviour, wrong script. - `ExecStart` pointed at the agent file directly. It has no shebang, so the kernel tried to run it as a binary: "Exec format error", a service in a restart loop, and nothing on the controller. Now invoked through `python3` explicitly, and the file is 644 rather than 755 because it is source. - `sed -i` with no backup suffix is GNU-only; BSD reads the next argument as a suffix, so on macOS it would leave the unit unfiltered and install hardening directives that cannot work. Replaced with a portable filter. - Pairing a name already in use surfaced as a bare 500 from a UniqueViolation. Now `node_name_taken`, with an answer that says pairing does not replace an existing node. - The agent logged "claim failed: node_scope_required" every second, forever. A read-only credential can never gain compute:claim, so this is permanent, and a warning that cannot change trains a reader to ignore warnings. Announced once. ## A privilege bug the pinned tests caught The bootstrap renderer declared an administrator dependency while sitting on the *node* router. A node could have called it to mint itself a credential for any node name. It is now `POST /api/compute/pairing-script` on the admin router, above the agent-surface marker so `test_every_node_route_uses_node_credential_authentication` keeps meaning something. Also worth noting: a node cannot be DELETEd from the database, because the pairing-code reuse trigger refuses the `ON DELETE SET NULL` cascade. That is correct - the status enum has `retired` and the admin route drains and retires, so removal is a status change, not a row deletion. Found by trying to clean up after a failed test run. 525 passed, 27 skipped. Test nodes boot-test-1..3 are retired, not deleted. |
||
|
umbra audio
|
c56ef46190 |
Measure capability by executing instructions, not by reading cpuinfo
/proc/cpuinfo is a declaration. On node 238 it reports 'AMD Ryzen 9 3900X' and then lists 65 CPUID flags that stop at SSE4.2 - no avx, avx2 or fma, on silicon that has all three. Executing an AVX2 instruction there faults with SIGILL, so the hypervisor is presenting a CPU that does not exist. That is the same defect the compute design already has, one layer down: the agent sets throughput_factor by string-matching a GPU model (40.0 if '4070' in name else 20.0), a declared factor wearing a measured name. Here the hypervisor makes the same kind of lie about instructions. So a feature is recorded as one of three things and never two: present - an instruction used it and produced the right answer absent - an instruction used it and the kernel killed the process unverified - it could not be executed at all Only 'present' counts as usable. An unverified feature is treated as unavailable on purpose: the cost of pessimism is a slower path that works, and the cost of optimism is a job dispatched to a machine that dies on it. Measured on node 238: sse2, sse4_2, bmi2, aes_ni usable; avx, avx2, fma, avx512f all SIGILL. On oldbox: everything but avx512f. Four bugs found while building the probe, each of which would have made it report something false: - Inline asm with wrong operand counts, for all eight features. Replaced with intrinsics, verified by disassembly. - One binary for every feature could not compile: -mavx2 alone makes the FMA, AVX-512 and AES intrinsics fail always_inline. Every feature reported unverified on a machine that has most of them. Now one translation unit per feature. - GCC folded the probe at -O2 with constant inputs and emitted nothing, so it would have reported 'present' for every CPU on earth. Inputs are volatile and the expected mnemonic is checked in the disassembly; test_a_folded_probe_is_reported_unverified builds a deliberately foldable probe and asserts it is caught. - The SSE4.2 probe stored a comparison mask through _mm_storeu_pd and read a double back, getting NaN - reported as 'executed but returned a wrong result' on every machine. Also: a caller passing an explicit compiler path that does not exist got eight build errors instead of one clear sentence; the compiler is now verified to work, and a missing or broken one is a False rather than an exception. 525 passed, 27 skipped. |
||
|
umbra audio
|
11f27f5ef9 |
Capability profile for node 238, measured over SSH
Reached 238 via public-key enrolment - no password anywhere. The first attempt failed because the key had never landed in authorized_keys; the second worked once the fingerprint matched. Findings worth keeping beyond this node: The VM reports 'AMD Ryzen 9 3900X' and then lists 65 CPUID flags that stop at SSE4.2. No avx, no avx2, no fma. That silicon has AVX2 and FMA, so VirtualBox is masking them and /proc/cpuinfo is a declaration, not a measurement. Recorded as *unverified* rather than absent, because confirming it needs an actual AVX instruction executed and there is no gcc on the guest - a flag list cannot settle it. This is the defect the compute design already has, one layer down. The agent sets throughput_factor by string-matching a GPU model (40.0 if '4070' in name else 20.0), which is a declared factor wearing a measured one's name. Here the hypervisor makes the same kind of lie. Capability discovery therefore has to probe by executing instructions and treat an unverifiable feature as unavailable. Also: cpu.max is 'max 100000' so the guest is not CPU-capped, memory is 47.0 GiB unballooned, no GPU and no nested virt flags, and umbra has no sudo. Recorded in docs/ node-238-capability-profile.md with how each value was obtained. |
||
|
umbra audio
|
0c35eb2d03 |
Allow the address the service binds to as a Host value
The deployment listens on 192.168.123.14 and TrustedHostMiddleware rejected that same address, so every request by IP returned a bare 400 Invalid host header - including /api/health, which is indistinguishable from the service being down. It has cost this project twice. Found when the first pairing attempt from 192.168.123.238 came back 400 with no body and no logged detail. The pairing code was fine and went unspent; the request never reached the route. Both umbratrans.bdmurphy.me and 192.168.123.14 are now allowed. The 0600 on the env file is unchanged, and the correction is recorded in the alignment doc's transport note so the audio node is not sent looking for a service outage. |
||
|
umbra audio
|
32dc580c74 |
Pair nodes with a short code instead of a copied credential
Issuing a node credential by hand required writing the key into a 0600 file on the controller and having a person copy it onto another machine. That key landed in shell history and a terminal buffer, and it made the controller a courier: nothing could be enrolled without me in the middle, which is the opposite of what a fleet of heterogeneous nodes needs. `POST /api/compute/pairing-codes` (admin session) mints a six-digit code for a *named* node. `POST /api/node/pair` exchanges it, once, for a credential returned in the response. The node never sees a secret typed by a person, and the controller is no longer in the path. ## Why a six-digit code is defensible 1,000,000 possibilities is nothing on its own. What makes it work is everything around it, and any one of these alone would be too thin: - **Single use.** Redemption is `SELECT ... FOR UPDATE` then `UPDATE ... WHERE used_at IS NULL`, in one transaction. - **Expiring.** Default ten minutes, hard-bounded to 60-1800s. Long enough to walk to a machine; short enough that a code left in a terminal buffer is not a credential for the day. - **Bound to a node name.** A code read out of a terminal buffer cannot pair a machine called something else. Without this, "confirm it is the machine" is decoration. - **Argon2id at rest.** A database read or a support transcript yields nothing replayable. - **Rate limited per source address**, charged *before* any code lookup, so guessing across many codes costs the same as guessing at one. Twenty per quarter-hour against a million-code space is about 21 years. This is the control that makes the length defensible; the others are cheap additions to it. Every rejection is worded identically - "no such code", "wrong code", "expired", "already used" all read the same - so the response is not an oracle for the rate limiter to work against. The one exception is a name mismatch, which says so plainly: it is not secret, and an operator staring at a failure needs to be told the code is for a different machine rather than left guessing. A name mismatch does *not* spend the code, because the obvious implementation spends it and then the retry with the right name cannot work. ## Why not a SECURITY DEFINER function The obvious shape is a plpgsql function that verifies and redeems, so the node surface never touches the tables. It cannot be done in SQL here: the codebase hashes with Argon2id in Python, and the only hashing PostgreSQL offers in this database is `crypt()` (bcrypt and MD5). Writing pairing against bcrypt would claim an algorithm choice the rest of the system does not make. So verification is in the repository, which means the pairing tables need grants for the application role when G2 lands, and the path needs re-testing as a non-owner role. Recorded in the migration header. ## Three defects found by writing the tests - **Redemption order.** Claiming the code before inserting the node it references violates the foreign key, so *every* redemption failed. Found by nine failing tests with an FK error that had nothing to do with pairing logic. - **A race that produced a 500.** Two nodes verifying one code simultaneously both created a node, and the loser hit a unique-constraint violation on `(organization_id, name)` - an unhandled error for an entirely ordinary condition. Fixed with `SELECT ... FOR UPDATE` on the code row before the insert, so the second caller blocks, re-evaluates `used_at IS NULL`, and gets the ordinary refusal. The concurrency test fails when the lock is removed and passes when it is present. - **A test that could not fail.** Removing the `used_at IS NULL` from the *claim* left all tests passing, because the candidate query's `used_at IS NULL` filter masks it. Sequential redemption cannot distinguish the two guards; only a concurrent one can. That is now `test_a_concurrent_redemption_produces_exactly_one_node`, and `test_a_spent_code_is_not_offered_as_a_candidate` pins the first guard directly. The reuse trigger needed a correction too. It originally refused any change to `expires_at`, which would have made it impossible to revoke an unused code - the normal way to withdraw a code issued to the wrong person. It now refuses identity changes and expiry *extension*, and permits expiry to move earlier only. Two existing tests flagged the new unauthenticated route, correctly: `test_the_node_surface_is_exactly_what_a_node_may_do` and `test_every_node_route_uses_node_credential_authentication` both assumed every `/api/node` route needs a credential. Rather than weaken either, `/pair` is recorded as a named exception in both, with a check that the named route still exists - so a second unauthenticated node route has to be added deliberately with a reason. 509 passed, 27 skipped. Deployment: 0004 applied after a backup, health 200, and the live endpoint verified to refuse a bogus code with 403 and a wrong node name with a `pairing_name_mismatch` that does not spend the code. G2 is still open, and the pairing tables will need grants when it closes. |
||
|
umbra audio
|
382ca8fc78 |
Derive node status on read; a heartbeat loop for the audio node
Two problems, both found by the audio node running Stage 1.
## A node that went quiet still looked available
`compute_nodes.status` is promoted to `online` by a heartbeat and demoted to
`offline` by an administrator. Nothing demoted it on the passage of time: grep found
`offline` appearing in exactly one place, the heartbeat's promotion CASE. So an agent
that was killed left the row reading `online` indefinitely - measured on the
deployment, where `umbraaudio-m4` showed `online` against a last heartbeat nine hours
old.
That is the failure invariant I24 exists to prevent. `create_node` refuses to mark a
node ready before its agent has ever spoken, and then nothing noticed when it stopped
speaking.
Derived on read instead, in `effective_status`, called from `_node_from_row` so every
caller - console, heartbeat response, placement - gets the same answer:
offline: no recent heartbeat (derived; the stored status is still 'online')
Three decisions inside it:
- Only `online` is derived away. `draining` and `retired` are administrator
decisions about intent, and a machine going quiet does not retract an order to stop
taking work. Demoting a draining node would make the console report it as gone
rather than winding down, and blame the wrong thing.
- Nothing is written back. The stored value stays as-is and is carried on the model as
`stored_status`, so a GET has no side effect and two readers cannot disagree about
what the column says. `NodeResponse` reports it only when it differs from `status`.
- The stale-window check that was hand-rolled in the placement loop is gone. Two copies
of one rule is how they drift; the loop now reads the same function as everything
else, which is what makes the console and the scheduler agree.
A sweeper was the alternative and is how expired leases are handled, so it was not a
strange idea. It loses because it is a periodic task that must actually run on a
deployment that has already been taken down three boots by a stray process - and while
it is not running, the stored value is wrong again. Computing on read has no window and
no background dependency.
## The document was wrong about what happens when a node stops
It said "miss that window and the node goes offline". That was false, which is how this
was found. The audio node followed the instruction, sent one heartbeat, and nothing
told it or anybody that a second one was expected.
`poc/umbra_stage1.py` is a loop rather than a description of one, so the audio node can
drop it in unchanged. One `curl` is one request: it returns to the prompt when done,
which is correct for curl and indistinguishable from a healthy node that never started.
The script runs in the foreground so closing the terminal stops it, which makes the
first few minutes obvious - press Ctrl-C and the node goes quiet, and about 45 seconds
later the controller says so. It exits on 401 and 403 rather than retrying once every
ten seconds for an hour, because a credential problem does not resolve itself, and it
sends an empty report rather than nothing when the report script is missing, since a
node that stops heartbeating because its report broke is indistinguishable from a dead
machine.
Verified against the live controller: three consecutive 200s at a 10s interval, and a
wrong key producing 401 and a clean exit.
Verified on the deployment: a node quiet for 50s reads `offline` while the column still
says `online`, and reads `online` again one second after a heartbeat.
488 passed, 27 skipped. The three staleness tests were confirmed to fail with the
derivation neutered, and the export test caught `effective_status` missing from
`__all__`.
|
||
|
umbra audio
|
2ff322dcd2 |
Close all thirteen app functions in the migration, not by hand
B4 was recorded as open against the deployment. It was not open there - the ten original functions had been closed by hand - but it was open everywhere else. A database restored from a backup of the deployment comes back with 10 of 13 PUBLIC-executable, because a hand-applied GRANT is a property of one database and does not survive a restore. Observed on scratch, built from the pre-0003 dump. So 0003 now revokes from PUBLIC on all thirteen, by full signature taken from pg_proc.oid::regprocedure: a bare () does not resolve for a function taking one argument, and REVOKE on a nonexistent name aborts the whole migration. Three findings while doing it, none of which would have shown up without running the tests: - My own three guard functions went in PUBLIC-executable, reproducing the exact defect B4 is about. - Revoking PUBLIC without regranting broke five tenancy tests with a permission denied error on the platform-admin helper. Every RLS policy calls it, so a missing grant here looks like a broken policy rather than a missing grant. Regranted to the roles that actually read and write, guarded by EXISTS so a role added later is covered by the next migration rather than by luck. - The revoke block originally sat before the functions were created, so it failed on a fresh restore and passed on the already-migrated database - passing by accident, the failure mode this project keeps hitting. Both revoke blocks now sit after the triggers, and the fresh-restore path is the one verified. Verified by restoring the pre-0003 backup into scratch, applying 0003, and running the suite: 0 of 13 PUBLIC-executable, 4 guards, 474 passed, 27 skipped. Re-applying is a no-op. Deployment data unchanged (users=2 orgs=3 audit=43 perms=20); health 200, heartbeat 401 without a credential and 200 with one, so the revoke broke nothing. G2 is still open: 26 tables have RLS enabled but not forced, and the application connects as the owner. |