The first version of the drift check asked two yes/no questions per
paragraph: is anything missing, is anything extra. Measured on the real
ch-15 window it was wrong in both directions:
* It flagged five paragraphs that were fine. The model reported "the adverb
'then' is missing" and "the adverb 'Enthusiastically' is missing" on
translations that contain both, and it called a singular/plural difference
a missing sentence. A stage whose action is to delete a line cannot be
allowed to over-report at that rate: five of fifteen flags were wrong.
* It missed two paragraphs that are drifted, because those paragraphs are
short and their Chinese looks like a plausible rendering of their own
English when seen alone.
So the check now asks a different question - which paragraph does this
translation actually render - and shows the model the neighbouring
paragraphs, because that question cannot be answered without them. The
verdict is one of aligned / incomplete / drift, and only a translation that
belongs to another paragraph is dropped.
The paragraphs are then judged as a window. A re-flowed passage cannot be
judged one paragraph at a time: the model calls the two paragraphs before a
real drift "aligned" when their Chinese is demonstrably the neighbouring
paragraph's, and the start of a drift is not knowable from any single
paragraph. Inside a window that contains a confirmed drift, a fragment the
model called "aligned" is dropped as well; one it called "incomplete" keeps
its text and is flagged, because that verdict says the translation really
does render its own leading content.
On the real ch-15 window this now finds exactly the drift: ten paragraphs
(p0095-p0104) dropped, and p0094 kept and flagged because its Chinese is a
correct translation of its first two sentences. The earlier version dropped
p0094 too, losing two good lines, and kept the two drifted paragraphs the
model wrongly called aligned.
The cost of the choice is explicit: a fragment inside a broken window can be
dropped on an unreliable verdict, so the audiobook has a gap there. The
alternative is a line in the audiobook that belongs to a different sentence.
58 tests, ruff clean.
Measured on the real Chinese edition: the drafter sometimes re-flows the
passage across paragraph ids. Pass 0 splits the original passage at sentence
boundaries, so many paragraphs are sentence fragments ("The chair in the
middle has", ""operations""). A fragment is exactly where an intent-based
drafter is told to write natural target-language prose, and once it does
that, the id-to-text correspondence drifts: the Chinese is good, but it is
attached to the wrong paragraph.
In ch-15 the drift runs from p0095 to p0104 - ten paragraphs - and the
Chinese for p0102 ("Yes, Captain.") is the translation of p0101. The audio
would say the right words at the wrong place.
The alignment pass could not see it. A one-to-one paragraph is a pure copy,
so there is nothing to divide and nothing to disagree; the drift is invisible
to an alignment check by construction.
--verify now checks the paragraphs where the drift is possible: it starts at
a sentence fragment and follows the drift until the passage re-synchronises.
The check asks two questions - is content missing, and is content extra - and
a paragraph that carries content its source does not have is marked
"mismatch" and dropped from the delivery. A visible gap is better than a
confident wrong line in the audiobook.
6 new tests, 51 total, ruff clean.
The client writes each answer as it arrives, so a cache directory that was
never created turns every model call into an error - and the fallback would
quietly turn the whole book into a mechanical split while the run still
reported success.
The directory is now created by the command line entry point, and the client
is constructed with it rather than having it assigned afterwards. A test
pins it.
45 tests, ruff clean.
The translation pipeline works on paragraphs, because epub and pdf are
paragraph media. The audiobook needs utterances: UmbraAudio has verified who
speaks every one of them, and each clip is voiced separately by that
speaker's voice. A paragraph-level translation cannot be spoken.
This is the pass that was missing. tools/translation/README.md said so
plainly: "It does not produce a 1:1 utterance alignment for TTS ... splitting
or aligning back to the 4,319 utterances is a separate pass."
The division is by meaning, not by position. An intent-based translation
merges and splits sentences for the target language's sake, so the sentence
counts do not line up on their own - that is the whole point of Passes 1-2.
A model is given the source utterances and the translation and asked which
part of the translation renders which source utterance.
Two rules the model cannot break:
* a source utterance may only be absorbed into another utterance by the same
speaker - a line must not be handed to a different character;
* the partition must reproduce the translated text exactly, modulo
whitespace, so the alignment can never rewrite the translation.
An answer that drops, duplicates or invents text is rejected and the
paragraph falls back to a mechanical split, which is reported as needing
review. A source utterance the target does not voice is delivered as
voiced=false with the reason, which is what UmbraAudio's
utterance_translations.voiced flag is for: the row is kept so the gap stays
visible, and no clip is generated for it.
The output is a delivery payload in the shape UmbraAudio's
translations.import_translations already accepts, so neither side has to
invent a second contract.
Measured on the real book, 42 sampled multi-utterance paragraphs across seven
chapters: 41 aligned by the model, 1 fell back to the mechanical split, 0
partition mismatches. The one fallback was a model answer that did not
reproduce the translated text; the content check caught it.
44 tests, ruff clean.
- Add migration 0010: utterance_translations table with UNIQUE(utterance_id, language_code)
- Add schemas: OrganizationCreateRequest/Response, ProgramCreateRequest/Response,
UtteranceTranslationItem/Request/Response/ListResponse, SpeakerSyncRequest/Response
- Add service methods: create_organization, create_program, sync_speakers,
upsert_utterance_translations, get_utterance_translations
- Add routes_controller.py: POST /api/orgs, POST /api/programs,
POST /api/speakers/sync, POST /api/projects/{id}/translations,
GET /api/projects/{id}/translations/{language_code}
- Mount controller_router at /api in app.py
- Create bdumbra account (member, BDMurphy org) for integration testing
Add REST endpoints and frontend wiring for per-language project actions:
- POST /api/workspace/{project}/translation/{lang}/queue — queue translation
- POST /api/workspace/{project}/audio/{lang}/queue — queue audio generation
- GET/POST /api/workspace/{project}/format/{lang} — format metadata (EPUB/PDF)
- GET /api/workspace/{project}/review/{lang}/chapters/{chapter} — utterance review
- GET/POST /api/workspace/{project}/front-matter/{lang} — front matter editor
- GET /api/workspace/{project}/chapters/{lang} — chapter list
- GET /api/workspace/{project}/chapter-content/{lang}/{chapter} — chapter text
Frontend fixes:
- Review page shows per-language text (English source, Chinese translation, etc.)
- Non-translated languages show 'Queue translation' button only
- Speaker data from English source matched by paragraph ID
- UmbraAudio column layout: Speaker, Variation, Text, Actions
- Chapter tree in left sidebar: Front matter, Back matter, Body
- Audio Format label at top of sidebar
- Pandoc formatting toolbar for front matter text editing
- Added exception handler for BundleError in app.py that returns HTTP 422
- Response includes code='invalid_bundle' and message from error argument
- Previously returned unstructured 500; now consistent with other validation errors
- Apply migration 0008_audio_source_import (additive schema for source import)
- Apply migration 0009_translation_import_runs (ledger table for import runs)
- Wire PostgresAudioImportStore into HTTP commit path with org from node credential
- Commit path uses PG store with org from node, no payload org
- Bundle hash 21ce5687… remains frozen until v2; no authority transferred
docs/prompt-identity.md, and tests that assert the mechanism rather than describe it.
The question this came from: are UmbraAudio's LLM prompts code, or data that gets pulled
and sent? The answer is neither, exactly - they are markdown files read at runtime:
PROMPT_PATH = Path(__file__).parent / "CHARACTER_SEGMENTATION_PROMPT.md"
prompt = template.replace("{chapter_text}", text)
client.chat.completions.create(messages=[{"role": "user", "content": prompt}])
That is a good design for *using* a prompt, and it is worth keeping. Editing the file and
re-running needs no code change and nothing to redeploy. Three things are missing, and all
three bite the moment a prompt is iterated on - which is the normal way to improve one, and
certainly how the intent scan will be developed.
1. No identity on the output. `utterances` records text, speaker, emotion and pace, and
nothing about which prompt produced them. After an edit and a re-run, "which paragraphs
were scanned under which version" is not difficult to answer but unanswerable.
2. No caching, so no invalidation to reason about. `llm_segmenter` has no cache references
at all, so re-running `analyze` re-sends every chapter. Editing a prompt costs a full
book scan, which is the cost that makes people avoid iterating on prompts.
3. Re-analysis appends. `insert_utterance` is an unconditional INSERT with no upsert key, so
a second `analyze` adds duplicate utterances rather than superseding the old ones.
The precedent for fixing this already exists one layer too late. `generation/staleness.py`
solves exactly this for audio - a sidecar per wav recording spoken text, speaker, voice_fp
and sha256 - and its docstring explains why: a "file exists" check silently reuses audio
that no longer matches the utterance text. The reasoning is right and the implementation is
careful, and it applies to audio only. The expensive artefact is protected and the cheap
one is not. This design extends the existing mechanism one layer earlier rather than
inventing one.
Three fields travel with everything an LLM produced: producer, prompt_id, prompt_sha.
prompt_id is the label a human reasons with; prompt_sha is the mechanical guarantee, taken
over the *rendered* prompt rather than the template - because hashing the template would
report two runs over different chapters as the same call, which is the one distinction the
mechanism exists to make.
The load-bearing test is that a version and its content must agree. A prompt file edited
without a version bump makes "v1" name two different prompts, and every query written
against it silently returns half the answer while looking like provenance. That
combination is refused rather than recorded, because a label that can quietly become untrue
is worse than no label.
test_prompt_identity.py states the mechanism as behaviour: that any edit changes the hash,
that different chapters render differently, that an unbumped edit is refused, that a bump
makes it legitimate, and that v1 still resolves after v2 exists. One of those tests failed
on the way because the registry helper rewrote v1's hash - backwards for that case - which
is recorded rather than quietly fixed.
Mutation-verified: disabling the version/content guard fails 1 test, and hashing the
template placeholder instead of the rendered prompt fails 3.
Does not cache responses - that is a separate decision needing a separate answer about when
a cached response is wrong, and it comes after this, not before. Does not touch UmbraAudio's
own prompts, which need the same treatment for the same reasons; that is a change to their
repository.
716 passed, 4 skipped.
Asks for what is needed to write a correct mapping, in a form that can be verified rather
than read. Two documents:
docs/audio-system-data-request.md - what to send, in what format, and why that format.
docs/audio-system-bundle-validator.py - ships *with* the request rather than after it, so
the sender can check their own work before it travels. Standard library only: whoever runs
it runs it on the audio system, and adding a dependency to someone's working machine to
validate a file is not a reasonable thing to ask. That means the schema checking is the
subset possible without a library - a smoke alarm, not an inspector - and the file says so
rather than implying more.
The reasoning behind the format is the substance. Prose cannot be validated: every mismatch
is something a human has to notice, and the ones that get missed are the interesting ones -
derived chapter order, two fields that mean slightly different things, a path relative to
something unspecified. A validator that names the record and the field turns a round trip
into a checklist, and reports every problem rather than stopping at the first, because a
validator that stops at the first failure makes a three-round-trip conversation.
Also asks for the things that are cheapest to supply and most expensive to guess: the
vocabulary table (guessing terminology is how a mapping looks correct and reads the wrong
field), which fields are derived (a derived chapter order treated as authoritative will
disagree the first time somebody reorders a chapter by hand), and what the TTS engine
actually is - fork or patch, what changed, whether it loads a model and how big. That last
one decides whether a 48 GB CPU-only node can run it at all, so the speak work function
cannot be written until it is answered.
Explicitly not asked for: media files, source code, screenshots, or anything the sender
would have to reconstruct. "We do not record this" is a complete answer and far better than
an estimate presented as fact.
The validator was itself tested against a deliberately broken bundle, which is the only way
to know a check can fail. It caught a dangling reference, a unitless duration, two unanchored
paths and a planted credential, and passed a correct bundle. Two flaws in it were found
that way and fixed: it reported the same path problem twice, and every credential twice,
because a tool whose only job is to be trusted about specifics loses credibility the first
time it says the same thing twice.
704 passed, 4 skipped - no product code touched.
The audio node is the master for the audiobooks. UmbraTrans holds the metadata needed to
reason about a file and a reference to where it is; it does not hold the bytes. DECISION-6
(object storage) stays unresolved, and this is what lets the system be useful before it is
resolved rather than after.
POST /api/compute/artifacts takes a manifest against a project and declares each entry:
content hash, size, media type, kind, and a storage_uri naming where the file actually is.
GET /api/compute/artifacts lists the catalogue. There were no artifact routes at all
before this - the tables existed since migration 0003 and nothing had ever written to them,
so "the controller has the artifacts to show a job is complete" had nothing behind it.
Every declared row lands integrity='unverified', and that is the point rather than a
caveat. Ingesting is a claim: a node reporting the hash of a file is a statement about
bytes nothing here has read, because there is no object storage to read them from. A design
where submitting a manifest marks the catalogue verified would make the first ingest the
most reassuring and least true event in the system's life. verified_at is set by reading
the bytes back and hashing them, and nothing in this path does that.
storage_node_id is part of the reference rather than an afterthought, because a bare path
means nothing six months later: /audio/book.mp3 is on whichever machine happened to serve
it that day.
Idempotent on (project_id, content_hash), so the same manifest can be submitted twice, or by
two operators who did not talk to each other, without the catalogue doubling - and a
partly-present manifest reports what was new rather than just "done". An operator who
submitted 300 entries and was told "done" would not know two were malformed and dropped, so
malformed entries are named in the response instead.
The listing is scoped by organization on every call. artifacts is platform-level, as every
table a node serves from is, so a missing predicate there is a cross-tenant read rather
than a style issue. A project in another tenant's schema is refused as *not found*, not as
*not permitted*, because a refusal that says "this exists but is not yours" answers the
question the caller was actually asking.
Migration 0007 applied to the deployment.
Mutation-verified: marking an ingest verified fails 1, dropping the idempotency check fails
2, and dropping the organization predicate from the listing fails 7.
704 passed, 4 skipped.
The 720px breakpoint contained three rules:
.admin-sidebar { display: none; }
.rail-link:not(.rail-link--active) { display: none; }
.topbar-admin-link { display: none; }
Every one hides navigation rather than relocating it. On a phone the console showed the
page you were on and no way to reach any other: the admin sidebar was gone, every rail
link except the current one was gone, and the route from the workspace to the control
panel was gone. There was no hamburger to find because nothing had ever been built for a
narrow screen - the breakpoint assumed the sidebar could simply go away, which is only true
when nothing else is left to navigate with.
A toggle appears below the breakpoint and reveals the navigation as a drawer. The drawer is
positioned rather than shown inline, because an inline nav pushes the page down by its own
height every time it opens, so on a short viewport the page the operator asked for ends up
below the fold. Escape closes it, a click on the page behind it closes it, following a link
closes it, and crossing the breakpoint settles the state - without which the drawer class
survives into the wide layout and the bug only reappears when the phone is rotated back.
Above the breakpoint the toggle is hidden, because a control that opens what is already
open misreports the state of the page.
nodes.html is deliberately not given a drawer. Its navigation is inline in <main> rather
than in a sidebar, so there is nothing to put in a drawer and a hamburger there would
toggle nothing; its nav was a single horizontal scroll strip instead, which hides links off
the right edge with no sign that there are more, so it now wraps.
test_navigation.py states the rule - navigation hidden at a width must still be reachable
at that width - and reads the breakpoint with comments stripped. Two things that check got
wrong on the way, both recorded in the file:
- It flagged the drawer's own "display: none" as a violation. Hiding a closed drawer is
correct, or it would push the page down on every load; what is wrong is hiding navigation
with no way to open it. The test now requires a "body.nav-open" rule alongside any hide.
- It only required that *some* page carry a toggle, so deleting it from the admin console
- the page that was reported broken - passed, because the workspace still had one. Now
per page.
Mutation-verified: restoring the original defect fails 1, removing the toggle from the
admin console fails 1.
695 passed, 4 skipped.
Replaces claiming with assignment. The controller hands a job to one or more nodes and a
node asks what it has been given; jobs.lease_owner could name only one holder, so a job
fanned out to several machines was inexpressible, and 'what was assigned and what is not
complete' had no answer. job_assignments is one row per (job, node, attempt), so both are
now readable.
Nodes poll rather than being pushed, which is not a simplification: nodes sit behind NAT
and the controller has no route to them. The only path that exists is a node calling out.
The rule with teeth: a job is complete when the controller has the artifacts to show it.
Reporting is not believing. A node that finishes and hands over nothing gets its
assignment failed, with the reason naming what was missing, and the work stays available
for another machine. The tempting implementation - mark complete, then verify afterwards -
cannot fail, because the only thing it could compare against is the report it just
trusted. artifacts_verified_at is written by verification alone, so 'the node says it is
done' and 'the controller confirmed it' are different facts that cannot look alike.
fan_out defaults to 1 and is never inferred. More than one is redundant execution: several
machines do the same work and the first verified result wins, the rest are cancelled.
Worth having for cheap idempotent work, wasteful for anything costing GPU-seconds, so a
caller has to say so. The losers are cancelled rather than left running - otherwise every
machine on the list burns its time on a result nobody will read.
Four defects found while building it, all by tests written first:
- Naming a node explicitly bypassed the drain check. An administrator could drain a
machine, see it read as drained, and still hand it work: the control doing nothing while
appearing to work. Status is now checked on both routes.
- attempts were counted from jobs.attempt, which only claim_job incremented, so under
assignments max_attempts was never reached and a failing job was re-offered forever.
Counted from failed assignments now, so a fan-out to three costs one attempt, not three.
- start_assignment reported 'not startable' when the real problem was that the assignment
belonged to another node. Ownership is checked first, because an operator handed the
first will look in the wrong place.
- The artifact API accepted values the database rejects: role 'output' where
job_artifacts_role_check allows only 'consumed' and 'produced', an unbounded status, a
content_hash that was not 64 hex characters, and kind at 80 where the limit is 64. Every
completion would have been a 500. The schema had fixed this vocabulary all along and
the API had to learn it rather than invent it.
Artifacts are recorded even when verification fails: a short run still produced evidence,
and discarding it would destroy the record of why. storage_key is never treated as proof
the bytes exist, because there is no object storage to ask yet (DECISION-6).
Claim still exists. It is listed in the pinned node surface as pending removal and the
next change takes it out - two paths handing out work is the state this project keeps
finding and criticising.
Mutation-verified: trusting the node fails 3 tests, dropping the drain check fails 1,
leaving fan-out losers running fails 1.
674 passed, 4 skipped.
node_credentials.revoked_at has existed since the first migration and
resolve_node_credential has always honoured it. Nothing could set it. The ability to cut a
node off was present in the data model and absent from every API, which is the shape of
defect worth naming: the column was the capability.
POST /nodes/{id}/revoke cuts a node off and leaves it - right for a compromised key or a
machine about to be reimaged. POST /nodes/{id}/remove takes the machine out: credentials
revoked, devices, heartbeats and model reports deleted, status 'removed'. 'removed' is
deliberately not settable through PATCH /nodes/{id}/status, because typing 'remove' as a
status change would skip the revocation and the release of held work, which are the two
things that make removal mean anything.
The row is kept and cannot be deleted. job_events.node_id and
node_pairing_codes.used_by_node_id both reference compute_nodes ON DELETE NO ACTION, so
the DELETE is refused; changing them to SET NULL does not help, because that is an UPDATE
of job_events and job_events_append_only raises on every update by design. So a node that
has ever done work cannot be removed without weakening the trail's immutability, and the
trail is worth more than a tidy DELETE. A trigger makes 'removed' terminal, and
set_node_status refuses it first so the caller gets a reason rather than a constraint
violation.
Work the node was holding is released at removal rather than left to its lease. An
operator removing dead hardware already knows the work did not finish, and making them
wait out a lease to find that out is the wrong way round - for a long lease, most of an
hour of queue stalled behind a machine nobody is going to switch on.
Also creates job_assignments, which is how the controller will hand work to a node:
one row per (job, node, attempt), so a job can be open on several machines at once and
the controller can say which of them finished it. state is deliberately not the job's
status, and artifacts_verified_at is written by verification alone - that column is the
whole of 'a job is complete when the controller has the artifacts to show it'.
Migration applied to the deployment inside a transaction that first normalised any
'online' node to 'provisioning', since the new terminal-state trigger refuses that
transition and those rows would otherwise have blocked it.
Mutation-verified: not releasing held work fails 6 tests, leaving the credential working
fails 1, and a revocation that sets revoked_at back to NULL fails 2.
652 passed, 4 skipped.
Several nodes legitimately want to be called VM238 - one per hypervisor, one per rack, one
per testbed. The unique constraint on compute_nodes is (organization_id, name), so until
now the second machine to pair was refused with node_name_taken for a name that looks free
to the person typing it, and the fix of 'pick another name' pushed the bookkeeping onto
the operator for something the controller can derive.
A node's name is now its base name plus the last four hex digits of its MAC: VM238-AC4B.
Four digits because the name appears in a UI column, a command line and a systemd unit.
Sixteen bits narrows collisions rather than eliminating them, so a letter fallback sits
behind it - VM238-AC4B-A - and the sequence continues past Z, because a sequence that
silently runs out hands out a name that is already taken.
The code stays bound to the *base* name. The controller cannot know a MAC before the
machine speaks, so the final name is settled at redeem, where 'is this taken' can be
asked of the database rather than guessed at by the script on the machine. The script
reads the first non-loopback, non-virtual interface, sends the MAC, and then adopts the
name the controller returns rather than the one it asked for - reporting one while
registering the other is how a node ends up appearing under a name it does not have.
Three things this deliberately does not do:
- A MAC that is not six octets of hex is rejected rather than cleaned up. The permissive
version turns 'not-a-mac' into AMAC, and VM238-AMAC is worse than a node admitting it
does not know its own address.
- Without a MAC the plain base name is used and a second node of that name is still
refused. There is no evidence the machines differ, and inventing a letter would turn
one ambiguous input into unbounded identities.
- Re-running bootstrap on an already-onboarded machine now succeeds as VM238-AC4B-A
rather than being refused. That follows the specified rule - on conflict, add a letter -
and it is called out in the test that asserts it: a stale node cannot be assumed gone
because its hardware came back.
Mutation-verified: the octet bug fails 10 tests, removing the letter fallback fails 2, and
the bootstrap script ceasing to send the MAC fails 1.
641 passed, 4 skipped.
Two defects, either of which alone would have stopped a node doing work, and both of
which looked finished.
Pairing issued a read-only credential. node_agent.py already calls claim, renew,
complete, fail and decompose, and no credential pairing could ever issue was permitted
any of them - so the agent looked ready, every endpoint existed, and not one of those
requests could succeed. The same shape as the login form with no method attribute:
everything present, nothing working.
A paired node now receives the whole node scope set. The pairing code is the
authorization: minted by an authenticated administrator, single-use, expiring, bound to
one node name, rate-limited per source address. An administrator who mints a code has
already decided to put that machine on the fleet. This does not weaken the boundary that
matters - NODE_SCOPES contains nothing that reads project content, and a node can hold
several credentials, so a monitoring-only machine is still issued a narrow one and this
one revoked.
Separately, claim_job never looked at the node. set_node_status documents that draining
'stops new claims and lets running work finish', and the add-node page offers drain,
return and retire; none of them did anything to the claim path. A machine drained before
a reboot kept being handed work by an agent that was still running. The check is inside
the claim transaction and takes FOR UPDATE on the node row, so an administrator draining
a node cannot land between the check and the claim. It reads effective_status, so a node
whose agent has died stops being offered work rather than taking a lease it will never
renew - and deliberately not is_schedulable, which also requires a free device, since
claiming a job is what makes a device busy.
That gate found four tests in test_node_tenancy.py claiming work from nodes that had
never authenticated - the state I24 exists to prevent. They now bring their nodes online
first. Confirmed the tenancy property is untouched: restoring the tenant filter still
fails 5 tests, the same count as before.
Every fix mutation-verified - removing the status gate fails 5, restoring read-only
scopes fails 2 including a new end-to-end test that pairs a node over HTTP and has it
claim and complete a real job.
605 passed, 4 skipped.
The PostgreSQL adapter suite refuses to run against a database that already holds rows,
because it truncates indiscriminately and cannot do so selectively - audit_events is
append-only, with a trigger that raises on delete. The refusal is correct. Two separate
faults meant it was firing when it should not have.
The guard sampled the live tables at the moment it was first requested. pytest runs files
alphabetically and several files before it create organizations and tenant schemas, so by
the time it looked, this session's own rows read as pre-existing data. It skipped 27 of
63 tests on a scratch database that had been empty moments earlier, and '567 passed'
carried no hint of it. The sample is now taken in pytest_sessionstart, before collection.
A session-scoped autouse fixture was tried first and is not early enough - it is
instantiated lazily and does not precede the other session fixtures that write.
Nothing cleaned up after the session. Ten files open their own connection to the DSN
outside the postgres fixture, each inserted organization provisions a tenant schema, and
no test dropped it: 78 schemas per run, which after about fourteen runs exhausted
max_locks_per_transaction and errored the integration tests with a message reading like
a PostgreSQL defect.
And the fix for that nearly caused the disaster the guard exists to prevent. The first
version of the cleanup truncated unconditionally on teardown, so pointing the suite at a
populated database destroyed it while every test that could have touched it skipped
itself. Found by planting an organization, watching the suite refuse, and finding the
organization gone afterwards. It now leaves an occupied database exactly as found, and
says so - through pytest's terminal reporter, because a plain write to stderr is
swallowed by file-descriptor capture and was seen by nobody.
test_scratch_database_safety.py covers all three, and each was confirmed to fail when the
protection is removed. Three mistakes in writing it are recorded in the file: a leading
test file that cleaned up after itself and so reproduced nothing; assertions that passed
because '-rs' was missing and pytest had printed no skip reasons; and a check that
stopped at the warnings header, which pytest prints its summary line after.
598 passed, 4 skipped. The 4 are environmental: two pages with no password field, two
needing passwordless sudo for device permissions.
The hint said to run 'sudo visudo' as though the account running the command already had
sudo. On a machine set up the way this one was - a separate administrator, then a
daily-use account - the daily-use account is not in the sudo group, so that instruction
could not be followed by the person who needed it. Replaced with the two steps that
actually apply, in order:
1. The account running the command must be in the sudo group. Check with 'id -nG'; if it
is not there, an administrator adds it with 'usermod -aG sudo <account>' and the
account signs out and back in, because group membership is fixed at sign-in. This is
the step that was missing, and it is the one that cannot be done by the person stuck.
2. Passwordless sudo, from an account that already has sudo: 'sudo visudo -f
/etc/sudoers.d/umbra-node', add the NOPASSWD line, save. Optional - without it the
command still works and simply prompts.
Written after a run stopped at a sudo password prompt with nothing on screen but 'Sorry,
try again.', which is sudo's own message and appears nowhere in this codebase, so the
failure looked like the system refusing rather than the shell asking.
Checked against 238: umbra is already in the sudo group, so only step 2 applies there.
The commit immediately before this one - the password-toggle fix - deleted a closing
brace and did not replace it. app.js did not parse. The browser reported it in a console
nobody was watching, and every script-driven behaviour on the sign-in page stopped.
The consequence was not a cosmetic one. The login form's submit handler, which is the
only thing calling preventDefault(), never ran, so the browser performed a native form
submission. Neither form declared a method, and a form with no method attribute submits
as GET. Both password fields went into the query string:
GET /?email=umbra%40bdmurphy.me&password=...
The server logged it in plain text, three times, before it was noticed. It would also
have sat in browser history and in any proxy access log between here and the public
internet.
Two fixes, the second of which is the one that matters:
- The toggle fix is reapplied correctly to the last known-good version of app.js, with
nothing else in the file touched - the diff is four lines against thirteen and two
against one.
- **Every form on every page now declares method=post.** The script still intercepts
and submits via fetch; this decides what happens when the script does not run, which is
precisely the case that failed. Five more forms carried the same latent defect,
including password reset and email verification.
test_form_safety.py checks both, and was confirmed to fail when app.js is broken the
same way. It uses esprima rather than a hand-written brace counter, which gave two false
readings during this work - one of them 'balanced' for a file that did not parse at all.
A check that can be wrong about a syntax error is worse than no check.
The account password should be treated as exposed and changed. So should any password
entered into that form while the script was broken.
The variable was called 'reveal' and held `input.type === 'password'` - which is true
when the password is HIDDEN - and was then used to pick the label. So a masked field
offered 'Hide', and clicking it revealed the password and relabelled itself 'Show'. The
click handler inverted the same way, so the two errors cancelled and the visible result
was a control that appeared to work while doing the opposite of what it said.
Renamed to 'visible', which is what the value now means. A correct-looking name is not
what prevents this; the value and the name agreeing is.
Affects the login field and both fields on the change-password screen.
Verified by running the two states rather than by reading the edit: masked says 'Show',
plain says 'Hide', and two clicks return to the start.
Three navigation faults, all found from the service request log rather than by looking
at the page:
- The console had no link to the nodes page at all. It was a standalone file I created
and never wired in, so leaving the nodes page meant there was no way back except the
browser. It is in the nav now, and every href on the console was checked against the
running service rather than assumed.
- VIEWS omitted 'projects' although the section and its nav link both exist, so clicking
Projects silently fell through to the dashboard. A link that looks live and does
nothing is worse than no link. Flagged in an earlier session as not-my-problem and left
alone; it was two characters.
- A 401 left the nodes page sitting there saying the session had ended, with nothing on
the page that could work without one. It now redirects to the console, which is what
the console does itself.
The empty organization dropdown was this too: with no session the organizations request
is 401, the list is empty, and the form is disabled with the new banner - which is
correct behaviour for a genuine no-organizations system and a misleading one for an
expired session. Redirecting first is what tells those two apart at the point where it
matters.
547 passed, 29 skipped.
The organization dropdown rendered empty. Three separate defects behind it, found from
the service's own request log rather than by looking at the page:
- /api/uadmin/organizations returned 200 - the data was arriving - while
/api/compute/nodes returned 422 on every load. ORGANIZATION_ID was only set when the
account could see exactly one organization, so for anyone who can see several the node
list was requested with no organization and the compute routes refused, as they are
designed to. The list is now fetched when a choice is made, not at load.
- The page had no case for *no* organizations. A node belongs to exactly one, so there is
nowhere to put it: the form is now disabled and a banner explains why, with a link to
the organizations screen. A dropdown with nothing in it and no explanation is the same
as a broken page.
- A 401 - an expired session - was indistinguishable from an empty list. It now says the
session ended, which is the one failure the operator can act on.
Also removed eight CSS classes I had invented rather than looked up: card, card-body,
card-note, field, form-actions, table-wrap, alert--warn and button--quiet. None exist in
styles.css, so the page was rendering unstyled. Replaced with what the working console
page uses - admin-card, form-field, modal-actions, admin-table-wrap, admin-banner--
warning, button--ghost. Every class on the page is now verified present in the stylesheet
rather than assumed, and the markup is checked as balanced by a parser rather than by
counting tags by eye.
0005 was applied to the deployment to unblock startup: the tenant-schema contract check
runs at boot and correctly refused to start a service whose schema it could not find. That
is the check working. Backup taken and checksum-verified first; three tenant schemas
created, one per organization, with users=2 orgs=3 nodes=9 audit=46 unchanged.
547 passed, 29 skipped.
It described the control as a hint whose effect was decided at run time, which was
true and misleading: the paths are chosen here, at render time, and the command
follows.
as_root selects which paths the rendered script writes to: /opt/umbra and
/etc/umbra/node.json, or the invoking user's own directories. The command did not follow
it. Ticking the box produced a script that wanted to write to /etc while being run by an
unprivileged shell, which fails at the first mkdir - after the pairing code has already
been spent, so it also leaves the operator with a dead code and nothing to show for it.
The command now carries sudo when the box is ticked. The script still checks for root at
run time and takes whichever branch is actually available, so this is belt and braces
rather than the mechanism; but a control that produces a command which cannot work is
worse than no control at all.
The checkbox's help text described it as a hint whose effect was decided at run time,
which was true and misleading: the effect is decided here, at render time.
Qwen3-TTS as modified on the audio node, kept. Quality over time is the principle that
settles it, and the engine already in use is the one worth keeping.
The modifications are on the audio node and unknown here, so the speak work function
cannot be written blind. Listed what is needed: what changed and why, whether it is a fork
or a carryable patch, the input contract, and whether it loads a model and how large -
which decides whether node 238 can run it at all.
Parallel synthesis is a requirement, not an optimisation. The mechanism exists already
(submit_children and the decompose route), which means the work-function protocol has to
be designed for decomposition rather than have parallelism retrofitted into it.
`claim_job` took an `organization_id` and matched it against the job's. Read plainly
that says: a machine may only do work for the tenant that owns it. Both paired nodes
belong to BDMurphy and The Last Dwarf belongs to WorstAuthor, so a node could never do a
word of the audio node's work however idle it was. The only way to have found that out
was to queue the job and watch it sit there.
`place()` had the same fault - it listed candidate nodes by organization - so the
scheduler could not use the fleet across tenants either. Both are fixed.
A node is a machine. It belongs to whoever administers it, not to a tenant, and a
tenant's data reaches it inside one job's payload for the duration of that job, never by
widening what the node itself can see. `kind` and the placement requirements decide what
a node may do; ownership decides who may switch it off. What did not change: a node still
returns one job's payload and nothing else, so it still cannot enumerate another tenant's
work.
test_node_tenancy.py asserts the behaviour, and was confirmed to detect the old one: five
of its seven tests fail when the tenant filter is restored.
Also adds docs/plan-translation-pipeline.md, which separates the critical path - the
work-function protocol, TTS, assembly, lineage, and one finished episode of The Last
Dwarf - from the work that can proceed beside it.
Writing those tests cost more time than it should have and the reason is worth recording:
four of them failed for reasons that had nothing to do with tenancy. A job left behind by
one test is claimed by the next, which reads as the rule breaking. `TRUNCATE ... CASCADE`
on a table shared with other fixtures destroyed `project_publishers` and surfaced two
files away as "relation does not exist". The queue is now left alone and the claims filter
on a job kind only this file creates, so a neighbour's leftovers cannot be returned and
the assertions can be exact.
547 passed, 29 skipped.
`projects` moved into a per-tenant schema and `jobs` stayed in `public`. A foreign key
cannot cross that boundary without postgres_fdw, so 0005 dropped
`jobs_project_id_fkey`, `artifacts_project_id_fkey` and a composite `user_roles` key. The
migration recorded the consequence honestly - "the application still refuses it; the
server no longer does" - but the application did not.
So after the split nothing stopped a `public.jobs` row naming a project in a different
tenant, which is precisely the leak the schema split exists to prevent. `submit_job` now
verifies it.
Two things that were wrong in the first attempt, both caught by running the tests:
- The lookup was unqualified, relying on `search_path`. `ComputeRepository._tx` does not
set it - only the account repository does - so the check ran against `public` and could
never see any project, refusing every project-scoped job. The schema is now derived from
the organization id and named explicitly: the check is self-contained, and does not
depend on ambient state deciding whether a security check runs at all.
- It filtered on `projects.organization_id`. That column no longer exists - the schema the
row sits in *is* the tenant, which is the whole point of the split. There is no
predicate to write.
`test_a_nonexistent_project_is_refused` asserts that a caller cannot distinguish "no such
project" from "another tenant's project". From inside a tenant schema the two are the same
observation, and they should be.
Confirmed the tests detect the gap: with the check removed, three of five fail, including
`test_the_refused_job_is_not_created` - which exists because a refusal that still writes a
row is not a refusal.
547 passed, 29 skipped. Deployment untouched: 30 tables, 0 tenant schemas.
ruff format disagrees with about two dozen files in this repository, most of them
untouched by this work. Reformatting them would bury the tenant-schema change in
noise, and reformatting only some of a file is its own kind of noise - so only the
lines written here are brought into line, and the pre-existing drift stays where
it is for whoever owns it.
The cleanup I added dropped every tenant schema in a single transaction, and a
hundred runs' worth of them exhausts the lock table: the session teardown failed
with 'out of shared memory ... max_locks_per_transaction', which reads like a
PostgreSQL defect and is not one. It is the same constraint the 0005 rollback
documents and I wrote there before hitting it here.
Committing after each drop releases the locks, and the teardown now leaves zero
schemas behind, so a run does not make the next one slower.
Same shape as the append-only guards: the project row is created in the
organization's own schema and the cleanup drops that schema. The old cleanup
deleted from public.projects, which since 0005 has cleaned nothing up while
looking like it did - the sort of teardown that reads as careful and is not.
public.projects does not exist since 0005, and this fixture was the last place
still writing to it. The project row moves into the organization's own schema
under an unqualified name, which is also the shape production writes it in - so
the artifact and job rows guarded by these tests are being set up the way the
application would set them up rather than the way the old shared table allowed.
Three changes, all of them consequences of the migration rather than new
questions.
test_rls_coverage.py drops project_authors and project_publishers from the
policy list - they are per-tenant now and there is nothing for a policy to
filter - and replaces that entry with checks that are stronger than what it
replaces. A policy on a shared table proved the table was filtered; a per-tenant
table cannot be filtered at all, so the properties worth asserting are that no
shared copy exists in public, that no policy lingers on a tenant table, that
every tenant schema carries all four tables, and that organization_id is gone
from all of them. That last one matters because it was the column every removed
policy filtered on: leaving it behind would leave an application filtering by a
value nobody maintains.
It also gains a check that POLICY_TABLES names tables that are actually in
public. Without it, moving a table out of the policy list and out of public in
the same change makes both existing checks pass by not looking at it - which is
how a table can quietly stop being policed and still look policed.
test_tenancy.py covers the search_path function directly, and the case that
matters is the negative one: a request with no organization gets public alone.
A wildcard, or leaving the connection's default in place, would put every
tenant's tables in reach of a request that has not decided which tenant it is
acting for.
test_postgres_repository.py proves the new startup contract check holds against
the real app function, and that it can fail. A check that cannot fail is the
same shape as a policy nobody wrote: it reports success while enforcing
nothing.
The emptiness guard now also looks inside the tenant schemas and the run drops
them at the end. Not tidiness: a database holding nothing in public can still
hold somebody's books, and a guard that only looked at public would pass on a
database it was about to empty.
The old file proved a policy refusing a row. With no shared table there is no
policy and nothing to refuse, so the tested property is gone rather than
rewritten - and Umbratools_2.md 5.0 says outright that this trade should be
noticed rather than discovered later. This commit is the noticing.
What is proved instead is stronger, not differently shaped. A statement with no
organization in hand asks for `projects` unqualified and is told the relation
does not exist: not zero rows, an error. That is a different claim from "the
policy filtered it", because a policy that was silently not enforced returned
zero rows and looked fine - which is the failure G2 was, and the reason the old
test had to run as a non-owner role to mean anything.
The new tests keep the non-owner discipline for the part that needs it. Naming
another tenant's schema is refused outright; repointing the search path at it
and using an ordinary name finds no table at all, because a role without USAGE
on a schema does not see its contents when names are resolved. That second one is
the outcome an application actually experiences, so both are asserted rather than
the convenient one.
Positive cases are here because a suite that only proves refusal passes just as
happily against a database where every write is refused: a write that must be
allowed is performed and then counted in both tenants.
What the file cannot prove is written down in its docstring rather than left to
be assumed. The application owns every tenant schema, so it can reach another
tenant's tables by naming that schema, and schema-per-tenant cannot change that
with one role serving every tenant. What it changes is that there is no shared
relation to reach through, and reaching another tenant means writing down another
organization's name - one call site in tenancy.py rather than a predicate in
every query.
Every project statement now says `projects` unqualified and takes its
organization through search_path. A shared table filtered by
`organization_id = %s` is a filter somebody has to remember, and there is always
a write path that would not have it; an unqualified name resolves into the
caller's own schema or it does not resolve at all. The strongest possible
mistake - forgetting to scope a query - now raises instead of returning a wider
result set.
Project.organization_id survives in the model and is filled in from the schema
the answer came from. Callers still see which tenant a row belongs to; it is
derived from where the row is rather than from what the row says about itself.
Two consequences that are worth more than the mechanical change:
- assign_user_to_project now takes the organization as an argument. It used to
read it off the project row and copy it into user_roles, and there is no
column to read now. user_roles is platform-level, so it cannot be written to
by looking inside a tenant schema - so the caller, which has already resolved
the project, has to say it. The protocol declares it, and the in-memory store
accepts and ignores it rather than being the one implementation with a
different signature.
- A platform administrator listing projects without naming an organization now
gets the answer assembled across tenant schemas, because there is no shared
table to ask. That is one round trip per organization and it is paid only by
that one caller; every per-tenant path, which is what an ordinary request
does, is still a single statement. The merge reproduces the ordering and the
total a shared table would have given, so paging across the boundary is the
same as paging inside one.
Two things now go onto a pooled connection before a statement runs: the caller's
identity, which the surviving platform policies read, and the caller's schema,
which decides which tenant tables the statement can even name.
search_path is set on every transaction rather than only the ones that have an
organization, so a platform context gets public alone instead of whatever the
connection's own default happens to be. That is the difference between a design
and a hope: a default naming a tenant schema is a silent leak with no line of
code mentioning it.
verify_tenant_schema_contract exists for the same reason verify_rls_contract
does. The schema name is derived in two languages - this one and app's - and
nothing joins them. A disagreement is silent and total: every tenant query
resolves into a schema that is not there, every project request for every
organization raises "relation does not exist", and the error names the statement
rather than the cause. That presents as an empty database, which is the one
diagnosis nobody reaches for.
tenant_search_path is the whole isolation model in one function: given the
organization a request resolved to, return the schema its tables are in; given
none, return public alone. That second case is the point. Returning a wildcard
- or leaving the connection's own default in place, which may name a schema
nobody chose - would put every tenant's tables in reach of a request that has not
decided which tenant it is acting for.
The name is derived from the organization id rather than stored, for the reason
the migration derives it too: a recorded name that disagreed with the schema the
rows are in would describe data that no computed name can reach, and that state
would be indistinguishable from "this organization has no data". Deriving makes
it unrepresentable.
This lives in tenancy.py rather than in the adapter because it is the same
decision as scope_for, not a second decision next to it. A module that picked
the organization and another that picked the schema would be two places for the
boundary to be got wrong, and the schema one is the silent one: an unqualified
projects query simply finds nothing, and reads as an empty database.
The value is quoted rather than interpolated because search_path parses its value
as a *list*. A name carrying a comma would not name a schema, it would append
one. The check before that is unreachable from a UUID and is there anyway.
Umbratools_2.md 5.0 replaces shared-tables-plus-RLS with one schema per
organization, because RLS was measured to be unenforceable twice: the
application connects as the table owner so the policies did not apply to it
(gate G2), and one app function was PUBLIC-executable until it was fixed by
hand against one database (defect B4). A model whose correctness depends on a
grant somebody remembers to make is not a boundary.
The classification comes before the SQL because the SQL's blast radius depends
on it, and the obvious reading of "move the tenant tables" is wrong for eight of
the tables carrying an organization_id. Four move: projects, project_authors,
project_publishers, invitations. The other twenty-six stay, each for a reason
read out of the actual columns and foreign keys rather than the table names.
Two findings the plan did not price, both recorded in the classification:
- user_roles cannot move. It is the index that resolves a session to an
organization, so putting it in a tenant schema means finding the tenant by
searching every tenant schema, on the login path. It also holds
platform-scope rows, which have no organization to belong to. So one table
stays shared, and 5.0's "no table is shared between organizations" is not
fully achievable while the session-to-tenant resolution is a table.
- moving projects forces dropping three foreign keys that point at it from
artifacts, jobs and user_roles, because a foreign key cannot name a table
whose schema depends on the row. That is the one place the decision costs a
database-enforced guarantee, and it is written down rather than papered over
with a trigger that would have to rediscover which tenant is meant.
The migration names each tenant schema from the organization id rather than
storing it, so a name cannot disagree with the rows and the rollback can rebuild
the organization_id column from the schema. It is idempotent - verified by
applying it twice - and the rollback is a file that runs rather than a comment
block, because a rollback for a data migration that only exists as a comment is a
rollback nobody runs. Both were applied to the scratch database, rolled back, and
applied again.
Feedback on the add-node page: the command was hard to read and hard to copy.
Both the command and the six-digit code now use .otp-value-row / .otp-value - the
wide monospace value with its copy button beside it that the provisioning dialog
already uses. Reusing that rather than writing new CSS, because a second styling of
the same idea is how the two drift apart.
The command was a <textarea> carrying a class named `code-block` that exists nowhere
in the stylesheet - I had invented it - so it rendered as a default textarea rather
than the monospace block it was meant to be. It is now a <code>, which means it is
written with setText rather than .value; setting .value on a <code> silently does
nothing, which is the same class of silent failure as the missing toast element.
The standalone "Copy the command" and "Copy the code" buttons at the bottom are gone:
each value now has its own button next to it, so there is one obvious way to copy each
thing and no need to match a button to a field.
Done is now labelled to say what it does - "clear this from the page" - because the
code is only in the page's memory and the controller cannot produce it again.
Reported as "I entered info, clicked create, and nothing happened". Three separate
faults, each of which fails quietly, so the page looked like it simply did not work.
**1. The toast element had the wrong id.** api.js's showToast() looks for
`id="toast"` and returns silently when it is absent. The page had
`id="toast-region"`. Every message - success and failure alike - was appended to
nothing. This is the direct cause of "nothing happened": the request may well have
succeeded and the answer was thrown away. The working console page uses id="toast";
mine should have matched it.
**2. Seven `el()` calls passed a className where api.js wants a props object.**
`el(tag, props, children)` assigns `props.class`. `el('td', 'table-note')` therefore
set an attribute literally named "table-note" and no class at all, so the node list
would have rendered unstyled. Caught by reading the helper instead of assuming its
signature from the name.
**3. A platform administrator could not use the page at all.** The compute routes
refuse to infer an organization - they raise `organization_required` and ask for
`?organization_id=...`. The first version read an `organization_id` out of
/api/uadmin/summary, which has no such field, so it was always empty and every create
was a 422.
The page now loads organizations from /api/uadmin/organizations and presents a picker,
sending the chosen id as a **query parameter**. It went in the JSON body at first,
where the route declares it as a Query - a body field of the same name is silently
ignored, which looks exactly like it worked. An organization administrator has exactly
one and sees it preselected rather than being asked to choose.
Also added `defer` to both script tags, as the working console page has, so api.js is
defined before nodes.js boots.
Verified the corrected request path end to end: the same payload the browser now sends
produces a code, and that code pairs a node on 238 through the `curl | sh` flow.
525 passed, 27 skipped.
The page an administrator uses to bring a node online: name it, get a one-line command
and a six-digit code, hand both to whoever is at the machine. That is the last piece
between "someone runs a command" and self-service onboarding.
POST /api/compute/pairing-script now returns JSON - script, code, expiry and the
command - rather than the bare script. There are two callers with two needs: a person at
a terminal wants a script to pipe to sh, and the console wants all three values so it can
show them once and then forget them. Returning text would have forced the console to
parse a shell script to recover a six-digit number, which works until someone reformats
the script.
The code is shown once and there is no button to fetch it again, because the controller
keeps only an Argon2id hash. That is the server's limit, not a UI choice, and offering a
"show code" button would be a lie about what it can do.
## Three defects the template flow exposed
Running `curl ... | sh` on node 238 - the flow the console hands out - found that the
template could never pair, for three separate reasons:
- It had no prompt and no code. `000000` is the placeholder that means "nobody filled
this in", so the script now asks. A rendered script from the console has the code
substituted and skips the prompt, so nobody is asked for a number they do not need.
- It had no node name, and the name is what the code is bound to. It now asks too. The
first version failed with "check the code has not expired" while the code was fine and
the *name* was the placeholder - an error pointing at the wrong thing.
- It hardcoded `controller.invalid` as its controller address, so every deployment but
one got a script that could not resolve its own controller. The address now comes from
the Host header the template was fetched with, and `UMBRA_CONTROLLER` overrides it.
The error message itself was part of the problem: a DNS failure reported "check the code
has not expired", which is a worse fault than no message because it sends an operator to
check the one thing that was working.
## Verified
`curl -fsSL http://192.168.123.14:8442/api/node/bootstrap | sh` on 238, code and name
typed at the prompts: paired, agent installed, service registered, node `online`.
`/api/node/bootstrap` is unauthenticated and returns only the inert template - a script
that can check a machine and then stop. It is listed by name in the test that pins
unauthenticated node routes, alongside /pair, with the reason each is safe.
## Also found, not fixed
`VIEWS` in web/assets/admin.js is `['dashboard','accounts','organizations','audit']` but
`view-projects` exists in the markup, so the projects view can never be activated. It
predates this work and I have left it alone rather than change navigation behaviour as a
side effect of adding a page.
525 passed, 27 skipped.
Enrolling a node meant a person reading instructions and running them, with a
credential typed into a terminal. That does not scale, and it makes the controller a
courier. The flow is now: an administrator renders a script, hands over one curl, and
the node sets itself up.
`GET /api/node/agent` serves the agent to a credentialed node. A node credential
already implies "run what the controller sends" - that is what claiming a job is - so
this adds no authority; it removes the hand-copying. `POST /api/compute/pairing-script`
renders the bootstrap, minting a pairing code server-side so a code pasted into a
request body cannot be one issued earlier for something else.
## The exclusive/shared distinction, encoded
The script asks whether it has root and branches. With root: `/opt/umbra`, a system
unit, and the agent running as a dedicated `umbra-node` account with NoNewPrivileges and
ProtectSystem=strict. Without: the user's own directories and a user unit. Both layouts
are in the output and the runtime decides, because whoever renders the script does not
always know how the person at the machine will run it.
The credential goes in a 0600 file referenced by systemd's EnvironmentFile. Three
alternatives were rejected and the reasons are in the code: a unit file is
world-readable, a config holding a secret ends up in a backup, and the agent
deliberately refuses to read the key from its config when the environment can carry it.
## Verified on node 238, not just reasoned about
Ran it as `umbra` with no root: paired, installed, registered, and the agent went
`online`. Six defects found only by running it:
- `$BOOTSTRAP_VERSION` was interpolated into the shell body but never declared as a
shell variable. `set -u` caught it on the first line that used it.
- The agent fetch sent no credential and got a 401 - correct behaviour, wrong script.
- `ExecStart` pointed at the agent file directly. It has no shebang, so the kernel
tried to run it as a binary: "Exec format error", a service in a restart loop, and
nothing on the controller. Now invoked through `python3` explicitly, and the file is
644 rather than 755 because it is source.
- `sed -i` with no backup suffix is GNU-only; BSD reads the next argument as a suffix,
so on macOS it would leave the unit unfiltered and install hardening directives that
cannot work. Replaced with a portable filter.
- Pairing a name already in use surfaced as a bare 500 from a UniqueViolation. Now
`node_name_taken`, with an answer that says pairing does not replace an existing node.
- The agent logged "claim failed: node_scope_required" every second, forever. A
read-only credential can never gain compute:claim, so this is permanent, and a
warning that cannot change trains a reader to ignore warnings. Announced once.
## A privilege bug the pinned tests caught
The bootstrap renderer declared an administrator dependency while sitting on the *node*
router. A node could have called it to mint itself a credential for any node name. It
is now `POST /api/compute/pairing-script` on the admin router, above the agent-surface
marker so `test_every_node_route_uses_node_credential_authentication` keeps meaning
something.
Also worth noting: a node cannot be DELETEd from the database, because the pairing-code
reuse trigger refuses the `ON DELETE SET NULL` cascade. That is correct - the status
enum has `retired` and the admin route drains and retires, so removal is a status
change, not a row deletion. Found by trying to clean up after a failed test run.
525 passed, 27 skipped. Test nodes boot-test-1..3 are retired, not deleted.
/proc/cpuinfo is a declaration. On node 238 it reports 'AMD Ryzen 9 3900X' and then
lists 65 CPUID flags that stop at SSE4.2 - no avx, avx2 or fma, on silicon that has
all three. Executing an AVX2 instruction there faults with SIGILL, so the hypervisor is
presenting a CPU that does not exist.
That is the same defect the compute design already has, one layer down: the agent sets
throughput_factor by string-matching a GPU model (40.0 if '4070' in name else 20.0), a
declared factor wearing a measured name. Here the hypervisor makes the same kind of lie
about instructions. So a feature is recorded as one of three things and never two:
present - an instruction used it and produced the right answer
absent - an instruction used it and the kernel killed the process
unverified - it could not be executed at all
Only 'present' counts as usable. An unverified feature is treated as unavailable on
purpose: the cost of pessimism is a slower path that works, and the cost of optimism is
a job dispatched to a machine that dies on it.
Measured on node 238: sse2, sse4_2, bmi2, aes_ni usable; avx, avx2, fma, avx512f all
SIGILL. On oldbox: everything but avx512f.
Four bugs found while building the probe, each of which would have made it report
something false:
- Inline asm with wrong operand counts, for all eight features. Replaced with
intrinsics, verified by disassembly.
- One binary for every feature could not compile: -mavx2 alone makes the FMA, AVX-512
and AES intrinsics fail always_inline. Every feature reported unverified on a machine
that has most of them. Now one translation unit per feature.
- GCC folded the probe at -O2 with constant inputs and emitted nothing, so it would
have reported 'present' for every CPU on earth. Inputs are volatile and the expected
mnemonic is checked in the disassembly; test_a_folded_probe_is_reported_unverified
builds a deliberately foldable probe and asserts it is caught.
- The SSE4.2 probe stored a comparison mask through _mm_storeu_pd and read a double back,
getting NaN - reported as 'executed but returned a wrong result' on every machine.
Also: a caller passing an explicit compiler path that does not exist got eight build
errors instead of one clear sentence; the compiler is now verified to work, and a
missing or broken one is a False rather than an exception.
525 passed, 27 skipped.
Reached 238 via public-key enrolment - no password anywhere. The first attempt failed
because the key had never landed in authorized_keys; the second worked once the
fingerprint matched.
Findings worth keeping beyond this node:
The VM reports 'AMD Ryzen 9 3900X' and then lists 65 CPUID flags that stop at SSE4.2.
No avx, no avx2, no fma. That silicon has AVX2 and FMA, so VirtualBox is masking them
and /proc/cpuinfo is a declaration, not a measurement. Recorded as *unverified* rather
than absent, because confirming it needs an actual AVX instruction executed and there
is no gcc on the guest - a flag list cannot settle it.
This is the defect the compute design already has, one layer down. The agent sets
throughput_factor by string-matching a GPU model (40.0 if '4070' in name else 20.0),
which is a declared factor wearing a measured one's name. Here the hypervisor makes the
same kind of lie. Capability discovery therefore has to probe by executing
instructions and treat an unverifiable feature as unavailable.
Also: cpu.max is 'max 100000' so the guest is not CPU-capped, memory is 47.0 GiB
unballooned, no GPU and no nested virt flags, and umbra has no sudo. Recorded in docs/
node-238-capability-profile.md with how each value was obtained.
The deployment listens on 192.168.123.14 and TrustedHostMiddleware rejected that same
address, so every request by IP returned a bare 400 Invalid host header - including
/api/health, which is indistinguishable from the service being down. It has cost this
project twice.
Found when the first pairing attempt from 192.168.123.238 came back 400 with no body
and no logged detail. The pairing code was fine and went unspent; the request never
reached the route.
Both umbratrans.bdmurphy.me and 192.168.123.14 are now allowed. The 0600 on the env
file is unchanged, and the correction is recorded in the alignment doc's transport
note so the audio node is not sent looking for a service outage.
Issuing a node credential by hand required writing the key into a 0600 file on the
controller and having a person copy it onto another machine. That key landed in shell
history and a terminal buffer, and it made the controller a courier: nothing could be
enrolled without me in the middle, which is the opposite of what a fleet of
heterogeneous nodes needs.
`POST /api/compute/pairing-codes` (admin session) mints a six-digit code for a *named*
node. `POST /api/node/pair` exchanges it, once, for a credential returned in the
response. The node never sees a secret typed by a person, and the controller is no
longer in the path.
## Why a six-digit code is defensible
1,000,000 possibilities is nothing on its own. What makes it work is everything around
it, and any one of these alone would be too thin:
- **Single use.** Redemption is `SELECT ... FOR UPDATE` then `UPDATE ... WHERE
used_at IS NULL`, in one transaction.
- **Expiring.** Default ten minutes, hard-bounded to 60-1800s. Long enough to walk to a
machine; short enough that a code left in a terminal buffer is not a credential for
the day.
- **Bound to a node name.** A code read out of a terminal buffer cannot pair a machine
called something else. Without this, "confirm it is the machine" is decoration.
- **Argon2id at rest.** A database read or a support transcript yields nothing
replayable.
- **Rate limited per source address**, charged *before* any code lookup, so guessing
across many codes costs the same as guessing at one. Twenty per quarter-hour against
a million-code space is about 21 years. This is the control that makes the length
defensible; the others are cheap additions to it.
Every rejection is worded identically - "no such code", "wrong code", "expired",
"already used" all read the same - so the response is not an oracle for the rate
limiter to work against. The one exception is a name mismatch, which says so plainly:
it is not secret, and an operator staring at a failure needs to be told the code is for
a different machine rather than left guessing. A name mismatch does *not* spend the
code, because the obvious implementation spends it and then the retry with the right
name cannot work.
## Why not a SECURITY DEFINER function
The obvious shape is a plpgsql function that verifies and redeems, so the node surface
never touches the tables. It cannot be done in SQL here: the codebase hashes with
Argon2id in Python, and the only hashing PostgreSQL offers in this database is
`crypt()` (bcrypt and MD5). Writing pairing against bcrypt would claim an algorithm
choice the rest of the system does not make. So verification is in the repository, which
means the pairing tables need grants for the application role when G2 lands, and the
path needs re-testing as a non-owner role. Recorded in the migration header.
## Three defects found by writing the tests
- **Redemption order.** Claiming the code before inserting the node it references
violates the foreign key, so *every* redemption failed. Found by nine failing tests
with an FK error that had nothing to do with pairing logic.
- **A race that produced a 500.** Two nodes verifying one code simultaneously both
created a node, and the loser hit a unique-constraint violation on
`(organization_id, name)` - an unhandled error for an entirely ordinary condition.
Fixed with `SELECT ... FOR UPDATE` on the code row before the insert, so the second
caller blocks, re-evaluates `used_at IS NULL`, and gets the ordinary refusal. The
concurrency test fails when the lock is removed and passes when it is present.
- **A test that could not fail.** Removing the `used_at IS NULL` from the *claim* left
all tests passing, because the candidate query's `used_at IS NULL` filter masks it.
Sequential redemption cannot distinguish the two guards; only a concurrent one can.
That is now `test_a_concurrent_redemption_produces_exactly_one_node`, and
`test_a_spent_code_is_not_offered_as_a_candidate` pins the first guard directly.
The reuse trigger needed a correction too. It originally refused any change to
`expires_at`, which would have made it impossible to revoke an unused code - the
normal way to withdraw a code issued to the wrong person. It now refuses identity
changes and expiry *extension*, and permits expiry to move earlier only.
Two existing tests flagged the new unauthenticated route, correctly:
`test_the_node_surface_is_exactly_what_a_node_may_do` and
`test_every_node_route_uses_node_credential_authentication` both assumed every
`/api/node` route needs a credential. Rather than weaken either, `/pair` is recorded as
a named exception in both, with a check that the named route still exists - so a second
unauthenticated node route has to be added deliberately with a reason.
509 passed, 27 skipped. Deployment: 0004 applied after a backup, health 200, and the
live endpoint verified to refuse a bogus code with 403 and a wrong node name with a
`pairing_name_mismatch` that does not spend the code.
G2 is still open, and the pairing tables will need grants when it closes.
