master
67 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
B.D. Murphy
|
3f05489e25 |
Parameterise the translation pipeline by target language
The pipeline was hard-coded to Simplified Chinese in three places: the drafter was told to translate into Chinese, the critic was told to compare English with Chinese, and the glossary's translated forms were read from a "zh" key. A production that declared target_language: es-ES was therefore translated into Chinese and filed under a Spanish name - measured on oldbox, deal-hunter-es held 3,038 paragraphs of Chinese and was reported complete. A new registry, tools/translation/languages.py, is the single source of truth for what a target language is: its names, its script, its punctuation, its glossary key, its typography and its chapter-numbering convention. Ten languages are registered. The prompt scaffolding stays in Chinese - that is what the models read best - and the target is named with its Chinese, English and native names so there is no room to misread it. The registry now reaches every stage: - prompts.py: the analyst, drafter, critic and glossary blocks take the target. The hard rules about this book - Deal Hunter is the ship's name, the AI's name and the book's title, so it stays in English - are the language's own table for Chinese and the global rules for everything else, so a language with no table does not lose them. The Chinese rules are byte-for-byte what they were. - pipeline.py: --target-language, a run-directory guard that refuses to reuse a directory belonging to another language (from the manifest, or by measuring the text it already holds), a draft-keyed-by-target reader, and a refusal of a draft that came back in the wrong script. - book.py: per-language chapter-title and subtitle tables. A language with no table falls back to the English title, never to another language's; a language with a table and a missing chapter gets a visible marker in its own language. - qa.py: the target's quote marks, punctuation, script count and forbidden title forms. The Latin-residue check is skipped for a Latin target, where it would flag the whole book. - align.py: the aligner and verifier prompts name the target, and the sentence and clause splitting uses the target's own terminators. - render.py: the fonts, the language and country codes, the writing direction, the chapter headings and the fixed editorial strings around the translation are the target's. The Latin-run span is skipped for a Latin target, where marking the text as English would tell the renderer it is English. - project_status.py: the dashboard measures a sample of a production's text and reports a production whose text is not in the language it declares as "mismatched", not "complete". A chapter count is a claim about the files; the script check is a measurement of the text. Also fixes: the dashboard gated every language's production on the source production existing, so a Spanish production could never be reported even when it was there. The Chinese render is byte-identical to before on the real book, and the Chinese rules are unchanged. Suite: 620 passed, the same 68 pre-existing environment failures, +49 new tests. |
||
|
umbra audio
|
fbbe23d7f9 |
Add output generation, cover image management, and back-translation tool
- Add cover image CRUD endpoints: GET/POST/DELETE /api/projects/{id}/cover
- Add output generation endpoints: POST /api/projects/{id}/output/{format}
- Add cover image and output queue service methods
- Add CoverImageResponse and CoverImageUploadResponse schemas
- Add umbra-api.js request handling for FormData (cover uploads)
- Add output generation buttons (EPUB/PDF/Audio) to Format UI in umbra-app.js
- Add backtranslate.py tool for verifying translation quality via back-translation
- Add cover image support to render.py EPUB generation
- Copy cover image to deal-hunter-zh source directory
- Spanish translation pipeline started (running in background)
|
||
|
umbra audio
|
be04e19d04 |
Add UmbraAudio controller integration endpoints
- Add migration 0010: utterance_translations table with UNIQUE(utterance_id, language_code)
- Add schemas: OrganizationCreateRequest/Response, ProgramCreateRequest/Response,
UtteranceTranslationItem/Request/Response/ListResponse, SpeakerSyncRequest/Response
- Add service methods: create_organization, create_program, sync_speakers,
upsert_utterance_translations, get_utterance_translations
- Add routes_controller.py: POST /api/orgs, POST /api/programs,
POST /api/speakers/sync, POST /api/projects/{id}/translations,
GET /api/projects/{id}/translations/{language_code}
- Mount controller_router at /api in app.py
- Create bdumbra account (member, BDMurphy org) for integration testing
|
||
|
umbra audio
|
3e0568b9d1 |
Generate backend endpoints for language-project workflow
Add REST endpoints and frontend wiring for per-language project actions:
- POST /api/workspace/{project}/translation/{lang}/queue — queue translation
- POST /api/workspace/{project}/audio/{lang}/queue — queue audio generation
- GET/POST /api/workspace/{project}/format/{lang} — format metadata (EPUB/PDF)
- GET /api/workspace/{project}/review/{lang}/chapters/{chapter} — utterance review
- GET/POST /api/workspace/{project}/front-matter/{lang} — front matter editor
- GET /api/workspace/{project}/chapters/{lang} — chapter list
- GET /api/workspace/{project}/chapter-content/{lang}/{chapter} — chapter text
Frontend fixes:
- Review page shows per-language text (English source, Chinese translation, etc.)
- Non-translated languages show 'Queue translation' button only
- Speaker data from English source matched by paragraph ID
- UmbraAudio column layout: Speaker, Variation, Text, Actions
- Chapter tree in left sidebar: Front matter, Back matter, Body
- Audio Format label at top of sidebar
- Pandoc formatting toolbar for front matter text editing
|
||
|
umbra audio
|
bd8238f3be |
fix BundleError handler: map malformed bundles to 422 with structured envelope
- Added exception handler for BundleError in app.py that returns HTTP 422 - Response includes code='invalid_bundle' and message from error argument - Previously returned unstructured 500; now consistent with other validation errors |
||
|
umbra audio
|
dda079f605 |
commit audio ingest: apply 0008+0009 migrations, wire Postgres adapter to commit path
- Apply migration 0008_audio_source_import (additive schema for source import) - Apply migration 0009_translation_import_runs (ledger table for import runs) - Wire PostgresAudioImportStore into HTTP commit path with org from node credential - Commit path uses PG store with org from node, no payload org - Bundle hash 21ce5687… remains frozen until v2; no authority transferred |
||
|
umbra audio
|
4160ec49a0 | audio: Phase 2 — Postgres AudioImportStore adapter + 0009 ledger; use PG store on commit with org from node; 0008 header updated; tests | ||
|
umbra audio
|
e35d7a144f | Add Apache-2.0 LICENSE, NOTICE, and SPDX headers | ||
|
umbra audio
|
d7f84075db |
Prompt identity: knowing which prompt produced a result
docs/prompt-identity.md, and tests that assert the mechanism rather than describe it.
The question this came from: are UmbraAudio's LLM prompts code, or data that gets pulled
and sent? The answer is neither, exactly - they are markdown files read at runtime:
PROMPT_PATH = Path(__file__).parent / "CHARACTER_SEGMENTATION_PROMPT.md"
prompt = template.replace("{chapter_text}", text)
client.chat.completions.create(messages=[{"role": "user", "content": prompt}])
That is a good design for *using* a prompt, and it is worth keeping. Editing the file and
re-running needs no code change and nothing to redeploy. Three things are missing, and all
three bite the moment a prompt is iterated on - which is the normal way to improve one, and
certainly how the intent scan will be developed.
1. No identity on the output. `utterances` records text, speaker, emotion and pace, and
nothing about which prompt produced them. After an edit and a re-run, "which paragraphs
were scanned under which version" is not difficult to answer but unanswerable.
2. No caching, so no invalidation to reason about. `llm_segmenter` has no cache references
at all, so re-running `analyze` re-sends every chapter. Editing a prompt costs a full
book scan, which is the cost that makes people avoid iterating on prompts.
3. Re-analysis appends. `insert_utterance` is an unconditional INSERT with no upsert key, so
a second `analyze` adds duplicate utterances rather than superseding the old ones.
The precedent for fixing this already exists one layer too late. `generation/staleness.py`
solves exactly this for audio - a sidecar per wav recording spoken text, speaker, voice_fp
and sha256 - and its docstring explains why: a "file exists" check silently reuses audio
that no longer matches the utterance text. The reasoning is right and the implementation is
careful, and it applies to audio only. The expensive artefact is protected and the cheap
one is not. This design extends the existing mechanism one layer earlier rather than
inventing one.
Three fields travel with everything an LLM produced: producer, prompt_id, prompt_sha.
prompt_id is the label a human reasons with; prompt_sha is the mechanical guarantee, taken
over the *rendered* prompt rather than the template - because hashing the template would
report two runs over different chapters as the same call, which is the one distinction the
mechanism exists to make.
The load-bearing test is that a version and its content must agree. A prompt file edited
without a version bump makes "v1" name two different prompts, and every query written
against it silently returns half the answer while looking like provenance. That
combination is refused rather than recorded, because a label that can quietly become untrue
is worse than no label.
test_prompt_identity.py states the mechanism as behaviour: that any edit changes the hash,
that different chapters render differently, that an unbumped edit is refused, that a bump
makes it legitimate, and that v1 still resolves after v2 exists. One of those tests failed
on the way because the registry helper rewrote v1's hash - backwards for that case - which
is recorded rather than quietly fixed.
Mutation-verified: disabling the version/content guard fails 1 test, and hashing the
template placeholder instead of the rendered prompt fails 3.
Does not cache responses - that is a separate decision needing a separate answer about when
a cached response is wrong, and it comes after this, not before. Does not touch UmbraAudio's
own prompts, which need the same treatment for the same reasons; that is a change to their
repository.
716 passed, 4 skipped.
|
||
|
umbra audio
|
9432366183 |
The catalogue: declare files that already exist, without moving the bytes
The audio node is the master for the audiobooks. UmbraTrans holds the metadata needed to reason about a file and a reference to where it is; it does not hold the bytes. DECISION-6 (object storage) stays unresolved, and this is what lets the system be useful before it is resolved rather than after. POST /api/compute/artifacts takes a manifest against a project and declares each entry: content hash, size, media type, kind, and a storage_uri naming where the file actually is. GET /api/compute/artifacts lists the catalogue. There were no artifact routes at all before this - the tables existed since migration 0003 and nothing had ever written to them, so "the controller has the artifacts to show a job is complete" had nothing behind it. Every declared row lands integrity='unverified', and that is the point rather than a caveat. Ingesting is a claim: a node reporting the hash of a file is a statement about bytes nothing here has read, because there is no object storage to read them from. A design where submitting a manifest marks the catalogue verified would make the first ingest the most reassuring and least true event in the system's life. verified_at is set by reading the bytes back and hashing them, and nothing in this path does that. storage_node_id is part of the reference rather than an afterthought, because a bare path means nothing six months later: /audio/book.mp3 is on whichever machine happened to serve it that day. Idempotent on (project_id, content_hash), so the same manifest can be submitted twice, or by two operators who did not talk to each other, without the catalogue doubling - and a partly-present manifest reports what was new rather than just "done". An operator who submitted 300 entries and was told "done" would not know two were malformed and dropped, so malformed entries are named in the response instead. The listing is scoped by organization on every call. artifacts is platform-level, as every table a node serves from is, so a missing predicate there is a cross-tenant read rather than a style issue. A project in another tenant's schema is refused as *not found*, not as *not permitted*, because a refusal that says "this exists but is not yours" answers the question the caller was actually asking. Migration 0007 applied to the deployment. Mutation-verified: marking an ingest verified fails 1, dropping the idempotency check fails 2, and dropping the organization predicate from the listing fails 7. 704 passed, 4 skipped. |
||
|
umbra audio
|
e757630da2 |
A phone had no navigation at all
The 720px breakpoint contained three rules:
.admin-sidebar { display: none; }
.rail-link:not(.rail-link--active) { display: none; }
.topbar-admin-link { display: none; }
Every one hides navigation rather than relocating it. On a phone the console showed the
page you were on and no way to reach any other: the admin sidebar was gone, every rail
link except the current one was gone, and the route from the workspace to the control
panel was gone. There was no hamburger to find because nothing had ever been built for a
narrow screen - the breakpoint assumed the sidebar could simply go away, which is only true
when nothing else is left to navigate with.
A toggle appears below the breakpoint and reveals the navigation as a drawer. The drawer is
positioned rather than shown inline, because an inline nav pushes the page down by its own
height every time it opens, so on a short viewport the page the operator asked for ends up
below the fold. Escape closes it, a click on the page behind it closes it, following a link
closes it, and crossing the breakpoint settles the state - without which the drawer class
survives into the wide layout and the bug only reappears when the phone is rotated back.
Above the breakpoint the toggle is hidden, because a control that opens what is already
open misreports the state of the page.
nodes.html is deliberately not given a drawer. Its navigation is inline in <main> rather
than in a sidebar, so there is nothing to put in a drawer and a hamburger there would
toggle nothing; its nav was a single horizontal scroll strip instead, which hides links off
the right edge with no sign that there are more, so it now wraps.
test_navigation.py states the rule - navigation hidden at a width must still be reachable
at that width - and reads the breakpoint with comments stripped. Two things that check got
wrong on the way, both recorded in the file:
- It flagged the drawer's own "display: none" as a violation. Hiding a closed drawer is
correct, or it would push the page down on every load; what is wrong is hiding navigation
with no way to open it. The test now requires a "body.nav-open" rule alongside any hide.
- It only required that *some* page carry a toggle, so deleting it from the admin console
- the page that was reported broken - passed, because the workspace still had one. Now
per page.
Mutation-verified: restoring the original defect fails 1, removing the toggle from the
admin console fails 1.
695 passed, 4 skipped.
|
||
|
umbra audio
|
d6cebe4d55 |
The controller assigns work, and a job is done only when the artifacts say so
Replaces claiming with assignment. The controller hands a job to one or more nodes and a node asks what it has been given; jobs.lease_owner could name only one holder, so a job fanned out to several machines was inexpressible, and 'what was assigned and what is not complete' had no answer. job_assignments is one row per (job, node, attempt), so both are now readable. Nodes poll rather than being pushed, which is not a simplification: nodes sit behind NAT and the controller has no route to them. The only path that exists is a node calling out. The rule with teeth: a job is complete when the controller has the artifacts to show it. Reporting is not believing. A node that finishes and hands over nothing gets its assignment failed, with the reason naming what was missing, and the work stays available for another machine. The tempting implementation - mark complete, then verify afterwards - cannot fail, because the only thing it could compare against is the report it just trusted. artifacts_verified_at is written by verification alone, so 'the node says it is done' and 'the controller confirmed it' are different facts that cannot look alike. fan_out defaults to 1 and is never inferred. More than one is redundant execution: several machines do the same work and the first verified result wins, the rest are cancelled. Worth having for cheap idempotent work, wasteful for anything costing GPU-seconds, so a caller has to say so. The losers are cancelled rather than left running - otherwise every machine on the list burns its time on a result nobody will read. Four defects found while building it, all by tests written first: - Naming a node explicitly bypassed the drain check. An administrator could drain a machine, see it read as drained, and still hand it work: the control doing nothing while appearing to work. Status is now checked on both routes. - attempts were counted from jobs.attempt, which only claim_job incremented, so under assignments max_attempts was never reached and a failing job was re-offered forever. Counted from failed assignments now, so a fan-out to three costs one attempt, not three. - start_assignment reported 'not startable' when the real problem was that the assignment belonged to another node. Ownership is checked first, because an operator handed the first will look in the wrong place. - The artifact API accepted values the database rejects: role 'output' where job_artifacts_role_check allows only 'consumed' and 'produced', an unbounded status, a content_hash that was not 64 hex characters, and kind at 80 where the limit is 64. Every completion would have been a 500. The schema had fixed this vocabulary all along and the API had to learn it rather than invent it. Artifacts are recorded even when verification fails: a short run still produced evidence, and discarding it would destroy the record of why. storage_key is never treated as proof the bytes exist, because there is no object storage to ask yet (DECISION-6). Claim still exists. It is listed in the pinned node surface as pending removal and the next change takes it out - two paths handing out work is the state this project keeps finding and criticising. Mutation-verified: trusting the node fails 3 tests, dropping the drain check fails 1, leaving fan-out losers running fails 1. 674 passed, 4 skipped. |
||
|
umbra audio
|
02fa70d8ad |
Revoke a credential, and take a machine out of the fleet for good
node_credentials.revoked_at has existed since the first migration and
resolve_node_credential has always honoured it. Nothing could set it. The ability to cut a
node off was present in the data model and absent from every API, which is the shape of
defect worth naming: the column was the capability.
POST /nodes/{id}/revoke cuts a node off and leaves it - right for a compromised key or a
machine about to be reimaged. POST /nodes/{id}/remove takes the machine out: credentials
revoked, devices, heartbeats and model reports deleted, status 'removed'. 'removed' is
deliberately not settable through PATCH /nodes/{id}/status, because typing 'remove' as a
status change would skip the revocation and the release of held work, which are the two
things that make removal mean anything.
The row is kept and cannot be deleted. job_events.node_id and
node_pairing_codes.used_by_node_id both reference compute_nodes ON DELETE NO ACTION, so
the DELETE is refused; changing them to SET NULL does not help, because that is an UPDATE
of job_events and job_events_append_only raises on every update by design. So a node that
has ever done work cannot be removed without weakening the trail's immutability, and the
trail is worth more than a tidy DELETE. A trigger makes 'removed' terminal, and
set_node_status refuses it first so the caller gets a reason rather than a constraint
violation.
Work the node was holding is released at removal rather than left to its lease. An
operator removing dead hardware already knows the work did not finish, and making them
wait out a lease to find that out is the wrong way round - for a long lease, most of an
hour of queue stalled behind a machine nobody is going to switch on.
Also creates job_assignments, which is how the controller will hand work to a node:
one row per (job, node, attempt), so a job can be open on several machines at once and
the controller can say which of them finished it. state is deliberately not the job's
status, and artifacts_verified_at is written by verification alone - that column is the
whole of 'a job is complete when the controller has the artifacts to show it'.
Migration applied to the deployment inside a transaction that first normalised any
'online' node to 'provisioning', since the new terminal-state trigger refuses that
transition and those rows would otherwise have blocked it.
Mutation-verified: not releasing held work fails 6 tests, leaving the credential working
fails 1, and a revocation that sets revoked_at back to NULL fails 2.
652 passed, 4 skipped.
|
||
|
umbra audio
|
9eb419ffc8 |
Nodes get a unique name from their MAC, so two machines can share one
Several nodes legitimately want to be called VM238 - one per hypervisor, one per rack, one per testbed. The unique constraint on compute_nodes is (organization_id, name), so until now the second machine to pair was refused with node_name_taken for a name that looks free to the person typing it, and the fix of 'pick another name' pushed the bookkeeping onto the operator for something the controller can derive. A node's name is now its base name plus the last four hex digits of its MAC: VM238-AC4B. Four digits because the name appears in a UI column, a command line and a systemd unit. Sixteen bits narrows collisions rather than eliminating them, so a letter fallback sits behind it - VM238-AC4B-A - and the sequence continues past Z, because a sequence that silently runs out hands out a name that is already taken. The code stays bound to the *base* name. The controller cannot know a MAC before the machine speaks, so the final name is settled at redeem, where 'is this taken' can be asked of the database rather than guessed at by the script on the machine. The script reads the first non-loopback, non-virtual interface, sends the MAC, and then adopts the name the controller returns rather than the one it asked for - reporting one while registering the other is how a node ends up appearing under a name it does not have. Three things this deliberately does not do: - A MAC that is not six octets of hex is rejected rather than cleaned up. The permissive version turns 'not-a-mac' into AMAC, and VM238-AMAC is worse than a node admitting it does not know its own address. - Without a MAC the plain base name is used and a second node of that name is still refused. There is no evidence the machines differ, and inventing a letter would turn one ambiguous input into unbounded identities. - Re-running bootstrap on an already-onboarded machine now succeeds as VM238-AC4B-A rather than being refused. That follows the specified rule - on conflict, add a letter - and it is called out in the test that asserts it: a stale node cannot be assumed gone because its hardware came back. Mutation-verified: the octet bug fails 10 tests, removing the letter fallback fails 2, and the bootstrap script ceasing to send the MAC fails 1. 641 passed, 4 skipped. |
||
|
umbra audio
|
851d14ee30 |
The node write path was dead, and drain did not drain
Two defects, either of which alone would have stopped a node doing work, and both of which looked finished. Pairing issued a read-only credential. node_agent.py already calls claim, renew, complete, fail and decompose, and no credential pairing could ever issue was permitted any of them - so the agent looked ready, every endpoint existed, and not one of those requests could succeed. The same shape as the login form with no method attribute: everything present, nothing working. A paired node now receives the whole node scope set. The pairing code is the authorization: minted by an authenticated administrator, single-use, expiring, bound to one node name, rate-limited per source address. An administrator who mints a code has already decided to put that machine on the fleet. This does not weaken the boundary that matters - NODE_SCOPES contains nothing that reads project content, and a node can hold several credentials, so a monitoring-only machine is still issued a narrow one and this one revoked. Separately, claim_job never looked at the node. set_node_status documents that draining 'stops new claims and lets running work finish', and the add-node page offers drain, return and retire; none of them did anything to the claim path. A machine drained before a reboot kept being handed work by an agent that was still running. The check is inside the claim transaction and takes FOR UPDATE on the node row, so an administrator draining a node cannot land between the check and the claim. It reads effective_status, so a node whose agent has died stops being offered work rather than taking a lease it will never renew - and deliberately not is_schedulable, which also requires a free device, since claiming a job is what makes a device busy. That gate found four tests in test_node_tenancy.py claiming work from nodes that had never authenticated - the state I24 exists to prevent. They now bring their nodes online first. Confirmed the tenancy property is untouched: restoring the tenant filter still fails 5 tests, the same count as before. Every fix mutation-verified - removing the status gate fails 5, restoring read-only scopes fails 2 including a new end-to-end test that pairs a node over HTTP and has it claim and complete a real job. 605 passed, 4 skipped. |
||
|
umbra audio
|
8d1c6a6406 |
27 of 63 database tests were skipping, and the summary said nothing
The PostgreSQL adapter suite refuses to run against a database that already holds rows, because it truncates indiscriminately and cannot do so selectively - audit_events is append-only, with a trigger that raises on delete. The refusal is correct. Two separate faults meant it was firing when it should not have. The guard sampled the live tables at the moment it was first requested. pytest runs files alphabetically and several files before it create organizations and tenant schemas, so by the time it looked, this session's own rows read as pre-existing data. It skipped 27 of 63 tests on a scratch database that had been empty moments earlier, and '567 passed' carried no hint of it. The sample is now taken in pytest_sessionstart, before collection. A session-scoped autouse fixture was tried first and is not early enough - it is instantiated lazily and does not precede the other session fixtures that write. Nothing cleaned up after the session. Ten files open their own connection to the DSN outside the postgres fixture, each inserted organization provisions a tenant schema, and no test dropped it: 78 schemas per run, which after about fourteen runs exhausted max_locks_per_transaction and errored the integration tests with a message reading like a PostgreSQL defect. And the fix for that nearly caused the disaster the guard exists to prevent. The first version of the cleanup truncated unconditionally on teardown, so pointing the suite at a populated database destroyed it while every test that could have touched it skipped itself. Found by planting an organization, watching the suite refuse, and finding the organization gone afterwards. It now leaves an occupied database exactly as found, and says so - through pytest's terminal reporter, because a plain write to stderr is swallowed by file-descriptor capture and was seen by nobody. test_scratch_database_safety.py covers all three, and each was confirmed to fail when the protection is removed. Three mistakes in writing it are recorded in the file: a leading test file that cleaned up after itself and so reproduced nothing; assertions that passed because '-rs' was missing and pytest had printed no skip reasons; and a check that stopped at the warnings header, which pytest prints its summary line after. 598 passed, 4 skipped. The 4 are environmental: two pages with no password field, two needing passwordless sudo for device permissions. |
||
|
umbra audio
|
3419485aa7 |
Fix a syntax error I introduced, which put a password in a URL
The commit immediately before this one - the password-toggle fix - deleted a closing
brace and did not replace it. app.js did not parse. The browser reported it in a console
nobody was watching, and every script-driven behaviour on the sign-in page stopped.
The consequence was not a cosmetic one. The login form's submit handler, which is the
only thing calling preventDefault(), never ran, so the browser performed a native form
submission. Neither form declared a method, and a form with no method attribute submits
as GET. Both password fields went into the query string:
GET /?email=umbra%40bdmurphy.me&password=...
The server logged it in plain text, three times, before it was noticed. It would also
have sat in browser history and in any proxy access log between here and the public
internet.
Two fixes, the second of which is the one that matters:
- The toggle fix is reapplied correctly to the last known-good version of app.js, with
nothing else in the file touched - the diff is four lines against thirteen and two
against one.
- **Every form on every page now declares method=post.** The script still intercepts
and submits via fetch; this decides what happens when the script does not run, which is
precisely the case that failed. Five more forms carried the same latent defect,
including password reset and email verification.
test_form_safety.py checks both, and was confirmed to fail when app.js is broken the
same way. It uses esprima rather than a hand-written brace counter, which gave two false
readings during this work - one of them 'balanced' for a file that did not parse at all.
A check that can be wrong about a syntax error is worse than no check.
The account password should be treated as exposed and changed. So should any password
entered into that form while the script was broken.
|
||
|
umbra audio
|
0e29d863fb |
The as_root checkbox now changes the command it produces
as_root selects which paths the rendered script writes to: /opt/umbra and /etc/umbra/node.json, or the invoking user's own directories. The command did not follow it. Ticking the box produced a script that wanted to write to /etc while being run by an unprivileged shell, which fails at the first mkdir - after the pairing code has already been spent, so it also leaves the operator with a dead code and nothing to show for it. The command now carries sudo when the box is ticked. The script still checks for root at run time and takes whichever branch is actually available, so this is belt and braces rather than the mechanism; but a control that produces a command which cannot work is worse than no control at all. The checkbox's help text described it as a hint whose effect was decided at run time, which was true and misleading: the effect is decided here, at render time. |
||
|
umbra audio
|
4ce121470e |
A node may do work for any tenant, and the pipeline plan
`claim_job` took an `organization_id` and matched it against the job's. Read plainly that says: a machine may only do work for the tenant that owns it. Both paired nodes belong to BDMurphy and The Last Dwarf belongs to WorstAuthor, so a node could never do a word of the audio node's work however idle it was. The only way to have found that out was to queue the job and watch it sit there. `place()` had the same fault - it listed candidate nodes by organization - so the scheduler could not use the fleet across tenants either. Both are fixed. A node is a machine. It belongs to whoever administers it, not to a tenant, and a tenant's data reaches it inside one job's payload for the duration of that job, never by widening what the node itself can see. `kind` and the placement requirements decide what a node may do; ownership decides who may switch it off. What did not change: a node still returns one job's payload and nothing else, so it still cannot enumerate another tenant's work. test_node_tenancy.py asserts the behaviour, and was confirmed to detect the old one: five of its seven tests fail when the tenant filter is restored. Also adds docs/plan-translation-pipeline.md, which separates the critical path - the work-function protocol, TTS, assembly, lineage, and one finished episode of The Last Dwarf - from the work that can proceed beside it. Writing those tests cost more time than it should have and the reason is worth recording: four of them failed for reasons that had nothing to do with tenancy. A job left behind by one test is claimed by the next, which reads as the rule breaking. `TRUNCATE ... CASCADE` on a table shared with other fixtures destroyed `project_publishers` and surfaced two files away as "relation does not exist". The queue is now left alone and the claims filter on a job kind only this file creates, so a neighbour's leftovers cannot be returned and the assertions can be exact. 547 passed, 29 skipped. |
||
|
umbra audio
|
715b7bc69f |
Restore the guarantee migration 0005 had to drop
`projects` moved into a per-tenant schema and `jobs` stayed in `public`. A foreign key cannot cross that boundary without postgres_fdw, so 0005 dropped `jobs_project_id_fkey`, `artifacts_project_id_fkey` and a composite `user_roles` key. The migration recorded the consequence honestly - "the application still refuses it; the server no longer does" - but the application did not. So after the split nothing stopped a `public.jobs` row naming a project in a different tenant, which is precisely the leak the schema split exists to prevent. `submit_job` now verifies it. Two things that were wrong in the first attempt, both caught by running the tests: - The lookup was unqualified, relying on `search_path`. `ComputeRepository._tx` does not set it - only the account repository does - so the check ran against `public` and could never see any project, refusing every project-scoped job. The schema is now derived from the organization id and named explicitly: the check is self-contained, and does not depend on ambient state deciding whether a security check runs at all. - It filtered on `projects.organization_id`. That column no longer exists - the schema the row sits in *is* the tenant, which is the whole point of the split. There is no predicate to write. `test_a_nonexistent_project_is_refused` asserts that a caller cannot distinguish "no such project" from "another tenant's project". From inside a tenant schema the two are the same observation, and they should be. Confirmed the tests detect the gap: with the check removed, three of five fail, including `test_the_refused_job_is_not_created` - which exists because a refusal that still writes a row is not a refusal. 547 passed, 29 skipped. Deployment untouched: 30 tables, 0 tenant schemas. |
||
|
umbra audio
|
24d7ea6a41 |
Format the lines this change wrote, and leave the rest of the drift alone
ruff format disagrees with about two dozen files in this repository, most of them untouched by this work. Reformatting them would bury the tenant-schema change in noise, and reformatting only some of a file is its own kind of noise - so only the lines written here are brought into line, and the pre-existing drift stays where it is for whoever owns it. |
||
|
umbra audio
|
0179355fe7 |
Drop the tenant schemas one transaction at a time
The cleanup I added dropped every tenant schema in a single transaction, and a hundred runs' worth of them exhausts the lock table: the session teardown failed with 'out of shared memory ... max_locks_per_transaction', which reads like a PostgreSQL defect and is not one. It is the same constraint the 0005 rollback documents and I wrote there before hitting it here. Committing after each drop releases the locks, and the teardown now leaves zero schemas behind, so a run does not make the next one slower. |
||
|
umbra audio
|
9a84a9d227 |
The staleness fixture's project goes into the tenant schema, and comes out of it
Same shape as the append-only guards: the project row is created in the organization's own schema and the cleanup drops that schema. The old cleanup deleted from public.projects, which since 0005 has cleaned nothing up while looking like it did - the sort of teardown that reads as careful and is not. |
||
|
umbra audio
|
7b12964a9b |
Seed the guard fixture's project into the tenant schema it now lives in
public.projects does not exist since 0005, and this fixture was the last place still writing to it. The project row moves into the organization's own schema under an unqualified name, which is also the shape production writes it in - so the artifact and job rows guarded by these tests are being set up the way the application would set them up rather than the way the old shared table allowed. |
||
|
umbra audio
|
152735f145 |
Assert the tenant boundary where it now lives, not where it used to
Three changes, all of them consequences of the migration rather than new questions. test_rls_coverage.py drops project_authors and project_publishers from the policy list - they are per-tenant now and there is nothing for a policy to filter - and replaces that entry with checks that are stronger than what it replaces. A policy on a shared table proved the table was filtered; a per-tenant table cannot be filtered at all, so the properties worth asserting are that no shared copy exists in public, that no policy lingers on a tenant table, that every tenant schema carries all four tables, and that organization_id is gone from all of them. That last one matters because it was the column every removed policy filtered on: leaving it behind would leave an application filtering by a value nobody maintains. It also gains a check that POLICY_TABLES names tables that are actually in public. Without it, moving a table out of the policy list and out of public in the same change makes both existing checks pass by not looking at it - which is how a table can quietly stop being policed and still look policed. test_tenancy.py covers the search_path function directly, and the case that matters is the negative one: a request with no organization gets public alone. A wildcard, or leaving the connection's default in place, would put every tenant's tables in reach of a request that has not decided which tenant it is acting for. test_postgres_repository.py proves the new startup contract check holds against the real app function, and that it can fail. A check that cannot fail is the same shape as a policy nobody wrote: it reports success while enforcing nothing. The emptiness guard now also looks inside the tenant schemas and the run drops them at the end. Not tidiness: a database holding nothing in public can still hold somebody's books, and a guard that only looked at public would pass on a database it was about to empty. |
||
|
umbra audio
|
4781468181 |
Test the property that replaced the one a policy used to prove
The old file proved a policy refusing a row. With no shared table there is no policy and nothing to refuse, so the tested property is gone rather than rewritten - and Umbratools_2.md 5.0 says outright that this trade should be noticed rather than discovered later. This commit is the noticing. What is proved instead is stronger, not differently shaped. A statement with no organization in hand asks for `projects` unqualified and is told the relation does not exist: not zero rows, an error. That is a different claim from "the policy filtered it", because a policy that was silently not enforced returned zero rows and looked fine - which is the failure G2 was, and the reason the old test had to run as a non-owner role to mean anything. The new tests keep the non-owner discipline for the part that needs it. Naming another tenant's schema is refused outright; repointing the search path at it and using an ordinary name finds no table at all, because a role without USAGE on a schema does not see its contents when names are resolved. That second one is the outcome an application actually experiences, so both are asserted rather than the convenient one. Positive cases are here because a suite that only proves refusal passes just as happily against a database where every write is refused: a write that must be allowed is performed and then counted in both tenants. What the file cannot prove is written down in its docstring rather than left to be assumed. The application owns every tenant schema, so it can reach another tenant's tables by naming that schema, and schema-per-tenant cannot change that with one role serving every tenant. What it changes is that there is no shared relation to reach through, and reaching another tenant means writing down another organization's name - one call site in tenancy.py rather than a predicate in every query. |
||
|
umbra audio
|
79b3484083 |
Reach the tenant tables by name, not by a filter somebody remembers
Every project statement now says `projects` unqualified and takes its
organization through search_path. A shared table filtered by
`organization_id = %s` is a filter somebody has to remember, and there is always
a write path that would not have it; an unqualified name resolves into the
caller's own schema or it does not resolve at all. The strongest possible
mistake - forgetting to scope a query - now raises instead of returning a wider
result set.
Project.organization_id survives in the model and is filled in from the schema
the answer came from. Callers still see which tenant a row belongs to; it is
derived from where the row is rather than from what the row says about itself.
Two consequences that are worth more than the mechanical change:
- assign_user_to_project now takes the organization as an argument. It used to
read it off the project row and copy it into user_roles, and there is no
column to read now. user_roles is platform-level, so it cannot be written to
by looking inside a tenant schema - so the caller, which has already resolved
the project, has to say it. The protocol declares it, and the in-memory store
accepts and ignores it rather than being the one implementation with a
different signature.
- A platform administrator listing projects without naming an organization now
gets the answer assembled across tenant schemas, because there is no shared
table to ask. That is one round trip per organization and it is paid only by
that one caller; every per-tenant path, which is what an ordinary request
does, is still a single statement. The merge reproduces the ordering and the
total a shared table would have given, so paging across the boundary is the
same as paging inside one.
|
||
|
umbra audio
|
2264a6cc42 |
Publish the caller's schema onto every transaction
Two things now go onto a pooled connection before a statement runs: the caller's identity, which the surviving platform policies read, and the caller's schema, which decides which tenant tables the statement can even name. search_path is set on every transaction rather than only the ones that have an organization, so a platform context gets public alone instead of whatever the connection's own default happens to be. That is the difference between a design and a hope: a default naming a tenant schema is a silent leak with no line of code mentioning it. verify_tenant_schema_contract exists for the same reason verify_rls_contract does. The schema name is derived in two languages - this one and app's - and nothing joins them. A disagreement is silent and total: every tenant query resolves into a schema that is not there, every project request for every organization raises "relation does not exist", and the error names the statement rather than the cause. That presents as an empty database, which is the one diagnosis nobody reaches for. |
||
|
umbra audio
|
e1bbfdb604 |
The session-to-schema step, next to the session-to-organization step
tenant_search_path is the whole isolation model in one function: given the organization a request resolved to, return the schema its tables are in; given none, return public alone. That second case is the point. Returning a wildcard - or leaving the connection's own default in place, which may name a schema nobody chose - would put every tenant's tables in reach of a request that has not decided which tenant it is acting for. The name is derived from the organization id rather than stored, for the reason the migration derives it too: a recorded name that disagreed with the schema the rows are in would describe data that no computed name can reach, and that state would be indistinguishable from "this organization has no data". Deriving makes it unrepresentable. This lives in tenancy.py rather than in the adapter because it is the same decision as scope_for, not a second decision next to it. A module that picked the organization and another that picked the schema would be two places for the boundary to be got wrong, and the schema one is the silent one: an unqualified projects query simply finds nothing, and reads as an empty database. The value is quoted rather than interpolated because search_path parses its value as a *list*. A name carrying a comma would not name a schema, it would append one. The check before that is unreachable from a UUID and is there anyway. |
||
|
umbra audio
|
2cdebf5ac3 |
Schema per organization: the migration, and why these four tables
Umbratools_2.md 5.0 replaces shared-tables-plus-RLS with one schema per
organization, because RLS was measured to be unenforceable twice: the
application connects as the table owner so the policies did not apply to it
(gate G2), and one app function was PUBLIC-executable until it was fixed by
hand against one database (defect B4). A model whose correctness depends on a
grant somebody remembers to make is not a boundary.
The classification comes before the SQL because the SQL's blast radius depends
on it, and the obvious reading of "move the tenant tables" is wrong for eight of
the tables carrying an organization_id. Four move: projects, project_authors,
project_publishers, invitations. The other twenty-six stay, each for a reason
read out of the actual columns and foreign keys rather than the table names.
Two findings the plan did not price, both recorded in the classification:
- user_roles cannot move. It is the index that resolves a session to an
organization, so putting it in a tenant schema means finding the tenant by
searching every tenant schema, on the login path. It also holds
platform-scope rows, which have no organization to belong to. So one table
stays shared, and 5.0's "no table is shared between organizations" is not
fully achievable while the session-to-tenant resolution is a table.
- moving projects forces dropping three foreign keys that point at it from
artifacts, jobs and user_roles, because a foreign key cannot name a table
whose schema depends on the row. That is the one place the decision costs a
database-enforced guarantee, and it is written down rather than papered over
with a trigger that would have to rediscover which tenant is meant.
The migration names each tenant schema from the organization id rather than
storing it, so a name cannot disagree with the rows and the rollback can rebuild
the organization_id column from the schema. It is idempotent - verified by
applying it twice - and the rollback is a file that runs rather than a comment
block, because a rollback for a data migration that only exists as a comment is a
rollback nobody runs. Both were applied to the scratch database, rolled back, and
applied again.
|
||
|
umbra audio
|
c3a3e506ec |
Add a node from the console, and make curl | sh actually work
The page an administrator uses to bring a node online: name it, get a one-line command and a six-digit code, hand both to whoever is at the machine. That is the last piece between "someone runs a command" and self-service onboarding. POST /api/compute/pairing-script now returns JSON - script, code, expiry and the command - rather than the bare script. There are two callers with two needs: a person at a terminal wants a script to pipe to sh, and the console wants all three values so it can show them once and then forget them. Returning text would have forced the console to parse a shell script to recover a six-digit number, which works until someone reformats the script. The code is shown once and there is no button to fetch it again, because the controller keeps only an Argon2id hash. That is the server's limit, not a UI choice, and offering a "show code" button would be a lie about what it can do. ## Three defects the template flow exposed Running `curl ... | sh` on node 238 - the flow the console hands out - found that the template could never pair, for three separate reasons: - It had no prompt and no code. `000000` is the placeholder that means "nobody filled this in", so the script now asks. A rendered script from the console has the code substituted and skips the prompt, so nobody is asked for a number they do not need. - It had no node name, and the name is what the code is bound to. It now asks too. The first version failed with "check the code has not expired" while the code was fine and the *name* was the placeholder - an error pointing at the wrong thing. - It hardcoded `controller.invalid` as its controller address, so every deployment but one got a script that could not resolve its own controller. The address now comes from the Host header the template was fetched with, and `UMBRA_CONTROLLER` overrides it. The error message itself was part of the problem: a DNS failure reported "check the code has not expired", which is a worse fault than no message because it sends an operator to check the one thing that was working. ## Verified `curl -fsSL http://192.168.123.14:8442/api/node/bootstrap | sh` on 238, code and name typed at the prompts: paired, agent installed, service registered, node `online`. `/api/node/bootstrap` is unauthenticated and returns only the inert template - a script that can check a machine and then stop. It is listed by name in the test that pins unauthenticated node routes, alongside /pair, with the reason each is safe. ## Also found, not fixed `VIEWS` in web/assets/admin.js is `['dashboard','accounts','organizations','audit']` but `view-projects` exists in the markup, so the projects view can never be activated. It predates this work and I have left it alone rather than change navigation behaviour as a side effect of adding a page. 525 passed, 27 skipped. |
||
|
umbra audio
|
bc031dd079 |
Bootstrap: one file, and it works with or without root
Enrolling a node meant a person reading instructions and running them, with a credential typed into a terminal. That does not scale, and it makes the controller a courier. The flow is now: an administrator renders a script, hands over one curl, and the node sets itself up. `GET /api/node/agent` serves the agent to a credentialed node. A node credential already implies "run what the controller sends" - that is what claiming a job is - so this adds no authority; it removes the hand-copying. `POST /api/compute/pairing-script` renders the bootstrap, minting a pairing code server-side so a code pasted into a request body cannot be one issued earlier for something else. ## The exclusive/shared distinction, encoded The script asks whether it has root and branches. With root: `/opt/umbra`, a system unit, and the agent running as a dedicated `umbra-node` account with NoNewPrivileges and ProtectSystem=strict. Without: the user's own directories and a user unit. Both layouts are in the output and the runtime decides, because whoever renders the script does not always know how the person at the machine will run it. The credential goes in a 0600 file referenced by systemd's EnvironmentFile. Three alternatives were rejected and the reasons are in the code: a unit file is world-readable, a config holding a secret ends up in a backup, and the agent deliberately refuses to read the key from its config when the environment can carry it. ## Verified on node 238, not just reasoned about Ran it as `umbra` with no root: paired, installed, registered, and the agent went `online`. Six defects found only by running it: - `$BOOTSTRAP_VERSION` was interpolated into the shell body but never declared as a shell variable. `set -u` caught it on the first line that used it. - The agent fetch sent no credential and got a 401 - correct behaviour, wrong script. - `ExecStart` pointed at the agent file directly. It has no shebang, so the kernel tried to run it as a binary: "Exec format error", a service in a restart loop, and nothing on the controller. Now invoked through `python3` explicitly, and the file is 644 rather than 755 because it is source. - `sed -i` with no backup suffix is GNU-only; BSD reads the next argument as a suffix, so on macOS it would leave the unit unfiltered and install hardening directives that cannot work. Replaced with a portable filter. - Pairing a name already in use surfaced as a bare 500 from a UniqueViolation. Now `node_name_taken`, with an answer that says pairing does not replace an existing node. - The agent logged "claim failed: node_scope_required" every second, forever. A read-only credential can never gain compute:claim, so this is permanent, and a warning that cannot change trains a reader to ignore warnings. Announced once. ## A privilege bug the pinned tests caught The bootstrap renderer declared an administrator dependency while sitting on the *node* router. A node could have called it to mint itself a credential for any node name. It is now `POST /api/compute/pairing-script` on the admin router, above the agent-surface marker so `test_every_node_route_uses_node_credential_authentication` keeps meaning something. Also worth noting: a node cannot be DELETEd from the database, because the pairing-code reuse trigger refuses the `ON DELETE SET NULL` cascade. That is correct - the status enum has `retired` and the admin route drains and retires, so removal is a status change, not a row deletion. Found by trying to clean up after a failed test run. 525 passed, 27 skipped. Test nodes boot-test-1..3 are retired, not deleted. |
||
|
umbra audio
|
c56ef46190 |
Measure capability by executing instructions, not by reading cpuinfo
/proc/cpuinfo is a declaration. On node 238 it reports 'AMD Ryzen 9 3900X' and then lists 65 CPUID flags that stop at SSE4.2 - no avx, avx2 or fma, on silicon that has all three. Executing an AVX2 instruction there faults with SIGILL, so the hypervisor is presenting a CPU that does not exist. That is the same defect the compute design already has, one layer down: the agent sets throughput_factor by string-matching a GPU model (40.0 if '4070' in name else 20.0), a declared factor wearing a measured name. Here the hypervisor makes the same kind of lie about instructions. So a feature is recorded as one of three things and never two: present - an instruction used it and produced the right answer absent - an instruction used it and the kernel killed the process unverified - it could not be executed at all Only 'present' counts as usable. An unverified feature is treated as unavailable on purpose: the cost of pessimism is a slower path that works, and the cost of optimism is a job dispatched to a machine that dies on it. Measured on node 238: sse2, sse4_2, bmi2, aes_ni usable; avx, avx2, fma, avx512f all SIGILL. On oldbox: everything but avx512f. Four bugs found while building the probe, each of which would have made it report something false: - Inline asm with wrong operand counts, for all eight features. Replaced with intrinsics, verified by disassembly. - One binary for every feature could not compile: -mavx2 alone makes the FMA, AVX-512 and AES intrinsics fail always_inline. Every feature reported unverified on a machine that has most of them. Now one translation unit per feature. - GCC folded the probe at -O2 with constant inputs and emitted nothing, so it would have reported 'present' for every CPU on earth. Inputs are volatile and the expected mnemonic is checked in the disassembly; test_a_folded_probe_is_reported_unverified builds a deliberately foldable probe and asserts it is caught. - The SSE4.2 probe stored a comparison mask through _mm_storeu_pd and read a double back, getting NaN - reported as 'executed but returned a wrong result' on every machine. Also: a caller passing an explicit compiler path that does not exist got eight build errors instead of one clear sentence; the compiler is now verified to work, and a missing or broken one is a False rather than an exception. 525 passed, 27 skipped. |
||
|
umbra audio
|
32dc580c74 |
Pair nodes with a short code instead of a copied credential
Issuing a node credential by hand required writing the key into a 0600 file on the controller and having a person copy it onto another machine. That key landed in shell history and a terminal buffer, and it made the controller a courier: nothing could be enrolled without me in the middle, which is the opposite of what a fleet of heterogeneous nodes needs. `POST /api/compute/pairing-codes` (admin session) mints a six-digit code for a *named* node. `POST /api/node/pair` exchanges it, once, for a credential returned in the response. The node never sees a secret typed by a person, and the controller is no longer in the path. ## Why a six-digit code is defensible 1,000,000 possibilities is nothing on its own. What makes it work is everything around it, and any one of these alone would be too thin: - **Single use.** Redemption is `SELECT ... FOR UPDATE` then `UPDATE ... WHERE used_at IS NULL`, in one transaction. - **Expiring.** Default ten minutes, hard-bounded to 60-1800s. Long enough to walk to a machine; short enough that a code left in a terminal buffer is not a credential for the day. - **Bound to a node name.** A code read out of a terminal buffer cannot pair a machine called something else. Without this, "confirm it is the machine" is decoration. - **Argon2id at rest.** A database read or a support transcript yields nothing replayable. - **Rate limited per source address**, charged *before* any code lookup, so guessing across many codes costs the same as guessing at one. Twenty per quarter-hour against a million-code space is about 21 years. This is the control that makes the length defensible; the others are cheap additions to it. Every rejection is worded identically - "no such code", "wrong code", "expired", "already used" all read the same - so the response is not an oracle for the rate limiter to work against. The one exception is a name mismatch, which says so plainly: it is not secret, and an operator staring at a failure needs to be told the code is for a different machine rather than left guessing. A name mismatch does *not* spend the code, because the obvious implementation spends it and then the retry with the right name cannot work. ## Why not a SECURITY DEFINER function The obvious shape is a plpgsql function that verifies and redeems, so the node surface never touches the tables. It cannot be done in SQL here: the codebase hashes with Argon2id in Python, and the only hashing PostgreSQL offers in this database is `crypt()` (bcrypt and MD5). Writing pairing against bcrypt would claim an algorithm choice the rest of the system does not make. So verification is in the repository, which means the pairing tables need grants for the application role when G2 lands, and the path needs re-testing as a non-owner role. Recorded in the migration header. ## Three defects found by writing the tests - **Redemption order.** Claiming the code before inserting the node it references violates the foreign key, so *every* redemption failed. Found by nine failing tests with an FK error that had nothing to do with pairing logic. - **A race that produced a 500.** Two nodes verifying one code simultaneously both created a node, and the loser hit a unique-constraint violation on `(organization_id, name)` - an unhandled error for an entirely ordinary condition. Fixed with `SELECT ... FOR UPDATE` on the code row before the insert, so the second caller blocks, re-evaluates `used_at IS NULL`, and gets the ordinary refusal. The concurrency test fails when the lock is removed and passes when it is present. - **A test that could not fail.** Removing the `used_at IS NULL` from the *claim* left all tests passing, because the candidate query's `used_at IS NULL` filter masks it. Sequential redemption cannot distinguish the two guards; only a concurrent one can. That is now `test_a_concurrent_redemption_produces_exactly_one_node`, and `test_a_spent_code_is_not_offered_as_a_candidate` pins the first guard directly. The reuse trigger needed a correction too. It originally refused any change to `expires_at`, which would have made it impossible to revoke an unused code - the normal way to withdraw a code issued to the wrong person. It now refuses identity changes and expiry *extension*, and permits expiry to move earlier only. Two existing tests flagged the new unauthenticated route, correctly: `test_the_node_surface_is_exactly_what_a_node_may_do` and `test_every_node_route_uses_node_credential_authentication` both assumed every `/api/node` route needs a credential. Rather than weaken either, `/pair` is recorded as a named exception in both, with a check that the named route still exists - so a second unauthenticated node route has to be added deliberately with a reason. 509 passed, 27 skipped. Deployment: 0004 applied after a backup, health 200, and the live endpoint verified to refuse a bogus code with 403 and a wrong node name with a `pairing_name_mismatch` that does not spend the code. G2 is still open, and the pairing tables will need grants when it closes. |
||
|
umbra audio
|
382ca8fc78 |
Derive node status on read; a heartbeat loop for the audio node
Two problems, both found by the audio node running Stage 1.
## A node that went quiet still looked available
`compute_nodes.status` is promoted to `online` by a heartbeat and demoted to
`offline` by an administrator. Nothing demoted it on the passage of time: grep found
`offline` appearing in exactly one place, the heartbeat's promotion CASE. So an agent
that was killed left the row reading `online` indefinitely - measured on the
deployment, where `umbraaudio-m4` showed `online` against a last heartbeat nine hours
old.
That is the failure invariant I24 exists to prevent. `create_node` refuses to mark a
node ready before its agent has ever spoken, and then nothing noticed when it stopped
speaking.
Derived on read instead, in `effective_status`, called from `_node_from_row` so every
caller - console, heartbeat response, placement - gets the same answer:
offline: no recent heartbeat (derived; the stored status is still 'online')
Three decisions inside it:
- Only `online` is derived away. `draining` and `retired` are administrator
decisions about intent, and a machine going quiet does not retract an order to stop
taking work. Demoting a draining node would make the console report it as gone
rather than winding down, and blame the wrong thing.
- Nothing is written back. The stored value stays as-is and is carried on the model as
`stored_status`, so a GET has no side effect and two readers cannot disagree about
what the column says. `NodeResponse` reports it only when it differs from `status`.
- The stale-window check that was hand-rolled in the placement loop is gone. Two copies
of one rule is how they drift; the loop now reads the same function as everything
else, which is what makes the console and the scheduler agree.
A sweeper was the alternative and is how expired leases are handled, so it was not a
strange idea. It loses because it is a periodic task that must actually run on a
deployment that has already been taken down three boots by a stray process - and while
it is not running, the stored value is wrong again. Computing on read has no window and
no background dependency.
## The document was wrong about what happens when a node stops
It said "miss that window and the node goes offline". That was false, which is how this
was found. The audio node followed the instruction, sent one heartbeat, and nothing
told it or anybody that a second one was expected.
`poc/umbra_stage1.py` is a loop rather than a description of one, so the audio node can
drop it in unchanged. One `curl` is one request: it returns to the prompt when done,
which is correct for curl and indistinguishable from a healthy node that never started.
The script runs in the foreground so closing the terminal stops it, which makes the
first few minutes obvious - press Ctrl-C and the node goes quiet, and about 45 seconds
later the controller says so. It exits on 401 and 403 rather than retrying once every
ten seconds for an hour, because a credential problem does not resolve itself, and it
sends an empty report rather than nothing when the report script is missing, since a
node that stops heartbeating because its report broke is indistinguishable from a dead
machine.
Verified against the live controller: three consecutive 200s at a 10s interval, and a
wrong key producing 401 and a clean exit.
Verified on the deployment: a node quiet for 50s reads `offline` while the column still
says `online`, and reads `online` again one second after a heartbeat.
488 passed, 27 skipped. The three staleness tests were confirmed to fail with the
derivation neutered, and the export test caught `effective_status` missing from
`__all__`.
|
||
|
umbra audio
|
2ff322dcd2 |
Close all thirteen app functions in the migration, not by hand
B4 was recorded as open against the deployment. It was not open there - the ten original functions had been closed by hand - but it was open everywhere else. A database restored from a backup of the deployment comes back with 10 of 13 PUBLIC-executable, because a hand-applied GRANT is a property of one database and does not survive a restore. Observed on scratch, built from the pre-0003 dump. So 0003 now revokes from PUBLIC on all thirteen, by full signature taken from pg_proc.oid::regprocedure: a bare () does not resolve for a function taking one argument, and REVOKE on a nonexistent name aborts the whole migration. Three findings while doing it, none of which would have shown up without running the tests: - My own three guard functions went in PUBLIC-executable, reproducing the exact defect B4 is about. - Revoking PUBLIC without regranting broke five tenancy tests with a permission denied error on the platform-admin helper. Every RLS policy calls it, so a missing grant here looks like a broken policy rather than a missing grant. Regranted to the roles that actually read and write, guarded by EXISTS so a role added later is covered by the next migration rather than by luck. - The revoke block originally sat before the functions were created, so it failed on a fresh restore and passed on the already-migrated database - passing by accident, the failure mode this project keeps hitting. Both revoke blocks now sit after the triggers, and the fresh-restore path is the one verified. Verified by restoring the pre-0003 backup into scratch, applying 0003, and running the suite: 0 of 13 PUBLIC-executable, 4 guards, 474 passed, 27 skipped. Re-applying is a no-op. Deployment data unchanged (users=2 orgs=3 audit=43 perms=20); health 200, heartbeat 401 without a credential and 200 with one, so the revoke broke nothing. G2 is still open: 26 tables have RLS enabled but not forced, and the application connects as the owner. |
||
|
umbra audio
|
0501739a88 |
append-only guards, and a fixture that could not set itself up
Three defects, found by applying 0003 to the deployment after a restore-tested backup. Each was caught by a check failing for a reason that had nothing to do with what it claimed to check. artifacts had no immutability guard, so content addressing was a convention. Its identity is the bytes: an UPDATE that changes content_hash or storage_key rewrites history for every job that consumed the row, and the lineage in job_artifacts would point at content that never existed. job_artifacts and job_events had the same gap - provenance and the event trail behind I25/I26, both editable in place. The guard on artifacts is UPDATE only, and that is a decision rather than an oversight. Guarding DELETE as well would make storage reclamation impossible: no artifact row could ever be removed, and the table would grow for the life of the deployment. The provenance guarantee does not rest on that guard. job_artifacts references artifacts ON DELETE CASCADE, so deleting an artifact any job consumed fires the guard on job_artifacts and is refused. An artifact nothing consumed is by definition nothing anyone's record depends on. Four cases are tested in both directions: each test was confirmed to fail when its trigger was disabled or widened, so none of them passes because its mechanism failed. test_rls_tenancy's fixture inserted a role with a fixed id and ON CONFLICT DO NOTHING. The unique constraint is on (scope, name), so the insert silently did nothing - organization_admin already existed under a different id - and the fixture then assigned user_roles to a role that was not there. validate_user_role_scope raised inside the fixture and all eight tests errored rather than failed, which reads as "the thing under test is broken". It now looks the role up and says so if the role is absent. test_append_only_guards.py needs no teardown: job_artifacts guards DELETE, so its rows cannot be removed, and fixed ids would collide on the next run. Ids are generated per run instead. The database is disposable by construction, which is what UMBRATRANS_TEST_DATABASE_IS_DISPOSABLE=1 asserts. 474 passed, 27 skipped, 0 errors. |
||
|
umbra
|
898906ab04 |
Forward-only link agreed, and the connection sequence the audio node asked for
D16: a book_local character revealed to recur keeps its released entity and gains a recurs_as edge, rather than being merged. Its cost is written down rather than discovered later - a retelling from the earlier book's point of view treats the two as different people. Chosen because the freeze is what protects reproducibility and a merge puts it at risk for a gain that is real but retrospective. Asked whether the document was ready to feed back and how the audio node connects. Reading it against that question found four gaps, all of which were mine: - One report skeleton was given for nine phases, which specifies nothing. evidence now has a tabulated shape per phase, so the reports are mechanically comparable rather than approximately so. - The identity probe said "we send you a fixed set" without saying how the set was fixed, which would have made phase 3 either unreproducible or impossible. Now: you pick the project, sort by source_unit_key, every 17th, cap 200, and report the stride. - Section 9 said "contact the controller on 192.168.123.14:8442" and stopped, which is how this session's own 400s happened twice. - There was no connection sequence at all. Section 10 adds it, in four stages. Stages 0 and 1 are possible today and carry no project data: Stage 0 is reachability, where a 401 from the node surface is the correct answer and the evidence; Stage 1 is the node report they already wrote, sent as a heartbeat. Stage 1 needs no schema change and no new endpoint, because node_heartbeats.raw is an open jsonb. Stages 2 and 3 are ours to build and are listed as such. Two transport facts stated plainly, both learned the hard way here. The deployment accepts exactly one Host header and returns 400 on everything otherwise, including health. And there is no TLS, despite the service description claiming "TLS terminated upstream by Caddy" - Caddy is inactive and its configuration is empty. Tolerable for a read-only node report on a LAN, not tolerable for the ingest path, so putting Caddy in front of the port is a Stage 3 prerequisite rather than an extra. It starts read-only for the same reason the compute substrate does: a node is online because it heartbeated, not because it was created. A mirror that begins with data cannot distinguish "the node is broken" from "the data is bad", and their speaker counts already disagree. One stale block survived a previous edit - a replace that matched nothing and reported success. It now asserts before replacing. |
||
|
umbra
|
a032e4f542 |
Record the GPU boundary as measured, on all three accounts
Fourth consecutive boot with all five nodes 0660 root:video, the timer firing, and the unit's journal showing the helper working - so the mechanism holds. What was still inference is now measurement. ollama and umbra-poc both report the RTX 3060, and umbra - which runs the controller - is refused. The device bits and the video memberships were identical to earlier boots where nvidia-smi had been run under those accounts, but identical configuration is not an observed result. Worth noting how this was reached: nvidia-device-access.sh show had been reporting 'could not check' for the two non-root accounts since the reboot, because there was no cached sudo credential. That is the honest answer, and it is why the direct run mattered - the tool had been accurately describing its own blindness while the boundary was being claimed from configuration alone. |
||
|
umbra
|
de08cfe38e |
Stop rate-limiting the deployment: any finite budget is exhaustible
The GPU permissions survived a third consecutive boot - all five nodes 660 root:video - so that is now established rather than hopeful. The deployment was down again for the same reason, and this time the widened start limit I committed two boots ago turned out not to have fixed it. It exhausted twenty restarts and returned to the same permanent failed, with the port free and nobody touching the service. That fix's verification missed it. The port was freed after three restarts, inside the window, so the test exercised the easy case and the commit said "fixed". The real case is a fault that outlasts the budget, which is the only case that matters. The rate limit is the wrong mechanism, not the wrong value. Any finite budget can be exhausted by a fault that lasts longer than it, and the failure is silent and permanent - `failed` looks like a decision rather than a give-up. StartLimitIntervalSec=0 disables it: the unit retries for as long as the fault lasts and binds the moment it clears. Verified against the case that broke it twice. Held the port for 200 seconds, which drove 31 restarts - past the old limit of 20 - with the unit never reaching failed. Freeing the port gave 200 from /api/health within eight seconds, with nothing having started it. A genuine crash loop now logs every 5s instead of stopping: noisier, and much better than a deployment that is quietly down. 469 pass with a database, 428 without. Deployment healthy, GPU boundary intact, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, persistenced up, perms timer firing, zero stray agents. |
||
|
umbra
|
1dbf5fb384 |
The GPU permissions survive a reboot, and the deployment stops giving up
Second boot since the permission fix, and it held: all five nodes 0660 root:video, the timer firing on schedule, and umbra - which runs the controller - still refused the card. That claim is now an observation rather than an inference. The same reboot surfaced an unrelated fault worth fixing. The deployment did not come up: a `python3 -m http.server` from a development session held port 8442, serving web/ - which the backend mounts itself and does not need - and the service failed five times in ten seconds. With StartLimitIntervalSec=120 and StartLimitBurst=5 that exhausted the budget, and systemd then never tried again. The deployment stayed down after the port was free and needed someone to notice, which is the second reboot in a row to end this way. A transient condition had become a permanent outage. Widened to 500s and a burst of 20, which keeps retrying across a slow boot or a settling reverse proxy while still stopping a genuine crash loop. Verified by holding the port with a squatter and watching: the unit cycles active/activating without giving up, and the instant the port is freed the next retry binds and /api/health returns 200. My first attempt at this test was void - the squatter could not bind because the service still held the port, so it proved nothing. Note that two consecutive reboots have been taken down by a background shell in this session holding the deployment's port. It returns after each boot. Stopping that shell, or moving it to another port, is the durable fix; killing it again only defers it. 469 pass with a database, 428 without. Deployment healthy, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, agent account intact, zero stray agents. |
||
|
umbra
|
4937b9beed |
Record that the GPU permissions now hold, and how that was verified
Cleared the unit's stale active(exited) state so the corrected version could run, then broke the permission deliberately the way nvidia-persistenced does and watched the timer repair it: 666 -> 660 root:video, with the helper reporting 'restricted 5 node(s)' in the journal. ollama and umbra-poc can measure the GPU; umbra, which runs the controller, cannot. Deployment healthy, ollama serving on 11434, database unchanged. Records what is still unverified: no boot has been observed end to end since the change. The unit, timer and udev hooks are installed and exercised individually, and the persistenced interaction is handled by ordering rather than luck, but 'works after a reboot' is a claim until a reboot confirms it. show answers it in one command. |
||
|
umbra
|
0ab1f0f1de |
The GPU boundary did not survive a reboot, and neither did the fix for it
This is the verification I said could not be done until the machine rebooted. The reboot proved the boot path wrong, and uncovering why took three rounds. **The boundary was gone.** /dev/nvidiactl came back 0666 root:root, so the controller could open the card again - the exact thing the udev rule was applied to prevent. **Why: the unit ran before anything existed.** umbra-nvidia-devices ran at 05:38:24, ordered After=systemd-udev-settle, globbed /dev, found no nodes, and exited 0. udev-settle means "udev has finished its queue", not "the proprietary driver has finished initialising". The nodes appeared moments later at 0666 and nothing revisited them. The helper ended in an unconditional `exit 0` - I had written that deliberately, so an early boot would not fail a unit - and it is precisely what made "I fixed the permissions" indistinguishable from "there were nothing to fix". **Why the first repair also failed: nvidia-persistenced resets the mode.** It chmods /dev/nvidiactl back to 0666 on every initialisation, via nvidia-modprobe. That is out of udev's reach entirely, because udev does not manage these nodes. So the fix is no longer an ordering problem but an eventual one: the unit now runs After=nvidia-persistenced.service, and a two-minute timer re-asserts the permissions because the trigger that breaks them can fire at any time. **Why the timer did nothing either.** RemainAfterExit=yes on the oneshot keeps it "active (exited)", so systemd treated every activation as already satisfied: the timer fired on schedule, the service reported Result=success and ExecMainStatus=0, and the permission was still wrong two minutes later. That is the original bug repeated one level out, inside the mechanism built to repair it. Removed, and a test now pins it - by directive name, with comments stripped, since the unit documents the omission by naming it and a substring search failed on the explanation. **I also broke nvidia-persistenced.** It drops to uid 122 with setuid and without initgroups(), so it never receives the video supplementary group and failed with "Insufficient Permissions" once the nodes were 0660 root:video. A systemd drop-in runs it as root - upstream's own default - which works without widening the device permissions. It was running fine before. The helper now waits for the nodes, verifies the modes afterwards, and exits non-zero under --strict rather than reporting a no-op as success. It takes UMBRA_NVIDIA_DEVDIR so its three paths can be tested without root and without a GPU; the first attempt at that test pointed CANARY at a non-existent path and still exited 0, because the fix loop globbed /dev directly and the override only reached the wait. Also: nvidia-device-access.sh `show` reported "cannot measure the GPU" when a cached sudo credential had simply expired - a finding about the machine when the truth was that the tool could not ask. It now distinguishes no from could-not-check, because I was misled by it during this check. **Still outstanding, and I could not finish it.** The repository is correct; the installed state is stale, because clearing the unit's active (exited) state needs root and that authorization was declined. As of now /dev/nvidiactl is 0666 and the controller can open it. To finish: systemctl stop umbra-nvidia-devices.service systemctl start umbra-nvidia-devices.service stat -c '%a %U:%G %n' /dev/nvidiactl and expect 660 root:video. Otherwise the system came through the reboot intact: the deployment was down only because a leftover `python3 -m http.server` from an earlier session was holding port 8442, and the backend mounts the same files itself. Deployment healthy, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, agent account intact, zero stray agents. 471 pass with a database, 430 without. |
||
|
umbra
|
1ab9c2595d |
Verify the datastore policies as a role that does not own the tables
They had never been observed to refuse anything, and until now every
statement this project made about tenant isolation was a statement about
the adapter's own WHERE clauses.
The reason was structural. RLS does not apply to a table's owner until
FORCE ROW LEVEL SECURITY, and every connection here is the owner. A probe
run as the owner showed organization B's administrator successfully
updating organization A's imprint - exactly what WITH CHECK was written
to prevent. A policy that cannot refuse is indistinguishable from a policy
that is absent, which is why the previous commit said so plainly rather
than claiming the migration was verified.
test_rls_tenancy.py runs as rls_probe, a role with plain table privileges
and no ownership. Eight tests, each checked against a mutation: each
organization sees only its own artifacts, an unattached principal sees
nothing on any datastore table, a publisher is readable across tenants but
writable only by its owner, a cross-tenant UPDATE changes zero rows, and a
cross-tenant INSERT is refused. Two of the eight also assert the positive
cases, because a policy that refuses everything would satisfy every
refusal above.
Two behaviours asserted so a change is noticed: a filtered UPDATE or
DELETE raises nothing and simply affects zero rows, while an INSERT
violating WITH CHECK raises InsufficientPrivilege. The first is a trap
for application code that checks "did it raise?" instead of "did it
change a row?", and it is why the earlier manual probe reported a
cross-tenant update as "allowed" when the statement had matched nothing.
Two mistakes in the tests, both worth recording. A policy violation aborts
the transaction, so the second refusal in a test could not even be
attempted and reported as "DID NOT RAISE" when nothing had run; each
assertion now gets a savepoint. And a module-level pytest.skip() is a
collection error rather than a skip, which broke the whole suite instead
of quietly omitting the file - the guard now evaluates to a boolean.
This does not close gate G2. The application still connects as the owner,
so in the running system these policies still enforce nothing. What is
established is that they are correct, which makes FORCE ROW LEVEL SECURITY
plus a non-owner application role a configuration change with a
known-good target rather than an untested hope. README.md says exactly
that, and the gate stays open.
Creating the role needs superuser and was authorized this time:
sudo -u postgres psql -d umbratrans_scratch -c \
"CREATE ROLE rls_probe NOLOGIN; GRANT rls_probe TO umbra;"
465 pass with a database, 424 without. Deployment untouched: users=2
orgs=3 projects=0 audit=43, no compute or datastore tables.
|
||
|
umbra
|
5fc38d4708 |
The datastore, and two parties that never sign in
Compute capability is not LLM work. It is mechanical split, analysis,
text to speech, PDF formatting and audio analysis, assigned as needed,
with the code on every node. That needs a capability vocabulary rather
than VRAM - which is noted and not yet built - and, more urgently, a
place to put the output. jobs.result is a jsonb column: right for
{"echo": "hello from the queue"}, wrong for "the English audiobook, four
hours twelve minutes, derived from these thirty-seven chapters and this
cast".
So 0003 gains section 5: artifacts, job_artifacts, publishers,
project_publishers, authors, project_authors, and eight permission
keys. Folded in here rather than deferred to 0004 because 0003 has still
never been applied to the deployment, and an unapplied migration can
still be edited.
The word "account" is now absent from every new column, table and
comment. It has meant two incompatible things in this system - a person
who signs in, and a house a work is published under - and a schema
using one word for both will use it for both in one constraint. An
organization owns the work. A user is a worker and owns nothing. A
publisher is the party a work is published under and has no password, no
session and no row in user_roles, because a name that appears on a work
is not a party that can be asked to prove who it is.
A byline is a third party and is neither. In Umbra -> WorstAuthor ->
B.D. Murphy, WorstAuthor is the publisher and B.D. Murphy is the
byline; different lifetimes, and the spec had been calling both "brand".
project_authors carries a credit_role rather than a single author_id,
because the author of the English edition is the translator of the
French one and the narrator is frequently neither.
owner_organization_id is the one deliberately cross-tenant column, and
the policies are asymmetric about it: reading a publisher is open to any
authenticated principal because publishers are shared reference data,
while setting that column is refused to anyone but a platform
administrator. There is no second policy carving that out, because
PostgreSQL combines permissive policies with OR and an "admins may also"
policy widens access for everyone. The WITH CHECK on the single FOR ALL
policy is doing that job alone.
Found while testing rather than while writing: RLS was not enabled on
four of the six new tables. PostgreSQL creates a policy on a table where
it cannot apply and reports success, so pg_policies listed them and
nothing was enforced - and a check for "does the policy exist" would
have passed throughout. test_rls_coverage.py asserts relrowsecurity
instead, and each of its six assertions has been checked against a
mutation that breaks it. One needed strengthening: dropping the
dedicated SELECT policy left the table looking fully policed, because
FOR ALL includes SELECT, so the surviving policy quietly gated reading on
organization ownership and hid shared reference data.
Also: the nvidia tests failed for want of a sudo credential rather than
because anything was wrong. They skip now, with the reason stated. A
test that fails for environmental reasons teaches the reader to ignore
red.
The policies have never been observed to refuse anything, and README.md
says so plainly. A probe as the owner showed organization B's
administrator editing organization A's imprint - what WITH CHECK was
written to prevent - because RLS does not apply to a table's owner until
FORCE ROW LEVEL SECURITY. That is gate G2, not a new problem. Testing it
for real needs a non-owner role and superuser; the request was not
granted, so it is not done.
455 pass with a database, 422 without.
|
||
|
umbra
|
c1741456e2 |
Apply the GPU boundary, and fix three checks that verified themselves
The udev rule I wrote last commit could not have worked. The nvidia driver creates /dev/nvidia* in the kernel without registering them in sysfs, so `udevadm info --name=/dev/nvidiactl` answers "Unknown device" and udev never evaluates a rule for them - which is also why they are 0666, since nothing sets a MODE. The rule hooks the driver bind event instead and runs a script, by which point the nodes exist. The first version of that hook triggered 0000:01:00.0, which on this machine is an AMD root complex, vendor 0x1022 rather than 0x10de. The trigger fired on the wrong device, the RUN key never ran, and the permissions looked right because the apply script had just fixed them itself. The address is discovered now. Verified by resetting to 0666, firing only the bind event, and watching it come back 0660 - the script never invoked. Applied and verified: ollama can open the card, umbra-poc can, umbra cannot. Traced, nvidia-smi opens /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm, so umbra-poc is in video or the agent starts, heartbeats, and reports a machine with no GPU - a failure that presents as a scheduling problem and is a permissions one. It is the agent's only supplementary group and grants a device, not a filesystem, so setup's "no supplementary groups" check now allows exactly video and rejects everything else, verified by adding users and watching it fail. Three tests, each verified against a mutation. The rule test resets the permissions and fires only the event, so it fails if the rule is inert. The end-state test fails rather than skips when the rule is missing - skipping made deleting the rule quieter, three passed becoming one passed and two skips, which reads as an improvement in the wrong direction. Also: the check "the controller cannot open the card" was testing root, which bypasses permissions entirely, and reported a working rule as a failure. The account is named now. Then four runs in a row leaked agents, and the cause was upstream of the leak. The victim reused the first agent's credential, and a node is identified by its credential, not by its config's node_name - so it heartbeated as poc-agent-node, was never offered the long job, and sat there for the whole step while a correct liveness check reported that the worker it had killed had survived. It has a third node and its own credential now, so the requeued job is not immediately reclaimed by an agent already stopped for the step. The last leak was a worker asleep in a 600-second job, unable to see a stop file because the check is at the top of its loop rather than inside the work. Four runs left one each. The job is 10 seconds now: the victim exits the instant it claims it and never runs it, so the length was never the point. PoC node names and idempotency keys carry a per-run suffix, so a second run against the same database does not die on a UniqueViolation - and there is no route to delete a node. Three consecutive runs, zero failures, zero leaks. 451 pass with a database, 424 without. Deployment untouched: users=2 orgs=3 projects=0 audit=43, no compute tables. |
||
|
umbra
|
0f5c1e3408 |
Run the agent as the service account it claims to be
The compute proof of concept had never been run with the agent as umbra-poc. Running it that way found three defects, all the same shape: a check that passed because its mechanism did nothing. The demonstration never reached the agent at all. It held the pid it spawned, which under `sudo -n -u` is the sudo wrapper - sudo forks the agent before exec'ing. Signalling that killed the wrapper and left the worker running, renewing its lease and holding the job indefinitely, under a printed note saying "exactly like a power cut". An unprivileged process cannot signal another account's process at all, and kill(2) on a process group succeeds as soon as one member is signalled, so the wrapper's death read as the agent's. Every run leaked an agent. Lease recovery was a diagnosis rather than a mechanism. sweep_expired_leases documented itself as "called by the controller's own timer" and there was no timer; a killed node's job came back only when somebody noticed a stalled job and posted to an administrative route by hand. The proof of concept swept by hand and recorded the result as automatic. app.py now runs the loop for the life of the application, building the compute repository at startup so a restart mid-job still recovers it. The power cut is now the agent's own: os._exit(137) the instant it claims a job, before running it, because a process that does not own the agent cannot signal it. A clean stop goes through a stop_file the agent polls, for the same reason. Liveness is measured by finding the worker's own pid in /proc matched on its config path, sampled repeatedly and accumulated - the wrapper and the worker are not simultaneous, so a single sample finds only the wrapper. The node key travels by `sudo --preserve-env` rather than as VAR=value in argv, because argv is world-readable through /proc/<pid>/cmdline and the environment is not. The demonstration checks whether the key will survive before starting and falls back to a pipe, which node_agent.py --node-key-stdin accepts. Neither a file nor argv ever holds it. Eight tests, each verified to fail against a mutation of the code it covers: three for the sweeper (it must return an expired lease, must survive a failing round, must not invent a repository it has no database for) and five driving a real agent process against a stub controller (stops on the stop file, exits 137 holding an uncompleted job, takes the key from both routes and neither, reports an empty pipe as an empty pipe). 448 pass with a database, 421 without. The deployment database is untouched: users=2 orgs=3 projects=0 audit=43, no compute tables. |
||
|
umbra audio
|
d13664a607 |
A compute substrate: node inventory, a leased queue, and an agent
Implements §6.16-6.20 and §7 of the specification, and proves it end to end on this box: a controller and an agent, the agent with no database credential at all, ten checks against the properties whose failure modes are silent. Migration 0003 adds compute_nodes, compute_devices, node_credentials, model_registry, node_models, node_heartbeats and job_events, and creates jobs - which §6.8 had specified and 0001 had never migrated. Every column added to jobs is nullable or defaulted, every policy is written against the same app.* helpers the existing eleven tables use, and re-applying is safe: CREATE POLICY has no IF NOT EXISTS, which is how 0001 learned the same lesson for ADD CONSTRAINT. Four properties, and each exists because the alternative fails silently: - **I25** claiming is FOR UPDATE SKIP LOCKED. Two agents claiming at once get different jobs. The proof of purpose counts the 'claimed' events rather than counting successes, because two agents both succeeding is not evidence. - **I26** a lease that expires returns the job to the queue, and the proof of purpose SIGKILLs the holder rather than stopping it, because a clean stop would release the lease and a clean stop is not the failure being tested. - **I24** a node is online because it heartbeated. Creation is not readiness. - **I28** a node credential reaches queue operations and nothing else. The organization comes from the credential, so a forged tenant identifier has nothing to attach to. And the one that is hardest to get right in the cheap direction: **a device that is slow is eligible, and one that is too small is not.** A model that does not fit in VRAM runs on system RAM, slowly, and a job nothing can run is 'blocked' with the reason rather than left looking like it is progressing. The proof of purpose found six defects, listed in §7.8. The two worth naming here: - Mounting one router at two prefixes published the administrator's routes behind node-credential authentication, so a node could create nodes and issue itself more authority. Two routers now, with a test that pins the surfaces disjoint and the node surface exactly. - **The correction I was given about unified memory was in the specification and not in the code.** Eligibility was VRAM-only, so every CPU and every shared-memory machine was ineligible, which would have reduced the fleet to one machine. Writing the implementation is what kept the error alive. Also: the CSRF middleware blocked the whole node API, because a bearer-authenticated agent has no ambient cookie for a cross-site request to ride on. Exempted by prefix, with a test asserting that no cookie-authenticated route joined it. 419 tests pass with the integration suite against a real database. The deployment database is untouched and has no compute tables. |
||
|
umbra audio
|
bfde763f00 |
Fix a 500 on /api/health from an uninterpolated SQL placeholder
The first deploy of the previous commit took the health endpoint down. The
tenant-row predicate had been added to `count_users_with_role` as a literal
`{TENANT_ROW_PREDICATE}` inside a plain triple-quoted string rather than an
f-string, so the braces were sent to PostgreSQL as text and the statement failed
to parse. That query backs `setup_is_required`, which every unauthenticated
health probe calls, so `/api/health` returned 500 and the deployment reported
itself broken.
Nothing caught it before production, and the reasons are worth keeping:
- the unit tests use the in-memory repository, which has no SQL;
- the integration tests did not call `count_users_with_role`;
- and the source-level test I wrote to pin the predicate's presence was
satisfied just as well by the broken version, because the broken version does
contain the text. A test that checks a predicate was *written down* cannot tell
a working predicate from an inert one.
Two tests now, each catching the regression on its own:
- `test_no_sql_string_carries_an_uninterpolated_placeholder` walks the parse tree
rather than the text, so a `{...}` in a non-f-string fails whether or not the
surrounding code mentions the constant. It reports the offending query, not a
line number: `inspect.getsource` returns a class body lifted out of the file, so
its line numbers do not point anywhere a reader could open.
- `test_the_health_probe_queries_run_against_the_real_schema` runs the probe's
own queries against the real schema, and
`test_setup_is_required_ignores_project_assignments` holds the predicate's
purpose: a project row is never mistaken for a platform administrator.
413 tests pass with the integration suite; 388 without a database.
|
||
|
umbra audio
|
ed72f61975 |
Separate organizations, and give members their organization's projects
An organization's work is separate from every other organization's, and the separation is now one module rather than a property each route remembers. `tenancy.py` resolves the caller's scope in a single place. A member is pinned to the organization on their own assignment, expressed as a concrete identifier rather than a flag, so "no filter" is unreachable except for a platform administrator. Every read and write resolves through it, so there is no per-route variation to drift. Within an organization, every account sees every project. The brief is explicit that two people working on different books both see both, and assignment says who is *working* on a project rather than who may *know* it exists. This diverges from `app.can_access_project`, which restricts visibility to assigned members; the divergence, and what it means if row-level security is ever switched on, are recorded in migrations/README.md because that would otherwise be a silent tenant-internal regression no test here would catch. Two rules, both learned by getting them wrong first: - A read naming another organization is ignored, because refusing it would confirm the identifier exists. A write is refused rather than retargeted, because a project landing in an organization the caller did not ask for is somewhere they will not look. - A missing project and a foreign one get the same error, so an identifier cannot be probed across a boundary. Projects: the `projects` table, its statuses, and the project-scoped arm of `user_roles` were already in the schema and unused. No migration. The dashboard is one server-scoped call, `/api/workspace`, rather than several assembled in the browser — pieces from different tenants that nobody notices are not a failure mode worth shipping. Filters narrow the list already in the page, so switching cannot mix two fetches, and an empty list is distinguishable from a failed one. **A sign-in lockout, found by the integration suite.** An assignment is a second active `user_roles` row, and `_resolve_assignment` refused an account with more than one. So assigning anybody to anything made them unable to authenticate — locked out of the page the assignment was meant to show them. `_sync_assignment` had the same defect. Six queries were scope-blind; three were harmless only because the assignment role is named `viewer` and never matched a tenant role name, which is luck rather than a property, so a structural test now pins the predicate on every one. The guard against two *tenant* rows survives, narrowed to the ambiguity that actually matters. Also fixed: `get_project_by_slug` cast `slug::text`, making citext's case-insensitivity a case-*sensitive* comparison that appeared to work everywhere else. An assignment grants the `viewer` project role, since no `member` role exists at project scope; it is a named constant, not a query literal, because the project workflow is not defined yet. 410 tests pass with the integration suite against a real database; 386 without. |