master
22 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
umbra audio
|
be04e19d04 |
Add UmbraAudio controller integration endpoints
- Add migration 0010: utterance_translations table with UNIQUE(utterance_id, language_code)
- Add schemas: OrganizationCreateRequest/Response, ProgramCreateRequest/Response,
UtteranceTranslationItem/Request/Response/ListResponse, SpeakerSyncRequest/Response
- Add service methods: create_organization, create_program, sync_speakers,
upsert_utterance_translations, get_utterance_translations
- Add routes_controller.py: POST /api/orgs, POST /api/programs,
POST /api/speakers/sync, POST /api/projects/{id}/translations,
GET /api/projects/{id}/translations/{language_code}
- Mount controller_router at /api in app.py
- Create bdumbra account (member, BDMurphy org) for integration testing
|
||
|
umbra audio
|
4160ec49a0 | audio: Phase 2 — Postgres AudioImportStore adapter + 0009 ledger; use PG store on commit with org from node; 0008 header updated; tests | ||
|
umbra audio
|
9432366183 |
The catalogue: declare files that already exist, without moving the bytes
The audio node is the master for the audiobooks. UmbraTrans holds the metadata needed to reason about a file and a reference to where it is; it does not hold the bytes. DECISION-6 (object storage) stays unresolved, and this is what lets the system be useful before it is resolved rather than after. POST /api/compute/artifacts takes a manifest against a project and declares each entry: content hash, size, media type, kind, and a storage_uri naming where the file actually is. GET /api/compute/artifacts lists the catalogue. There were no artifact routes at all before this - the tables existed since migration 0003 and nothing had ever written to them, so "the controller has the artifacts to show a job is complete" had nothing behind it. Every declared row lands integrity='unverified', and that is the point rather than a caveat. Ingesting is a claim: a node reporting the hash of a file is a statement about bytes nothing here has read, because there is no object storage to read them from. A design where submitting a manifest marks the catalogue verified would make the first ingest the most reassuring and least true event in the system's life. verified_at is set by reading the bytes back and hashing them, and nothing in this path does that. storage_node_id is part of the reference rather than an afterthought, because a bare path means nothing six months later: /audio/book.mp3 is on whichever machine happened to serve it that day. Idempotent on (project_id, content_hash), so the same manifest can be submitted twice, or by two operators who did not talk to each other, without the catalogue doubling - and a partly-present manifest reports what was new rather than just "done". An operator who submitted 300 entries and was told "done" would not know two were malformed and dropped, so malformed entries are named in the response instead. The listing is scoped by organization on every call. artifacts is platform-level, as every table a node serves from is, so a missing predicate there is a cross-tenant read rather than a style issue. A project in another tenant's schema is refused as *not found*, not as *not permitted*, because a refusal that says "this exists but is not yours" answers the question the caller was actually asking. Migration 0007 applied to the deployment. Mutation-verified: marking an ingest verified fails 1, dropping the idempotency check fails 2, and dropping the organization predicate from the listing fails 7. 704 passed, 4 skipped. |
||
|
umbra audio
|
02fa70d8ad |
Revoke a credential, and take a machine out of the fleet for good
node_credentials.revoked_at has existed since the first migration and
resolve_node_credential has always honoured it. Nothing could set it. The ability to cut a
node off was present in the data model and absent from every API, which is the shape of
defect worth naming: the column was the capability.
POST /nodes/{id}/revoke cuts a node off and leaves it - right for a compromised key or a
machine about to be reimaged. POST /nodes/{id}/remove takes the machine out: credentials
revoked, devices, heartbeats and model reports deleted, status 'removed'. 'removed' is
deliberately not settable through PATCH /nodes/{id}/status, because typing 'remove' as a
status change would skip the revocation and the release of held work, which are the two
things that make removal mean anything.
The row is kept and cannot be deleted. job_events.node_id and
node_pairing_codes.used_by_node_id both reference compute_nodes ON DELETE NO ACTION, so
the DELETE is refused; changing them to SET NULL does not help, because that is an UPDATE
of job_events and job_events_append_only raises on every update by design. So a node that
has ever done work cannot be removed without weakening the trail's immutability, and the
trail is worth more than a tidy DELETE. A trigger makes 'removed' terminal, and
set_node_status refuses it first so the caller gets a reason rather than a constraint
violation.
Work the node was holding is released at removal rather than left to its lease. An
operator removing dead hardware already knows the work did not finish, and making them
wait out a lease to find that out is the wrong way round - for a long lease, most of an
hour of queue stalled behind a machine nobody is going to switch on.
Also creates job_assignments, which is how the controller will hand work to a node:
one row per (job, node, attempt), so a job can be open on several machines at once and
the controller can say which of them finished it. state is deliberately not the job's
status, and artifacts_verified_at is written by verification alone - that column is the
whole of 'a job is complete when the controller has the artifacts to show it'.
Migration applied to the deployment inside a transaction that first normalised any
'online' node to 'provisioning', since the new terminal-state trigger refuses that
transition and those rows would otherwise have blocked it.
Mutation-verified: not releasing held work fails 6 tests, leaving the credential working
fails 1, and a revocation that sets revoked_at back to NULL fails 2.
652 passed, 4 skipped.
|
||
|
umbra audio
|
2cdebf5ac3 |
Schema per organization: the migration, and why these four tables
Umbratools_2.md 5.0 replaces shared-tables-plus-RLS with one schema per
organization, because RLS was measured to be unenforceable twice: the
application connects as the table owner so the policies did not apply to it
(gate G2), and one app function was PUBLIC-executable until it was fixed by
hand against one database (defect B4). A model whose correctness depends on a
grant somebody remembers to make is not a boundary.
The classification comes before the SQL because the SQL's blast radius depends
on it, and the obvious reading of "move the tenant tables" is wrong for eight of
the tables carrying an organization_id. Four move: projects, project_authors,
project_publishers, invitations. The other twenty-six stay, each for a reason
read out of the actual columns and foreign keys rather than the table names.
Two findings the plan did not price, both recorded in the classification:
- user_roles cannot move. It is the index that resolves a session to an
organization, so putting it in a tenant schema means finding the tenant by
searching every tenant schema, on the login path. It also holds
platform-scope rows, which have no organization to belong to. So one table
stays shared, and 5.0's "no table is shared between organizations" is not
fully achievable while the session-to-tenant resolution is a table.
- moving projects forces dropping three foreign keys that point at it from
artifacts, jobs and user_roles, because a foreign key cannot name a table
whose schema depends on the row. That is the one place the decision costs a
database-enforced guarantee, and it is written down rather than papered over
with a trigger that would have to rediscover which tenant is meant.
The migration names each tenant schema from the organization id rather than
storing it, so a name cannot disagree with the rows and the rollback can rebuild
the organization_id column from the schema. It is idempotent - verified by
applying it twice - and the rollback is a file that runs rather than a comment
block, because a rollback for a data migration that only exists as a comment is a
rollback nobody runs. Both were applied to the scratch database, rolled back, and
applied again.
|
||
|
umbra audio
|
32dc580c74 |
Pair nodes with a short code instead of a copied credential
Issuing a node credential by hand required writing the key into a 0600 file on the controller and having a person copy it onto another machine. That key landed in shell history and a terminal buffer, and it made the controller a courier: nothing could be enrolled without me in the middle, which is the opposite of what a fleet of heterogeneous nodes needs. `POST /api/compute/pairing-codes` (admin session) mints a six-digit code for a *named* node. `POST /api/node/pair` exchanges it, once, for a credential returned in the response. The node never sees a secret typed by a person, and the controller is no longer in the path. ## Why a six-digit code is defensible 1,000,000 possibilities is nothing on its own. What makes it work is everything around it, and any one of these alone would be too thin: - **Single use.** Redemption is `SELECT ... FOR UPDATE` then `UPDATE ... WHERE used_at IS NULL`, in one transaction. - **Expiring.** Default ten minutes, hard-bounded to 60-1800s. Long enough to walk to a machine; short enough that a code left in a terminal buffer is not a credential for the day. - **Bound to a node name.** A code read out of a terminal buffer cannot pair a machine called something else. Without this, "confirm it is the machine" is decoration. - **Argon2id at rest.** A database read or a support transcript yields nothing replayable. - **Rate limited per source address**, charged *before* any code lookup, so guessing across many codes costs the same as guessing at one. Twenty per quarter-hour against a million-code space is about 21 years. This is the control that makes the length defensible; the others are cheap additions to it. Every rejection is worded identically - "no such code", "wrong code", "expired", "already used" all read the same - so the response is not an oracle for the rate limiter to work against. The one exception is a name mismatch, which says so plainly: it is not secret, and an operator staring at a failure needs to be told the code is for a different machine rather than left guessing. A name mismatch does *not* spend the code, because the obvious implementation spends it and then the retry with the right name cannot work. ## Why not a SECURITY DEFINER function The obvious shape is a plpgsql function that verifies and redeems, so the node surface never touches the tables. It cannot be done in SQL here: the codebase hashes with Argon2id in Python, and the only hashing PostgreSQL offers in this database is `crypt()` (bcrypt and MD5). Writing pairing against bcrypt would claim an algorithm choice the rest of the system does not make. So verification is in the repository, which means the pairing tables need grants for the application role when G2 lands, and the path needs re-testing as a non-owner role. Recorded in the migration header. ## Three defects found by writing the tests - **Redemption order.** Claiming the code before inserting the node it references violates the foreign key, so *every* redemption failed. Found by nine failing tests with an FK error that had nothing to do with pairing logic. - **A race that produced a 500.** Two nodes verifying one code simultaneously both created a node, and the loser hit a unique-constraint violation on `(organization_id, name)` - an unhandled error for an entirely ordinary condition. Fixed with `SELECT ... FOR UPDATE` on the code row before the insert, so the second caller blocks, re-evaluates `used_at IS NULL`, and gets the ordinary refusal. The concurrency test fails when the lock is removed and passes when it is present. - **A test that could not fail.** Removing the `used_at IS NULL` from the *claim* left all tests passing, because the candidate query's `used_at IS NULL` filter masks it. Sequential redemption cannot distinguish the two guards; only a concurrent one can. That is now `test_a_concurrent_redemption_produces_exactly_one_node`, and `test_a_spent_code_is_not_offered_as_a_candidate` pins the first guard directly. The reuse trigger needed a correction too. It originally refused any change to `expires_at`, which would have made it impossible to revoke an unused code - the normal way to withdraw a code issued to the wrong person. It now refuses identity changes and expiry *extension*, and permits expiry to move earlier only. Two existing tests flagged the new unauthenticated route, correctly: `test_the_node_surface_is_exactly_what_a_node_may_do` and `test_every_node_route_uses_node_credential_authentication` both assumed every `/api/node` route needs a credential. Rather than weaken either, `/pair` is recorded as a named exception in both, with a check that the named route still exists - so a second unauthenticated node route has to be added deliberately with a reason. 509 passed, 27 skipped. Deployment: 0004 applied after a backup, health 200, and the live endpoint verified to refuse a bogus code with 403 and a wrong node name with a `pairing_name_mismatch` that does not spend the code. G2 is still open, and the pairing tables will need grants when it closes. |
||
|
umbra audio
|
2ff322dcd2 |
Close all thirteen app functions in the migration, not by hand
B4 was recorded as open against the deployment. It was not open there - the ten original functions had been closed by hand - but it was open everywhere else. A database restored from a backup of the deployment comes back with 10 of 13 PUBLIC-executable, because a hand-applied GRANT is a property of one database and does not survive a restore. Observed on scratch, built from the pre-0003 dump. So 0003 now revokes from PUBLIC on all thirteen, by full signature taken from pg_proc.oid::regprocedure: a bare () does not resolve for a function taking one argument, and REVOKE on a nonexistent name aborts the whole migration. Three findings while doing it, none of which would have shown up without running the tests: - My own three guard functions went in PUBLIC-executable, reproducing the exact defect B4 is about. - Revoking PUBLIC without regranting broke five tenancy tests with a permission denied error on the platform-admin helper. Every RLS policy calls it, so a missing grant here looks like a broken policy rather than a missing grant. Regranted to the roles that actually read and write, guarded by EXISTS so a role added later is covered by the next migration rather than by luck. - The revoke block originally sat before the functions were created, so it failed on a fresh restore and passed on the already-migrated database - passing by accident, the failure mode this project keeps hitting. Both revoke blocks now sit after the triggers, and the fresh-restore path is the one verified. Verified by restoring the pre-0003 backup into scratch, applying 0003, and running the suite: 0 of 13 PUBLIC-executable, 4 guards, 474 passed, 27 skipped. Re-applying is a no-op. Deployment data unchanged (users=2 orgs=3 audit=43 perms=20); health 200, heartbeat 401 without a credential and 200 with one, so the revoke broke nothing. G2 is still open: 26 tables have RLS enabled but not forced, and the application connects as the owner. |
||
|
umbra audio
|
0501739a88 |
append-only guards, and a fixture that could not set itself up
Three defects, found by applying 0003 to the deployment after a restore-tested backup. Each was caught by a check failing for a reason that had nothing to do with what it claimed to check. artifacts had no immutability guard, so content addressing was a convention. Its identity is the bytes: an UPDATE that changes content_hash or storage_key rewrites history for every job that consumed the row, and the lineage in job_artifacts would point at content that never existed. job_artifacts and job_events had the same gap - provenance and the event trail behind I25/I26, both editable in place. The guard on artifacts is UPDATE only, and that is a decision rather than an oversight. Guarding DELETE as well would make storage reclamation impossible: no artifact row could ever be removed, and the table would grow for the life of the deployment. The provenance guarantee does not rest on that guard. job_artifacts references artifacts ON DELETE CASCADE, so deleting an artifact any job consumed fires the guard on job_artifacts and is refused. An artifact nothing consumed is by definition nothing anyone's record depends on. Four cases are tested in both directions: each test was confirmed to fail when its trigger was disabled or widened, so none of them passes because its mechanism failed. test_rls_tenancy's fixture inserted a role with a fixed id and ON CONFLICT DO NOTHING. The unique constraint is on (scope, name), so the insert silently did nothing - organization_admin already existed under a different id - and the fixture then assigned user_roles to a role that was not there. validate_user_role_scope raised inside the fixture and all eight tests errored rather than failed, which reads as "the thing under test is broken". It now looks the role up and says so if the role is absent. test_append_only_guards.py needs no teardown: job_artifacts guards DELETE, so its rows cannot be removed, and fixed ids would collide on the next run. Ids are generated per run instead. The database is disposable by construction, which is what UMBRATRANS_TEST_DATABASE_IS_DISPOSABLE=1 asserts. 474 passed, 27 skipped, 0 errors. |
||
|
umbra
|
898906ab04 |
Forward-only link agreed, and the connection sequence the audio node asked for
D16: a book_local character revealed to recur keeps its released entity and gains a recurs_as edge, rather than being merged. Its cost is written down rather than discovered later - a retelling from the earlier book's point of view treats the two as different people. Chosen because the freeze is what protects reproducibility and a merge puts it at risk for a gain that is real but retrospective. Asked whether the document was ready to feed back and how the audio node connects. Reading it against that question found four gaps, all of which were mine: - One report skeleton was given for nine phases, which specifies nothing. evidence now has a tabulated shape per phase, so the reports are mechanically comparable rather than approximately so. - The identity probe said "we send you a fixed set" without saying how the set was fixed, which would have made phase 3 either unreproducible or impossible. Now: you pick the project, sort by source_unit_key, every 17th, cap 200, and report the stride. - Section 9 said "contact the controller on 192.168.123.14:8442" and stopped, which is how this session's own 400s happened twice. - There was no connection sequence at all. Section 10 adds it, in four stages. Stages 0 and 1 are possible today and carry no project data: Stage 0 is reachability, where a 401 from the node surface is the correct answer and the evidence; Stage 1 is the node report they already wrote, sent as a heartbeat. Stage 1 needs no schema change and no new endpoint, because node_heartbeats.raw is an open jsonb. Stages 2 and 3 are ours to build and are listed as such. Two transport facts stated plainly, both learned the hard way here. The deployment accepts exactly one Host header and returns 400 on everything otherwise, including health. And there is no TLS, despite the service description claiming "TLS terminated upstream by Caddy" - Caddy is inactive and its configuration is empty. Tolerable for a read-only node report on a LAN, not tolerable for the ingest path, so putting Caddy in front of the port is a Stage 3 prerequisite rather than an extra. It starts read-only for the same reason the compute substrate does: a node is online because it heartbeated, not because it was created. A mirror that begins with data cannot distinguish "the node is broken" from "the data is bad", and their speaker counts already disagree. One stale block survived a previous edit - a replace that matched nothing and reported success. It now asserts before replacing. |
||
|
umbra
|
a032e4f542 |
Record the GPU boundary as measured, on all three accounts
Fourth consecutive boot with all five nodes 0660 root:video, the timer firing, and the unit's journal showing the helper working - so the mechanism holds. What was still inference is now measurement. ollama and umbra-poc both report the RTX 3060, and umbra - which runs the controller - is refused. The device bits and the video memberships were identical to earlier boots where nvidia-smi had been run under those accounts, but identical configuration is not an observed result. Worth noting how this was reached: nvidia-device-access.sh show had been reporting 'could not check' for the two non-root accounts since the reboot, because there was no cached sudo credential. That is the honest answer, and it is why the direct run mattered - the tool had been accurately describing its own blindness while the boundary was being claimed from configuration alone. |
||
|
umbra
|
de08cfe38e |
Stop rate-limiting the deployment: any finite budget is exhaustible
The GPU permissions survived a third consecutive boot - all five nodes 660 root:video - so that is now established rather than hopeful. The deployment was down again for the same reason, and this time the widened start limit I committed two boots ago turned out not to have fixed it. It exhausted twenty restarts and returned to the same permanent failed, with the port free and nobody touching the service. That fix's verification missed it. The port was freed after three restarts, inside the window, so the test exercised the easy case and the commit said "fixed". The real case is a fault that outlasts the budget, which is the only case that matters. The rate limit is the wrong mechanism, not the wrong value. Any finite budget can be exhausted by a fault that lasts longer than it, and the failure is silent and permanent - `failed` looks like a decision rather than a give-up. StartLimitIntervalSec=0 disables it: the unit retries for as long as the fault lasts and binds the moment it clears. Verified against the case that broke it twice. Held the port for 200 seconds, which drove 31 restarts - past the old limit of 20 - with the unit never reaching failed. Freeing the port gave 200 from /api/health within eight seconds, with nothing having started it. A genuine crash loop now logs every 5s instead of stopping: noisier, and much better than a deployment that is quietly down. 469 pass with a database, 428 without. Deployment healthy, GPU boundary intact, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, persistenced up, perms timer firing, zero stray agents. |
||
|
umbra
|
1dbf5fb384 |
The GPU permissions survive a reboot, and the deployment stops giving up
Second boot since the permission fix, and it held: all five nodes 0660 root:video, the timer firing on schedule, and umbra - which runs the controller - still refused the card. That claim is now an observation rather than an inference. The same reboot surfaced an unrelated fault worth fixing. The deployment did not come up: a `python3 -m http.server` from a development session held port 8442, serving web/ - which the backend mounts itself and does not need - and the service failed five times in ten seconds. With StartLimitIntervalSec=120 and StartLimitBurst=5 that exhausted the budget, and systemd then never tried again. The deployment stayed down after the port was free and needed someone to notice, which is the second reboot in a row to end this way. A transient condition had become a permanent outage. Widened to 500s and a burst of 20, which keeps retrying across a slow boot or a settling reverse proxy while still stopping a genuine crash loop. Verified by holding the port with a squatter and watching: the unit cycles active/activating without giving up, and the instant the port is freed the next retry binds and /api/health returns 200. My first attempt at this test was void - the squatter could not bind because the service still held the port, so it proved nothing. Note that two consecutive reboots have been taken down by a background shell in this session holding the deployment's port. It returns after each boot. Stopping that shell, or moving it to another port, is the durable fix; killing it again only defers it. 469 pass with a database, 428 without. Deployment healthy, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, agent account intact, zero stray agents. |
||
|
umbra
|
4937b9beed |
Record that the GPU permissions now hold, and how that was verified
Cleared the unit's stale active(exited) state so the corrected version could run, then broke the permission deliberately the way nvidia-persistenced does and watched the timer repair it: 666 -> 660 root:video, with the helper reporting 'restricted 5 node(s)' in the journal. ollama and umbra-poc can measure the GPU; umbra, which runs the controller, cannot. Deployment healthy, ollama serving on 11434, database unchanged. Records what is still unverified: no boot has been observed end to end since the change. The unit, timer and udev hooks are installed and exercised individually, and the persistenced interaction is handled by ordering rather than luck, but 'works after a reboot' is a claim until a reboot confirms it. show answers it in one command. |
||
|
umbra
|
1ab9c2595d |
Verify the datastore policies as a role that does not own the tables
They had never been observed to refuse anything, and until now every
statement this project made about tenant isolation was a statement about
the adapter's own WHERE clauses.
The reason was structural. RLS does not apply to a table's owner until
FORCE ROW LEVEL SECURITY, and every connection here is the owner. A probe
run as the owner showed organization B's administrator successfully
updating organization A's imprint - exactly what WITH CHECK was written
to prevent. A policy that cannot refuse is indistinguishable from a policy
that is absent, which is why the previous commit said so plainly rather
than claiming the migration was verified.
test_rls_tenancy.py runs as rls_probe, a role with plain table privileges
and no ownership. Eight tests, each checked against a mutation: each
organization sees only its own artifacts, an unattached principal sees
nothing on any datastore table, a publisher is readable across tenants but
writable only by its owner, a cross-tenant UPDATE changes zero rows, and a
cross-tenant INSERT is refused. Two of the eight also assert the positive
cases, because a policy that refuses everything would satisfy every
refusal above.
Two behaviours asserted so a change is noticed: a filtered UPDATE or
DELETE raises nothing and simply affects zero rows, while an INSERT
violating WITH CHECK raises InsufficientPrivilege. The first is a trap
for application code that checks "did it raise?" instead of "did it
change a row?", and it is why the earlier manual probe reported a
cross-tenant update as "allowed" when the statement had matched nothing.
Two mistakes in the tests, both worth recording. A policy violation aborts
the transaction, so the second refusal in a test could not even be
attempted and reported as "DID NOT RAISE" when nothing had run; each
assertion now gets a savepoint. And a module-level pytest.skip() is a
collection error rather than a skip, which broke the whole suite instead
of quietly omitting the file - the guard now evaluates to a boolean.
This does not close gate G2. The application still connects as the owner,
so in the running system these policies still enforce nothing. What is
established is that they are correct, which makes FORCE ROW LEVEL SECURITY
plus a non-owner application role a configuration change with a
known-good target rather than an untested hope. README.md says exactly
that, and the gate stays open.
Creating the role needs superuser and was authorized this time:
sudo -u postgres psql -d umbratrans_scratch -c \
"CREATE ROLE rls_probe NOLOGIN; GRANT rls_probe TO umbra;"
465 pass with a database, 424 without. Deployment untouched: users=2
orgs=3 projects=0 audit=43, no compute or datastore tables.
|
||
|
umbra
|
5fc38d4708 |
The datastore, and two parties that never sign in
Compute capability is not LLM work. It is mechanical split, analysis,
text to speech, PDF formatting and audio analysis, assigned as needed,
with the code on every node. That needs a capability vocabulary rather
than VRAM - which is noted and not yet built - and, more urgently, a
place to put the output. jobs.result is a jsonb column: right for
{"echo": "hello from the queue"}, wrong for "the English audiobook, four
hours twelve minutes, derived from these thirty-seven chapters and this
cast".
So 0003 gains section 5: artifacts, job_artifacts, publishers,
project_publishers, authors, project_authors, and eight permission
keys. Folded in here rather than deferred to 0004 because 0003 has still
never been applied to the deployment, and an unapplied migration can
still be edited.
The word "account" is now absent from every new column, table and
comment. It has meant two incompatible things in this system - a person
who signs in, and a house a work is published under - and a schema
using one word for both will use it for both in one constraint. An
organization owns the work. A user is a worker and owns nothing. A
publisher is the party a work is published under and has no password, no
session and no row in user_roles, because a name that appears on a work
is not a party that can be asked to prove who it is.
A byline is a third party and is neither. In Umbra -> WorstAuthor ->
B.D. Murphy, WorstAuthor is the publisher and B.D. Murphy is the
byline; different lifetimes, and the spec had been calling both "brand".
project_authors carries a credit_role rather than a single author_id,
because the author of the English edition is the translator of the
French one and the narrator is frequently neither.
owner_organization_id is the one deliberately cross-tenant column, and
the policies are asymmetric about it: reading a publisher is open to any
authenticated principal because publishers are shared reference data,
while setting that column is refused to anyone but a platform
administrator. There is no second policy carving that out, because
PostgreSQL combines permissive policies with OR and an "admins may also"
policy widens access for everyone. The WITH CHECK on the single FOR ALL
policy is doing that job alone.
Found while testing rather than while writing: RLS was not enabled on
four of the six new tables. PostgreSQL creates a policy on a table where
it cannot apply and reports success, so pg_policies listed them and
nothing was enforced - and a check for "does the policy exist" would
have passed throughout. test_rls_coverage.py asserts relrowsecurity
instead, and each of its six assertions has been checked against a
mutation that breaks it. One needed strengthening: dropping the
dedicated SELECT policy left the table looking fully policed, because
FOR ALL includes SELECT, so the surviving policy quietly gated reading on
organization ownership and hid shared reference data.
Also: the nvidia tests failed for want of a sudo credential rather than
because anything was wrong. They skip now, with the reason stated. A
test that fails for environmental reasons teaches the reader to ignore
red.
The policies have never been observed to refuse anything, and README.md
says so plainly. A probe as the owner showed organization B's
administrator editing organization A's imprint - what WITH CHECK was
written to prevent - because RLS does not apply to a table's owner until
FORCE ROW LEVEL SECURITY. That is gate G2, not a new problem. Testing it
for real needs a non-owner role and superuser; the request was not
granted, so it is not done.
455 pass with a database, 422 without.
|
||
|
umbra audio
|
d13664a607 |
A compute substrate: node inventory, a leased queue, and an agent
Implements §6.16-6.20 and §7 of the specification, and proves it end to end on this box: a controller and an agent, the agent with no database credential at all, ten checks against the properties whose failure modes are silent. Migration 0003 adds compute_nodes, compute_devices, node_credentials, model_registry, node_models, node_heartbeats and job_events, and creates jobs - which §6.8 had specified and 0001 had never migrated. Every column added to jobs is nullable or defaulted, every policy is written against the same app.* helpers the existing eleven tables use, and re-applying is safe: CREATE POLICY has no IF NOT EXISTS, which is how 0001 learned the same lesson for ADD CONSTRAINT. Four properties, and each exists because the alternative fails silently: - **I25** claiming is FOR UPDATE SKIP LOCKED. Two agents claiming at once get different jobs. The proof of purpose counts the 'claimed' events rather than counting successes, because two agents both succeeding is not evidence. - **I26** a lease that expires returns the job to the queue, and the proof of purpose SIGKILLs the holder rather than stopping it, because a clean stop would release the lease and a clean stop is not the failure being tested. - **I24** a node is online because it heartbeated. Creation is not readiness. - **I28** a node credential reaches queue operations and nothing else. The organization comes from the credential, so a forged tenant identifier has nothing to attach to. And the one that is hardest to get right in the cheap direction: **a device that is slow is eligible, and one that is too small is not.** A model that does not fit in VRAM runs on system RAM, slowly, and a job nothing can run is 'blocked' with the reason rather than left looking like it is progressing. The proof of purpose found six defects, listed in §7.8. The two worth naming here: - Mounting one router at two prefixes published the administrator's routes behind node-credential authentication, so a node could create nodes and issue itself more authority. Two routers now, with a test that pins the surfaces disjoint and the node surface exactly. - **The correction I was given about unified memory was in the specification and not in the code.** Eligibility was VRAM-only, so every CPU and every shared-memory machine was ineligible, which would have reduced the fleet to one machine. Writing the implementation is what kept the error alive. Also: the CSRF middleware blocked the whole node API, because a bearer-authenticated agent has no ambient cookie for a cross-site request to ride on. Exempted by prefix, with a test asserting that no cookie-authenticated route joined it. 419 tests pass with the integration suite against a real database. The deployment database is untouched and has no compute tables. |
||
|
umbra audio
|
ed72f61975 |
Separate organizations, and give members their organization's projects
An organization's work is separate from every other organization's, and the separation is now one module rather than a property each route remembers. `tenancy.py` resolves the caller's scope in a single place. A member is pinned to the organization on their own assignment, expressed as a concrete identifier rather than a flag, so "no filter" is unreachable except for a platform administrator. Every read and write resolves through it, so there is no per-route variation to drift. Within an organization, every account sees every project. The brief is explicit that two people working on different books both see both, and assignment says who is *working* on a project rather than who may *know* it exists. This diverges from `app.can_access_project`, which restricts visibility to assigned members; the divergence, and what it means if row-level security is ever switched on, are recorded in migrations/README.md because that would otherwise be a silent tenant-internal regression no test here would catch. Two rules, both learned by getting them wrong first: - A read naming another organization is ignored, because refusing it would confirm the identifier exists. A write is refused rather than retargeted, because a project landing in an organization the caller did not ask for is somewhere they will not look. - A missing project and a foreign one get the same error, so an identifier cannot be probed across a boundary. Projects: the `projects` table, its statuses, and the project-scoped arm of `user_roles` were already in the schema and unused. No migration. The dashboard is one server-scoped call, `/api/workspace`, rather than several assembled in the browser — pieces from different tenants that nobody notices are not a failure mode worth shipping. Filters narrow the list already in the page, so switching cannot mix two fetches, and an empty list is distinguishable from a failed one. **A sign-in lockout, found by the integration suite.** An assignment is a second active `user_roles` row, and `_resolve_assignment` refused an account with more than one. So assigning anybody to anything made them unable to authenticate — locked out of the page the assignment was meant to show them. `_sync_assignment` had the same defect. Six queries were scope-blind; three were harmless only because the assignment role is named `viewer` and never matched a tenant role name, which is luck rather than a property, so a structural test now pins the predicate on every one. The guard against two *tenant* rows survives, narrowed to the ambiguity that actually matters. Also fixed: `get_project_by_slug` cast `slug::text`, making citext's case-insensitivity a case-*sensitive* comparison that appeared to work everywhere else. An assignment grants the `viewer` project role, since no `member` role exists at project scope; it is a named constant, not a query literal, because the project workflow is not defined yet. 410 tests pass with the integration suite against a real database; 386 without. |
||
|
umbra audio
|
e034061ec1 |
Add password reset and self-service account settings
An administrator could not reset a password for bdmurph73@gmail.com, and there was no way for a signed-in person to change their own. Both are now in place. Migration 0002 gives single-use tokens a purpose. 0001 reused the control plane's unused bootstrap_tokens for email verification, and the table has no notion of what a token is for, which was invisible while there was only one kind. With a second kind it stops being invisible: a reset link clicked by anyone holding it would confirm an address, and a verification link would set a password. The column defaults to the existing purpose, so no row is rewritten and verification is unchanged. Every token lookup now filters on purpose, and the adapter reads the purpose from the row rather than hardcoding it. An administrator may start a reset but never finish one. The link goes to the address on the account, only its holder can complete it, and the endpoint returns a bool rather than a password. An administrator who can set someone's password can also sign in as them indefinitely. A reset revokes every session, because a reset is what someone reaches for when they think their password is compromised, and revoking only the session that completed it would leave a stolen cookie working. Reset links live one hour against the two days a verification link gets. A verification link confirms an address the person already controls; a reset link is a live credential for as long as it is valid. Three bugs found while building it, all mine: - request_password_reset did not call _authorize_target, so an organization administrator could start a reset on an account in another tenant. Caught by a test written for the scoping rule that revoke_sessions already followed. - The in-memory repository never checked token expiry, only PostgreSQL did. So an expired link was honoured in every test and refused in production - the wrong direction for a divergence between two stores of the same rule. - The PostgreSQL adapter's get_live_verification_token took a purpose but not the now its in-memory counterpart needed, so the first call against a real database raised TypeError and returned a 500. The conformance test only checked that a method existed. It now compares signatures, which immediately caught that the protocol itself was missing the parameter. The reported failure was a deadlock rather than a missing feature. The account was invited: it could not sign in because the address was unverified, and I refused to reset anything not yet active. It is now allowed, and completing the reset also confirms the address, since holding the link proves control of it. Without that the flow would not converge - reset the password, sign in, refused again, using a verification link that was never sent. Suspended and deactivated accounts are still refused, because a reset restores sign-in and would otherwise undo a suspension without going through the status controls. Verified live: the account that could not be reset is now active, verified and able to sign in, with password.reset_requested, password.reset_completed and email.verified all in the audit trail. A reset token is refused by the verification endpoint, an invented token gives the same answer as a used one, and a token in a reset URL is redacted from the access log. 327 tests pass, 337 with the integration suite. |
||
|
umbra audio
|
64af39eb68 |
Run the PostgreSQL integration suite against the real schema
The integration suite had never been executed. Running it against a PostgreSQL 16 instance carrying an exact copy of the control-plane schema found seven real defects, all of which were invisible without a database. Migration: - `ADD CONSTRAINT ... UNIQUE USING INDEX` failed when the index was already attached, for both `organizations_slug_key` and `users_username_key`. Because the file runs in one transaction, that error rolled back every other change, leaving a half-applied schema rather than an obviously untouched one. Both are now guarded, so the file applies cleanly to the real schema and is idempotent on re-apply. The adapter: - `get_user_by_email` was never implemented. The protocol stub returned `None`, so email login found no account against a real database while working perfectly against the in-memory one. - The four verification-token methods were never implemented either, so email verification links never worked against PostgreSQL. - The RLS identity GUC was wrong. `app.current_user_id()` reads `app.user_id`, not `app.current_user_id`; setting the similarly named GUC left every policy helper resolving to NULL, so every policy denied and every query returned zero rows. Added `verify_rls_contract()`, called at startup, which proves the two agree. It has teeth: pointed at a wrong GUC it refuses. - `SET` takes no bind parameters, so the statement timeout was a syntax error on every transaction. The identity GUC now uses `set_config`, which is parameterisable and keeps the value bound. - `append_audit_event` returned only `id` and then mapped the whole row, and passed a text `target_id` where a `UUID` was required. The domain: - A member with no organization is storable in the development JSON store and rejected by `user_roles_check`, so the same call succeeded in development and failed in production. Enforced in the service now, and 43 tests were relying on the JSON store's leniency. - `create_account` accepted `require_email_verification` and silently dropped it on the operator path, returning an unverified account to a caller that had asked for a verified one. Now refused, with the reason. - All 26 protocol stubs returned `None` instead of raising, so a missing method was indistinguishable from an empty database. They now raise, and a conformance test checks each repository implements every declared method rather than merely inheriting it. The old conformance test used `hasattr`, which inheritance always satisfies; it passed while five methods were missing. - The bootstrap CLI refused an unsatisfiable role only after the operator had typed a one-time password. The migration README now records which review questions are answered and which remain, including that RLS is enabled but not forced, so the adapter's own scoping is currently the only tenant control and the isolation tests below it are still outstanding. 255 pass without a database, 263 against one. |
||
|
umbra audio
|
31fd8720ec |
Add the PostgreSQL adapter for the authoritative store
Implements AccountRepository against the control-plane schema plus the phase-1 identity migration, and wires it in so production can select it. The adapter is not a thin SQL wrapper, because the control plane does not store the two things the application needs on the users row. Role and organization come from user_roles. There is no role column and no organization column on users; a role is assigned through user_roles with a scope, and app.validate_user_role_scope() enforces that the role's scope matches the assignment. Resolving one role per user is therefore a join, and a user holding more than one active assignment is refused rather than resolved to an arbitrary row. The organization on that assignment becomes the scope of every administrative decision, so guessing would hand out the wrong tenant's authority. Reconciling this with project-scoped membership stays an open review question, recorded as such. Email became a required field on account creation. users.email is NOT NULL UNIQUE, so accepting a missing address would let account creation succeed against the development repositories and then fail on the authoritative store. It is now required in the service, the API schema, and the bootstrap command, so it fails identically everywhere. Audit identifiers now belong to the store. audit_events.id is a bigint identity, so the adapter reports that value; the application correlation UUID is preserved in the detail and never displaces the row key. AuditEvent.id is a string for that reason, which removes a type the two stores disagreed about. Three further fail-closed behaviours: a users.status of invited or locked is refused rather than coerced onto active; a row with mfa_required is refused, because this phase cannot satisfy a second factor and authenticating it anyway would grant access to someone who was supposed to present one; and a missing organization blocks access rather than granting it. save_user deliberately does not write username or email, since both are identity keys with no reviewed change path and no audit event behind them. Every operation runs in one transaction on one pooled connection and publishes the caller with SET LOCAL app.current_user_id, so a returned connection cannot carry one request's identity into the next. That is inert until the reviewed RLS policies exist; the explicit WHERE clauses enforce scope today. A statement timeout bounds how long a pooled connection can be held. Production fails closed at two independent gates: the settings layer refuses without a DSN, and the repository boundary refuses to build a PostgreSQL store without one. Verified by running both, including that an unreachable host refuses to start rather than degrading. The migration gains an index on audit_events (target_type, target_id), which the administrator audit filter needs and nothing covered, and the review questions now include the three things that must be checked before provisioning: the roles rows and their scopes, any row with mfa_required set, and any user holding more than one active assignment. Verification, and its limit. 236 tests pass with 3 skipped. The adapter's row mapping, role resolution, conflict mapping, and configuration are unit tested and run everywhere. Its SQL, transactions, and pooled reuse are NOT executed: this environment has no database credentials, because sudo to the postgres user needs a password, there is no umbra database role, and there is no .pgpass. The integration suite is written and selected by UMBRATRANS_TEST_DSN, and its fixture refuses any DSN whose database name contains umbracontrol. A green run here says nothing about whether the SQL is correct. |
||
|
umbra audio
|
0b9c4d4220 |
Add organization tenancy and same-origin frontend mounting
The administrator interface needs organizations to manage, and the slice had none: users.organization_id was a bare UUID pointing at nothing, so there was no tenant to create, list, or validate an account against. Tenancy: - Organization record with name, slug, and status active|suspended|archived, matching the control-plane vocabulary rather than reusing the account one. - Reads are open to any administrator; every write is platform-admin only, so the tenant boundary stays under a single authority. An organization admin is pinned to the organization on their own principal. - A derived slug is normalized from the display name, but a slug the operator typed must already be canonical, so nothing is silently rewritten behind their back. archived is terminal. - GET /api/uadmin/summary for the dashboard, recomputed per request from the same scoped queries the list endpoints use, so a summary cannot disclose a number the caller could not reach by listing. Two integrity gaps closed: 1. Account creation accepted any UUID as an organization. It now requires the organization to exist and to be active. 2. Suspending an organization revoked member sessions but did not stop new logins, because each member's own account status stays active. Both authenticate and resolve_session now refuse a member of a non-active organization, and a missing organization record blocks access rather than granting it. A test asserts the member is still active yet locked out. Migration, still unapplied: section 4 creates organizations with CREATE TABLE IF NOT EXISTS so a control-plane table is skipped, never redefined, and the rollback drops it only when this migration actually created it. The membership divergence is recorded as the top open review question rather than decided: user_roles is the control plane's membership source, this phase uses a column, and the migration adds no membership column so the choice stays open. Serving: UMBRATRANS_STATIC_ROOT mounts the browser client at / after the API routes, so the session cookie and CSRF header work without CORS. HTML responses get their own CSP allowing only same-origin script and style, which is why the frontend carries no inline script or style. Verification: 204 tests pass (21 new organization tests plus the existing suite), ruff check and format clean. Also exercised against a real uvicorn server rather than only the test client: bootstrap, forced password change, organization creation, one-time password issue, member lockout on organization suspension, and rejection of an account assigned to a suspended tenant. |
||
|
umbra audio
|
b474995414 |
Add phase-1 account management backend slice
Implements the account and user-management system behind the new web UI against development repositories only. Nothing here connects to, reads, or modifies the production umbracontrol database, and no account is created in any approved environment. Account service and API: - Local temporary-password and OIDC enrollment, separated by type. - Argon2id credentials, must-change enforcement, server-side sessions with hashed tokens, idle and absolute deadlines, rotation and revocation. - Administrator account management: list, get, create, suspend, reactivate, deactivate, revoke sessions, org-scoped audit view. - Hidden-prompt operator command for the first administrator, refusing production mode and refusing to run without a TTY. Temporary passwords are never accepted from a client. The request schema has no password field and refuses unknown fields; the administrator path generates the one-time password in process, hashes it immediately, and returns it once in the creation response. The account can reach no route, including /api/auth/me, until it replaces that password, after which the one-time value no longer authenticates. Security posture: fail-closed configuration, Secure cookies in production, double-submit CSRF, constant-time token comparison, generic authentication errors, audit detail allow-listing, and logger-tree secret redaction. Reviewed but unapplied: migrations/0001_phase1_identity.sql adds users.username, a separate user_credentials table, and additive session columns, with a rollback and open review questions. The PostgreSQL adapter is deliberately absent, so the application refuses to start in production rather than run against an unreviewed store. Verification: 169 tests pass (148 new, 21 pre-existing); ruff check, ruff format, compileall, and make web-check are clean. |