dda079f60580f1681e4bf89b84ba35f090794a8f
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
umbra audio
|
dda079f605 |
commit audio ingest: apply 0008+0009 migrations, wire Postgres adapter to commit path
- Apply migration 0008_audio_source_import (additive schema for source import) - Apply migration 0009_translation_import_runs (ledger table for import runs) - Wire PostgresAudioImportStore into HTTP commit path with org from node credential - Commit path uses PG store with org from node, no payload org - Bundle hash 21ce5687… remains frozen until v2; no authority transferred |
||
|
umbra audio
|
e35d7a144f | Add Apache-2.0 LICENSE, NOTICE, and SPDX headers | ||
|
umbra audio
|
bc031dd079 |
Bootstrap: one file, and it works with or without root
Enrolling a node meant a person reading instructions and running them, with a credential typed into a terminal. That does not scale, and it makes the controller a courier. The flow is now: an administrator renders a script, hands over one curl, and the node sets itself up. `GET /api/node/agent` serves the agent to a credentialed node. A node credential already implies "run what the controller sends" - that is what claiming a job is - so this adds no authority; it removes the hand-copying. `POST /api/compute/pairing-script` renders the bootstrap, minting a pairing code server-side so a code pasted into a request body cannot be one issued earlier for something else. ## The exclusive/shared distinction, encoded The script asks whether it has root and branches. With root: `/opt/umbra`, a system unit, and the agent running as a dedicated `umbra-node` account with NoNewPrivileges and ProtectSystem=strict. Without: the user's own directories and a user unit. Both layouts are in the output and the runtime decides, because whoever renders the script does not always know how the person at the machine will run it. The credential goes in a 0600 file referenced by systemd's EnvironmentFile. Three alternatives were rejected and the reasons are in the code: a unit file is world-readable, a config holding a secret ends up in a backup, and the agent deliberately refuses to read the key from its config when the environment can carry it. ## Verified on node 238, not just reasoned about Ran it as `umbra` with no root: paired, installed, registered, and the agent went `online`. Six defects found only by running it: - `$BOOTSTRAP_VERSION` was interpolated into the shell body but never declared as a shell variable. `set -u` caught it on the first line that used it. - The agent fetch sent no credential and got a 401 - correct behaviour, wrong script. - `ExecStart` pointed at the agent file directly. It has no shebang, so the kernel tried to run it as a binary: "Exec format error", a service in a restart loop, and nothing on the controller. Now invoked through `python3` explicitly, and the file is 644 rather than 755 because it is source. - `sed -i` with no backup suffix is GNU-only; BSD reads the next argument as a suffix, so on macOS it would leave the unit unfiltered and install hardening directives that cannot work. Replaced with a portable filter. - Pairing a name already in use surfaced as a bare 500 from a UniqueViolation. Now `node_name_taken`, with an answer that says pairing does not replace an existing node. - The agent logged "claim failed: node_scope_required" every second, forever. A read-only credential can never gain compute:claim, so this is permanent, and a warning that cannot change trains a reader to ignore warnings. Announced once. ## A privilege bug the pinned tests caught The bootstrap renderer declared an administrator dependency while sitting on the *node* router. A node could have called it to mint itself a credential for any node name. It is now `POST /api/compute/pairing-script` on the admin router, above the agent-surface marker so `test_every_node_route_uses_node_credential_authentication` keeps meaning something. Also worth noting: a node cannot be DELETEd from the database, because the pairing-code reuse trigger refuses the `ON DELETE SET NULL` cascade. That is correct - the status enum has `retired` and the admin route drains and retires, so removal is a status change, not a row deletion. Found by trying to clean up after a failed test run. 525 passed, 27 skipped. Test nodes boot-test-1..3 are retired, not deleted. |
||
|
umbra
|
c1741456e2 |
Apply the GPU boundary, and fix three checks that verified themselves
The udev rule I wrote last commit could not have worked. The nvidia driver creates /dev/nvidia* in the kernel without registering them in sysfs, so `udevadm info --name=/dev/nvidiactl` answers "Unknown device" and udev never evaluates a rule for them - which is also why they are 0666, since nothing sets a MODE. The rule hooks the driver bind event instead and runs a script, by which point the nodes exist. The first version of that hook triggered 0000:01:00.0, which on this machine is an AMD root complex, vendor 0x1022 rather than 0x10de. The trigger fired on the wrong device, the RUN key never ran, and the permissions looked right because the apply script had just fixed them itself. The address is discovered now. Verified by resetting to 0666, firing only the bind event, and watching it come back 0660 - the script never invoked. Applied and verified: ollama can open the card, umbra-poc can, umbra cannot. Traced, nvidia-smi opens /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm, so umbra-poc is in video or the agent starts, heartbeats, and reports a machine with no GPU - a failure that presents as a scheduling problem and is a permissions one. It is the agent's only supplementary group and grants a device, not a filesystem, so setup's "no supplementary groups" check now allows exactly video and rejects everything else, verified by adding users and watching it fail. Three tests, each verified against a mutation. The rule test resets the permissions and fires only the event, so it fails if the rule is inert. The end-state test fails rather than skips when the rule is missing - skipping made deleting the rule quieter, three passed becoming one passed and two skips, which reads as an improvement in the wrong direction. Also: the check "the controller cannot open the card" was testing root, which bypasses permissions entirely, and reported a working rule as a failure. The account is named now. Then four runs in a row leaked agents, and the cause was upstream of the leak. The victim reused the first agent's credential, and a node is identified by its credential, not by its config's node_name - so it heartbeated as poc-agent-node, was never offered the long job, and sat there for the whole step while a correct liveness check reported that the worker it had killed had survived. It has a third node and its own credential now, so the requeued job is not immediately reclaimed by an agent already stopped for the step. The last leak was a worker asleep in a 600-second job, unable to see a stop file because the check is at the top of its loop rather than inside the work. Four runs left one each. The job is 10 seconds now: the victim exits the instant it claims it and never runs it, so the length was never the point. PoC node names and idempotency keys carry a per-run suffix, so a second run against the same database does not die on a UniqueViolation - and there is no route to delete a node. Three consecutive runs, zero failures, zero leaks. 451 pass with a database, 424 without. Deployment untouched: users=2 orgs=3 projects=0 audit=43, no compute tables. |
||
|
umbra
|
0f5c1e3408 |
Run the agent as the service account it claims to be
The compute proof of concept had never been run with the agent as umbra-poc. Running it that way found three defects, all the same shape: a check that passed because its mechanism did nothing. The demonstration never reached the agent at all. It held the pid it spawned, which under `sudo -n -u` is the sudo wrapper - sudo forks the agent before exec'ing. Signalling that killed the wrapper and left the worker running, renewing its lease and holding the job indefinitely, under a printed note saying "exactly like a power cut". An unprivileged process cannot signal another account's process at all, and kill(2) on a process group succeeds as soon as one member is signalled, so the wrapper's death read as the agent's. Every run leaked an agent. Lease recovery was a diagnosis rather than a mechanism. sweep_expired_leases documented itself as "called by the controller's own timer" and there was no timer; a killed node's job came back only when somebody noticed a stalled job and posted to an administrative route by hand. The proof of concept swept by hand and recorded the result as automatic. app.py now runs the loop for the life of the application, building the compute repository at startup so a restart mid-job still recovers it. The power cut is now the agent's own: os._exit(137) the instant it claims a job, before running it, because a process that does not own the agent cannot signal it. A clean stop goes through a stop_file the agent polls, for the same reason. Liveness is measured by finding the worker's own pid in /proc matched on its config path, sampled repeatedly and accumulated - the wrapper and the worker are not simultaneous, so a single sample finds only the wrapper. The node key travels by `sudo --preserve-env` rather than as VAR=value in argv, because argv is world-readable through /proc/<pid>/cmdline and the environment is not. The demonstration checks whether the key will survive before starting and falls back to a pipe, which node_agent.py --node-key-stdin accepts. Neither a file nor argv ever holds it. Eight tests, each verified to fail against a mutation of the code it covers: three for the sweeper (it must return an expired lease, must survive a failing round, must not invent a repository it has no database for) and five driving a real agent process against a stub controller (stops on the stop file, exits 137 holding an uncompleted job, takes the key from both routes and neither, reports an empty pipe as an empty pipe). 448 pass with a database, 421 without. The deployment database is untouched: users=2 orgs=3 projects=0 audit=43, no compute tables. |
||
|
umbra audio
|
d13664a607 |
A compute substrate: node inventory, a leased queue, and an agent
Implements §6.16-6.20 and §7 of the specification, and proves it end to end on this box: a controller and an agent, the agent with no database credential at all, ten checks against the properties whose failure modes are silent. Migration 0003 adds compute_nodes, compute_devices, node_credentials, model_registry, node_models, node_heartbeats and job_events, and creates jobs - which §6.8 had specified and 0001 had never migrated. Every column added to jobs is nullable or defaulted, every policy is written against the same app.* helpers the existing eleven tables use, and re-applying is safe: CREATE POLICY has no IF NOT EXISTS, which is how 0001 learned the same lesson for ADD CONSTRAINT. Four properties, and each exists because the alternative fails silently: - **I25** claiming is FOR UPDATE SKIP LOCKED. Two agents claiming at once get different jobs. The proof of purpose counts the 'claimed' events rather than counting successes, because two agents both succeeding is not evidence. - **I26** a lease that expires returns the job to the queue, and the proof of purpose SIGKILLs the holder rather than stopping it, because a clean stop would release the lease and a clean stop is not the failure being tested. - **I24** a node is online because it heartbeated. Creation is not readiness. - **I28** a node credential reaches queue operations and nothing else. The organization comes from the credential, so a forged tenant identifier has nothing to attach to. And the one that is hardest to get right in the cheap direction: **a device that is slow is eligible, and one that is too small is not.** A model that does not fit in VRAM runs on system RAM, slowly, and a job nothing can run is 'blocked' with the reason rather than left looking like it is progressing. The proof of purpose found six defects, listed in §7.8. The two worth naming here: - Mounting one router at two prefixes published the administrator's routes behind node-credential authentication, so a node could create nodes and issue itself more authority. Two routers now, with a test that pins the surfaces disjoint and the node surface exactly. - **The correction I was given about unified memory was in the specification and not in the code.** Eligibility was VRAM-only, so every CPU and every shared-memory machine was ineligible, which would have reduced the fleet to one machine. Writing the implementation is what kept the error alive. Also: the CSRF middleware blocked the whole node API, because a bearer-authenticated agent has no ambient cookie for a cross-site request to ride on. Exempted by prefix, with a test asserting that no cookie-authenticated route joined it. 419 tests pass with the integration suite against a real database. The deployment database is untouched and has no compute tables. |