master
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
umbra audio
|
e35d7a144f | Add Apache-2.0 LICENSE, NOTICE, and SPDX headers | ||
|
umbra audio
|
382ca8fc78 |
Derive node status on read; a heartbeat loop for the audio node
Two problems, both found by the audio node running Stage 1.
## A node that went quiet still looked available
`compute_nodes.status` is promoted to `online` by a heartbeat and demoted to
`offline` by an administrator. Nothing demoted it on the passage of time: grep found
`offline` appearing in exactly one place, the heartbeat's promotion CASE. So an agent
that was killed left the row reading `online` indefinitely - measured on the
deployment, where `umbraaudio-m4` showed `online` against a last heartbeat nine hours
old.
That is the failure invariant I24 exists to prevent. `create_node` refuses to mark a
node ready before its agent has ever spoken, and then nothing noticed when it stopped
speaking.
Derived on read instead, in `effective_status`, called from `_node_from_row` so every
caller - console, heartbeat response, placement - gets the same answer:
offline: no recent heartbeat (derived; the stored status is still 'online')
Three decisions inside it:
- Only `online` is derived away. `draining` and `retired` are administrator
decisions about intent, and a machine going quiet does not retract an order to stop
taking work. Demoting a draining node would make the console report it as gone
rather than winding down, and blame the wrong thing.
- Nothing is written back. The stored value stays as-is and is carried on the model as
`stored_status`, so a GET has no side effect and two readers cannot disagree about
what the column says. `NodeResponse` reports it only when it differs from `status`.
- The stale-window check that was hand-rolled in the placement loop is gone. Two copies
of one rule is how they drift; the loop now reads the same function as everything
else, which is what makes the console and the scheduler agree.
A sweeper was the alternative and is how expired leases are handled, so it was not a
strange idea. It loses because it is a periodic task that must actually run on a
deployment that has already been taken down three boots by a stray process - and while
it is not running, the stored value is wrong again. Computing on read has no window and
no background dependency.
## The document was wrong about what happens when a node stops
It said "miss that window and the node goes offline". That was false, which is how this
was found. The audio node followed the instruction, sent one heartbeat, and nothing
told it or anybody that a second one was expected.
`poc/umbra_stage1.py` is a loop rather than a description of one, so the audio node can
drop it in unchanged. One `curl` is one request: it returns to the prompt when done,
which is correct for curl and indistinguishable from a healthy node that never started.
The script runs in the foreground so closing the terminal stops it, which makes the
first few minutes obvious - press Ctrl-C and the node goes quiet, and about 45 seconds
later the controller says so. It exits on 401 and 403 rather than retrying once every
ten seconds for an hour, because a credential problem does not resolve itself, and it
sends an empty report rather than nothing when the report script is missing, since a
node that stops heartbeating because its report broke is indistinguishable from a dead
machine.
Verified against the live controller: three consecutive 200s at a 10s interval, and a
wrong key producing 401 and a clean exit.
Verified on the deployment: a node quiet for 50s reads `offline` while the column still
says `online`, and reads `online` again one second after a heartbeat.
488 passed, 27 skipped. The three staleness tests were confirmed to fail with the
derivation neutered, and the export test caught `effective_status` missing from
`__all__`.
|
||
|
umbra
|
0ab1f0f1de |
The GPU boundary did not survive a reboot, and neither did the fix for it
This is the verification I said could not be done until the machine rebooted. The reboot proved the boot path wrong, and uncovering why took three rounds. **The boundary was gone.** /dev/nvidiactl came back 0666 root:root, so the controller could open the card again - the exact thing the udev rule was applied to prevent. **Why: the unit ran before anything existed.** umbra-nvidia-devices ran at 05:38:24, ordered After=systemd-udev-settle, globbed /dev, found no nodes, and exited 0. udev-settle means "udev has finished its queue", not "the proprietary driver has finished initialising". The nodes appeared moments later at 0666 and nothing revisited them. The helper ended in an unconditional `exit 0` - I had written that deliberately, so an early boot would not fail a unit - and it is precisely what made "I fixed the permissions" indistinguishable from "there were nothing to fix". **Why the first repair also failed: nvidia-persistenced resets the mode.** It chmods /dev/nvidiactl back to 0666 on every initialisation, via nvidia-modprobe. That is out of udev's reach entirely, because udev does not manage these nodes. So the fix is no longer an ordering problem but an eventual one: the unit now runs After=nvidia-persistenced.service, and a two-minute timer re-asserts the permissions because the trigger that breaks them can fire at any time. **Why the timer did nothing either.** RemainAfterExit=yes on the oneshot keeps it "active (exited)", so systemd treated every activation as already satisfied: the timer fired on schedule, the service reported Result=success and ExecMainStatus=0, and the permission was still wrong two minutes later. That is the original bug repeated one level out, inside the mechanism built to repair it. Removed, and a test now pins it - by directive name, with comments stripped, since the unit documents the omission by naming it and a substring search failed on the explanation. **I also broke nvidia-persistenced.** It drops to uid 122 with setuid and without initgroups(), so it never receives the video supplementary group and failed with "Insufficient Permissions" once the nodes were 0660 root:video. A systemd drop-in runs it as root - upstream's own default - which works without widening the device permissions. It was running fine before. The helper now waits for the nodes, verifies the modes afterwards, and exits non-zero under --strict rather than reporting a no-op as success. It takes UMBRA_NVIDIA_DEVDIR so its three paths can be tested without root and without a GPU; the first attempt at that test pointed CANARY at a non-existent path and still exited 0, because the fix loop globbed /dev directly and the override only reached the wait. Also: nvidia-device-access.sh `show` reported "cannot measure the GPU" when a cached sudo credential had simply expired - a finding about the machine when the truth was that the tool could not ask. It now distinguishes no from could-not-check, because I was misled by it during this check. **Still outstanding, and I could not finish it.** The repository is correct; the installed state is stale, because clearing the unit's active (exited) state needs root and that authorization was declined. As of now /dev/nvidiactl is 0666 and the controller can open it. To finish: systemctl stop umbra-nvidia-devices.service systemctl start umbra-nvidia-devices.service stat -c '%a %U:%G %n' /dev/nvidiactl and expect 660 root:video. Otherwise the system came through the reboot intact: the deployment was down only because a leftover `python3 -m http.server` from an earlier session was holding port 8442, and the backend mounts the same files itself. Deployment healthy, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, agent account intact, zero stray agents. 471 pass with a database, 430 without. |
||
|
umbra
|
c1741456e2 |
Apply the GPU boundary, and fix three checks that verified themselves
The udev rule I wrote last commit could not have worked. The nvidia driver creates /dev/nvidia* in the kernel without registering them in sysfs, so `udevadm info --name=/dev/nvidiactl` answers "Unknown device" and udev never evaluates a rule for them - which is also why they are 0666, since nothing sets a MODE. The rule hooks the driver bind event instead and runs a script, by which point the nodes exist. The first version of that hook triggered 0000:01:00.0, which on this machine is an AMD root complex, vendor 0x1022 rather than 0x10de. The trigger fired on the wrong device, the RUN key never ran, and the permissions looked right because the apply script had just fixed them itself. The address is discovered now. Verified by resetting to 0666, firing only the bind event, and watching it come back 0660 - the script never invoked. Applied and verified: ollama can open the card, umbra-poc can, umbra cannot. Traced, nvidia-smi opens /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm, so umbra-poc is in video or the agent starts, heartbeats, and reports a machine with no GPU - a failure that presents as a scheduling problem and is a permissions one. It is the agent's only supplementary group and grants a device, not a filesystem, so setup's "no supplementary groups" check now allows exactly video and rejects everything else, verified by adding users and watching it fail. Three tests, each verified against a mutation. The rule test resets the permissions and fires only the event, so it fails if the rule is inert. The end-state test fails rather than skips when the rule is missing - skipping made deleting the rule quieter, three passed becoming one passed and two skips, which reads as an improvement in the wrong direction. Also: the check "the controller cannot open the card" was testing root, which bypasses permissions entirely, and reported a working rule as a failure. The account is named now. Then four runs in a row leaked agents, and the cause was upstream of the leak. The victim reused the first agent's credential, and a node is identified by its credential, not by its config's node_name - so it heartbeated as poc-agent-node, was never offered the long job, and sat there for the whole step while a correct liveness check reported that the worker it had killed had survived. It has a third node and its own credential now, so the requeued job is not immediately reclaimed by an agent already stopped for the step. The last leak was a worker asleep in a 600-second job, unable to see a stop file because the check is at the top of its loop rather than inside the work. Four runs left one each. The job is 10 seconds now: the victim exits the instant it claims it and never runs it, so the length was never the point. PoC node names and idempotency keys carry a per-run suffix, so a second run against the same database does not die on a UniqueViolation - and there is no route to delete a node. Three consecutive runs, zero failures, zero leaks. 451 pass with a database, 424 without. Deployment untouched: users=2 orgs=3 projects=0 audit=43, no compute tables. |
||
|
umbra
|
220bc00823 |
An upgrade path for the agent, and the GPU device boundary
Two questions worth answering properly rather than by assertion. How an agent is updated. `setup` has always doubled as an upgrade - it installs the file unconditionally, which is most of what an upgrade needs and the one thing it should not do: a broken file replaces a working agent on every node with no way back. `update` names the version being replaced, keeps the previous file, and verifies that the new agent *starts as umbra-poc* rather than that it parses. A file that imports cleanly under umbra's virtualenv may not be readable by the account that runs it. Verified against a deliberately broken agent: installed, detected, previous restored. The agent does not update itself. An agent that can fetch and execute code has a remote-code-execution channel, and the point of the account is that it holds no credentials and reaches nothing. Updates are an out-of-band deploy per node. Nothing enforces a minimum agent version either - the console can report "behind", but refusing a heartbeat would strand a node mid-job for tidiness. The GPU. /dev/nvidia* is 0666 root:root because the driver creates it that way and its own 71-nvidia.rules sets no MODE or GROUP. Measured, not assumed: nvidia-smi -L succeeds for umbra, ollama and umbra-poc alike - so the controller can open the card today, and the only thing stopping it from loading a model on top of the node's work is the scheduler choosing not to. 72-umbra-nvidia.rules sets 0660 root:video, sorting after the driver's rule so it wins. video already contains ollama, which is right: ollama is the account that opens the GPU. umbra-poc must be added to it, and traced nvidia-smi opens /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm - so without the membership the agent starts, heartbeats, and reports a machine with no GPU. That fails silently, as a scheduling problem rather than a permissions one. It is the agent's only supplementary group and grants a device, not a filesystem, so it is added by the apply script rather than by setup. What it does not buy: separation between two agents. Both are umbra-poc, both in video, both could open the card. On one box with one GPU, placement is the control and the kernel is not. What it buys is that every other account stops being able to touch the card, the controller above all. apply verifies the rule took, that ollama and the agent can still measure the GPU, and that ollama still serves models - and reverts itself if any of those fail. Also fixes a check this session broke: a sed meant for new code rewrote verify's agent-start line to run a command called `x`, so it had been failing on every run and I read past it. `bash -n` cannot see that class of damage; verify did, which is the argument for it. |
||
|
umbra
|
0f5c1e3408 |
Run the agent as the service account it claims to be
The compute proof of concept had never been run with the agent as umbra-poc. Running it that way found three defects, all the same shape: a check that passed because its mechanism did nothing. The demonstration never reached the agent at all. It held the pid it spawned, which under `sudo -n -u` is the sudo wrapper - sudo forks the agent before exec'ing. Signalling that killed the wrapper and left the worker running, renewing its lease and holding the job indefinitely, under a printed note saying "exactly like a power cut". An unprivileged process cannot signal another account's process at all, and kill(2) on a process group succeeds as soon as one member is signalled, so the wrapper's death read as the agent's. Every run leaked an agent. Lease recovery was a diagnosis rather than a mechanism. sweep_expired_leases documented itself as "called by the controller's own timer" and there was no timer; a killed node's job came back only when somebody noticed a stalled job and posted to an administrative route by hand. The proof of concept swept by hand and recorded the result as automatic. app.py now runs the loop for the life of the application, building the compute repository at startup so a restart mid-job still recovers it. The power cut is now the agent's own: os._exit(137) the instant it claims a job, before running it, because a process that does not own the agent cannot signal it. A clean stop goes through a stop_file the agent polls, for the same reason. Liveness is measured by finding the worker's own pid in /proc matched on its config path, sampled repeatedly and accumulated - the wrapper and the worker are not simultaneous, so a single sample finds only the wrapper. The node key travels by `sudo --preserve-env` rather than as VAR=value in argv, because argv is world-readable through /proc/<pid>/cmdline and the environment is not. The demonstration checks whether the key will survive before starting and falls back to a pipe, which node_agent.py --node-key-stdin accepts. Neither a file nor argv ever holds it. Eight tests, each verified to fail against a mutation of the code it covers: three for the sweeper (it must return an expired lease, must survive a failing round, must not invent a repository it has no database for) and five driving a real agent process against a stub controller (stops on the stop file, exits 137 holding an uncompleted job, takes the key from both routes and neither, reports an empty pipe as an empty pipe). 448 pass with a database, 421 without. The deployment database is untouched: users=2 orgs=3 projects=0 audit=43, no compute tables. |
||
|
umbra audio
|
1ab0fa2498 |
A service-account setup and boundary check for the node agent
The agent is stdlib-only - no package imports, no third-party dependencies - so it needs no virtualenv and no access to the controller's home directory. That is the whole reason the setup is short: an account with no group memberships at all, running one file from /opt on the system python. poc/poc_node_user.sh has four subcommands. setup needs root and creates the account, installs the agent, and writes a config that deliberately contains no key. verify asserts the boundary. run does both. status reports what exists and what is reachable, needing no privilege at all. The check that matters: the agent must not be able to read ~/.config/umbratrans/umbratrans.env. It cannot, and there is a second layer behind it - peer authentication maps a unix user to a database role, and umbra-poc has none, so even a leaked DSN would not connect. The two are independent and neither alone is the boundary. Writing verify produced the third instance of a bug class that has come up repeatedly in this work: a check that passes because its mechanism failed. When sudo -u could not run, every positive check correctly failed and every negative check passed - because `test -r` on an unreadable file returns false, which is exactly what "correctly denied" looks like. The script would have reported a boundary as holding while testing nothing. It now confirms the privilege drop once, up front, and abandons the run rather than reporting; status prints "unknown (no privilege drop)" rather than "no". Found by running the checks against an account it could not drop to, which is the only way to see it. Also: the agent's config now travels in a 0750 directory, because mkdtemp gives 0700 and a different user could not have read its own config out of it. The script is dry-run against an absent account, an existing unprivileged account, a missing controller, and a rootless setup invocation. It has not been run end to end, because umbra-poc does not exist yet. |
||
|
umbra audio
|
d13664a607 |
A compute substrate: node inventory, a leased queue, and an agent
Implements §6.16-6.20 and §7 of the specification, and proves it end to end on this box: a controller and an agent, the agent with no database credential at all, ten checks against the properties whose failure modes are silent. Migration 0003 adds compute_nodes, compute_devices, node_credentials, model_registry, node_models, node_heartbeats and job_events, and creates jobs - which §6.8 had specified and 0001 had never migrated. Every column added to jobs is nullable or defaulted, every policy is written against the same app.* helpers the existing eleven tables use, and re-applying is safe: CREATE POLICY has no IF NOT EXISTS, which is how 0001 learned the same lesson for ADD CONSTRAINT. Four properties, and each exists because the alternative fails silently: - **I25** claiming is FOR UPDATE SKIP LOCKED. Two agents claiming at once get different jobs. The proof of purpose counts the 'claimed' events rather than counting successes, because two agents both succeeding is not evidence. - **I26** a lease that expires returns the job to the queue, and the proof of purpose SIGKILLs the holder rather than stopping it, because a clean stop would release the lease and a clean stop is not the failure being tested. - **I24** a node is online because it heartbeated. Creation is not readiness. - **I28** a node credential reaches queue operations and nothing else. The organization comes from the credential, so a forged tenant identifier has nothing to attach to. And the one that is hardest to get right in the cheap direction: **a device that is slow is eligible, and one that is too small is not.** A model that does not fit in VRAM runs on system RAM, slowly, and a job nothing can run is 'blocked' with the reason rather than left looking like it is progressing. The proof of purpose found six defects, listed in §7.8. The two worth naming here: - Mounting one router at two prefixes published the administrator's routes behind node-credential authentication, so a node could create nodes and issue itself more authority. Two routers now, with a test that pins the surfaces disjoint and the node surface exactly. - **The correction I was given about unified memory was in the specification and not in the code.** Eligibility was VRAM-only, so every CPU and every shared-memory machine was ineligible, which would have reduced the fleet to one machine. Writing the implementation is what kept the error alive. Also: the CSRF middleware blocked the whole node API, because a bearer-authenticated agent has no ambient cookie for a cross-site request to ride on. Exempted by prefix, with a test asserting that no cookie-authenticated route joined it. 419 tests pass with the integration suite against a real database. The deployment database is untouched and has no compute tables. |