0ab1f0f1de
This is the verification I said could not be done until the machine rebooted. The reboot proved the boot path wrong, and uncovering why took three rounds. **The boundary was gone.** /dev/nvidiactl came back 0666 root:root, so the controller could open the card again - the exact thing the udev rule was applied to prevent. **Why: the unit ran before anything existed.** umbra-nvidia-devices ran at 05:38:24, ordered After=systemd-udev-settle, globbed /dev, found no nodes, and exited 0. udev-settle means "udev has finished its queue", not "the proprietary driver has finished initialising". The nodes appeared moments later at 0666 and nothing revisited them. The helper ended in an unconditional `exit 0` - I had written that deliberately, so an early boot would not fail a unit - and it is precisely what made "I fixed the permissions" indistinguishable from "there were nothing to fix". **Why the first repair also failed: nvidia-persistenced resets the mode.** It chmods /dev/nvidiactl back to 0666 on every initialisation, via nvidia-modprobe. That is out of udev's reach entirely, because udev does not manage these nodes. So the fix is no longer an ordering problem but an eventual one: the unit now runs After=nvidia-persistenced.service, and a two-minute timer re-asserts the permissions because the trigger that breaks them can fire at any time. **Why the timer did nothing either.** RemainAfterExit=yes on the oneshot keeps it "active (exited)", so systemd treated every activation as already satisfied: the timer fired on schedule, the service reported Result=success and ExecMainStatus=0, and the permission was still wrong two minutes later. That is the original bug repeated one level out, inside the mechanism built to repair it. Removed, and a test now pins it - by directive name, with comments stripped, since the unit documents the omission by naming it and a substring search failed on the explanation. **I also broke nvidia-persistenced.** It drops to uid 122 with setuid and without initgroups(), so it never receives the video supplementary group and failed with "Insufficient Permissions" once the nodes were 0660 root:video. A systemd drop-in runs it as root - upstream's own default - which works without widening the device permissions. It was running fine before. The helper now waits for the nodes, verifies the modes afterwards, and exits non-zero under --strict rather than reporting a no-op as success. It takes UMBRA_NVIDIA_DEVDIR so its three paths can be tested without root and without a GPU; the first attempt at that test pointed CANARY at a non-existent path and still exited 0, because the fix loop globbed /dev directly and the override only reached the wait. Also: nvidia-device-access.sh `show` reported "cannot measure the GPU" when a cached sudo credential had simply expired - a finding about the machine when the truth was that the tool could not ask. It now distinguishes no from could-not-check, because I was misled by it during this check. **Still outstanding, and I could not finish it.** The repository is correct; the installed state is stale, because clearing the unit's active (exited) state needs root and that authorization was declined. As of now /dev/nvidiactl is 0666 and the controller can open it. To finish: systemctl stop umbra-nvidia-devices.service systemctl start umbra-nvidia-devices.service stat -c '%a %U:%G %n' /dev/nvidiactl and expect 660 root:video. Otherwise the system came through the reboot intact: the deployment was down only because a leftover `python3 -m http.server` from an earlier session was holding port 8442, and the backend mounts the same files itself. Deployment healthy, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, agent account intact, zero stray agents. 471 pass with a database, 430 without.
24 lines
822 B
SYSTEMD
24 lines
822 B
SYSTEMD
# Backstop for the GPU device permissions.
|
|
#
|
|
# nvidia-persistenced chmods /dev/nvidiactl back to 0666 every time it
|
|
# initialises, which it does at boot and again on any restart. udev does not manage
|
|
# these nodes, so nothing can react to that event. This re-checks every two
|
|
# minutes; the helper is a handful of stat and chmod calls and a no-op when the
|
|
# permissions are already right.
|
|
#
|
|
# It is a timer rather than an event hook because the event is not ours to see. On
|
|
# a box that also runs a web service, a security invariant that depends on nothing
|
|
# else touching it is an invariant that eventually is not true.
|
|
|
|
[Unit]
|
|
Description=Periodically re-assert the NVIDIA device permissions
|
|
|
|
[Timer]
|
|
OnBootSec=90s
|
|
OnUnitActiveSec=120s
|
|
AccuracySec=10s
|
|
Unit=umbra-nvidia-devices.service
|
|
|
|
[Install]
|
|
WantedBy=timers.target
|