0ab1f0f1de
This is the verification I said could not be done until the machine rebooted. The reboot proved the boot path wrong, and uncovering why took three rounds. **The boundary was gone.** /dev/nvidiactl came back 0666 root:root, so the controller could open the card again - the exact thing the udev rule was applied to prevent. **Why: the unit ran before anything existed.** umbra-nvidia-devices ran at 05:38:24, ordered After=systemd-udev-settle, globbed /dev, found no nodes, and exited 0. udev-settle means "udev has finished its queue", not "the proprietary driver has finished initialising". The nodes appeared moments later at 0666 and nothing revisited them. The helper ended in an unconditional `exit 0` - I had written that deliberately, so an early boot would not fail a unit - and it is precisely what made "I fixed the permissions" indistinguishable from "there were nothing to fix". **Why the first repair also failed: nvidia-persistenced resets the mode.** It chmods /dev/nvidiactl back to 0666 on every initialisation, via nvidia-modprobe. That is out of udev's reach entirely, because udev does not manage these nodes. So the fix is no longer an ordering problem but an eventual one: the unit now runs After=nvidia-persistenced.service, and a two-minute timer re-asserts the permissions because the trigger that breaks them can fire at any time. **Why the timer did nothing either.** RemainAfterExit=yes on the oneshot keeps it "active (exited)", so systemd treated every activation as already satisfied: the timer fired on schedule, the service reported Result=success and ExecMainStatus=0, and the permission was still wrong two minutes later. That is the original bug repeated one level out, inside the mechanism built to repair it. Removed, and a test now pins it - by directive name, with comments stripped, since the unit documents the omission by naming it and a substring search failed on the explanation. **I also broke nvidia-persistenced.** It drops to uid 122 with setuid and without initgroups(), so it never receives the video supplementary group and failed with "Insufficient Permissions" once the nodes were 0660 root:video. A systemd drop-in runs it as root - upstream's own default - which works without widening the device permissions. It was running fine before. The helper now waits for the nodes, verifies the modes afterwards, and exits non-zero under --strict rather than reporting a no-op as success. It takes UMBRA_NVIDIA_DEVDIR so its three paths can be tested without root and without a GPU; the first attempt at that test pointed CANARY at a non-existent path and still exited 0, because the fix loop globbed /dev directly and the override only reached the wait. Also: nvidia-device-access.sh `show` reported "cannot measure the GPU" when a cached sudo credential had simply expired - a finding about the machine when the truth was that the tool could not ask. It now distinguishes no from could-not-check, because I was misled by it during this check. **Still outstanding, and I could not finish it.** The repository is correct; the installed state is stale, because clearing the unit's active (exited) state needs root and that authorization was declined. As of now /dev/nvidiactl is 0666 and the controller can open it. To finish: systemctl stop umbra-nvidia-devices.service systemctl start umbra-nvidia-devices.service stat -c '%a %U:%G %n' /dev/nvidiactl and expect 660 root:video. Otherwise the system came through the reboot intact: the deployment was down only because a leftover `python3 -m http.server` from an earlier session was holding port 8442, and the backend mounts the same files itself. Deployment healthy, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, agent account intact, zero stray agents. 471 pass with a database, 430 without.
35 lines
2.0 KiB
Plaintext
35 lines
2.0 KiB
Plaintext
# Restrict the NVIDIA device nodes to the accounts that should have the GPU.
|
|
#
|
|
# The nvidia driver creates /dev/nvidia* in the kernel itself. It does not
|
|
# register them in sysfs and it does not let udev process them:
|
|
#
|
|
# $ udevadm info --name=/dev/nvidiactl
|
|
# Unknown device "/dev/nvidiactl": No such device
|
|
#
|
|
# So a rule that matches the device nodes - KERNEL=="nvidiactl" and so on -
|
|
# never runs, because udev never evaluates a rule for a device it does not
|
|
# know about. That is also why the nodes are 0666: nothing in
|
|
# /usr/lib/udev/rules.d/71-nvidia.rules sets a MODE or a GROUP, and the driver
|
|
# created them world-accessible. A first attempt at this file was exactly those
|
|
# node-matching rules; it was inert, and only the permission check afterwards
|
|
# showed it.
|
|
#
|
|
# What udev *does* know about is the PCI device. This file therefore hooks the
|
|
# event where the driver binds, which is after modprobe has created the nodes,
|
|
# and fixes them from a script. It sorts after the driver's own 71-nvidia.rules
|
|
# so the module-load rules there have already run.
|
|
#
|
|
# `nvidia-restrict-permissions` is the script that does the work. Keeping it
|
|
# out of the rule file matters: a RUN key with a shell pipeline is one long line
|
|
# to review, and this is a security boundary.
|
|
#
|
|
# The video group already contains `ollama`, which is the account that actually
|
|
# opens the GPU - the agent talks to ollama over HTTP and runs `nvidia-smi`, it
|
|
# does not open these nodes itself. `umbra-poc` is added to the group by
|
|
# nvidia-device-access.sh, not by hand: `nvidia-smi` opens /dev/nvidia0,
|
|
# /dev/nvidiactl and /dev/nvidia-uvm, so without it the agent would still start,
|
|
# still heartbeat, and report a machine with no GPU - a failure that presents as
|
|
# a scheduling problem and is a permissions one.
|
|
|
|
ACTION=="bind", SUBSYSTEM=="pci", ATTRS{vendor}=="0x10de", RUN+="/usr/lib/udev/nvidia-restrict-permissions --wait=30"
|
|
ACTION=="add", DEVPATH=="/bus/pci/drivers/nvidia", RUN+="/usr/lib/udev/nvidia-restrict-permissions --wait=30" |