0ab1f0f1de
This is the verification I said could not be done until the machine rebooted. The reboot proved the boot path wrong, and uncovering why took three rounds. **The boundary was gone.** /dev/nvidiactl came back 0666 root:root, so the controller could open the card again - the exact thing the udev rule was applied to prevent. **Why: the unit ran before anything existed.** umbra-nvidia-devices ran at 05:38:24, ordered After=systemd-udev-settle, globbed /dev, found no nodes, and exited 0. udev-settle means "udev has finished its queue", not "the proprietary driver has finished initialising". The nodes appeared moments later at 0666 and nothing revisited them. The helper ended in an unconditional `exit 0` - I had written that deliberately, so an early boot would not fail a unit - and it is precisely what made "I fixed the permissions" indistinguishable from "there were nothing to fix". **Why the first repair also failed: nvidia-persistenced resets the mode.** It chmods /dev/nvidiactl back to 0666 on every initialisation, via nvidia-modprobe. That is out of udev's reach entirely, because udev does not manage these nodes. So the fix is no longer an ordering problem but an eventual one: the unit now runs After=nvidia-persistenced.service, and a two-minute timer re-asserts the permissions because the trigger that breaks them can fire at any time. **Why the timer did nothing either.** RemainAfterExit=yes on the oneshot keeps it "active (exited)", so systemd treated every activation as already satisfied: the timer fired on schedule, the service reported Result=success and ExecMainStatus=0, and the permission was still wrong two minutes later. That is the original bug repeated one level out, inside the mechanism built to repair it. Removed, and a test now pins it - by directive name, with comments stripped, since the unit documents the omission by naming it and a substring search failed on the explanation. **I also broke nvidia-persistenced.** It drops to uid 122 with setuid and without initgroups(), so it never receives the video supplementary group and failed with "Insufficient Permissions" once the nodes were 0660 root:video. A systemd drop-in runs it as root - upstream's own default - which works without widening the device permissions. It was running fine before. The helper now waits for the nodes, verifies the modes afterwards, and exits non-zero under --strict rather than reporting a no-op as success. It takes UMBRA_NVIDIA_DEVDIR so its three paths can be tested without root and without a GPU; the first attempt at that test pointed CANARY at a non-existent path and still exited 0, because the fix loop globbed /dev directly and the override only reached the wait. Also: nvidia-device-access.sh `show` reported "cannot measure the GPU" when a cached sudo credential had simply expired - a finding about the machine when the truth was that the tool could not ask. It now distinguishes no from could-not-check, because I was misled by it during this check. **Still outstanding, and I could not finish it.** The repository is correct; the installed state is stale, because clearing the unit's active (exited) state needs root and that authorization was declined. As of now /dev/nvidiactl is 0666 and the controller can open it. To finish: systemctl stop umbra-nvidia-devices.service systemctl start umbra-nvidia-devices.service stat -c '%a %U:%G %n' /dev/nvidiactl and expect 660 root:video. Otherwise the system came through the reboot intact: the deployment was down only because a leftover `python3 -m http.server` from an earlier session was holding port 8442, and the backend mounts the same files itself. Deployment healthy, database unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or datastore tables, ollama serving, agent account intact, zero stray agents. 471 pass with a database, 430 without.
100 lines
3.8 KiB
Bash
Executable File
100 lines
3.8 KiB
Bash
Executable File
#!/bin/sh
|
|
# Put the NVIDIA device nodes in the video group, mode 0660.
|
|
#
|
|
# Called from udev when the driver binds, and by umbra-nvidia-devices.service at
|
|
# boot. Idempotent, and safe to run when the nodes are not there yet.
|
|
#
|
|
# WHY THIS ISN'T JUST A CHMOD LOOP
|
|
#
|
|
# The nvidia driver creates /dev/nvidia* in the kernel. It does not register them
|
|
# in sysfs and it does not let udev process them, so a rule matching the device
|
|
# nodes never runs - `udevadm info --name=/dev/nvidiactl` answers "Unknown
|
|
# device", and the driver's own 71-nvidia.rules sets no MODE and no GROUP. That is
|
|
# why they arrive as 0666.
|
|
#
|
|
# Which leaves a scheduling problem: nothing udev manages announces when these
|
|
# nodes appear, so a boot-time oneshot ordered after `systemd-udev-settle` runs
|
|
# *before* the driver has finished creating them. That is exactly what happened on
|
|
# 2026-10-01: the unit ran at 05:38:24, found nothing, exited 0, and the nodes
|
|
# came up 0666 a moment later. The fix has to be eventual rather than ordered,
|
|
# which is what the wait below is for.
|
|
#
|
|
# --strict exists because the first version of this script ended in an
|
|
# unconditional `exit 0`, so "I fixed the permissions" and "there were no
|
|
# permissions to fix" were indistinguishable from the outside. Under systemd
|
|
# that is a unit reporting success while having done nothing. With --strict, a run
|
|
# that found no nodes fails loudly instead.
|
|
|
|
set -e
|
|
|
|
# Overridable so the waiting and the strictness can be tested without root and
|
|
# without a GPU. The first attempt at testing --strict pointed CANARY at a
|
|
# non-existent path and it still exited 0, because the fix loop globbed /dev
|
|
# directly and the override only affected the wait - so the test proved nothing.
|
|
DEVDIR="${UMBRA_NVIDIA_DEVDIR:-/dev}"
|
|
|
|
WAIT=0
|
|
STRICT=0
|
|
for argument in "$@"; do
|
|
case "$argument" in
|
|
--wait=*) WAIT="${argument#--wait=}" ;;
|
|
--wait) WAIT=30 ;;
|
|
--strict) STRICT=1 ;;
|
|
esac
|
|
done
|
|
|
|
# /dev/nvidiactl is the canary: it is created by the core nvidia module, before
|
|
# the uvm and modeset ones, so waiting for it means the rest are close behind.
|
|
CANARY="$DEVDIR/nvidiactl"
|
|
waited=0
|
|
while [ ! -e "$CANARY" ] && [ "$waited" -lt "$WAIT" ]; do
|
|
sleep 1
|
|
waited=$((waited + 1))
|
|
done
|
|
|
|
fixed=0
|
|
for node in "$DEVDIR"/nvidiactl "$DEVDIR"/nvidia[0-9]* "$DEVDIR"/nvidia-modeset \
|
|
"$DEVDIR"/nvidia-uvm "$DEVDIR"/nvidia-uvm-tools; do
|
|
[ -e "$node" ] || continue
|
|
chgrp video "$node" 2>/dev/null || true
|
|
chmod 0660 "$node" 2>/dev/null || true
|
|
fixed=$((fixed + 1))
|
|
done
|
|
|
|
# The capability nodes carry a privileged bit and nothing here needs to set it.
|
|
for node in "$DEVDIR"/nvidia-caps/nvidia-cap*; do
|
|
[ -e "$node" ] || continue
|
|
chgrp video "$node" 2>/dev/null || true
|
|
chmod 0440 "$node" 2>/dev/null || true
|
|
done
|
|
|
|
# Verify rather than assume. `chmod` on a device node can fail - read-only /dev,
|
|
# a node held open - and a script that exits 0 anyway reports a boundary that is
|
|
# not there.
|
|
wrong=""
|
|
for node in "$DEVDIR"/nvidiactl "$DEVDIR"/nvidia[0-9]* "$DEVDIR"/nvidia-modeset \
|
|
"$DEVDIR"/nvidia-uvm "$DEVDIR"/nvidia-uvm-tools; do
|
|
[ -e "$node" ] || continue
|
|
mode=$(stat -c '%a' "$node" 2>/dev/null || echo "?")
|
|
group=$(stat -c '%G' "$node" 2>/dev/null || echo "?")
|
|
if [ "$mode" != "660" ] || [ "$group" != "video" ]; then
|
|
wrong="$wrong $node($mode,$group)"
|
|
fi
|
|
done
|
|
|
|
if [ -n "$wrong" ]; then
|
|
echo "nvidia-restrict-permissions: still not restricted:$wrong" >&2
|
|
exit 1
|
|
fi
|
|
|
|
if [ "$fixed" -eq 0 ]; then
|
|
if [ "$STRICT" -eq 1 ]; then
|
|
echo "nvidia-restrict-permissions: waited ${WAIT}s and found no /dev/nvidia* nodes" >&2
|
|
exit 1
|
|
fi
|
|
echo "nvidia-restrict-permissions: no nodes yet; a later run will catch them" >&2
|
|
exit 0
|
|
fi
|
|
|
|
echo "nvidia-restrict-permissions: restricted $fixed node(s)${wrong:+ (waited ${waited}s)}"
|
|
exit 0 |