umbragit/UmbraTrans
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

The GPU boundary did not survive a reboot, and neither did the fix for it

Browse Source 
This is the verification I said could not be done until the machine
rebooted. The reboot proved the boot path wrong, and uncovering why took
three rounds.

**The boundary was gone.** /dev/nvidiactl came back 0666 root:root, so
the controller could open the card again - the exact thing the udev rule
was applied to prevent.

**Why: the unit ran before anything existed.** umbra-nvidia-devices ran
at 05:38:24, ordered After=systemd-udev-settle, globbed /dev, found no
nodes, and exited 0. udev-settle means "udev has finished its queue", not
"the proprietary driver has finished initialising". The nodes appeared
moments later at 0666 and nothing revisited them. The helper ended in an
unconditional `exit 0` - I had written that deliberately, so an early boot
would not fail a unit - and it is precisely what made "I fixed the
permissions" indistinguishable from "there were nothing to fix".

**Why the first repair also failed: nvidia-persistenced resets the mode.**
It chmods /dev/nvidiactl back to 0666 on every initialisation, via
nvidia-modprobe. That is out of udev's reach entirely, because udev does
not manage these nodes. So the fix is no longer an ordering problem but an
eventual one: the unit now runs After=nvidia-persistenced.service, and a
two-minute timer re-asserts the permissions because the trigger that
breaks them can fire at any time.

**Why the timer did nothing either.** RemainAfterExit=yes on the oneshot
keeps it "active (exited)", so systemd treated every activation as already
satisfied: the timer fired on schedule, the service reported Result=success
and ExecMainStatus=0, and the permission was still wrong two minutes later.
That is the original bug repeated one level out, inside the mechanism built
to repair it. Removed, and a test now pins it - by directive name, with
comments stripped, since the unit documents the omission by naming it and a
substring search failed on the explanation.

**I also broke nvidia-persistenced.** It drops to uid 122 with setuid and
without initgroups(), so it never receives the video supplementary group and
failed with "Insufficient Permissions" once the nodes were 0660 root:video.
A systemd drop-in runs it as root - upstream's own default - which works
without widening the device permissions. It was running fine before.

The helper now waits for the nodes, verifies the modes afterwards, and
exits non-zero under --strict rather than reporting a no-op as success. It
takes UMBRA_NVIDIA_DEVDIR so its three paths can be tested without root and
without a GPU; the first attempt at that test pointed CANARY at a
non-existent path and still exited 0, because the fix loop globbed /dev
directly and the override only reached the wait.

Also: nvidia-device-access.sh `show` reported "cannot measure the GPU" when
a cached sudo credential had simply expired - a finding about the machine
when the truth was that the tool could not ask. It now distinguishes no from
could-not-check, because I was misled by it during this check.

**Still outstanding, and I could not finish it.** The repository is
correct; the installed state is stale, because clearing the unit's active
(exited) state needs root and that authorization was declined. As of now
/dev/nvidiactl is 0666 and the controller can open it. To finish:
  systemctl stop umbra-nvidia-devices.service
  systemctl start umbra-nvidia-devices.service
  stat -c '%a %U:%G %n' /dev/nvidiactl
and expect 660 root:video.

Otherwise the system came through the reboot intact: the deployment was
down only because a leftover `python3 -m http.server` from an earlier
session was holding port 8442, and the backend mounts the same files
itself. Deployment healthy, database unchanged at users=2 orgs=3
projects=0 audit=43 with no compute or datastore tables, ollama serving,
agent account intact, zero stray agents. 471 pass with a database, 430
without.
This commit is contained in:
umbra
2026-10-01 06:23:21 -07:00
parent 1ab9c2595d
commit 0ab1f0f1de
6 changed files with 307 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
poc/72-umbra-nvidia.rules
View File

@@ -31,5 +31,5 @@
# still heartbeat, and report a machine with no GPU - a failure that presents as
# a scheduling problem and is a permissions one.
ACTION=="bind", SUBSYSTEM=="pci", ATTRS{vendor}=="0x10de", RUN+="/usr/lib/udev/nvidia-restrict-permissions"
ACTION=="add", DEVPATH=="/bus/pci/drivers/nvidia", RUN+="/usr/lib/udev/nvidia-restrict-permissions"
ACTION=="bind", SUBSYSTEM=="pci", ATTRS{vendor}=="0x10de", RUN+="/usr/lib/udev/nvidia-restrict-permissions --wait=30"
ACTION=="add", DEVPATH=="/bus/pci/drivers/nvidia", RUN+="/usr/lib/udev/nvidia-restrict-permissions --wait=30"

28
poc/nvidia-device-access.sh
View File

@@ -58,13 +58,25 @@ in_group() {
# the node as long as the mode is 0666, which is the state we are leaving. This
# asks the only question that matters, which is whether a *measurement* works,
# because that is what the agent actually needs.
# Prints one of: yes, no, or "cannot check". The third case matters: a cached
# sudo credential expires, `sudo -n` then fails, and a two-valued check reports
# "cannot measure the GPU" - a finding about the machine, when the truth is that
# this tool could not ask. That was read as a real result during the 2026-10-01
# reboot check, and it would have sent anyone reading the output to debug a
# permission problem that did not exist.
sees_gpu() {
local user="$1"
if [[ "$user" == "$(id -un)" ]]; then
nvidia-smi -L >/dev/null 2>&1
else
sudo -n -u "$user" -- nvidia-smi -L >/dev/null 2>&1
nvidia-smi -L >/dev/null 2>&1 && { printf yes; return 0; }
printf no
return 1
fi
sudo -n -u "$user" -- true >/dev/null 2>&1 || { printf "cannot check"; return 2; }
if sudo -n -u "$user" -- nvidia-smi -L >/dev/null 2>&1; then
printf yes; return 0
fi
printf no
return 1
}
do_show() {
@@ -84,11 +96,11 @@ do_show() {
note ""
for user in "$GPU_USER" "$AGENT_USER" "$CONTROLLER_USER"; do
id "$user" >/dev/null 2>&1 || continue
if sees_gpu "$user"; then
good "$user can measure the GPU"
else
warn "$user cannot measure the GPU"
fi
case "$(sees_gpu "$user")" in
yes) good "$user can measure the GPU" ;;
no) warn "$user CANNOT measure the GPU - if that is unexpected, add the account to the video group" ;;
*) warn "$user: could not check (no passwordless sudo; the answer is unknown, not 'no')" ;;
esac
done
if [[ -f "$RULE_DST" ]] && command -v udevadm >/dev/null 2>&1; then
note ""

94
poc/nvidia-restrict-permissions
View File

@@ -1,32 +1,100 @@
#!/bin/sh
# Put the NVIDIA device nodes in the video group, mode 0660.
#
# Called from udev when the driver binds to the card, and by
# umbra-nvidia-devices.service at boot. Idempotent, and safe to run when the
# nodes are not there yet.
# Called from udev when the driver binds, and by umbra-nvidia-devices.service at
# boot. Idempotent, and safe to run when the nodes are not there yet.
#
# Why this is a script and not a MODE= in the rule file: the nvidia driver
# creates /dev/nvidia* in the kernel, without registering them in sysfs and
# without letting udev process them. `udevadm info --name=/dev/nvidiactl`
# answers "Unknown device", so a rule that matches the device nodes never runs.
# That is why the nodes are 0666, and it is the reason a rule file cannot fix it
# - only something triggered by the *driver binding* can, by which time modprobe
# has created the nodes.
# WHY THIS ISN'T JUST A CHMOD LOOP
#
# The nvidia driver creates /dev/nvidia* in the kernel. It does not register them
# in sysfs and it does not let udev process them, so a rule matching the device
# nodes never runs - `udevadm info --name=/dev/nvidiactl` answers "Unknown
# device", and the driver's own 71-nvidia.rules sets no MODE and no GROUP. That is
# why they arrive as 0666.
#
# Which leaves a scheduling problem: nothing udev manages announces when these
# nodes appear, so a boot-time oneshot ordered after `systemd-udev-settle` runs
# *before* the driver has finished creating them. That is exactly what happened on
# 2026-10-01: the unit ran at 05:38:24, found nothing, exited 0, and the nodes
# came up 0666 a moment later. The fix has to be eventual rather than ordered,
# which is what the wait below is for.
#
# --strict exists because the first version of this script ended in an
# unconditional `exit 0`, so "I fixed the permissions" and "there were no
# permissions to fix" were indistinguishable from the outside. Under systemd
# that is a unit reporting success while having done nothing. With --strict, a run
# that found no nodes fails loudly instead.
set -e
for node in /dev/nvidiactl /dev/nvidia[0-9]* /dev/nvidia-modeset \
/dev/nvidia-uvm /dev/nvidia-uvm-tools; do
# Overridable so the waiting and the strictness can be tested without root and
# without a GPU. The first attempt at testing --strict pointed CANARY at a
# non-existent path and it still exited 0, because the fix loop globbed /dev
# directly and the override only affected the wait - so the test proved nothing.
DEVDIR="${UMBRA_NVIDIA_DEVDIR:-/dev}"
WAIT=0
STRICT=0
for argument in "$@"; do
case "$argument" in
--wait=*) WAIT="${argument#--wait=}" ;;
--wait) WAIT=30 ;;
--strict) STRICT=1 ;;
esac
done
# /dev/nvidiactl is the canary: it is created by the core nvidia module, before
# the uvm and modeset ones, so waiting for it means the rest are close behind.
CANARY="$DEVDIR/nvidiactl"
waited=0
while [ ! -e "$CANARY" ] && [ "$waited" -lt "$WAIT" ]; do
sleep 1
waited=$((waited + 1))
done
fixed=0
for node in "$DEVDIR"/nvidiactl "$DEVDIR"/nvidia[0-9]* "$DEVDIR"/nvidia-modeset \
"$DEVDIR"/nvidia-uvm "$DEVDIR"/nvidia-uvm-tools; do
[ -e "$node" ] || continue
chgrp video "$node" 2>/dev/null || true
chmod 0660 "$node" 2>/dev/null || true
fixed=$((fixed + 1))
done
# The capability nodes carry a privileged bit and nothing here needs to set it.
for node in /dev/nvidia-caps/nvidia-cap*; do
for node in "$DEVDIR"/nvidia-caps/nvidia-cap*; do
[ -e "$node" ] || continue
chgrp video "$node" 2>/dev/null || true
chmod 0440 "$node" 2>/dev/null || true
done
# Verify rather than assume. `chmod` on a device node can fail - read-only /dev,
# a node held open - and a script that exits 0 anyway reports a boundary that is
# not there.
wrong=""
for node in "$DEVDIR"/nvidiactl "$DEVDIR"/nvidia[0-9]* "$DEVDIR"/nvidia-modeset \
"$DEVDIR"/nvidia-uvm "$DEVDIR"/nvidia-uvm-tools; do
[ -e "$node" ] || continue
mode=$(stat -c '%a' "$node" 2>/dev/null || echo "?")
group=$(stat -c '%G' "$node" 2>/dev/null || echo "?")
if [ "$mode" != "660" ] || [ "$group" != "video" ]; then
wrong="$wrong $node($mode,$group)"
fi
done
if [ -n "$wrong" ]; then
echo "nvidia-restrict-permissions: still not restricted:$wrong" >&2
exit 1
fi
if [ "$fixed" -eq 0 ]; then
if [ "$STRICT" -eq 1 ]; then
echo "nvidia-restrict-permissions: waited ${WAIT}s and found no /dev/nvidia* nodes" >&2
exit 1
fi
echo "nvidia-restrict-permissions: no nodes yet; a later run will catch them" >&2
exit 0
fi
echo "nvidia-restrict-permissions: restricted $fixed node(s)${wrong:+ (waited ${waited}s)}"
exit 0

48
poc/umbra-nvidia-devices.service
View File

@@ -1,27 +1,45 @@
# Second path to the same permissions, for the case the udev event misses.
# Restrict the NVIDIA device nodes to the video group, and keep them that way.
#
# The nvidia driver creates /dev/nvidia* in the kernel at module load, which
# happens during boot; the udev rule fires on the driver's bind event and
# normally covers it. This unit is here because "normally" is doing real work in
# that sentence: if the module is loaded before the rules are in place, or the
# ordering shifts on a future kernel, the nodes would come up 0666 and stay that
# way until the next reboot noticed.
# Two facts about the nvidia driver shape this unit, and both were learned from a
# reboot that went wrong (2026-10-01).
#
# Deliberately not ordered Before= the nvidia driver, because it does not matter:
# the ExecStart runs the same idempotent script udev runs, and if the nodes are
# not there yet it fixes whatever exists and exits 0. The next trigger catches
# the rest.
# 1. The driver creates /dev/nvidia* in the kernel without registering them in
# sysfs and without letting udev process them. A rule matching the device nodes
# never runs; the udev side hooks the driver's bind event instead.
#
# UNVERIFIED until the next reboot - a unit cannot test its own boot behaviour.
# 2. **nvidia-persistenced resets /dev/nvidiactl to 0666 every time it
# initialises**, via nvidia-modprobe, and that happens after any boot-time
# oneshot has run. Ordering `After=systemd-udev-settle` was not enough: on
# 2026-10-01 the unit ran at 05:38:24, before the driver had created any node,
# found nothing, exited 0, and the permissions were 0666 from then on.
#
# So this unit runs *after* nvidia-persistenced and waits for the nodes rather
# than assuming them, and umbra-nvidia-devices.timer re-checks periodically
# because the trigger that resets the mode is outside udev's reach and can fire
# at any time.
#
# --strict turns "the nodes never appeared" into a failed unit. A unit that
# reports success while having changed nothing is worse than one that fails: the
# failure is visible, and that is the whole difference between this being caught
# in a log and this being caught in six days.
[Unit]
Description=Restrict the NVIDIA device nodes to the video group
After=systemd-udev-settle.service
After=systemd-udev-settle.service nvidia-persistenced.service
Wants=nvidia-persistenced.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/lib/udev/nvidia-restrict-permissions
# Deliberately NOT RemainAfterExit=yes.
#
# The timer activates this unit, and a oneshot with RemainAfterExit stays
# "active (exited)" after it completes - so systemd treats every subsequent
# activation as already satisfied and runs nothing. The first version of the
# timer did exactly that: the timer fired on schedule, the service reported
# Result=success and ExecMainStatus=0, and /dev/nvidiactl was still 0666 two
# minutes later. A timer that reports success while doing nothing is the same
# failure as the boot unit that did, one level further out.
ExecStart=/usr/lib/udev/nvidia-restrict-permissions --wait=60 --strict
[Install]
WantedBy=multi-user.target

23
poc/umbra-nvidia-devices.timer Normal file
View File

@@ -0,0 +1,23 @@
# Backstop for the GPU device permissions.
#
# nvidia-persistenced chmods /dev/nvidiactl back to 0666 every time it
# initialises, which it does at boot and again on any restart. udev does not manage
# these nodes, so nothing can react to that event. This re-checks every two
# minutes; the helper is a handful of stat and chmod calls and a no-op when the
# permissions are already right.
#
# It is a timer rather than an event hook because the event is not ours to see. On
# a box that also runs a web service, a security invariant that depends on nothing
# else touching it is an invariant that eventually is not true.
[Unit]
Description=Periodically re-assert the NVIDIA device permissions
[Timer]
OnBootSec=90s
OnUnitActiveSec=120s
AccuracySec=10s
Unit=umbra-nvidia-devices.service
[Install]
WantedBy=timers.target

149
umbratrans_web/tests/test_nvidia_devices.py
View File

@@ -10,6 +10,7 @@ cannot be what fixes it.
from __future__ import annotations
import os
import shutil
import subprocess
from pathlib import Path
@@ -155,4 +156,150 @@ def test_the_helper_is_idempotent_and_tolerates_absent_nodes(tmp_path: Path) ->
# rather than by running it against the real devices, which needs root.
assert '[ -e "$node" ] || continue' in helper, "no existence guard on the node loop"
assert "exit 0" in helper, "does not end in success"
assert "chmod 0666" not in helper, "it should never widen permissions"
assert "chmod 0666" not in helper, "it should never widen permissions"
# ---------------------------------------------------------------------------
# The boot path, which the 2026-10-01 reboot proved does not work as written.
# ---------------------------------------------------------------------------
#
# On that reboot the oneshot ran at 05:38:24, ordered after systemd-udev-settle,
# before the driver had created any node. The helper globbed /dev, found nothing,
# and exited 0. The nodes came up 0666 a moment later and the boundary was gone
# until it was noticed. `udev-settle` means "udev has finished its queue", not "the
# proprietary driver has finished initialising".
#
# The helper takes UMBRA_NVIDIA_DEVDIR so these paths can be exercised without root
# and without a GPU. The first attempt at testing --strict pointed CANARY at a
# non-existent path and still exited 0, because the fix loop globbed /dev directly
# and the override only affected the wait - a test that could not fail.
HELPER = REPO / "poc" / "nvidia-restrict-permissions"
def _run_helper(devdir: Path, *args: str):
return subprocess.run(
[str(HELPER), *args],
capture_output=True,
text=True,
env={**os.environ, "UMBRA_NVIDIA_DEVDIR": str(devdir)},
timeout=60,
)
def test_strict_mode_fails_when_the_nodes_never_appear(tmp_path: Path) -> None:
"""The failure that matters, and the one that was previously invisible.
Without --strict the helper exits 0 having done nothing, which is correct for
the udev event - a later trigger will catch them. Under systemd it is a unit
reporting success while the boundary is not there.
"""
empty = tmp_path / "dev"
empty.mkdir()
result = _run_helper(empty, "--wait=1", "--strict")
assert result.returncode != 0, (
f"the helper reported success with no nodes present: {result.stdout!r}"
)
assert "found no" in result.stderr, result.stderr
def test_non_strict_mode_is_quiet_when_the_nodes_have_not_appeared_yet(tmp_path: Path) -> None:
"""The udev path must not fail a boot because a device is not there.
A machine with no NVIDIA card, or a trigger that fires before module load, is
an ordinary situation rather than an error.
"""
empty = tmp_path / "dev"
empty.mkdir()
result = _run_helper(empty, "--wait=1")
assert result.returncode == 0, result.stderr
assert "no nodes yet" in result.stderr, result.stderr
def test_the_helper_reports_a_failure_it_could_not_complete(tmp_path: Path) -> None:
"""Verification, not assumption.
`chmod` on a device node can fail. The helper re-reads the modes afterwards
and exits non-zero if any are still 0666 or not in the video group, rather
than reporting a boundary that is not there.
"""
faked = tmp_path / "dev"
faked.mkdir()
for name in ("nvidiactl", "nvidia0"):
(faked / name).write_text("")
(faked / name).chmod(0o666)
# Run as an unprivileged user against a directory it does not own, so neither
# chgrp video nor the mode change can succeed.
result = _run_helper(faked, "--wait=1", "--strict")
if result.returncode == 0:
pytest.skip("this process can re-permission the test directory; the failure "\
"path needs an unprivileged user against a root-owned one")
assert "still not restricted" in result.stderr, result.stderr
def test_the_boot_unit_waits_rather_than_assuming(tmp_path: Path) -> None:
"""The unit's ordering is the bug that let the reboot through.
Asserted on the file, because the failure mode is exactly that a
well-behaved unit runs at the wrong moment and succeeds anyway.
"""
unit = (REPO / "poc" / "umbra-nvidia-devices.service").read_text()
assert "ExecStart=/usr/lib/udev/nvidia-restrict-permissions" in unit, (
"the unit does not run the shared helper"
)
assert "--strict" in unit, (
"the boot unit can report a no-op as success; --strict is what turns "
"'the nodes never appeared' into a failed unit"
)
assert "--wait=" in unit, (
"the boot unit assumes the nodes already exist, which is the bug that let "
"the 2026-10-01 reboot through with the boundary gone"
)
# An earlier version of this test asserted `X or True`, which cannot fail and
# therefore asserted nothing. Ordering is left unasserted on purpose: it is not
# what makes the unit correct, and pinning a comment about it would only
# ossify prose.
def test_the_udev_rules_wait_too(tmp_path: Path) -> None:
"""Both hooks, because the bind event fires before module load as well."""
rules = (REPO / "poc" / "72-umbra-nvidia.rules").read_text()
hooks = [line for line in rules.splitlines() if "RUN+=" in line and "nvidia" in line]
assert hooks, "no udev hook found in the rule file"
for line in hooks:
assert "--wait=" in line, f"a udev hook that does not wait: {line.strip()}"
def test_the_boot_service_can_actually_be_re_run_by_its_timer() -> None:
"""The timer activates the service, and RemainAfterExit would make that a no-op.
Found by watching it: the timer fired on schedule, the service reported
`Result=success` and `ExecMainStatus=0`, and the permission was still wrong two
minutes later. A `Type=oneshot` unit with `RemainAfterExit=yes` stays
"active (exited)" once it has run, so systemd considers every later activation
already satisfied.
That is the original bug - a unit reporting success while having changed
nothing - repeated one level out, in the mechanism built to repair it.
"""
# Comments are stripped first. The unit documents why this directive is absent
# and does so by naming it, so a naive substring search fails on the
# explanation - which is how the first version of this assertion failed while
# the file was correct.
unit = (REPO / "poc" / "umbra-nvidia-devices.service").read_text()
directives = [
line.strip()
for line in unit.split("[Service]", 1)[-1].splitlines()
if line.strip() and not line.strip().startswith("#")
]
service_section = "\n".join(directives)
assert "RemainAfterExit=yes" not in service_section, (
"the boot service sets RemainAfterExit=yes, so the backstop timer cannot "
"re-run it and the permission fix becomes a no-op after the first run"
)
assert "Type=oneshot" in service_section, "expected a oneshot"