The GPU permissions survive a reboot, and the deployment stops giving up

Second boot since the permission fix, and it held: all five nodes
0660 root:video, the timer firing on schedule, and umbra - which runs the
controller - still refused the card. That claim is now an observation
rather than an inference.

The same reboot surfaced an unrelated fault worth fixing. The deployment
did not come up: a `python3 -m http.server` from a development session
held port 8442, serving web/ - which the backend mounts itself and does
not need - and the service failed five times in ten seconds. With
StartLimitIntervalSec=120 and StartLimitBurst=5 that exhausted the budget,
and systemd then never tried again. The deployment stayed down after the
port was free and needed someone to notice, which is the second reboot in
a row to end this way.

A transient condition had become a permanent outage. Widened to 500s and
a burst of 20, which keeps retrying across a slow boot or a settling
reverse proxy while still stopping a genuine crash loop.

Verified by holding the port with a squatter and watching: the unit
cycles active/activating without giving up, and the instant the port is
freed the next retry binds and /api/health returns 200. My first attempt
at this test was void - the squatter could not bind because the service
still held the port, so it proved nothing.

Note that two consecutive reboots have been taken down by a background
shell in this session holding the deployment's port. It returns after
each boot. Stopping that shell, or moving it to another port, is the
durable fix; killing it again only defers it.

469 pass with a database, 428 without. Deployment healthy, database
unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or
datastore tables, ollama serving, agent account intact, zero stray
agents.

systemd/umbratrans-web.service

@@ -6,8 +6,20 @@ Wants=network-online.target
 
 # A crash loop should stop trying rather than spin, but a clean stop (for
 # example after `systemctl --user restart`) always starts again.
-StartLimitIntervalSec=120
-StartLimitBurst=5
+#
+# The window used to be 120s with a burst of 5, and with RestartSec=5s that is
+# twenty-five seconds of trying. That is not long enough to ride out anything, and
+# it turns a *transient* condition into a permanent outage: on 2026-09-30 and again
+# on 2026-10-01 something else held port 8442 at boot, the service failed five
+# times in ten seconds, systemd gave up, and the deployment then stayed down
+# indefinitely - including after the port was free again. Nothing recovers it but a
+# person.
+#
+# Five hundred seconds with a burst of twenty keeps retrying for the whole window a
+# slow boot or a settling reverse proxy can plausibly need, while still stopping a
+# genuine crash loop rather than spinning on it forever.
+StartLimitIntervalSec=500
+StartLimitBurst=20
 
 [Service]
 Type=simple
@@ -34,6 +46,8 @@ ExecStart=/home/umbra/src/UmbraTrans/.venv/bin/python -m umbratrans_web serve --
 # Kept in step with ExecStart: a dependency change means re-running `uv sync` and
 # then `systemctl --user restart`, not editing this file.
 
+# 5s is short enough that a restart is nearly immediate and long enough that twenty
+# of them still spread across the start-limit window above.
 Restart=on-failure
 RestartSec=5s
 UMask=0077

umbratrans_web/migrations/README.md

@@ -402,10 +402,28 @@ with the journal showing the helper running and reporting *restricted 5 node(s)*
 currently holding, and it now distinguishes "no" from "could not check" — the
 latter being a fact about the tool, not about the machine.
 
-**Not yet verified:** behaviour across a *second* reboot. The unit, the timer and
-the udev hooks are installed and individually exercised, and the persistenced
-interaction is handled by ordering rather than by luck, but no boot has been
-observed end to end since the change. Check with `show` after the next one.
+**Verified across a reboot** on 2026-10-01 06:37, the second boot since the
+change: all five nodes `0660 root:video`, the timer firing on schedule, and
+`umbra` — the controller — still refused the card. The claim is now an
+observation.
+
+**A second finding from the same reboot, unrelated to the GPU.** The deployment
+did not come up. Something else in a development session held port 8442 — a
+`python3 -m http.server` serving `web/`, which the backend mounts itself and does
+not need — and the service failed five times in ten seconds. With
+`StartLimitIntervalSec=120` and `StartLimitBurst=5` that exhausted the budget, and
+systemd then **never tried again**: the deployment stayed down after the port was
+free, and needed a person to notice. A transient condition had become a permanent
+outage, twice.
+
+Widened to 500 seconds and a burst of 20. Verified by holding the port with a
+squatter, watching the unit cycle `active`/`activating` without giving up, then
+freeing the port: the next retry bound and `/api/health` returned 200. Under the
+old limits the same test would have ended in `failed`.
+
+The generalisable point is the one this file keeps arriving at: **a service that
+stops retrying has converted a recoverable fault into an outage, and it does so
+quietly.** `failed` looks like a decision.
 
 ## Gates that must pass before approval