Stop rate-limiting the deployment: any finite budget is exhaustible

The GPU permissions survived a third consecutive boot - all five nodes
660 root:video - so that is now established rather than hopeful.

The deployment was down again for the same reason, and this time the
widened start limit I committed two boots ago turned out not to have
fixed it. It exhausted twenty restarts and returned to the same permanent
failed, with the port free and nobody touching the service.

That fix's verification missed it. The port was freed after three
restarts, inside the window, so the test exercised the easy case and the
commit said "fixed". The real case is a fault that outlasts the budget,
which is the only case that matters.

The rate limit is the wrong mechanism, not the wrong value. Any finite
budget can be exhausted by a fault that lasts longer than it, and the
failure is silent and permanent - `failed` looks like a decision rather
than a give-up. StartLimitIntervalSec=0 disables it: the unit retries for
as long as the fault lasts and binds the moment it clears.

Verified against the case that broke it twice. Held the port for 200
seconds, which drove 31 restarts - past the old limit of 20 - with the
unit never reaching failed. Freeing the port gave 200 from /api/health
within eight seconds, with nothing having started it. A genuine crash
loop now logs every 5s instead of stopping: noisier, and much better than
a deployment that is quietly down.

469 pass with a database, 428 without. Deployment healthy, GPU boundary
intact, database unchanged at users=2 orgs=3 projects=0 audit=43 with no
compute or datastore tables, ollama serving, persistenced up, perms timer
firing, zero stray agents.

systemd/umbratrans-web.service

@@ -4,22 +4,33 @@ Documentation=file:///home/umbra/src/UmbraTrans/docs/umbratrans-backend-design.m
 After=network-online.target
 Wants=network-online.target
 
-# A crash loop should stop trying rather than spin, but a clean stop (for
-# example after `systemctl --user restart`) always starts again.
+# `failed` must not be a resting state for this service.
 #
-# The window used to be 120s with a burst of 5, and with RestartSec=5s that is
-# twenty-five seconds of trying. That is not long enough to ride out anything, and
-# it turns a *transient* condition into a permanent outage: on 2026-09-30 and again
-# on 2026-10-01 something else held port 8442 at boot, the service failed five
-# times in ten seconds, systemd gave up, and the deployment then stayed down
-# indefinitely - including after the port was free again. Nothing recovers it but a
-# person.
+# Two attempts to get this right, both wrong in the same way.
 #
-# Five hundred seconds with a burst of twenty keeps retrying for the whole window a
-# slow boot or a settling reverse proxy can plausibly need, while still stopping a
-# genuine crash loop rather than spinning on it forever.
-StartLimitIntervalSec=500
-StartLimitBurst=20
+# The original was StartLimitIntervalSec=120 with a burst of 5 and RestartSec=5s,
+# which is twenty-five seconds of trying. On three separate boots something else
+# held port 8442 - a development shell serving web/, which this backend mounts
+# itself - the service exhausted the budget, and systemd then never tried again.
+# The deployment stayed down after the port was free, needing a person to notice.
+#
+# Widening it to 500s with a burst of 20 did not fix that. It only bought more
+# time: on the next boot the squatter outlasted twenty restarts and the unit
+# returned to the same permanent `failed`. The verification missed it because the
+# port was freed after three restarts, inside the window - so the test exercised
+# the easy case and the claim said "fixed".
+#
+# The lesson is that the rate limit is the wrong mechanism entirely, not that it
+# was set too low. Any finite budget can be exhausted by a fault that lasts longer
+# than the budget, and the failure is silent and permanent.
+#
+# So: interval 0 disables rate limiting entirely. This service retries for as long
+# as the fault lasts and recovers by itself the moment it clears. A genuine crash
+# loop logs every RestartSec instead of stopping - noisier, and vastly better than
+# a deployment that is quietly down. There is a `systemctl --user stop` for when
+# you mean it.
+StartLimitIntervalSec=0
+StartLimitBurst=0
 
 [Service]
 Type=simple
@@ -46,8 +57,6 @@ ExecStart=/home/umbra/src/UmbraTrans/.venv/bin/python -m umbratrans_web serve --
 # Kept in step with ExecStart: a dependency change means re-running `uv sync` and
 # then `systemctl --user restart`, not editing this file.
 
-# 5s is short enough that a restart is nearly immediate and long enough that twenty
-# of them still spread across the start-limit window above.
 Restart=on-failure
 RestartSec=5s
 UMask=0077

umbratrans_web/migrations/README.md

@@ -416,14 +416,28 @@ systemd then **never tried again**: the deployment stayed down after the port wa
 free, and needed a person to notice. A transient condition had become a permanent
 outage, twice.
 
-Widened to 500 seconds and a burst of 20. Verified by holding the port with a
-squatter, watching the unit cycle `active`/`activating` without giving up, then
-freeing the port: the next retry bound and `/api/health` returned 200. Under the
-old limits the same test would have ended in `failed`.
+Widening that to 500 seconds and a burst of 20 **did not fix it.** It bought more
+time and nothing else: on the next boot the squatter outlasted twenty restarts,
+the unit returned to the same permanent `failed`, and the deployment stayed down
+with the port free.
 
-The generalisable point is the one this file keeps arriving at: **a service that
-stops retrying has converted a recoverable fault into an outage, and it does so
-quietly.** `failed` looks like a decision.
+That fix's verification missed it, and the reason is worth recording. The port was
+freed after three restarts — inside the window — so the test exercised the easy
+case and the commit message said *fixed*. The real case is a fault that lasts
+longer than the budget, which is exactly the case the budget exists to stop
+handling.
+
+So the rate limit is the wrong mechanism, not merely the wrong value: **any finite
+budget can be exhausted by a fault that outlasts it, and the resulting failure is
+silent and permanent.** `StartLimitIntervalSec=0` disables rate limiting; the unit
+retries for as long as the fault lasts and binds the moment it clears.
+
+Verified against the case that broke it twice: port held for 200 seconds, which
+drove **31 restarts** — past the old limit of 20 — with the unit never reaching
+`failed`. Freeing the port produced `200` from `/api/health` within eight seconds,
+with nothing having touched the service. A genuine crash loop now logs every 5
+seconds instead of stopping, which is noisier and vastly better than a deployment
+that is quietly down. There is `systemctl --user stop` for when you mean it.
 
 ## Gates that must pass before approval