umbragit/UmbraTrans
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stop rate-limiting the deployment: any finite budget is exhaustible

Browse Source 
The GPU permissions survived a third consecutive boot - all five nodes
660 root:video - so that is now established rather than hopeful.

The deployment was down again for the same reason, and this time the
widened start limit I committed two boots ago turned out not to have
fixed it. It exhausted twenty restarts and returned to the same permanent
failed, with the port free and nobody touching the service.

That fix's verification missed it. The port was freed after three
restarts, inside the window, so the test exercised the easy case and the
commit said "fixed". The real case is a fault that outlasts the budget,
which is the only case that matters.

The rate limit is the wrong mechanism, not the wrong value. Any finite
budget can be exhausted by a fault that lasts longer than it, and the
failure is silent and permanent - `failed` looks like a decision rather
than a give-up. StartLimitIntervalSec=0 disables it: the unit retries for
as long as the fault lasts and binds the moment it clears.

Verified against the case that broke it twice. Held the port for 200
seconds, which drove 31 restarts - past the old limit of 20 - with the
unit never reaching failed. Freeing the port gave 200 from /api/health
within eight seconds, with nothing having started it. A genuine crash
loop now logs every 5s instead of stopping: noisier, and much better than
a deployment that is quietly down.

469 pass with a database, 428 without. Deployment healthy, GPU boundary
intact, database unchanged at users=2 orgs=3 projects=0 audit=43 with no
compute or datastore tables, ollama serving, persistenced up, perms timer
firing, zero stray agents.
This commit is contained in:
umbra
2026-10-01 07:11:13 -07:00
parent 1dbf5fb384
commit de08cfe38e
2 changed files with 46 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
systemd/umbratrans-web.service
View File

@@ -4,22 +4,33 @@ Documentation=file:///home/umbra/src/UmbraTrans/docs/umbratrans-backend-design.m
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
# A crash loop should stop trying rather than spin, but a clean stop (for # `failed` must not be a resting state for this service.
# example after `systemctl --user restart`) always starts again.
# #
# The window used to be 120s with a burst of 5, and with RestartSec=5s that is # Two attempts to get this right, both wrong in the same way.
# twenty-five seconds of trying. That is not long enough to ride out anything, and
# it turns a *transient* condition into a permanent outage: on 2026-09-30 and again
# on 2026-10-01 something else held port 8442 at boot, the service failed five
# times in ten seconds, systemd gave up, and the deployment then stayed down
# indefinitely - including after the port was free again. Nothing recovers it but a
# person.
# #
# Five hundred seconds with a burst of twenty keeps retrying for the whole window a # The original was StartLimitIntervalSec=120 with a burst of 5 and RestartSec=5s,
# slow boot or a settling reverse proxy can plausibly need, while still stopping a # which is twenty-five seconds of trying. On three separate boots something else
# genuine crash loop rather than spinning on it forever. # held port 8442 - a development shell serving web/, which this backend mounts
StartLimitIntervalSec=500 # itself - the service exhausted the budget, and systemd then never tried again.
StartLimitBurst=20 # The deployment stayed down after the port was free, needing a person to notice.
#
# Widening it to 500s with a burst of 20 did not fix that. It only bought more
# time: on the next boot the squatter outlasted twenty restarts and the unit
# returned to the same permanent `failed`. The verification missed it because the
# port was freed after three restarts, inside the window - so the test exercised
# the easy case and the claim said "fixed".
#
# The lesson is that the rate limit is the wrong mechanism entirely, not that it
# was set too low. Any finite budget can be exhausted by a fault that lasts longer
# than the budget, and the failure is silent and permanent.
#
# So: interval 0 disables rate limiting entirely. This service retries for as long
# as the fault lasts and recovers by itself the moment it clears. A genuine crash
# loop logs every RestartSec instead of stopping - noisier, and vastly better than
# a deployment that is quietly down. There is a `systemctl --user stop` for when
# you mean it.
StartLimitIntervalSec=0
StartLimitBurst=0
[Service] [Service]
Type=simple Type=simple
@@ -46,8 +57,6 @@ ExecStart=/home/umbra/src/UmbraTrans/.venv/bin/python -m umbratrans_web serve --
# Kept in step with ExecStart: a dependency change means re-running `uv sync` and # Kept in step with ExecStart: a dependency change means re-running `uv sync` and
# then `systemctl --user restart`, not editing this file. # then `systemctl --user restart`, not editing this file.
# 5s is short enough that a restart is nearly immediate and long enough that twenty
# of them still spread across the start-limit window above.
Restart=on-failure Restart=on-failure
RestartSec=5s RestartSec=5s
UMask=0077 UMask=0077

28
umbratrans_web/migrations/README.md
View File

@@ -416,14 +416,28 @@ systemd then **never tried again**: the deployment stayed down after the port wa
free, and needed a person to notice. A transient condition had become a permanent free, and needed a person to notice. A transient condition had become a permanent
outage, twice. outage, twice.
Widened to 500 seconds and a burst of 20. Verified by holding the port with a Widening that to 500 seconds and a burst of 20 **did not fix it.** It bought more
squatter, watching the unit cycle `active`/`activating` without giving up, then time and nothing else: on the next boot the squatter outlasted twenty restarts,
freeing the port: the next retry bound and `/api/health` returned 200. Under the the unit returned to the same permanent `failed`, and the deployment stayed down
old limits the same test would have ended in `failed`. with the port free.
The generalisable point is the one this file keeps arriving at: **a service that That fix's verification missed it, and the reason is worth recording. The port was
stops retrying has converted a recoverable fault into an outage, and it does so freed after three restarts — inside the window — so the test exercised the easy
quietly.** `failed` looks like a decision. case and the commit message said *fixed*. The real case is a fault that lasts
longer than the budget, which is exactly the case the budget exists to stop
handling.
So the rate limit is the wrong mechanism, not merely the wrong value: **any finite
budget can be exhausted by a fault that outlasts it, and the resulting failure is
silent and permanent.** `StartLimitIntervalSec=0` disables rate limiting; the unit
retries for as long as the fault lasts and binds the moment it clears.
Verified against the case that broke it twice: port held for 200 seconds, which
drove **31 restarts** — past the old limit of 20 — with the unit never reaching
`failed`. Freeing the port produced `200` from `/api/health` within eight seconds,
with nothing having touched the service. A genuine crash loop now logs every 5
seconds instead of stopping, which is noisier and vastly better than a deployment
that is quietly down. There is `systemctl --user stop` for when you mean it.
## Gates that must pass before approval ## Gates that must pass before approval