diff --git a/systemd/umbratrans-web.service b/systemd/umbratrans-web.service index 3204bfd..9af58c4 100644 --- a/systemd/umbratrans-web.service +++ b/systemd/umbratrans-web.service @@ -4,22 +4,33 @@ Documentation=file:///home/umbra/src/UmbraTrans/docs/umbratrans-backend-design.m After=network-online.target Wants=network-online.target -# A crash loop should stop trying rather than spin, but a clean stop (for -# example after `systemctl --user restart`) always starts again. +# `failed` must not be a resting state for this service. # -# The window used to be 120s with a burst of 5, and with RestartSec=5s that is -# twenty-five seconds of trying. That is not long enough to ride out anything, and -# it turns a *transient* condition into a permanent outage: on 2026-09-30 and again -# on 2026-10-01 something else held port 8442 at boot, the service failed five -# times in ten seconds, systemd gave up, and the deployment then stayed down -# indefinitely - including after the port was free again. Nothing recovers it but a -# person. +# Two attempts to get this right, both wrong in the same way. # -# Five hundred seconds with a burst of twenty keeps retrying for the whole window a -# slow boot or a settling reverse proxy can plausibly need, while still stopping a -# genuine crash loop rather than spinning on it forever. -StartLimitIntervalSec=500 -StartLimitBurst=20 +# The original was StartLimitIntervalSec=120 with a burst of 5 and RestartSec=5s, +# which is twenty-five seconds of trying. On three separate boots something else +# held port 8442 - a development shell serving web/, which this backend mounts +# itself - the service exhausted the budget, and systemd then never tried again. +# The deployment stayed down after the port was free, needing a person to notice. +# +# Widening it to 500s with a burst of 20 did not fix that. It only bought more +# time: on the next boot the squatter outlasted twenty restarts and the unit +# returned to the same permanent `failed`. The verification missed it because the +# port was freed after three restarts, inside the window - so the test exercised +# the easy case and the claim said "fixed". +# +# The lesson is that the rate limit is the wrong mechanism entirely, not that it +# was set too low. Any finite budget can be exhausted by a fault that lasts longer +# than the budget, and the failure is silent and permanent. +# +# So: interval 0 disables rate limiting entirely. This service retries for as long +# as the fault lasts and recovers by itself the moment it clears. A genuine crash +# loop logs every RestartSec instead of stopping - noisier, and vastly better than +# a deployment that is quietly down. There is a `systemctl --user stop` for when +# you mean it. +StartLimitIntervalSec=0 +StartLimitBurst=0 [Service] Type=simple @@ -46,8 +57,6 @@ ExecStart=/home/umbra/src/UmbraTrans/.venv/bin/python -m umbratrans_web serve -- # Kept in step with ExecStart: a dependency change means re-running `uv sync` and # then `systemctl --user restart`, not editing this file. -# 5s is short enough that a restart is nearly immediate and long enough that twenty -# of them still spread across the start-limit window above. Restart=on-failure RestartSec=5s UMask=0077 diff --git a/umbratrans_web/migrations/README.md b/umbratrans_web/migrations/README.md index 5a22bf5..b53fe86 100644 --- a/umbratrans_web/migrations/README.md +++ b/umbratrans_web/migrations/README.md @@ -416,14 +416,28 @@ systemd then **never tried again**: the deployment stayed down after the port wa free, and needed a person to notice. A transient condition had become a permanent outage, twice. -Widened to 500 seconds and a burst of 20. Verified by holding the port with a -squatter, watching the unit cycle `active`/`activating` without giving up, then -freeing the port: the next retry bound and `/api/health` returned 200. Under the -old limits the same test would have ended in `failed`. +Widening that to 500 seconds and a burst of 20 **did not fix it.** It bought more +time and nothing else: on the next boot the squatter outlasted twenty restarts, +the unit returned to the same permanent `failed`, and the deployment stayed down +with the port free. -The generalisable point is the one this file keeps arriving at: **a service that -stops retrying has converted a recoverable fault into an outage, and it does so -quietly.** `failed` looks like a decision. +That fix's verification missed it, and the reason is worth recording. The port was +freed after three restarts — inside the window — so the test exercised the easy +case and the commit message said *fixed*. The real case is a fault that lasts +longer than the budget, which is exactly the case the budget exists to stop +handling. + +So the rate limit is the wrong mechanism, not merely the wrong value: **any finite +budget can be exhausted by a fault that outlasts it, and the resulting failure is +silent and permanent.** `StartLimitIntervalSec=0` disables rate limiting; the unit +retries for as long as the fault lasts and binds the moment it clears. + +Verified against the case that broke it twice: port held for 200 seconds, which +drove **31 restarts** — past the old limit of 20 — with the unit never reaching +`failed`. Freeing the port produced `200` from `/api/health` within eight seconds, +with nothing having touched the service. A genuine crash loop now logs every 5 +seconds instead of stopping, which is noisier and vastly better than a deployment +that is quietly down. There is `systemctl --user stop` for when you mean it. ## Gates that must pass before approval