The GPU permissions survived a third consecutive boot - all five nodes
660 root:video - so that is now established rather than hopeful.
The deployment was down again for the same reason, and this time the
widened start limit I committed two boots ago turned out not to have
fixed it. It exhausted twenty restarts and returned to the same permanent
failed, with the port free and nobody touching the service.
That fix's verification missed it. The port was freed after three
restarts, inside the window, so the test exercised the easy case and the
commit said "fixed". The real case is a fault that outlasts the budget,
which is the only case that matters.
The rate limit is the wrong mechanism, not the wrong value. Any finite
budget can be exhausted by a fault that lasts longer than it, and the
failure is silent and permanent - `failed` looks like a decision rather
than a give-up. StartLimitIntervalSec=0 disables it: the unit retries for
as long as the fault lasts and binds the moment it clears.
Verified against the case that broke it twice. Held the port for 200
seconds, which drove 31 restarts - past the old limit of 20 - with the
unit never reaching failed. Freeing the port gave 200 from /api/health
within eight seconds, with nothing having started it. A genuine crash
loop now logs every 5s instead of stopping: noisier, and much better than
a deployment that is quietly down.
469 pass with a database, 428 without. Deployment healthy, GPU boundary
intact, database unchanged at users=2 orgs=3 projects=0 audit=43 with no
compute or datastore tables, ollama serving, persistenced up, perms timer
firing, zero stray agents.
Second boot since the permission fix, and it held: all five nodes
0660 root:video, the timer firing on schedule, and umbra - which runs the
controller - still refused the card. That claim is now an observation
rather than an inference.
The same reboot surfaced an unrelated fault worth fixing. The deployment
did not come up: a `python3 -m http.server` from a development session
held port 8442, serving web/ - which the backend mounts itself and does
not need - and the service failed five times in ten seconds. With
StartLimitIntervalSec=120 and StartLimitBurst=5 that exhausted the budget,
and systemd then never tried again. The deployment stayed down after the
port was free and needed someone to notice, which is the second reboot in
a row to end this way.
A transient condition had become a permanent outage. Widened to 500s and
a burst of 20, which keeps retrying across a slow boot or a settling
reverse proxy while still stopping a genuine crash loop.
Verified by holding the port with a squatter and watching: the unit
cycles active/activating without giving up, and the instant the port is
freed the next retry binds and /api/health returns 200. My first attempt
at this test was void - the squatter could not bind because the service
still held the port, so it proved nothing.
Note that two consecutive reboots have been taken down by a background
shell in this session holding the deployment's port. It returns after
each boot. Stopping that shell, or moving it to another port, is the
durable fix; killing it again only defers it.
469 pass with a database, 428 without. Deployment healthy, database
unchanged at users=2 orgs=3 projects=0 audit=43 with no compute or
datastore tables, ollama serving, agent account intact, zero stray
agents.
bdmurphy.me's MX records point at privateemail.com. Its submission host is
smtp.privateemail.com:587 with STARTTLS, offering PLAIN and LOGIN over
TLS 1.3. Not the MX host: an MX serves inbound mail and its submission
ports are closed, so the obvious place to look has nothing listening. A
test message to umbra@bdmurphy.me was delivered, and so was a real
confirmation message through the provisioning path, leaving the account in
`invited` awaiting confirmation.
Flipping to production surfaced two guards that had become limitations
rather than protections, and both are fixed rather than routed around:
- `serve` refused any production configuration. It is now the deployment
command as well as the development one, so it takes --allow-production
by name. That flag is an acknowledgement, not the control: the
configuration gate independently refuses a deployment without a
PostgreSQL DSN, Secure cookies, CSRF, an absolute verification base
URL, and a mail transport.
- `create_app` refused to build a production app unless a repository was
injected, with a message saying the PostgreSQL adapter "is not part of
this slice". It is. The check is now about which adapter is in use
rather than where it came from, stated twice: once against the
configured kind before anything is opened, and once against the built
object, because an injected repository bypasses the first.
Two tests asserted the old behaviour and were rewritten rather than
deleted. One claimed a production app must be refused for want of an
injected repository, which would have made a production deployment
impossible to start; it now covers both routes to a development store.
The other built a production app on an in-memory repository to assert a
Set-Cookie header, which the corrected rule refuses; it now asserts the
cookie flags on settings that ask for them, with the production half
covered by the configuration gate that refuses insecure cookies.
/api/health now reports environment=production, repository=postgres,
mail_deliverable=true, setup_required=false, and the service survives a
restart with its data intact.
Brings up a working deployment and, in the process, fixes a unit that
could not start and a test suite that had been writing into it.
- Created the dedicated `umbratrans` database owned by the non-superuser
role `umbra`, applied the control-plane schema and the phase-1
migration. The migration is idempotent, verified by applying it twice.
The existing `umbracontrol` control plane is untouched, which is the
point of a separate database.
- The unit could not start: `ProtectHome=read-only` forbids the write that
`uv run` makes to ~/.cache, so it exited 2 and restart-looped until
systemd's start limit stopped it. Switched to the venv interpreter
directly, which is the right shape for a service anyway: a fixed
interpreter, nothing to resolve at boot, and no developer tool's cache
able to change what runs.
- The integration suite leaves the database as it found it. It creates
real accounts, sessions, credentials and audit rows and used to leave
every one behind, which was only visible because the scratch database
was disposable. Running it against the deployment database filled it
with ten test accounts, and first-run setup then reported that an
administrator already existed — a confusing failure with no visible
cause. Teardown truncates the rows the suite's code creates and leaves
the schema's reference data alone.
- Bound to 192.168.123.14:8442, the private LAN address the reverse proxy
reaches, replacing a static file server that had no API behind it. The
public hostname now serves the real application.
First-run setup is open, and the deployment runs staging with the mail
requirement explicitly waived: /api/health reports mail_deliverable=false,
the control panel shows a standing banner, and account creation is
refused there. Sign-in and setup work; nothing needing a delivered email
does, until SMTP credentials arrive.
288 tests pass against this database, and leave it clean.
The gap this closes: the operator command that creates the first
administrator refuses to run in production, and production refuses to
start without SMTP credentials that only the operator has. So a real
deployment could not be brought up at all. Everything else already worked.
Setup is offered only when both of these hold, checked against the store
on every request rather than cached at boot:
* no platform administrator exists, and
* the token generated at that boot is presented.
The token is what keeps this from being first-requester-wins: anyone who
reached the URL before the operator would otherwise become the platform
administrator. It is written to the journal, so control of the service is
still what authorises the first administrator, and it is regenerated on
restart.
It is spent when it authorises rather than when the account is created, so
a password the policy rejects cannot leave a shown token usable. Five
failed attempts burn it. Comparison is constant-time. The status endpoint
never returns it, and there is no endpoint that would.
One deliberate exception to the log filter: the announcement uses no
key=value shape, so redact_text leaves it alone, and a test pins both
halves — the setup code still gets through, and `setup_token=` still does
not. The filter is not weakened, disabled, or special-cased.
Also:
- a `staging` environment, so a deployment can be brought up before SMTP
exists. It holds every production rule except that one, and the waiver
that permits it is refused in production outright, reported by
/api/health as mail_deliverable, and shown as a standing banner in the
control panel, which then refuses to create accounts. A staging
deployment cannot become a convenient way to run something production
would not allow.
- `serve` accepts `--allow-lan-bind` for a named private address, because
the reverse proxy runs on another host. It refuses a wildcard and a
public address, so the flag cannot open the port to the internet. AF_UNIX
was added to the unit's address-family set, without which the PostgreSQL
socket connection cannot be made.
- `count_users_with_role` on the repository, counting active assignments
rather than profile rows, because a suspended administrator is not one.
- setup.html and the sign-in link that appears only while setup is
outstanding, plus the deployment runbook and a rootless unit.
A test of mine was asserting over every credential row in the shared
scratch database rather than the one it created, so it passed only while
no account had replaced its one-time password. Scoped it, and added the
other half of the property.
279 pass without a database, 288 against one.
The prototype was served by a bare `python3 -m http.server` in a foreground
shell. It has no restart policy, so it died silently and the public URL
returned a 502 from Caddy with valid DNS and a valid certificate already in
place. This unit makes the failure mode visible and recoverable.
umbratrans-web-prototype.service:
- Restart=on-failure with a 5s delay and a 120s/5-attempt start limit, so a
crash loop stops retrying but a clean stop always starts again.
- Binds 0.0.0.0:8442 serving web/; Caddy keeps terminating TLS upstream.
- Sandboxed: runs unprivileged with an empty capability bounding set, no new
privileges, read-only home and system, private tmp and devices, system call
filtering, and no access to kernel or control-group interfaces.
- Journal output to the user journal for inspection.
PrivateUsers=yes is documented as deliberately omitted. It hides /etc/group
from the manager, so systemd cannot resolve supplementary groups and the unit
fails with status=216/GROUP. A probe unit confirmed the remaining sandbox
directives are all valid in user scope.
Verified: all three routes and the icon return 200 locally and the public
HTTPS URL returns 200. SIGKILL of the main PID recovered automatically within
the restart delay. `systemd-analyze verify` is clean.
Two follow-ups requiring root are recorded in web/README.md: the rootless
install must drop User= because a --user manager cannot setuid, and
`loginctl enable-linger umbra` is needed for the service to survive logout and
start at boot.
