0f5c1e3408
The compute proof of concept had never been run with the agent as umbra-poc. Running it that way found three defects, all the same shape: a check that passed because its mechanism did nothing. The demonstration never reached the agent at all. It held the pid it spawned, which under `sudo -n -u` is the sudo wrapper - sudo forks the agent before exec'ing. Signalling that killed the wrapper and left the worker running, renewing its lease and holding the job indefinitely, under a printed note saying "exactly like a power cut". An unprivileged process cannot signal another account's process at all, and kill(2) on a process group succeeds as soon as one member is signalled, so the wrapper's death read as the agent's. Every run leaked an agent. Lease recovery was a diagnosis rather than a mechanism. sweep_expired_leases documented itself as "called by the controller's own timer" and there was no timer; a killed node's job came back only when somebody noticed a stalled job and posted to an administrative route by hand. The proof of concept swept by hand and recorded the result as automatic. app.py now runs the loop for the life of the application, building the compute repository at startup so a restart mid-job still recovers it. The power cut is now the agent's own: os._exit(137) the instant it claims a job, before running it, because a process that does not own the agent cannot signal it. A clean stop goes through a stop_file the agent polls, for the same reason. Liveness is measured by finding the worker's own pid in /proc matched on its config path, sampled repeatedly and accumulated - the wrapper and the worker are not simultaneous, so a single sample finds only the wrapper. The node key travels by `sudo --preserve-env` rather than as VAR=value in argv, because argv is world-readable through /proc/<pid>/cmdline and the environment is not. The demonstration checks whether the key will survive before starting and falls back to a pipe, which node_agent.py --node-key-stdin accepts. Neither a file nor argv ever holds it. Eight tests, each verified to fail against a mutation of the code it covers: three for the sweeper (it must return an expired lease, must survive a failing round, must not invent a repository it has no database for) and five driving a real agent process against a stub controller (stops on the stop file, exits 137 holding an uncompleted job, takes the key from both routes and neither, reports an empty pipe as an empty pipe). 448 pass with a database, 421 without. The deployment database is untouched: users=2 orgs=3 projects=0 audit=43, no compute tables.
15 KiB
Deploying the UmbraTrans account service
Current state, and what is left to do. Everything in "Done" has been run, not just written.
What is ready
| Piece | State |
|---|---|
| Application | Serves the API and the frontend from one origin, no CORS |
| PostgreSQL adapter | Tested against a real schema carrying a copy of the control plane |
| Email login and verification | Single-use links, no account-existence oracle, no plaintext at rest |
| First-run setup | Creates the first platform administrator through the browser |
| Environments | development, test, staging, production |
| Service unit | systemd/umbratrans-web.service |
| Environment template | systemd/umbratrans.env.example |
Tests: 279 without a database, 288 against one.
Deployed, and what is still outstanding
Done. A dedicated umbratrans database, owned by the non-superuser role
umbra, carrying the control-plane schema with the phase-1 migration applied
(idempotently, verified twice). The service runs as a rootless systemd user unit
on 192.168.123.14:8442, the private LAN address the reverse proxy on the other
host reaches, and the public hostname answers through it. The dedicated database
means the existing umbracontrol control plane is untouched.
Done: SMTP. The domain's MX records point at privateemail.com, whose
submission host is smtp.privateemail.com:587 with STARTTLS, offering PLAIN and
LOGIN. That is not the MX host: an MX serves inbound mail and its submission
ports are closed, which is why looking there finds nothing.
The deployment runs production with that transport configured and, more
importantly, verified: /api/health reports mail_deliverable: true because
the process authenticated, not because a host and a password are present. A
credential the server rejects reports false, and the control panel then refuses
account creation rather than accepting accounts whose confirmation links can
never be delivered.
Note that a domain mailbox is often the right credential for a deployment like this precisely because it is a role address rather than a person's account, and because it can be given its own app password. It is still a credential in a file on this host.
Account recovery and self-service
An administrator can start a password reset for any account in their scope, from the row actions on the accounts page. It does not set a password: the link goes to the address on the account and only its holder can complete it, because an administrator who can choose someone's password can also sign in as them indefinitely. Completing a reset revokes every session on the account, so a cookie captured beforehand stops working.
Anyone signed in can change their own password at /settings.html, which needs
the current password. Changing it also revokes every other session.
Reset links live one hour, against the two days a verification link gets, and the
two are not interchangeable: bootstrap_tokens carries a purpose and every
lookup filters on it, so a reset link cannot confirm an address and a verification
link cannot set a password. Migration 0002 added that column.
An account that has never confirmed its address can be reset, and doing so also confirms it. Holding the link proves control of the address, which is what verification asserts, and without it the flow does not converge: the person resets the password, signs in, and is refused for the same reason, using a verification link they never received.
A note on row-level security
The application owns the tables in this database, and RLS is enabled but not
forced, so the policies are not currently enforced and tenant isolation rests
entirely on the adapter's own WHERE clauses. That is tested; the policies are
not. Giving the service a non-owner role would switch the policies on, and that
configuration has never been run. It should be tested before this database holds
real tenant data. See the open items in umbratrans_web/migrations/README.md.
Running the tests, and a database they will empty
The integration suite writes real rows and empties users, organizations
and audit_events when it finishes. It cannot clean up selectively:
audit_events is append-only, with a trigger that raises on delete, so
truncating is the only option and truncating is indiscriminate.
Two guards, because one was not enough:
UMBRATRANS_TEST_DATABASE_IS_DISPOSABLE=1must accompany the DSN. A database name is not evidence of anything, and an empty database gives no way to tell a scratch one from a live deployment nobody has set up yet.- The suite then refuses to start if the target holds any rows, and names them.
Never point it at the deployment database. It has been used that way by mistake, and the accounts and organization in it were destroyed. A scratch database is the only correct target:
psql -d umbratrans_scratch -c 'TRUNCATE public.users, public.organizations, public.audit_events CASCADE;'
UMBRATRANS_TEST_DSN="postgresql:///umbratrans_scratch?host=/run/postgresql&user=umbra" \
UMBRATRANS_TEST_DATABASE_IS_DISPOSABLE=1 uv run pytest umbratrans_web
Tenant separation and projects
An organization's work is separate from every other organization's. Within an
organization, every account sees every project; a project-scoped assignment
says who is working on one, not who may look at it.
The rule lives in one module, umbratrans_web/tenancy.py, and is not
per-route: a member is pinned to the organization on their own assignment, so
naming a different one in a query string or body cannot widen it. A pinned scope
is a concrete identifier rather than a flag, so "no filter" is unreachable except
for a platform administrator.
Two rules worth holding onto, because both were got wrong first:
- A read naming another organization is ignored; a write is refused. Refusing
a read would confirm the identifier exists. Silently retargeting a write would
put the project in an organization the caller did not ask for and would not
look in —
tenancy.resolve_targetexists for exactly that. - A missing project and a foreign one get the same error, so an identifier cannot be probed for existence across a tenant boundary.
An assignment writes a second user_roles row. That is why every tenancy query
carries TENANT_ROW_PREDICATE, and why the review gate's own SQL in
migrations/README.md needed a project_id IS NULL predicate. Read that section
before approving the migration; the failure mode is an assigned member who cannot
sign in, and it was found by the integration suite and not by any unit test.
Lease recovery runs itself now, and that changed what a deploy must include
A killed node's job is returned by the controller's own sweeper, started with the
application lifespan (LEASE_SWEEP_SECONDS, 10 s). It did not used to, and that
was not a design choice: sweep_expired_leases carried a docstring saying
"Called by the controller's own timer", and there was no timer. Recovery depended
on a human noticing a stalled job and posting to POST /api/compute/leases/sweep
by hand. The route is still there — driving it by hand is how you watch the
mechanism work the first time — but it is a diagnostic now, not the mechanism.
Two consequences for an operator:
- A restart mid-job no longer strands it. The compute repository is built at startup rather than on first use, specifically so the sweeper has something to sweep. A deployment with no database configured gets no repository and no sweeper, which is not a degraded compute tier but the absence of one.
- Watch for
returned N job(s) whose lease expiredin the log. The sweeper logs a warning per recovery round, because a sweeper that dies quietly is worse than no sweeper: the queue simply stops returning work.
The sweeper survives a failing round and logs the traceback. A loop that stopped on the first error would look identical to a queue with nothing in it.
The agent cannot be signalled, by anyone who does not own it
A node agent runs as its own unix account, so its supervisor cannot kill(2) it —
that fails across accounts. Stopping one therefore goes through a stop_file in
its config, which the agent polls each iteration. Fault injection
(--fault-exit-after-claim N) makes it os._exit instead, for exercising lease
recovery against a process that really vanished.
Two sudo details, both learned by running it rather than by reading about it:
sudoresets the environment, soUMBRA_NODE_KEYdoes not reach the agent unless the sudoers policy preserves it.sudo -n -u … --preserve-env=UMBRA_NODE_KEYis what the proof of concept uses, because the alternative (VAR=valuebefore the command) puts the key in argv, which every user on the host can read from/proc/<pid>/cmdline. Where the policy refuses, the agent takes the key on stdin instead.sudoforks before exec'ing, so the process you spawned is the wrapper and the agent is its child. Signalling the wrapper kills the wrapper, leaves the agent running, and renews its lease for ever.
Deploying this needs no migration
The projects table, its (id, organization_id) unique constraint, the
on_hold/released/archived statuses, and the project-scoped arm of
user_roles were all already in the schema. This change uses them and alters
nothing, so a deploy is a restart with no database step:
systemctl --user restart umbratrans-web
The project assignment writes the viewer project role, because there is no
member role at project scope. It is a constant,
PostgresAccountRepository.PROJECT_ASSIGNMENT_ROLE, pending a project workflow.
After a restart, check /api/health first
It is the unauthenticated endpoint a load balancer or an uptime check polls, and
it is not a safe place for a broken query: it calls setup_is_required, which
asks whether any platform administrator exists, so a statement that cannot parse
there is a 500 on the endpoint used to decide whether the deployment is up.
That happened on the first deploy of this change. The tenant-row predicate had
been added to count_users_with_role as a literal {TENANT_ROW_PREDICATE}
inside a plain string instead of an f-string, so the braces went to PostgreSQL
verbatim. Every unit test passed, because none of them run that query.
test_no_sql_string_carries_an_uninterpolated_placeholder now checks the parse
tree so a placeholder in a non-f-string cannot reach a database again, and
test_the_health_probe_queries_run_against_the_real_schema runs the probe's own
queries against the real schema.
The original two open questions, for the record
1. A database (resolved: a dedicated one)
The alternative was the existing umbracontrol database, the real control plane
with umbracontrol_migrator, umbracontrol_api, and umbracontrol_worker
roles. The phase-1 migration has never been applied to it, because applying it is
gated on review and approval, and the test fixture refuses any DSN containing
umbracontrol. A dedicated database avoids both.
2. SMTP credentials (resolved: smtp.privateemail.com:587)
Every account created through the control panel starts unverified and needs a confirmation link. With no mail transport, no new account can ever be activated, so the control panel refuses account creation rather than creating accounts that cannot work.
A staging deployment can be started without this to reach first-run setup, using
UMBRATRANS_MAIL_DELIVERY_WAIVED=1. That waiver is refused in production and is
reported by /api/health as mail_deliverable: false, and the control panel
shows a standing banner. It buys exactly one thing: reaching first-run setup
before credentials exist. It should be removed as soon as SMTP is configured, and
UMBRA_ENV set to production at the same time.
Bring-up, once those two exist
# 1. Environment. 0600, owned by the service user, never committed.
install -d -m 700 ~/.config/umbratrans
install -m 600 systemd/umbratrans.env.example ~/.config/umbratrans/umbratrans.env
$EDITOR ~/.config/umbratrans/umbratrans.env
# systemd's EnvironmentFile is not shell syntax. Do not `source` it to check a
# value: an unquoted `&` in a DSN is a command separator to a shell and will
# silently truncate it. Use:
systemctl --user show umbratrans-web -p Environment
# 2. Unit. A user unit, because linger is enabled, so it survives logout.
# No User= line: the manager already runs as umbra and setting it fails with
# status=216/GROUP here.
install -m 644 systemd/umbratrans-web.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now umbratrans-web
# 3. Check it came up, and read the setup code. This is the only place it appears.
systemctl --user status umbratrans-web
journalctl --user -u umbratrans-web -n 40 | grep -A3 'FIRST-RUN SETUP'
Then open https://umbratrans.bdmurphy.me/setup.html, paste the code, and create
the first administrator.
The sign-in page offers the setup link on its own while setup is outstanding, and hides it afterwards, so there is no way to reach the page for a deployment that is already configured.
What replaces what
Port 8442 is currently a python3 -m http.server serving the static prototype.
It has no API behind it, so nothing on the public hostname can sign in. Promoting
the service means stopping that process and binding 8442 instead:
pid=$(ss -ltnp | grep ':8442 ' | grep -oP 'pid=\K[0-9]+' | head -1)
[ -n "$pid" ] && kill "$pid"
The unit already binds 192.168.123.14:8442, this host's private LAN address,
which is what the reverse proxy on the other host reaches. --allow-lan-bind is
required for that and refuses a wildcard or a public address, so the port is not
opened to the internet. The proxy terminates TLS, so :8442 itself is plain
HTTP on the local network and must not be exposed beyond it.
Reversing the promotion
The static prototype can be brought back with the unit already in the tree:
systemctl --user disable --now umbratrans-web
python3 -m http.server 8442 --bind 0.0.0.0 --directory web &
That returns the public hostname to a sign-in page with no backend, which is worse than the service but better than nothing being wrong.
What is deliberately not here
- No CORS middleware. The frontend and the API share an origin because the service serves both. Adding CORS would mean two origins and two sets of credentials.
- No inline script or style. The content security policy is
script-src 'self'; style-src 'self', and no page violates it. - No way for the control panel to configure SMTP or TLS. A setup wizard that reconfigures its own mail or certificate is how deployments end up with an open relay or a broken certificate. Transport configuration is an environment file, read at start, owned by the operator.
- No OIDC callback yet. The schema and the adapter support the credential type; the provider handshake is not implemented.