The compute proof of concept had never been run with the agent as
umbra-poc. Running it that way found three defects, all the same
shape: a check that passed because its mechanism did nothing.
The demonstration never reached the agent at all. It held the pid it
spawned, which under `sudo -n -u` is the sudo wrapper - sudo forks the
agent before exec'ing. Signalling that killed the wrapper and left the
worker running, renewing its lease and holding the job indefinitely,
under a printed note saying "exactly like a power cut". An
unprivileged process cannot signal another account's process at all,
and kill(2) on a process group succeeds as soon as one member is
signalled, so the wrapper's death read as the agent's. Every run
leaked an agent.
Lease recovery was a diagnosis rather than a mechanism.
sweep_expired_leases documented itself as "called by the controller's
own timer" and there was no timer; a killed node's job came back only
when somebody noticed a stalled job and posted to an administrative
route by hand. The proof of concept swept by hand and recorded the
result as automatic. app.py now runs the loop for the life of the
application, building the compute repository at startup so a restart
mid-job still recovers it.
The power cut is now the agent's own: os._exit(137) the instant it
claims a job, before running it, because a process that does not own
the agent cannot signal it. A clean stop goes through a stop_file the
agent polls, for the same reason. Liveness is measured by finding the
worker's own pid in /proc matched on its config path, sampled
repeatedly and accumulated - the wrapper and the worker are not
simultaneous, so a single sample finds only the wrapper.
The node key travels by `sudo --preserve-env` rather than as VAR=value
in argv, because argv is world-readable through /proc/<pid>/cmdline
and the environment is not. The demonstration checks whether the key
will survive before starting and falls back to a pipe, which
node_agent.py --node-key-stdin accepts. Neither a file nor argv ever
holds it.
Eight tests, each verified to fail against a mutation of the code it
covers: three for the sweeper (it must return an expired lease, must
survive a failing round, must not invent a repository it has no
database for) and five driving a real agent process against a stub
controller (stops on the stop file, exits 137 holding an uncompleted
job, takes the key from both routes and neither, reports an empty pipe
as an empty pipe).
448 pass with a database, 421 without. The deployment database is
untouched: users=2 orgs=3 projects=0 audit=43, no compute tables.
0f5c1e3408