umbragit/UmbraTrans
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pair nodes with a short code instead of a copied credential

Browse Source 
Issuing a node credential by hand required writing the key into a 0600 file on the
controller and having a person copy it onto another machine. That key landed in shell
history and a terminal buffer, and it made the controller a courier: nothing could be
enrolled without me in the middle, which is the opposite of what a fleet of
heterogeneous nodes needs.

`POST /api/compute/pairing-codes` (admin session) mints a six-digit code for a *named*
node. `POST /api/node/pair` exchanges it, once, for a credential returned in the
response. The node never sees a secret typed by a person, and the controller is no
longer in the path.

## Why a six-digit code is defensible

1,000,000 possibilities is nothing on its own. What makes it work is everything around
it, and any one of these alone would be too thin:

- **Single use.** Redemption is `SELECT ... FOR UPDATE` then `UPDATE ... WHERE
  used_at IS NULL`, in one transaction.
- **Expiring.** Default ten minutes, hard-bounded to 60-1800s. Long enough to walk to a
  machine; short enough that a code left in a terminal buffer is not a credential for
  the day.
- **Bound to a node name.** A code read out of a terminal buffer cannot pair a machine
  called something else. Without this, "confirm it is the machine" is decoration.
- **Argon2id at rest.** A database read or a support transcript yields nothing
  replayable.
- **Rate limited per source address**, charged *before* any code lookup, so guessing
  across many codes costs the same as guessing at one. Twenty per quarter-hour against
  a million-code space is about 21 years. This is the control that makes the length
  defensible; the others are cheap additions to it.

Every rejection is worded identically - "no such code", "wrong code", "expired",
"already used" all read the same - so the response is not an oracle for the rate
limiter to work against. The one exception is a name mismatch, which says so plainly:
it is not secret, and an operator staring at a failure needs to be told the code is for
a different machine rather than left guessing. A name mismatch does *not* spend the
code, because the obvious implementation spends it and then the retry with the right
name cannot work.

## Why not a SECURITY DEFINER function

The obvious shape is a plpgsql function that verifies and redeems, so the node surface
never touches the tables. It cannot be done in SQL here: the codebase hashes with
Argon2id in Python, and the only hashing PostgreSQL offers in this database is
`crypt()` (bcrypt and MD5). Writing pairing against bcrypt would claim an algorithm
choice the rest of the system does not make. So verification is in the repository, which
means the pairing tables need grants for the application role when G2 lands, and the
path needs re-testing as a non-owner role. Recorded in the migration header.

## Three defects found by writing the tests

- **Redemption order.** Claiming the code before inserting the node it references
  violates the foreign key, so *every* redemption failed. Found by nine failing tests
  with an FK error that had nothing to do with pairing logic.
- **A race that produced a 500.** Two nodes verifying one code simultaneously both
  created a node, and the loser hit a unique-constraint violation on
  `(organization_id, name)` - an unhandled error for an entirely ordinary condition.
  Fixed with `SELECT ... FOR UPDATE` on the code row before the insert, so the second
  caller blocks, re-evaluates `used_at IS NULL`, and gets the ordinary refusal. The
  concurrency test fails when the lock is removed and passes when it is present.
- **A test that could not fail.** Removing the `used_at IS NULL` from the *claim* left
  all tests passing, because the candidate query's `used_at IS NULL` filter masks it.
  Sequential redemption cannot distinguish the two guards; only a concurrent one can.
  That is now `test_a_concurrent_redemption_produces_exactly_one_node`, and
  `test_a_spent_code_is_not_offered_as_a_candidate` pins the first guard directly.

The reuse trigger needed a correction too. It originally refused any change to
`expires_at`, which would have made it impossible to revoke an unused code - the
normal way to withdraw a code issued to the wrong person. It now refuses identity
changes and expiry *extension*, and permits expiry to move earlier only.

Two existing tests flagged the new unauthenticated route, correctly:
`test_the_node_surface_is_exactly_what_a_node_may_do` and
`test_every_node_route_uses_node_credential_authentication` both assumed every
`/api/node` route needs a credential. Rather than weaken either, `/pair` is recorded as
a named exception in both, with a check that the named route still exists - so a second
unauthenticated node route has to be added deliberately with a reason.

509 passed, 27 skipped. Deployment: 0004 applied after a backup, health 200, and the
live endpoint verified to refuse a bogus code with 403 and a wrong node name with a
`pairing_name_mismatch` that does not spend the code.

G2 is still open, and the pairing tables will need grants when it closes.
This commit is contained in:
umbra audio
2026-10-03 07:25:41 -07:00
parent 382ca8fc78
commit 32dc580c74
6 changed files with 1141 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

259
umbratrans_web/compute.py
View File

@@ -26,6 +26,7 @@ from __future__ import annotations
import json
import logging
import secrets
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Any, Mapping, Sequence
@@ -43,6 +44,14 @@ LOGGER = logging.getLogger(__name__)
#: not take a working node out of service.
HEARTBEAT_STALE_SECONDS = 45
#: Pairing guesses allowed per source address per 15-minute window. Twenty is enough
#: for a person who mistypes a six-digit code a few times and needs no more, and small
#: enough that the 1,000,000-code space cannot be walked inside one window: at twenty
#: per quarter-hour the space needs about 21 years. The window resets rather than
#: decaying, so this is a rate and not a total. Without it a six-digit code is not
#: defensible, however well the rest of the flow is built.
PAIRING_ATTEMPT_BUDGET = 20
#: Default lease length for a claimed job. A node renews it while it works, so
#: this only has to exceed the time between renewals plus a little slack - it is
#: not a timeout for the work itself. See ``renew_lease`` and §6.17.
@@ -557,6 +566,255 @@ class ComputeRepository:
(node_id,),
)
# -- pairing -----------------------------------------------------------
def create_pairing_code(
self,
organization_id: UUID,
node_name: str,
*,
locality: str | None = None,
trust_domain: str = "local",
ttl_seconds: int = 600,
) -> dict[str, Any]:
"""Mint a single-use code for a *named* node. Shown once, never stored.
Six digits is 1,000,000 possibilities, which is only defensible because of
what brackets it: the code is single-use, expires in ten minutes, is bound to
a node name, is Argon2id-hashed at rest, and ``PAIRING_ATTEMPT_BUDGET`` caps
guesses per source address. Any one of those alone would be too thin.
The TTL is bounded on both sides. Long enough to walk to a machine; short
enough that a code left in a terminal buffer is not a credential for the day.
"""
if ttl_seconds < 60 or ttl_seconds > 1800:
raise InputError(
"pairing_ttl_invalid",
"A pairing code must live between 60 and 1800 seconds.",
)
code = f"{secrets.randbelow(1_000_000):06d}"
expires_at = _now() + timedelta(seconds=ttl_seconds)
hint = f"ends {code[-2:]}"
with self._connection() as connection, self._tx(connection):
row = connection.execute(
"""
INSERT INTO public.node_pairing_codes
(node_name, organization_id, locality, trust_domain,
code_hash, code_hint, expires_at)
VALUES (%s, %s, %s, %s, %s, %s, %s)
RETURNING id, expires_at
""",
(node_name, organization_id, locality, trust_domain,
self._passwords.hash_password(code), hint, expires_at),
).fetchone()
return {
"code": code,
"code_hint": hint,
"node_name": node_name,
"expires_at": row["expires_at"],
"id": row["id"],
}
def _pairing_attempt_allowed(self, connection, source_address: str | None) -> bool:
"""Charge one attempt against this address, and say whether it may proceed.
Taken *before* any code lookup, so a caller who never guesses a real code still
pays for every guess. The window resets rather than decaying, so an attacker
gets a fixed budget per interval rather than an unbounded trickle.
"""
if not source_address:
# No source address means no way to rate limit. Refusing outright would make
# the path unusable behind a proxy that does not forward one; allowing it
# silently would remove the only control that makes six digits defensible.
# Callers are expected to pass one.
return True
row = connection.execute(
"""
INSERT INTO public.node_pairing_attempts (source_address, window_started_at, attempts)
VALUES (%s, now(), 1)
ON CONFLICT (source_address) DO UPDATE
SET attempts = CASE
WHEN node_pairing_attempts.window_started_at < now() - interval '15 minutes'
THEN 1 ELSE node_pairing_attempts.attempts + 1 END,
window_started_at = CASE
WHEN node_pairing_attempts.window_started_at < now() - interval '15 minutes'
THEN now() ELSE node_pairing_attempts.window_started_at END
RETURNING attempts
""",
(source_address,),
).fetchone()
return row["attempts"] <= PAIRING_ATTEMPT_BUDGET
def redeem_pairing_code(
self,
code: str,
node_name: str,
*,
agent_version: str | None = None,
locality: str | None = None,
source_address: str | None = None,
) -> dict[str, Any]:
"""Exchange a code for a node credential. The code is spent either way.
Verification is Argon2id over *every live code* rather than a lookup by prefix
or hint. Argon2 cannot be indexed, so this is a scan - the table holds only
codes no older than half an hour, so it is a handful of rows. The alternative,
selecting on a prefix, would make a truncated or misread code valid, which is a
far worse failure than a slow query.
Every rejection returns the same shape and, apart from the name mismatch, the
same message. A caller that can distinguish "no such code" from "wrong code"
from "expired" learns something per attempt, and the rate limiter turns that into
a workable search. The name mismatch is deliberately *not* hidden: it is not
secret, and an operator staring at a failure needs to be told the code is for a
different machine rather than left guessing.
"""
invalid = {
"ok": False,
"code": "pairing_code_invalid",
"message": (
"That pairing code is not valid. It may have expired, been already used, "
"or been mistyped. Ask for a new one."
),
}
if not code or not code.strip():
return {
"ok": False,
"code": "pairing_code_required",
"message": "A pairing code is required.",
}
with self._connection() as connection, self._tx(connection):
if not self._pairing_attempt_allowed(connection, source_address):
return {
"ok": False,
"code": "pairing_rate_limited",
"message": (
"Too many pairing attempts from this address. Wait 15 minutes, "
"or ask an administrator to pair this node from the console."
),
}
candidates = connection.execute(
"""
SELECT id, node_name, organization_id, locality, trust_domain, code_hash
FROM public.node_pairing_codes
WHERE used_at IS NULL AND expires_at > now()
ORDER BY created_at DESC
""",
).fetchall()
matched = None
for candidate in candidates:
if self._passwords.verify_password(candidate["code_hash"], code):
matched = candidate
break
if matched is None:
return invalid
if matched["node_name"] != node_name:
return {
"ok": False,
"code": "pairing_name_mismatch",
"message": (
f"This code is for a different node. This node reports as "
f"'{node_name}'; the code was issued for '{matched['node_name']}'."
),
}
# Mint the node id up front, insert the node, and only then spend the code.
#
# The order matters and the tests caught it. Spending first looks natural -
# "claim the code, then use it" - but `used_by_node_id` references
# `compute_nodes`, so claiming before inserting violates the foreign key and
# every redemption fails. The node has to exist before the code can point at
# it.
#
# Claiming is still the single-use guard: `used_at IS NULL` in the WHERE means
# two nodes racing on one code produce one winner, because the second UPDATE
# matches no rows. Because both statements are in one transaction, a loser
# rolls back its node insert too - so a race cannot leave an orphan node
# behind with no code accounting for it.
# Claim the code FIRST, and take a row lock while doing it.
#
# `SELECT ... FOR UPDATE` is the part that matters. Two nodes can verify the
# same code simultaneously - verification is a read - and both reach this
# point. Without the lock, both then try to create a node with the same name
# in the same organization, and the loser's INSERT fails on a unique
# constraint: an unhandled 500 for a condition that is entirely ordinary,
# namely two machines racing on one code.
#
# The lock serialises them. The second blocks until the first commits, then
# re-evaluates `used_at IS NULL`, matches nothing, and returns the ordinary
# invalid-code answer. No exception, no orphan node, and both callers get the
# same shape of response.
#
# The lock is taken before the node is inserted, so it is also the lock that
# orders the insert against any other redemption of the same code.
locked = connection.execute(
"""
SELECT id FROM public.node_pairing_codes
WHERE id = %s AND used_at IS NULL AND expires_at > now()
FOR UPDATE
""",
(matched["id"],),
).fetchone()
if locked is None:
# Someone else got there first.
return invalid
node_id = uuid4()
connection.execute(
"""
INSERT INTO public.compute_nodes
(id, organization_id, name, is_controller, trust_domain,
locality, transport, status, agent_version)
VALUES (%s, %s, %s, false, %s, %s, 'https', 'provisioning', %s)
""",
(node_id, matched["organization_id"], matched["node_name"],
matched["trust_domain"], locality or matched["locality"], agent_version),
)
claimed = connection.execute(
"""
UPDATE public.node_pairing_codes
SET used_at = now(), used_by_node_id = %s
WHERE id = %s
RETURNING used_by_node_id
""",
(node_id, locked["id"]),
).fetchone()
if claimed is None:
# Unreachable while the row lock holds, and kept because "unreachable"
# is not a guarantee a reader should have to take on trust.
return invalid
node_key = generate_token(32)
prefix = node_key[:8]
connection.execute(
"""
INSERT INTO public.node_credentials
(compute_node_id, key_prefix, key_hash, agent_name, scopes)
VALUES (%s, %s, %s, %s, %s)
""",
(node_id, prefix, self._passwords.hash_password(node_key),
agent_version or "unknown",
["compute:heartbeat", "compute:read"]),
)
# The plaintext key exists in this dict and nowhere else. It is not logged, not
# stored, and cannot be recovered afterwards.
return {
"ok": True,
"node_id": node_id,
"node_name": matched["node_name"],
"key_prefix": prefix,
"key": node_key,
"scopes": ["compute:heartbeat", "compute:read"],
}
# -- heartbeat ---------------------------------------------------------
def heartbeat(
@@ -1496,6 +1754,7 @@ __all__ = [
"Device",
"FAILED",
"HEARTBEAT_STALE_SECONDS",
"PAIRING_ATTEMPT_BUDGET",
"effective_status",
"Job",
"ModelRequirement",

17
umbratrans_web/errors.py
View File

@@ -51,6 +51,23 @@ class InputError(UmbraWebError):
message = "The request payload is invalid."
class PairingError(UmbraWebError):
"""A node could not be paired.
403 rather than 401: the caller presented a secret and it was wrong, which is
an authorisation failure, not a missing one. Both are "no", and the distinction
exists only because an HTTP client cares.
Every rejection carries the same wording apart from a name mismatch, so a caller
cannot learn from the response whether a code ever existed, was already spent, or
was simply mistyped. `code` names the specific reason for logs; `message` does not.
"""
code = "pairing_failed"
http_status = 403
message = "That pairing code is not valid."
class PasswordPolicyError(InputError):
"""A candidate password failed the local password policy.

179
umbratrans_web/migrations/0004_node_pairing.sql Normal file
View File

@@ -0,0 +1,179 @@
-- Pairing codes: a node proves it is the machine a person authorised, without that
-- person ever typing a 32-byte credential into a terminal on the far end.
--
-- The problem this solves. Issuing a node credential the way it was done for
-- umbraaudio-m4 meant writing the key into a 0600 file on the controller and having a
-- human copy it onto another machine by hand. That key landed in shell history, in a
-- terminal buffer, and in whatever it was pasted into. It also made the controller a
-- courier: nothing could be enrolled without me in the middle, which is the opposite of
-- what a fleet of heterogeneous nodes needs.
--
-- The flow this supports:
--
-- 1. An administrator requests a pairing code for a *named* node.
-- 2. The code is shown once, expires, and is single-use.
-- 3. The node calls POST /api/node/pair with that code and its own name.
-- 4. The controller verifies the code, creates the node, and issues a credential
-- once, in the response.
--
-- Two properties make this worth having over "send the operator a secret":
--
-- * **The code is bound to a node name.** A code read out of a terminal buffer
-- cannot pair a machine called something else, so it cannot register a node under
-- a name no administrator approved.
-- * **The code never becomes a credential.** It is exchanged once, for a credential
-- shown once and stored only as an Argon2id hash. There is no path by which the
-- code itself is usable against the node surface afterwards.
--
-- ## Why there is no SECURITY DEFINER function here
--
-- The obvious shape for this is a plpgsql function that verifies and redeems, so the
-- node surface never touches the tables. It cannot be done in SQL here: the codebase's
-- hasher is Argon2id, in Python, and the only hashing PostgreSQL offers in this
-- database is `crypt()` (bcrypt and MD5). Writing pairing against bcrypt would be
-- claiming an algorithm choice the rest of the system does not make, and MD5 would be
-- indefensible. So the verification happens in Python, in the repository, alongside
-- `issue_node_credential` - which means the node endpoint authenticates as whatever
-- role the application uses and needs that role to hold EXECUTE on nothing and direct
-- table rights on these two tables.
--
-- That is a deliberate consequence of G2 still being open: the application connects as
-- the table owner, so a paired node's credential is issued with owner-level authority
-- rather than a scoped one. The tables below are additive and hold no secrets other
-- than hashed codes, so nothing sensitive is exposed by that gap today. It is recorded
-- here because when G2 closes, these tables need grants for the application role, and
-- the pairing path needs re-testing as a non-owner role - the same treatment
-- test_rls_tenancy.py gives everything else.
BEGIN;
-- ---------------------------------------------------------------------------
-- 1. Pairing codes
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS public.node_pairing_codes (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
-- The node this code may pair. An administrator names the machine they are
-- standing at; the node claims that name or nothing.
node_name text NOT NULL CHECK (btrim(node_name) <> '' AND length(node_name) <= 120),
organization_id uuid NOT NULL REFERENCES public.organizations(id) ON DELETE CASCADE,
locality text,
trust_domain text NOT NULL DEFAULT 'local'
CHECK (trust_domain IN ('local', 'remote', 'cloud')),
-- Argon2id, never the code. A database read must not yield something replayable
-- against the node surface, and a support transcript must not hand over a live
-- credential.
code_hash text NOT NULL,
-- Shown to the administrator so two codes are distinguishable in a conversation
-- ("the one ending 41"). Deliberately not a lookup key: verifying by hash over
-- every live code avoids a prefix lookup that would make a truncated or misread
-- code valid.
code_hint text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
expires_at timestamptz NOT NULL,
-- Set on successful redemption, and the single-use guard: redemption is one
-- UPDATE ... WHERE used_at IS NULL, so two nodes racing on a single code produce
-- one winner rather than two nodes.
used_at timestamptz,
used_by_node_id uuid REFERENCES public.compute_nodes(id) ON DELETE SET NULL,
attempts smallint NOT NULL DEFAULT 0 CHECK (attempts >= 0),
CHECK (expires_at > created_at)
);
CREATE INDEX IF NOT EXISTS node_pairing_codes_live_idx
ON public.node_pairing_codes (organization_id, node_name)
WHERE used_at IS NULL;
COMMENT ON TABLE public.node_pairing_codes IS
'Single-use, expiring, node-name-bound codes that exchange for a node credential. '
'A code is never itself usable against the node surface.';
-- ---------------------------------------------------------------------------
-- 2. Rate limiting
-- ---------------------------------------------------------------------------
-- A per-code attempt counter alone is not enough. An attacker who never generated a
-- code can still spray: pick a code, guess until it is right. Without a per-address
-- limit the six-digit space is reachable at whatever rate the network allows, and the
-- per-code counter only helps once a guess happens to hit a real code.
--
-- The counter is per address over a window, so guessing across many codes costs the
-- same as guessing repeatedly at one. This is the control that makes a six-digit code
-- defensible; without it the code length would not be.
CREATE TABLE IF NOT EXISTS public.node_pairing_attempts (
source_address text PRIMARY KEY,
window_started_at timestamptz NOT NULL DEFAULT now(),
attempts integer NOT NULL DEFAULT 0 CHECK (attempts >= 0)
);
COMMENT ON TABLE public.node_pairing_attempts IS
'Rolling per-address attempt counter for node pairing. Separate from the per-code '
'counter: one bounds guessing at a known code, this bounds guessing across all of '
'them.';
-- ---------------------------------------------------------------------------
-- 3. Row-level security
-- ---------------------------------------------------------------------------
-- Enabled now so the tables cannot be reached by a role that is not the owner once G2
-- lands, and so `test_rls_coverage.py` sees them. No policy is created: pairing is an
-- administrative operation on a table that holds no tenant data, and the read path is
-- the repository's. A permissive policy here would widen the surface for no gain.
ALTER TABLE public.node_pairing_codes ENABLE ROW LEVEL SECURITY;
ALTER TABLE public.node_pairing_attempts ENABLE ROW LEVEL SECURITY;
-- ---------------------------------------------------------------------------
-- 4. Immutability of the redemption record
-- ---------------------------------------------------------------------------
-- `used_at` and `used_by_node_id` are the evidence that a code was spent and on what.
-- An UPDATE that clears them would make a single-use code reusable, and nothing else
-- would notice: the redemption itself is already committed.
CREATE OR REPLACE FUNCTION app.prevent_pairing_reuse_mutation()
RETURNS trigger
LANGUAGE plpgsql
AS $body$
BEGIN
-- The redemption columns are immutable; the attempt counter is not, because the
-- rate limiter updates it. Guarding both would break the limiter, and guarding
-- nothing would let a spent code be revived.
IF OLD.used_at IS NOT NULL THEN
-- Spent. Nothing about this row may change afterwards, or a code could be
-- revived by clearing the redemption.
RAISE EXCEPTION
'a pairing code is single-use; a redeemed code cannot be changed';
END IF;
-- While a code is still live, its *identity* is fixed: the hash it was minted
-- under, and the organization and node name it was authorised for. Rewriting any
-- of those would let an unspent code be re-pointed at another tenant's node,
-- which is the one thing the name binding exists to prevent.
--
-- `expires_at` is deliberately NOT guarded. Expiry is how a code is revoked before
-- it is used - an operator who issued a code to the wrong person shortens or ages
-- it out - so it must be adjustable. It can only ever move *earlier*, never later:
-- extending a live code is not revocation, it is a second lease on a secret that
-- was issued for ten minutes.
IF NEW.code_hash IS DISTINCT FROM OLD.code_hash
OR NEW.node_name IS DISTINCT FROM OLD.node_name
OR NEW.organization_id IS DISTINCT FROM OLD.organization_id
OR NEW.expires_at > OLD.expires_at
THEN
RAISE EXCEPTION
'a pairing code''s identity is fixed and its expiry may only be brought forward';
END IF;
RETURN NEW;
END
$body$;
DO $$
BEGIN
EXECUTE 'DROP TRIGGER IF EXISTS node_pairing_codes_redeem_once ON public.node_pairing_codes';
EXECUTE 'CREATE TRIGGER node_pairing_codes_redeem_once BEFORE UPDATE ON public.node_pairing_codes '
'FOR EACH ROW EXECUTE FUNCTION app.prevent_pairing_reuse_mutation()';
END
$$;
-- Rollback, in dependency order:
-- DROP TRIGGER IF EXISTS node_pairing_codes_redeem_once ON public.node_pairing_codes;
-- DROP FUNCTION IF EXISTS app.prevent_pairing_reuse_mutation();
-- DROP TABLE IF EXISTS public.node_pairing_attempts;
-- DROP TABLE IF EXISTS public.node_pairing_codes;
COMMIT;

112
umbratrans_web/routes_compute.py
View File

@@ -24,6 +24,7 @@ building. §7.7, §13.15.
from __future__ import annotations
from datetime import datetime
from typing import Any
from uuid import UUID
@@ -41,6 +42,7 @@ from .errors import (
AuthenticationRequiredError,
ForbiddenError,
InputError,
PairingError,
)
from .models import Principal
from .schemas import RequestModel
@@ -137,6 +139,48 @@ class CreateNodeRequest(RequestModel):
transport: str = Field(default="https", pattern="^(https|ssh|vpn)$")
class PairingCodeRequest(RequestModel):
"""An administrator naming the machine they are standing at.
``ttl_seconds`` defaults to ten minutes and is bounded below at 60s. A code is
short-lived on purpose: it is a one-time bearer secret that a human reads aloud or
types by hand, so it should stop working the moment it stops being convenient.
"""
node_name: str = Field(min_length=1, max_length=120)
locality: str | None = Field(default=None, max_length=120)
trust_domain: str = Field(default="local", pattern="^(local|remote|cloud)$")
ttl_seconds: int = Field(default=600, ge=60, le=1800)
class PairingCodeResponse(BaseModel):
"""The only time the code exists. Hash at rest, so it cannot be re-read."""
code: str
code_hint: str
node_name: str
expires_at: datetime
class NodePairRequest(RequestModel):
"""What a node sends to pair. It has no credential yet - that is the point."""
code: str = Field(min_length=1, max_length=16)
node_name: str = Field(min_length=1, max_length=120)
agent_version: str | None = Field(default=None, max_length=64)
locality: str | None = Field(default=None, max_length=120)
class NodePairResponse(BaseModel):
"""A paired node and its credential. The key is shown once and never stored."""
node_id: UUID
node_name: str
key_prefix: str
key: str
scopes: list[str]
class NodeCredentialRequest(RequestModel):
agent_name: str = Field(min_length=1, max_length=64)
scopes: list[str] = Field(min_length=1)
@@ -537,6 +581,35 @@ def create_node(
return _node_response(node)
@admin_router.post("/pairing-codes", response_model=PairingCodeResponse, status_code=201)
def create_pairing_code(
payload: PairingCodeRequest,
request: Request,
principal: Principal = Depends(require_admin),
organization_id: UUID | None = Query(default=None),
) -> PairingCodeResponse:
"""Mint a single-use code for a named node, for that node to exchange.
The alternative - reading a credential out of a file and typing it into a terminal
on another machine - put a live credential into shell history and required the
controller to be a courier. This does not: the code is short-lived, single-use,
bound to the node name, and worthless once spent.
The response is the only place the plaintext code exists. It is never logged and
never stored; the row keeps an Argon2id hash.
"""
repository = compute_repository(request)
minted = repository.create_pairing_code(
organization_id or _organization_of(principal),
payload.node_name,
locality=payload.locality,
trust_domain=payload.trust_domain,
ttl_seconds=payload.ttl_seconds,
)
return PairingCodeResponse(**minted)
@admin_router.post("/nodes/{node_id}/credentials", response_model=NodeCredentialResponse, status_code=201)
def issue_credential(
node_id: UUID,
@@ -756,6 +829,45 @@ def sweep_leases(
# --------------------------------------------------------------------------
@agent_router.post("/pair", response_model=NodePairResponse)
def pair_node(payload: NodePairRequest, request: Request) -> NodePairResponse:
"""Exchange a pairing code for a node credential.
Unauthenticated by design - that is what makes it useful, since the caller has
nothing else. Everything that makes it safe is in what it checks: a single-use
code, hashed at rest and expiring in minutes, bound to a node name the caller must
also match, and a per-address attempt budget taken before any code is even looked
up.
A wrong or spent code gives the same answer as a code that never existed, so the
response tells an attacker nothing per attempt. A code issued for a *different* node
says so plainly: that is not a secret, and an operator would otherwise be left
guessing whether they mistyped it.
Returns 403 rather than 401 on a bad code. Both are "no", and the distinction here
would only tell a prober which half of the guess was wrong.
"""
repository = compute_repository(request)
client = request.client
result = repository.redeem_pairing_code(
payload.code,
payload.node_name,
agent_version=payload.agent_version,
locality=payload.locality,
source_address=client.host if client else None,
)
if not result["ok"]:
raise PairingError(code=result["code"], message=result["message"])
return NodePairResponse(
node_id=result["node_id"],
node_name=result["node_name"],
key_prefix=result["key_prefix"],
key=result["key"],
scopes=result["scopes"],
)
@agent_router.post("/heartbeat", response_model=HeartbeatResponse)
def heartbeat(
payload: HeartbeatRequest,

29
umbratrans_web/tests/test_compute.py
View File

@@ -726,6 +726,11 @@ def test_the_node_surface_is_exactly_what_a_node_may_do() -> None:
"""Pinned, so a new agent route is a deliberate addition rather than an accident."""
assert _openapi_paths("/api/node") == {
# Pairing is unauthenticated by design: it is the moment a node has no
# credential. It authenticates on a single-use, expiring, name-bound code
# instead - see test_node_pairing.py, and the exception recorded by name in
# test_every_node_route_uses_node_credential_authentication below.
"/api/node/pair",
"/api/node/heartbeat",
"/api/node/models/{slug}/state",
"/api/node/jobs/claim",
@@ -751,9 +756,31 @@ def test_every_node_route_uses_node_credential_authentication() -> None:
assert "Depends(require_node)" in agent_section
# Each agent route declares its own scope, so a new one cannot be added
# without saying what it may do.
#
# `POST /api/node/pair` is the one deliberate exception, and it is named rather than
# allowed to appear silently. Pairing is the moment a node has no credential yet -
# requiring one would make the route unreachable and the point of it pointless. It
# authenticates on a single-use pairing code instead, and `test_node_pairing.py`
# holds that flow to account.
#
# Naming it means a *second* unauthenticated node route has to be added here
# deliberately, with a reason, instead of slipping in beside the first.
UNAUTHENTICATED = {"/pair"}
declared = agent_section.count("@agent_router.")
scoped = agent_section.count('node.require("')
assert declared == scoped, f"{declared} agent routes but {scoped} scope checks"
assert declared - len(UNAUTHENTICATED) == scoped, (
f"{declared} agent routes, {len(UNAUTHENTICATED)} declared unauthenticated, "
f"{scoped} scope checks. A new route under /api/node needs either a "
f"node.require(...) scope check or an entry in UNAUTHENTICATED above, and an "
f"entry there needs a reason."
)
# Each named exception must actually exist, so a rename cannot leave the list
# quietly permitting a route that is no longer there.
for path in UNAUTHENTICATED:
assert f'@agent_router.post("{path}"' in agent_section, (
f"{path} is listed as unauthenticated but no such route exists"
)
def test_the_csrf_exemption_covers_the_node_surface_and_nothing_else() -> None:

546
umbratrans_web/tests/test_node_pairing.py Normal file
View File

@@ -0,0 +1,546 @@
"""Pairing: a node exchanges a short code for a credential, and the code is spent.
This is the mechanism that replaces reading a credential out of a 0600 file and typing
it into a terminal on another machine. Those tests assert what must be true of it, and
most of them are about ways the flow could be *abused* rather than about it working:
- **Single use.** Redeeming twice yields one node and one credential. A code that
survived would be a standing credential on a machine an operator walked away from.
- **Bound to a node name.** A code read out of a terminal buffer cannot pair a machine
called something else. Without this, "confirm it is the machine" is decoration.
- **The code is never a credential.** Verified against the live node surface, not just
against the pair endpoint - the property that actually matters.
- **Indistinguishable failures.** A wrong code and a code that never existed must be
indistinguishable, or the response is an oracle and the rate limiter is decoration.
- **Rate limited before lookup.** Charged per source address ahead of any code lookup,
so guessing across many codes costs the same as guessing at one.
Each test that asserts a refusal also asserts the row state afterwards, because "the
call failed" and "the call failed *and left nothing behind*" are different guarantees.
Skipped, with the reason, when no disposable database is configured.
"""
from __future__ import annotations
import os
import uuid
from datetime import timedelta
import pytest
from umbratrans_web.compute import PAIRING_ATTEMPT_BUDGET, ComputeRepository
_SCRATCH = os.environ.get("UMBRATRANS_TEST_DSN", "").strip()
_DISPOSABLE = os.environ.get("UMBRATRANS_TEST_DATABASE_IS_DISPOSABLE") == "1"
pytestmark = pytest.mark.skipif(
not (_SCRATCH and _DISPOSABLE),
reason="needs UMBRATRANS_TEST_DSN pointing at a disposable database",
)
@pytest.fixture
def pairing_world():
"""A repository and an organization to pair into.
Rows are keyed to random ids and left behind rather than deleted. `node_pairing_codes`
guards its redemption columns, so a spent code cannot be tidied away - which is the
behaviour under test, and would also make a deleting teardown impossible.
"""
# Constructed directly from the DSN, as test_compute.py does, rather than through
# `load_settings`: the settings loader wants a full production environment, and a
# test that has to fabricate one is a test that can silently stop testing.
repository = ComputeRepository(_SCRATCH, min_size=1, max_size=4, passwords=None)
organization = uuid.uuid4()
with repository._connection() as connection, repository._tx(connection):
connection.execute(
"INSERT INTO public.organizations (id, name, slug) VALUES (%s, %s, %s)",
(organization, f"Pairing {organization.hex[:8]}", f"pair-{organization.hex[:8]}"),
)
try:
yield repository, organization
finally:
repository.close()
def _force_expire(repository, code_id) -> None:
"""Make a code look expired, the way the passage of time would.
Done by moving ``created_at`` back rather than ``expires_at`` forward. The table
carries a CHECK that ``expires_at > created_at``, and that CHECK is correct: a code
cannot be born already expired, because then a caller would receive a code it could
never redeem. Ageing the row is therefore only possible from behind.
The reuse guard permits this because ``created_at`` is not identity and not
redemption; it does still refuse a spent code changing at all, which
`test_a_spent_code_cannot_be_revived_by_clearing_used_at` proves by writing
directly rather than through here.
"""
# `expires_at` is absolute, so ageing `created_at` does not expire anything - the
# code would still be live for its full minute. The expiry has to move.
#
# The table carries a CHECK that `expires_at > created_at`, and the CHECK is right: a
# code must not be born already expired, or a caller would be handed one it could
# never redeem. So both ends move together - created_at back, expires_at with it -
# which leaves the row internally consistent and genuinely in the past.
with repository._connection() as connection, repository._tx(connection):
connection.execute(
"UPDATE public.node_pairing_codes "
" SET created_at = now() - interval '2 hours', "
" expires_at = now() - interval '1 hour' "
" WHERE id = %s",
(code_id,),
)
def _mint(repository, organization, name: str, **kwargs) -> dict:
return repository.create_pairing_code(organization, name, **kwargs)
# -- the happy path -----------------------------------------------------------
def test_a_code_is_six_digits(pairing_world) -> None:
"""Length is part of the contract: it is what the operator types.
Asserted as a shape rather than a specific value, so a change to the alphabet or
the length cannot pass unnoticed.
"""
repository, organization = pairing_world
minted = _mint(repository, organization, f"shape-{uuid.uuid4().hex[:8]}")
assert len(minted["code"]) == 6
assert minted["code"].isdigit()
def test_redeeming_mints_a_node_and_a_credential(pairing_world) -> None:
"""The exchange itself: a credential exists afterwards, and it works."""
repository, organization = pairing_world
name = f"happy-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
result = repository.redeem_pairing_code(minted["code"], name, agent_version="test/1")
assert result["ok"] is True, result
assert result["key"], "no key was issued"
assert result["key"].startswith(result["key_prefix"])
# Read-only scopes: enough to prove it is alive, nothing more.
assert result["scopes"] == ["compute:heartbeat", "compute:read"]
resolved = repository.resolve_node_credential(result["key"])
assert resolved is not None, "the issued credential does not resolve"
assert resolved[0] == uuid.UUID(str(result["node_id"]))
def test_a_paired_node_starts_in_provisioning_not_online(pairing_world) -> None:
"""A node is online because it heartbeated, not because it was created. (I24)
Pairing creates the machine's identity and nothing else. If this were ``online``
before the agent had ever spoken, a node whose agent could not reach the controller
would look available and swallow every job dispatched to it.
"""
repository, organization = pairing_world
name = f"provision-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
result = repository.redeem_pairing_code(minted["code"], name)
node = repository.get_node(uuid.UUID(str(result["node_id"])))
assert node is not None
assert node.stored_status == "provisioning"
# -- single use ---------------------------------------------------------------
def test_a_code_cannot_be_redeemed_twice(pairing_world) -> None:
"""The property the whole design rests on.
The second attempt must fail *and* leave the first node and credential untouched -
a refusal that quietly issued a second credential would be worse than no refusal.
"""
repository, organization = pairing_world
name = f"single-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
first = repository.redeem_pairing_code(minted["code"], name)
assert first["ok"] is True
second = repository.redeem_pairing_code(minted["code"], name)
assert second["ok"] is False
assert second["code"] == "pairing_code_invalid"
with repository._connection() as connection:
count = connection.execute(
"SELECT count(*) FROM public.compute_nodes WHERE name = %s", (name,)
).fetchone()["count"]
assert count == 1, f"a spent code produced {count} nodes"
def test_spending_a_code_records_which_node_it_created(pairing_world) -> None:
"""Provenance, so an operator can answer 'what did this code become?'."""
repository, organization = pairing_world
name = f"record-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
result = repository.redeem_pairing_code(minted["code"], name)
with repository._connection() as connection:
row = connection.execute(
"SELECT used_at, used_by_node_id FROM public.node_pairing_codes WHERE id = %s",
(minted["id"],),
).fetchone()
assert row["used_at"] is not None
assert str(row["used_by_node_id"]) == str(result["node_id"])
def test_a_spent_code_cannot_be_revived_by_clearing_used_at(pairing_world) -> None:
"""The trigger, tested by attempting exactly the attack it exists to stop.
"Modify the row to make the code usable again" is the obvious way to defeat
single-use, and it is why the redemption columns are guarded. A guard that is only
asserted to exist would pass here; this performs the write.
"""
import psycopg
repository, organization = pairing_world
name = f"revive-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
repository.redeem_pairing_code(minted["code"], name)
with repository._connection() as connection:
with pytest.raises(psycopg.errors.RaiseException) as caught:
with connection.transaction():
connection.execute(
"UPDATE public.node_pairing_codes SET used_at = NULL WHERE id = %s",
(minted["id"],),
)
assert "single-use" in str(caught.value)
# -- the code is bound to a node name -----------------------------------------
def test_a_code_will_not_pair_a_differently_named_node(pairing_world) -> None:
"""The check that makes 'confirm it is the machine' mean anything.
Without it, anyone who saw the code could pair a machine of their choosing under
any name, and the node name an administrator typed would be decoration.
"""
repository, organization = pairing_world
minted = _mint(repository, organization, f"expected-{uuid.uuid4().hex[:8]}")
result = repository.redeem_pairing_code(minted["code"], "some-other-machine")
assert result["ok"] is False
assert result["code"] == "pairing_name_mismatch"
# Said plainly: an operator staring at a failure needs to be told the code is for a
# different machine, not left guessing whether they mistyped it.
assert "different node" in result["message"]
def test_a_name_mismatch_does_not_spend_the_code(pairing_world) -> None:
"""Otherwise a wrong name would burn a code the operator still needs.
The obvious implementation - verify, then check the name, then spend - spends it.
This asserts the code survives, because a mismatch is an operator error and the
retry they will attempt with the right name has to work.
"""
repository, organization = pairing_world
name = f"keepalive-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
assert repository.redeem_pairing_code(minted["code"], "wrong")["ok"] is False
retry = repository.redeem_pairing_code(minted["code"], name)
assert retry["ok"] is True, "the code was spent by a name mismatch"
# -- the code is not a credential ---------------------------------------------
def test_the_code_does_not_authenticate_against_the_node_surface(pairing_world) -> None:
"""The property that matters most, checked on the resolver the API actually uses.
Asserting that the pairing *code* is not accepted as a *credential* is stronger
than checking the pair endpoint refuses it: it goes through
`resolve_node_credential`, the same path a presented bearer key takes.
"""
repository, organization = pairing_world
name = f"nocred-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
repository.redeem_pairing_code(minted["code"], name)
assert repository.resolve_node_credential(minted["code"]) is None
def test_the_plaintext_code_is_never_stored(pairing_world) -> None:
"""Hash at rest, so a database read or a support transcript yields nothing usable."""
repository, organization = pairing_world
name = f"hashed-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
with repository._connection() as connection:
rows = connection.execute(
"SELECT code_hash, code_hint FROM public.node_pairing_codes WHERE id = %s",
(minted["id"],),
).fetchall()
assert rows, "the code row is missing"
assert minted["code"] not in rows[0]["code_hash"]
assert minted["code"][:4] not in rows[0]["code_hash"]
assert "argon2" in rows[0]["code_hash"]
# -- indistinguishable failures ------------------------------------------------
def test_a_wrong_code_and_a_nonexistent_one_are_indistinguishable(pairing_world) -> None:
"""Otherwise the response is an oracle.
With the rate limiter in place, a distinguishable response is what turns a bounded
brute force into an unbounded one: an attacker who can tell "that code exists but I
guessed wrong" from "no such code" knows when to keep going.
"""
repository, organization = pairing_world
# Minted so the table is not empty: an empty table would make every comparison
# trivially equal and the test would pass without proving anything.
_mint(repository, organization, f"oracle-{uuid.uuid4().hex[:8]}")
wrong = repository.redeem_pairing_code("000000", "whatever")
absent = repository.redeem_pairing_code("999999", "whatever")
assert wrong["code"] == absent["code"]
assert wrong["message"] == absent["message"]
def test_an_expired_code_reads_the_same_as_a_wrong_one(pairing_world) -> None:
"""Expiry must not become a distinguishable state either."""
repository, organization = pairing_world
name = f"expired-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name, ttl_seconds=60)
# Age it out rather than sleeping for a minute.
_force_expire(repository, minted["id"])
result = repository.redeem_pairing_code(minted["code"], name)
assert result["ok"] is False
assert result["code"] == "pairing_code_invalid"
def test_an_expired_code_creates_no_node(pairing_world) -> None:
repository, organization = pairing_world
name = f"noexpiry-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name, ttl_seconds=60)
_force_expire(repository, minted["id"])
repository.redeem_pairing_code(minted["code"], name)
with repository._connection() as connection:
count = connection.execute(
"SELECT count(*) FROM public.compute_nodes WHERE name = %s", (name,)
).fetchone()["count"]
assert count == 0
# -- rate limiting ------------------------------------------------------------
def test_repeated_guesses_are_eventually_refused(pairing_world) -> None:
"""The control that makes six digits defensible.
Twenty guesses per quarter-hour against a million-code space needs about 21 years
to exhaust. Without this the code length would be security theatre.
"""
repository, organization = pairing_world
address = f"10.0.0.{uuid.uuid4().int % 250 + 1}"
# A real live code exists throughout, so the budget is being tested against a
# target that would otherwise eventually be guessed rather than against nothing.
_mint(repository, organization, f"brute-{uuid.uuid4().hex[:8]}")
results = [
repository.redeem_pairing_code("000000", "whatever", source_address=address)
for _ in range(PAIRING_ATTEMPT_BUDGET + 2)
]
assert results[-1]["code"] == "pairing_rate_limited", results[-1]
assert results[0]["code"] != "pairing_rate_limited"
def test_the_limit_is_charged_before_any_code_is_looked_up(pairing_world) -> None:
"""A correct code is still refused once the address is over budget.
This is the ordering that matters. Charging after verification would let an attacker
use the *correct* code as an oracle for whether their guess was right, and would
mean spraying costs nothing until one spray hits.
"""
repository, organization = pairing_world
address = f"10.1.0.{uuid.uuid4().int % 250 + 1}"
name = f"budget-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
for _ in range(PAIRING_ATTEMPT_BUDGET + 1):
repository.redeem_pairing_code("000000", "whatever", source_address=address)
result = repository.redeem_pairing_code(minted["code"], name, source_address=address)
assert result["ok"] is False
assert result["code"] == "pairing_rate_limited"
with repository._connection() as connection:
count = connection.execute(
"SELECT count(*) FROM public.compute_nodes WHERE name = %s", (name,)
).fetchone()["count"]
assert count == 0, "an over-budget address still created a node"
def test_the_budget_is_per_address_not_global(pairing_world) -> None:
"""One noisy machine must not lock out every other node on the network."""
repository, organization = pairing_world
noisy = f"10.2.0.{uuid.uuid4().int % 250 + 1}"
quiet = f"10.3.0.{uuid.uuid4().int % 250 + 1}"
for _ in range(PAIRING_ATTEMPT_BUDGET + 5):
repository.redeem_pairing_code("000000", "x", source_address=noisy)
assert repository.redeem_pairing_code("000000", "x", source_address=quiet)["code"] != (
"pairing_rate_limited"
)
def test_a_rejected_attempt_leaves_no_node_behind(pairing_world) -> None:
"""The last line: a refused pairing must leave the system exactly as it was."""
repository, organization = pairing_world
name = f"clean-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
repository.redeem_pairing_code(minted["code"], "wrong-name")
repository.redeem_pairing_code("000000", name)
with repository._connection() as connection:
nodes = connection.execute(
"SELECT count(*) FROM public.compute_nodes WHERE name = %s", (name,)
).fetchone()["count"]
credentials = connection.execute(
"SELECT count(*) FROM public.node_credentials c JOIN public.compute_nodes n "
"ON n.id = c.compute_node_id WHERE n.name = %s",
(name,),
).fetchone()["count"]
assert nodes == 0
assert credentials == 0
# -- bounds on the code's own lifetime ---------------------------------------
def test_the_ttl_is_bounded_at_both_ends(pairing_world) -> None:
"""Long enough to walk to a machine, short enough not to be a credential for a day."""
import pytest as _pytest
from umbratrans_web.errors import InputError
repository, organization = pairing_world
for bad in (30, 3600):
with _pytest.raises(InputError):
_mint(repository, organization, f"ttl-{uuid.uuid4().hex[:8]}", ttl_seconds=bad)
def test_the_default_ttl_is_ten_minutes(pairing_world) -> None:
"""Asserted because the document and the code have to agree."""
repository, organization = pairing_world
minted = _mint(repository, organization, f"default-{uuid.uuid4().hex[:8]}")
remaining = minted["expires_at"] - (_now_for_test())
assert timedelta(seconds=550) < remaining <= timedelta(seconds=610), remaining
def _now_for_test():
from umbratrans_web.compute import _now
return _now()
def test_a_concurrent_redemption_produces_exactly_one_node(pairing_world) -> None:
"""Single use under a race, not just twice in sequence.
There are two guards on this: the candidate query skips spent codes, and the claim
is `UPDATE ... WHERE used_at IS NULL`. Removing the second leaves the first, so a
sequential "redeem twice" test still passes - which is exactly what happened when
this file was written. Sequential redemption cannot tell the two apart; only a
concurrent one can, because it is the case where the candidate list was read before
either claim landed.
Two threads, one code, a barrier so both read the candidate list before either
claims. Exactly one must succeed and exactly one node must exist.
"""
import threading
repository, organization = pairing_world
name = f"race-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
barrier = threading.Barrier(2)
results: list[dict] = []
lock = threading.Lock()
def attempt() -> None:
# A separate repository per thread: sharing one pool across threads is its own
# source of confusion, and the race being tested is in the database, not the pool.
thread_repository = ComputeRepository(_SCRATCH, min_size=1, max_size=2, passwords=None)
try:
barrier.wait(timeout=10)
outcome = thread_repository.redeem_pairing_code(minted["code"], name)
with lock:
results.append(outcome)
finally:
thread_repository.close()
threads = [threading.Thread(target=attempt) for _ in range(2)]
for thread in threads:
thread.start()
for thread in threads:
thread.join(timeout=30)
assert len(results) == 2, [r.get("code") for r in results]
successes = [r for r in results if r["ok"]]
assert len(successes) == 1, f"{len(successes)} redemptions succeeded from one code"
with repository._connection() as connection:
count = connection.execute(
"SELECT count(*) FROM public.compute_nodes WHERE name = %s", (name,)
).fetchone()["count"]
assert count == 1, f"a raced code produced {count} nodes"
def test_a_spent_code_is_not_offered_as_a_candidate(pairing_world) -> None:
"""The filter that made the sequential single-use test unable to fail.
Worth its own assertion because it is the *first* of the two guards, and the one
that does the work in every ordinary case. If this were dropped, redemption of a
spent code would reach the claim, and only the claim's `used_at IS NULL` would stop
it - one mechanism instead of two, with the race window opened up accordingly.
"""
repository, organization = pairing_world
name = f"filter-{uuid.uuid4().hex[:8]}"
minted = _mint(repository, organization, name)
repository.redeem_pairing_code(minted["code"], name)
with repository._connection() as connection:
live = connection.execute(
"SELECT count(*) FROM public.node_pairing_codes WHERE used_at IS NULL AND id = %s",
(minted["id"],),
).fetchone()["count"]
assert live == 0, "a spent code is still offered as a candidate"