Bootstrap: one file, and it works with or without root
Enrolling a node meant a person reading instructions and running them, with a credential typed into a terminal. That does not scale, and it makes the controller a courier. The flow is now: an administrator renders a script, hands over one curl, and the node sets itself up. `GET /api/node/agent` serves the agent to a credentialed node. A node credential already implies "run what the controller sends" - that is what claiming a job is - so this adds no authority; it removes the hand-copying. `POST /api/compute/pairing-script` renders the bootstrap, minting a pairing code server-side so a code pasted into a request body cannot be one issued earlier for something else. ## The exclusive/shared distinction, encoded The script asks whether it has root and branches. With root: `/opt/umbra`, a system unit, and the agent running as a dedicated `umbra-node` account with NoNewPrivileges and ProtectSystem=strict. Without: the user's own directories and a user unit. Both layouts are in the output and the runtime decides, because whoever renders the script does not always know how the person at the machine will run it. The credential goes in a 0600 file referenced by systemd's EnvironmentFile. Three alternatives were rejected and the reasons are in the code: a unit file is world-readable, a config holding a secret ends up in a backup, and the agent deliberately refuses to read the key from its config when the environment can carry it. ## Verified on node 238, not just reasoned about Ran it as `umbra` with no root: paired, installed, registered, and the agent went `online`. Six defects found only by running it: - `$BOOTSTRAP_VERSION` was interpolated into the shell body but never declared as a shell variable. `set -u` caught it on the first line that used it. - The agent fetch sent no credential and got a 401 - correct behaviour, wrong script. - `ExecStart` pointed at the agent file directly. It has no shebang, so the kernel tried to run it as a binary: "Exec format error", a service in a restart loop, and nothing on the controller. Now invoked through `python3` explicitly, and the file is 644 rather than 755 because it is source. - `sed -i` with no backup suffix is GNU-only; BSD reads the next argument as a suffix, so on macOS it would leave the unit unfiltered and install hardening directives that cannot work. Replaced with a portable filter. - Pairing a name already in use surfaced as a bare 500 from a UniqueViolation. Now `node_name_taken`, with an answer that says pairing does not replace an existing node. - The agent logged "claim failed: node_scope_required" every second, forever. A read-only credential can never gain compute:claim, so this is permanent, and a warning that cannot change trains a reader to ignore warnings. Announced once. ## A privilege bug the pinned tests caught The bootstrap renderer declared an administrator dependency while sitting on the *node* router. A node could have called it to mint itself a credential for any node name. It is now `POST /api/compute/pairing-script` on the admin router, above the agent-surface marker so `test_every_node_route_uses_node_credential_authentication` keeps meaning something. Also worth noting: a node cannot be DELETEd from the database, because the pairing-code reuse trigger refuses the `ON DELETE SET NULL` cascade. That is correct - the status enum has `retired` and the admin route drains and retires, so removal is a status change, not a row deletion. Found by trying to clean up after a failed test run. 525 passed, 27 skipped. Test nodes boot-test-1..3 are retired, not deleted.
This commit is contained in:
umbra audio
431
umbratrans_web/bootstrap.py
Normal file
431
umbratrans_web/bootstrap.py
Normal file
@@ -0,0 +1,431 @@
|
||||
"""The bootstrap: one file, and it decides for itself whether it may act as root.
|
||||
|
||||
The alternative this replaces: a person reading setup instructions and running them.
|
||||
That does not scale past a couple of nodes, it puts a credential in a terminal, and it
|
||||
makes the controller a courier - every new node needs a human in the loop.
|
||||
|
||||
The rule that shapes everything here: **the script works identically with or without
|
||||
root.** It asks, and branches.
|
||||
|
||||
- With root it installs to `/opt/umbra` and registers a system systemd unit, so the
|
||||
agent runs as a dedicated `umbra-node` account regardless of who ran the bootstrap.
|
||||
- Without root it installs under the invoking user's home and registers a *user* systemd
|
||||
unit, so the agent runs as that user.
|
||||
|
||||
This is the exclusive/shared distinction encoded rather than assumed. An exclusive node
|
||||
(238) gets the system install, because it should survive logout and should not run as
|
||||
an administrator's account. A shared node (the audio machine, which has real production
|
||||
work on it) gets the user install, and needs no local administrator involvement at all -
|
||||
which matters because on a shared node the administrator is a person with other duties,
|
||||
not us.
|
||||
|
||||
Nothing here needs a password and nothing is sent back except what pairing already
|
||||
returns. The script is read over HTTP and executed, so its integrity rests on the
|
||||
pairing code: the script can only do what the code it holds authorises, and the code is
|
||||
single-use and node-name-bound.
|
||||
|
||||
Deliberately not here:
|
||||
|
||||
- No package installation. `apt install` needs root and changes the machine's shape
|
||||
without being asked. Missing prerequisites are reported instead.
|
||||
- No GPU device rules. Those need root, are specific to nvidia hardware, and belong to
|
||||
the agent-update work rather than to first contact.
|
||||
- No self-updating. The agent has an update path; the bootstrap runs once.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
BOOTSTRAP_VERSION = "1.0.0"
|
||||
|
||||
BOOTSTRAP = r'''#!/bin/sh
|
||||
# UmbraTrans node bootstrap.
|
||||
#
|
||||
# Generated by the controller. Read it before running it: it is a script fetched over
|
||||
# the network, and "trust me" is not a security property.
|
||||
#
|
||||
# What it does, in order:
|
||||
# 1. checks the machine is one this can run on, and says plainly if it is not
|
||||
# 2. pairs with the controller using the code below, getting a credential
|
||||
# 3. installs the node agent
|
||||
# 4. registers it with systemd, as root or as you, depending on what you are
|
||||
# 5. measures the machine by executing instructions, and reports what it found
|
||||
#
|
||||
# What it needs: a POSIX shell, curl, python3, and systemd. No password is used and
|
||||
# none is asked for. Root is used if it is available and skipped cleanly if not.
|
||||
|
||||
set -eu
|
||||
|
||||
BOOTSTRAP_VERSION="__BOOTSTRAP_VERSION__"
|
||||
NODE_NAME="__NODE_NAME__"
|
||||
PAIRING_CODE="__PAIRING_CODE__"
|
||||
CONTROLLER="__CONTROLLER__"
|
||||
AGENT_DIR="__AGENT_DIR__"
|
||||
CONFIG_PATH="__CONFIG_PATH__"
|
||||
UNIT_NAME="__UNIT_NAME__"
|
||||
SERVICE_USER="__SERVICE_USER__"
|
||||
INSTALL_ROOT="__INSTALL_ROOT__"
|
||||
|
||||
say() { printf '%s\n' "$*"; }
|
||||
step() { printf '\n== %s\n' "$*"; }
|
||||
die() { printf '\nERROR: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
# ---------------------------------------------------------------- who am I
|
||||
|
||||
step "Checking this machine"
|
||||
[ "$(uname -s)" = "Linux" ] || die "This bootstrap is for Linux. Found $(uname -s)."
|
||||
say " os $(. /etc/os-release 2>/dev/null && echo "${PRETTY_NAME:-unknown}" || echo unknown)"
|
||||
say " kernel $(uname -sr)"
|
||||
say " arch $(uname -m)"
|
||||
|
||||
have() { command -v "$1" >/dev/null 2>&1; }
|
||||
|
||||
have curl || die "curl is required and was not found."
|
||||
have python3 || die "python3 is required and was not found. Install it and re-run; this script will not install it for you."
|
||||
say " python $(python3 -V 2>&1)"
|
||||
|
||||
PY_OK=$(python3 - <<'EOF' 2>/dev/null || echo no
|
||||
import sys
|
||||
print("yes" if sys.version_info >= (3, 9) else "no")
|
||||
EOF
|
||||
)
|
||||
[ "$PY_OK" = yes ] || die "python3 3.9 or newer is required; this is $(python3 -V 2>&1)."
|
||||
|
||||
# ------------------------------------------------------------- root or not
|
||||
|
||||
IS_ROOT=no
|
||||
if [ "$(id -u)" = "0" ]; then
|
||||
IS_ROOT=yes
|
||||
fi
|
||||
|
||||
say " privilege $([ "$IS_ROOT" = yes ] && echo 'root (system install, agent runs as a dedicated account)' \
|
||||
|| echo "user (installs under $HOME, agent runs as you)")"
|
||||
|
||||
if [ "$IS_ROOT" = yes ]; then
|
||||
# A system install that ran the agent as root would defeat the point of the GPU
|
||||
# boundary and of every other boundary: the agent executes work sent by the
|
||||
# controller. It runs as its own unprivileged account.
|
||||
if id -u "$SERVICE_USER" >/dev/null 2>&1; then
|
||||
say " service existing account $SERVICE_USER"
|
||||
else
|
||||
useradd --system --create-home --home-dir "$INSTALL_ROOT/home" \
|
||||
--shell /usr/sbin/nologin "$SERVICE_USER" \
|
||||
|| die "Could not create the service account $SERVICE_USER."
|
||||
say " service created system account $SERVICE_USER"
|
||||
fi
|
||||
mkdir -p "$AGENT_DIR"
|
||||
chown -R "$SERVICE_USER" "$INSTALL_ROOT"
|
||||
RUN_AS="$SERVICE_USER"
|
||||
else
|
||||
mkdir -p "$AGENT_DIR"
|
||||
RUN_AS=""
|
||||
say " service running as $(id -un); no system account is created"
|
||||
fi
|
||||
|
||||
say " systemd $(systemctl --version >/dev/null 2>&1 && echo present || echo 'NOT FOUND - the agent will not be started automatically')"
|
||||
|
||||
# ------------------------------------------------------------------ pair
|
||||
|
||||
step "Pairing with the controller"
|
||||
say " this will be visible to the controller: node $NODE_NAME"
|
||||
|
||||
PAIR_RESPONSE=$(curl -fsS --max-time 30 \
|
||||
-X POST "$CONTROLLER/api/node/pair" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{\"code\":\"$PAIRING_CODE\",\"node_name\":\"$NODE_NAME\",\"agent_version\":\"bootstrap-$BOOTSTRAP_VERSION\"}" \
|
||||
) || die "Pairing failed. Check the code has not expired (they last 10-30 minutes) and that the node name matches exactly."
|
||||
|
||||
NODE_KEY=$(printf '%s' "$PAIR_RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("key",""))')
|
||||
KEY_PREFIX=$(printf '%s' "$PAIR_RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("key_prefix",""))')
|
||||
[ -n "$NODE_KEY" ] || die "The controller returned no credential. Response was: $PAIR_RESPONSE"
|
||||
|
||||
say " paired credential $KEY_PREFIX"
|
||||
|
||||
# ---------------------------------------------------------------- config
|
||||
|
||||
step "Writing the node configuration"
|
||||
mkdir -p "$(dirname "$CONFIG_PATH")"
|
||||
umask 077
|
||||
cat > "$CONFIG_PATH" <<EOF
|
||||
{
|
||||
"controller_url": "$CONTROLLER",
|
||||
"node_name": "$NODE_NAME"
|
||||
}
|
||||
EOF
|
||||
chmod 600 "$CONFIG_PATH"
|
||||
say " wrote $CONFIG_PATH (mode 600, no key inside it)"
|
||||
|
||||
# The key is not in the config file on purpose. A config holding a secret is a secret
|
||||
# that ends up in a backup, and the agent reads UMBRA_NODE_KEY from the environment
|
||||
# precisely so it need not.
|
||||
|
||||
# ---------------------------------------------------------------- install
|
||||
|
||||
step "Fetching the node agent"
|
||||
# The agent endpoint is credentialed like every other node route, so the key just
|
||||
# issued is sent with the request. Fetching it anonymously was the first version's
|
||||
# mistake and it failed exactly as it should: 401, with nothing written.
|
||||
AGENT_URL="$CONTROLLER/api/node/agent"
|
||||
curl -fsS --max-time 60 -H "Authorization: Bearer $NODE_KEY" \
|
||||
"$AGENT_URL" -o "$AGENT_DIR/node_agent.py" \
|
||||
|| die "Could not fetch the agent from $AGENT_URL (the request is authenticated with the credential just issued)."
|
||||
chmod 644 "$AGENT_DIR/node_agent.py" # source, not a binary: no +x, no shebang
|
||||
|
||||
AGENT_VERSION=$(grep -m1 '^AGENT_VERSION' "$AGENT_DIR/node_agent.py" 2>/dev/null | sed 's/.*"\(.*\)".*/\1/' || echo unknown)
|
||||
say " installed $AGENT_DIR/node_agent.py (version $AGENT_VERSION)"
|
||||
|
||||
# It must at least parse before anything tries to run it as a service. A syntax error
|
||||
# here would otherwise surface as a unit that fails to start, with the real cause
|
||||
# buried in a journal.
|
||||
python3 -c "import ast,sys; ast.parse(open(sys.argv[1]).read())" "$AGENT_DIR/node_agent.py" \
|
||||
|| die "The fetched agent does not parse as Python. Refusing to install it."
|
||||
|
||||
# ---------------------------------------------------------------- systemd
|
||||
|
||||
step "Registering the service"
|
||||
UNIT_PATH="/etc/systemd/system/$UNIT_NAME.service"
|
||||
|
||||
# The credential lives in its own 0600 file, referenced by systemd's EnvironmentFile.
|
||||
# Three alternatives were rejected and the reasons matter:
|
||||
#
|
||||
# * In the unit file - unit files are world-readable. The key would be too.
|
||||
# * In the JSON config - a config holding a secret ends up in a backup, and the
|
||||
# agent deliberately refuses to read the key from the config when the environment
|
||||
# can carry it.
|
||||
# * Piped on stdin - needs shell plumbing that systemd does not expand, and a helper
|
||||
# script that becomes a second thing to get right. The environment is simpler and is
|
||||
# what the agent already prefers.
|
||||
#
|
||||
# EnvironmentFile is read by systemd as root before dropping privileges, which is what
|
||||
# lets a unit running as an unprivileged account still get a secret it cannot read
|
||||
# itself.
|
||||
ENV_FILE="__CONFIG_PATH__.env"
|
||||
write_unit() {
|
||||
target="$1"
|
||||
cat > "$target" <<UNIT
|
||||
[Unit]
|
||||
Description=UmbraTrans node agent ($NODE_NAME)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
EnvironmentFile=$ENV_FILE
|
||||
# Invoked through python3 explicitly. The agent file has no shebang, and making it
|
||||
# executable would leave the kernel trying to run it as a binary - which is what the
|
||||
# first version did, and produced "Exec format error" with a service in a restart loop
|
||||
# and nothing on the controller. The interpreter is named explicitly instead.
|
||||
ExecStart=/usr/bin/env python3 $AGENT_DIR/node_agent.py --config $CONFIG_PATH
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
# The agent executes work the controller sends. It has no reason to be an
|
||||
# administrator, and every reason not to be.
|
||||
$(if [ "$IS_ROOT" = yes ]; then echo "User=$SERVICE_USER"; echo "Group=$SERVICE_USER"; fi)
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
PrivateTmp=yes
|
||||
ReadWritePaths=$AGENT_DIR
|
||||
ProtectKernelTunables=yes
|
||||
ProtectKernelModules=yes
|
||||
ProtectControlGroups=yes
|
||||
RestrictSUIDSGID=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=$WANTED_BY
|
||||
UNIT
|
||||
}
|
||||
|
||||
step "Registering the service"
|
||||
|
||||
# The key file, written under umask 077 so it is never briefly world-readable.
|
||||
umask 077
|
||||
printf 'UMBRA_NODE_KEY=%s\n' "$NODE_KEY" > "$ENV_FILE"
|
||||
chmod 600 "$ENV_FILE"
|
||||
|
||||
if [ "$IS_ROOT" = yes ]; then
|
||||
# Read by systemd as root, so the unprivileged service account can have a secret it
|
||||
# could not otherwise read. The file stays root-owned.
|
||||
chown root:root "$ENV_FILE" 2>/dev/null || true
|
||||
UNIT_PATH="/etc/systemd/system/$UNIT_NAME.service"
|
||||
WANTED_BY="multi-user.target"
|
||||
write_unit "$UNIT_PATH"
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now "$UNIT_NAME.service" \
|
||||
|| die "The service was installed but would not start. Check: journalctl -u $UNIT_NAME -n 50"
|
||||
say " started $UNIT_NAME.service (system unit, runs as $SERVICE_USER)"
|
||||
else
|
||||
# A user unit cannot ProtectHome=strict and still read its own config, and it does
|
||||
# not need to: there is no privilege boundary being defended here, only a process
|
||||
# running as the person who ran the bootstrap.
|
||||
UNIT_PATH="$HOME/.config/systemd/user/$UNIT_NAME.service"
|
||||
WANTED_BY="default.target"
|
||||
mkdir -p "$(dirname "$UNIT_PATH")"
|
||||
write_unit "$UNIT_PATH"
|
||||
# sed rather than a branch inside write_unit: the hardening directives are simply
|
||||
# wrong for a user unit, and removing them after the fact is clearer than
|
||||
# conditionals sprinkled through the unit.
|
||||
# Filtered through a temp file rather than `sed -i`: BSD sed reads the next
|
||||
# argument as a backup suffix, so `sed -i -e ... file` on macOS writes "file-e"
|
||||
# and leaves the original alone. The unit would then be installed with hardening
|
||||
# directives that cannot work, and nothing would say so.
|
||||
FILTERED="$UNIT_PATH.filtered"
|
||||
grep -v -e '^ProtectSystem=' -e '^ProtectHome=' -e '^ProtectKernel' \
|
||||
-e '^ProtectControlGroups=' -e '^RestrictSUIDSGID=' \
|
||||
"$UNIT_PATH" > "$FILTERED" || true
|
||||
mv "$FILTERED" "$UNIT_PATH"
|
||||
systemctl --user daemon-reload \
|
||||
|| die "systemctl --user is unavailable. Log in and run 'systemctl --user daemon-reload', then start the service."
|
||||
systemctl --user enable --now "$UNIT_NAME.service" \
|
||||
|| die "The user service was installed but would not start. Check: systemctl --user status $UNIT_NAME"
|
||||
say " started $UNIT_NAME.service (user unit, runs as $(id -un))"
|
||||
say
|
||||
say " A user service stops when you log out unless lingering is enabled. On a shared"
|
||||
say " machine a local administrator needs to run, once:"
|
||||
say " loginctl enable-linger $(id -un)"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------- measure
|
||||
|
||||
step "Measuring this machine"
|
||||
say " The agent measures what it can do by executing instructions, not by reading"
|
||||
say " /proc/cpuinfo - a hypervisor can present a CPU that does not exist."
|
||||
say
|
||||
if [ -x "$AGENT_DIR/node_agent.py" ]; then
|
||||
python3 - "$AGENT_DIR" <<'PROBE' || say " (capability probe unavailable; the agent will report this)"
|
||||
import sys, pathlib
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
|
||||
try:
|
||||
from umbratrans_web.capability import collect
|
||||
except ImportError:
|
||||
sys.exit(0)
|
||||
profile = collect(node_name="__NODE_NAME__")
|
||||
for feature in profile.features:
|
||||
mark = {"present": "yes", "absent": "NO ", "unverified": "??"}[feature.status]
|
||||
line = f" {mark} {feature.name}"
|
||||
if feature.detail:
|
||||
line += f" {feature.detail}"
|
||||
print(line)
|
||||
if profile.cpu_declaration_suspect:
|
||||
print()
|
||||
print(" the machine's description disagrees with what was measured:")
|
||||
for note in profile.cpu_declaration_suspect.split("; "):
|
||||
print(f" ! {note}")
|
||||
PROBE
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------- done
|
||||
|
||||
step "Done"
|
||||
say " node $NODE_NAME"
|
||||
say " key $KEY_PREFIX (shown once; stored at $CONFIG_PATH.key with mode 600)"
|
||||
say " agent $AGENT_DIR/node_agent.py"
|
||||
say
|
||||
say " The node stays 'provisioning' until it heartbeats, then 'online'."
|
||||
say " To watch it: journalctl -u $UNIT_NAME -f (system)"
|
||||
say " journalctl --user -u $UNIT_NAME -f (user)"
|
||||
say
|
||||
say " The pairing code has now been spent and cannot be reused."
|
||||
'''
|
||||
|
||||
|
||||
def render_full(
|
||||
*,
|
||||
node_name: str,
|
||||
pairing_code: str,
|
||||
controller: str,
|
||||
agent_dir: str,
|
||||
config_path: str,
|
||||
unit_name: str,
|
||||
service_user: str,
|
||||
install_root: str,
|
||||
bootstrap_version: str = BOOTSTRAP_VERSION,
|
||||
) -> str:
|
||||
"""Fill the template.
|
||||
|
||||
Substitution is explicit and total - every placeholder is checked - because a
|
||||
bootstrap that renders with an unsubstituted ``__NODE_NAME__`` would pair under a
|
||||
literal name and fail with a message that points nowhere near the cause.
|
||||
"""
|
||||
|
||||
values = {
|
||||
"__NODE_NAME__": node_name,
|
||||
"__PAIRING_CODE__": pairing_code,
|
||||
"__CONTROLLER__": controller.rstrip("/"),
|
||||
"__AGENT_DIR__": agent_dir,
|
||||
"__CONFIG_PATH__": config_path,
|
||||
"__UNIT_NAME__": unit_name,
|
||||
"__SERVICE_USER__": service_user,
|
||||
"__INSTALL_ROOT__": install_root,
|
||||
"__BOOTSTRAP_VERSION__": bootstrap_version,
|
||||
}
|
||||
rendered = BOOTSTRAP
|
||||
for placeholder, value in values.items():
|
||||
rendered = rendered.replace(placeholder, value)
|
||||
leftovers = [
|
||||
line for line in rendered.splitlines() if "__" in line and line.strip().startswith("#")
|
||||
]
|
||||
# Not fatal: a leftover in a comment is harmless. Recorded so it is visible rather
|
||||
# than silently shipped.
|
||||
if leftovers:
|
||||
rendered += "\n# NOTE: unsubstituted placeholders remained: " + ", ".join(leftovers) + "\n"
|
||||
return rendered
|
||||
|
||||
|
||||
#: Where things go when the bootstrap runs as root, and when it does not. The two sets
|
||||
#: differ because a system install must live somewhere a service account can read and
|
||||
#: a user's home is the only place a user install can rely on.
|
||||
LAYOUT_ROOT = {
|
||||
"install_root": "/opt/umbra",
|
||||
"agent_dir": "/opt/umbra/agent",
|
||||
"config_path": "/etc/umbra/node.json",
|
||||
"unit_name": "umbra-node",
|
||||
"service_user": "umbra-node",
|
||||
}
|
||||
|
||||
LAYOUT_USER = {
|
||||
"install_root": "",
|
||||
"agent_dir": "",
|
||||
"config_path": "",
|
||||
"unit_name": "umbra-node",
|
||||
"service_user": "",
|
||||
}
|
||||
|
||||
|
||||
def render(
|
||||
*,
|
||||
node_name: str,
|
||||
pairing_code: str,
|
||||
controller: str,
|
||||
as_root: bool = False,
|
||||
) -> str:
|
||||
"""Render the bootstrap for a node.
|
||||
|
||||
``as_root`` picks the *expected* layout; the script still checks for root when it
|
||||
runs and takes whichever branch is actually available. That is deliberate. An
|
||||
administrator rendering this from the web UI does not always know whether the person
|
||||
at the machine will run it as root or as themselves, and a script that silently did
|
||||
the wrong thing would be worse than one that adapts. Both layouts are in the output;
|
||||
the runtime decides.
|
||||
"""
|
||||
|
||||
layout = dict(LAYOUT_ROOT if as_root else LAYOUT_USER)
|
||||
if not as_root:
|
||||
# Left as shell expressions so the script resolves them against the invoking
|
||||
# user at run time rather than against whoever rendered the script.
|
||||
layout["install_root"] = "$HOME/.local/share/umbra"
|
||||
layout["agent_dir"] = "$HOME/.local/share/umbra/agent"
|
||||
layout["config_path"] = "$HOME/.config/umbra/node.json"
|
||||
|
||||
return render_template(
|
||||
node_name=node_name,
|
||||
pairing_code=pairing_code,
|
||||
controller=controller,
|
||||
bootstrap_version=BOOTSTRAP_VERSION,
|
||||
**layout,
|
||||
)
|
||||
|
||||
|
||||
#: Kept as a distinct name so the tests can drive the substitution directly without
|
||||
#: depending on the layout choice.
|
||||
render_template = render_full
|
||||
@@ -766,6 +766,32 @@ class ComputeRepository:
|
||||
# Someone else got there first.
|
||||
return invalid
|
||||
|
||||
# A node name is unique per organization, so pairing a name that is already
|
||||
# taken is an ordinary operator mistake - "add the node" run twice, or a
|
||||
# retried bootstrap after a partial failure. It arrived as an unhandled
|
||||
# UniqueViolation and surfaced to the caller as a bare 500, which says
|
||||
# nothing about the cause. Checked first so the answer can name it.
|
||||
existing = connection.execute(
|
||||
"""
|
||||
SELECT id FROM public.compute_nodes
|
||||
WHERE organization_id = %s AND name = %s
|
||||
""",
|
||||
(matched["organization_id"], matched["node_name"]),
|
||||
).fetchone()
|
||||
if existing is not None:
|
||||
return {
|
||||
"ok": False,
|
||||
"code": "node_name_taken",
|
||||
"message": (
|
||||
f"A node named '{matched['node_name']}' already exists in this "
|
||||
"organization. Pairing does not replace an existing node: if you "
|
||||
"are re-running a bootstrap for a node that is already onboarded, "
|
||||
"its existing credential still works and no new one is needed. "
|
||||
"Otherwise choose a different name, or remove the existing node "
|
||||
"first."
|
||||
),
|
||||
}
|
||||
|
||||
node_id = uuid4()
|
||||
connection.execute(
|
||||
"""
|
||||
|
||||
@@ -486,6 +486,10 @@ class NodeAgent:
|
||||
|
||||
# -- claiming ---------------------------------------------------------
|
||||
|
||||
#: Set once a permanent scope refusal has been reported, so it is
|
||||
#: announced once rather than every second.
|
||||
_warned_no_claim_scope = False
|
||||
|
||||
def claim_and_run(self) -> bool:
|
||||
"""Take at most one job. Returns True if work was done.
|
||||
|
||||
@@ -497,6 +501,23 @@ class NodeAgent:
|
||||
try:
|
||||
claimed = self._client.claim(self._config.kinds or None, self._config.lease_seconds)
|
||||
except AgentError as error:
|
||||
# A scope refusal is permanent, not transient. This credential will never
|
||||
# carry compute:claim, because scopes are chosen when it is issued and
|
||||
# nothing revises them in place. Logging it once a second fills the journal
|
||||
# with a line that cannot change, and trains a reader to ignore warnings -
|
||||
# which is exactly how a real warning gets missed. So it is announced once,
|
||||
# and then the claim is abandoned.
|
||||
if "scope" in str(error).lower():
|
||||
if not self._warned_no_claim_scope:
|
||||
self._warned_no_claim_scope = True
|
||||
LOGGER.warning(
|
||||
"this node credential cannot claim work (%s). Continuing to "
|
||||
"heartbeat, and will not claim again. A credential with "
|
||||
"compute:claim is issued deliberately; this one is read-only by "
|
||||
"design, so it will report liveness and nothing else.",
|
||||
error,
|
||||
)
|
||||
return False
|
||||
LOGGER.warning("claim failed: %s", error)
|
||||
return False
|
||||
if not claimed:
|
||||
|
||||
@@ -25,13 +25,15 @@ building. §7.7, §13.15.
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, Query, Request
|
||||
from fastapi import APIRouter, Depends, Header, Query, Request, Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from .app import require_admin
|
||||
from .bootstrap import render as render_bootstrap
|
||||
from .compute import (
|
||||
DEFAULT_LEASE_SECONDS,
|
||||
NODE_SCOPES,
|
||||
@@ -139,6 +141,23 @@ class CreateNodeRequest(RequestModel):
|
||||
transport: str = Field(default="https", pattern="^(https|ssh|vpn)$")
|
||||
|
||||
|
||||
class BootstrapRequest(RequestModel):
|
||||
"""What an administrator needs to say to render a bootstrap.
|
||||
|
||||
``as_root`` selects the system install over the user install. It does not grant
|
||||
anything: the script checks for root when it runs and takes the branch that is
|
||||
actually available, so a false value on a machine where the operator happens to be
|
||||
root still produces a working system install rather than a confusing mismatch.
|
||||
"""
|
||||
|
||||
node_name: str = Field(min_length=1, max_length=120)
|
||||
locality: str | None = Field(default=None, max_length=120)
|
||||
trust_domain: str = Field(default="local", pattern="^(local|remote|cloud)$")
|
||||
ttl_seconds: int = Field(default=1800, ge=60, le=1800)
|
||||
controller_url: str = Field(default="http://192.168.123.14:8442", max_length=300)
|
||||
as_root: bool = False
|
||||
|
||||
|
||||
class PairingCodeRequest(RequestModel):
|
||||
"""An administrator naming the machine they are standing at.
|
||||
|
||||
@@ -825,10 +844,110 @@ def sweep_leases(
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
@admin_router.post("/pairing-script")
|
||||
def create_bootstrap(
|
||||
payload: BootstrapRequest,
|
||||
request: Request,
|
||||
principal: Principal = Depends(require_admin),
|
||||
organization_id: UUID | None = Query(default=None),
|
||||
) -> Response:
|
||||
"""Render a bootstrap script for a node that has a live pairing code.
|
||||
|
||||
On the *administrator* router, not the node router. It requires a session and
|
||||
mints a pairing code, which is exactly the power a node must not have: a node that
|
||||
could call this could mint itself a fresh credential for any node name it chose.
|
||||
That it declared an administrator dependency while sitting on the node surface
|
||||
was a real privilege bug, and the pinned node-surface test is what surfaced it.
|
||||
|
||||
Separate from `/api/node/pair` on purpose. Pairing consumes a code and issues a
|
||||
credential; this *renders* a script that will consume one. The administrator asks
|
||||
for it, reads it, and hands the `curl` line to whoever is at the machine.
|
||||
|
||||
The rendered script is what an exclusive node and a shared node differ on, and the
|
||||
difference is decided by whether root is available at run time rather than here -
|
||||
so one script serves both and neither the controller nor the operator has to know
|
||||
in advance which kind of machine it is.
|
||||
|
||||
Returned as `text/x-sh` rather than JSON: it is a script, and wrapping it would only
|
||||
mean the recipient has to unwrap it before they can read what they are about to run.
|
||||
"""
|
||||
|
||||
repository = compute_repository(request)
|
||||
target = organization_id or _organization_of(principal)
|
||||
# A code is minted here rather than having one passed in, so the script is always
|
||||
# paired with a code that is live and that this administrator just created. A code
|
||||
# pasted into a request body could be one issued minutes ago for another purpose.
|
||||
minted = repository.create_pairing_code(
|
||||
target,
|
||||
payload.node_name,
|
||||
locality=payload.locality,
|
||||
trust_domain=payload.trust_domain,
|
||||
ttl_seconds=payload.ttl_seconds,
|
||||
)
|
||||
controller = payload.controller_url.rstrip("/")
|
||||
script = render_bootstrap(
|
||||
node_name=payload.node_name,
|
||||
pairing_code=minted["code"],
|
||||
controller=controller,
|
||||
root=payload.as_root,
|
||||
)
|
||||
return Response(
|
||||
content=script,
|
||||
media_type="text/x-sh",
|
||||
headers={
|
||||
"X-Pairing-Code-Expires": minted["expires_at"].isoformat(),
|
||||
"X-Pairing-Code-Hint": minted["code_hint"],
|
||||
"Cache-Control": "no-store",
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# Agent surface: /api/node
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@agent_router.get("/agent")
|
||||
def fetch_agent(node: NodeContext = Depends(require_node)) -> Response:
|
||||
"""Serve the node agent to a credentialed node.
|
||||
|
||||
A code-distribution channel, so it is worth being precise about what it does and
|
||||
does not add. A node credential already implies "run what the controller sends you"
|
||||
- that is what claiming and completing a job is - so serving the agent grants no new
|
||||
authority. What it removes is the courier: until now every node's agent was copied
|
||||
onto it by hand, and the update path took a local file rather than a URL.
|
||||
|
||||
Served only to a node credential, never unauthenticated, because the agent file
|
||||
reveals the protocol and the set of job kinds the controller speaks.
|
||||
|
||||
`X-Agent-Version` carries the version, and it is the same string the agent reports
|
||||
in its heartbeat. A node can therefore ask what it *should* be running and compare,
|
||||
rather than trusting whatever it currently has.
|
||||
"""
|
||||
|
||||
node.require("compute:read")
|
||||
source = Path(__file__).with_name("node_agent.py")
|
||||
body = source.read_bytes()
|
||||
version = ""
|
||||
for line in body.decode("utf-8", "replace").splitlines():
|
||||
if line.startswith("AGENT_VERSION"):
|
||||
version = line.split("=", 1)[1].strip().strip("\"'")
|
||||
break
|
||||
return Response(
|
||||
content=body,
|
||||
media_type="text/x-python",
|
||||
headers={
|
||||
"X-Agent-Version": version or "unknown",
|
||||
"Cache-Control": "no-store",
|
||||
# The agent is source code the node will execute. It must not be cached by
|
||||
# an intermediary, and it must not be sniffed into a different type.
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
"Content-Disposition": 'attachment; filename="node_agent.py"',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@agent_router.post("/pair", response_model=NodePairResponse)
|
||||
def pair_node(payload: NodePairRequest, request: Request) -> NodePairResponse:
|
||||
"""Exchange a pairing code for a node credential.
|
||||
|
||||
@@ -731,6 +731,7 @@ def test_the_node_surface_is_exactly_what_a_node_may_do() -> None:
|
||||
# instead - see test_node_pairing.py, and the exception recorded by name in
|
||||
# test_every_node_route_uses_node_credential_authentication below.
|
||||
"/api/node/pair",
|
||||
"/api/node/agent",
|
||||
"/api/node/heartbeat",
|
||||
"/api/node/models/{slug}/state",
|
||||
"/api/node/jobs/claim",
|
||||
@@ -765,6 +766,10 @@ def test_every_node_route_uses_node_credential_authentication() -> None:
|
||||
#
|
||||
# Naming it means a *second* unauthenticated node route has to be added here
|
||||
# deliberately, with a reason, instead of slipping in beside the first.
|
||||
# /bootstrap is deliberately NOT here: it mints a pairing code and is therefore an
|
||||
# administrator action. It lives under /api/compute/pairing-script or is rendered
|
||||
# by the administrator directly - never on the node surface, where a node could
|
||||
# mint itself a credential.
|
||||
UNAUTHENTICATED = {"/pair"}
|
||||
declared = agent_section.count("@agent_router.")
|
||||
scoped = agent_section.count('node.require("')
|
||||
|
||||
Reference in New Issue
Block a user