A service-account setup and boundary check for the node agent
The agent is stdlib-only - no package imports, no third-party dependencies - so it needs no virtualenv and no access to the controller's home directory. That is the whole reason the setup is short: an account with no group memberships at all, running one file from /opt on the system python. poc/poc_node_user.sh has four subcommands. setup needs root and creates the account, installs the agent, and writes a config that deliberately contains no key. verify asserts the boundary. run does both. status reports what exists and what is reachable, needing no privilege at all. The check that matters: the agent must not be able to read ~/.config/umbratrans/umbratrans.env. It cannot, and there is a second layer behind it - peer authentication maps a unix user to a database role, and umbra-poc has none, so even a leaked DSN would not connect. The two are independent and neither alone is the boundary. Writing verify produced the third instance of a bug class that has come up repeatedly in this work: a check that passes because its mechanism failed. When sudo -u could not run, every positive check correctly failed and every negative check passed - because `test -r` on an unreadable file returns false, which is exactly what "correctly denied" looks like. The script would have reported a boundary as holding while testing nothing. It now confirms the privilege drop once, up front, and abandons the run rather than reporting; status prints "unknown (no privilege drop)" rather than "no". Found by running the checks against an account it could not drop to, which is the only way to see it. Also: the agent's config now travels in a 0750 directory, because mkdtemp gives 0700 and a different user could not have read its own config out of it. The script is dry-run against an absent account, an existing unprivileged account, a missing controller, and a rootless setup invocation. It has not been run end to end, because umbra-poc does not exist yet.
This commit is contained in:
umbra audio
335
poc/poc_node_user.sh
Executable file
335
poc/poc_node_user.sh
Executable file
@@ -0,0 +1,335 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Set up and verify the node-agent service account for the compute proof of concept.
|
||||
#
|
||||
# sudo poc/poc_node_user.sh setup create umbra-poc, install the agent (needs root)
|
||||
# poc/poc_node_user.sh verify assert the boundary holds (needs sudo -u)
|
||||
# poc/poc_node_user.sh run verify, then run the whole PoC
|
||||
# poc/poc_node_user.sh status what exists right now
|
||||
#
|
||||
# Why an account at all, when the agent is one file and no dependencies: because
|
||||
# the boundary worth proving is that a worker on a node has *no* database
|
||||
# credential and no access to the controller's files. Running the agent as
|
||||
# `umbra` would make that true by assertion; running it as somebody else makes it
|
||||
# true by construction. The account is the test.
|
||||
#
|
||||
# The agent is stdlib-only, so it needs no virtualenv and no access to
|
||||
# /home/umbra. That is the whole reason the setup is this short: no group
|
||||
# memberships, no path grants, nothing to widen later.
|
||||
#
|
||||
# Nothing here touches the deployment database or the deployment service.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Overridable so the checks can be exercised against a throwaway account while
|
||||
# developing the script, and so a real deployment that already has a service user
|
||||
# can point at it rather than creating a second one.
|
||||
AGENT_USER="${UMBRA_AGENT_USER:-umbra-poc}"
|
||||
AGENT_HOME="${UMBRA_AGENT_HOME:-/var/lib/umbra-poc}"
|
||||
AGENT_DIR="${UMBRA_AGENT_DIR:-/opt/umbra/node-agent}"
|
||||
AGENT_SRC="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/umbratrans_web/node_agent.py"
|
||||
ENV_FILE="/home/umbra/.config/umbratrans/umbratrans.env"
|
||||
REPO_DIR="/home/umbra/src/UmbraTrans"
|
||||
CONFIG="$AGENT_HOME/node.json"
|
||||
|
||||
# Facts about this host, discovered rather than assumed. Overridable so the
|
||||
# script is not welded to one machine.
|
||||
CONTROLLER_URL="${UMBRA_CONTROLLER_URL:-http://127.0.0.1:8543}"
|
||||
PYTHON_BIN="${UMBRA_PYTHON_BIN:-/usr/bin/python3}"
|
||||
|
||||
pass_count=0
|
||||
fail_count=0
|
||||
|
||||
bold() { printf '\033[1m%s\033[0m\n' "$*"; }
|
||||
ok() { printf ' \033[32mok\033[0m %s\n' "$*"; pass_count=$((pass_count + 1)); }
|
||||
bad() { printf ' \033[31mFAIL\033[0m %s\n' "$*"; fail_count=$((fail_count + 1)); }
|
||||
note() { printf ' %s\n' "$*"; }
|
||||
head_() { printf '\n\033[1m%s\033[0m\n' "$*"; }
|
||||
|
||||
# -n: fail fast rather than prompting. A prompt inside a loop of checks turns
|
||||
# into a hang, and worse, into a *false pass* - see require_agent_runner.
|
||||
as_agent() { sudo -n -u "$AGENT_USER" -- "$@"; }
|
||||
|
||||
# A check that passes because its mechanism failed is worse than a check that
|
||||
# fails. When `sudo -u` cannot run, the positive checks ("can read the agent")
|
||||
# correctly fail, but every negative check ("cannot read the DSN") passes for the
|
||||
# wrong reason - the test never ran, and `test -r` on an unreadable file returns
|
||||
# false, which is exactly what "correctly denied" looks like. So the runner is
|
||||
# confirmed once, up front, and a failure there aborts rather than reporting.
|
||||
# "no" and "could not check" are different answers and a status display that
|
||||
# conflates them will report a boundary as holding when nothing was tested.
|
||||
agent_can_read() {
|
||||
if ! as_agent true 2>/dev/null; then
|
||||
printf 'unknown (no privilege drop)'
|
||||
return
|
||||
fi
|
||||
if as_agent test -r "$1" 2>/dev/null; then printf 'yes'; else printf 'no'; fi
|
||||
}
|
||||
|
||||
require_agent_runner() {
|
||||
local actual
|
||||
if ! actual="$(as_agent id -un 2>/dev/null)"; then
|
||||
bad "cannot run commands as $AGENT_USER"
|
||||
note "sudo -n -u $AGENT_USER failed. Every \"must not reach\" check below"
|
||||
note "would pass for the wrong reason, so this run is abandoned rather than"
|
||||
note "than reported. Either run this with a valid sudo credential, or add"
|
||||
note "a NOPASSWD rule for 'sudo -n -u $AGENT_USER'."
|
||||
return 1
|
||||
fi
|
||||
if [[ "$actual" != "$AGENT_USER" ]]; then
|
||||
bad "sudo -u $AGENT_USER actually ran as '$actual'"
|
||||
return 1
|
||||
fi
|
||||
ok "can run commands as $AGENT_USER (id -un = $actual)"
|
||||
return 0
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# setup
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
do_setup() {
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
echo "setup must run as root: sudo $0 setup" >&2
|
||||
return 2
|
||||
fi
|
||||
if ! command -v "$PYTHON_BIN" >/dev/null 2>&1; then
|
||||
echo "no python at $PYTHON_BIN; set UMBRA_PYTHON_BIN" >&2
|
||||
return 2
|
||||
fi
|
||||
"$PYTHON_BIN" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 10) else 1)' || {
|
||||
echo "$PYTHON_BIN is older than 3.10 and the agent needs it" >&2
|
||||
return 2
|
||||
}
|
||||
|
||||
bold "Creating $AGENT_USER"
|
||||
|
||||
if id "$AGENT_USER" >/dev/null 2>&1; then
|
||||
note "$AGENT_USER already exists; leaving the account alone"
|
||||
else
|
||||
# --system : no aging, no mail, a service account rather than a person
|
||||
# --groups "" : no supplementary groups. This is the load-bearing part -
|
||||
# group `umbra` would grant read on the repo, group `users`
|
||||
# would grant other people's world-readable files, and
|
||||
# neither is needed for an agent that is one stdlib file.
|
||||
# --nologin : there is no reason to sign in as a worker
|
||||
useradd \
|
||||
--system \
|
||||
--home-dir "$AGENT_HOME" \
|
||||
--no-create-home \
|
||||
--shell /usr/sbin/nologin \
|
||||
--groups "" \
|
||||
"$AGENT_USER"
|
||||
ok "created $AGENT_USER with no supplementary groups and no login shell"
|
||||
fi
|
||||
|
||||
install -d -o "$AGENT_USER" -g "$AGENT_USER" -m 0750 "$AGENT_HOME"
|
||||
ok "$AGENT_HOME owned by $AGENT_USER, mode 0750"
|
||||
|
||||
install -d -o root -g root -m 0755 "$AGENT_DIR"
|
||||
install -o root -g root -m 0755 "$AGENT_SRC" "$AGENT_DIR/node_agent.py"
|
||||
ok "agent installed at $AGENT_DIR/node_agent.py (root-owned, world-readable)"
|
||||
|
||||
# A config with no key in it. The key travels in the environment, so this file
|
||||
# holds nothing worth protecting and 0640 is belt-and-braces rather than the
|
||||
# thing keeping the secret out of a backup.
|
||||
if [[ ! -f "$CONFIG" ]]; then
|
||||
cat > "$CONFIG" <<JSON
|
||||
{
|
||||
"controller_url": "$CONTROLLER_URL",
|
||||
"node_name": "poc-agent-node",
|
||||
"kinds": ["sleep"],
|
||||
"lease_seconds": 30,
|
||||
"poll_interval_seconds": 0.4
|
||||
}
|
||||
JSON
|
||||
chown "$AGENT_USER:$AGENT_USER" "$CONFIG"
|
||||
chmod 0640 "$CONFIG"
|
||||
ok "config written to $CONFIG (no key in it)"
|
||||
else
|
||||
note "$CONFIG exists; leaving it alone"
|
||||
fi
|
||||
|
||||
printf '\n'
|
||||
note "controller: $CONTROLLER_URL"
|
||||
note "next: $0 verify"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# verify
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
do_verify() {
|
||||
bold "Node-agent service account: $AGENT_USER"
|
||||
|
||||
if ! id "$AGENT_USER" >/dev/null 2>&1; then
|
||||
bad "$AGENT_USER does not exist yet"
|
||||
note "run: sudo $0 setup"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! require_agent_runner; then
|
||||
printf '\n\033[31mCannot verify without a working privilege drop; 0 checks attempted.\033[0m\n'
|
||||
return 1
|
||||
fi
|
||||
|
||||
head_ "The account itself"
|
||||
local shell groups
|
||||
shell="$(getent passwd "$AGENT_USER" | cut -d: -f7)"
|
||||
groups="$(id -nG "$AGENT_USER" 2>/dev/null | tr ' ' ',')"
|
||||
note "shell : $shell"
|
||||
note "groups : ${groups:-<none>}"
|
||||
if [[ "$shell" == */nologin || "$shell" == "false" ]]; then
|
||||
ok "cannot sign in as it"
|
||||
else
|
||||
bad "shell is '$shell'; expected nologin"
|
||||
fi
|
||||
# A single group is expected: its own. Anything more is a grant waiting to
|
||||
# be used, and the agent needs none of them.
|
||||
if [[ "$groups" == "$AGENT_USER" || -z "$groups" ]]; then
|
||||
ok "no supplementary groups"
|
||||
else
|
||||
bad "extra groups: $groups"
|
||||
fi
|
||||
if id -nG "$AGENT_USER" 2>/dev/null | grep -qw sudo; then
|
||||
bad "$AGENT_USER is in the sudo group"
|
||||
else
|
||||
ok "not in the sudo group"
|
||||
fi
|
||||
|
||||
head_ "What it can reach"
|
||||
if as_agent test -r "$AGENT_DIR/node_agent.py"; then
|
||||
ok "its own code is readable"
|
||||
else
|
||||
bad "cannot read $AGENT_DIR/node_agent.py"
|
||||
fi
|
||||
if as_agent test -r "$CONFIG"; then
|
||||
ok "its own config is readable"
|
||||
else
|
||||
bad "cannot read $CONFIG"
|
||||
fi
|
||||
if as_agent "$PYTHON_BIN" -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 10) else 1)'; then
|
||||
ok "the system python runs for it"
|
||||
else
|
||||
bad "$PYTHON_BIN is unusable as $AGENT_USER"
|
||||
fi
|
||||
if as_agent "$PYTHON_BIN" "$AGENT_DIR/node_agent.py" --help >/dev/null 2>&1; then
|
||||
ok "the agent starts and prints its usage"
|
||||
else
|
||||
bad "the agent does not start as $AGENT_USER"
|
||||
fi
|
||||
|
||||
head_ "What it must not reach (these are the point)"
|
||||
if as_agent test -r "$ENV_FILE"; then
|
||||
bad "CAN READ $ENV_FILE <- the database DSN and SMTP credentials"
|
||||
else
|
||||
ok "cannot read the controller's env file (DSN, SMTP credentials)"
|
||||
fi
|
||||
if as_agent test -r "$REPO_DIR"; then
|
||||
bad "CAN READ $REPO_DIR"
|
||||
else
|
||||
ok "cannot read the repository"
|
||||
fi
|
||||
if as_agent test -x /home/umbra; then
|
||||
bad "CAN ENTER /home/umbra"
|
||||
else
|
||||
ok "cannot enter /home/umbra"
|
||||
fi
|
||||
|
||||
head_ "Database access, denied twice over"
|
||||
if as_agent psql -d umbratrans -c 'select 1' >/dev/null 2>&1; then
|
||||
bad "CAN CONNECT to the database as $AGENT_USER"
|
||||
else
|
||||
ok "peer authentication refuses the connection"
|
||||
fi
|
||||
# Even with the DSN it could not connect, because peer auth maps the unix user
|
||||
# to a database role and there is no role for it. Checked directly so the
|
||||
# refusal is not mistaken for the file permission doing the work.
|
||||
if [[ -n "$(psql -d umbratrans -tAc "select rolname from pg_roles where rolname = '$AGENT_USER'" 2>/dev/null)" ]]; then
|
||||
bad "a PostgreSQL role named $AGENT_USER exists; peer auth would accept it"
|
||||
else
|
||||
ok "no PostgreSQL role named $AGENT_USER, so the DSN would not help either"
|
||||
fi
|
||||
|
||||
head_ "The GPU, and what it means"
|
||||
local perms
|
||||
perms="$(stat -c '%A %U' /dev/nvidiactl 2>/dev/null || echo 'absent')"
|
||||
note "/dev/nvidiactl is $perms"
|
||||
if [[ "$perms" == "crw-rw-rw-"* ]]; then
|
||||
note "World-accessible. The OS does NOT stop one node from using another's"
|
||||
note "share of VRAM - placement does, and nothing below the scheduler does."
|
||||
note "A udev rule (0660 root:video plus a group) would make the kernel agree."
|
||||
fi
|
||||
|
||||
printf '\n'
|
||||
if (( fail_count == 0 )); then
|
||||
printf '\033[32m%d check(s) passed, 0 failed.\033[0m\n' "$pass_count"
|
||||
return 0
|
||||
fi
|
||||
printf '\033[31m%d check(s) passed, %d FAILED.\033[0m\n' "$pass_count" "$fail_count"
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# run
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
do_run() {
|
||||
do_verify || true
|
||||
head_ "Running the proof of concept with the agent as $AGENT_USER"
|
||||
if ! curl -fsS --max-time 5 "$CONTROLLER_URL/api/health" >/dev/null 2>&1; then
|
||||
bad "no controller at $CONTROLLER_URL"
|
||||
note "start it with: UMBRATRANS_DATABASE_DSN=\$DSN UMBRA_ENV=test \\"
|
||||
note " uv run python -m umbratrans_web serve --host 127.0.0.1 --port 8543 --repository postgres"
|
||||
return 2
|
||||
fi
|
||||
ok "controller is up at $CONTROLLER_URL"
|
||||
|
||||
if [[ $fail_count -gt 0 ]]; then
|
||||
printf '\n\033[33mRunning anyway; the boundary problems above are the interesting part.\033[0m\n'
|
||||
fi
|
||||
|
||||
local repo_dir; repo_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
( cd "$repo_dir" && exec uv run python poc/run_poc.py \
|
||||
--controller "$CONTROLLER_URL" \
|
||||
--agent-user "$AGENT_USER" \
|
||||
"$@" )
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# status
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
do_status() {
|
||||
bold "$AGENT_USER"
|
||||
if id "$AGENT_USER" >/dev/null 2>&1; then
|
||||
id "$AGENT_USER"
|
||||
note "home : $(getent passwd "$AGENT_USER" | cut -d: -f6)"
|
||||
note "shell: $(getent passwd "$AGENT_USER" | cut -d: -f7)"
|
||||
for path in "$AGENT_DIR/node_agent.py" "$CONFIG" "$ENV_FILE" "$REPO_DIR"; do
|
||||
if [[ -e "$path" ]]; then
|
||||
note "$(stat -c '%A %U:%G %n' "$path")"
|
||||
else
|
||||
note "missing $path"
|
||||
fi
|
||||
done
|
||||
printf '\n'
|
||||
note "can read agent : $(agent_can_read "$AGENT_DIR/node_agent.py")"
|
||||
note "can read env file: $(agent_can_read "$ENV_FILE")"
|
||||
else
|
||||
note "not created. run: sudo $0 setup"
|
||||
fi
|
||||
printf '\n'
|
||||
note "controller: $(curl -fsS --max-time 3 "$CONTROLLER_URL/api/health" 2>/dev/null | head -c 80 || echo unreachable)"
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
setup) shift; do_setup "$@" ;;
|
||||
verify) shift; do_verify "$@" ;;
|
||||
run) shift; do_run "$@" ;;
|
||||
status) shift; do_status "$@" ;;
|
||||
*)
|
||||
sed -n '3,20p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
@@ -240,6 +240,9 @@ def start_agent(
|
||||
}
|
||||
config_path = workdir / f"node-{agent_name}.json"
|
||||
config_path.write_text(json.dumps(config, indent=2))
|
||||
# World-readable on purpose: the config holds no secret. The key travels in
|
||||
# the environment, so a config file that had to be protected would be a sign
|
||||
# the key was in the wrong place.
|
||||
config_path.chmod(0o644)
|
||||
|
||||
agent_script = Path(__file__).resolve().parents[1] / "umbratrans_web" / "node_agent.py"
|
||||
@@ -715,6 +718,12 @@ def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
workdir = args.workdir or Path(tempfile.mkdtemp(prefix="umbra-poc-"))
|
||||
workdir.mkdir(parents=True, exist_ok=True)
|
||||
# 0750, not the 0700 mkdtemp gives and not the 0755 a default umask would.
|
||||
# The agent runs as another user and has to be able to read its config out of
|
||||
# here, and the directory has to be traversable for that to work - while the
|
||||
# node key, which is the thing worth protecting, is in the environment rather
|
||||
# than in the file.
|
||||
workdir.chmod(0o750)
|
||||
print("\033[1mUmbraTrans compute proof of concept\033[0m")
|
||||
_note(f"controller : {args.controller}")
|
||||
_note(f"workdir : {workdir}")
|
||||
|
||||
Reference in New Issue
Block a user