dd26e7c751
Every nightly run had been failing since 20 Sep, writing the reason only to
a log file nobody reads. The last backup on the share was from 19 Sep.
Root cause: a sudoers rule that names arguments matches only that exact
argument list. The rule allowed
sudo /usr/bin/sqlite3 /var/.../db.sqlite3
but the script ran
sudo sqlite3 /var/.../db.sqlite3 ".backup '/tmp/.../db'"
The second argument was not in the rule, so it did not match, sudo fell
through to the password-required bdm ALL=(ALL) ALL, and cron has no
terminal to prompt on. Reproduced with -n, so neither call could prompt:
sudo -n sqlite3 <db> # 1 arg, matches -> exit 0
sudo -n sqlite3 <db> "SELECT 1" # 2 args, no match -> password required
Two further problems were hidden behind that early exit:
The cleanup phase called sudo mount/umount/find inline, but sudoers only
whitelisted helper scripts that existed and were never invoked. The 60-day
retention was dead too. The old cleanup also could not distinguish "share
not mounted" from "nothing to delete": find matched nothing and the script
exited 0 reporting a healthy run.
The helpers that *were* whitelisted lived under /home/bdm and are owned by
bdm, so the grant was equivalent to unrestricted root.
Changes:
Move each privileged step into a root-owned helper in /usr/local/sbin,
deployed from the repo as source of truth. This avoids depending on fragile
argument globs and lets the rule constrain the one argument that varies.
vw-backup-db re-validates the destination path itself rather than trusting
the sudoers pattern.
Call the helpers from the script and drop the inline sudo. Verify the
snapshot with PRAGMA integrity_check before uploading, and abort rather
than shipping a copy that does not verify. Check smbclient's exit status
directly instead of piping into grep, which would have let SIGPIPE mask a
failed upload.
Add a systemd staleness watchdog so this cannot hide again. It mounts the
share read-only and separates STALE (nothing recent) from CANNOT VERIFY
(could not mount), which mean different things.
Also chmod 0440 /etc/sudoers.d/bdm, which visudo was warning about.
Tested with a cold sudo cache, since timestamp_type=global means an
interactive sudo -v otherwise masks the failure. A real cron invocation
succeeded end to end: snapshot, integrity check, upload, and prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
66 lines
2.0 KiB
Bash
Executable File
66 lines
2.0 KiB
Bash
Executable File
#!/bin/bash
|
|
# Fail loudly when the vaultwarden backup has gone stale.
|
|
#
|
|
# The backup job itself is only visible by reading a log file, which is how a
|
|
# nine-day outage went unnoticed. This runs from a systemd timer as root, and
|
|
# exits non-zero when the newest backup on the share is older than the
|
|
# threshold, so the failure shows up in "systemctl status" and the journal.
|
|
#
|
|
# It covers both ways the backup can stop: the job failing, and the job not
|
|
# running at all.
|
|
|
|
set -uo pipefail
|
|
|
|
THRESHOLD_DAYS=2
|
|
MNT="/mnt/netbackup-check"
|
|
SMB_HOST="192.168.123.2"
|
|
SMB_SHARE="NetBackup"
|
|
CRED="/root/.smbcredentials"
|
|
TAG="vaultwarden-backup-check"
|
|
|
|
note() { logger -t "$TAG" -- "$*"; echo "$*"; }
|
|
|
|
cleanup() {
|
|
if mountpoint -q "$MNT"; then
|
|
umount "$MNT" 2>/dev/null || true
|
|
fi
|
|
rmdir "$MNT" 2>/dev/null || true
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
if [ ! -f "$CRED" ]; then
|
|
note "STALE CHECK FAILED - credentials missing at ${CRED}"
|
|
exit 2
|
|
fi
|
|
|
|
mkdir -p "$MNT"
|
|
|
|
# Read-only, and not a system mount, so an unreachable NAS cannot wedge the
|
|
# system. A failure to mount is reported as "cannot verify", not as "stale",
|
|
# because those mean different things.
|
|
if ! mount -t cifs "//${SMB_HOST}/${SMB_SHARE}" "$MNT" \
|
|
-o "credentials=${CRED},ro,iocharset=utf8,vers=3.0" 2>/dev/null; then
|
|
note "CANNOT VERIFY - could not mount //${SMB_HOST}/${SMB_SHARE} to check backup freshness"
|
|
exit 2
|
|
fi
|
|
|
|
newest=$(find "$MNT/Vault" -maxdepth 1 -type f -name 'vaultwarden-*.db' -printf '%T@ %p\n' 2>/dev/null \
|
|
| sort -rn | head -1 | cut -d' ' -f2-)
|
|
|
|
if [ -z "$newest" ]; then
|
|
note "STALE - no vaultwarden backups found on the share at all"
|
|
exit 1
|
|
fi
|
|
|
|
# Age in days, rounded down.
|
|
age_days=$(( ( $(date +%s) - $(stat -c %Y "$newest") ) / 86400 ))
|
|
newest_name=$(basename "$newest")
|
|
|
|
if [ "$age_days" -ge "$THRESHOLD_DAYS" ]; then
|
|
note "STALE - newest backup is ${newest_name}, ${age_days} day(s) old (threshold ${THRESHOLD_DAYS})"
|
|
exit 1
|
|
fi
|
|
|
|
note "OK - newest backup is ${newest_name}, ${age_days} day(s) old"
|
|
exit 0
|