Files
bdm dd26e7c751 Fix the vaultwarden backup failing silently for nine days
Every nightly run had been failing since 20 Sep, writing the reason only to
a log file nobody reads. The last backup on the share was from 19 Sep.

Root cause: a sudoers rule that names arguments matches only that exact
argument list. The rule allowed

    sudo /usr/bin/sqlite3 /var/.../db.sqlite3

but the script ran

    sudo sqlite3 /var/.../db.sqlite3 ".backup '/tmp/.../db'"

The second argument was not in the rule, so it did not match, sudo fell
through to the password-required bdm ALL=(ALL) ALL, and cron has no
terminal to prompt on. Reproduced with -n, so neither call could prompt:

    sudo -n sqlite3 <db>            # 1 arg, matches  -> exit 0
    sudo -n sqlite3 <db> "SELECT 1" # 2 args, no match -> password required

Two further problems were hidden behind that early exit:

The cleanup phase called sudo mount/umount/find inline, but sudoers only
whitelisted helper scripts that existed and were never invoked. The 60-day
retention was dead too. The old cleanup also could not distinguish "share
not mounted" from "nothing to delete": find matched nothing and the script
exited 0 reporting a healthy run.

The helpers that *were* whitelisted lived under /home/bdm and are owned by
bdm, so the grant was equivalent to unrestricted root.

Changes:

Move each privileged step into a root-owned helper in /usr/local/sbin,
deployed from the repo as source of truth. This avoids depending on fragile
argument globs and lets the rule constrain the one argument that varies.
vw-backup-db re-validates the destination path itself rather than trusting
the sudoers pattern.

Call the helpers from the script and drop the inline sudo. Verify the
snapshot with PRAGMA integrity_check before uploading, and abort rather
than shipping a copy that does not verify. Check smbclient's exit status
directly instead of piping into grep, which would have let SIGPIPE mask a
failed upload.

Add a systemd staleness watchdog so this cannot hide again. It mounts the
share read-only and separates STALE (nothing recent) from CANNOT VERIFY
(could not mount), which mean different things.

Also chmod 0440 /etc/sudoers.d/bdm, which visudo was warning about.

Tested with a cold sudo cache, since timestamp_type=global means an
interactive sudo -v otherwise masks the failure. A real cron invocation
succeeded end to end: snapshot, integrity check, upload, and prune.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-09-28 11:11:42 -07:00

13 KiB

Vaultwarden Backup & Restore System

This directory contains scripts for backing up, restoring, and converting Vaultwarden data.

Overview

Primary Backup: SQLite Database (Automated Daily)

The main backup runs daily at 3:00 AM via cron (/home/bdm/scripts/vaultwarden-backup.sh). It:

  1. Creates a consistent SQLite backup using sqlite3 .backup
  2. Uploads to SMB share (//192.168.123.2/NetBackup/Vault/)
  3. Cleans up backups older than 60 days

This is the complete backup for disaster recovery. It contains ALL users, their encrypted vaults, organizations, collections, sends, etc.

Restore Options

Use restore-sqlite.sh to restore the entire Vaultwarden instance:

./restore-sqlite.sh vaultwarden-2026-09-19_09-03-45.db

This replaces the database file and restarts Vaultwarden. All users log in with their existing master passwords.

Option 2: Individual User Vault Export/Import (JSON)

Use export-to-json.sh and import-from-json.sh for individual user vaults:

# Export (requires master password)
./export-to-json.sh user@example.com my-vault.json

# Import (requires master password)
./import-from-json.sh user@example.com my-vault.json

Note: JSON export requires the master password because the SQLite database stores all data ENCRYPTED. The master password is needed to decrypt the vault for JSON export.

Scripts

Script Purpose Requires Master Password
vaultwarden-backup.sh Daily automated SQLite backup (run by cron) No
restore-sqlite.sh Restore entire instance from SQLite backup No
export-to-json.sh Export single user vault to JSON Yes
import-from-json.sh Import JSON vault to user account Yes
vw-backup-db Take the consistent SQLite snapshot (root helper) No
mount-smb.sh Mount SMB share for cleanup (root helper) No
umount-smb.sh Unmount SMB share (root helper) No
cleanup-old-backups.sh Delete backups older than 60 days (root helper) No
vw-backup-check Staleness watchdog (root helper, run by systemd) No

Privileged Helpers

The four helpers that need root are kept in this directory as the source of truth and deployed to /usr/local/sbin/, owned by root:root, mode 755:

Source (here) Deployed to
vw-backup-db /usr/local/sbin/vw-backup-db
mount-smb.sh /usr/local/sbin/vw-mount-smb
umount-smb.sh /usr/local/sbin/vw-umount-smb
cleanup-old-backups.sh /usr/local/sbin/vw-cleanup-old
vw-backup-check /usr/local/sbin/vw-backup-check

They must stay root-owned. sudoers runs a script as root, so a NOPASSWD grant pointing at a file that bdm can rewrite is equivalent to granting unrestricted root. The previous sudoers file pointed at scripts under /home/bdm, which bdm owns.

To deploy after editing:

cd /home/bdm/scripts/vaultwarden-backup-helpers
sudo install -o root -g root -m 0755 vw-backup-db          /usr/local/sbin/vw-backup-db
sudo install -o root -g root -m 0755 mount-smb.sh          /usr/local/sbin/vw-mount-smb
sudo install -o root -g root -m 0755 umount-smb.sh         /usr/local/sbin/vw-umount-smb
sudo install -o root -g root -m 0755 cleanup-old-backups.sh /usr/local/sbin/vw-cleanup-old
sudo install -o root -g root -m 0755 vw-backup-check       /usr/local/sbin/vw-backup-check

Why the work is done in helpers, not inline sudo

A sudoers rule that names arguments matches only that exact argument list.

This was the cause of a silent nine-day backup outage. The rule was:

bdm ALL=(ALL) NOPASSWD: /usr/bin/sqlite3 /var/.../db.sqlite3

but the script ran sudo sqlite3 /var/.../db.sqlite3 ".backup '/tmp/...db'" - a second argument the rule did not name. The rule did not match, sudo fell through to the password-required bdm ALL=(ALL) ALL, and cron has no terminal, so every nightly run failed with sudo: a password is required and wrote the failure only to a log nobody read.

Demonstrating it, both with -n so neither can prompt:

sudo -n /usr/bin/sqlite3 /var/.../db.sqlite3                 # 1 arg, matches  -> works
sudo -n /usr/bin/sqlite3 /var/.../db.sqlite3 "SELECT 1;"     # 2 args, no match -> password required

Putting each privileged step in a root-owned helper avoids depending on fragile argument globs, and lets the sudoers rule constrain the one argument that varies. The vw-backup-db rule is still argument-constrained, so it cannot be used to write files outside the backup directory:

bdm ALL=(ALL) NOPASSWD: /usr/local/sbin/vw-backup-db /tmp/vaultwarden-backup/vaultwarden-*.db

Note the filename must match vaultwarden-*.db or sudo will not match the rule and will fall back to asking for a password.

Never replace these helper calls with inline sudo in the script. That is what broke it.

Staleness Watchdog

A backup that fails silently is worse than one that fails loudly, so a systemd timer checks the share every day and fails visibly.

  • Timer: vw-backup-check.timer, daily at 10:30 + up to 10 min jitter, Persistent=true so a missed run catches up
  • Service: vw-backup-check.service, runs as root, no sudo involved
  • Alarm threshold: 2 days

It mounts the share read-only and reports the three states separately, which matters because they mean different things:

Exit State Meaning
0 OK Newest backup is under 2 days old
1 STALE Newest backup is 2+ days old, or there are no backups at all
2 CANNOT VERIFY The share could not be mounted, so freshness is unknown

Check it by hand:

systemctl status vw-backup-check.service
journalctl -t vaultwarden-backup-check -n 20
sudo systemctl start vw-backup-check.service   # run it now

Backup Location

  • SMB Share: //192.168.123.2/NetBackup/Vault/
  • Local Temp: /tmp/vaultwarden-backup/ (cleaned up after upload)
  • Log File: /var/log/vaultwarden-backup.log
  • Journal: journalctl -t vaultwarden-backup (the log file is append-only and nobody reads it; the journal is the one to check)

Cron Schedule

Daily at 3:00 AM (see crontab -l):

0 3 * * * /home/bdm/scripts/vaultwarden-backup.sh >> /var/log/vaultwarden-backup.log 2>&1

Manual Backup

To run a manual backup:

/home/bdm/scripts/vaultwarden-backup.sh

Sudo Configuration

Sudoers rules are in /etc/sudoers.d/vaultwarden-backup and grant passwordless root for exactly four root-owned helpers in /usr/local/sbin/: the SQLite snapshot, the SMB mount, the SMB unmount, and the retention prune.

The old rules for a bare sqlite3 call and for chown are gone. The sqlite3 rule could not match the two-argument call the script actually makes, and chown is no longer needed because vw-backup-db hands the file back to the calling user using SUDO_UID/SUDO_GID.

Validate after any edit - visudo -c catches a bad rule before it can lock you out, and /etc/sudoers.d/*.bak is not read by sudo, so a broken file can simply be removed:

sudo visudo -c -f /path/to/newfile     # check before installing
sudo install -o root -g root -m 0440 /path/to/newfile /etc/sudoers.d/vaultwarden-backup
sudo visudo -c                         # then check the whole config

Two pre-existing quirks in this system's sudo setup are worth knowing about, because they explain confusing behaviour:

  • /etc/sudoers.d/010_pi-nopasswd contains bdm ALL=(ALL) NOPASSWD: ALL, which looks like blanket passwordless root. It is inert: sudo applies the last matching rule, and bdm ALL=(ALL) ALL in sudoers.d/bdm sorts after it and wins. Left alone deliberately - reordering would actually grant the blanket access that the file only appears to.
  • Defaults timestamp_type=global in 010_global-tty means a sudo -v in any terminal is reused by all of bdm's sessions, including cron, for 15 minutes. So sudo can appear to work from a script when it would actually fail on an unattended run. sudo -k clears it, which is the only honest way to test a NOPASSWD path.

Disaster Recovery Procedure

  1. New server setup:

    • Install Docker, Docker Compose
    • Deploy Vaultwarden container with same volume name (caddy_vaultwarden_data)
    • Start container once to create empty database
  2. Restore database:

    # Download backup from SMB (credentials read from the file below)
    smbclient //192.168.123.2/NetBackup -A "$HOME/.smbcredentials" -c "cd Vault; get vaultwarden-YYYY-MM-DD_HH-MM-SS.db"
    
    # Restore
    ./restore-sqlite.sh vaultwarden-YYYY-MM-DD_HH-MM-SS.db
    
  3. Verify: Users can log in with their existing credentials

JSON Import/Export Notes

  • JSON files contain DECRYPTED vault data - store securely!
  • Use BW_MASTER_PASSWORD env var for automation (less secure)
  • Interactive prompt is more secure
  • JSON format is Bitwarden-compatible (select "Bitwarden (json)" on import)
  • Import does not overwrite existing items - it adds/updates

Troubleshooting

Backup fails with "sqlite3 backup failed"

  • Check if container is running: docker ps | grep vaultwarden
  • Check volume path: ls -la /var/lib/docker/volumes/caddy_vaultwarden_data/_data/
  • Run manually to see errors: /home/bdm/scripts/vaultwarden-backup.sh

Backup fails with "sudo: a password is required"

The NOPASSWD rule did not match. Almost always an argument mismatch - see Why the work is done in helpers. Check what the rule expects against what the script passes:

sudo -n -l                                    # effective rules for bdm
sudo -k                                       # clear the cached credential first
sudo -n /usr/local/sbin/vw-backup-db /tmp/vaultwarden-backup/vaultwarden-probe.db

If that probe works with a cold cache, the grant is fine and the problem is in the script's arguments. If it fails, the sudoers file is wrong or was not installed.

Backup succeeds but nothing is on the share

Compare the two sides. The helper verifies integrity before uploading, so a successful run with no file on the NAS points at the upload rather than the snapshot:

tail -20 /var/log/vaultwarden-backup.log
smbclient //192.168.123.2/NetBackup -A "$HOME/.smbcredentials" -c 'cd Vault; ls'

The watchdog alarms but backups look fine

Exit 2 (CANNOT VERIFY) means the share could not be mounted, so the check could not read anything - that is a NAS or credential problem, not a backup problem. Exit 1 (STALE) means it mounted the share and genuinely found nothing recent.

SMB upload fails

  • Check network connectivity: ping 192.168.123.2
  • Check the SMB credentials file is present and readable: ls -l ~/.smbcredentials
  • Test manually: smbclient //192.168.123.2/NetBackup -A "$HOME/.smbcredentials" -c "ls"

Cron doesn't run

  • Check cron logs: journalctl -u cron -f
  • Verify crontab: crontab -l
  • Check log file: cat /var/log/vaultwarden-backup.log

SMB Credentials

The scripts do not contain the SMB password. It lives in an external file so that it stays out of version control and out of the process list:

  • Location: ~/.smbcredentials (override with the SMB_CRED_FILE env var)
  • Permissions: 600, owned by the user running the backup (bdm)

Create it once, with mode 600:

umask 077
printf 'username = %s\npassword = %s\n' 'bdm' 'YOUR_PASSWORD' > ~/.smbcredentials

The scripts pass it to smbclient with -A (authentication file), which is preferable to -U "user%pass" because a password passed via -U is visible to any local user through ps.

Note that /root/.smbcredentials is a separate file used only by the sudo mount -t cifs calls, which run as root. The two are independent.

Security Notes

  • SMB credentials are kept in ~/.smbcredentials (mode 600), not in the scripts
  • JSON exports contain plaintext passwords - encrypt them if storing long-term
  • SQLite backups contain encrypted data - safe to store but treat as sensitive
  • Master passwords are never stored - only used interactively or via env var
  • Every NOPASSWD grant points at a root-owned helper in /usr/local/sbin/, never at a file the bdm account can edit. A sudoers grant to a user-writable script is unrestricted root in practice.
  • vw-backup-db re-checks the destination path itself rather than trusting the sudoers argument glob, so the grant stays constrained even if the rule is broadened later
  • The database is in WAL mode with a large uncheckpointed WAL. cp would capture stale data; only sqlite3 .backup produces a consistent snapshot. The backup is also PRAGMA integrity_checked before upload, and a failure there aborts rather than shipping a bad copy.