backup-agent/RUNBOOK.md

Backup & Recovery Runbook

Covers borg-backup.sh (daily backup), dump_db.sh (MariaDB logical dumps, invoked by the backup script), and restore.sh (recovery).

All deployment-specific values (TARGET, REPO, DB_CONTAINER, etc.) live in a config file under configs/. Both scripts source it automatically — see §2 for how to point them at the right one.

---

1. One-Time Setup (per server)

Run once, by hand, on the server. Substitute values from your config file (configs/<deployment>.conf). The examples below use shell variables sourced from it:

source /opt/backup-agent/configs/nexusvoice.conf   # adapt path

1.1 Initialize the Borg repo

mkdir -p "$(dirname "$REPO")"
borg init --encryption=repokey-blake2 "$REPO"
# If running unencrypted by deliberate choice:
# borg init --encryption=none "$REPO"

1.2 Passphrase file (backup and restore both read it)

echo 'your-strong-passphrase' > "$BORG_PASSPHRASE_FILE"
chmod 600 "$BORG_PASSPHRASE_FILE"

Skip this step only if running without encryption and without a passphrase.

1.3 MariaDB backup user (used by dump_db.sh)

Read-only; --single-transaction makes the dump consistent without stopping the container.

CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';

echo 'a-strong-password' > /root/.mariadb-backup.pw
chmod 600 /root/.mariadb-backup.pw

1.4 MariaDB root password file (used only by restore.sh)

Restore needs CREATE/DROP privileges the backup user does not have.

echo 'the-mariadb-root-password' > "$ROOT_PASSWORD_FILE"
chmod 600 "$ROOT_PASSWORD_FILE"

1.5 Exclude the MariaDB raw data directory

Find the host directory bind-mounted into the container as its datadir and drop a marker file in it. Borg skips any directory that contains .nobackup (via --exclude-if-present), so live InnoDB files are never read:

touch "${TARGET}/mariadb/data/.nobackup"

This is what allows backups to run with the container up: only the logical dump under ${DUMP_SUBDIR}/ is ever archived or restored.

1.6 Configure the Scaleway rclone remote

rclone config
# Create a remote named "scaleway", type S3, with your Scaleway
# Object Storage credentials and region.

---

2. Deploying the Scripts

Single deployment

Copy borg-backup.sh, dump_db.sh, restore.sh, and the configs/ directory to /opt/backup-agent/. Symlink the relevant config as backup.conf beside the scripts, or pass BACKUP_CONF explicitly in the cron entry.

chmod +x /opt/backup-agent/borg-backup.sh \
         /opt/backup-agent/restore.sh \
         /opt/backup-agent/dump_db.sh

# Option A — symlink (scripts auto-discover backup.conf beside them):
ln -s /opt/backup-agent/configs/srv.conf /opt/backup-agent/backup.conf

# Option B — explicit BACKUP_CONF in the cron entry (see §3).

Multiple deployments on different servers

Each server gets its own config. Deploy the same three scripts to /opt/backup-agent/ on each server; point each cron entry at its config via BACKUP_CONF:

# /etc/cron.d/borg-backup  (nexusvoice server)
BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1

Before the first run, verify configs/nexusvoice.conf:

• TARGET — current path; update if files move (e.g. /home/acid/nexusvoice → /opt/nexusvoice)
• DB_CONTAINER — confirm the MariaDB container name on that server
• REPO — must not overlap with any other deployment's repo

---

3. Scheduling

Add a cron entry to run the backup daily, off-peak:

# /etc/cron.d/borg-backup  (srv — default config via symlink)
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1

Check the last run:

ls -lt /var/log/borg/backup-*.log | head -1   # latest log file
tail -50 /var/log/borg/backup-*.log           # inspect it

---

4. Day-2 Operations

Source your config first to get the right BORG_REPO, BORG_PASSCOMMAND, etc.:

source /opt/backup-agent/configs/nexusvoice.conf   # adapt
export BORG_REPO="$REPO"
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"

Then standard borg commands work without extra flags:

# List archives
./restore.sh --list-archives
# or directly:
borg list

# Repo size and health
borg info

# Rotate the passphrase
# Note: only re-encrypts the key, not the archive data; old passphrase
# still needed for archives created before the change until fully migrated.
borg key change-passphrase

# Break a stale lockfile (backup or restore aborted mid-run)
borg break-lock

---

5. Recovery Procedures

All restore.sh commands accept:

• --dry-run — preview exactly what would happen without touching anything
• --archive NAME — target a specific archive instead of the latest (get names via --list-archives)

Root DB credentials come from MYSQL_ROOT_PASSWORD in the environment if set, otherwise from the ROOT_PASSWORD_FILE in your config (default /root/.mariadb-root.pw). Restore logs go to /var/log/borg/restore-*.log; RESTORE_LOGDIR overrides the directory (useful for testing).

restore.sh shares borg-backup.sh's lockfile (LOCKFILE in the config), so a restore refuses to start while the nightly backup is mid-run and vice versa.

For all restore.sh commands, select the deployment with BACKUP_CONF:

export BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf

Or run without it if backup.conf is already symlinked beside the script.

5.1 Full disaster recovery (new or wiped server)

Use when the whole server/container is gone and you're rebuilding from scratch.

# 1. Reinstall borg, docker, and the mariadb container image/compose file
#    (not covered by restore.sh — this is infra provisioning).
# 2. Restore the passphrase file (from your password manager / secondary
#    backup — it is NOT stored in the repo it protects) to the path set in
#    BORG_PASSPHRASE_FILE, and the root DB password to ROOT_PASSWORD_FILE.
# 3. Preview:
./restore.sh full --dry-run
# 4. Run for real. --force is only required if $TARGET already has data in it
#    (e.g. a stale mount); omit it on a genuinely empty/fresh server:
./restore.sh full --force

This extracts the full content tree, starts the DB container and waits for it to report healthy, then restores every database dump (users/grants first).

Verify afterward:

• docker ps shows the DB container running and healthy.
• The application responds normally.
• Spot-check row counts on a couple of tables against what you'd expect.

5.2 Single database restore

Use when one database got corrupted or someone ran a bad migration/query — this drops and recreates that database.

./restore.sh db shopdb --dry-run     # preview
./restore.sh db shopdb               # prompts: type "shopdb" to confirm

Non-interactive (e.g. scripted from a monitoring alert): add --yes to skip the typed confirmation.

Verify afterward: connect to the database and check the tables/row counts you expect.

5.3 Single file/directory restore

Use for accidental deletion or to inspect an old version — never touches the running database or container.

./restore.sh file path/relative/to/target/some-file.txt --dest /tmp/recovered

The final location of the recovered item is printed at the end. Borg preserves the absolute path it was archived with, so the file lands under /tmp/recovered/<TARGET>/....

---

6. Restore Drill Cadence

Quarterly, run a real full extract into a scratch directory to confirm backups are actually usable.

borg extract always extracts into the current directory; the archive name must come from borg list --short (plain borg list prints a formatted line, not a bare name). Source your config to get the right values:

source /opt/backup-agent/configs/nexusvoice.conf   # adapt
export BORG_REPO="$REPO"
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"

mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
LATEST=$(borg list --short | tail -1)
borg extract --lock-wait 600 "::$LATEST"

Confirm the dump files under ${DUMP_SUBDIR}/ are present, non-empty, and importable (mysql -u root -p < mariadb/dump/somedb.sql against a throwaway MariaDB container). Log the drill date and outcome somewhere durable (e.g. a team wiki page).