kevin.batailleandClaude Sonnet 4.6
f3cf24a41c
- README: remove hardcoded srv paths; add configs/ to components table; document BACKUP_CONF usage for multi-server setups; note DB_CONTAINER= makes docker optional - RUNBOOK: generalize §1 setup to use config variables (source config, use $REPO / $BORG_PASSPHRASE_FILE / etc. instead of hardcoded paths); §4 day-2 ops now shows source-config pattern before direct borg commands; §5 recovery notes BACKUP_CONF for restore.sh; §6 drill uses config vars Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A3rQSEidP6Y61kaCtxjVV1
277 lines
8.4 KiB
Markdown
277 lines
8.4 KiB
Markdown
# Backup & Recovery Runbook
|
|
|
|
Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
|
|
dumps, invoked by the backup script), and `restore.sh` (recovery).
|
|
|
|
All deployment-specific values (`TARGET`, `REPO`, `DB_CONTAINER`, etc.) live in
|
|
a config file under `configs/`. Both scripts source it automatically — see
|
|
**§2** for how to point them at the right one.
|
|
|
|
---
|
|
|
|
## 1. One-Time Setup (per server)
|
|
|
|
Run once, by hand, on the server. Substitute values from your config file
|
|
(`configs/<deployment>.conf`). The examples below use shell variables sourced
|
|
from it:
|
|
|
|
```bash
|
|
source /opt/backup-agent/configs/nexusvoice.conf # adapt path
|
|
```
|
|
|
|
### 1.1 Initialize the Borg repo
|
|
|
|
```bash
|
|
mkdir -p "$(dirname "$REPO")"
|
|
borg init --encryption=repokey-blake2 "$REPO"
|
|
# If running unencrypted by deliberate choice:
|
|
# borg init --encryption=none "$REPO"
|
|
```
|
|
|
|
### 1.2 Passphrase file (backup and restore both read it)
|
|
|
|
```bash
|
|
echo 'your-strong-passphrase' > "$BORG_PASSPHRASE_FILE"
|
|
chmod 600 "$BORG_PASSPHRASE_FILE"
|
|
```
|
|
|
|
Skip this step only if running without encryption and without a passphrase.
|
|
|
|
### 1.3 MariaDB `backup` user (used by `dump_db.sh`)
|
|
|
|
Read-only; `--single-transaction` makes the dump consistent without stopping
|
|
the container.
|
|
|
|
```sql
|
|
CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
|
|
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';
|
|
```
|
|
|
|
```bash
|
|
echo 'a-strong-password' > /root/.mariadb-backup.pw
|
|
chmod 600 /root/.mariadb-backup.pw
|
|
```
|
|
|
|
### 1.4 MariaDB root password file (used only by `restore.sh`)
|
|
|
|
Restore needs CREATE/DROP privileges the `backup` user does not have.
|
|
|
|
```bash
|
|
echo 'the-mariadb-root-password' > "$ROOT_PASSWORD_FILE"
|
|
chmod 600 "$ROOT_PASSWORD_FILE"
|
|
```
|
|
|
|
### 1.5 Exclude the MariaDB raw data directory
|
|
|
|
Find the host directory bind-mounted into the container as its datadir and
|
|
drop a marker file in it. Borg skips any directory that contains `.nobackup`
|
|
(via `--exclude-if-present`), so live InnoDB files are never read:
|
|
|
|
```bash
|
|
touch "${TARGET}/mariadb/data/.nobackup"
|
|
```
|
|
|
|
This is what allows backups to run with the container up: only the logical
|
|
dump under `${DUMP_SUBDIR}/` is ever archived or restored.
|
|
|
|
### 1.6 Configure the Scaleway rclone remote
|
|
|
|
```bash
|
|
rclone config
|
|
# Create a remote named "scaleway", type S3, with your Scaleway
|
|
# Object Storage credentials and region.
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Deploying the Scripts
|
|
|
|
### Single deployment
|
|
|
|
Copy `borg-backup.sh`, `dump_db.sh`, `restore.sh`, and the `configs/` directory
|
|
to `/opt/backup-agent/`. Symlink the relevant config as `backup.conf` beside
|
|
the scripts, or pass `BACKUP_CONF` explicitly in the cron entry.
|
|
|
|
```bash
|
|
chmod +x /opt/backup-agent/borg-backup.sh \
|
|
/opt/backup-agent/restore.sh \
|
|
/opt/backup-agent/dump_db.sh
|
|
|
|
# Option A — symlink (scripts auto-discover backup.conf beside them):
|
|
ln -s /opt/backup-agent/configs/srv.conf /opt/backup-agent/backup.conf
|
|
|
|
# Option B — explicit BACKUP_CONF in the cron entry (see §3).
|
|
```
|
|
|
|
### Multiple deployments on different servers
|
|
|
|
Each server gets its own config. Deploy the same three scripts to
|
|
`/opt/backup-agent/` on each server; point each cron entry at its config via
|
|
`BACKUP_CONF`:
|
|
|
|
```
|
|
# /etc/cron.d/borg-backup (nexusvoice server)
|
|
BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf
|
|
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
|
|
```
|
|
|
|
Before the first run, verify `configs/nexusvoice.conf`:
|
|
- `TARGET` — current path; update if files move (e.g. `/home/acid/nexusvoice` → `/opt/nexusvoice`)
|
|
- `DB_CONTAINER` — confirm the MariaDB container name on that server
|
|
- `REPO` — must not overlap with any other deployment's repo
|
|
|
|
---
|
|
|
|
## 3. Scheduling
|
|
|
|
Add a cron entry to run the backup daily, off-peak:
|
|
|
|
```
|
|
# /etc/cron.d/borg-backup (srv — default config via symlink)
|
|
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
|
|
```
|
|
|
|
Check the last run:
|
|
|
|
```bash
|
|
ls -lt /var/log/borg/backup-*.log | head -1 # latest log file
|
|
tail -50 /var/log/borg/backup-*.log # inspect it
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Day-2 Operations
|
|
|
|
Source your config first to get the right `BORG_REPO`, `BORG_PASSCOMMAND`, etc.:
|
|
|
|
```bash
|
|
source /opt/backup-agent/configs/nexusvoice.conf # adapt
|
|
export BORG_REPO="$REPO"
|
|
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"
|
|
```
|
|
|
|
Then standard borg commands work without extra flags:
|
|
|
|
```bash
|
|
# List archives
|
|
./restore.sh --list-archives
|
|
# or directly:
|
|
borg list
|
|
|
|
# Repo size and health
|
|
borg info
|
|
|
|
# Rotate the passphrase
|
|
# Note: only re-encrypts the key, not the archive data; old passphrase
|
|
# still needed for archives created before the change until fully migrated.
|
|
borg key change-passphrase
|
|
|
|
# Break a stale lockfile (backup or restore aborted mid-run)
|
|
borg break-lock
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Recovery Procedures
|
|
|
|
All `restore.sh` commands accept:
|
|
- `--dry-run` — preview exactly what would happen without touching anything
|
|
- `--archive NAME` — target a specific archive instead of the latest (get names via `--list-archives`)
|
|
|
|
Root DB credentials come from `MYSQL_ROOT_PASSWORD` in the environment if set,
|
|
otherwise from the `ROOT_PASSWORD_FILE` in your config (default
|
|
`/root/.mariadb-root.pw`). Restore logs go to `/var/log/borg/restore-*.log`;
|
|
`RESTORE_LOGDIR` overrides the directory (useful for testing).
|
|
|
|
`restore.sh` shares `borg-backup.sh`'s lockfile (`LOCKFILE` in the config),
|
|
so a restore refuses to start while the nightly backup is mid-run and vice
|
|
versa.
|
|
|
|
For all `restore.sh` commands, select the deployment with `BACKUP_CONF`:
|
|
|
|
```bash
|
|
export BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf
|
|
```
|
|
|
|
Or run without it if `backup.conf` is already symlinked beside the script.
|
|
|
|
### 5.1 Full disaster recovery (new or wiped server)
|
|
|
|
Use when the whole server/container is gone and you're rebuilding from scratch.
|
|
|
|
```bash
|
|
# 1. Reinstall borg, docker, and the mariadb container image/compose file
|
|
# (not covered by restore.sh — this is infra provisioning).
|
|
# 2. Restore the passphrase file (from your password manager / secondary
|
|
# backup — it is NOT stored in the repo it protects) to the path set in
|
|
# BORG_PASSPHRASE_FILE, and the root DB password to ROOT_PASSWORD_FILE.
|
|
# 3. Preview:
|
|
./restore.sh full --dry-run
|
|
# 4. Run for real. --force is only required if $TARGET already has data in it
|
|
# (e.g. a stale mount); omit it on a genuinely empty/fresh server:
|
|
./restore.sh full --force
|
|
```
|
|
|
|
This extracts the full content tree, starts the DB container and waits for it
|
|
to report healthy, then restores every database dump (users/grants first).
|
|
|
|
**Verify afterward:**
|
|
- `docker ps` shows the DB container running and healthy.
|
|
- The application responds normally.
|
|
- Spot-check row counts on a couple of tables against what you'd expect.
|
|
|
|
### 5.2 Single database restore
|
|
|
|
Use when one database got corrupted or someone ran a bad migration/query —
|
|
this **drops and recreates** that database.
|
|
|
|
```bash
|
|
./restore.sh db shopdb --dry-run # preview
|
|
./restore.sh db shopdb # prompts: type "shopdb" to confirm
|
|
```
|
|
|
|
Non-interactive (e.g. scripted from a monitoring alert): add `--yes` to
|
|
skip the typed confirmation.
|
|
|
|
**Verify afterward:** connect to the database and check the tables/row counts
|
|
you expect.
|
|
|
|
### 5.3 Single file/directory restore
|
|
|
|
Use for accidental deletion or to inspect an old version — never touches the
|
|
running database or container.
|
|
|
|
```bash
|
|
./restore.sh file path/relative/to/target/some-file.txt --dest /tmp/recovered
|
|
```
|
|
|
|
The final location of the recovered item is printed at the end. Borg
|
|
preserves the absolute path it was archived with, so the file lands under
|
|
`/tmp/recovered/<TARGET>/...`.
|
|
|
|
---
|
|
|
|
## 6. Restore Drill Cadence
|
|
|
|
Quarterly, run a real `full` extract into a scratch directory to confirm
|
|
backups are actually usable.
|
|
|
|
`borg extract` always extracts into the current directory; the archive name
|
|
must come from `borg list --short` (plain `borg list` prints a formatted
|
|
line, not a bare name). Source your config to get the right values:
|
|
|
|
```bash
|
|
source /opt/backup-agent/configs/nexusvoice.conf # adapt
|
|
export BORG_REPO="$REPO"
|
|
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"
|
|
|
|
mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
|
|
LATEST=$(borg list --short | tail -1)
|
|
borg extract --lock-wait 600 "::$LATEST"
|
|
```
|
|
|
|
Confirm the dump files under `${DUMP_SUBDIR}/` are present, non-empty, and
|
|
importable (`mysql -u root -p < mariadb/dump/somedb.sql` against a throwaway
|
|
MariaDB container). Log the drill date and outcome somewhere durable (e.g. a
|
|
team wiki page).
|