# Backup & Recovery Runbook

Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
dumps, invoked by the backup script), and `restore.sh` (recovery).

All deployment-specific values (`TARGET`, `REPO`, `DB_CONTAINER`, etc.) live in
a config file under `configs/`. Both scripts source it automatically — see
**§2** for how to point them at the right one.

---

## 1. One-Time Setup (per server)

Run once, by hand, on the server. Substitute values from your config file
(`configs/<deployment>.conf`). The examples below use shell variables sourced
from it:

```bash
source /opt/backup-agent/configs/nexusvoice.conf   # adapt path
```

### 1.1 Initialize the Borg repo

```bash
mkdir -p "$(dirname "$REPO")"
borg init --encryption=repokey-blake2 "$REPO"
# If running unencrypted by deliberate choice:
# borg init --encryption=none "$REPO"
```

### 1.2 Passphrase file (backup and restore both read it)

```bash
echo 'your-strong-passphrase' > "$BORG_PASSPHRASE_FILE"
chmod 600 "$BORG_PASSPHRASE_FILE"
```

Skip this step only if running without encryption and without a passphrase.

### 1.3 MariaDB `backup` user (used by `dump_db.sh`)

Read-only; `--single-transaction` makes the dump consistent without stopping
the container.

```sql
CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';
```

```bash
echo 'a-strong-password' > /root/.mariadb-backup.pw
chmod 600 /root/.mariadb-backup.pw
```

### 1.4 MariaDB root password file (used only by `restore.sh`)

Restore needs CREATE/DROP privileges the `backup` user does not have.

```bash
echo 'the-mariadb-root-password' > "$ROOT_PASSWORD_FILE"
chmod 600 "$ROOT_PASSWORD_FILE"
```

### 1.5 Exclude the MariaDB raw data directory

Find the host directory bind-mounted into the container as its datadir and
drop a marker file in it. Borg skips any directory that contains `.nobackup`
(via `--exclude-if-present`), so live InnoDB files are never read:

```bash
touch "${TARGET}/mariadb/data/.nobackup"
```

This is what allows backups to run with the container up: only the logical
dump under `${DUMP_SUBDIR}/` is ever archived or restored.

### 1.6 Configure the Scaleway rclone remote

```bash
rclone config
# Create a remote named "scaleway", type S3, with your Scaleway
# Object Storage credentials and region.
```

---

## 2. Deploying the Scripts

### Single deployment

Copy `borg-backup.sh`, `dump_db.sh`, `restore.sh`, and the `configs/` directory
to `/opt/backup-agent/`. Symlink the relevant config as `backup.conf` beside
the scripts, or pass `BACKUP_CONF` explicitly in the cron entry.

```bash
chmod +x /opt/backup-agent/borg-backup.sh \
         /opt/backup-agent/restore.sh \
         /opt/backup-agent/dump_db.sh

# Option A — symlink (scripts auto-discover backup.conf beside them):
ln -s /opt/backup-agent/configs/srv.conf /opt/backup-agent/backup.conf

# Option B — explicit BACKUP_CONF in the cron entry (see §3).
```

### Multiple deployments on different servers

Each server gets its own config. Deploy the same three scripts to
`/opt/backup-agent/` on each server; point each cron entry at its config via
`BACKUP_CONF`:

```
# /etc/cron.d/borg-backup  (nexusvoice server)
BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
```

Before the first run, verify `configs/nexusvoice.conf`:
- `TARGET` — current path; update if files move (e.g. `/home/acid/nexusvoice` → `/opt/nexusvoice`)
- `DB_CONTAINER` — confirm the MariaDB container name on that server
- `REPO` — must not overlap with any other deployment's repo

---

## 3. Scheduling

Add a cron entry to run the backup daily, off-peak:

```
# /etc/cron.d/borg-backup  (srv — default config via symlink)
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
```

Check the last run:

```bash
ls -lt /var/log/borg/backup-*.log | head -1   # latest log file
tail -50 /var/log/borg/backup-*.log           # inspect it
```

---

## 4. Day-2 Operations

Source your config first to get the right `BORG_REPO`, `BORG_PASSCOMMAND`, etc.:

```bash
source /opt/backup-agent/configs/nexusvoice.conf   # adapt
export BORG_REPO="$REPO"
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"
```

Then standard borg commands work without extra flags:

```bash
# List archives
./restore.sh --list-archives
# or directly:
borg list

# Repo size and health
borg info

# Rotate the passphrase
# Note: only re-encrypts the key, not the archive data; old passphrase
# still needed for archives created before the change until fully migrated.
borg key change-passphrase

# Break a stale lockfile (backup or restore aborted mid-run)
borg break-lock
```

---

## 5. Recovery Procedures

All `restore.sh` commands accept:
- `--dry-run` — preview exactly what would happen without touching anything
- `--archive NAME` — target a specific archive instead of the latest (get names via `--list-archives`)

Root DB credentials come from `MYSQL_ROOT_PASSWORD` in the environment if set,
otherwise from the `ROOT_PASSWORD_FILE` in your config (default
`/root/.mariadb-root.pw`). Restore logs go to `/var/log/borg/restore-*.log`;
`RESTORE_LOGDIR` overrides the directory (useful for testing).

`restore.sh` shares `borg-backup.sh`'s lockfile (`LOCKFILE` in the config),
so a restore refuses to start while the nightly backup is mid-run and vice
versa.

For all `restore.sh` commands, select the deployment with `BACKUP_CONF`:

```bash
export BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf
```

Or run without it if `backup.conf` is already symlinked beside the script.

### 5.1 Full disaster recovery (new or wiped server)

Use when the whole server/container is gone and you're rebuilding from scratch.

```bash
# 1. Reinstall borg, docker, and the mariadb container image/compose file
#    (not covered by restore.sh — this is infra provisioning).
# 2. Restore the passphrase file (from your password manager / secondary
#    backup — it is NOT stored in the repo it protects) to the path set in
#    BORG_PASSPHRASE_FILE, and the root DB password to ROOT_PASSWORD_FILE.
# 3. Preview:
./restore.sh full --dry-run
# 4. Run for real. --force is only required if $TARGET already has data in it
#    (e.g. a stale mount); omit it on a genuinely empty/fresh server:
./restore.sh full --force
```

This extracts the full content tree, starts the DB container and waits for it
to report healthy, then restores every database dump (users/grants first).

**Verify afterward:**
- `docker ps` shows the DB container running and healthy.
- The application responds normally.
- Spot-check row counts on a couple of tables against what you'd expect.

### 5.2 Single database restore

Use when one database got corrupted or someone ran a bad migration/query —
this **drops and recreates** that database.

```bash
./restore.sh db shopdb --dry-run     # preview
./restore.sh db shopdb               # prompts: type "shopdb" to confirm
```

Non-interactive (e.g. scripted from a monitoring alert): add `--yes` to
skip the typed confirmation.

**Verify afterward:** connect to the database and check the tables/row counts
you expect.

### 5.3 Single file/directory restore

Use for accidental deletion or to inspect an old version — never touches the
running database or container.

```bash
./restore.sh file path/relative/to/target/some-file.txt --dest /tmp/recovered
```

The final location of the recovered item is printed at the end. Borg
preserves the absolute path it was archived with, so the file lands under
`/tmp/recovered/<TARGET>/...`.

---

## 6. Restore Drill Cadence

Quarterly, run a real `full` extract into a scratch directory to confirm
backups are actually usable.

`borg extract` always extracts into the current directory; the archive name
must come from `borg list --short` (plain `borg list` prints a formatted
line, not a bare name). Source your config to get the right values:

```bash
source /opt/backup-agent/configs/nexusvoice.conf   # adapt
export BORG_REPO="$REPO"
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"

mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
LATEST=$(borg list --short | tail -1)
borg extract --lock-wait 600 "::$LATEST"
```

Confirm the dump files under `${DUMP_SUBDIR}/` are present, non-empty, and
importable (`mysql -u root -p < mariadb/dump/somedb.sql` against a throwaway
MariaDB container). Log the drill date and outcome somewhere durable (e.g. a
team wiki page).