Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d467929f91 | ||
|
|
b0a72d1190 | ||
|
|
4975bd1bda | ||
|
|
f3cf24a41c | ||
|
|
74390b78af |
@@ -1,23 +1,38 @@
|
|||||||
# backup-agent
|
# backup-agent
|
||||||
|
|
||||||
Backup and disaster-recovery tooling for a Linux server: [Borg](https://borgbackup.readthedocs.io/)
|
Backup and disaster-recovery tooling for Linux servers: [Borg](https://borgbackup.readthedocs.io/)
|
||||||
backing up `/home/srv/files/content` — including a MariaDB database
|
backing up a target directory tree — including a MariaDB database running in
|
||||||
running in Docker — with an offsite mirror on Scaleway S3 via `rclone`.
|
Docker — with an offsite mirror on Scaleway S3 via `rclone`.
|
||||||
|
|
||||||
MariaDB is never stopped during backup: `dump_db.sh` takes a
|
MariaDB is never stopped during backup: `dump_db.sh` takes a
|
||||||
transactionally-consistent logical dump (`mysqldump --single-transaction`)
|
transactionally-consistent logical dump (`mysqldump --single-transaction`)
|
||||||
while the container keeps running, and the container's raw data directory
|
while the container keeps running, and the container's raw data directory is
|
||||||
is excluded from the archive entirely (a `.nobackup` marker file), so only
|
excluded from the archive entirely (a `.nobackup` marker file), so only the
|
||||||
the logical dump ever gets backed up. Zero DB downtime.
|
logical dump ever gets backed up. Zero DB downtime.
|
||||||
|
|
||||||
## Components
|
## Components
|
||||||
|
|
||||||
| File | Purpose |
|
| File | Purpose |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
| `configs/` | Per-deployment config files (one per server). Each sets `TARGET`, `REPO`, `DB_CONTAINER`, etc. |
|
||||||
| `borg-backup.sh` | Daily backup orchestrator (run from cron): dump → archive → prune → compact → integrity check → offsite sync. |
|
| `borg-backup.sh` | Daily backup orchestrator (run from cron): dump → archive → prune → compact → integrity check → offsite sync. |
|
||||||
| `dump_db.sh` | Per-database `mysqldump`/`mariadb-dump`, atomic staging/swap. Invoked by `borg-backup.sh`. |
|
| `dump_db.sh` | Per-database `mysqldump`/`mariadb-dump`, atomic staging/swap. Invoked by `borg-backup.sh`. |
|
||||||
| `restore.sh` | Recovery CLI: `full` (disaster recovery), `db <name>` (single database), `file <path>` (single file/dir), `--list-archives`. Every mode supports `--dry-run`. |
|
| `restore.sh` | Recovery CLI: `full` (disaster recovery), `db <name>` (single database), `file <path>` (single file/dir), `--list-archives`. Every mode supports `--dry-run`. |
|
||||||
|
|
||||||
|
## Multi-server use
|
||||||
|
|
||||||
|
Both scripts are deployment-agnostic. Point them at a config with `BACKUP_CONF`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# backup
|
||||||
|
BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf /opt/backup-agent/borg-backup.sh
|
||||||
|
|
||||||
|
# restore
|
||||||
|
BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf /opt/backup-agent/restore.sh --list-archives
|
||||||
|
```
|
||||||
|
|
||||||
|
See `configs/` for existing deployments and `RUNBOOK.md §2` for deployment steps.
|
||||||
|
|
||||||
## Quickstart
|
## Quickstart
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -37,16 +52,16 @@ Day to day:
|
|||||||
|
|
||||||
## Encryption
|
## Encryption
|
||||||
|
|
||||||
The Borg repo at `/home/srv/files/backups/borg-2025` is **unencrypted** by
|
The Borg repos on current deployments are **unencrypted** by deliberate operator
|
||||||
deliberate choice on this deployment — `borg-backup.sh` will keep printing
|
choice — `borg-backup.sh` will keep printing a warning about it on every run,
|
||||||
a warning about it on every run, which is expected. See `RUNBOOK.md` if
|
which is expected. See `RUNBOOK.md` if you want to switch to an encrypted repo.
|
||||||
you want to switch to an encrypted repo.
|
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
`borg`, `docker`, `rclone`, `flock`, a `mysql`/`mariadb` client — see
|
`borg`, `docker`, `rclone`, `flock`, a `mysql`/`mariadb` client. When
|
||||||
`REQUIRED_CMDS` in `borg-backup.sh`. Targets Linux; `flock(1)` doesn't
|
`DB_CONTAINER` is empty in the config, `docker` is not required. Targets
|
||||||
exist on macOS, so these scripts won't run as-is on a Mac.
|
Linux; `flock(1)` doesn't exist on macOS, so these scripts won't run as-is
|
||||||
|
on a Mac.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
+192
-118
@@ -3,68 +3,131 @@
|
|||||||
Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
|
Covers `borg-backup.sh` (daily backup), `dump_db.sh` (MariaDB logical
|
||||||
dumps, invoked by the backup script), and `restore.sh` (recovery).
|
dumps, invoked by the backup script), and `restore.sh` (recovery).
|
||||||
|
|
||||||
## 1. One-Time Setup
|
All deployment-specific values (`TARGET`, `REPO`, `DB_CONTAINER`, etc.) live in
|
||||||
|
a config file under `configs/`. Both scripts source it automatically — see
|
||||||
|
**§2** for how to point them at the right one.
|
||||||
|
|
||||||
Run once, by hand, on the server:
|
---
|
||||||
|
|
||||||
1. Initialize the encrypted Borg repo:
|
## 1. One-Time Setup (per server)
|
||||||
```bash
|
|
||||||
mkdir -p /home/srv/files/backups
|
Run once, by hand, on the server. Substitute values from your config file
|
||||||
borg init --encryption=repokey-blake2 /home/srv/files/backups/borg-2025
|
(`configs/<deployment>.conf`). The examples below use shell variables sourced
|
||||||
```
|
from it:
|
||||||
2. Create the passphrase file used by both backup and restore:
|
|
||||||
```bash
|
```bash
|
||||||
echo 'your-strong-passphrase' > /root/.borg-passphrase
|
source /opt/backup-agent/configs/nexusvoice.conf # adapt path
|
||||||
chmod 600 /root/.borg-passphrase
|
```
|
||||||
```
|
|
||||||
3. Create the MariaDB `backup` user used by `dump_db.sh` (read-only, no
|
### 1.1 Initialize the Borg repo
|
||||||
stop/lock of the server required thanks to `--single-transaction`):
|
|
||||||
```sql
|
```bash
|
||||||
CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
|
mkdir -p "$(dirname "$REPO")"
|
||||||
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';
|
borg init --encryption=repokey-blake2 "$REPO"
|
||||||
```
|
# If running unencrypted by deliberate choice:
|
||||||
```bash
|
# borg init --encryption=none "$REPO"
|
||||||
echo 'a-strong-password' > /root/.mariadb-backup.pw
|
```
|
||||||
chmod 600 /root/.mariadb-backup.pw
|
|
||||||
```
|
### 1.2 Passphrase file (backup and restore both read it)
|
||||||
4. Create the root password file used only by `restore.sh` (restore needs
|
|
||||||
CREATE/DROP privileges the `backup` user does not have):
|
```bash
|
||||||
```bash
|
echo 'your-strong-passphrase' > "$BORG_PASSPHRASE_FILE"
|
||||||
echo 'the-mariadb-root-password' > /root/.mariadb-root.pw
|
chmod 600 "$BORG_PASSPHRASE_FILE"
|
||||||
chmod 600 /root/.mariadb-root.pw
|
```
|
||||||
```
|
|
||||||
5. Exclude MariaDB's raw data directory from the archive. Find the
|
Skip this step only if running without encryption and without a passphrase.
|
||||||
directory bind-mounted into the container as its datadir and drop a
|
|
||||||
marker file in it:
|
### 1.3 MariaDB `backup` user (used by `dump_db.sh`)
|
||||||
```bash
|
|
||||||
touch /home/srv/files/content/mariadb/data/.nobackup
|
Read-only; `--single-transaction` makes the dump consistent without stopping
|
||||||
```
|
the container.
|
||||||
This is what allows backups to run with the container up: only the
|
|
||||||
logical dump under `mariadb/dump/` is ever archived or restored from.
|
```sql
|
||||||
6. Configure the `scaleway` rclone remote:
|
CREATE USER 'backup'@'%' IDENTIFIED BY 'a-strong-password';
|
||||||
```bash
|
GRANT SELECT, LOCK TABLES, SHOW VIEW, TRIGGER, PROCESS, RELOAD ON *.* TO 'backup'@'%';
|
||||||
rclone config
|
```
|
||||||
# create a remote named "scaleway", type S3, matching your Scaleway
|
|
||||||
# Object Storage credentials and region
|
```bash
|
||||||
```
|
echo 'a-strong-password' > /root/.mariadb-backup.pw
|
||||||
|
chmod 600 /root/.mariadb-backup.pw
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.4 MariaDB root password file (used only by `restore.sh`)
|
||||||
|
|
||||||
|
Restore needs CREATE/DROP privileges the `backup` user does not have.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
echo 'the-mariadb-root-password' > "$ROOT_PASSWORD_FILE"
|
||||||
|
chmod 600 "$ROOT_PASSWORD_FILE"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 1.5 Exclude the MariaDB raw data directory
|
||||||
|
|
||||||
|
Find the host directory bind-mounted into the container as its datadir and
|
||||||
|
drop a marker file in it. Borg skips any directory that contains `.nobackup`
|
||||||
|
(via `--exclude-if-present`), so live InnoDB files are never read:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
touch "${TARGET}/mariadb/data/.nobackup"
|
||||||
|
```
|
||||||
|
|
||||||
|
This is what allows backups to run with the container up: only the logical
|
||||||
|
dump under `${DUMP_SUBDIR}/` is ever archived or restored.
|
||||||
|
|
||||||
|
### 1.6 Configure the Scaleway rclone remote
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rclone config
|
||||||
|
# Create a remote named "scaleway", type S3, with your Scaleway
|
||||||
|
# Object Storage credentials and region.
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 2. Deploying the Scripts
|
## 2. Deploying the Scripts
|
||||||
|
|
||||||
Copy `borg-backup.sh`, `dump_db.sh`, and `restore.sh` to `/opt/backup-agent/`
|
### Single deployment
|
||||||
(this exact path is what `borg-backup.sh` invokes for `dump_db.sh`). Make
|
|
||||||
all three executable:
|
Copy `borg-backup.sh`, `dump_db.sh`, `restore.sh`, and the `configs/` directory
|
||||||
|
to `/opt/backup-agent/`. Symlink the relevant config as `backup.conf` beside
|
||||||
|
the scripts, or pass `BACKUP_CONF` explicitly in the cron entry.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
chmod +x /opt/backup-agent/borg-backup.sh /opt/backup-agent/restore.sh
|
chmod +x /opt/backup-agent/borg-backup.sh \
|
||||||
chmod +x /opt/backup-agent/dump_db.sh
|
/opt/backup-agent/restore.sh \
|
||||||
|
/opt/backup-agent/dump_db.sh
|
||||||
|
|
||||||
|
# Option A — symlink (scripts auto-discover backup.conf beside them):
|
||||||
|
ln -s /opt/backup-agent/configs/srv.conf /opt/backup-agent/backup.conf
|
||||||
|
|
||||||
|
# Option B — explicit BACKUP_CONF in the cron entry (see §3).
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Multiple deployments on different servers
|
||||||
|
|
||||||
|
Each server gets its own config. Deploy the same three scripts to
|
||||||
|
`/opt/backup-agent/` on each server; point each cron entry at its config via
|
||||||
|
`BACKUP_CONF`:
|
||||||
|
|
||||||
|
```
|
||||||
|
# /etc/cron.d/borg-backup (nexusvoice server)
|
||||||
|
BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf
|
||||||
|
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
Before the first run, verify `configs/nexusvoice.conf`:
|
||||||
|
- `TARGET` — current path; update if files move (e.g. `/home/acid/nexusvoice` → `/opt/nexusvoice`)
|
||||||
|
- `DB_CONTAINER` — confirm the MariaDB container name on that server
|
||||||
|
- `REPO` — must not overlap with any other deployment's repo
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 3. Scheduling
|
## 3. Scheduling
|
||||||
|
|
||||||
Add a cron entry to run the backup daily, off-peak:
|
Add a cron entry to run the backup daily, off-peak:
|
||||||
|
|
||||||
```
|
```
|
||||||
# /etc/cron.d/borg-backup
|
# /etc/cron.d/borg-backup (srv — default config via symlink)
|
||||||
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
|
30 2 * * * root /opt/backup-agent/borg-backup.sh >> /var/log/borg/cron.log 2>&1
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -75,128 +138,139 @@ ls -lt /var/log/borg/backup-*.log | head -1 # latest log file
|
|||||||
tail -50 /var/log/borg/backup-*.log # inspect it
|
tail -50 /var/log/borg/backup-*.log # inspect it
|
||||||
```
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 4. Day-2 Operations
|
## 4. Day-2 Operations
|
||||||
|
|
||||||
List archives:
|
Source your config first to get the right `BORG_REPO`, `BORG_PASSCOMMAND`, etc.:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
source /opt/backup-agent/configs/nexusvoice.conf # adapt
|
||||||
|
export BORG_REPO="$REPO"
|
||||||
|
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"
|
||||||
|
```
|
||||||
|
|
||||||
|
Then standard borg commands work without extra flags:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List archives
|
||||||
./restore.sh --list-archives
|
./restore.sh --list-archives
|
||||||
|
# or directly:
|
||||||
|
borg list
|
||||||
|
|
||||||
|
# Repo size and health
|
||||||
|
borg info
|
||||||
|
|
||||||
|
# Rotate the passphrase
|
||||||
|
# Note: only re-encrypts the key, not the archive data; old passphrase
|
||||||
|
# still needed for archives created before the change until fully migrated.
|
||||||
|
borg key change-passphrase
|
||||||
|
|
||||||
|
# Break a stale lockfile (backup or restore aborted mid-run)
|
||||||
|
borg break-lock
|
||||||
```
|
```
|
||||||
|
|
||||||
Check repo size and health:
|
---
|
||||||
|
|
||||||
```bash
|
|
||||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg info /home/srv/files/backups/borg-2025
|
|
||||||
```
|
|
||||||
|
|
||||||
Rotate the passphrase (creates a new key, re-encrypts nothing — old
|
|
||||||
archives still need the old passphrase to read, so keep both until fully
|
|
||||||
migrated):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg key change-passphrase /home/srv/files/backups/borg-2025
|
|
||||||
```
|
|
||||||
|
|
||||||
Stale lockfile (backup or restore aborted mid-run and left the repo
|
|
||||||
locked):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg break-lock /home/srv/files/backups/borg-2025
|
|
||||||
```
|
|
||||||
|
|
||||||
## 5. Recovery Procedures
|
## 5. Recovery Procedures
|
||||||
|
|
||||||
All `restore.sh` commands accept `--dry-run` to preview exactly what would
|
All `restore.sh` commands accept:
|
||||||
happen without touching anything, and `--archive NAME` to target a
|
- `--dry-run` — preview exactly what would happen without touching anything
|
||||||
specific archive instead of the latest (see archive names via
|
- `--archive NAME` — target a specific archive instead of the latest (get names via `--list-archives`)
|
||||||
`--list-archives`). Root DB credentials come from `MYSQL_ROOT_PASSWORD` in
|
|
||||||
the environment if set, otherwise from `/root/.mariadb-root.pw` — set
|
Root DB credentials come from `MYSQL_ROOT_PASSWORD` in the environment if set,
|
||||||
whichever is more convenient for how you're invoking it. Restore logs go
|
otherwise from the `ROOT_PASSWORD_FILE` in your config (default
|
||||||
to `/var/log/borg/restore-*.log` (the `RESTORE_LOGDIR` environment
|
`/root/.mariadb-root.pw`). Restore logs go to `/var/log/borg/restore-*.log`;
|
||||||
variable overrides the directory, mainly useful for testing). `full` and
|
`RESTORE_LOGDIR` overrides the directory (useful for testing).
|
||||||
`db` share `borg-backup.sh`'s lockfile, so a restore refuses to start
|
|
||||||
while the nightly backup is mid-run (and vice versa) rather than racing
|
`restore.sh` shares `borg-backup.sh`'s lockfile (`LOCKFILE` in the config),
|
||||||
it.
|
so a restore refuses to start while the nightly backup is mid-run and vice
|
||||||
|
versa.
|
||||||
|
|
||||||
|
For all `restore.sh` commands, select the deployment with `BACKUP_CONF`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
Or run without it if `backup.conf` is already symlinked beside the script.
|
||||||
|
|
||||||
### 5.1 Full disaster recovery (new or wiped server)
|
### 5.1 Full disaster recovery (new or wiped server)
|
||||||
|
|
||||||
Use when the whole server/container is gone and you're rebuilding from
|
Use when the whole server/container is gone and you're rebuilding from scratch.
|
||||||
scratch.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 1. Reinstall borg, docker, and the mariadb container image/compose file
|
# 1. Reinstall borg, docker, and the mariadb container image/compose file
|
||||||
# (not covered by restore.sh - this is infra provisioning).
|
# (not covered by restore.sh — this is infra provisioning).
|
||||||
# 2. Restore the passphrase file (from your password manager / secondary
|
# 2. Restore the passphrase file (from your password manager / secondary
|
||||||
# backup - it is NOT stored in the repo it protects) to
|
# backup — it is NOT stored in the repo it protects) to the path set in
|
||||||
# /root/.borg-passphrase, and the root DB password to
|
# BORG_PASSPHRASE_FILE, and the root DB password to ROOT_PASSWORD_FILE.
|
||||||
# /root/.mariadb-root.pw.
|
|
||||||
# 3. Preview:
|
# 3. Preview:
|
||||||
./restore.sh full --dry-run
|
./restore.sh full --dry-run
|
||||||
# 4. Run for real. --force is only required if /home/srv/files/content
|
# 4. Run for real. --force is only required if $TARGET already has data in it
|
||||||
# already has data in it (e.g. a stale mount); omit it on a genuinely
|
# (e.g. a stale mount); omit it on a genuinely empty/fresh server:
|
||||||
# empty/fresh server:
|
|
||||||
./restore.sh full --force
|
./restore.sh full --force
|
||||||
```
|
```
|
||||||
|
|
||||||
This extracts the full content tree from the archive, starts the
|
This extracts the full content tree, starts the DB container and waits for it
|
||||||
`mariadb` container and waits for it to report healthy, then restores
|
to report healthy, then restores every database dump (users/grants first).
|
||||||
every database dump (users/grants first) — the container must be running
|
|
||||||
before any of the dump restores, which is why it starts first.
|
|
||||||
|
|
||||||
**Verify afterward:**
|
**Verify afterward:**
|
||||||
- `docker ps` shows `mariadb` running and healthy.
|
- `docker ps` shows the DB container running and healthy.
|
||||||
- The application responds normally.
|
- The application responds normally.
|
||||||
- Spot-check row counts on a couple of tables against what you'd expect.
|
- Spot-check row counts on a couple of tables against what you'd expect.
|
||||||
|
|
||||||
### 5.2 Single database restore
|
### 5.2 Single database restore
|
||||||
|
|
||||||
Use when one database got corrupted or someone ran a bad migration/query
|
Use when one database got corrupted or someone ran a bad migration/query —
|
||||||
against it — this **drops and recreates** that database.
|
this **drops and recreates** that database.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./restore.sh db shopdb --dry-run # preview
|
./restore.sh db shopdb --dry-run # preview
|
||||||
./restore.sh db shopdb # prompts: type "shopdb" to confirm
|
./restore.sh db shopdb # prompts: type "shopdb" to confirm
|
||||||
```
|
```
|
||||||
|
|
||||||
Non-interactive (e.g. scripted from a monitoring alert): add `--yes` to
|
Non-interactive (e.g. scripted from a monitoring alert): add `--yes` to
|
||||||
skip the typed confirmation.
|
skip the typed confirmation.
|
||||||
|
|
||||||
**Verify afterward:** connect to the database and check the tables/row
|
**Verify afterward:** connect to the database and check the tables/row counts
|
||||||
counts you expect.
|
you expect.
|
||||||
|
|
||||||
### 5.3 Single file/directory restore
|
### 5.3 Single file/directory restore
|
||||||
|
|
||||||
Use for accidental deletion of a file, or to inspect an old version — this
|
Use for accidental deletion or to inspect an old version — never touches the
|
||||||
never touches the running database or container.
|
running database or container.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./restore.sh file path/relative/to/content/some-file.txt --dest /tmp/recovered
|
./restore.sh file path/relative/to/target/some-file.txt --dest /tmp/recovered
|
||||||
```
|
```
|
||||||
|
|
||||||
The final location of the recovered item is printed at the end (it lands
|
The final location of the recovered item is printed at the end. Borg
|
||||||
under `/tmp/recovered/home/srv/files/content/...` — Borg preserves the
|
preserves the absolute path it was archived with, so the file lands under
|
||||||
absolute path it was archived with).
|
`/tmp/recovered/<TARGET>/...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 6. Restore Drill Cadence
|
## 6. Restore Drill Cadence
|
||||||
|
|
||||||
Quarterly, run a real `full` restore into a scratch directory (not
|
Quarterly, run a real `full` extract into a scratch directory to confirm
|
||||||
`/home/srv/files/content`) to confirm backups are actually usable:
|
backups are actually usable.
|
||||||
|
|
||||||
`borg extract` always extracts into the current directory (there's no
|
`borg extract` always extracts into the current directory; the archive name
|
||||||
`--destination` flag — this is why `restore.sh` itself `cd`s into the
|
must come from `borg list --short` (plain `borg list` prints a formatted
|
||||||
destination before extracting), and the archive name must come from
|
line, not a bare name). Source your config to get the right values:
|
||||||
`borg list --short` (plain `borg list` prints a formatted line, not a bare
|
|
||||||
name), so:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
source /opt/backup-agent/configs/nexusvoice.conf # adapt
|
||||||
|
export BORG_REPO="$REPO"
|
||||||
|
export BORG_PASSCOMMAND="cat $BORG_PASSPHRASE_FILE"
|
||||||
|
|
||||||
mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
|
mkdir -p /tmp/restore-drill && cd /tmp/restore-drill
|
||||||
export BORG_REPO=/home/srv/files/backups/borg-2025
|
|
||||||
export BORG_PASSCOMMAND="cat /root/.borg-passphrase"
|
|
||||||
LATEST=$(borg list --short | tail -1)
|
LATEST=$(borg list --short | tail -1)
|
||||||
borg extract --lock-wait 600 "::$LATEST"
|
borg extract --lock-wait 600 "::$LATEST"
|
||||||
```
|
```
|
||||||
|
|
||||||
Confirm the dump files under `mariadb/dump/` are present, non-empty, and
|
Confirm the dump files under `${DUMP_SUBDIR}/` are present, non-empty, and
|
||||||
importable (`mysql -u root -p < mariadb/dump/somedb.sql` against a
|
importable (`mysql -u root -p < mariadb/dump/somedb.sql` against a throwaway
|
||||||
throwaway MariaDB container). Log the drill date and outcome somewhere
|
MariaDB container). Log the drill date and outcome somewhere durable (e.g. a
|
||||||
durable (e.g. a team wiki page).
|
team wiki page).
|
||||||
|
|||||||
+47
-49
@@ -1,6 +1,6 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Borg backup: /content + MariaDB -> local repo -> Scaleway S3 mirror
|
# Borg backup: $TARGET + MariaDB -> local borg repo -> Scaleway S3 mirror
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# CHANGES vs. original:
|
# CHANGES vs. original:
|
||||||
# • Encryption enabled (repokey-blake2 via BORG_PASSCOMMAND)
|
# • Encryption enabled (repokey-blake2 via BORG_PASSCOMMAND)
|
||||||
@@ -12,6 +12,7 @@
|
|||||||
# excluded from the archive (drop a .nobackup file in it - see
|
# excluded from the archive (drop a .nobackup file in it - see
|
||||||
# --exclude-if-present below) so no live InnoDB file is ever copied.
|
# --exclude-if-present below) so no live InnoDB file is ever copied.
|
||||||
# Zero DB downtime during backup.
|
# Zero DB downtime during backup.
|
||||||
|
# • Config split into per-deployment backup.conf (see configs/)
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
@@ -20,40 +21,28 @@ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
|||||||
umask 077
|
umask 077
|
||||||
|
|
||||||
# ========================= CONFIGURATION =========================
|
# ========================= CONFIGURATION =========================
|
||||||
|
# All deployment-specific constants live in a config file.
|
||||||
|
# Default: backup.conf next to this script.
|
||||||
|
# Override: BACKUP_CONF=/path/to/other.conf borg-backup.sh
|
||||||
|
# See configs/ for per-deployment examples.
|
||||||
|
|
||||||
NAME="chaudron"
|
_SCRIPT_DIR="$(cd "$(dirname "$(readlink -f "$0")")" && pwd)"
|
||||||
REPO="/home/srv/files/backups/$NAME"
|
BACKUP_CONF="${BACKUP_CONF:-${_SCRIPT_DIR}/backup.conf}"
|
||||||
TARGET="/home/srv/files/content"
|
[[ -r "$BACKUP_CONF" ]] || { echo "ERROR: config not found: $BACKUP_CONF"; exit 1; }
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "$BACKUP_CONF"
|
||||||
|
|
||||||
# dump_db.sh lives with the rest of this toolkit, not inside $TARGET - its
|
# Derived values (override in config only if you need a non-standard layout)
|
||||||
# own default dump dir is relative to wherever IT lives, so DUMP_DIR must be
|
# dump_db.sh lives with this toolkit; DUMP_DIR is passed explicitly so dumps
|
||||||
# passed explicitly (below) to keep dumps inside $TARGET where borg can see them.
|
# land inside $TARGET where borg can see them.
|
||||||
DUMP_SCRIPT="/opt/backup-agent/dump_db.sh"
|
export DUMP_DIR="${DUMP_DIR:-${TARGET}/${DUMP_SUBDIR}}"
|
||||||
export DUMP_DIR="${TARGET}/mariadb/dump"
|
|
||||||
|
|
||||||
REPO_MOUNT=""
|
# docker is only required when DB_CONTAINER is set
|
||||||
|
if [[ -n "${DB_CONTAINER:-}" ]]; then
|
||||||
LOGDIR="/var/log/borg"
|
REQUIRED_CMDS=(borg rclone docker timeout flock date find)
|
||||||
LOG_RETENTION_DAYS=90
|
else
|
||||||
|
REQUIRED_CMDS=(borg rclone timeout flock date find)
|
||||||
RCLONE_REMOTE="scaleway"
|
fi
|
||||||
RCLONE_PATH="cyanet-backups-prod/$NAME"
|
|
||||||
|
|
||||||
LOCKFILE="/var/lock/borg-backup.lock"
|
|
||||||
|
|
||||||
DB_CONTAINER="mariadb"
|
|
||||||
DB_START_TIMEOUT=180
|
|
||||||
|
|
||||||
CREATE_TIMEOUT="6h"
|
|
||||||
PRUNE_TIMEOUT="2h"
|
|
||||||
SYNC_TIMEOUT="12h"
|
|
||||||
CHECK_TIMEOUT="4h"
|
|
||||||
|
|
||||||
# Passphrase file: chmod 600, owned by the backup user.
|
|
||||||
# Create it with: echo 'your-strong-passphrase' > /root/.borg-passphrase
|
|
||||||
BORG_PASSPHRASE_FILE="${BORG_PASSPHRASE_FILE:-/root/.borg-passphrase}"
|
|
||||||
|
|
||||||
REQUIRED_CMDS=(borg rclone docker timeout flock date find)
|
|
||||||
|
|
||||||
# =================================================================
|
# =================================================================
|
||||||
|
|
||||||
@@ -127,7 +116,7 @@ cleanup() {
|
|||||||
|
|
||||||
# We never stop the container ourselves anymore, but if it crashed for
|
# We never stop the container ourselves anymore, but if it crashed for
|
||||||
# an unrelated reason during the backup window, try to bring it back.
|
# an unrelated reason during the backup window, try to bring it back.
|
||||||
if ! container_running "$DB_CONTAINER"; then
|
if [[ -n "${DB_CONTAINER:-}" ]] && ! container_running "$DB_CONTAINER"; then
|
||||||
step "Cleanup: $DB_CONTAINER is down, attempting restart"
|
step "Cleanup: $DB_CONTAINER is down, attempting restart"
|
||||||
start_db || { log "CRITICAL: $DB_CONTAINER is DOWN - manual action required"
|
start_db || { log "CRITICAL: $DB_CONTAINER is DOWN - manual action required"
|
||||||
[ "$exit_code" -eq 0 ] && exit_code=1; }
|
[ "$exit_code" -eq 0 ] && exit_code=1; }
|
||||||
@@ -191,8 +180,10 @@ preflight() {
|
|||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
container_running "$DB_CONTAINER" \
|
if [[ -n "${DB_CONTAINER:-}" ]]; then
|
||||||
|| { log "ERROR: $DB_CONTAINER is not running before we start"; return 1; }
|
container_running "$DB_CONTAINER" \
|
||||||
|
|| { log "ERROR: $DB_CONTAINER is not running before we start"; return 1; }
|
||||||
|
fi
|
||||||
|
|
||||||
log "Preflight OK"
|
log "Preflight OK"
|
||||||
}
|
}
|
||||||
@@ -223,20 +214,24 @@ preflight
|
|||||||
# --- 1. Logical dumps (container stays up the whole time) ---------------
|
# --- 1. Logical dumps (container stays up the whole time) ---------------
|
||||||
step "Step 1: MariaDB dumps"
|
step "Step 1: MariaDB dumps"
|
||||||
|
|
||||||
if [[ ! -x "$DUMP_SCRIPT" ]]; then
|
if [[ -n "${DB_CONTAINER:-}" ]]; then
|
||||||
log "ERROR: dump script missing or not executable: $DUMP_SCRIPT"
|
if [[ ! -x "$DUMP_SCRIPT" ]]; then
|
||||||
exit 1
|
log "ERROR: dump script missing or not executable: $DUMP_SCRIPT"
|
||||||
fi
|
exit 1
|
||||||
run_cmd "$DUMP_SCRIPT"
|
fi
|
||||||
|
run_cmd "$DUMP_SCRIPT"
|
||||||
|
|
||||||
if [[ -d "$DUMP_DIR" ]]; then
|
if [[ -d "$DUMP_DIR" ]]; then
|
||||||
fresh=$(find "$DUMP_DIR" -type f -size +1k -mmin -60 | wc -l)
|
fresh=$(find "$DUMP_DIR" -type f -size +1k -mmin -60 | wc -l)
|
||||||
empty=$(find "$DUMP_DIR" -type f -size -1k -mmin -60 | wc -l)
|
empty=$(find "$DUMP_DIR" -type f -size -1k -mmin -60 | wc -l)
|
||||||
log "Dumps: $fresh fresh non-trivial file(s), $empty suspiciously small"
|
log "Dumps: $fresh fresh non-trivial file(s), $empty suspiciously small"
|
||||||
(( fresh > 0 )) || { log "ERROR: no usable dumps produced"; exit 1; }
|
(( fresh > 0 )) || { log "ERROR: no usable dumps produced"; exit 1; }
|
||||||
(( empty == 0 )) || log "WARNING: $empty near-empty dump file(s) - check $DUMP_DIR"
|
(( empty == 0 )) || log "WARNING: $empty near-empty dump file(s) - check $DUMP_DIR"
|
||||||
|
else
|
||||||
|
log "WARNING: dump directory not found: $DUMP_DIR"
|
||||||
|
fi
|
||||||
else
|
else
|
||||||
log "WARNING: dump directory not found: $DUMP_DIR"
|
log "DB_CONTAINER not set - skipping database dump"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# --- 2. Create the archive -----------------------------------------------
|
# --- 2. Create the archive -----------------------------------------------
|
||||||
@@ -247,8 +242,10 @@ fi
|
|||||||
# are excluded too - they churn on every request, so backing them up adds
|
# are excluded too - they churn on every request, so backing them up adds
|
||||||
# noise and dedup overhead for no recovery value.
|
# noise and dedup overhead for no recovery value.
|
||||||
step "Step 2: Creating archive $ARCHIVE"
|
step "Step 2: Creating archive $ARCHIVE"
|
||||||
container_running "$DB_CONTAINER" \
|
if [[ -n "${DB_CONTAINER:-}" ]]; then
|
||||||
|| { log "ERROR: $DB_CONTAINER is not running - refusing to archive"; exit 1; }
|
container_running "$DB_CONTAINER" \
|
||||||
|
|| { log "ERROR: $DB_CONTAINER is not running - refusing to archive"; exit 1; }
|
||||||
|
fi
|
||||||
run_cmd timeout --signal=INT --kill-after=120s "$CREATE_TIMEOUT" \
|
run_cmd timeout --signal=INT --kill-after=120s "$CREATE_TIMEOUT" \
|
||||||
borg create \
|
borg create \
|
||||||
--lock-wait 600 \
|
--lock-wait 600 \
|
||||||
@@ -303,4 +300,5 @@ run_cmd timeout "$SYNC_TIMEOUT" \
|
|||||||
--retries-sleep 10s \
|
--retries-sleep 10s \
|
||||||
--low-level-retries 20 \
|
--low-level-retries 20 \
|
||||||
--s3-no-check-bucket \
|
--s3-no-check-bucket \
|
||||||
|
--max-delete "$RCLONE_MAX_DELETE" \
|
||||||
"$REPO" "${RCLONE_REMOTE}:${RCLONE_PATH}"
|
"$REPO" "${RCLONE_REMOTE}:${RCLONE_PATH}"
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# Deployment config: srv (primary server — /home/srv/files/content)
|
||||||
|
# Used by: borg-backup.sh, restore.sh
|
||||||
|
# Deploy: BACKUP_CONF=/opt/backup-agent/configs/srv.conf /opt/backup-agent/borg-backup.sh
|
||||||
|
# (or symlink configs/srv.conf -> ../backup.conf next to the scripts)
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
# === Repo & target ===
|
||||||
|
NAME="chaudron"
|
||||||
|
REPO="/home/srv/files/backups/${NAME}"
|
||||||
|
TARGET="/home/srv/files/content"
|
||||||
|
|
||||||
|
# === Database (Docker) ===
|
||||||
|
# Set DB_CONTAINER="" to skip all DB dump steps entirely.
|
||||||
|
DB_CONTAINER="mariadb"
|
||||||
|
DUMP_SUBDIR="mariadb/dump" # relative to $TARGET; dump_db.sh writes here
|
||||||
|
DUMP_SCRIPT="/opt/backup-agent/dump_db.sh"
|
||||||
|
|
||||||
|
# === Rclone offsite mirror ===
|
||||||
|
RCLONE_REMOTE="scaleway"
|
||||||
|
RCLONE_PATH="cyanet-backups-prod/${NAME}"
|
||||||
|
RCLONE_MAX_DELETE=200
|
||||||
|
|
||||||
|
# === Credentials ===
|
||||||
|
BORG_PASSPHRASE_FILE="/root/.borg-passphrase"
|
||||||
|
ROOT_PASSWORD_FILE="/root/.mariadb-root.pw" # restore only; needs CREATE/DROP
|
||||||
|
|
||||||
|
# === Paths ===
|
||||||
|
REPO_MOUNT="" # if non-empty, checked as a mountpoint
|
||||||
|
LOCKFILE="/var/lock/borg-backup.lock"
|
||||||
|
LOGDIR="/var/log/borg"
|
||||||
|
LOG_RETENTION_DAYS=90
|
||||||
|
|
||||||
|
# === Timeouts ===
|
||||||
|
DB_START_TIMEOUT=180
|
||||||
|
CREATE_TIMEOUT="6h"
|
||||||
|
PRUNE_TIMEOUT="2h"
|
||||||
|
SYNC_TIMEOUT="12h"
|
||||||
|
CHECK_TIMEOUT="4h"
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# =============================================================================
|
||||||
|
# Deployment config: nexusvoice (/home/acid/nexusvoice — may move to /opt/nexusvoice)
|
||||||
|
# Used by: borg-backup.sh, restore.sh
|
||||||
|
# Deploy: BACKUP_CONF=/opt/backup-agent/configs/nexusvoice.conf /opt/backup-agent/borg-backup.sh
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
# === Repo & target ===
|
||||||
|
NAME="nexusvoice"
|
||||||
|
REPO="/opt/backups/${NAME}" # adjust if you want the repo elsewhere
|
||||||
|
TARGET="/home/acid/nexusvoice" # update to /opt/nexusvoice when files move
|
||||||
|
|
||||||
|
# === Database (Docker) ===
|
||||||
|
# Set DB_CONTAINER="" to skip all DB dump steps entirely.
|
||||||
|
DB_CONTAINER="mariadb" # TODO: confirm container name on this server
|
||||||
|
DUMP_SUBDIR="mariadb/dump" # relative to $TARGET; dump_db.sh writes here
|
||||||
|
DUMP_SCRIPT="/home/acid/nexusvoice/mariadb/dump_db.sh"
|
||||||
|
|
||||||
|
# === Rclone offsite mirror ===
|
||||||
|
RCLONE_REMOTE="scaleway"
|
||||||
|
RCLONE_PATH="cyanet-backups-prod/cilaos" # different directory from srv in same bucket
|
||||||
|
RCLONE_MAX_DELETE=200
|
||||||
|
|
||||||
|
# === Credentials ===
|
||||||
|
BORG_PASSPHRASE_FILE="/root/.borg-passphrase"
|
||||||
|
ROOT_PASSWORD_FILE="/root/.mariadb-root.pw" # restore only; needs CREATE/DROP
|
||||||
|
|
||||||
|
# === Paths ===
|
||||||
|
REPO_MOUNT="" # if non-empty, checked as a mountpoint
|
||||||
|
LOCKFILE="/var/lock/borg-backup.lock"
|
||||||
|
LOGDIR="/var/log/borg"
|
||||||
|
LOG_RETENTION_DAYS=90
|
||||||
|
|
||||||
|
# === Timeouts ===
|
||||||
|
DB_START_TIMEOUT=180
|
||||||
|
CREATE_TIMEOUT="6h"
|
||||||
|
PRUNE_TIMEOUT="2h"
|
||||||
|
SYNC_TIMEOUT="12h"
|
||||||
|
CHECK_TIMEOUT="4h"
|
||||||
+12
-11
@@ -14,25 +14,26 @@ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PATH"
|
|||||||
umask 077
|
umask 077
|
||||||
|
|
||||||
# ========================= CONFIGURATION =========================
|
# ========================= CONFIGURATION =========================
|
||||||
|
# All deployment-specific constants live in a config file.
|
||||||
|
# Default: backup.conf next to this script.
|
||||||
|
# Override: BACKUP_CONF=/path/to/other.conf restore.sh <cmd>
|
||||||
|
# See configs/ for per-deployment examples.
|
||||||
|
|
||||||
NAME="borg-2025"
|
_SCRIPT_DIR="$(cd "$(dirname "$(readlink -f "$0")")" && pwd)"
|
||||||
REPO="/home/srv/files/backups/$NAME"
|
BACKUP_CONF="${BACKUP_CONF:-${_SCRIPT_DIR}/backup.conf}"
|
||||||
TARGET="/home/srv/files/content"
|
[[ -r "$BACKUP_CONF" ]] || { echo "ERROR: config not found: $BACKUP_CONF"; exit 1; }
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "$BACKUP_CONF"
|
||||||
|
|
||||||
|
# Derived values
|
||||||
ARCHIVE_TARGET_PATH="${TARGET#/}"
|
ARCHIVE_TARGET_PATH="${TARGET#/}"
|
||||||
|
|
||||||
DB_CONTAINER="mariadb"
|
|
||||||
DB_START_TIMEOUT=180
|
|
||||||
|
|
||||||
BORG_PASSPHRASE_FILE="${BORG_PASSPHRASE_FILE:-/root/.borg-passphrase}"
|
|
||||||
ROOT_PASSWORD_FILE="${ROOT_PASSWORD_FILE:-/root/.mariadb-root.pw}"
|
|
||||||
DUMP_SUBDIR="mariadb/dump"
|
|
||||||
|
|
||||||
# Same lockfile borg-backup.sh takes (via flock -n 9) before touching $TARGET
|
# Same lockfile borg-backup.sh takes (via flock -n 9) before touching $TARGET
|
||||||
# or the repo, so a restore and the nightly backup cron job can never run
|
# or the repo, so a restore and the nightly backup cron job can never run
|
||||||
# concurrently against each other.
|
# concurrently against each other.
|
||||||
LOCKFILE="${LOCKFILE:-/var/lock/borg-backup.lock}"
|
LOCKFILE="${LOCKFILE:-/var/lock/borg-backup.lock}"
|
||||||
|
|
||||||
LOGDIR="${RESTORE_LOGDIR:-/var/log/borg}"
|
LOGDIR="${RESTORE_LOGDIR:-${LOGDIR:-/var/log/borg}}"
|
||||||
mkdir -p "$LOGDIR" 2>/dev/null || LOGDIR="/tmp"
|
mkdir -p "$LOGDIR" 2>/dev/null || LOGDIR="/tmp"
|
||||||
LOGFILE="$LOGDIR/restore-$(date +%Y-%m-%d-%H%M%S).log"
|
LOGFILE="$LOGDIR/restore-$(date +%Y-%m-%d-%H%M%S).log"
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user