Understanding backups¶
A backup is only worth something if you can restore from it. Everything else in this chapter serves that one goal: after a deleted file, a broken disk, ransomware or a fire in the data center, you get your data back, and you know how long it takes.
The 3-2-1 rule¶
| Meaning | On tuxdex-srv | |
|---|---|---|
| 3 copies | The original plus two backups | Data on sda, backup on /mnt/backup, a second backup offsite |
| 2 different media | Not everything on the same disk or system | Local backup disk plus a storage at another provider |
| 1 copy offsite | At least one copy at another location | restic or borg repository on an external server or S3 |
Add one more rule: at least one copy must be out of reach of the server, so that an attacker or ransomware with root access can't delete it (append-only repository, object lock, or a backup server that pulls).
What to back up¶
| What | How | Why not just copy the files |
|---|---|---|
Configuration (/etc) |
restic or borg | Small, but expensive to rebuild |
Web content, uploads (/srv) |
restic or borg | |
| Databases | pg_dump, mysqldump first, then back up the dump |
Copying the data files of a running database gives a broken backup |
| Docker volumes | Dump of the database inside, plus the volume directories | Same reason |
| Package list | dpkg --get-selections, dnf history |
To set up the same software again |
Not needed: /proc, /sys, /dev, /run, /tmp, caches and anything that comes from
packages or Git.
restic or borg?¶
Both deduplicate (each block is stored only once), compress, encrypt and keep many versions with little space.
restic |
borg |
|
|---|---|---|
| Targets | Local, SFTP, S3, Azure, Google Cloud, REST server, rclone | Local or SSH with borg on the other side |
| Encryption | Always | Selectable, usually repokey |
| Speed | Fast | Very fast, very good compression |
| Typical use | Backups to cloud storage | Backups to own servers or Hetzner Storage Box, BorgBase |
A backup routine¶
02:30 database dumps pg_dump / mysqldump → /var/backups/db/
02:40 restic backup /etc /srv /var/backups/db → repository
03:30 restic forget keep 7 daily, 4 weekly, 6 monthly, then prune
Sunday restic check read part of the data and verify it
Monthly restore test restore one file and one database into a test system
Run it as a systemd timer or cron job, see crontab, and
let it report failures, for example to the journal with logger and to your monitoring.
What you need to install¶
| Debian | Ubuntu | RHEL | Arch | |
|---|---|---|---|---|
| restic | sudo apt install restic |
sudo apt install restic |
sudo dnf install restic (EPEL) |
sudo pacman -S restic |
| borg | sudo apt install borgbackup |
sudo apt install borgbackup |
sudo dnf install borgbackup (EPEL) |
sudo pacman -S borg |
| pg_dump | with postgresql-client |
with postgresql-client |
with postgresql |
with postgresql |
| mysqldump | with mariadb-client |
with mysql-client or mariadb-client |
with mysql or mariadb |
with mariadb-clients |
Test the restore
Schedule a restore test, for example once a month: restore a random file and a database dump into a test system and check them. Only then do you know that the backup works and how long a restore takes.
Without the password, the backup is gone
restic and borg encrypt everything. Store the repository password (for borg also the exported key) outside the server, for example in your password manager. If the server burns, the password must not burn with it.