Howto Backup With Borg And Borgmatic To Remote Location
Also see How to back up with borg on USB drive
What I wanted:
- no shell access on remote
- can’t delete on remote location
- deduplicating
- encryption password is not stored on remote server
And:
-
rsync and rrsync (
/usr/share/doc/rsync/scripts/rrsync
) turned out to not work very well - I got BORG_RELOCATED_REPO_ACCESS_IS_OK errors and it was very inefficientcp /usr/share/doc/rsync/scripts/rrsync /usr/local/bin/ command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAA...vp Automated remote backup
-
borg on source and destination server do the job
Source
Install borg and borgmatic on the source server. Borgmatic is declarative interface to borg configured by a YAML file:
apt-get install borgbackup
apt install python3-pip
sudo pip3 uninstall borgmatic
sudo pip3 install --upgrade borgmatic
Create borg repository on source server, I use the same password for source and destination repositories:
mkdir -p /backups/borg
cd /backups/borg
borg init --encryption=repokey-blake2 .
borg config . additional_free_space 2G
Update borgmatic config file with destination server:
$ cat /etc/borgmatic/config.yaml
Notice location, repositories, encryption_passphrase, before_backup, after_backup and retention sections.
# Where to look for files to backup, and where to store those backups. See
# https://borgbackup.readthedocs.io/en/stable/quickstart.html and
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create for details.
location:
# List of source directories to backup (required). Globs and tildes are expanded.
source_directories:
- /backups/borgcache
- /root
- /home
- /var/www
- /usr/local
# already backed by script as tar.gz file
# - /etc
# - /var/log/syslog*
# Paths to local or remote repositories (required). Tildes are expanded. Multiple
# repositories are backed up to in sequence. See ssh_command for SSH options like
# identity file or port.
repositories:
- /backups/borg/
- borger@destination_server.sk:backups/borg
# Stay in same file system (do not cross mount points).
one_file_system: true
# Use Borg's --read-special flag to allow backup of block and other special
# devices. Use with caution, as it will lead to problems if used when
# backing up special devices such as /dev/zero.
#read_special: false
# Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive. Defaults to true.
#bsd_flags: true
# Mode in which to operate the files cache. See
# https://borgbackup.readthedocs.io/en/stable/usage/create.html#description for
# details.
#files_cache: ctime,size,inode
# Alternate Borg local executable. Defaults to "borg".
#local_path: borg1
# Alternate Borg remote executable. Defaults to "borg".
#remote_path: borg1
# Any paths matching these patterns are included/excluded from backups. Globs are
# expanded. (Tildes are not.) Note that Borg considers this option experimental.
# See the output of "borg help patterns" for more details. Quote any value if it
# contains leading punctuation, so it parses correctly.
#patterns:
# - R /
# - '- /home/*/.cache'
# - + /home/susan
# - '- /home/*'
# Read include/exclude patterns from one or more separate named files, one pattern
# per line. Note that Borg considers this option experimental. See the output of
# "borg help patterns" for more details.
#patterns_from:
# - /etc/borgmatic/patterns
# Any paths matching these patterns are excluded from backups. Globs and tildes
# are expanded. See the output of "borg help patterns" for more details.
#exclude_patterns:
# - '*.pyc'
# - ~/*/.cache
# - /etc/ssl
# Read exclude patterns from one or more separate named files, one pattern per
# line. See the output of "borg help patterns" for more details.
#exclude_from:
# - /etc/borgmatic/excludes
# Exclude directories that contain a CACHEDIR.TAG file. See
# http://www.brynosaurus.com/cachedir/spec.html for details.
#exclude_caches: true
# Exclude directories that contain a file with the given filename.
#exclude_if_present: .nobackup
# Repository storage options. See
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and
# https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for
# details.
storage:
# The standard output of this command is used to unlock the encryption key. Only
# use on repositories that were initialized with passcommand/repokey encryption.
# Note that if both encryption_passcommand and encryption_passphrase are set,
# then encryption_passphrase takes precedence.
#encryption_passcommand: secret-tool lookup borg-repository repo-name
# Passphrase to unlock the encryption key with. Only use on repositories that were
# initialized with passphrase/repokey encryption. Quote the value if it contains
# punctuation, so it parses correctly. And backslash any quote or backslash
# literals as well.
encryption_passphrase: "my super strong password"
# Number of seconds between each checkpoint during a long-running backup. See
# https://borgbackup.readthedocs.io/en/stable/faq.html#if-a-backup-stops-mid-way-does-the-already-backed-up-data-stay-there
# for details. Defaults to checkpoints every 1800 seconds (30 minutes).
#checkpoint_interval: 1800
# Specify the parameters passed to then chunker (CHUNK_MIN_EXP, CHUNK_MAX_EXP,
# HASH_MASK_BITS, HASH_WINDOW_SIZE). See https://borgbackup.readthedocs.io/en/stable/internals.html
# for details.
#chunker_params: 19,23,21,4095
# Type of compression to use when creating archives. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create for details.
# Defaults to no compression.
compression: zstd
# Remote network upload rate limit in kiBytes/second.
#remote_rate_limit: 100
# Command to use instead of just "ssh". This can be used to specify ssh options.
#ssh_command: ssh -i /path/to/private/key
# Umask to be used for borg create.
#umask: 0077
# Maximum seconds to wait for acquiring a repository/cache lock.
#lock_wait: 5
# Name of the archive. Borg placeholders can be used. See the output of
# "borg help placeholders" for details. Default is
# "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this option, you must
# also specify a prefix in the retention section to avoid accidental pruning of
# archives with a different archive name format. And you should also specify a
# prefix in the consistency section as well.
#archive_name_format: '{hostname}-documents-{now}'
# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
# At least one of the "keep" options is required for pruning to work.
retention:
# Keep all archives within this time interval.
#keep_within: 3H
# Number of secondly archives to keep.
#keep_secondly: 60
# Number of minutely archives to keep.
#keep_minutely: 60
# Number of hourly archives to keep.
#keep_hourly: 24
# Number of daily archives to keep.
keep_daily: 7
# Number of weekly archives to keep.
#keep_weekly: 4
# Number of monthly archives to keep.
keep_monthly: 6
# Number of yearly archives to keep.
keep_yearly: 3
# When pruning, only consider archive names starting with this prefix.
# Borg placeholders can be used. See the output of "borg help placeholders" for
# details. Default is "{hostname}-".
#prefix: sourcehostname
# Consistency checks to run after backups. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check and
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-extract for details.
#consistency:
# List of one or more consistency checks to run: "repository", "archives", and/or
# "extract". Defaults to "repository" and "archives". Set to "disabled" to disable
# all consistency checks. "repository" checks the consistency of the repository,
# "archive" checks all of the archives, and "extract" does an extraction dry-run
# of just the most recent archive.
#checks:
# - repository
# - archives
# Paths to a subset of the repositories in the location section on which to run
# consistency checks. Handy in case some of your repositories are very large, and
# so running consistency checks on them would take too long. Defaults to running
# consistency checks on all repositories configured in the location section.
#check_repositories:
# - user@backupserver:sourcehostname.borg
# Restrict the number of checked archives to the last n. Applies only to the "archives" check.
#check_last: 3
# When performing the "archives" check, only consider archive names starting with
# this prefix. Borg placeholders can be used. See the output of
# "borg help placeholders" for details. Default is "{hostname}-".
#prefix: sourcehostname
# Shell commands or scripts to execute before and after a backup or if an error has occurred.
# IMPORTANT: All provided commands and scripts are executed with user permissions of borgmatic.
# Do not forget to set secure permissions on this file as well as on any script listed (chmod 0700) to
# prevent potential shell injection or privilege escalation.
hooks:
# List of one or more shell commands or scripts to execute before creating a backup.
before_backup:
- echo "`date` - Starting a backup job."
# these back up data into /backups/borgcache
- backup-etc
- backup-database
# List of one or more shell commands or scripts to execute after creating a backup.
after_backup:
- echo "`date` - Backup created."
# List of one or more shell commands or scripts to execute in case an exception has occurred.
#on_error:
# - echo "`date` - Error while creating a backup."
Add borgmatic into daily cron jobs, so it runs every day, apply chmod 755:
cat /etc/cron.daily/borgmatic
#!/bin/bash
borgmatic --verbosity 1
And generate and set up SSH key to destination:
ssh-keygen -a 100 -t ed25519 -C "borger"
# cat /root/.ssh/config
Host backupserver
HostName destination_server.sk
User borger
IdentityFile ~/.ssh/id_ed25519
Destination
As root install borg on destination server:
$ apt-get install borgbackup
Add user:
$ useradd borger
As borger user create repo, the password is the same as in borgmatic configuration:
mkdir -p ~/backups/borg
cd ~/backups/borg
borg init --encryption=repokey-blake2 .
borg config . additional_free_space 2G
borg config . append_only 1
append_only to not have anything removed, free_space for borg cannot prune while no free space and you will die.
As borger user add public SSH key of source user to destination server:
cat .ssh/authorized_keys
command="borg serve --restrict-to-path ~/backups/borg",restrict ssh-ed25519 AAAAmykey root@source
chmod 700 ~/.ssh
chod 600 ~/.ssh/id_ed25519
Running it
And now run it on source and you should see something like this (I have backups from August):
/etc/cron.daily/borgmatic
/etc/borgmatic/config.yaml: Running 4 commands for pre-backup hook
Sun dec 6 13:47:49 CET 2020 - Starting a backup job.
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
/backups/borg/: Pruning archives
/backups/borg/: Creating archive
Creating archive at "/backups/borg/::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
/backups/borg/: Running consistency checks
Starting repository check
Starting repository index check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive source-server-2020-07-31T06:35:24.333633 (1/12)
Analyzing archive source-server-2020-08-31T06:29:25.065722 (2/12)
Analyzing archive source-server-2020-09-30T06:31:43.033300 (3/12)
Analyzing archive source-server-2020-10-31T06:31:38.222528 (4/12)
Analyzing archive source-server-2020-11-30T06:30:23.411184 (5/12)
Analyzing archive source-server-2020-12-01T06:29:44.704769 (6/12)
Analyzing archive source-server-2020-12-02T06:30:25.994776 (7/12)
Analyzing archive source-server-2020-12-03T06:29:51.315834 (8/12)
Analyzing archive source-server-2020-12-04T06:29:35.802369 (9/12)
Analyzing archive source-server-2020-12-05T06:29:23.555802 (10/12)
Analyzing archive source-server-2020-12-06T06:29:51.908761 (11/12)
Analyzing archive source-server-2020-12-06T13:49:09.584833 (12/12)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
borger@destination_server.sk:backups/borg: Pruning archives
borger@destination_server.sk:backups/borg: Creating archive
Creating archive at "borger@destination_server.sk:backups/borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
borger@destination_server.sk:backups/borg: Running consistency checks
Remote: Starting repository check
Remote: Starting repository index check
Remote: Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive source-server-2020-07-31T06:41:25.875108 (1/12)
Analyzing archive source-server-2020-08-31T06:35:52.625629 (2/12)
Analyzing archive source-server-2020-09-30T06:40:06.208063 (3/12)
Analyzing archive source-server-2020-10-31T06:40:39.288971 (4/12)
Analyzing archive source-server-2020-11-30T06:40:58.945221 (5/12)
Analyzing archive source-server-2020-12-01T06:40:27.373710 (6/12)
Analyzing archive source-server-2020-12-02T06:41:14.762190 (7/12)
Analyzing archive source-server-2020-12-03T06:40:33.421019 (8/12)
Analyzing archive source-server-2020-12-04T06:40:24.263220 (9/12)
Analyzing archive source-server-2020-12-05T06:40:04.474899 (10/12)
Analyzing archive source-server-2020-12-06T06:40:34.384012 (11/12)
Analyzing archive source-server-2020-12-06T13:59:31.589336 (12/12)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
/etc/borgmatic/config.yaml: Running command for post-backup hook
Ne dec 6 14:11:26 CET 2020 - Backup created.
summary:
/etc/borgmatic/config.yaml: Successfully ran configuration file
List backups:
borg list --debug /backups/borg
Enter passphrase for key /backups/borg:
source-server-2020-07-31T06:35:24.333633 Fri, 2020-07-31 06:35:24 [58aa33f782413e5d9ffb941b880d3c3deca4015938c446cc6b1b017dd9633b44]
source-server-2020-08-31T06:29:25.065722 Mon, 2020-08-31 06:29:25 [30f1885dc224e32a6e76af7b4e9fdcc5cb064e8a343bade04ab6438d023df065]
source-server-2020-09-30T06:31:43.033300 Wed, 2020-09-30 06:31:43 [7944462de0c10584b9879e302bff5bfff2fd6ee6fde1778453ed52a42a93ff0c]
source-server-2020-10-31T06:31:38.222528 Sat, 2020-10-31 06:31:38 [67fbdf24ab59be5d3eaa8d123024e83d05e24983fa4f03a8c24fab08c7d0fdf2]
source-server-2020-11-30T06:30:23.411184 Mon, 2020-11-30 06:30:23 [c47c9c1ea704089e65c699e68625f1109d33d729e4440ffded0b5326157a26cc]
source-server-2020-12-01T06:29:44.704769 Tue, 2020-12-01 06:29:45 [e9db7e9afc4457db18def2346d8552230bd840e92069e3be1c6ccc1724c7ed4b]
source-server-2020-12-02T06:30:25.994776 Wed, 2020-12-02 06:30:26 [2f43b1591211d55a0968a7afa2e9e24744f18f9e7c42dd270e7145dea1ea4798]
source-server-2020-12-03T06:29:51.315834 Thu, 2020-12-03 06:29:51 [02a4bd337a0c34d984fdbe1d842d61e5c984ca67401b78c3b336dc698e74c217]
source-server-2020-12-04T06:29:35.802369 Fri, 2020-12-04 06:29:36 [ae8b54727a812ba5b1d56097852e63d5751532eb99a22516188754de42de316c]
source-server-2020-12-05T06:29:23.555802 Sat, 2020-12-05 06:29:24 [d44d3168f81722650e2a80f184bc7ab165a5373b3b45360756d45713739b30b4]
source-server-2020-12-06T06:29:51.908761 Sun, 2020-12-06 06:29:52 [999ba07baddf4e962ab63da0579e61a36a721d9aacb28bfad6bc33a35890050a]
source-server-2020-12-06T13:49:09.584833 Sun, 2020-12-06 13:49:10 [0306c63b9fc375165449709f987699b33775eb19d8c0fed488bce14d88dc7165]
You can get too many open files error when borgmatic is running:
/home/someuser/python: scandir: [Errno 24] Too many open files: '/home/someuser/python'
Add this into /etc/security/limits.conf
and relog:
root - nofile 1000000
By default, max open files is 1024:
$ ulimit | grep files
open files (-n) 1024
And remember, you need both repository key and password to decrypt it.
That’s all.
Add Comment