Mind Dump, Tech And Life Blog
written by Ivan Alenko
published under license Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)copy! share!
posted in category Digital Despotism / Software
posted at 09. Dec '20

Howto Backup With Borg And Borgmatic To Remote Location

Also see How to back up with borg on USB drive

What I wanted:

  • no shell access on remote
  • can’t delete on remote location
  • deduplicating
  • encryption password is not stored on remote server

And:

  • rsync and rrsync (/usr/share/doc/rsync/scripts/rrsync) turned out to not work very well - I got BORG_RELOCATED_REPO_ACCESS_IS_OK errors and it was very inefficient

    cp /usr/share/doc/rsync/scripts/rrsync /usr/local/bin/
    command="$HOME/bin/rrsync -ro ~/backups/",no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-rsa AAA...vp Automated remote backup
    
  • borg on source and destination server do the job

Source

Install borg and borgmatic on the source server. Borgmatic is declarative interface to borg configured by a YAML file:

apt-get install borgbackup

apt install python3-pip
sudo pip3 uninstall borgmatic
sudo pip3 install --upgrade borgmatic

Create borg repository on source server, I use the same password for source and destination repositories:

mkdir -p /backups/borg
cd /backups/borg
borg init --encryption=repokey-blake2 .
borg config . additional_free_space 2G

Update borgmatic config file with destination server:

$ cat /etc/borgmatic/config.yaml

Notice location, repositories, encryption_passphrase, before_backup, after_backup and retention sections.

# Where to look for files to backup, and where to store those backups. See
# https://borgbackup.readthedocs.io/en/stable/quickstart.html and
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create for details.
location:
    # List of source directories to backup (required). Globs and tildes are expanded.
    source_directories:
        - /backups/borgcache
        - /root
        - /home
        - /var/www
        - /usr/local
# already backed by script as tar.gz file
#        - /etc
#        - /var/log/syslog*

    # Paths to local or remote repositories (required). Tildes are expanded. Multiple
    # repositories are backed up to in sequence. See ssh_command for SSH options like
    # identity file or port.
    repositories:
        - /backups/borg/
        - borger@destination_server.sk:backups/borg

    # Stay in same file system (do not cross mount points).
    one_file_system: true

    # Use Borg's --read-special flag to allow backup of block and other special
    # devices. Use with caution, as it will lead to problems if used when
    # backing up special devices such as /dev/zero.
    #read_special: false

    # Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive. Defaults to true.
    #bsd_flags: true

    # Mode in which to operate the files cache. See
    # https://borgbackup.readthedocs.io/en/stable/usage/create.html#description for
    # details.
    #files_cache: ctime,size,inode

    # Alternate Borg local executable. Defaults to "borg".
    #local_path: borg1

    # Alternate Borg remote executable. Defaults to "borg".
    #remote_path: borg1

    # Any paths matching these patterns are included/excluded from backups. Globs are
    # expanded. (Tildes are not.) Note that Borg considers this option experimental.
    # See the output of "borg help patterns" for more details. Quote any value if it
    # contains leading punctuation, so it parses correctly.
    #patterns:
    #    - R /
    #    - '- /home/*/.cache'
    #    - + /home/susan
    #    - '- /home/*'

    # Read include/exclude patterns from one or more separate named files, one pattern
    # per line. Note that Borg considers this option experimental. See the output of
    # "borg help patterns" for more details.
    #patterns_from:
    #    - /etc/borgmatic/patterns

    # Any paths matching these patterns are excluded from backups. Globs and tildes
    # are expanded. See the output of "borg help patterns" for more details.
    #exclude_patterns:
    #    - '*.pyc'
    #    - ~/*/.cache
    #    - /etc/ssl

    # Read exclude patterns from one or more separate named files, one pattern per
    # line. See the output of "borg help patterns" for more details.
    #exclude_from:
    #    - /etc/borgmatic/excludes

    # Exclude directories that contain a CACHEDIR.TAG file. See
    # http://www.brynosaurus.com/cachedir/spec.html for details.
    #exclude_caches: true

    # Exclude directories that contain a file with the given filename.
    #exclude_if_present: .nobackup

# Repository storage options. See
# https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create and
# https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables for
# details.
storage:
    # The standard output of this command is used to unlock the encryption key. Only
    # use on repositories that were initialized with passcommand/repokey encryption.
    # Note that if both encryption_passcommand and encryption_passphrase are set,
    # then encryption_passphrase takes precedence.
    #encryption_passcommand: secret-tool lookup borg-repository repo-name

    # Passphrase to unlock the encryption key with. Only use on repositories that were
    # initialized with passphrase/repokey encryption. Quote the value if it contains
    # punctuation, so it parses correctly. And backslash any quote or backslash
    # literals as well.
    encryption_passphrase: "my super strong password"

    # Number of seconds between each checkpoint during a long-running backup. See
    # https://borgbackup.readthedocs.io/en/stable/faq.html#if-a-backup-stops-mid-way-does-the-already-backed-up-data-stay-there
    # for details. Defaults to checkpoints every 1800 seconds (30 minutes).
    #checkpoint_interval: 1800

    # Specify the parameters passed to then chunker (CHUNK_MIN_EXP, CHUNK_MAX_EXP,
    # HASH_MASK_BITS, HASH_WINDOW_SIZE). See https://borgbackup.readthedocs.io/en/stable/internals.html
    # for details.
    #chunker_params: 19,23,21,4095

    # Type of compression to use when creating archives. See
    # https://borgbackup.readthedocs.org/en/stable/usage.html#borg-create for details.
    # Defaults to no compression.
    compression: zstd

    # Remote network upload rate limit in kiBytes/second.
    #remote_rate_limit: 100

    # Command to use instead of just "ssh". This can be used to specify ssh options.
    #ssh_command: ssh -i /path/to/private/key

    # Umask to be used for borg create.
    #umask: 0077

    # Maximum seconds to wait for acquiring a repository/cache lock.
    #lock_wait: 5

    # Name of the archive. Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. Default is
    # "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this option, you must
    # also specify a prefix in the retention section to avoid accidental pruning of
    # archives with a different archive name format. And you should also specify a
    # prefix in the consistency section as well.
    #archive_name_format: '{hostname}-documents-{now}'

# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-prune for details.
# At least one of the "keep" options is required for pruning to work.
retention:
    # Keep all archives within this time interval.
    #keep_within: 3H

    # Number of secondly archives to keep.
    #keep_secondly: 60

    # Number of minutely archives to keep.
    #keep_minutely: 60

    # Number of hourly archives to keep.
    #keep_hourly: 24

    # Number of daily archives to keep.
    keep_daily: 7

    # Number of weekly archives to keep.
    #keep_weekly: 4

    # Number of monthly archives to keep.
    keep_monthly: 6

    # Number of yearly archives to keep.
    keep_yearly: 3

    # When pruning, only consider archive names starting with this prefix.
    # Borg placeholders can be used. See the output of "borg help placeholders" for
    # details. Default is "{hostname}-".
    #prefix: sourcehostname

# Consistency checks to run after backups. See
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-check and
# https://borgbackup.readthedocs.org/en/stable/usage.html#borg-extract for details.
#consistency:
    # List of one or more consistency checks to run: "repository", "archives", and/or
    # "extract". Defaults to "repository" and "archives". Set to "disabled" to disable
    # all consistency checks. "repository" checks the consistency of the repository,
    # "archive" checks all of the archives, and "extract" does an extraction dry-run
    # of just the most recent archive.
    #checks:
    #    - repository
    #    - archives

    # Paths to a subset of the repositories in the location section on which to run
    # consistency checks. Handy in case some of your repositories are very large, and
    # so running consistency checks on them would take too long. Defaults to running
    # consistency checks on all repositories configured in the location section.
    #check_repositories:
    #    - user@backupserver:sourcehostname.borg

    # Restrict the number of checked archives to the last n. Applies only to the "archives" check.
    #check_last: 3

    # When performing the "archives" check, only consider archive names starting with
    # this prefix. Borg placeholders can be used. See the output of
    # "borg help placeholders" for details. Default is "{hostname}-".
    #prefix: sourcehostname

# Shell commands or scripts to execute before and after a backup or if an error has occurred.
# IMPORTANT: All provided commands and scripts are executed with user permissions of borgmatic.
# Do not forget to set secure permissions on this file as well as on any script listed (chmod 0700) to
# prevent potential shell injection or privilege escalation.
hooks:
    # List of one or more shell commands or scripts to execute before creating a backup.
    before_backup:
        - echo "`date` - Starting a backup job."
        # these back up data into /backups/borgcache
        - backup-etc
        - backup-database

    # List of one or more shell commands or scripts to execute after creating a backup.
    after_backup:
        - echo "`date` - Backup created."
    # List of one or more shell commands or scripts to execute in case an exception has occurred.
    #on_error:
    #    - echo "`date` - Error while creating a backup."

Add borgmatic into daily cron jobs, so it runs every day, apply chmod 755:

cat /etc/cron.daily/borgmatic

#!/bin/bash
borgmatic --verbosity 1

And generate and set up SSH key to destination:

ssh-keygen -a 100 -t ed25519 -C "borger"

# cat /root/.ssh/config
Host backupserver
    HostName destination_server.sk
    User borger
    IdentityFile ~/.ssh/id_ed25519

Destination

As root install borg on destination server:

$ apt-get install borgbackup

Add user:

$ useradd borger

As borger user create repo, the password is the same as in borgmatic configuration:

mkdir -p ~/backups/borg
cd ~/backups/borg
borg init --encryption=repokey-blake2 .
borg config . additional_free_space 2G
borg config . append_only 1

append_only to not have anything removed, free_space for borg cannot prune while no free space and you will die.

As borger user add public SSH key of source user to destination server:

cat .ssh/authorized_keys
command="borg serve --restrict-to-path ~/backups/borg",restrict  ssh-ed25519 AAAAmykey root@source

chmod 700 ~/.ssh
chod 600 ~/.ssh/id_ed25519

Running it

And now run it on source and you should see something like this (I have backups from August):

/etc/cron.daily/borgmatic
/etc/borgmatic/config.yaml: Running 4 commands for pre-backup hook
Sun dec  6 13:47:49 CET 2020 - Starting a backup job.
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
/backups/borg/: Pruning archives
/backups/borg/: Creating archive
Creating archive at "/backups/borg/::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
/backups/borg/: Running consistency checks
Starting repository check
Starting repository index check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive source-server-2020-07-31T06:35:24.333633 (1/12)
Analyzing archive source-server-2020-08-31T06:29:25.065722 (2/12)
Analyzing archive source-server-2020-09-30T06:31:43.033300 (3/12)
Analyzing archive source-server-2020-10-31T06:31:38.222528 (4/12)
Analyzing archive source-server-2020-11-30T06:30:23.411184 (5/12)
Analyzing archive source-server-2020-12-01T06:29:44.704769 (6/12)
Analyzing archive source-server-2020-12-02T06:30:25.994776 (7/12)
Analyzing archive source-server-2020-12-03T06:29:51.315834 (8/12)
Analyzing archive source-server-2020-12-04T06:29:35.802369 (9/12)
Analyzing archive source-server-2020-12-05T06:29:23.555802 (10/12)
Analyzing archive source-server-2020-12-06T06:29:51.908761 (11/12)
Analyzing archive source-server-2020-12-06T13:49:09.584833 (12/12)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
borger@destination_server.sk:backups/borg: Pruning archives
borger@destination_server.sk:backups/borg: Creating archive
Creating archive at "borger@destination_server.sk:backups/borg::{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}"
borger@destination_server.sk:backups/borg: Running consistency checks
Remote: Starting repository check
Remote: Starting repository index check
Remote: Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive source-server-2020-07-31T06:41:25.875108 (1/12)
Analyzing archive source-server-2020-08-31T06:35:52.625629 (2/12)
Analyzing archive source-server-2020-09-30T06:40:06.208063 (3/12)
Analyzing archive source-server-2020-10-31T06:40:39.288971 (4/12)
Analyzing archive source-server-2020-11-30T06:40:58.945221 (5/12)
Analyzing archive source-server-2020-12-01T06:40:27.373710 (6/12)
Analyzing archive source-server-2020-12-02T06:41:14.762190 (7/12)
Analyzing archive source-server-2020-12-03T06:40:33.421019 (8/12)
Analyzing archive source-server-2020-12-04T06:40:24.263220 (9/12)
Analyzing archive source-server-2020-12-05T06:40:04.474899 (10/12)
Analyzing archive source-server-2020-12-06T06:40:34.384012 (11/12)
Analyzing archive source-server-2020-12-06T13:59:31.589336 (12/12)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
/etc/borgmatic/config.yaml: Running command for post-backup hook
Ne dec  6 14:11:26 CET 2020 - Backup created.

summary:
/etc/borgmatic/config.yaml: Successfully ran configuration file

List backups:

borg list --debug /backups/borg
Enter passphrase for key /backups/borg:
source-server-2020-07-31T06:35:24.333633     Fri, 2020-07-31 06:35:24 [58aa33f782413e5d9ffb941b880d3c3deca4015938c446cc6b1b017dd9633b44]
source-server-2020-08-31T06:29:25.065722     Mon, 2020-08-31 06:29:25 [30f1885dc224e32a6e76af7b4e9fdcc5cb064e8a343bade04ab6438d023df065]
source-server-2020-09-30T06:31:43.033300     Wed, 2020-09-30 06:31:43 [7944462de0c10584b9879e302bff5bfff2fd6ee6fde1778453ed52a42a93ff0c]
source-server-2020-10-31T06:31:38.222528     Sat, 2020-10-31 06:31:38 [67fbdf24ab59be5d3eaa8d123024e83d05e24983fa4f03a8c24fab08c7d0fdf2]
source-server-2020-11-30T06:30:23.411184     Mon, 2020-11-30 06:30:23 [c47c9c1ea704089e65c699e68625f1109d33d729e4440ffded0b5326157a26cc]
source-server-2020-12-01T06:29:44.704769     Tue, 2020-12-01 06:29:45 [e9db7e9afc4457db18def2346d8552230bd840e92069e3be1c6ccc1724c7ed4b]
source-server-2020-12-02T06:30:25.994776     Wed, 2020-12-02 06:30:26 [2f43b1591211d55a0968a7afa2e9e24744f18f9e7c42dd270e7145dea1ea4798]
source-server-2020-12-03T06:29:51.315834     Thu, 2020-12-03 06:29:51 [02a4bd337a0c34d984fdbe1d842d61e5c984ca67401b78c3b336dc698e74c217]
source-server-2020-12-04T06:29:35.802369     Fri, 2020-12-04 06:29:36 [ae8b54727a812ba5b1d56097852e63d5751532eb99a22516188754de42de316c]
source-server-2020-12-05T06:29:23.555802     Sat, 2020-12-05 06:29:24 [d44d3168f81722650e2a80f184bc7ab165a5373b3b45360756d45713739b30b4]
source-server-2020-12-06T06:29:51.908761     Sun, 2020-12-06 06:29:52 [999ba07baddf4e962ab63da0579e61a36a721d9aacb28bfad6bc33a35890050a]
source-server-2020-12-06T13:49:09.584833     Sun, 2020-12-06 13:49:10 [0306c63b9fc375165449709f987699b33775eb19d8c0fed488bce14d88dc7165]

You can get too many open files error when borgmatic is running:

/home/someuser/python: scandir: [Errno 24] Too many open files: '/home/someuser/python'

Add this into /etc/security/limits.conf and relog:

root             -       nofile          1000000

By default, max open files is 1024:

$ ulimit | grep files
open files                      (-n) 1024

And remember, you need both repository key and password to decrypt it.

That’s all.

Add Comment