Remote Borg backup guide

[gbertolottiKS]

One year ago, we wrote the guide on how to use Rclone to synchronize the Nextcloud backups to a remote server (as a fallback copy).
But after that, we realized that the solution had a big problem: in case of a malware attack, all the backup could be encrypted, and so the remote copies, making those useless.
We thought a bit on what to do, but at the end we decided that Borg itself (the technology behind the Nextcloud backups) could be the perfect solution.

So below there is a guide on how install, configure and use Borg (and Borgmatic) to create remote copies of the backups with the Append-Only feature, so that every day a new version is created (but deduplicated and compressed), and the previous ones are not touched: if your backups get corrupted, you can always use a previous version!

Let's get started!

As an off-site backup solution, in addition to Nextcloud's native backups, Borg backup technology (https://www.borgbackup.org/) was chosen, and as a remote hosting service, BorgBase's service (https://www.borgbase.com/).

BorgBase account creation.

Register at https://www.borgbase.com/ and select a plan from those offered. All of them provide the possibility to exceed the GB provided by paying an additional fee.

SSH key pair creation.

In order to perform any operation on BorgBase remotely, you need to create an SSH key pair.

Follow the official guide: https://docs.borgbase.com/setup/borg/cli#step-3-create-and-assign-ssh-key-for-authentication, or the steps below.

Using a user with root privileges, create the key:

cd /root/.ssh/
ssh-keygen -o -a 100 -t ed25519

When specifying the key name, it is preferable not to leave the default one, but to change it, defining a unique name, including the path.

However, this implies that every time an SSH connection is defined, it will be necessary to specify the key path (the system will automatically look for the default one: if it has been changed, it will fail, unless the new path is explicitly specified).

Show it in order to copy it (copy the whole string!):

cat ~/.ssh/<name-of-the-key>.pub

Copy it to the appropriate section of BorgBase; if multiple servers connect to the same repository (e.g., because one uploads backups, the other downloads them to do testing), you need to create a key for each:

Create repo on BorgBase

1. create a new repository, by going to Repositories > New Repo
2. choose a name for the repo, region, and type (i.e. Borg)
3. under the Access tab, choose the SSH key and set it in the Append-Only Access field (this ensures that a new version is created each time, and that previous versions are not overwritten)

Borg and Borgmatic installation and configuration.

Online guides recommend using pip for installation, which allows for the most up-to-date versions; we used the (older) version found on Ubuntu's apt repositories, for convenience.

Follow the official guide (https://docs.borgbase.com/setup/borg/cli) or the steps below.

Install Borg

apt install borgbackup

To see the installed version:

borg --version

Install Borgmatic

apt install borgmatic

To see the installed version:

borgmatic --version

Initialize remote Borg repository

Copy repository path, from Borgbase, via appropriate icon next to name:

Initialize environment variable, with the path you just copied:

export BORG_REPO=ssh://[email protected]/./repo

Initialize environment variable, with the path of the ssh key, e.g.:

export BORG_RSH='ssh -i /root/.ssh/nextcloud_borg'

Initialize the repository:

borg init -e repokey-blake2

These command were tested using the Borg version 1.2.0

Configure Borgmatic

Create directory for Borgmatic configuration file:

mkdir -p ~/.config/borgmatic

Create configuration file:

nano ~/.config/borgmatic/config.yaml

and as content, follow the guide: https://torsion.org/borgmatic/docs/reference/configuration/

Depending on the version of Borgmatic installed, the structure of the configuration file changes.

We used this configuration (remember to specify the path to the ssh key!), tested on the Borgmatic version 1.5.20:

location:
  source_directories:
    - /mnt/backup/borg

  repositories:
    - ssh://[email protected]/./repo

  exclude_caches: true
  exclude_patterns:
    - /mnt/backup/borg/aio-lockfile

storage:
  compression: auto,zstd
  encryption_passphrase: <password of the Borg repo's encryption>.
  ssh_command: ssh -i /root/.ssh/nextcloud_borg

retention:
  keep_within: 1m

consistency:
  checks:
    - disabled

  check_last: 3

hooks:
  before_backup:
    - /root/.config/borgmatic/on_before_backup.sh
  after_backup:
    - /root/.config/borgmatic/on_after_backup.sh
  on_error:
    - /root/.config/borgmatic/on_error_backup.sh {repository} {error} {output}

⚠️ The retention policy is up to your decision. For more information, see here: https://torsion.org/borgmatic/docs/reference/configuration/

Important: Repository disk space is not freed until you run borg compact. On BorgBase it can be automated via GUI:

Creating hook scripts.

Borgmatic allows you to define, via hooks, Shell files to be executed automatically:

• before backup execution
• after execution
• in case of errors

This is done by specifying in the config.yaml file the "hooks" section (see above).

Therefore:

• create 3 shell files under the same folder as the Borgmatic configuration file (/root/.config/borgmatic/):

on_before_backup.sh

#!/bin/bash

SOURCE_DIRECTORY="/mnt/backup/borg"

if [ "$EUID" -ne 0 ]; then
    echo "run as root"
    output 1
fi

if ! [ -d "$SOURCE_DIRECTORY" ]; then
    echo "The root directory does not exist."
    exit 1
fi

if [ -z "$(ls -A "$SOURCE_DIRECTORY/")" ]; then
    echo "The source directory is empty, which is not allowed."
    exit 1
fi

if [ -f "$SOURCE_DIRECTORY/lock.roster" ]; then
    echo "Unable to execute the script as the backup archive is currently modified. Please try again later."
    exit 1
fi

if [ -f "$SOURCE_DIRECTORY/aio-lockfile" ]; then
    echo "Cannot continue because aio-lockfile already exists."
    exit 1
fi

touch "$SOURCE_DIRECTORY/aio-lockfile"

on_after_backup.sh

#!/bin/bash

SOURCE_DIRECTORY="/mnt/backup/borg"

rm "$SOURCE_DIRECTORY/aio-lockfile"

if docker ps --format "{{.Names}}" | grep "^nextcloud-aio-nextcloud$"; then
    docker exec -en nextcloud-aio-nextcloud bash /notify.sh "Remote Borg backup successful!" "Synchronised the backup repository successfully!"
else
    echo "Synchronised the backup repository successfully."
fi

on_error_backup.sh

#!/bin/bash

if docker ps --format "{{.Names}}" | grep "^nextcloud-aio-nextcloud$"; then
    docker exec -en nextcloud-aio-nextcloud bash /notify.sh "Remote Borg backup FAILED!!!" "Repository: $1 failed to synchronise with error: $2 when executing command: $3>.
else
    echo "Repository: $1 failed to synchronise with error: $2 when executing command: $3"
fi

• assign the correct read, write, and execute permissions to each:

chmod 0700 on_before_backup.sh
chmod 0700 on_after_backup.sh
chmod 0700 on_error_backup.sh

• check that in the config.yaml file the paths match

• in the case of the on_error hook, you can pass parameters in curly brackets to the shell file; refer to the documentation https://torsion.org/borgmatic/docs/how-to/monitor-your-backups/?ref=noted#error-hooks

Initialize Borgmatic repository

borgmatic init --encryption repokey-blake2

Manual import/export.

Import (on BorgBase)

Using Borgmatic:

borgmatic --verbosity 2

Export (on server)

Obviously, the server where you want to download the backup must also have Borg and Borgmatic installed.

Show all versions of backups in the remote archive (BorgBase):

borgmatic --list --info

Download a specific version:

borgmatic extract --archive <full archive name>

Set up crontab

To have backups run periodically automatically, you need to invoke Borgmatic via crontab.

• open the cronjob in edit mode: crontab -e

• set the desired schedule; for example, if you want to run it every day at 05, run:

0 5 * * * borgmatic

Make sure that the set time does not overlap with the time when backups are created on Nextcloud.

Extra: retention policy

To better understand the retention policy, we suggest visiting the example in https://borgbackup.readthedocs.io/en/stable/usage/prune.html.
We also made a scheme to visualize our current one used to copy the local AIO backup to the remote repository ("keep-within: 1m")

[gbertolottiKS]

CC @szaimen

[szaimen]

Looks good, thanks for creating it! I added it to the readme as well: 31fa780

[kF13Ld]

Thanks for this guide!

Two questions though: in order to restore a backup with this solution, I would have to restore the borg backup from the remote server which again includes borg backups for Nextcloud AIO, right?

Does a borg backup of an encrypted borg backup play nicely concerning deduplication and incremental backup?

[gbertolottiKS]

Hi.
I'm happy you found this guide useful.

1. Yes: because AIO doesn't ship with an option to support remote backups, it's necessary to do this secondary encapsulation. But in the end, it's pretty simple: once you extract a version from your remote server, you get a ready-to-go AIO backup that you can use to restore a complete AIO environment.

2. Yes: Borg it's smart, and it handles the compression and deduplication of anything, including an archive that has been already compressed.

[kF13Ld]

For second point: I guess I am just unsure about how the encryption works with borg. If a change of one byte in the original data results in just one byte changed in the encrypted backup data, then I agree, that the second borg backup should be able to work as expected.
But the more I think about it, the more it has to be like that. Otherwise even one borg instance would have problems creating light incremental backups.

[jdumas]

@gbertolottiKS I'm also worried about the point raised by @kF13Ld. Since you have been using this method for a while, do you have some data to share regarding the size of (1) the primary Borg archive created by Nextcloud, and (2) the secondary remote archives created by Borgmatic?

[gbertolottiKS]

Hi.
We found that, based on the suggestion given to us by @kostiya (#4391 (reply in thread)), there was an error on the Borgmatic configuration file, so we weren't actually pruning any archives. We updated the guide accordingly.

To answer your question, we made some research:

• our Nextcloud instance is around 339 GB
• the local (primary) AIO borg backup is around 144 GB
• the remote (secondary) archive on BorgBase is around 174 GB, considering that we have now a retention policy of 1 month (so we maintain the last 30 copies)

[jdumas]

Thanks. So you're using the default retention policy on AIO (6 month), and keep 1 month of the secondary archive? I'll try to do something similar with restic (my backup solution for other data), and see how it goes.

[gbertolottiKS]

Hi.
No, we do not change the default retention on AIO, so we are using:
keep-within: 7d
keep-weekly: 4
keep-monthly: 6

Every day, we copy all the AIO (primary) archive to Borgabase, with the retention policy of 1 month. So, in the end, on the secondary remote repository, we have circa 30 copies of the primary one. For reference, see the scheme in the "Extra: retention policy" section of our guide

[AiroPi]

Hello, I am looking for a solution to use borg backup remote solution over ssh.
Currently I am using Borg over a network file system, but it is very slow (sometimes I can't use my nextcloud at all for more than 24h just because I added like 30Go of files...

According to borg, by using SSH:

If Borg is installed on the remote host, big performance gains can be achieved compared to using a network filesystem (sshfs, nfs, …).

So my question is: does your solution require to have a local borg backup, or is it a 100% remote solution ? The only solutions mentioned in the AIO readme are solutions to copy the backup once it has been done on a local (or network) filesystem :s this would divide my main storage by 2...

[kiwijunglist]

Anyone interested in writing a similar guide for running your own borgserver in docker and then either borg client or borgmatic client or vorta client to backup to the borgserver. I haven't been able to get it working properly.

I've been using this docker - https://hub.docker.com/r/nold360/borgserver

Having problems specifying the port (non-standard 2222) and the ssh keys from the client.

[kostiya]

Hello, I used this guide. borgmatic doesn't seem to follow my retention policy - I have at least 15 days of archives even thow I configured 7.
I have :

• borg 1.2.4
• borgmatic 1.7.7

Here is the appropriate part in config.ymal:

retention:
  keep_daily: 7
  keep_weekly: 4
  keep_monthly: 6
  prefix: "old"

[kostiya]

Found the problem,
Removed the prefix statement.

[cbrunet]

• With newer versions of borgmatic, you should put all configs in global scope, instead of using retention, storage, hooks, etc. subsections.
• The source directory in on_before_script.sh should be /mnt/backup/borg(not borgmatic)
• The first if in on_before_script.sh should only have one [.
• There is a truncated line in the on_after_script.sh. It should read as docker exec -en nextcloud-aio-nextcloud bash /notify.sh "Remote Borg backup FAILED!!!" "Repository: $1 failed to synchronise with error: $2 when executing command: $3"

[gbertolottiKS]

Thanks, updated

[gbertolottiKS]

Updated the guide based on feedbacks