← All notes

Keeping Nextcloud and a file server in sync automatically

Published · last modified

NextCloudrcloneBackupLinuxSelf-Hosting

Also in: Deutsch, español

A hosted Nextcloud is convenient as long as the internet works. If the connection fails, however, the data is out of reach. A file server on the home network that holds a complete copy of the Nextcloud closes this gap and is also the ideal starting point for local backups. nc-bisync.sh keeps this copy up to date automatically with rclone bisync. It runs unattended via cron and only raises an alarm when something looks wrong.

Download: nc-bisync.sh (comments and messages in the script are in German)

The setup

The Nextcloud is the central store, and the file server mirrors it into the home network:

Laptops, phones  ⇄  Nextcloud client  ⇄  Nextcloud
                                             ⇅   rclone bisync (automatic)
                                         File server  →  SMB shares on the home network
                                             │
                                             └─ rsnapshot (daily, versioned) → external drive

Laptops and phones sync with the Nextcloud client as usual. The file server syncs with the Nextcloud regularly and is reachable on the home network through its SMB shares. If the internet connection fails, a complete copy of all data is still there. Because the sync runs in both directions, files saved directly on the file server also make it into the Nextcloud. Backups are made from the file server, independently of the provider.

rclone bisync is well suited to this sync: rclone talks to WebDAV directly and needs no davfs mount. It also comes with safeguards that would otherwise require a wrapper script of your own.

What the script does

One directory pair per share. Each share on the file server is synced with the corresponding folder in Nextcloud, independently of the others. Pairs can be run individually or in groups, even at different frequencies. If needed, a single subfolder, such as a large project, becomes a pair of its own and is automatically excluded from the parent pair.

Archive only on demand. A subfolder archiv inside a pair is never synced in a normal run, only with --archiv. Internally it is a separate pair with its own state. That is more robust than a folder that a filter sometimes includes and sometimes excludes.

Protection against mass damage. Three built-in mechanisms guard against larger damage:

  • --check-access: Each root contains a file named RCLONE_TEST. If it is missing on one side, because a path is wrong or a drive is not mounted, the run aborts. This also prevents an empty bind mount from being pushed to the Nextcloud as “everything deleted”.
  • --max-delete 10: If more than 10 % of the files would be deleted, nothing happens at all.
  • --backup-dir1: Files overwritten or deleted on the file server end up in a daily folder and are cleaned up after 30 days.

Conflicts. If a file was changed on both sides, the newer version wins (--conflict-resolve newer). The older one is kept under a new name.

Robustness. With --resilient and --recover, bisync usually survives interrupted runs without a manual --resync. A lock file prevents overlapping runs.

Notifications. On errors you get a message by email or via ntfy. The script distinguishes whether a resync is needed, the delete limit was hit or the access check failed. Every run is logged with its duration in summary.log.

Filters. Temporary files from Office, macOS and Windows are never synced.

Requirements

On the file server:

  • Linux with bash version 4 or later and the standard tools flock (from util-linux), find and cmp. On Debian, Ubuntu and Raspberry Pi OS all of these are preinstalled.
  • rclone 1.66 or newer, see step 1.
  • A user with write access to the synced directories, under which the script runs.
  • Optionally curl for push messages via ntfy and a working mail setup for email notifications, see step 7.

In Nextcloud:

  • An account with WebDAV access and an app password.
  • The target folders; --init creates missing ones itself.

Setup

The script does not run as root but as the user who owns the files on the file server. Otherwise files newly fetched from the Nextcloud would belong to root and could no longer be written by the other users of the share. sudo is only needed for the installation.

1. Install rclone

The script needs rclone 1.66 or newer. Many distributions ship older packages, so it is better to use the official install script:

curl https://rclone.org/install.sh | sudo bash
rclone version

2. Set up the Nextcloud remote

In Nextcloud, create an app password under Settings → Security. Then set up the remote as the user the script will run as later (without sudo):

rclone config
# n) New remote
# name> nc
# Storage> webdav
# url> https://cloud.example.com/remote.php/dav/files/USERNAME/
# vendor> nextcloud
# user> USERNAME
# pass> (app password)

To test:

rclone lsd nc:

3. Install and adapt the script

sudo install -m 755 nc-bisync.sh /usr/local/bin/nc-bisync.sh
sudo install -d -o USER /var/lib/nc-bisync /var/log/nc-bisync
sudo install -d -o USER -m 700 /mnt/USB-Crucial/.bisync-backup

Mode 755 makes the script executable for all users, but only root can change it. The second line creates the state and log directories and hands them over to the user the script runs as. The third creates the backup directory (BACKUP_ROOT); mode 700, because overwritten files from all shares end up there and only the script’s user should see them.

In the configuration block at the top, enter the pairs, one per line with name, local path and path in Nextcloud:

PAIRS=(
    "administration|/mnt/shares/administration|nc:Administration"
    "bibliothek|/mnt/shares/bibliothek|nc:Bibliothek"
    "dokumente|/mnt/shares/dokumente|nc:Dokumente"
    "fotos|/mnt/shares/fotos|nc:Fotos"
    "projekte|/mnt/shares/projekte|nc:Projekte"
    "unternehmen|/mnt/shares/unternehmen|nc:Unternehmen"
)

Further settings:

Variable Meaning Example
ARCHIVE_SUBDIR Subfolder that only runs with --archiv archiv
BACKUP_ROOT Storage for overwritten files /mnt/USB-Crucial/.bisync-backup
MAX_DELETE Delete limit in percent 10
NOTIFY_MAIL Recipient for error messages (optional) admin@example.com
NTFY_URL ntfy topic for push messages (optional) https://ntfy.sh/my-topic

Make sure BACKUP_ROOT is outside all pairs. nc-bisync.sh --list shows which pairs are configured and which have an archive subfolder.

4. Group permissions on the file server

The script creates new files with umask 002 so that they stay writable for the group. To make sure they also get the right group rather than the user’s primary group, the synced directories should have the setgid bit. The setgid bit on a directory makes files and subfolders newly created in it automatically inherit its group instead of getting their creator’s primary group.

For the pairs from the example above, set it on all existing directories like this:

for d in administration bibliothek dokumente fotos projekte unternehmen; do
    sudo find "/mnt/shares/$d" -type d -exec chmod g+s {} +
done

You can tell it is set by an s in the group permissions:

$ ls -ld /mnt/shares/dokumente
drwxrws--- 10 stefan familie 4,0K 21. Jul 13:19 /mnt/shares/dokumente

5. Initial sync

Test with --dry-run first, then run the initial sync:

nc-bisync.sh --init --dry-run
nc-bisync.sh --init

--init creates the RCLONE_TEST files in every pair and every existing archive subfolder and syncs everything with --resync. Where they differ, the newer file wins. With a lot of data it makes sense to set up the pairs one at a time, for example nc-bisync.sh --init fotos.

In a dry run the script changes nothing: it does not create the RCLONE_TEST files either and therefore skips the access check.

6. Schedule

Add the cron jobs with crontab -e to the crontab of the user the script runs as. Because each pair has its own lock, frequently changed shares can run more often than rarely changed ones:

# frequently changed: every 2 hours
0 */2 * * *  /usr/local/bin/nc-bisync.sh dokumente fotos projekte unternehmen administration
# rarely changed: at night
30 2 * * *   /usr/local/bin/nc-bisync.sh bibliothek

With many files, the bottleneck is not the transfer but listing everything over WebDAV, and that happens in full on every run. So start with two hours and check the run times after a few days:

tail /var/log/nc-bisync/summary.log

As a rule of thumb, the interval should be about three times as long as a run. Do not sync too often, because some providers throttle large numbers of WebDAV requests.

7. Set up notifications (optional)

If NOTIFY_MAIL and NTFY_URL are left empty, the script only writes errors to summary.log. For unattended operation you should set up at least one of the two; they can also be combined.

Email (NOTIFY_MAIL). The script sends the message with the mail command. This requires a suitable package such as mailutils or bsd-mailx, and the server must be able to deliver mail to the outside world. The easiest way is msmtp-mta, which sends mail through your mail provider’s SMTP account. To check that it works, run this as the script’s user:

echo "Test" | mail -s "Test from nc-bisync" admin@example.com

If the mail command is missing, the script skips sending email without an error message.

Push message (NTFY_URL). This needs curl on the server. Install the ntfy app on your phone and subscribe to a topic there. The public server ntfy.sh needs no account, but anyone who knows the topic name can read along. So choose a long, random name, for example https://ntfy.sh/nc-bisync-7f3k9x2q. To test:

curl -d "Test from nc-bisync" https://ntfy.sh/nc-bisync-7f3k9x2q

A self-hosted ntfy server works the same way. Topics that require a login or an access token are not supported by the script in its current form.

Day-to-day use

Command When
nc-bisync.sh sync all pairs, without archive
nc-bisync.sh fotos dokumente sync only the pairs named
nc-bisync.sh --archiv projekte after archiving files in projekte
nc-bisync.sh --dry-run fotos see beforehand what would happen
nc-bisync.sh --archiv --force projekte when the delete limit was hit while archiving
nc-bisync.sh --list show the configured pairs

Always pass rclone options that take a value as --option=value; otherwise the script takes the value for the name of a pair.

Archiving files

Move the files within the share into the subfolder archiv, then run the sync for that share:

nc-bisync.sh --archiv projekte

The script syncs the archive first and only then the rest of the share. That way the files are already in the Nextcloud archive before they disappear from the rest of the share there. If the archive run fails, the rest of the share is not touched at all.

If you move a lot of files at once, the delete limit may kick in. In that case, take a quick look at the log and run again with --force.

To add an archive later, create the folder archiv locally and run nc-bisync.sh --init NAME once.

Individual subfolders as a pair of their own

If a share becomes too large for frequent runs, a subfolder can get a pair of its own:

"tango|/mnt/shares/projekte/Tango|nc:Projekte/Tango"

The script then automatically excludes Tango from the pair projekte. Because this changes its filter, nc-bisync.sh --init projekte tango is needed once. The subfolder must sit in the corresponding place in Nextcloud, otherwise the script aborts with a message.

When a notification arrives

  • Resync needed: Check the cause in the log mentioned. After that, nc-bisync.sh --init NAME usually fixes the state, ideally with --dry-run first.
  • Too many deletions: Check whether the deletions are intended. If so, run with --force.
  • Access check failed: An RCLONE_TEST file is missing. Check that the path is correct, the Nextcloud is reachable and the drive is mounted.

Troubleshooting

Connection drops during large transfers

Symptoms. Especially during the initial sync of large amounts of data, errors like this appear in the log for individual files:

ERROR : 2014/…/IMG_0929.PNG: Failed to copy: unchunked simple update failed: Put "https://…":
http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY]
after Request.Body was written; define Request.GetBody to avoid this error

The run does not abort but carries on transferring the remaining files. At the end of an initial sync, however, bisync still reports Bisync aborted. Must run --resync to recover., because with --resync a single failed file is enough. The summary then shows, for example, 100 % transferred but several errors.

What goes wrong with HTTP/2. HTTP/2 sends many requests at the same time over a single connection. At some point the server or a proxy in front of it closes this connection in an orderly way with a GOAWAY, for example after a fixed number of requests or on a restart. Requests that have not started yet can simply be sent over a new connection. For uploads whose data was already on its way, that is not possible: the file content has already been sent and cannot be sent a second time at this point. The file fails.

What is different with HTTP/1.1. Here each request runs over its own connection. If the server closes one of them, only that one transfer is affected, and rclone retries it over a new connection with its normal retries. Because rclone transfers several files in parallel anyway, HTTP/1.1 is hardly slower in practice.

Workaround for a single run. Often it is enough to simply repeat the run: the drops only happen sporadically, and on the next attempt the failed files are usually transferred after all. Files already transferred are only compared, not uploaded again. Otherwise you can fall back to HTTP/1.1. The script passes every option it does not know itself on to rclone. This is how you repeat a failed initial sync with HTTP/1.1:

nc-bisync.sh --init fotos --disable-http2

Switching it off permanently. Every rclone option can also be set as an environment variable RCLONE_<OPTION>. Add this line to the script’s configuration block:

export RCLONE_DISABLE_HTTP2=true

All rclone calls of the script then use HTTP/1.1, including those from the cron job.

Limits

rclone bisync is not a backup. The sync also carries over accidental changes; only mass deletions are caught. If ransomware encrypts files on a laptop, for example, they reach the Nextcloud via the Nextcloud client and the file server via bisync, because files are overwritten, not deleted. The overwritten versions do remain in the backup directory for 30 days, but the real protection comes from the versioned rsnapshot backups and Nextcloud’s file versions. The retention period should be longer than the time it takes to notice damage.