If you ever tried to setup ZFS replication between 2 TrueNAS instance, you might have been very surprised by how easy it is. But what if you don’t want or don’t need a dedicated TrueNAS instance for your backup ? Maybe you want it to replicate to a Linux machine that does other type of backup for example ? In this post, I’ll show you how I’ve done it in a secure and reliable way, and hopefully make it way easier for you to do the same.

The basic.

For 4Y now, i have been running FreeNAS (now TrueNAS) as my main NAS . It’s reliable, and I never had any issue with it. I have a Zpool of 2 Raidz2 with six 10To disk each. As you may guess, i don’t backup everything, but when I started saving important document to it, I knew an offsite backup would be mandatory !

To do so, the first point was to make a dedicated dataset for thing to backup. Actually, I made multiple, every user’s home have a “Documents” dataset who is backed up off-site every day.
I also needed to create snapshots policy that would allow this replication. To make it simpler I choses to create dedicated snapshot for the replication part in addition to the standard hourly, daily and monthly snapshots I created for most of the dataset.
I called those Rsync-%Y-%m-%d_%H-%M-2W, and choose to keep them for 2 week just in case i’m having issue with the replication.

We now have the basic we need, let’s configure the remote server.

Preparing the remote linux server.

We will work in a LAN area for now, how to send it from site A to site B is out of scope for this post, but please don’t expose your server SSH port to the internet. I will show in another post about how to safely link multiple site for 2€ per month using a VPS .

I’m using debian here, it’s not the most up-to-date but it’s very reliable and my main server OS. Most of what we will talk about can be transposed to other distro without much work.

ZFS Pool.

First point of business, you will need ZFS on your server. OpenZFS for linux is a pretty young but already more than stable port of ZFS on linux. If it wasn’t very safe to use it 4Y ago, it is now the base of TrueNAS scale and you can trust it.

Install the following packages on your server:

sudo apt install \
    zfs-dkms \
    zfsutils-linux \
    lz4

Then create the zpool you want to use for backup (here also called backup). I’m using an old HP microserver with 4 3.5" HDD, so I choose to make a simple RaidZ1 as I needed capacity over failure tolerance since this is already a backup.

sudo zpool create backup raidz sda sdb sdc sdd

You also want to disable atime to minimize disk IO, and enable maximum compression to maximize space utilization.

sudo zfs set atime=off backup
sudo zfs set compression=gzip-9 backup

The server I used a couple year ago was under debian buster and didn’t include automatic scrubbing. I made the following service and timer to do so. Newer version available for debian bullseye seem to have those, I would recommends checking if they exist, are enabled and working since it depend of you distro.

...
zfsutils-linux: /lib/systemd/system/zfs-mount.service
zfsutils-linux: /lib/systemd/system/zfs-scrub-monthly@.timer
zfsutils-linux: /lib/systemd/system/zfs-scrub-weekly@.timer
zfsutils-linux: /lib/systemd/system/zfs-scrub@.service
...

Here are my 2 unit for this.

zpool_scrub.timer

[Unit]
Description=Monthly zpool scrub

[Timer]
OnCalendar=monthly
AccuracySec=2h
Persistent=true

[Install]
WantedBy=timers.target

zpool_scrub.service

[Unit]
Description=zpool scrub

[Service]
KillSignal=SIGINT
Type=oneshot
Nice=19
ExecStart=/usr/sbin/zpool scrub backup

Our pool is now ready to receive the replication, but our server really isn’t yet.

Creating a dedicated user.

I’ll write it once ok…Do NOT allow SSH as root.
Sadly TrueNAS replication script aren’t made to use sudo, and ZFS require sudo to run. Even the zfs bin is in /usr/sbin, which is not in the path of normal user.
We are going to work around that.

First, create a user, mine is called zfs_sync, and give it the following permission.

ALL=(ALL:ALL) NOPASSWD: /usr/sbin/zpool
ALL=(ALL:ALL) NOPASSWD: /usr/sbin/zfs

You should also give him the public ssh key that TrueNAS will use to connect. But you’ll create it a bit later. Then you need to create a script at /usr/local/bin/zfs containing the following:

#!/bin/bash
sudo /usr/sbin/zfs $@

And give it a chmod 0755.

Good, now you have a dedicated user who is allowed to use the zfs binary, a wrapper to trick TrueNAS into using it, and a service to scrub the pool every month. You are now ready to start your first replication.

Configuring the replication on TrueNAS.

On your TrueNAS webUI, in system -> SSH Keypairs, Generate an SSH key for your server if you haven’t done already. Then in system -> SSH Connections, Add your server, specify the username we created and select the SSH key.

In Tasks -> Replication Tasks create a new replication. I enable advanced replication because i’m use to it.

In General, name your replication task, choose the SSH connection you just created and enable stream compression if you want to. LZ4 is very lite on the cpu so i default to it usually.

In Source, select what you want to replicate, check Recursive if you need to, but make sure everything is already snapshoted. Check Include Dataset Properties (Default) and in Periodic Snapshot Tasks, select all the snapshots you made for this replication. You can check Save Pending Snapshots if you want to make sure the incremental path never disappear because the replication was broken for a long time.

In Destination if your SSH connection work, you should see it remotely load your distant pool when you unfold the folder icon. if it doesn’t show up, make sure the the ssh connection work my hand, and test that you can use the ZFS command without sudo as the user TrueNAS is using.
Select your pool, if you want to order it another way you will have to create the dataset manually on your server. \ Select a retention policy that fits your need, I made it 1Y.

Save it, run now and check the log. It will automatically run when your snapshots finish, just make sure to have mailing enabled in order to be warned if it fail at some point.

Warp-up

Your done ! Add some monitoring on your remote server, ship-it to your parent house or what-not and you now have a trustworthy external backup who can resist cryptolocker and NAS destruction.

Well … as trustworthy as your test, so don’t forget to try that restoration from time to time.