So I was happily using sanoid, when someone made me aware of pyznap (thanks !). Ever since that comment, it was on my to-do list to check out pyznap, but if a system works, why change it ? Definitely a important cog such as automated snapshots.
The new version of sanoid, breaks comparability with older versions (at least at config level) and is not documented well at the moment; One has to look into the pull requests to actually understand what is required to get it running. I know open-source project are sometimes in large changes, and its all run on love & joy, but its a sorry state for a project of ~39 contributors. I also find it useful if a project or tool is simple in its setup and understanding for people not looking at this daily. I might sound critically, but I still think sanoid is a wonderful tool, but personally I just need to get it up & running in 10 minutes and then move on. The feature list of sanoid and its companion syncoid seem ever growing and with it the complexity to find out what is going wrong, that it was time to take pyznap for a run.
And pyznap is actually a shining gem, I got it up & running in 10 minutes. Nothing fancy, just works out of the box. So here’s how I set it up;
First I needed to install Python 3.5+ version on Centos 7, I won’t go into detail, cause its basically part of every fresh install these days.
yum install yum-utils groupinstall development
yum install https://centos7.iuscommunity.org/ius-release.rpm
yum install python36u python36u-pip
After that, install pyznap using pip; we need to specify the version otherwise it will take 2.7 (still default on Centos 7)
pip3.6 install pyznap
This installs all dependency’s, optional you can install pv and mbuffer to visualize transfer speed on sending snapshots to backup locations.
yum install pv mbuffer
Now onto the setup/configuration; This tool wil generate a default config and directory if you set it up :
[[email protected] ~]# pyznap setup -p /etc/pyznap
Feb 21 16:06:30 INFO: Starting pyznap...
Feb 21 16:06:30 INFO: Initial setup...
Feb 21 16:06:30 INFO: Creating directory /etc/pyznap...
Feb 21 16:06:30 INFO: Creating sample config /etc/pyznap/pyznap.conf...
Feb 21 16:06:30 INFO: Finished successfully...
After the setup; its time for configuration, all items are clearly documented. One remark : don’t put # (hashtag) behind to comment; this will generate errors as only lines started with # (hashtag) are ignored;
# default settings parent
[data]
# every $cron runtime (~15 minutes)
frequent = 2
hourly = 6
daily = 3
weekly = 1
monthly = 0
yearly = 0
# take snapshots ?
snap = no
# clean snapshots ?
clean = no
[data/brick1]
snap = yes
clean = yes
dest = ssh:22:[email protected]:backup/brick1
dest_key = /root/.ssh/id_rsa_backup
[data/brick2]
daily = 24
snap = yes
clean = yes
To give some information I have one pool split up in multiple sub-file systems. (data is the parent and data/brick* are the actually location for data) This means that I can setup defaults for the data but don’t really want snapshots of that, as no data resides there; Only in the bricks, so I overwrite the defaults (snap/clean);
A special case is dest = that’s a build-in backup system; just make a password-less ssh login for server and you can leverage this feature; One thing to remark is that the backup server needs to be a ZFS filesystem and not the actually physical location. (a bit of trickery cause the location is /backup/brick1 but one needs to remove the first / for ZFS) if you use this feature, perhaps its worth downgrading a package cryptography cause the latest version generates a insane amount of warnings about upcoming deprecation of some function calls. See the issue.
pip3.6 install cryptography==2.4.2
A way to clean up the backup location is also provided by pyznap, one can setup a remote cleanup job for the brick by adding :
# cleanup on backup
[ssh:22:[email protected]:backup/brick1]
frequent = 2
hourly = 6
daily = 3
weekly = 1
key = /root/.ssh/id_rsa
clean = yes
The only thing to do now is to either make a cron, which is the easiest :
*/15 * * * * root /usr/bin/pyznap snap | logger
0 * * * * root /usr/bin/pyznap send | logger
Take snapshots every 15 minutes (frequent) and sync to the backup location (dest=) once per hour; Other snapshots are taken based on the need; (note | logger sends output to /var/log/messages, by being processed by rsyslog, you could also log to a static file, such as explained by the docs)
Alternative on systemd systems like Centos 7, you can leverage systemd timers. Instead of cron, but who wants to go into that mess ?
And that’s it, automated snapshots <3. Thanks yboetz for an small but useful tool 🙂
