So I was happily using sanoid, when someone made me aware of pyznap (thanks !). Ever since that comment, it was on my to-do list to check out pyznap, but if a system works, why change it ? Definitely a important cog such as automated snapshots.

The new version of sanoid, breaks comparability with older versions (at least at config level) and is not documented well at the moment; One has to look into the pull requests to actually understand what is required to get it running. I know open-source project are sometimes in large changes, and its all run on love & joy, but its a sorry state for a project of ~39 contributors. I also find it useful if a project or tool is simple in its setup and understanding for people not looking at this daily.  I might sound critically, but I still think sanoid is a wonderful tool, but personally I just need to get it up & running in 10 minutes and then move on. The feature list of sanoid and its companion syncoid seem ever growing and with it the complexity to find out what is going wrong, that it was time to take pyznap for a run.

And pyznap is actually a shining gem, I got it up & running in 10 minutes. Nothing fancy, just works out of the box. So here’s how I set it up;

First I needed to install Python 3.5+ version on Centos 7, I won’t go into detail, cause its basically part of every fresh install these days.

yum install yum-utils groupinstall development
yum install https://centos7.iuscommunity.org/ius-release.rpm
yum install python36u python36u-pip

After that, install pyznap using pip; we need to specify the version otherwise it will take 2.7 (still default on Centos 7)

pip3.6 install pyznap

This installs all dependency’s, optional you can install pv and mbuffer to visualize transfer speed on sending snapshots to backup locations.

yum install pv mbuffer

Now onto the setup/configuration; This tool wil generate a default config and directory if you set it up :

[[email protected] ~]# pyznap setup -p /etc/pyznap
Feb 21 16:06:30 INFO: Starting pyznap...
Feb 21 16:06:30 INFO: Initial setup...
Feb 21 16:06:30 INFO: Creating directory /etc/pyznap...
Feb 21 16:06:30 INFO: Creating sample config /etc/pyznap/pyznap.conf...
Feb 21 16:06:30 INFO: Finished successfully...

After the setup; its time for configuration, all items are clearly documented. One remark : don’t put # (hashtag) behind to comment; this will generate errors as only lines started with # (hashtag) are ignored;

# default settings parent
[data]
  # every $cron runtime (~15 minutes)
  frequent = 2
  hourly = 6
  daily = 3
  weekly = 1
  monthly = 0
  yearly = 0
  # take snapshots ?
  snap = no
  # clean snapshots ?
  clean = no
  
[data/brick1]
  snap = yes
  clean = yes
  dest = ssh:22:[email protected]:backup/brick1
  dest_key = /root/.ssh/id_rsa_backup
  
[data/brick2]
  daily = 24
  snap = yes
  clean = yes

To give some information I have one pool split up in multiple sub-file systems. (data is the parent and data/brick* are the actually location for data) This means that I can setup defaults for the data but don’t really want snapshots of that, as no data resides there; Only in the bricks, so I overwrite the defaults (snap/clean);

A special case is dest = that’s a build-in backup system; just make a password-less ssh login for server and you can leverage this feature; One thing to remark is that the backup server needs to be a ZFS filesystem and not the actually physical location. (a bit of trickery cause the location is /backup/brick1 but one needs to remove the first / for ZFS) if you use this feature, perhaps its worth downgrading a package cryptography cause the latest version generates a insane amount of warnings about upcoming deprecation of some function calls. See the issue.

pip3.6 install cryptography==2.4.2

A way to clean up the backup location is also provided by pyznap, one can setup a remote cleanup job for the brick by adding :

# cleanup on backup
[ssh:22:[email protected]:backup/brick1]
        frequent = 2
        hourly = 6
        daily = 3
        weekly = 1
        key = /root/.ssh/id_rsa
        clean = yes

The only thing to do now is to either make a cron, which is the easiest :

*/15 * * * *   root    /usr/bin/pyznap snap | logger
0 * * * *   root    /usr/bin/pyznap send | logger

Take snapshots every 15 minutes (frequent) and sync to the backup location (dest=) once per hour; Other snapshots are taken based on the need; (note | logger sends output to /var/log/messages, by being processed by rsyslog, you could also log to a static file, such as explained by the docs)

Alternative on systemd systems like Centos 7, you can leverage systemd timers. Instead of cron, but who wants to go into that mess ?

And that’s it, automated snapshots <3. Thanks yboetz for an small but useful tool 🙂

I installed PHP 7.1 through webtatic, except with Apache instead of Nginx, so not the -fpm.  Installing PHP stats package is easy, if you know how, so follow along.

First we need to provide our system with a compiler, and the pear package which contains pecl (yum whatprovides pecl)

yum install gcc php71w-pear

After that you can run pecl to install it :

pecl install stats

This will however fail, and tell you you need a PHP 5.* version … however if you specify the latest version it wil just install fine :

pecl install stats-2.0.3

And bam, installed. Now we need to activate it in the PHP config :  (new file, or else /etc/php.ini)

nano /etc/php.d/stats.ini

add :

extension=stats.so

After that, just restart apache and you are good to go :

systemctl restart httpd

I installed a new kernel (kernel-ml, from mainline 4.18) to make sure on next boot it got selected this is what I had to do : (Since its headless and I won’t see grub when it boots)

Install kernel mainline :

yum install kernel-ml

Find the current installed kernels :

cat /boot/grub2/grub.cfg | grep ^menuentry | cut -c1-50 | nl -v 0

for me :

0  menuentry 'CentOS Linux (4.18.16-1.el7.elrepo.x86_
1  menuentry 'CentOS Linux (3.10.0-693.5.2.el7.x86_64
2  menuentry 'CentOS Linux (0-rescue-daa04fd470a24a8f

So I want the first kernel (starting from 0) as default; This can be added in /etc/default/grub :

GRUB_DEFAULT=0

Then remake grub config :

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot and the new kernel is :

[[email protected] ~]# uname -a
Linux svennd.local 4.18.16-1.el7.elrepo.x86_64 #1 SMP Sat Oct 20 12:52:50 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

Auch, I got the same error as ma.ttias.be pretty much two years ago. I have no idea why it failed, however the solution seemed to work similar to what he suggested.

[Mon Dec 03 15:07:55.871866 2018] [auth_digest:notice] [pid 3352] AH01757: generating secret for digest authentication ...
[Mon Dec 03 15:07:55.871892 2018] [auth_digest:error] [pid 3352] (2)No such file or directory: AH01762: Failed to create shared memory segment on file /run/httpd/authdigest_shm.3352
[Mon Dec 03 15:07:55.871897 2018] [auth_digest:error] [pid 3352] (2)No such file or directory: AH01760: failed to initialize shm - all nonce-count checking, one-time nonces, and MD5-sess algorithm disabled
[Mon Dec 03 15:07:55.871901 2018] [:emerg] [pid 3352] AH00020: Configuration Failed, exiting

The directory /run/httpd is gone. So recreate it and give correct permissions; This in my case (on Centos 7X was apache instead of httpd)

mkdir /run/httpd
chown root.apache /run/httpd
chmod 0710 /run/httpd

Another solution, sadly the mystery is not solved.

Ordinarily I use “static” references in /etc/fstab to mount NFS shares to a server. Doing this on a Rocks Cluster however is a bit “hacky” and adapting every node can be automated by using rocks run host however there is a alternative way. (not sure if its better 😉 ) but Rocks uses Autofs to load NFS mounts when they are required. So I felt brave and wanted to learn something new. This is the documentation on that journey.  

Read More