ZFS has this less documented feature, called share[nfs|smb]; I tried it once, it “did not work on first attempt”™ So I ignored it; However now we faced an issue where we normally exported ZFS volumes using /etc/exports (NFS); and mounted using /etc/fstab; but got an empty directory where there was a “sub” zpool volume; This seems counter intuitive as on the NFS exporting system you don’t see any difference in child pools directory structure;

For example : (on NFS server)

[[email protected] /]# zfs list
tinky              31.4T  49.0T      256K  /tinky
tinky/archive       219K  49.0T      219K  /tinky/archive
tinky/upload        219K  49.0T      219K  /tinky/upload

In order to export the full /tinky directory; we should do the following in /etc/exports :


This we can mount; and should give :


Which it does, yaaay ! Wrong ! All the data stored under /tinky/archive and /tinky/upload is invisible on the mount ! In fact these directories are empty.

Since zpool children are really easy (unlike human children) it would be really annoying to have to create a /etc/export and /etc/fstab entry for each of those childeren; There is a better way; using sharenfs option in ZFS and autofs (which I played around with already before, here).

In order to start the share with ZFS simply do :

zfs sharenfs=on tinky

You can check the result :

[[email protected] /]# zfs get all | grep nfs
tinky                 sharenfs              on                     local
tinky/archive         sharenfs              on                     inherited from tinky
tinky/upload          sharenfs              on                     inherited from tinky


[[email protected] ~]# showmount -e
Export list for tinky:
/tinky             *
/tinky/archive     *
/tinky/upload      *

If you want to restrict the shares or adapt the options; you can use :

zfs set sharenfs='[email protected]/16,' tinky

in this case I restrict to two IP ranges; but go crazy. On issuing this command, they are automatically exported; which you can verify using exportfs

[[email protected] /]# exportfs

note : reduced output.

In case you don’t see it, you can try to share manually :

zfs share -a

and surprising, if you wish to un-share :

zfs unshare -a

On the remote machine, you can verify if you have access to the NFS mounts using showmount -e , using the IP of the NFS server;

# showmount -e
Export list for

The auto-mount itself now seems smarter then before; although I’m unsure exactly how, simply adding the root of the share is enough to see all the subvolumes in ZFS :

tinky       -rw,hard,intr,rsize=32768,wsize=32768,nfsvers=3\

Happy ZFS sharing 😉

The nproc are the number of proces units a user can start; This is managed in the file : /etc/security/limits.confbut can be overwritten in /etc/security/limits.d/ normally in /etc/security/limits.d/20-nproc.conf which has priority over the limits.conf file. To check how many nproc’s you are using, you need to include the amount of threads, this can be seen using :

[[email protected] ~]# ps -lfu root | wc -l

The default limit for Centos 7 is 4096 for a user and unlimited for root;

# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     4096
root       soft    nproc     unlimited

On Centos 5 this was based on the amount of threads-max the kernel could handle; (50% of that)

[[email protected] ~]# sysctl -a | grep threads-max
kernel.threads-max = 1029577

In Centos 6 the limit was 1024; and on Centos 7 the limit got increased to 4096; While its already higher, we could argue to increase it even further; As threads are relatively cheap.

I increased it to a randomly selected 10000 hard and 8192 soft; Except for the root, although a limit might be useful, I’m afraid if a fork bomb happens, you need the unlimited power to stop it … (perhaps soft ?)

cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     8192
*          hard    nproc     10000
root       soft    nproc     unlimited
root       hard    nproc     unlimited

A similar thing can be done for open files per user; since there is no specific number (yet?) I used 30-nofile.conf You could base this on the maximum open files the system can handle cat /proc/sys/fs/file-max but that seems a bit excessive !

cat /etc/security/limits.d/30-nofile.conf
# max cat /proc/sys/fs/file-max
# 13163513
*       soft    nofile  100000
*       hard    nofile  100001

How to activate it ? Just open a new shell, and the limits are active; check :

[[email protected] ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514788
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 20001
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Useful links :

This is a quick guide on how I got a simple samba (4.10.x) share to work on Centos 8.1.1911 (Core) and Windows 10 Enterprise (1903). By god this should not be so hard, but it f*cking is.

Generic setup

Install samba :

yum install samba

edit /etc/samba/smb.conf

change the workgroup, netbios name and security, add log file and log level:

        workgroup = SERVERWORKGROUP
        netbios name = server
        security = user

        log file = /var/log/samba/%m.log
        log level = 1

Create user & share

Create a user :

useradd machine

optional, if only used for samba, you can not get them a /home directory and no shell using :

useradd -M -s /sbin/nologin machine

In order to enable the user, they require a password, however this won’t be used (unless if you gave them normal user shell & home dir)

passwd machine

Give them a samba password (used to authenticate) :

smbpasswd -a machine

Should give :

[[email protected] ~]# smbpasswd -a machine
New SMB password:
Retype new SMB password:
tdbsam_open: Converting version 0.0 database to version 4.0.
tdbsam_convert_backup: updated /var/lib/samba/private/passdb.tdb file.
Added user machine.

Enable the user :

smbpasswd -e machine

Now once we done that, its time to create the share folder :

Create the share path :

mkdir /server/upload/machine

Give the new user permissions :

chown machine.machine /server/upload/machine

then add the share in the configuration; (smb.conf)

Add :

        path = /server/upload/machine
        read only = no

now for the full smb.conf : test it !

[[email protected] upload]# testparm
Load smb config files from /etc/samba/smb.conf
Loaded services file OK.

Press enter to see a dump of your service definitions

# Global parameters
        log file = /var/log/samba/%m.log
        security = USER
        workgroup = MACHINEWORKGROUP
        idmap config * : backend = tdb

        path = /server/upload/machine
        read only = No


Whether you use iptables or firewalld; you need to open some ports for samba :

firewall-cmd --permanent --add-service=samba
firewall-cmd --reload

Start Service

Then we should be able to start the service and connect windows to it :

service smb start

note : I did not start nmb or winbind;

A happy ending for samba after all;

$ yum info samba
Last metadata expiration check: 0:26:43 ago on Mon 17 Feb 2020 03:52:39 PM CET.
Installed Packages
Name         : samba
Version      : 4.10.4
Release      : 101.el8_1
Architecture : x86_64
Size         : 2.4 M
Source       : samba-4.10.4-101.el8_1.src.rpm
Repository   : @System
From repo    : BaseOS
Summary      : Server and Client software to interoperate with Windows machines
URL          : http://www.samba.org/
License      : GPLv3+ and LGPLv3+
Description  : Samba is the standard Windows interoperability suite of programs for Linux
             : and Unix.

It’s the first time I’m upgrading Proxmox’s; There is luckily good documentation on the upgrade path; So this is only a small summary of how I did the upgrade of a few standalone proxmox machines; (big note : this is for non cluster proxmox)

First I upgrade everything to the latest :

apt-get update && apt-get dist-upgrade

Should work flawless :

[email protected]:~# apt-get update
Ign:1 http://ftp.be.debian.org/debian stretch InRelease
Get:2 http://security.debian.org stretch/updates InRelease [94.3 kB]
Get:3 http://ftp.be.debian.org/debian stretch-updates InRelease [91.0 kB]
Hit:4 http://ftp.be.debian.org/debian stretch Release
Hit:5 http://download.proxmox.com/debian stretch InRelease
Fetched 185 kB in 0s (308 kB/s)
Reading package lists... Done
[email protected]:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

In case you have kernel updates; its advisable to reboot the machine, so you work from the latest and greatest.

After this use the proxmox upgrade checktool :


This delivered :


TOTAL:    16
PASSED:   13

The only warning is :

WARN: 2 running guest(s) detected - consider migrating or stopping them.

On a different machine some SSH Ciphers where no longer supported; to fix this, remove them; from :


Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc

Then change the repo’s being used from stretch to buster, one can use this handy one liner; unless you have a subscription then you need to also change /etc/apt/sources.list.d/pve-enterprise

sed -i 's/stretch/buster/g' /etc/apt/sources.list

after that update the list :

apt-get update

if you want minimal downtime, you could first download the updates, while the machines are still running; using :

apt dist-upgrade --download-only

make sure all machines are down now (I used the webinterface); then run the upgrade : (and pray & take a coffee)

apt dist-upgrade

Some accept’s are required during the installation; (just enter to accept the defaults) these should not be essential to proxmox;

If everything is going fine; reboot and unleash the Proxmox 6.1*goodies, such as ZFS 0.8.2 🙂

I could already read ntfs file systems  but creating one seems to require more packages; I was surprised as I installed ntfs-3g but still could no run :

[[email protected] ~]# mkfs -t ntfs /dev/sda1
mkfs.ntfs: No such file or directory

The solution is a tiny other package :

yum install ntfsprogs

And magically :

mkfs -t ntfs /dev/sda1
Cluster size has been automatically set to 4096 bytes.

note : this will initialize the disk all with 0’s which could take an insane amount of time;

Therefore you could use the :

-f flag for skiping zero’ing and

-Q flag for skiping checking on bad sectors :

mkfs -t ntfs -f -Q /dev/sda1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.


Here we are again, ‘a brand new’ Proxmox : 6.1 based on debian Buster (10.2) including ZFS 0.8.2 (zfs on Linux). However updating by default will fail due to the inclusion of only the enterprise (paying) repo’s; They do however offer free repo’s that are marked as testing; To activate those and get rid of this ugly error, follow along;

remove or disable pve-enterprise repo :

rm -f /etc/apt/sources.list.d/pve-enterprise.list

add to nano /etc/apt/sources.list

deb http://download.proxmox.com/debian buster pve-no-subscription

Now you can run those updates 🙂  take note only  apt-get update and apt-get dist-upgrade are supported by Proxmox !

Update & upgrade

apt-get update
apt-get dist-upgrade

happy virtualizing !

While installing hdf5 I got this error :

# yum install hdf5.x86_64
Last metadata expiration check: 1:05:47 ago on Tue 14 Jan 2020 01:27:49 PM CET.
 Problem: conflicting requests
  - nothing provides libsz.so.2()(64bit) needed by hdf5-1.10.5-4.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

As it turns out, I’m not the only one. Turns out that installing epel-release does not cut it any longer; You need to install the PowerTools repo:

dnf config-manager --set-enabled PowerTools
yum update

After that; It installed fine :

 yum install hdf5.x86_64                                      Last metadata expiration check: 0:00:09 ago on Tue 14 Jan 2020 02:37:30 PM CET.
Dependencies resolved.
 Package              Arch            Version                   Repository           Size
 hdf5                 x86_64          1.10.5-4.el8              epel                2.1 M
Installing dependencies:
 libgfortran          x86_64          8.2.1-3.5.el8             BaseOS              636 k
 libquadmath          x86_64          8.2.1-3.5.el8             BaseOS              167 k
 libaec               x86_64          1.0.2-3.el8               PowerTools           39 k

Transaction Summary
Install  4 Packages

Total download size: 3.0 M
Installed size: 12 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): libaec-1.0.2-3.el8.x86_64.rpm                      1.4 MB/s |  39 kB     00:00
(2/4): libquadmath-8.2.1-3.5.el8.x86_64.rpm               4.0 MB/s | 167 kB     00:00
(3/4): libgfortran-8.2.1-3.5.el8.x86_64.rpm               8.2 MB/s | 636 kB     00:00
(4/4): hdf5-1.10.5-4.el8.x86_64.rpm                        11 MB/s | 2.1 MB     00:00
Total                                                     938 kB/s | 3.0 MB     00:03
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                  1/1
  Installing       : libaec-1.0.2-3.el8.x86_64                                        1/4
  Installing       : libquadmath-8.2.1-3.5.el8.x86_64                                 2/4
  Running scriptlet: libquadmath-8.2.1-3.5.el8.x86_64                                 2/4
  Installing       : libgfortran-8.2.1-3.5.el8.x86_64                                 3/4
  Running scriptlet: libgfortran-8.2.1-3.5.el8.x86_64                                 3/4
  Installing       : hdf5-1.10.5-4.el8.x86_64                                         4/4
  Running scriptlet: hdf5-1.10.5-4.el8.x86_64                                         4/4
  Verifying        : libgfortran-8.2.1-3.5.el8.x86_64                                 1/4
  Verifying        : libquadmath-8.2.1-3.5.el8.x86_64                                 2/4
  Verifying        : libaec-1.0.2-3.el8.x86_64                                        3/4
  Verifying        : hdf5-1.10.5-4.el8.x86_64                                         4/4

  hdf5-1.10.5-4.el8.x86_64                    libgfortran-8.2.1-3.5.el8.x86_64
  libquadmath-8.2.1-3.5.el8.x86_64            libaec-1.0.2-3.el8.x86_64



As its written in the extra note’s :

NOTE for CentOS 8 users
EPEL packages assume that the 'PowerTools' repository is enabled.
You can do this with: dnf config-manager --set-enabled PowerTools
NOTE for CentOS users
You can install EPEL by running yum install epel-release. 
The package is included in the CentOS Extras repository, enabled by default.

I got this seemingly wrong error message from Centos 8 Linux shell :

[[email protected] httpd]# ll
total 566
drwxrwxr-x  3 svennd svennd      3 Dec 11 16:23 apps
drwxrwxr-x 20 svennd svennd      20 Dec 11 16:23 exts
drwxrwxr-x  4 svennd svennd      10 Dec 11 16:23 lib
lrwxrwxrwx  1 svennd svennd      8 Dec 11 16:23 tclsh -> tclsh8.4
-rwxrwxr-x  1 svennd svennd      781385 Dec 11 16:23 tclsh8.4
drwxrwxr-x  3 svennd svennd      5 Dec 11 16:23 work
[[email protected] httpd]# ./tclsh8.4
bash: ./tclsh8.4: No such file or directory

It’s there right ? So let’s check whats going on with this “No such file”.

[[email protected] httpd]# file tclsh8.4
tclsh8.4: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.9, with debug_info, not stripped

The issue is in the 32-bit information you get here, all modern Linux distributions (here Centos 8) are 64-bit; By default they don’t come with 32-bit library’s; You could try to find out what is going on by using ldd & strace. 

But in this case it gave little information :

[[email protected] httpd]# ldd ./tclsh8.4
        not a dynamic executable

The solution is to install glibc for i686 (32 bit);

yum install glibc-devel.i686

And voila, magically the error disappeared and I could run this piece of ancient history.

Set IPMI to use DHCP

4 December, 2019

If you installed ipmitools, you can change the IP address source from static to DHCP or otherwise.

To change to DHCP :

ipmitool lan set 1 ipsrc dhcp

Check what is now setup :

ipmitool lan print

example :

[[email protected]~]# ipmitool lan print
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD
Auth Type Enable        : Callback : MD2 MD5 PASSWORD
                        : User     : MD2 MD5 PASSWORD
                        : Operator : MD2 MD5 PASSWORD
                        : Admin    : MD2 MD5 PASSWORD
                        : OEM      : MD2 MD5 PASSWORD
IP Address Source       : DHCP Address
IP Address              : -ethernet-
Subnet Mask             :
MAC Address             : -mac-
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP      : -gateway-
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       :
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max   : XaaaXXaaaXXaaXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : 0
Invalid password disable: no
Attempt Count Reset Int.: 0
User Lockout Interval   : 0


Thanks to this IBM documentation.

Disable 2FA in Proxmox 6

21 November, 2019

I was checking the new version of Proxmox, when I noticed there is an option to enable 2FA; (see the docs) It works really fine, but then I realized my colleague’s probably don’t have the key. To disable however, I couldn’t find it online; So I digged in the guts of Proxmox /etc/pve/user.cfg there is a line similar to :

user:[email protected]:1:0:::[email protected]::x!oath:

I changed this to :

user:[email protected]:1:0:::[email protected]:::


That change let me log in without the 2FA.