fatal error: ffi.h: No such file or directory

Missing library’s are painful, this one I hit upon while compiling some python dependency’s.

gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
    c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
     #include <ffi.h>
    compilation terminated.
    error: command 'gcc' failed with exit status 1

Command "/usr/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-EHadKY/cffi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-5wnmzT-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-EHadKY/cffi/

for Centos 6/7 (and all other redhat related distro’s)

yum install libffi-devel

for Debian brothers :

apt-get install libffi-dev



Bored this weekend and interested in free/open-source ? Perhaps consider a trip to Brussels, for this awesome event :

FOSDEM is a free and non-commercial event organised by the community for the community. The goal is to provide free and open source software developers and communities a place to meet to:

  • get in touch with other developers and projects;
  • be informed about the latest developments in the free software world;
  • be informed about the latest developments in the open source world;
  • attend interesting talks and presentations on various topics by project leaders and committers;
  • to promote the development and benefits of free software and open source solutions

Read more at the FOSDEM website.

dehydrated using cloudflare hook

After the fairy dust has settled on Let’s Encrypt, allot of good solutions are out there to get a valid certificate using the ACME protocol (see client list). Most of the technique’s use webroot as verification technique, while this is fine for most websites, it has some downsides. For one, you need to adapt (let it be adapted) the webserver configuration, to allow connection to the webroot, for allot of webservers this is difficult or annoying. The alternative solution is to add a TXT record in the DNS manager and verify the domain using that method. This however, is more difficult to automate, but if you use cloudflare like I do (at least for DNS) you can automate it pretty easely, using dehydrated (previously known as letsencrypt.sh) and the cloudflare hook (by kappataumu). This is how I generally install it on non-public servers that require https.


Some of these might already be installed but for a clean server they aren’t.  You do need epel-release, so after that install, we need to update yum so that the packages and dependency’s can be found.

yum install epel-release
yum update
yum install git python-pip gcc python-devel libffi-devel openssl-devel


Now its time to get the software that runs this setup : I for one do this in /opt but its totally up to you.

cd /opt
git clone https://github.com/lukas2511/dehydrated

Next we are going to download the hook for cloudflare DNS :

cd dehydrated
mkdir hooks
git clone https://github.com/kappataumu/letsencrypt-cloudflare-hook hooks/cloudflare

Now we need to run the requirements of python, on Centos 6/7, the default python version is 2.X So run :

pip install -r hooks/cloudflare/requirements-python-2.txt

For linux distro’s where Python 3.x is default use :

pip install -r hooks/cloudflare/requirements.txt

If that finishes ok, its time to make a config file for dehydrated. (not strictly necessary but for a cron & automating allot easier!)


Dehydrated uses a file “config”, I however dislike this and renamed it config.sh (as its shell code) and linked a config file. (you could even put it in /etc where this would fit better) We also need a file domains.txt this is a space separated list of the domain + sub-domains, and if you have multiple domains, split them by a line break.

cd /opt/dehydrated
touch config.sh
ln -s config.sh config
touch domains.txt

You can find an example config + domains.txt in docs/examples/.

This is a valid domains.txt :

svennd.be www.svennd.be
divebug.com www.divebug.com demo.divebug.com

This will request two certificates, one for svennd.be and one for divebug.com the www is a sub-domein and also demo is a sub-domain.

My config : (stripped of sensitive information)

# cloudflare settings
export CF_EMAIL=""
export CF_KEY=""

# letsencrypt.sh
export CHALLENGETYPE="dns-01"
export CONTACT_EMAIL="valid@mail.ext"
export HOOK="hooks/cloudflare/hook.py"
export RENEW_DAYS="60"

This will check if the current certificates are valid for at least 60 days (RENEW_DAYS) if not it will get a new certificate. note that you need to change CONTACT_EMAIL, as this will inform you if the renew hasn’t happened and you did not notice.

CF_EMAIL is the login email for Cloudflare. The CF_KEY is an API key used to access Cloudflare API’s you can find it under my account

Just so you know : I removed my api key for this screenshot 🙂

Getting certificate

First time you need to register & accept the agreement of Let’s Encrypt :

./dehydrated --register --accept-terms

If setup properly, you should now be able to run ./dehydrated -c this would generate valid certificates : (something similar like 🙂

# ./dehydrated -c
# INFO: Using main config file /opt/dehydrated/config
Processing svennd.be with alternative names: www.svennd.be
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Apr  2 09:21:00 2017 GMT (Less than 91 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for svennd.be...
 + Requesting challenge for www.svennd.be...
 + CloudFlare hook executing: deploy_challenge
 + Settling down for 10s...
 + Responding to challenge for svennd.be...
 + CloudFlare hook executing: clean_challenge
 + Challenge is valid!
 + CloudFlare hook executing: deploy_challenge
 + Settling down for 10s...
 + Responding to challenge for www.svennd.be...
 + CloudFlare hook executing: clean_challenge
 + Challenge is valid!
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
 + CloudFlare hook executing: deploy_cert
 + ssl_certificate: /opt/letsencrypt/certs/svennd.be/fullchain.pem
 + ssl_certificate_key: /opt/letsencrypt/certs/svennd.be/privkey.pem
 + Done!
Processing divebug.com with alternative names: www.divebug.com
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Apr  2 09:22:00 2017 GMT (Less than 91 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for divebug.com...
 + Requesting challenge for www.divebug.com...
 + CloudFlare hook executing: deploy_challenge
 + Settling down for 10s...
 + Responding to challenge for divebug.com...
 + CloudFlare hook executing: clean_challenge
 + Challenge is valid!
 + CloudFlare hook executing: deploy_challenge
 + Settling down for 10s...
 + Responding to challenge for www.divebug.com...
 + CloudFlare hook executing: clean_challenge
 + Challenge is valid!
 + Creating fullchain.pem...
 + CloudFlare hook executing: deploy_cert
 + ssl_certificate: /opt/letsencrypt/certs/divebug.com/fullchain.pem
 + ssl_certificate_key: /opt/letsencrypt/certs/divebug.com/privkey.pem
 + Done!

note : I cheated and put the renew_days from 60 to 91, since let’s encrypt certificates are valid for 90 days, this script would always request new certificates which is not needed.

I run this script in /etc/cron.weekly/cert.cron :

/opt/dehydrated/dehydrated -c

For debugging purpose it is sometimes useful to run it without the need for a fully working config, in that case you can use :

./dehydrated -c -d svennd.be -t dns-01 -k 'hooks/cloudflare/hook.py'

That’s it ! Fully configured certificate renewal process 🙂

Tuning of ZFS module

Tuning of ZFS module

The more difficult part of ZOL is the fact that there are plenty of tune able kernel module parameters, and hence ZFS can be used in many kinds of systems for many different reasons. (laptops, file-servers, database-servers, file-clusters) However some of the parameters come bad out of the box for file serving systems. Here are some of the options that might help you tune ZFS on Linux.


The ARC (Adaptive Replacement Cache) is a cache that’s used to speed up read actions, (also async writes can be grouped here before flushed to disk (?)) by default zfs_arc_max is going to take (all physical memory – 1 GB) if the total physical memory is larger then 4GB. At least if I can believe this document however I think ZOL uses 75% by default. On a storage-only device, this can be increased, however on anything else, while the memory will be used for ARC, the ARC size will lower when applications require it. The only disadvantage is; some applications will require a certain amount of memory to start, or request memory on a rate that is quicker then ARC size can lower, resulting in a crash/swapping.  The zfs_arc_min is the smallest value the ARC will become when there is memory pressure. You could lower it on for example laptops/desktops that only use ZFS for a part of the system (backup storage).

The options are expressed in bytes…

# i did not adapt zfs_arc_min
# options zfs_arc_min=

# i increased zfs_arc_max as this is a 128GB memory machine only serving files
options zfs_arc_max=100000000000

Possible options to check are arcstat :

# /opt/zfs/arcstat.py -f "time,read,hit%,hits,miss%,miss,arcsz,c" 1
    time  read  hit%  hits  miss%  miss  arcsz     c
16:21:42   52K    98   52K      1   716    48G   48G
16:21:43   69K    98   68K      1   937    48G   48G
16:21:44   38K    98   38K      1   459    48G   48G
16:21:45   41K    98   40K      1   623    48G   48G
16:21:46   47K    98   46K      1   483    48G   48G

As you see, ZFS dynamically adapts ARC, seems my server is not working very hard at this moment, and hence the arcsize is only ~48GB large. Useful stats are the hit%. If you get a low value, this would mean ARC is not catching much requests, so increasing might help when you expect read requests should be able to be cached.

It is also possible to disable ARC per dataset, or reduce what it is used for, of course this will slow any file action hugely and you should only do it when you really don’t care about speed in that particular data-set.  There is a primarycache (RAM), and a secondarycache (cache devices, such as SSD’s), by default they are both used for both metadata and data. The options are :

  • all : both metadata and data are cached
  • metadata : only metadata is cached (file names, sizes, attributes, …)
  • none : no cache is used
# zfs get all jbod1 | grep cache
jbod1  primarycache          all                    default
jbod1  secondarycache        all                    default

I left as is (default is all), and would advice you (based on this mailinglist) to also leave it default. A much better option is to lower the zfs_arc_max since ZFS will still be able to cache some of the data/metadata.


ZFS has five I/O queues for each of the different I/O types :

  • sync reads : normal read operation by applications
  • async reads : reads by ZFS prefetcher, this will try to “guess” what writes an application might need and request them and cache them
  • sync writes : a write that requires fsync(); to be written to “long term storage”, basically disk or when a separate ZIL is available written to ZIL (intent log).
  • async writes :  a ‘normal’ write. (called bulk writes of dirty data)
  • scrub :  those are scrub and resilver operation, scrub checks for wrong checksums, resilver tries to repair. (resilver : I would say, raid rebuild)

This is straight from the ZFS on Linux sourcecode :

The ratio of the queues’ max_actives determines the balance of performance between reads, writes, and scrubs. E.g., increasing zfs_vdev_scrub_max_active will cause the scrub or resilver to complete more quickly, but reads and writes to have higher latency and lower throughput

These value’s however are low for modern hardware and can be increased. Now I don’t completely understand what is at play here, so feel free to research on this, but these value’s seem to increase throughput at cost of latency. For example; Scrubbing is now a few factors quicker … from a few days to 18h / ~100TB. (that makes it I can cron it, more on scrub frequently)

Each of the zfs_vdev_*_*_max_active limits the number of IO’s issues to a single vdev.  Finding the right value’s can be done by monitoring IO throughput and latency under load, and increase the value until you find the point where throughput no longer increased and latency is still acceptable. These are the value’s I use :

# increase them so scrub/resilver is more quickly at the cost of other work
options zfs zfs_vdev_scrub_min_active=24
options zfs zfs_vdev_scrub_max_active=64

# sync write
options zfs zfs_vdev_sync_write_min_active=8
options zfs zfs_vdev_sync_write_max_active=32

# sync reads (normal)
options zfs zfs_vdev_sync_read_min_active=8
options zfs zfs_vdev_sync_read_max_active=32

# async reads : prefetcher
options zfs zfs_vdev_async_read_min_active=8
options zfs zfs_vdev_async_read_max_active=32

# async write : bulk writes
options zfs zfs_vdev_async_write_min_active=8
options zfs zfs_vdev_async_write_max_active=32

At current writing these are the default value’s in the ZoL repo:

zfs_vdev_sync_read_min_active = 10;
zfs_vdev_sync_read_max_active = 10;
zfs_vdev_sync_write_min_active = 10;
zfs_vdev_sync_write_max_active = 10;
zfs_vdev_async_read_min_active = 1;
zfs_vdev_async_read_max_active = 3;
zfs_vdev_async_write_min_active = 1;
zfs_vdev_async_write_max_active = 10;
zfs_vdev_scrub_min_active = 1;
zfs_vdev_scrub_max_active = 2;

As far as I understand, dirty data are non-sync writes, so basically the applications says to the file system, write this down, but doesn’t wait or expects a reply. Since I have a L2ARC and ARC, this should cache it up first and only then we should start writing it to “slow” disks. This value limits the amount of data that can be at any given time in the pool. (the cap is at 25% of physmem)

Default out of the box is 10%;

options zfs zfs_dirty_data_max_percent=40

Maximum number of scrub I/O per top-level vdev, by default 32. Increases zfs scrub speed. (at what cost, no idea)

options zfs zfs_top_maxinflight=320

These regulate the default ticks between two actions (either scrub or resilver), I would echo 0 > /sys/module/zfs/parameters/zfs_resilver_delay when a disk has broken, but that is just me.

options zfs zfs_resilver_delay = 2
options zfs zfs_scrub_delay = 4

There is a time before async writes are written to disk, this makes it possible for ZFS to write a larger piece. The old default was 30 seconds, but due to fluctuating write performance on some machines was lowered to 5 seconds. Decent servers should be able to hold a bit of data, so I increased it to 15 seconds, more might be fine, depending on the workload.

options zfs zfs_txg_timeout=15

I see this online frequently where this setting is set; I contacted an expert and  Noop and Deadline are both fine.  You can read more about noop scheduler and the deadline scheduler, but I don’t understand enough to give a fair value. I did get the feeling deadline is more focused on databases pattern but I could be wrong. I did not change it.

# default : noop
options zfs zfs_vdev_scheduler=deadline

ZFS can predict what data will be requested next, and can cache this until the request comes, this is easier and faster for spinning disks, by default its disabled (it was a surprise to me) setting this value to zero will activate the setting.

# use the prefetch method
options zfs zfs_prefetch_disable=0

Only applies if you have cache device such as a ssd, when ZFS was created, ssd’s where new and could only be written to a few times, so zfs has some prehistoric limits to save the SSD of the hard labor. l2arc_write_max is such a value, by default only 8mb/s can be written to the ssd. Clearly you can increase this. (at the cost of more SSD use)

# max write speed to l2arc
# tradeoff between write/read and durability of ssd (?)
# default : 8 * 1024 * 1024
# setting here : 500 * 1024 * 1024
options zfs l2arc_write_max=524288000

Number of max devices writes to precache, can be increased.

# number of max device writes to precache
# default : 2
options zfs l2arc_headroom=12
 zfs_immediate_write_sz || zil_slog_limit

zfs_log_write() handles TX_WRITE transactions. The specified callback is called as soon as the write is on stable storage (be it via a DMU sync or a * ZIL commit).

a useful article on this tune-able parameter.  On the other hand, zil_slog_limit is the max commit in byte to the separate log device, in short another attempt to not overuse the slog device. (ssd)

# default : 32768
options zfs zfs_immediate_write_sz=131072 

# default : 1024*1024 = 1mb
options zfs zil_slog_limit=536870912


Important note : I received / found these value’s floating on the internet, if you did not yet realize I am not an expert at this, this is my “best” attempt to understand these value’s and settings, and this information might just be plain wrong (or dated when you read it), do your research before changing value’s in production. You can’t blame me if your servers turns out to be the next skynet.

Using ZOHO SMTP mail on wordpress

After three hours reading up on configuring postfix/dovecot/… I finally thrown in the towel. There are some resources on the interwebs on how to setup a clean postfix on Centos/Linux, but they are dated or they just throw in a configuration file without explanation, and while I eventually got postfix mail to work. Sending a mail was slow (load time, no idea why) and gmail/google showed the mail being from root@linux_host_name.svennd.be which I did not like, since I wanted it to come from no-reply@svennd.be. On top of that, google put it in the spam directory, even after I put a valid SPF record. (for some reason this also wasn’t accepted)


So I gave it my best shot, it’s time to accept defeat. Instead of trying to do it myself, I used zoho mail (referral link), this article is about how I set up mailing using the free service of zoho on my wordpress blog.


Nginx basic HTTP authentication

A short howto, create a basic HTTP authentication on Nginx, Centos 6.8.

  • Nginx is setup correctly
  • root access

First lets get htpasswd, this “manages” the password file and creates users.

yum install httpd-tools


The first run you need to create a new .htpasswd file :

htpasswd -c /var/www/html/.htpasswd svennd

This would create the passwd file and a user svennd (you will be prompted for the password)

Next the nginx configuration in the server {} block :

location /my_secret {
        auth_basic "restricted";
        auth_basic_user_file /var/www/html/.htpasswd;

To restrict access to my_secret directory. Next is to check if nginx configuration is fine :

# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

And reload nginx :

service nginx reload

Bam, that easy !

Basic ZFS tune tips

Lets start out with saying that ZFS on Linux (ZoL) is pretty awesome, snapshots, compression-on-the-fly, quota’s, … I love it. However ZoL also has issue’s, I even blogged about some in the past. (a full ZFS pool, failing to load ZFS module) Most cases where my own fault, but still, its very uncommon for a file system to be such a pain in the ass.

The last few days, I tried to find out why, my installation was so slow. Slow in this context is hard to define. Sometimes users complained about slow ls runs, while write speed was decent enough … read speed was also fine, perhaps NFS was at fault, perhaps disks, controller, … ?

Long story short, I looked for some tuning options, as the defaults most likely aren’t perfect. Before I take all the credit, I was masterly helped by senior systems engineer, ewwhite in this matter. So lets go over some “simple” tune options during initial creation.

The setup

  • Centos 7.3 (up-to-date, nightly)
  • 128GB RAM
  • Xeon E5-2620 v3
  • 3 * raidz2 with each 12 x 4Tb drives SAS 3 (HGST Ultrastar 7K6000)
  • 2 * Intel SSD DC P3600 Series PCI-E, 400GB (used for cache and in mirror logs, partitioned)
  • ZFS on Linux : v0.6.5.8

Bragging rights, right ? Sadly its for work… (or happy for my power bill)

Tuning during creation

Setting these value’s can be done on a live system, however most of them will only apply for new data or rewrites, so try to do them when creating a pool. Setting them to a pool is done :

# in case you want to apply to a specific subpool
zfs set parameter=value pool/subpool

# pool and all not locally set subpools, will receive this parameter
zfs set parameter=value pool

Sub-pools inherit from the pool above them, so setting these features will auto-magically inherit from the top level setting when you create sub-pools. (Eg. tank/users will inherit from tank unless defined otherwise)

  • xattr=sa, default xattr=on
    • What happens on default behavior is that for every xattr object a file in a hidden directory is created, so browsing or ls‘ing requires up to three disk seeks, on top of that it seems its not really well cached. The reason it was not changed to default behavior seems to be compatibility, which is a good reason if you work with other systems such as Solaris. But for Linux only environment its not relevant.
    • This option only works for newly written files, so if you only notice this tune option post-creation, you will need to remove and copy them in the pool again to take effect. Perhaps you can use zfs send and zfs receive. First set the xattr=sa then zfs send to an image, destroy the location, and zfs receive in the (old) location. (note : Not sure if this works, you could also use rsync here, which most likely will work.)
    • original implementation, I’m not alone award.
    • zfs set xattr=sa tank/users
      zfs send tank/users > tank/users_disk.img
      zfs destroy tank/users
      zfs receive tank/users < tank/users_disk.img
  • acltype=posixacl, default acltype=off
    • The reason behind this is fuzzy to me, but they parameters come up together regularly, so they are most likely related somehow. (need to research further)
  • compression=lz4, default compression=off
    • I did activate this option when I first started, basically the file system will try and compress all files, but when an application requests them, it will decompress them, without anyone noticing, of-course this creates a bit of CPU but any decent server should be able to do it. On a ~100 TB system :
      [root@huginn jbod1]# zfs get all huginn| grep compres
      huginn  compressratio         1.36x                  -
      huginn  compression           lz4                    local

      That’s 36 TB of extra space… note : allot of data is already compressed more heavy using gzip. This is a must have. Not only does it help you get more storage, writing uncompressed data is most likely taking longer then writing compressed data.

  • atime=off
    • atime is short for “last” access time, so every time a file is read a update has to flush to disk to reflect this access item. For most applications that’s totally irrelevant and a quit overhead.  I always set it to off. In cases where you wish to use access time, look to relatime.
  • relatime=off
    • relatime was introduced in Linux 2.6.20 kernel. It only updates the atime when the previous atime is older then mtime (modification time) or ctime (changed time) or when the atime is older then 24h (based on setting). I am assuming the zfs implementation is using this kernel feature. I don’t require relatime or atime, but this is the preferred option. Perhaps in the future we might see lazytime (linux kernel 4.X) which uses memory to store atime updates and write them during normal I/O.

And that’s it, next time lets look at the kernel module 🙂

single line power : create a swap file in linux

In case the machine has no swap, (VPS are typical clients but also containers) you can just create a file and assign it to be used as swap. So here is the magic line : (this creates a 512mb swapfile)

dd if=/dev/zero of=/swapfile bs=1024 count=512k && chown root:root /swapfile &&  chmod 0600 /swapfile && mkswap /swapfile -f && swapon /swapfile


In steps ? OK :

(1) create an file filled with zero’s of 1024*512k = 512M

dd if=/dev/zero of=/swapfile bs=1024 count=512k

(2) make user:group both root

chown root:root /swapfile

(3) only allow read/write (2+4) on the file

chmod 0600 /swapfile

(4) prepare this file for swap usage

mkswap /swapfile -f

(5) tell the kernel a new swap location is ready for use

swapon /swapfile

(6) not in the single line but you should also add this file to /etc/fstab in case it reboots

/swapfile    none    swap    sw    0   0

two remarks :

  • Don’t use this on desktops to hibernate
  • SSD’s won’t like that kind of file, prefer not to place it on SSD’s if you can chose.