ZFS quota

19 April, 2017

ZFS is loaded with useful features, just a while ago I took a peak at setting quotes. Using just a few commands you limit the amount  of storage a certain pool can take.

First let’s check if there is no quota set :

zfs get quota

This would result in something like :

# zfs get quota
NAME                                                       PROPERTY  VALUE  SOURCE
huginn                                                     quota     none   default
huginn/dataset                                           quota     none   default
huginn/dataset@autosnap_2017-04-01_00:00:02_monthly      quota     -      -
huginn/dataset@autosnap_2017-04-10_00:00:01_daily        quota     -      -
huginn/dataset@autosnap_2017-04-19_14:00:01_hourly       quota     -      -
jbod1                                                      quota     none   default
jbod1/users                                                quota     none   default
jbod1/users/servers                                        quota     -     -

As you see, no quota’s. Now let’s add one :

zfs set quota=5TB jbod1/users/server

The format : zfs set quota=$size $pool

Let’s check again :

[root@huginn HG00731]# zfs get -r quota jbod1/users/wdecoster
NAME                                                       PROPERTY  VALUE  SOURCE
jbod1/users/server                                      quota     5T     local
jbod1/users/server@autosnap_2017-04-01_00:00:02_daily   quota     -      -
jbod1/users/server@autosnap_2017-04-02_00:00:01_daily   quota     -      -
jbod1/users/server@autosnap_2017-04-03_00:00:01_daily   quota     -      -
jbod1/users/server@autosnap_2017-04-03_13:00:01_hourly  quota     -      -
jbod1/users/server@autosnap_2017-04-03_14:00:01_hourly  quota     -      -

Simple enough; now some practical warnings :

  • full is full; you can’t remove, edit or overwrite data once its 100% full. Even tricks like “echo > big_file” won’t work. You need to either increase or drop the quota before its possible
  • You can’t decrease a quota … once its allowed 5TB only destroying it will release the limit.

One of those things I never spend much time on was reading about NFS. There is no need for it, it just kinda works out the box. Most of the time that is. The headache starts when it stops working, or pushes your cluster to a grind. One of the first resources you will find is to check /proc/net/rpc/nfsd. But that does not help you much further, so I recently started monitoring the content. Let’s see what is what.

Read More

ZFS replace a broken disk

28 March, 2017

After a reboot, I got greeted with this error :

root@server:~# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 10.1G in 0h26m with 0 errors on Mon Mar 27 16:35:05 2017

        NAME                     STATE     READ WRITE CKSUM
        rpool                    DEGRADED     0     0     0
          mirror-0               DEGRADED     0     0     0
            sda                  ONLINE       0     0     0
            2748060340541772838  UNAVAIL      0     0     0  was /dev/sdb2
            sdc2                 ONLINE       0     0     0
            sdd2                 ONLINE       0     0     0

I replaced the disk, and then forced a replace.

zpool replace -f rpool 2748060340541772838 /dev/sdb

zpool replace [-f] $poolname $old_device $new_device

The status is now :

root@server:~# zpool list
rpool   928G  10.2G   918G     16.0E     3%     1%  1.00x  DEGRADED  -
root@server:~# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Mar 28 11:31:57 2017
    14.3M scanned out of 10.2G at 1.02M/s, 2h50m to go
    14.2M resilvered, 0.14% done

        NAME                       STATE     READ WRITE CKSUM
        rpool                      DEGRADED     0     0     0
          mirror-0                 DEGRADED     0     0     0
            sda                    ONLINE       0     0     0
            replacing-1            UNAVAIL      0     0     0
              2748060340541772838  UNAVAIL      0     0     0  was /dev/sdb2
              sdb                  ONLINE       0     0     0  (resilvering)
            sdc2                   ONLINE       0     0     0
            sdd2                   ONLINE       0     0     0

errors: No known data errors


error libwrap missing

22 March, 2017

While compiling bareos I got this error :

checking for socket... yes
using socket from libc
checking for library containing nanosleep... no
checking for libwrap... configure: error: *** libwrap missing

The missing library on Centos is :


Fix :

yum install tcp_wrappers-devel

Since most of us work in a mixed environment (at best), we have to deal with all environments when it comes to backups. So here is how to install Bareos on Windows. There are two methods described in the manual. However I’m doing the GUI method. (It’s Windows after all) You can download the latest Windows Bareos binary package here.

Read More

The well known, good’ol LAMP stack has slowly made place for a LEMP stack in many of my configurations. Enough articles have been written on why Apache or Nginx are better in certain situations, but I like the clean way of configuration that Nginx uses. (pronounced Engine X, hence the E in LEMP). So I generally use Nginx. I however never made an article about it, so here it is, long overdue.

Read More

bareos : 2902 Bad storage

17 March, 2017

While working with bareos backups, I got this error :

Fatal error: Bad response to Storage command: wanted 2000 OK storage , got 2902 Bad storage

Read More

We recently moved over from Slack to a self-hosted communication app : Rocket.chat . One of the things I really enjoyed on slack, was the fact that there is a huge amount of integrations. Even from bash one can write to slack. This is particularly nice for reports of cron jobs, previously I relied on e-mails, but those tend to get ignored/spammed/… not read. So I let cron jobs also slack me. On Rocket.chat there are also incoming hooks, but I could not find a script that made it dummy prove to use like the one from Sul Aga so I took his code and adapted it to work with Rocket.chat. Here is how you can do the same.

Read More

After setting up my first job in bareos, I got this error :

sysadmin-fd JobId 4: Error: bsock_tcp.c:187 bnet_host2ipaddrs() for host "bareos" failed: ERR=Name or service not known
sysadmin-fd JobId 4: Fatal error: Failed to connect to Storage daemon: bareos:9103
bareos-dir JobId 4: Fatal error: Bad response to Storage command: wanted 2000 OK storage
, got 2902 Bad storage

host2ipaddr should have given it away, but it still took some fiddling for me. I found out that bareos expects to have a DNS server set up properly to connect to clients and in this case to “bareos” (itself). If you are just testing, you can add the client/server’s directly in /etc/hosts or in Windows C:\Windows\System32\drivers\etc\hosts but if you plan to go beyond the testing phase soon, it would be best to set up a FQDN that solves from both the clients and the server.

So quick and dirty :


add : bareos

And the errors melted like snow under the sun 🙂

I’m still a fan of iptables, I know firewalld is most likely the “wrapped iptables” future, but let’s hold on to iptables for just a bit longer shall we ? This is a short how-to on cleaning up PREROUTING NAT rules. PREROUTING can’t be flushed using iptables -F so its a bit different.
Read More