Bareos is really powerful, but also pretty complex. If we take it step by step, we will get stuff up and running soon enough. Let’s start simpel by adding a client and setting up a backup job for the configuration of Linux servers (next post). Since I like to run yum-cron its quit common for configuration files to get added or changed. It’s also common for configuration to be made and forgotten about, but during a bare metal recovery of that server, its a great resource to have config files ! So let’s go ahead and back that up.

Read More

Bareos (Backup Archiving Recovery Open Sourced) is a backup solution, its a fork of the better known Bacula. At work we use a commercial package called retrospect, it has proven itself in many situations, never the less, its good to look out what free and open-source alternatives are out there. Bareos caught my eye in particular, cause they seem very active in the FOSS community (see fosdem 2017), always a good sign.

Bareos has a huge documentation, sadly not everything is up-to-date or easy to understand for beginners like myself. The 500+ page manual is also not something you browse through quickly. That’s why I will post my guide(s) here, not cause the documentation is incomplete or perhaps not up-to-date. But to fellow sysadmin’s looking for a quick run at bareos without having to read through all the documentation. Like any open-source project, the more people who are involved the better the chance a project can survive on the long term.

Read More

Create a ZFS mirror pool

8 March, 2017

I recently revived a “old” compute cluster. While its hardware was formidable back in the day, now it has been replaced with a younger version, which has more compute nodes. I wanted to create a data partition of two disks, normally I have a raid controller but on this server, there is none.  An open invitation for a software RAID it is, so I could go with mdadm as software raid, but I did not want to read up on all those commands (again). When in fact ZFS is already in my head.

Creating a mirror or RAID 1 can be done using : zpool create $poolname mirror $first_disk $second_disk
This would make a mirror and mount it on /data, now that is the theory, in practice you most likely will be shown this error :

invalid vdev specification
use '-f' to override the following errors:
/dev/sdb does not contain an EFI label but it may contain partition information in the MBR.
/dev/sdc does not contain an EFI label but it may contain partition information in the MBR.

So be sure to add -f , it is however a good idea to double check if those are the disks you want to use.

zpool create -f data mirror /dev/sdb /dev/sdc

After that you can see

[root@lserver-01 ~]# zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
data   216K  3.51T    96K  /data
[root@server-01 ~]# zpool status
  pool: data
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0

errors: No known data errors

One of the tuning options for pools is ashift, which can be 9 (for 512 sector drives) or 12 (for 4k sector). However, this can only be set at creation. This can be done using the option -o ashift=value so why did I not tell you ? Cause ZOL (ZFS on Linux) since a while, will try and find the correct value.  From my finding (on the internet) almost all disks these days are 4k sector drives or advanced format drives. This you can check using hdparm -i /dev/sdb (you might need to install this)

[root@server ~]# hdparm -I /dev/sdb
        Model Number:       WDC WD4Y0                  
        Firmware Revision:  80.00A80
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Supported: 9 8 7 6 5 
        Likely used: 9
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors: 7814037168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:     3815447 MBytes
        device size with M = 1000*1000:     4000787 MBytes (4000 GB)

As you see the Logical sector size, is 512 bytes, this is for backwards compatibility, but the physical sector size is 4k. So in this situation a ashift=12 would be ideal. You can verify what your ashift is, by using the zdb tool :

[root@lungo-01 ~]# zdb | grep ashift
            ashift: 12

From what I read in the repo, it seems that 512 bytes in some cases can give you more storage if you have allot of very tiny files, compared to 4k, but that 4k is in almost all cases allot more performant. In general terms unless you really have a corner case, default ZFS will most likely guess the best option.

After creating this pool, I would recommend you read up on basic tuning, in short :

zfs set xattr=sa data
zfs set acltype=posixacl data
zfs set compression=lz4 data
zfs set atime=off data
zfs set relatime=off data

And that’s it folks !

Once ipmitools are installed (if not see this post) its easy :

ipmitool -I open user set password 2 NEW_PASSWORD

(the capitals are the new password)

The 2 stands for second user, all the machines I have seen 1 is “guest” who has no access.

During installing mysql2 gem on ruby :

checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lmygcc... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile

Its basically saying you miss a library, good thing someone tracked it down (see askubuntu)

So for Centos 7 : (for Centos 6 most likely mysql-devel)

yum install mariadb-devel

for Debian/Ubuntu :

apt-get install libmysqlclient-dev

Damn you Ruby.

I got this weird error, which -to be fair- I haven’t exactly found out why it happens but I found a way around it. If you ever figure out how please let me know.

There are some markers of this issue :

Trying to log in remote, seems to work but ends up with : (the console won’t open)

Server refused to allocate pty

In the /var/log/secure :

Feb 22 09:59:22 enterprise sshd[702]: Accepted password for root from some_IP port 54406 ssh2
Feb 22 09:59:22 enterprise sshd[702]: pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 22 09:59:22 enterprise sshd[702]: error: openpty: No such file or directory
Feb 22 09:59:22 enterprise sshd[702]: error: session_pty_req: session 0 alloc failed

Essential here are the openpty : no such file and session_pty_req : session 0 alloc failed. 

This may or may not be related to this issue but in /var/log/messages allot of errors started popping up :

eb 21 14:23:41 enterprise udevd-work[903]: inotify_add_watch(6, /dev/zd16, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4100]: inotify_add_watch(6, /dev/loop5, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4113]: inotify_add_watch(6, /dev/ram12, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4108]: inotify_add_watch(6, /dev/ram1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4095]: inotify_add_watch(6, /dev/loop1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4093]: inotify_add_watch(6, /dev/ram13, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4094]: inotify_add_watch(6, /dev/zd0, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4108]: inotify_add_watch(6, /dev/ram4, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4095]: inotify_add_watch(6, /dev/zd16, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4093]: inotify_add_watch(6, /dev/zd0p1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4094]: inotify_add_watch(6, /dev/zd0p2, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise ata_id[4098]: unable to open '/dev/.tmp-block-8:16'
Feb 21 14:23:41 enterprise ata_id[4087]: unable to open '/dev/.tmp-block-8:80'
Feb 21 14:23:41 enterprise ata_id[4083]: unable to open '/dev/.tmp-block-8:64'
Feb 21 14:23:41 enterprise ata_id[4081]: unable to open '/dev/.tmp-block-8:48'
Feb 21 14:23:41 enterprise ata_id[4114]: unable to open '/dev/.tmp-block-8:32'
Feb 21 14:23:41 enterprise ata_id[4102]: unable to open '/dev/.tmp-block-8:128'
Feb 21 14:23:41 enterprise ata_id[4122]: unable to open '/dev/.tmp-block-8:0'
Feb 21 14:23:41 enterprise ata_id[4089]: unable to open '/dev/.tmp-block-8:112'
Feb 21 14:23:41 enterprise ata_id[4105]: unable to open '/dev/.tmp-block-8:96'
Feb 21 14:23:41 enterprise ata_id[4176]: unable to open '/dev/.tmp-block-11:0'
Feb 21 14:23:41 enterprise ata_id[4177]: unable to open '/dev/.tmp-block-11:0'
Feb 21 14:23:41 enterprise udevd-work[903]: inotify_add_watch(6, /dev/sdd, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4085]: inotify_add_watch(6, /dev/sdh, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4082]: inotify_add_watch(6, /dev/sdf, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4080]: inotify_add_watch(6, /dev/sdd2, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4084]: inotify_add_watch(6, /dev/sdg1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4085]: inotify_add_watch(6, /dev/sde1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4106]: inotify_add_watch(6, /dev/sdb2, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[903]: inotify_add_watch(6, /dev/sdd1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4082]: inotify_add_watch(6, /dev/sdd9, 10) failed: Operation not permitted

The solution

Comment in the container /etc/rc.sysinit this line : (around line 155, nano +155 /etc/rc.sysinit)

/sbin/start_udev

Getting in the container can be done using lxc-attach -n ID after commenting that line reboot the container and it worked again for me.

Best of luck !

top explained visually

21 February, 2017

A while ago, I read this blog post htop explained visually. I  liked the idea of visually representing htop, but beside the result of a visual representation it gives a good opportunity to dive in to those columns you silently have been ignoring for a large time. The author of the idea commented “I hope this post and your comment inspire people to create visual explanations for other unix commands, especially those that pack a lot of information on one screen :)” hence my attempt to do something similar with the much more widespread command : top. As an extra point, I also used Inkscape the more open version of Illustrator/Photoshop. (please note that you can make way, way more beautiful drawing in inkscape then what I made)

Read More

Install ZFS on Centos

17 February, 2017

Installing ZFS on Centos has been ironed out pretty much, so just follow along. There are three ways I know of to install ZFS on Centos. The two methods are recommended, as they use a repository, the last is just compiling from source, which I like, since I decide when its updated. (as kernel updates have caused some issue’s after reboots for ZFS)

The ZOL (ZFS on Linux) advice is to use kABI-tracking kmod, so that on updating to a newer kernel, ZFS does not need to be rebuild, however the default repository points to DKMS style tracking. This does work, but I had issue’s in the past,  so I chose to compile from source and update when I know we can handle some downtime.

Read More

I did not -physically- make it to Fosdem 2017.  Luckily the guys from Fosdem offered a live-stream for most part of the event, this gave the advantage for digital lurkers, like me, to jump from one talk to the next mid presentation, without being rude to presenter. While randomly wandering between two presentations, I stumbled upon this passbolt presentation, I had read of passbolt before, but I was put off by the fact that it required a browser plugin.

In case you don’t know passbolt, it’s a password managers aimed at managing passwords for teams. But don’t take my word for it, go over to the passbolt website and find out.

passbolt install on centos

passbolt install on centos

Read More

Trying to install GnuPG and failing with :

configure: error: Please reinstall the gpgme distribution
ERROR: `/var/tmp/gnupg/configure --with-php-config=/usr/bin/php-config' failed

I hit on this error while installing using pecl :

[root@tetra tetra]# pecl install gnupg
downloading gnupg-1.4.0.tgz ...
Starting to download gnupg-1.4.0.tgz (28,349 bytes)
.........done: 28,349 bytes
6 source files, building
running: phpize
Configuring for:
PHP Api Version:         20151012
Zend Module Api No:      20151012
Zend Extension Api No:   320151012
building in /var/tmp/pear-build-root7FLwJg/gnupg-1.4.0
running: /var/tmp/gnupg/configure --with-php-config=/usr/bin/php-config
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for a sed that does not truncate output... /usr/bin/sed
checking for cc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether cc accepts -g... yes
checking for cc option to accept ISO C89... none needed
checking how to run the C preprocessor... cc -E
checking for icc... no
checking for suncc... no
checking whether cc understands -c and -o together... yes
checking for system library directory... lib
checking if compiler supports -R... no
checking if compiler supports -Wl,-rpath,... yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for PHP prefix... /usr
checking for PHP includes... -I/usr/include/php -I/usr/include/php/main -I/usr/include/php/TSRM -I/usr/include/php/Zend -I/usr/include/php/ext -I/usr/include/php/ext/date/lib
checking for PHP extension directory... /usr/lib64/php/modules
checking for PHP installed headers prefix... /usr/include/php
checking if debug is enabled... no
checking if zts is enabled... no
checking for re2c... no
configure: WARNING: You will need re2c 0.13.4 or later if you want to regenerate PHP parsers.
checking for gawk... gawk
checking for gnupg support... yes, shared
checking for gnupg files in default path... not found
configure: error: Please reinstall the gpgme distribution
ERROR: `/var/tmp/gnupg/configure --with-php-config=/usr/bin/php-config' failed

What it is missing is some development files that gpgme supply; On Centos this is the dependency that is missing :

yum install gpgme-devel

On Debian :

apt-get install libgpgme11-dev

After that, it installed perfectly, don’t forget to add a file /etc/php.d/ (on Centos) with content :

#load gnupg extension
extension=gnupg.so