I hit on this annoying little bugger while setting up snmpd :

snmpwalk -c servers compute-0-1 SNMPv2-MIB::sysDescr.0  No log handling enabled - turning on stderr logging
snmpwalk: No securityName specified

Version of snmp :

snmpd -v

NET-SNMP version:  5.5
Web:               http://www.net-snmp.org/
Email:             net-snmp-coders@lists.sourceforge.net

In this particilar case I had to specificaly add the security version (v1,v2c,v3) :

snmpwalk -v 2c -c servers compute-0-1 SNMPv2-MIB::sysDescr.0

And it works ūüôā

Just playing around with bareos-webui and suddenly I’m greeted with a 500 error on the website. The director was running nicely and /var/log/messages did not show anything. Since this is a web server error, luckily there is something logged in the¬†/var/log/httpd/error.log¬†:

[Fri Mar 10 16:38:08.009365 2017] [:error] [pid 11266] [client hidden-ip:64292] PHP Fatal error:  Uncaught exception 'Zend\\Session\\Exception\\RuntimeException' with message 'Session validation failed' in /usr/share/bareos-webui/vendor/zendframework/zend-session/src/SessionManager.php:130\nStack trace:\n#0 /usr/share/bareos-webui/module/Application/Module.php(76): Zend\\Session\\SessionManager->start()\n#1 /usr/share/bareos-webui/module/Application/Module.php(43): Application\\Module->initSession(Object(Zend\\Mvc\\MvcEvent))\n#2 [internal function]: Application\\Module->onBootstrap(Object(Zend\\Mvc\\MvcEvent))\n#3 /usr/share/bareos-webui/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(Zend\\Mvc\\MvcEvent))\n#4 /usr/share/bareos-webui/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\\EventManager\\EventManager->triggerListeners('bootstrap', Object(Zend\\Mvc\\MvcEvent), Array)\n#5 /usr/share/bareos-webui/vendor/zendframework/zend-mvc/src/Application.php(157): Zend\\EventManager\\EventManager->trigger('bootstrap', Object(Zend\\Mvc\\MvcEvent))\n#6 /usr/share/bareos-web in /usr/share/bareos-webui/vendor/zendframework/zend-session/src/SessionManager.php on line 130

It seems its a known error for bareos-webui, the workaround is deleting the cookie related to bareos-webui. Thanks to this google post.

In the last posts, we setup bareos and connected a client to the bareos server, there are however no backup jobs auto-magically added to the bareos server so that’s our job. At first creating a job seems difficult and complex, but most of the options only become interesting in specific applications and a simple basic job can be configured pretty easy. That’s what I am going to do now, create a backup job that will incremental backup the /etc directory of my Linux client.
Read More

Bareos is really powerful, but also pretty complex. If we take it step by step, we will get stuff up¬†and running soon enough. Let’s start simpel by adding a client and setting up a backup job for the configuration of Linux servers (next post). Since I like to run¬†yum-cron¬†its quit common for configuration files to get added or changed. It’s also common for configuration to be made and forgotten about, but during a bare metal recovery of that server, its a great resource to have config files ! So let’s go ahead and back that up.

Read More

Bareos (Backup Archiving Recovery Open Sourced) is a backup solution, its a fork of the better known Bacula. At work we use a commercial package called retrospect, it has proven itself in many situations, never the less, its good to look out what free and open-source alternatives are out there. Bareos caught my eye in particular, cause they seem very active in the FOSS community (see fosdem 2017), always a good sign.

Bareos has a huge documentation, sadly¬†not everything is up-to-date or easy to understand for beginners like myself. The 500+ page manual is also not something you browse through quickly. That’s why I will post my guide(s) here, not cause the documentation is incomplete or perhaps not up-to-date. But to fellow sysadmin’s looking for a quick run at bareos without having to read through all the documentation. Like any open-source project, the more people who are involved the better the chance a¬†project can survive on the long term.

Read More

Create a ZFS mirror pool

8 March, 2017

I recently revived¬†a “old” compute cluster. While its hardware was formidable back in the day, now it has been replaced with a younger version, which has more compute nodes. I¬†wanted to create a data partition of two disks, normally I have a raid controller but on this server, there is none. ¬†An open invitation for a software RAID it is,¬†so I could go with¬†mdadm as software raid, but I did not want to read up on all those commands (again). When in fact ZFS is already in my head.

Creating a mirror or RAID 1 can be done using : zpool create $poolname mirror $first_disk $second_disk
This would make a mirror and mount it on /data, now that is the theory, in practice you most likely will be shown this error :

invalid vdev specification
use '-f' to override the following errors:
/dev/sdb does not contain an EFI label but it may contain partition information in the MBR.
/dev/sdc does not contain an EFI label but it may contain partition information in the MBR.

So be sure to add -f , it is however a good idea to double check if those are the disks you want to use.

zpool create -f data mirror /dev/sdb /dev/sdc

After that you can see

[root@lserver-01 ~]# zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
data   216K  3.51T    96K  /data
[root@server-01 ~]# zpool status
  pool: data
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0

errors: No known data errors

One of the tuning options for pools is ashift, which can be 9 (for 512 sector drives) or 12 (for 4k sector). However, this can only be set at creation. This can be done using the option -o ashift=value so why did I not tell you ? Cause ZOL (ZFS on Linux) since a while, will try and find the correct value.  From my finding (on the internet) almost all disks these days are 4k sector drives or advanced format drives. This you can check using hdparm -i /dev/sdb (you might need to install this)

[root@server ~]# hdparm -I /dev/sdb
        Model Number:       WDC WD4Y0                  
        Firmware Revision:  80.00A80
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Supported: 9 8 7 6 5 
        Likely used: 9
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors: 7814037168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:     3815447 MBytes
        device size with M = 1000*1000:     4000787 MBytes (4000 GB)

As you see the Logical sector size, is 512 bytes, this is for backwards compatibility, but the physical sector size is 4k. So in this situation a ashift=12 would be ideal. You can verify what your ashift is, by using the zdb tool :

[root@lungo-01 ~]# zdb | grep ashift
            ashift: 12

From what I read in the repo, it seems that 512 bytes in some cases can give you more storage if you have allot of very tiny files, compared to 4k, but that 4k is in almost all cases allot more performant. In general terms unless you really have a corner case, default ZFS will most likely guess the best option.

After creating this pool, I would recommend you read up on basic tuning, in short :

zfs set xattr=sa data
zfs set acltype=posixacl data
zfs set compression=lz4 data
zfs set atime=off data
zfs set relatime=off data

And that’s it folks !

Once ipmitools are installed (if not see this post) its easy :

ipmitool -I open user set password 2 NEW_PASSWORD

(the capitals are the new password)

The 2 stands for second user, all the machines I have seen 1 is “guest” who has no access.

During installing mysql2 gem on ruby :

checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lmygcc... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile

Its basically saying you miss a library, good thing someone tracked it down (see askubuntu)

So for Centos 7 : (for Centos 6 most likely mysql-devel)

yum install mariadb-devel

for Debian/Ubuntu :

apt-get install libmysqlclient-dev

Damn you Ruby.

I got this weird error, which -to be fair- I haven’t exactly found out why it happens but I found a way around it. If you ever figure out how please let me know.

There are some markers of this issue :

Trying to log in remote, seems to work but ends up with : (the console won’t open)

Server refused to allocate pty

In the /var/log/secure :

Feb 22 09:59:22 enterprise sshd[702]: Accepted password for root from some_IP port 54406 ssh2
Feb 22 09:59:22 enterprise sshd[702]: pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 22 09:59:22 enterprise sshd[702]: error: openpty: No such file or directory
Feb 22 09:59:22 enterprise sshd[702]: error: session_pty_req: session 0 alloc failed

Essential here are the openpty : no such file and session_pty_req : session 0 alloc failed. 

This may or may not be related to this issue but in /var/log/messages allot of errors started popping up :

eb 21 14:23:41 enterprise udevd-work[903]: inotify_add_watch(6, /dev/zd16, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4100]: inotify_add_watch(6, /dev/loop5, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4113]: inotify_add_watch(6, /dev/ram12, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4108]: inotify_add_watch(6, /dev/ram1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4095]: inotify_add_watch(6, /dev/loop1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4093]: inotify_add_watch(6, /dev/ram13, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4094]: inotify_add_watch(6, /dev/zd0, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4108]: inotify_add_watch(6, /dev/ram4, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4095]: inotify_add_watch(6, /dev/zd16, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4093]: inotify_add_watch(6, /dev/zd0p1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4094]: inotify_add_watch(6, /dev/zd0p2, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise ata_id[4098]: unable to open '/dev/.tmp-block-8:16'
Feb 21 14:23:41 enterprise ata_id[4087]: unable to open '/dev/.tmp-block-8:80'
Feb 21 14:23:41 enterprise ata_id[4083]: unable to open '/dev/.tmp-block-8:64'
Feb 21 14:23:41 enterprise ata_id[4081]: unable to open '/dev/.tmp-block-8:48'
Feb 21 14:23:41 enterprise ata_id[4114]: unable to open '/dev/.tmp-block-8:32'
Feb 21 14:23:41 enterprise ata_id[4102]: unable to open '/dev/.tmp-block-8:128'
Feb 21 14:23:41 enterprise ata_id[4122]: unable to open '/dev/.tmp-block-8:0'
Feb 21 14:23:41 enterprise ata_id[4089]: unable to open '/dev/.tmp-block-8:112'
Feb 21 14:23:41 enterprise ata_id[4105]: unable to open '/dev/.tmp-block-8:96'
Feb 21 14:23:41 enterprise ata_id[4176]: unable to open '/dev/.tmp-block-11:0'
Feb 21 14:23:41 enterprise ata_id[4177]: unable to open '/dev/.tmp-block-11:0'
Feb 21 14:23:41 enterprise udevd-work[903]: inotify_add_watch(6, /dev/sdd, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4085]: inotify_add_watch(6, /dev/sdh, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4082]: inotify_add_watch(6, /dev/sdf, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4080]: inotify_add_watch(6, /dev/sdd2, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4084]: inotify_add_watch(6, /dev/sdg1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4085]: inotify_add_watch(6, /dev/sde1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4106]: inotify_add_watch(6, /dev/sdb2, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[903]: inotify_add_watch(6, /dev/sdd1, 10) failed: Operation not permitted
Feb 21 14:23:41 enterprise udevd-work[4082]: inotify_add_watch(6, /dev/sdd9, 10) failed: Operation not permitted

The solution

Comment in the container /etc/rc.sysinit this line : (around line 155, nano +155 /etc/rc.sysinit)

/sbin/start_udev

Getting in the container can be done using lxc-attach -n ID after commenting that line reboot the container and it worked again for me.

Best of luck !

top explained visually

21 February, 2017

A while ago, I read this blog post¬†htop explained visually. I ¬†liked the idea of visually representing¬†htop, but beside the result of a visual representation it gives a good opportunity to dive in to those columns you silently have been ignoring for a large time. The author of the idea commented “I hope this post and your comment inspire people to create visual explanations for other unix commands, especially those that pack a lot of information on one screen :)”¬†hence my attempt to do something similar with the much more widespread command : top. As an extra point, I also used Inkscape¬†the more open version of Illustrator/Photoshop. (please note that you can make way, way more beautiful drawing in inkscape then what I made)

Read More