crond: (root) BAD FILE MODE

27 November, 2015

AAAAAAAAAAAAAAAAAAAAAAAAA you’re cron is not working.

You would see this error in /var/log/cron-* or /var/log/cron or in some configurations /var/log/messages. Its panic time!

Just kidding, bad file mode is just permissions.  You fix it with :

chmod 0644 /etc/cron.d/your.cron

Also a common one is WRONG FILE OWNER again rather straight forward :

chown root.root /etc/cron.d/your.cron

Note that you need root to be the owner, even tho you can execute it as a non root user.



A great find today, I’d like to share : LibreNMS ! Its a really open source fork of observium. I was a fan of observium, but I dislike the fact that they took “our” kickstarter money and just force you to pay for the latest codebase. The “community” version is just updated twice a year. Clearly SNMP is a bit of a black box and every vendor changes stuff when they feel like it, (I hear it has something to do with the moon/sun-cycle)  so its needed to update regularly, allot of functions are not available in community edition.

I don’t mind these guys making money, I just don’t find it useful to pay for a license for something that is only considered ‘good practice’ where I work. (I only maintain ~20 server and downtime is not something we get killed for, just beaten up a bit)

Comes to save my day : LibreNMS.

I really wanted to add a few lines of code to observium, but since it was closed source, could not do that, as such I wanted to build a own simple system. Good thing some heroes are out there that saved me the extra headache!

Happy network monitoring!

On a virtual server,  date was responding with the correct time/date. However logs where filling up with a wrong time. I normally change the timezone like this : (adapt as needed)

ln -sf /usr/share/zoneinfo/Europe/Brussels /etc/localtime
[root@main ~]# date
Sat Nov 14 21:55:22 CET 2015

Also in /etc/sysconfig/clock change to : (adapt as needed)


Even when those are done the logs kept on showing up logs very long time ago, the cause was rsyslog running behind… never happened before so :

service rsyslog restart

This was also causing my fail2ban to be to easy on this spammers I believe! Questions, remarks, just shoot below !


Update : seems /etc/sysconfig/clock is not per definition the problem, its rsyslog that is keeping the old date, just restarting ryslog seems to be enough!

A little annoying problem, not blocking but easy to solve!

[root@host~]# service httpd configtest
httpd: apr_sockaddr_info_get() failed for host.ext
httpd: Could not reliably determine the server's fully qualified domain name, using for ServerName
Syntax OK

We could simply add it in /etc/hosts

# fix apache warning host.ext

fail2ban add ip manually

9 November, 2015

My love for fail2ban is slowly decreasing, I had a problem a while ago, while fixed, it was not easy to find. (see fail2ban 0.9.9 with centos 6.7 not adding ips to firewall) Today, again, I noticed that my iptables -L was empty, while my /var/log/secure was full (~70k lines, in one day) and fail2ban was doing absolutly nothing … 🙁

I thought my older version of iptables (16.el6, 1.4.7) was the cause again, but it seems its not. Since I can use fail2ban to add ip’s to iptables. I found the command, on the website by the original patch creator 🙂  Great tool if you wanne ban a range quickly!

fail2ban-client set ssh-iptables banip 123.456.789.001


Today I was searching for tools to centralize the logging of some 20 odd Linux servers, while this is no endpoint in my research, I “logged” the method I used to setup my test/demo servers using good old rsyslog.

While there are allot possibilities towards logging, I’d like :

  • local + remote logging
    • It would be nice if I could log into one “remote” machine/service and see the log files of all machines I need to “maintain”. However keeping a local log file is also important since not everybody will have access to the central machine logging server/service.
  • KISS
    • As little as possible dependencies, preferably it should run up from a ancient Centos 5 to a bleeding edie nighly Ubuntu virtual machine.  So preferably a default package in the common linux distro’s.
  • little overhead
    • The overhead for client servers should be as small as possible, since they do have a real job beside logging.

Most of these points are checked off when working with rsyslog, so I took that solution out for a spin. With rsyslog we can filter out some irrelevant messages (like DHCP requests), use different logging servers for different levels/labels or service … its pretty powerful and best of all, the package is in Centos by default. 🙂

my "work" servers send their log files to the central Rsyslog server, while keeping a local log file also.

my “work” servers send their log files to the central Rsyslog server, while keeping a local log file also.

Server Setup

This machine is going to be the central logging server. (rsyslog server)

# lets be good to our logging server
yum update -y

# install if not yet here
yum install rsyslog rsyslog-doc

# edit
nano +15 /etc/rsyslog.conf


# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

note : this only enables UDP logging, modload imtcp does TCP. I picked UDP since I don’t care for specific order of the log messages, even if a messages get lost now and again that’s ok. 

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

Reload the service and check if it is listening.

# lets load the new config
systemctl restart rsyslog

# or if you are like me and like the good'ol way 
service rsyslog start

# now lets check if our server is listening to the port
netstat -anup | grep 514
udp        0      0   *                           27285/rsyslogd      
udp6       0      0 :::514                  :::*                                27285/rsyslogd     

# or if you also enabled tcp 
netstat -antup | grep 514

Also open ports on the firewall! Example with iptables (restricting on INPUT) :

# don't forget to allow this info in firewall
# example for iptables on UDP
iptables -I INPUT -p udp --dport 514 -j ACCEPT

# for tcp
iptables -I INPUT -p tcp --dport 514 -j ACCEPT

You’re server is ready to start logging more servers, onto the client (server 2) !

Client Setup

# again be good sysadmin
yum update -y

# install if not there
yum install rsyslog rsyslog-doc

# edit the config
nano +92 /etc/rsyslog.conf

# add, this would log everything
# possible you would wanne restrict this a bit 
# see man rsyslog.conf
*.* @SERVER_IP:514

Testing the client setup :

# (optional) check connection
# with tcp this could be used
telnet SERVER_IP 514
Trying SERVER_IP ...
Connected to SERVER_IP.
Escape character is '^]'.

# with udp client side (package used : nmap-ncat, tcpdump)
# on server
tcpdump 'port 514'

# on client
nc -u SERVER_IP 514
some message
some more messages

# After this restart the logger
systemctl restart rsyslog
# or <3
service rsyslog restart


on server :

tail -f /var/log/messages

on client :

logger -t sysadmin good job kid

Thats it, so simple, why did I not find this faster ? Good luck with the logging!

Some more resources :

I updated this guide due to the public beta, as well as better support for python 2.6. See the new post.

I’m a huge fan of Let’s Encrypt, generally https is safer then http why ? Well when you send data (login data) over http its child play to read out your password and login name. With https a hacker would only see jumble. Until recently only a few free options where available to the webmaster/sysadmin/devops/… first was self signed certificates, these give you a horrible warning that the website is insecure, ironically its much saver then http. The second option was to use free certificate providers, such as sadly it takes allot of work and even if you are used to their workflow it takes some time to redo them every year. Let’s Encrypt is to be the game changer in the field, they will deliver free certificates with only a few commands, on top of that they focus on automating the proces,they also deliver a certificate that is trusted by browsers!

So where is the catch ? Well there is not really one, expect they work with open-source and community driven development, which means, not everything is available when they are going to launch. Such as support for Centos 6.X (due to the python 2.7 requirement!)

It is however rather easy to install python 2.7 on Centos! Not even other repo’s are required!

yum install centos-release-SCL && yum update

# install python 2.7
yum install python27

# activate it
scl enable python27 bash

# install other python dependencies
yum install python27-python-devel python27-python-setuptools python27-python-tools python27-python-virtualenv

# these would be installed automaticly by the client but I prefer to do it myself
yum install augeas-libs dialog gcc libffi-devel openssl-devel python-devel

After python 2.7 is installed you are ready to follow up with let’s encrypt default tool :

git clone
cd letsencrypt
./letsencrypt-auto --verbose

# during beta
./letsencrypt-auto --verbose --agree-dev-preview --server certonly

Let’s encrypt the web!

update: Seems let’s encrypt is working on support for python2.6 (and centos as a result)
update 9 nov. :
1) updated article based on experience during beta of lets encrypt.
2) this method only works on 64bit machines, since SCL is only available for 64bit os
3) public beta has been pushed back to 3 december. (source)

mariadb on centos 7

23 October, 2015

I’m just not very happy with Centos 7, things change and I can’t handle change. Then again, most changes probably have a good reason, and its all good fun for the brain. Keeping us all awake. Still i’d like to know why eth[0-9]* was a bad name ? Why do we need enp8s0f[0-9]* ?  Well anyways, another change -probably for the better- is that MySQL has been swapped with MariaDB, to be honest I have been living under a rock for the last few years, so I had no idea why or what MariaDB was.

Good thing the folks at digitalocean know what’s going on :

MariaDB is an open source fork of MySQL developed and worked on by the original MySQL developers, lead by Michael “Monty” Widenius. It was created and embraced by the open source community as an effective alternative to MySQL. Although MySQL is still an open source project, it is owned by Oracle, purveyors of their own enterprise software. Worries about the progress of MySQL as well as the status of MySQL as an open source project have prompted the migration to MariaDB. A bug snafu that removed the GPL license from MySQL’s man page caused additional consternation in the open source community as it seemed to restrict the replication of the man pages. Oracle quickly reported this issue as a bug and corrected the copyright notice. Nonetheless, this event was another reminder of how quickly corporate policies could affect the MySQL community.

When summarizing Wikipedia’s move to MariaDB, the announcer Asher Feldman, included a hint as to why he switched: “…as supporters of the free culture movement, the Wikimedia Foundation strongly prefers free software projects; that includes a preference for projects without bifurcated code bases between differently licensed free and enterprise editions.” Oracle’s different treatment of the enterprise and community versions is another factor that caused a stir. At this point, the future of MySQL, subject to decisions of Oracle, remains murky.

So generally its the latest open-source MySQL, great, lets install it on Centos 7

yum install mariadb mariadb-server

Its nothing special, so lets fire it up :

systemctl start mariadb

# or (not supported)
service mariadb start

sadly fresh from the repo, this failed 🙁 The error log is located : /var/log/mariadb/mariadb.log. It contained a weird error on a fresh install :

[ERROR] mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)

Error 13 is mostly used around permission errors, and in this case it was the same.  cat /etc/my.cnf | grep datadir will tell you where the datadir is; this is the directory with most of the files mysql mariadb uses. For me /var/lib/mysql/mysql, /var/lib/mysql/performance_schema and /var/lib/mysql/test where all owned by root. Changing those to mysql ownership fixed this problem 🙂

# give good permissions
chown -R mysql.mysql /var/lib/mysql/mysql
chown -R mysql.mysql /var/lib/mysql/performance_schema
chown -R mysql.mysql /var/lib/mysql/test

# restart it
service mariadb start

centos 7 from usb with YUMI

23 October, 2015

This happened while installing Centos 7 from usb, with YUMI : (just after mounted configuration file system)

dracut-initqueue[685]: Warning: /dev/root does not exist

It will stop there, and you are pretty much at a dead end; The error happens, cause there are multiple ways to install it in the iso and they break each other … weird stuff.

Anyways, while this guy seemed to have found a work around, I remade the stick using good’ol fashion data destruct-or better known as dd.

dd bs=4M if=CentOS-7-x86_64-Minimal-1503-01.iso of=/dev/sdd

note : sdd is my USB stick, don’t try it with sda, or do and see the world burn.

Now my server runs as a baby. (it’s also as slow as a baby)

On the public site, allot of errors where being logged in /var/log/messages, but no ip’s showed up on

iptables -L

meaning there where no active bans, however fail2ban reported (in /var/log/messages)

fail2ban.actions[]: NOTICE [ssh-iptables] 43.229.**.** already banned

Meaning it found an attacker, that it knows about (in its database/logging mechanism) but still kept on hitting the server. After some digging I found this rather large and verbose error :

fail2ban.action[]: ERROR iptables -w -D INPUT -p tcp --dport ssh -j f2b-SSH#012iptables -w -F f2b-SSH#012iptables -w -X f2b-SSH -- stderr: "iptables v1.4.7: option `-w' requires an argument\nTry `iptables -h' or 'iptables --help' for more information.\niptables v1.4.7: option `-w' requires an argument\nTry `iptables -h' or 'iptables --help' for more information.\niptables v1.4.7: option `-w' requires an argument\nTry `iptables -h' or 'iptables --help' for more information.\n"

After some debugging of the jail.local, I found nothing out of it, and googling fail2ban iptables did not bring much to the table, until at page two of google I found this Russian support forum telling me that the -w option should be removed, while I believe it, my iptables (1.4.7, 1.el6) has -w option.

iptables v1.4.7: option `-w' requires an argument

In the end, I found the original enhancement request, and fixing it is easy :

find in /etc/fail2ban/action.d/iptables-common.conf

# Option:  lockingopt
# Notes.:  Option was introduced to iptables to prevent multiple instances from
#          running concurrently and causing irratic behavior.  -w was introduced
#          in iptables 1.4.20, so might be absent on older systems
#          See
# Values:  STRING
lockingopt = -w

Replace it with

# Option:  lockingopt
# Notes.:  Option was introduced to iptables to prevent multiple instances from
#          running concurrently and causing irratic behavior.  -w was introduced
#          in iptables 1.4.20, so might be absent on older systems
#          See
# Values:  STRING
lockingopt =

Reloading fail2ban (service fail2ban restart) will get the bad boys banned in no time again !