A little annoying problem, not blocking but easy to solve!

[root@host~]# service httpd configtest
httpd: apr_sockaddr_info_get() failed for host.ext
httpd: Could not reliably determine the server's fully qualified domain name, using for ServerName
Syntax OK

We could simply add it in /etc/hosts

# fix apache warning host.ext

fail2ban add ip manually

9 November, 2015

My love for fail2ban is slowly decreasing, I had a problem a while ago, while fixed, it was not easy to find. (see fail2ban 0.9.9 with centos 6.7 not adding ips to firewall) Today, again, I noticed that my iptables -L was empty, while my /var/log/secure was full (~70k lines, in one day) and fail2ban was doing absolutly nothing … 🙁

I thought my older version of iptables (16.el6, 1.4.7) was the cause again, but it seems its not. Since I can use fail2ban to add ip’s to iptables. I found the command, on the website by the original patch creator 🙂  Great tool if you wanne ban a range quickly!

fail2ban-client set ssh-iptables banip 123.456.789.001


Today I was searching for tools to centralize the logging of some 20 odd Linux servers, while this is no endpoint in my research, I “logged” the method I used to setup my test/demo servers using good old rsyslog.

While there are allot possibilities towards logging, I’d like :

  • local + remote logging
    • It would be nice if I could log into one “remote” machine/service and see the log files of all machines I need to “maintain”. However keeping a local log file is also important since not everybody will have access to the central machine logging server/service.
  • KISS
    • As little as possible dependencies, preferably it should run up from a ancient Centos 5 to a bleeding edie nighly Ubuntu virtual machine.  So preferably a default package in the common linux distro’s.
  • little overhead
    • The overhead for client servers should be as small as possible, since they do have a real job beside logging.

Most of these points are checked off when working with rsyslog, so I took that solution out for a spin. With rsyslog we can filter out some irrelevant messages (like DHCP requests), use different logging servers for different levels/labels or service … its pretty powerful and best of all, the package is in Centos by default. 🙂

my "work" servers send their log files to the central Rsyslog server, while keeping a local log file also.

my “work” servers send their log files to the central Rsyslog server, while keeping a local log file also.

Server Setup

This machine is going to be the central logging server. (rsyslog server)

# lets be good to our logging server
yum update -y

# install if not yet here
yum install rsyslog rsyslog-doc

# edit
nano +15 /etc/rsyslog.conf


# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

note : this only enables UDP logging, modload imtcp does TCP. I picked UDP since I don’t care for specific order of the log messages, even if a messages get lost now and again that’s ok. 

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

Reload the service and check if it is listening.

# lets load the new config
systemctl restart rsyslog

# or if you are like me and like the good'ol way 
service rsyslog start

# now lets check if our server is listening to the port
netstat -anup | grep 514
udp        0      0   *                           27285/rsyslogd      
udp6       0      0 :::514                  :::*                                27285/rsyslogd     

# or if you also enabled tcp 
netstat -antup | grep 514

Also open ports on the firewall! Example with iptables (restricting on INPUT) :

# don't forget to allow this info in firewall
# example for iptables on UDP
iptables -I INPUT -p udp --dport 514 -j ACCEPT

# for tcp
iptables -I INPUT -p tcp --dport 514 -j ACCEPT

You’re server is ready to start logging more servers, onto the client (server 2) !

Client Setup

# again be good sysadmin
yum update -y

# install if not there
yum install rsyslog rsyslog-doc

# edit the config
nano +92 /etc/rsyslog.conf

# add, this would log everything
# possible you would wanne restrict this a bit 
# see man rsyslog.conf http://linux.die.net/man/5/rsyslog.conf
*.* @SERVER_IP:514

Testing the client setup :

# (optional) check connection
# with tcp this could be used
telnet SERVER_IP 514
Trying SERVER_IP ...
Connected to SERVER_IP.
Escape character is '^]'.

# with udp client side (package used : nmap-ncat, tcpdump)
# on server
tcpdump 'port 514'

# on client
nc -u SERVER_IP 514
some message
some more messages

# After this restart the logger
systemctl restart rsyslog
# or <3
service rsyslog restart


on server :

tail -f /var/log/messages

on client :

logger -t sysadmin good job kid

Thats it, so simple, why did I not find this faster ? Good luck with the logging!

Some more resources :

I updated this guide due to the public beta, as well as better support for python 2.6. See the new post.

I’m a huge fan of Let’s Encrypt, generally https is safer then http why ? Well when you send data (login data) over http its child play to read out your password and login name. With https a hacker would only see jumble. Until recently only a few free options where available to the webmaster/sysadmin/devops/… first was self signed certificates, these give you a horrible warning that the website is insecure, ironically its much saver then http. The second option was to use free certificate providers, such as startssl.com sadly it takes allot of work and even if you are used to their workflow it takes some time to redo them every year. Let’s Encrypt is to be the game changer in the field, they will deliver free certificates with only a few commands, on top of that they focus on automating the proces,they also deliver a certificate that is trusted by browsers!

So where is the catch ? Well there is not really one, expect they work with open-source and community driven development, which means, not everything is available when they are going to launch. Such as support for Centos 6.X (due to the python 2.7 requirement!)

It is however rather easy to install python 2.7 on Centos! Not even other repo’s are required!

yum install centos-release-SCL && yum update

# install python 2.7
yum install python27

# activate it
scl enable python27 bash

# install other python dependencies
yum install python27-python-devel python27-python-setuptools python27-python-tools python27-python-virtualenv

# these would be installed automaticly by the client but I prefer to do it myself
yum install augeas-libs dialog gcc libffi-devel openssl-devel python-devel

After python 2.7 is installed you are ready to follow up with let’s encrypt default tool :

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
./letsencrypt-auto --verbose

# during beta
./letsencrypt-auto --verbose --agree-dev-preview --server https://acme-v01.api.letsencrypt.org/directory certonly

Let’s encrypt the web!

update: Seems let’s encrypt is working on support for python2.6 (and centos as a result)
update 9 nov. :
1) updated article based on experience during beta of lets encrypt.
2) this method only works on 64bit machines, since SCL is only available for 64bit os
3) public beta has been pushed back to 3 december. (source)

mariadb on centos 7

23 October, 2015

I’m just not very happy with Centos 7, things change and I can’t handle change. Then again, most changes probably have a good reason, and its all good fun for the brain. Keeping us all awake. Still i’d like to know why eth[0-9]* was a bad name ? Why do we need enp8s0f[0-9]* ?  Well anyways, another change -probably for the better- is that MySQL has been swapped with MariaDB, to be honest I have been living under a rock for the last few years, so I had no idea why or what MariaDB was.

Good thing the folks at digitalocean know what’s going on :

MariaDB is an open source fork of MySQL developed and worked on by the original MySQL developers, lead by Michael “Monty” Widenius. It was created and embraced by the open source community as an effective alternative to MySQL. Although MySQL is still an open source project, it is owned by Oracle, purveyors of their own enterprise software. Worries about the progress of MySQL as well as the status of MySQL as an open source project have prompted the migration to MariaDB. A bug snafu that removed the GPL license from MySQL’s man page caused additional consternation in the open source community as it seemed to restrict the replication of the man pages. Oracle quickly reported this issue as a bug and corrected the copyright notice. Nonetheless, this event was another reminder of how quickly corporate policies could affect the MySQL community.

When summarizing Wikipedia’s move to MariaDB, the announcer Asher Feldman, included a hint as to why he switched: “…as supporters of the free culture movement, the Wikimedia Foundation strongly prefers free software projects; that includes a preference for projects without bifurcated code bases between differently licensed free and enterprise editions.” Oracle’s different treatment of the enterprise and community versions is another factor that caused a stir. At this point, the future of MySQL, subject to decisions of Oracle, remains murky.

So generally its the latest open-source MySQL, great, lets install it on Centos 7

yum install mariadb mariadb-server

Its nothing special, so lets fire it up :

systemctl start mariadb

# or (not supported)
service mariadb start

sadly fresh from the repo, this failed 🙁 The error log is located : /var/log/mariadb/mariadb.log. It contained a weird error on a fresh install :

[ERROR] mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)

Error 13 is mostly used around permission errors, and in this case it was the same.  cat /etc/my.cnf | grep datadir will tell you where the datadir is; this is the directory with most of the files mysql mariadb uses. For me /var/lib/mysql/mysql, /var/lib/mysql/performance_schema and /var/lib/mysql/test where all owned by root. Changing those to mysql ownership fixed this problem 🙂

# give good permissions
chown -R mysql.mysql /var/lib/mysql/mysql
chown -R mysql.mysql /var/lib/mysql/performance_schema
chown -R mysql.mysql /var/lib/mysql/test

# restart it
service mariadb start

centos 7 from usb with YUMI

23 October, 2015

This happened while installing Centos 7 from usb, with YUMI : (just after mounted configuration file system)

dracut-initqueue[685]: Warning: /dev/root does not exist

It will stop there, and you are pretty much at a dead end; The error happens, cause there are multiple ways to install it in the iso and they break each other … weird stuff.

Anyways, while this guy seemed to have found a work around, I remade the stick using good’ol fashion data destruct-or better known as dd.

dd bs=4M if=CentOS-7-x86_64-Minimal-1503-01.iso of=/dev/sdd

note : sdd is my USB stick, don’t try it with sda, or do and see the world burn.

Now my server runs as a baby. (it’s also as slow as a baby)

On the public site, allot of errors where being logged in /var/log/messages, but no ip’s showed up on

iptables -L

meaning there where no active bans, however fail2ban reported (in /var/log/messages)

fail2ban.actions[]: NOTICE [ssh-iptables] 43.229.**.** already banned

Meaning it found an attacker, that it knows about (in its database/logging mechanism) but still kept on hitting the server. After some digging I found this rather large and verbose error :

fail2ban.action[]: ERROR iptables -w -D INPUT -p tcp --dport ssh -j f2b-SSH#012iptables -w -F f2b-SSH#012iptables -w -X f2b-SSH -- stderr: "iptables v1.4.7: option `-w' requires an argument\nTry `iptables -h' or 'iptables --help' for more information.\niptables v1.4.7: option `-w' requires an argument\nTry `iptables -h' or 'iptables --help' for more information.\niptables v1.4.7: option `-w' requires an argument\nTry `iptables -h' or 'iptables --help' for more information.\n"

After some debugging of the jail.local, I found nothing out of it, and googling fail2ban iptables did not bring much to the table, until at page two of google I found this Russian support forum telling me that the -w option should be removed, while I believe it, my iptables (1.4.7, 1.el6) has -w option.

iptables v1.4.7: option `-w' requires an argument

In the end, I found the original enhancement request, and fixing it is easy :

find in /etc/fail2ban/action.d/iptables-common.conf

# Option:  lockingopt
# Notes.:  Option was introduced to iptables to prevent multiple instances from
#          running concurrently and causing irratic behavior.  -w was introduced
#          in iptables 1.4.20, so might be absent on older systems
#          See https://github.com/fail2ban/fail2ban/issues/1122
# Values:  STRING
lockingopt = -w

Replace it with

# Option:  lockingopt
# Notes.:  Option was introduced to iptables to prevent multiple instances from
#          running concurrently and causing irratic behavior.  -w was introduced
#          in iptables 1.4.20, so might be absent on older systems
#          See https://github.com/fail2ban/fail2ban/issues/1122
# Values:  STRING
lockingopt =

Reloading fail2ban (service fail2ban restart) will get the bad boys banned in no time again !

HTTP is broken. It’s time for the distributed, permanent web.

Source: HTTP is obsolete. It’s time for the distributed, permanent web

rpm -qlp fil.rpm

Source: What Files Are In a RPM Package?

Another useful tool  from package yum-utils is repoquery. Even non-installed or downloaded packages can be inspected this way :

# install the tool
yum install yum-utils

# check what is in yum-utils package
repoquery -l yum-utils

# peek inside the nano package
repoquery -l nano

// update 13/01/2016 – I added repoquery, handy tool !

I’m a bit a fan of observium, sadly they switched from “open-source” to a fremium model. The Community version is free, but only gets two updates a year. Its also not visible when a new version is out and the download is “latest” not version/revision …. So you end up gambling when there is an update.  Even within the structure there is no clear version given.

So the only way to know if its a new version is to compare the two directories but guess what, its as easy as :

diff -rq /path/to/observium /path/to/observium_new?

Good little tool. By the way it seems I was up-to-date ! (no new eye-candy)