I received this weird error a few days ago:


End of system activity file unexpected

By received I mean received in the /var/spool/mail/root mail :

From root@server  Thu Jan 19 23:53:01 2017
Return-Path: <root@server>
X-Original-To: root
Delivered-To: root@server
Received: by server (Postfix, from userid 0)
        id D352D25133; Thu, 19 Jan 2017 23:53:01 +0100 (CET)
From: root@server (Cron Daemon)
To: root@server
Subject: Cron <root@server> /usr/lib64/sa/sa2 -A
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20170119225301.D352D25133@server>
Date: Thu, 19 Jan 2017 23:53:01 +0100 (CET)

End of system activity file unexpected

What does this error means ? I had no clue.  The first thing I checked was a full disk, nothing popped up for me :

[root@server data]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdw2             9.5G  8.0G  1.5G  85% /
tmpfs                  16G   68K   16G   1% /dev/shm
/dev/sdw1             241G  206M  228G   1% /boot
/dev/sdw5             1.6T   70M  1.5T   1% /home
/dev/sdx1             109T   98T   11T  90% /data
jbod2                  22T  4.7G   22T   1% /jbod2

Nothing weird, at first sight. I checked to see what the /var/log/cron was doing during that time (Thu Jan 19 23:53:01 2017) This is /var/log/cron

Jan 23 23:50:01 localhost CROND[19121]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 23 23:53:01 localhost CROND[19274]: (root) CMD (/usr/lib64/sa/sa2 -A)
Jan 24 00:00:01 localhost CROND[19635]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jan 24 00:01:01 localhost CROND[19689]: (root) CMD (run-parts /etc/cron.hourly)
Jan 24 00:01:01 localhost run-parts(/etc/cron.hourly)[19689]: starting 0anacron
Jan 24 00:01:01 localhost anacron[19700]: Anacron started on 2017-01-24
Jan 24 00:01:01 localhost run-parts(/etc/cron.hourly)[19702]: finished 0anacron
Jan 24 00:01:01 localhost run-parts(/etc/cron.hourly)[19689]: starting rsync_cron.sh
Jan 24 00:01:01 localhost anacron[19700]: Jobs will be executed sequentially
Jan 24 00:01:01 localhost anacron[19700]: Normal exit (0 jobs run)

So what is sa1/sa2 ? The man of sa1/2 to the rescue :

sa2 - Write a daily report in the /var/log/sa directory

Aaha, /root write access; now lets look at the first line in df -h :

/dev/sdw2             9.5G  8.0G  1.5G  85% /

1.5 G free is not that much but enough for a simple log file no ?  The reall mystery was solved when I read back on the /var/spool/mail/root :

From root@server  Thu Jan 19 06:23:15 2017
Return-Path: <root@server>
X-Original-To: root
Delivered-To: root@server
Received: byserver (Postfix, from userid 0)
        id 7692325002; Thu, 19 Jan 2017 06:23:14 +0100 (CET)
From: Anacron <root@server>
To: root@server
Content-Type: text/plain; charset="ANSI_X3.4-1968"
Subject: Anacron job 'cron.daily' on server
Message-Id: <20170119052314.7692325002@server>
Date: Thu, 19 Jan 2017 06:23:14 +0100 (CET)

/etc/cron.daily/mlocate.cron:

/usr/bin/updatedb: I/O error while writing to `/var/lib/mlocate/mlocate.db.E6o5gC': No space left on device

mlocate is running an update on all the filesystem, which on this particular file server is quit extensive. Disabling the cron, and removing the mlocate database freed enough to keep the machine running without this error.

rm -rf /var/lib/mlocate/mlocate.db
sudo chmod -x /etc/cron.daily/mlocate.cron

Note that I don’t use locate (find database backed file search), this process will update the database with all the files nightly. So for me removing it and disabling the cron was the best option. In case you use locate, you could soft-link it to a better location. (ln -s)

 

fix :

rm -rf /var/lib/mlocate/mlocate.db

sudo chmod -x /etc/cron.daily/mlocate.cron

Additional I dropped the amount of kernels that is kept :

yum install yum-utils
package-cleanup --oldkernels --count=2

Then edit /etc/yum.conf

installonly_limit=3

note : always good practice to keep a few older kernels