SvennD

Rocks Grid Engine error states

27 June, 2017

I have not found easy explanation on what kind of error states are in Rocks, Grid Engine, so I am collecting them here as I find them.

Show states

First let’s find the overview of the nodes; this can be done using qstaf -f

qstat -f

Result should be something like :

# qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
----------------------------------------------------------------
all.q@compute-0-0.local        BIP   0/0/24         0.00     linux-x64 
----------------------------------------------------------------
all.q@compute-0-1.local        BIP   0/0/24         0.00     linux-x64
----------------------------------------------------------------
all.q@compute-0-2.local        BIP   0/0/24         0.02     linux-x64
----------------------------------------------------------------
all.q@compute-0-3.local        BIP   0/0/24         0.00     linux-x64 
----------------------------------------------------------------
all.q@compute-0-4.local        BIP   0/0/24         0.00     linux-x64
----------------------------------------------------------------
all.q@compute-0-5.local        BIP   0/0/24         0.00     linux-x64    
----------------------------------------------------------------
Error state : E

E stands for (hard) error, which means something bad is going on. The result is a decision by the headnode to not use this node anymore until manual intervention. This happens, to make sure there is not a job sinkhole created.

Example :

qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------
all.q@compute-0-0.local        BIP   0/0/24         0.01     linux-x64     E
---------------------------------------------------------------
all.q@compute-0-1.local        BIP   0/0/24         0.00     linux-x64
---------------------------------------------------------------
all.q@compute-0-2.local        BIP   0/0/24         0.02     linux-x64
---------------------------------------------------------------
all.q@compute-0-3.local        BIP   0/0/24         0.00     linux-x64     E
---------------------------------------------------------------
all.q@compute-0-4.local        BIP   0/0/24         0.00     linux-x64
---------------------------------------------------------------
all.q@compute-0-5.local        BIP   0/0/24         0.00     linux-x64     E
---------------------------------------------------------------

The error states here where probably due to a full root disk on these nodes. There is a tool for finding out which jobs failed to find out what was happening at the time (-explain E)

qstat -f -explain E
queuename                      qtype resv/used/tot. load_avg arch          states
-------------------------------------------------------------
all.q@compute-0-0.local        BIP   0/0/24         0.00     linux-x64     E
        queue all.q marked QERROR as result of job 542032's failure at host compute-0-0.local
        queue all.q marked QERROR as result of job 542033's failure at host compute-0-0.local
        queue all.q marked QERROR as result of job 542036's failure at host compute-0-0.local

What is more important, is the fact that Error state will survive a reboot. So we should clean it up if the underlying issue has been resolved : (this will clear all errors)

qmod -c '*'

source

Disabled state : d

d means the node has been disabled, this normally should not happen automatically. We can disable a node from getting anymore jobs, but the running jobs will continue to run.

You can disable a node from further jobs using the qmod command.

qmod -d all.q@compute-0-5.local

You can re-enable a node again using

qmod -e all.q@compute-0-5.local

Example :

[root@server ~]# qmod -d all.q@compute-0-5.local
root@server.local changed state of "all.q@compute-0-5.local" (disabled)
[root@server ~]# qmod -e all.q@compute-0-5.local
root@server.local changed state of "all.q@compute-0-5.local" (enabled)

 

au : Alarm, Unreachable

The state au, u means unreachable this happens when sge_execd on the node does not respond to the sge_qmaster on the headnode within a configured timeout window. The a state is alarm, this will happen when the node does not report the load, in which case a load of 99.99 is assumed. This results in the scheduler to not assign more work to the node. The au state can happen when a NFS server is being hammered and the complete node is waiting for the “slow” network disk.  (when hard mounted nfs) This state can resolve itself if the problem gets resolved.

source

[root@server ~]# qstat -f
queuename                      qtype resv/used/tot. load_avg arch          states
---------------------------------------------------------------------------------
all.q@compute-0-0.local        BIP   0/0/24         0.00     linux-x64     d
---------------------------------------------------------------------------------
all.q@compute-0-1.local        BIP   0/3/24         3.25     linux-x64     d
---------------------------------------------------------------------------------
all.q@compute-0-2.local        BIP   0/6/24         6.32     linux-x64     d
---------------------------------------------------------------------------------
all.q@compute-0-3.local        BIP   0/0/24         0.00     linux-x64     dE
---------------------------------------------------------------------------------
all.q@compute-0-4.local        BIP   0/5/24         5.28     linux-x64     d
---------------------------------------------------------------------------------
all.q@compute-0-5.local        BIP   0/0/24         0.00     linux-x64     dE
---------------------------------------------------------------------------------
all.q@compute-0-7.local        BIP   0/0/24         -NA-     linux-x64     adu

Useful link :

Linux, SysAdmin

Enhance backup schedule for Bareos

21 May, 2017

In the previous article on Bareos, we setup a quick and dirty backup job to run every night. This was pretty easy, but it has some flaws. (1) the first flaws, -after a full backup- only increment backups are created, forever. This makes it difficult to get a restore going down the line, as all the incremental backups need to poke the initial full backup before you can start recovering. (2) A second flaw, we ran backups but never checked if we can restore them. We need to take into account the Schrodinger’s backup : “The condition of any backup is unknown until a restore is attempted”. Perhaps a bit out of this scope but (3) we did not look where the backups are stored, and how they are being stored. There are plenty of options in Bareos, so let’s take a look and fine tune the backup setup.

Read More

SysAdmin

screen : Cannot open your terminal ‘/dev/pts/0’ – please check.

13 May, 2017

I hit upon this error :

Cannot open your terminal '/dev/pts/0' - please check.

after being logged in under root and su to a user profile;

su user

and then willing to resume a screen :

# screen -r copy.pid
Cannot open your terminal '/dev/pts/3' - please check.

The issue is resolved using :

script /dev/null

I previously had similar issues, but this was using lxc-attach. See this previous post.

Linux, SysAdmin

Error contacting apcupsd @ localhost:3551: Connection refused

11 May, 2017

When installing apcupsd on proxmox (similar to centos) I received this error :

Error contacting apcupsd @ localhost:3551: Connection refused

When running apcaccess. I got more info on the error using checking the service status :

root@rocky:~# service apcupsd status
● apcupsd.service - LSB: Starts apcupsd daemon
   Loaded: loaded (/etc/init.d/apcupsd)
   Active: active (exited) since Thu 2017-05-11 15:46:28 CEST; 6s ago
  Process: 11837 ExecStop=/etc/init.d/apcupsd stop (code=exited, status=0/SUCCESS)
  Process: 11841 ExecStart=/etc/init.d/apcupsd start (code=exited, status=0/SUCCESS)

May 11 15:46:28 rocky apcupsd[11841]: Please check your configuration ISCONFIGURED in /etc/default/apcupsd
May 11 15:46:28 rocky systemd[1]: Started LSB: Starts apcupsd daemon.

Found it !  In /etc/default/apcupsd there is a fail-safe variable ISCONFIGURED. This protects you against accidentally shutdowns when you are testing/freshly installing.

Just change the no into a yes; resulting in :

# Defaults for apcupsd initscript

# Apcupsd-devel internal configuration
APCACCESS=/sbin/apcaccess
ISCONFIGURED=yes

And restart the service :

service apcupsd restart

Now apcaccess works fine 🙂

SysAdmin

crond: (root) BAD FILE MODE

10 May, 2017

I found these errors today in my the cron log : /var/log/cron

 May 10 09:53:01 sysadmin CROND[2267]: (librenms) CMD ( /opt/librenms/alerts.php >> /dev/null 2>&1) May 10 09:54:01 sysadmin crond[63]: (root) BAD FILE MODE (/etc/cron.d/librenms.cron)

I wrongly assumed a 755 permission was needed for crons, seems I was wrong. For cron execution we only need 644. Surprisingly the issue gets resolved then and cron does execute the cron. I believe this is new behavior in from Centos 7.

Chmod to the solution :

chmod 0644 /etc/cron.d/librenms.cron
Linux, SysAdmin

SNMP OID to Text

4 May, 2017

I have been searching high and low for this, and it’s not even the first time. Since my brain did not store this long term; I’m sharing it here.

When extending snmpd like this : (in /etc/snmp/snmpd.conf)

extend zfs-arcstat /usr/bin/cat /proc/spl/kstat/zfs/arcstats

The output can be queried using :

NET-SNMP-EXTEND-MIB::nsExtendOutputFull."zfs-arcstat"

“zfs-arcstat” is a free to chose name. My search, is how can you represent this free name in a single numeric OID;  This can be done using snmptranslate, with the flags -0n.

Take note that one has to escape the quotes.

# snmptranslate -On NET-SNMP-EXTEND-MIB::nsExtendOutputFull.\"zfs-arcstat\"
.1.3.6.1.4.1.8072.1.3.2.3.1.2.11.122.102.115.45.97.114.99.115.116.97.116

The opposite is also possible, without any flags :

# snmptranslate .1.3.6.1.2.1.1.3.0
SNMPv2-MIB::sysUpTime.0

It was in the snmp docs all along.

Well perhaps writing this post will help me remember;

SysAdmin

ZFS quota

19 April, 2017

ZFS is loaded with useful features, just a while ago I took a peak at setting quotes. Using just a few commands you limit the amount  of storage a certain pool can take.

First let’s check if there is no quota set :

zfs get quota

This would result in something like :

# zfs get quota
NAME                                                       PROPERTY  VALUE  SOURCE
huginn                                                     quota     none   default
huginn/dataset                                           quota     none   default
huginn/dataset@autosnap_2017-04-01_00:00:02_monthly      quota     -      -
huginn/dataset@autosnap_2017-04-10_00:00:01_daily        quota     -      -
[...]
huginn/dataset@autosnap_2017-04-19_14:00:01_hourly       quota     -      -
jbod1                                                      quota     none   default
jbod1/users                                                quota     none   default
jbod1/users/servers                                        quota     -     -

As you see, no quota’s. Now let’s add one :

zfs set quota=5TB jbod1/users/server

The format : zfs set quota=$size $pool

Let’s check again :

[root@huginn HG00731]# zfs get -r quota jbod1/users/wdecoster
NAME                                                       PROPERTY  VALUE  SOURCE
jbod1/users/server                                      quota     5T     local
jbod1/users/server@autosnap_2017-04-01_00:00:02_daily   quota     -      -
jbod1/users/server@autosnap_2017-04-02_00:00:01_daily   quota     -      -
jbod1/users/server@autosnap_2017-04-03_00:00:01_daily   quota     -      -
jbod1/users/server@autosnap_2017-04-03_13:00:01_hourly  quota     -      -
jbod1/users/server@autosnap_2017-04-03_14:00:01_hourly  quota     -      -

Simple enough; now some practical warnings :

  • full is full; you can’t remove, edit or overwrite data once its 100% full. Even tricks like “echo > big_file” won’t work. You need to either increase or drop the quota before its possible
  • You can’t decrease a quota … once its allowed 5TB only destroying it will release the limit.
Linux, SysAdmin, ZFS

nfsd stats explained : /proc/net/rpc/nfsd

17 April, 2017

One of those things I never spend much time on was reading about NFS. There is no need for it, it just kinda works out the box. Most of the time that is. The headache starts when it stops working, or pushes your cluster to a grind. One of the first resources you will find is to check /proc/net/rpc/nfsd. But that does not help you much further, so I recently started monitoring the content. Let’s see what is what.

Read More

Linux, SysAdmin

ZFS replace a broken disk

28 March, 2017

After a reboot, I got greeted with this error :

root@server:~# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 10.1G in 0h26m with 0 errors on Mon Mar 27 16:35:05 2017
config:

        NAME                     STATE     READ WRITE CKSUM
        rpool                    DEGRADED     0     0     0
          mirror-0               DEGRADED     0     0     0
            sda                  ONLINE       0     0     0
            2748060340541772838  UNAVAIL      0     0     0  was /dev/sdb2
            sdc2                 ONLINE       0     0     0
            sdd2                 ONLINE       0     0     0

I replaced the disk, and then forced a replace.

zpool replace -f rpool 2748060340541772838 /dev/sdb

zpool replace [-f] $poolname $old_device $new_device

The status is now :

root@server:~# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool   928G  10.2G   918G     16.0E     3%     1%  1.00x  DEGRADED  -
root@server:~# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Mar 28 11:31:57 2017
    14.3M scanned out of 10.2G at 1.02M/s, 2h50m to go
    14.2M resilvered, 0.14% done
config:

        NAME                       STATE     READ WRITE CKSUM
        rpool                      DEGRADED     0     0     0
          mirror-0                 DEGRADED     0     0     0
            sda                    ONLINE       0     0     0
            replacing-1            UNAVAIL      0     0     0
              2748060340541772838  UNAVAIL      0     0     0  was /dev/sdb2
              sdb                  ONLINE       0     0     0  (resilvering)
            sdc2                   ONLINE       0     0     0
            sdd2                   ONLINE       0     0     0

errors: No known data errors

 

SysAdmin, ZFS

error libwrap missing

22 March, 2017

While compiling bareos I got this error :

checking for socket... yes
using socket from libc
checking for library containing nanosleep... no
checking for libwrap... configure: error: *** libwrap missing

The missing library on Centos is :

tcp_wrappers-devel

Fix :

yum install tcp_wrappers-devel
Linux
Newer Posts
Older Posts