Slack as alternative to e-mail warnings.

When monitor systems are made, there are only a few options, e-mail, telephone and some constricting alternative nobody ever heard off. Well goodbye to those. E-mails get lost, or in spam, telephones are fine, only Y2K skype called, and they want there platform back. So here goes my next attempt : slack ūüôā LibreNMS¬†already polls all my devices over the network, and can report straight to slack. (and also e-mail and 50 other options) Time to also update my emailing cron jobs to slack’s.

I used slack for bash, written by Sul Aga. Using it is easy :

Install

cd /opt
git clone https://github.com/sulhome/bash-slack
ln -s /opt/bash-slack/postToSlack.sh /usr/local/bin/postToSlack

Use 

postToSlack.sh -t "title of your message" -b "content of the message" -c "serverroom_channel" -u "https://hooks.slack.com/services/..." > /dev/null

The URL can be generated from an incoming webhook. Messages look like this :

webhook using bash

Note that the output will tell you the status code (200: OK) I ignore it, since there is no other way then to … mail on failure ?

how to install SNMP service on Proxmox

In this guide I just install configure SNMP deamon on Proxmox. SNMP is used to read out the state of the machine, this includes CPU, network (-traffic, -settings,…), memory, … I can suggest LibreNMS, for the monitor. Back to setting SNMP deamon¬†up !

Install the service

apt-get install snmpd

Config the service

I like to backup the original configuration (for documentation) :

mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.ori

Then I create the config file : /etc/snmp/snmpd.conf

# this create a  SNMPv1/SNMPv2c community named "my_servers"
# and restricts access to LAN adresses 192.168.0.0/16 (last two 0's are ranges)
rocommunity my_servers 192.168.0.0/16

# setup info
syslocation  "rack 1, room 3, Antwerpen serverrroom"
syscontact  "Svenn"

# open up
agentAddress  udp:161

# run as
agentuser  root

# dont log connection from UDP:
dontLogTCPWrappersConnects yes

If needed add an exception to the firewall :

iptables -A INPUT -s 192.168.0.0/16 -p udp --dport 161 -j ACCEPT

And start the service :

# debian starts services on install
service snmpd restart

And enable by default :

update-rc.d snmpd enable

Test

after that its time to test if the connection works from the server that is going to listen to the snmpd :

nmpwalk -c my_server -v1 servername SNMPv2-MIB::sysDescr.0

This should return something like :

SNMPv2-MIB::sysDescr.0 = STRING: Linux  ...

That’s it, happy tracking !

install supermicro’s IPMI

IPMI is cool, this is my manual to “set it up” on¬†Centos (RHEL) and Debian (Ubuntu, Proxmox, …)

Centos

yum install ipmitool

or on older machines (Centos 5)

yum install OpenIPMI OpenIPMI-toolschkconfig ipmi

Debian

apt-get install openipmi ipmitool

also need to load these modules for some reason :

modprobe ipmi_msghandler
modprobe ipmi_devintf
modprobe ipmi_si

Next I setup LAN setup and change the password, I mostly do that using the web-interface, but its possible to do from the console.

# give information on ipmi (ip, etc.)
ipmitool lan print 1

# create user and set password
ipmitool user set name 2 admin
ipmitool user set password 2

 

full ZFS filesystem : No space left on device

Starting the new year with :

No space left on device

I looked to the usual suspects (/tmp, /data, …) ¬†but nothing excessive … It was my sanoid setup, woops! A¬†gazillion snapshots ! For some reason one of my containers changes around ~300 Gb a day, and guess who keeps 60 days of snapshots, on top of 6 months snapshots ? –> you guessed right.

Its annoying ZFS does not warn you, or that sanoid doesn’t take care, not to fill it up 100% . Not that’s its there task, but still, it was holidays for everyone guys ! So how to fix this ? Simply remove excessive snapshots (those can be deleted)

zfs destroy jbod1/users@autosnap_2016-12-09_14:00:01_hourly

Note that during a 100% full you can’t remove data due to Copy On Write (COW) so rm`ing a big file would result in :

# rm /that/big/file
Can't remove /that/big/file. No space left on the device.

What you can do (if you can’t remove snapshots, or you just filled it up the old fashion way)

echo > /that/big/file

Credits for this great solution go to this guy.

Best wishes for 2017 ūüôā

New theme :)

Almost 2017, so I though its time to browse for another wordpress theme, and you are looking at it ! I liked the previous one, but there was allot of “lost” space in the header, so this one does slightly better. Hope you like it, I do ! ūüôā Thanks go to¬†templateexpress¬†for the previous¬†Discovery theme, now its time for Sparkeling from¬†colorlib, thanks guys/gals !

old layout :

old layour svennd

Shiny new layout :

new layout

mount: unknown filesystem type ‘zfs_member’

After mounting a NTFS partiton in read/write, a NFS partition and just last week a LVM2_member partition. Its time for a new episode in the series: How to mount an unknown filetype in linux. As they say, the saga continues.

Note that I use proxmox (debian spin) on this machines, this makes that ZFS¬†was already installed. Installing ZFS is easy, and I done it before on Mint, the official ZFS (on linux) information is also a huge help, so head over there if you wan’t up-to-date install & configuration help.

Every good story always starts when someone or something f*cks up. So here goes, right in the nuts :

root@svennd:~# mount /dev/sdd3 /mnt/datadisk/
mount: unknown filesystem type 'zfs_member'

That’s fine, its time for our hero to move in and gain information about our target:¬†fdisk -l

root@svennd:~# fdisk -l /dev/sdg

Disk /dev/sdd: 74.5 GiB, 80026361856 bytes, 156301488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D4DA51A9-6084-4E52-ACE7-AD0272064251

Device      Start       End   Sectors  Size Type
/dev/sdd1    2048      4095      2048    1M BIOS boot
/dev/sdd2    4096    266239    262144  128M EFI System
/dev/sdd3  266240 156299439 156033200 74.4G Linux filesystem

The “BIOS boot” and “EFI System” partition are irrelevant to me collecting data, but “Linux filesystem” is where the data is. Sadly this does not give us allot to go on. I was hoping this was part of a ZFS mirror. If it where a RAIDZ[1-3] a single disk is not going to cough up its data to easy. ¬†ZFS comes with allot of handy tools, one is actually scanning storage devices for ZFS partitions that aren’t yet active on the device and contrary to other RAID systems, ZFS is always in a state where it can be transferred between machines. So I tried to run zpool import :

root@svennd:~# zpool import
 pool: rpool
    id: 1395200144405345736
 state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices.  The
       fault tolerance of the pool may be compromised if imported.
  see: http://zfsonlinux.org/msg/ZFS-8000-4J
config:

       rpool                    DEGRADED
         mirror-0               DEGRADED
           sdd3                 ONLINE
           4085699320674626661  UNAVAIL

Bam ! It found the partition on the disk! (/dev/sdd3) This also tells me that it is in fact a ZFS mirror setup. Now once we found it, we can simply import it, this is done with zpool import <name_dataset>

root@svennd:~# zpool import rpool
cannot import 'rpool': a pool with that name already exists
use the form 'zpool import <pool | id> <newpool>' to give it a new name

Woops. Good thing, ZFS helps us out here :

root@svennd:~# zpool import rpool temp_rpool
cannot import 'rpool': pool may be in use from other system, it was last accessed by (none) (hostid: 0xa8c00802) on Tue Mar 31 15:58:04 2015
use '-f' to import anyway

Oke, ZFS likes -f, so lets add it ; zpool import -f poolname new_poolname

root@svennd:~# zpool import -f rpool temp_rpool
cannot mount '/': directory is not empty

The error on there means this was mounted on / (root) before, this obv. is not empty so it does not mount it. To change that first look up what the current mountpoint is, and change that, this can all be done using zfs get and zfs set.

root@svennd:~# zfs get all temp_rpool |grep mountpoint
temp_rpool  mountpoint            /                   default
root@svennd:~# zfs set mountpoint=/mnt/datadisk temp_rpool

Once that’s done you can mount the ZFS pool as using zfs mount.

zfs mount temp_rpool

Another “un-mountable” partition beats the dust. Good luck!

mount: unknown filesystem type LVM2_member

So I’m planning on making a series, “mount: unknown filesystem type $TYPE”. I already have how to mount a ntfs partition, also how to mount a nfs¬†on proxmox, now to be continued by another fun filesystem. I was going through old disks, so I came across one that had LVM2_member.

root@svennd:~# mount /dev/sdd2 /mnt/disk
mount: unknown filesystem type 'LVM2_member'

The fdisk -l already told me its a LVM :

root@svennd:~# fdisk -l /dev/sdd

Disk /dev/sdd: 233.8 GiB, 251000193024 bytes, 490234752 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0009345d

Device     Boot  Start       End   Sectors   Size Id Type
/dev/sdd1  *        63    208844    208782   102M 83 Linux
/dev/sdd2       208845 488247479 488038635 232.7G 8e Linux LVM

 (/dev/sdi1 is /boot partition, /dev/sdi2 is where the /home data resides)

Seems lvm2 tools also provide a way to check if its lvm or not, using lvmdiskscan (/dev/sdd2 here)

root@svennd:~# lvmdiskscan
  /dev/sdb1  [       1.82 TiB]
  /dev/sdc2  [     149.04 GiB]
  /dev/sdd1  [     101.94 MiB]
  /dev/sdd2  [     232.71 GiB] LVM physical volume
  0 disks
  4 partitions
  0 LVM physical volume whole disks
  1 LVM physical volume

Fine, now lets scan what lv’s are to be found using¬†lvscan

root@svennd:~# lvscan
 inactive '/dev/VolGroup00/LogVol00' [230.75 GiB] inherit
 inactive '/dev/VolGroup00/LogVol01' [1.94 GiB] inherit

Since this is a old disk in an enclosure, its not activated on system boot. So we need to “activate” this lvm volume.

root@svennd:~# vgchange -ay
 2 logical volume(s) in volume group "VolGroup00" now active

and bam, ready to mount :

root@svennd:~# lvscan
  ACTIVE            '/dev/VolGroup00/LogVol00' [230.75 GiB] inherit
  ACTIVE            '/dev/VolGroup00/LogVol01' [1.94 GiB] inherit

now to mount :

mount /dev/VolGroup00/LogVol00 /mnt/disk

succes !