We bought our first 10GBase-T switch. Pretty nice, but setting it up proved a bit more complex then our SuperMicro switches. Here I documented my search to get servers to talk to each other over a VLAN using the switch to assign IP addresses, using DHCP.

Login & Connect

Logging in proved rather easy and was similar to SuperMicro’s, connect “through the back” using a RJ45 alike cable that is actually a serial cable to USB. (I don’t think this was included, but there alternatives) Once connected I could find the “COM” port windows assigned checking Bluetooth devices :

windows screen of bluetooth & other devices

One could also open the device manager and find it under COM ports. I used putty as my VT100 terminal emulator, on any Linux distro there are plenty of options but the easiest option would be screen. The setting of this switch are : 115200 baud speed, 8 data bits, 1 stop bits, none parity, none flow control.

Login & connect option 2

Provided by Netgear is a USB to UART, you need this driver (on Win 10) to connect; you still need the VT100 terminal emulator, so that’s the same 🙂

Connect and enter twice and voila you can see the cli interface, the default login is “admin” and no password.

To get the assigned IP address :

# set ip
ip management vlan 1 8.8.8.8

# get ip
show ip management

# or 
show ip vlan

After you get the IP address or guess it, you can go to the more simple web-interface configuration option. In my case the IP was 169.254.100.100. to be greeted with a login page.

Make VLAN

First I needed to make a VLAN, Switching -> VLAN -> basic -> VLAN Configuration. Giving a VLAN ID and a name (disable static).  If everything works it should look like :

vlan configuration m4300

After this, you need to define the VLAN membership this can be done in Switching -> VLAN -> Advanced -> VLAN Membership. Go to VLAN ID you just made (in my case 400) and select group operation tag all then untag all, so that all ports show as untagged. (U).

untag VLAN 400 members

From my basic understanding of VLAN’s and tagged packets, I figured that a port can only be part of one untagged VLAN. I’m probably wrong, but this “worked”. So I made all my ports that I intend to use for data to this new VLAN. Switching -> VLAN -> Advanced -> Port PVID Configuration. I changed VLAN member for all except my management port I’m using (1/0/1). Do this by entering the VLAN ID into VLAN Member and accepting the changes. It should look like :

port VLAN PVID configuration

Now for some reason, ports are still not able to communicate. Time to add a routing interface.

Routing Interface VLAN

note : changed from 1 to 254.

This is totally new to me so, bare along : Routing -> VLAN -> VLAN Routing Configuration; We need to add routing configuration, from the dropdown select our PVID (400) and create a IP address for the routing (end with 254) I decided on 10.0.0.254  for the IP and subnet mask 255.255.255.0 so eventually IP’s allowed will be 10.0.0.[0-255].

It’s important to remember this IP adres, as we will re-use it once we setup the DHCP.

Setup DHCP service

Seamless next step, would you not say ? We need to setup the DHCP, which can be found : System -> Services -> DHCP Server -> DHCP Server Configuration.

We need to set the Admin Mode to enabled, and exclude our routing interface, so that this is not given out by the DHCP. It should look like :

After this we are ready to create the DHCP pool, do this under System -> Services -> DHCP Server -> DHCP Pool Configuration.

Pool name : (random) Type of Binding : Dynamic. Network Addressshould be the one we picked for the routing interface and excluded in the previous step. But instead of 1 use 0. In our case Network Address is 10.0.0.0 and the netmask is 255.255.255.0 or length 24. I setup lease time, to 7 days, which is a weekly thing. Should end up looking like :

After this the only thing remaining is testing & saving the configuration. If you are happy with how it works save your configuration.  Maintenance -> Save Configuration

 

 

useful resource : https://kb.netgear.com/30818/How-to-configure-routing-VLANs-on-a-NETGEAR-managed-switch-with-shared-internet-access

While updating our network on the Rocks Cluster, the nodes had to reinstall (this is default protocol). Now however the nodes got stuck during PXE (over the net automatic installation) on the setting of the language.

install language Centos

a screen like this (source).

This is annoying as it would mean connect a screen and a keyboard to every node to install. This however is an indication that something is wrong all together, however finding what proved a little bit tricky, that’s why I share it here.

To find if you face the same issue, execute :

sudo -u apache /opt/rocks/bin/rocks list host profile compute-0-0

This will show the configuration the node pulled from the head-node.  In my case it looked like :

Traceback (most recent call last):
  File "/opt/rocks/bin/rocks", line 301, in <module>
    command.runWrapper(name, args[i:])
  File 
"/opt/rocks/lib/python2.6/site-packages/rocks/commands/__init__.py", 
line 2194, in runWrapper
    self.run(self._params, self._args)
  File 
"/opt/rocks/lib/python2.6/site-packages/rocks/commands/list/host/profile/__init__.py", 
line 301, in run
    for host in self.getHostnames(args):
  File 
"/opt/rocks/lib/python2.6/site-packages/rocks/commands/__init__.py", 
line 773, in getHostnames
    min,max = self.db.fetchone()
TypeError: 'NoneType' object is not iterable

While this should be a large XML file structure like. After allot of extensive google skills (and this 2013 topic) I found out that a simple MySQL update had dropped the root password out of the global configuration, this can be found :

/opt/rocks/etc/my.cnf

a update, generally saves it here :

/opt/rocks/etc/my.cnf.rpmsave

it should look like this :

[mysqld]
user            = rocksdb
port            = 40000
socket          = /var/opt/rocks/mysql/mysql.sock
datadir         = /var/opt/rocks/mysql

[client]
user            = rocksdb
port            = 40000
socket          = /var/opt/rocks/mysql/mysql.sock
password        = <password>

You don’t have to restart the MySQL or the service, just let the node reboot and it will install properly 🙂

Good luck

Rocks distro is a cluster system. It comes with SNMP configured out of the box. It is polled using Ganglia. Which is working nicely, but I like to have all SNMP data in my favorite SNMP system, LibreNMS. Changing the SNMP configuration to be able to poll from LibreNMS should be a rather straight forward process, however those nodes have no connection to the public network. They have a private VLAN to talk to the head-node and a private VLAN to communicate with the storage array. So to get the SNMP data to Librenms we will have to get crafty with Iptables to get this data to LibreNMS on the public net.

forward snmp from 161 to 3161

Configuration

First let’s check the /etc/snmp/snmpd.conf file from the Rocks installation :

com2sec        notConfigUser   default         public
group  notConfigGroup  v1              notConfigUser
group  notConfigGroup  v2c             notConfigUser
view    all             included        .1             80

access  notConfigGroup  "" any noauth exact all all all

This config is a bit complex and I figure I won’t go back, so I commented it. I decided not to remove it completely, since I don’t want to break the possibility to go back to Ganglia should it be an important system in Rocks. (note : it’s not) I added :

# this create a  SNMPv1/SNMPv2c community named "my_servers"
# and restricts access to LAN adresses 192.168.0.0/16 (last two 0's are ranges)
rocommunity my_servers <our_public_net>/16
rocommunity my_servers 10.1.0.0/16

# setup info
syscontact  "svennD"

# open up
agentAddress  udp:161

# run as
agentuser  root

# dont log connection from UDP:
dontLogTCPWrappersConnects yes

Important here is, that I added two IP ranges, I’m not sure if the private VLAN (10.1.0.0/16) is even required, but since traffic is going over those devices I just added it.

Next thing is setting up the Iptables on the head-node. Since Rocks is already pretty protective (good !) I had to add an extra rule to even allow SNMP polling from the device :

-A INPUT -p udp -m udp --dport 161 -j ACCEPT

Allow the head-node to be polled by LibreNMS by accepting incoming UDP packets over port 161.

To receive packets send from the node on port 161 to the head-node, but forward this to port 3161 externally to LibreNMS (circumventing most known ports and the REJECT rule in Rocks for port 1-1023.) can be done with prerouting rule :

-A PREROUTING -i eth1 -p udp -m udp --dport 3161 -j DNAT --to-destination 10.1.255.244:161
-A POSTROUTING -o eth1 -j MASQUERADE

Note : 10.1.255.244 is the private IP of the node.

So from now on packets should come in, this can be checked using tcpdump, which came in handy during the debugging of this project : (on the node)

tcpdump 'port 161'

To be able to let snmpd answer we needed the information to be forwarded on the head-node, this can be done with a forward rule :

-A FORWARD -d 10.1.255.244/32 -p udp -m udp --dport 161 -m state --state NEW,RELATED,ESTABLISHED -j

note : again 10.1.255.244 is the ip of the node.

Surprisingly the node was unable to answer to the incoming requests. This was due to the fact that, the default route (route -n) was pointing towards one of the storage servers. To add a default gateway we can add it using route :

route add default gw 10.1.1.1 eth0

note : 10.1.1.1 is the private VLAN ip of the head-node.

Conclusion

Bam! LibreNMS can talk to the node using the public IP of the head-node on port 3161 and to the head-node on port 161. One issue that remains unsolved is on reboot, this setup is lost. Rocks by default will reinstall nodes that reboot. This can be resolved by adapting the configurations on Rocks and rebuild the distribution (rocks create distro). However this is rather advanced and (IMHO) difficult to debug. So I did not use that system for this project. Another problem is that its rather work-intense to add all the configuration to all the nodes. (this is only for a single node) This can be resolved most easily using scripts and using rocks run host to execute bits on all the nodes. I decided that I only want one node to be polled as a sample. I already track opengridscheduler using an extend on the head-node. So this is mostly for debugging. Good luck !

Oh noes, while starting a container, it failed cause of a corrupt quota file…

Starting container ...
vzquota : (error) quota file is corrupted
vzquota on failed [4]
TASK ERROR: command 'vzctl start 114' failed: exit code 60

The fix is to remove it and rebuild it (this takes a long time, depending on the size of the container)

# this failed
vzquota off 114

# delete the quota
quota drop 114

We can rebuild the quota using vzquota but that is done automatically if you restart the container :

# vzctl start 114
Starting container ...
Initializing quota ...

The vzquota currently running is : (for reference)

/usr/sbin/vzquota init 114 -b 8388608100 -B 9227468900 -i 1600000100 -I 1760000100 -p /data/private/114 -e 0 -n 0 -s 0

Sit back and relax, cause this will take time.

Got this error :

mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

You forgot the rpcbind service !

yum install rpcbind
service rpcbind start
chkconfig rpcbind on

Glad to help 😉

Mount NFS on LXC Proxmox

8 August, 2018

I’m a long time user of Proxmox (a few years), and recently I had the chance to upgrade an by-now ancient Proxmox 3.4 to current 5.2. In that time frame the developers have changed from OpenVZ to LXC and made a script to migrate the data. One key element however, mounting (remote) NFS shares are no longer possible from within the containers, at least not native.

Within the container the error is rather lacking information and is pointing towards the NFS server issue.

Aug  8 09:09:51 svennd mount: mount.nfs: access denied by server while mounting nfs_server:/data

However, on the Proxmox host, in /var/log/messages you can find that apparmor is the problem.

apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/storage/nfs_server/data/" pid=25086 comm="mount.nfs" fstype="nfs" srcname="nfs_server:/data" flags="ro, noatime"

It seems this is a feature .

Well now, lets try and undo this security feature, in my case the profile that is causing it to block is lxc-container-default-cgns. You can find this file : /etc/apparmor.d/lxc/lxc-default-cgns Also some other configs can be found there (not sure when what profile is loaded) I added :

mount fstype=rpc_pipefs,
mount fstype=nfs,

below mount fstype=cgroup -> /sys/fs/cgroup/**, resulting in this final file :

# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-cgns flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=cgroup -> /sys/fs/cgroup/**,
  mount fstype=rpc_pipefs,
  mount fstype=nfs,
}

After that we need to reload Apparmor, I’m not sure what made it work again, but it was one of these :

apparmor_parser -r /etc/apparmor.d/lxc-containers
systemctl apparmor reload

And now we can mount from within once more ! 🙂

There is an alternative, but from what I read here you need to remap user ID’s, and need to use mountpoints on the host and draw them inside the container.

A small list for important Proxmox files & directories.

For LXC (Linux containers) :

config: /var/lib/lxc/$ct_id/config

For KVM (Kernel-based Virtual Machine, mostly Windows machines)

config: /etc/pve/qemu-server/$kvm_id.conf

Cluster files (if you use Proxmox cluster)

per node configuration : /etc/pve/nodes/$node_name/*

Location of the ISO files that can then be loaded from the web interface :

/var/lib/vz/template/iso/

In order to increase the size of files you can upload in WordPress, we need to adapt the software powering the software.

Let’s begin with PHP, I use Nginx + php-fpm to run PHP, so I need to edit

/etc/php.ini

on line 799, upload_max_filesize determines the maximum allowed size a PHP file will accept for upload.

upload_max_filesize = 2M

This should not be to big, but a few megabytes will work just fine. I increased this to 10M (~10MB)

After that its time to update Nginx to allow larger upload body’s, this I already had an error documented in the past. So in short :

/etc/nginx/nginx.conf

add after the

http {

# increase upload size to 10MB
client_max_body_size 10M;

Once that is done, just restart Nginx and php-fpm.

systemctl restart nginx
systemctl restart php-fpm

And voila

Some powerful and or useful find commands; As with anything on the web, test before running ! (I forget the syntax so this is mainly self-documenting)

 

 

Find all *.tsv larger 1Mb, compress them with the super fast lz4 on high compression and remove the source file after this.

find . -name "*.tsv" -size +1M -exec lz4 -9 --rm '{}' \;

In the same line; compress all files ending with *.fastq and gzip them, also they cannot end with *.gz (in this case redundant but its an extra safety)

find . -type f -name "*.fastq" ! -name '*.gz' -exec gzip "{}" \;

 

Recursive remove all directory’s matching the name *.tsv.index in a rm or echo single command. This makes it possible to easily swap out rm for echo as a test.

find . -type d -name "*.tsv.index" -exec echo {} +
find . -type d -name "*.tsv.index" -exec rm -rf {} +

A combination of a few commands, calculate the storage use from all files size larger then 1M, with no hardlinks, ending with *.tsv.

find . -name "*.tsv" -size +1M -links 1 -print0 | du -hc | tail -n 1

(edit: might not work as intended)

find . -name "log_jobs" -exec du -hc {} +

 

Find files, that are newer then 5 minutes :

find . -type f -mmin -5

and older :

find . -type f -mmin +5

Hard links are nice, but also a (enter curse-word) to track, luckily we have find to locate it :

find /data -samefile file.txt -xdev

This would find all the files that are exactly the same as file.txt (so only hard links, no soft links or copy’s) considering hard links can only be in one file system its logical to add -xdev which tells find not to enter other file-systems since hard links can not be across file-systems. If you are also looking for soft links remove -xdev and add -L

Generate a md5sum for every file in this current directory except files “mylog.log” and “md5.lst”.

find . -type f ! -name "mylog.log" ! -name "md5.lst" -exec md5sum "{}" + > md5.lst

 

A quick and dirty way to find directories (=experiments) that have been made in the last 90 days, sorted on date (removing hard linked .save dirs) This is a sort.

find . -maxdepth 1 -name "*_machine_ID_*" -type d -ctime -90 | grep -v .save | sort -t_ -k 2

 

Ignore certain files, can be done using ! -name “*file” for example. This finds all directories starting with 17, and not ending with .save (hard link for us) and shows the size of those directories.

find . -maxdepth 1 -name "17*" ! -name "*.save" -type d -exec du -hs '{}' +

 

Count certain file type in a single directory (not recursively)

find . -maxdepth 1 -name "*.fastq" | wc -l

 

I could not remove this folder in Windows. It gave an error that is was not there anymore … uh-oh … corruption ? I checked the disk using the windows utility but the disk seemed fine (raid was a-OK).

To my surprise the only solution was to go to the windows console / command prompt and remove the directory directly from there. So Windows is becoming more like Linux, I like it !

The command to be used is rd :

rd /S \\?\D:\arch_micro_data\svennd\data\2015

rd stands for remove directory, the /S is the flag for recursively remove the tree.

After running this command, and accepting the removal, bloop, directory was gone.

What the \\? means I have not found, but the solution came from spiceworks.