Set IPMI to use DHCP

4 December, 2019

If you installed ipmitools, you can change the IP address source from static to DHCP or otherwise.

To change to DHCP :

ipmitool lan set 1 ipsrc dhcp

Check what is now setup :

ipmitool lan print

example :

[[email protected]~]# ipmitool lan print
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD
Auth Type Enable        : Callback : MD2 MD5 PASSWORD
                        : User     : MD2 MD5 PASSWORD
                        : Operator : MD2 MD5 PASSWORD
                        : Admin    : MD2 MD5 PASSWORD
                        : OEM      : MD2 MD5 PASSWORD
IP Address Source       : DHCP Address
IP Address              : -ethernet-
Subnet Mask             : 255.255.0.0
MAC Address             : -mac-
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP      : -gateway-
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max   : XaaaXXaaaXXaaXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : 0
Invalid password disable: no
Attempt Count Reset Int.: 0
User Lockout Interval   : 0

 

Thanks to this IBM documentation.

Disable 2FA in Proxmox 6

21 November, 2019

I was checking the new version of Proxmox, when I noticed there is an option to enable 2FA; (see the docs) It works really fine, but then I realized my colleague’s probably don’t have the key. To disable however, I couldn’t find it online; So I digged in the guts of Proxmox /etc/pve/user.cfg there is a line similar to :

user:[email protected]:1:0:::[email protected]::x!oath:

I changed this to :

user:[email protected]:1:0:::[email protected]:::

 

That change let me log in without the 2FA.

 

I got this weird error;

Warning: mysql_connect(): Headers and client library minor version mismatch. Headers:50162 Library:50524
Filename: mysqli/mysqli_driver.php

After checking PHP seems there was a different version of MariaDB installed; 5.5 vs 10.1;

php -i |grep API
Server API => Command Line Interface
PHP API => 20160303
Zend Extension Build => API320160303,NTS
PHP Extension Build => API20160303,NTS
MHASH API Version => Emulated Support
Client API library version => 10.1.23-MariaDB
Client API header version => 5.5.52-MariaDB
Client API version => 10.1.23-MariaDB
Phar API version => 1.1.1

To fix it I installed ‘mysql native driver’ instead of the MySQL lib what is mysqlnd ? See this writeup by ulf-wendel.

fix :

yum remove php71w-mysql
yum install php71w-mysqlnd

[[email protected] /]# service php-fpm restart
Redirecting to /bin/systemctl restart php-fpm.service
[[email protected] /]#

See also stackoverflow and MySQL explanation.

Finally got time to give Centos 8 a spin in a Proxmox Linux container (lxc); I downloaded the template, and started a container only to find its not possible :

extracting archive '/var/lib/vz/template/cache/centos-8-default_20191016_amd64.tar.xz'
Total bytes read: 673566720 (643MiB, 37MiB/s)
Detected container architecture: amd64
TASK ERROR: unable to create CT 100 - unsupported centos release 'CentOS Linux release 8.0.1905 (Core) '

I tried update & upgrade :

apt-get update && apt-get dist-upgrade

To no avail; Luckely Geekshack SecLabs and his sed wizardry provided a workaround :

sed -i 's/ < 8) {/ < 9) {/' /usr/share/perl5/PVE/LXC/Setup/CentOS.pm

Then restart the pve daemon :

systemctl restart pveproxy pvedaemon

and bam, container is installed  :

extracting archive '/var/lib/vz/template/cache/centos-8-default_20191016_amd64.tar.xz'
Total bytes read: 673566720 (643MiB, 38MiB/s)
Detected container architecture: amd64
[-cut-]
TASK OK

 

This is a mount issue, that I haven’t faced lately anymore; It was however in my drafts for so long; I might as wel write it up;

linux_raid_member is a filesystem that is part of a software RAID meaning you need at least two disks to read the data in a “normal” state. So it largely depends on what RAID level was setup and how many disks where involved.  Most common RAID types are 0, 1, 5, 6, 10. RAID 0, you need all disks, RAID 1 you could access the data with only half of all the disks, with RAID 5, you need all disks – 1; raid 6 you need all disk -2; raid 10 you need half of all disks, but you need the full set.

You might not know what kind of RAID level was picked, lucky you can examine that from the metadata, using mdadm :

# mdadm --examine /dev/sdc1
          Magic :
        Version : 
    Feature Map :
     Array UUID : 
           Name : 
  Creation Time : 
     Raid Level : raid1
   Raid Devices : 2

- snap -

This learns us that there where only 2 devices and the level was RAID 1; meaning all data is on both disks, and should be equal !

We can start the raid again using mdadm :

mdadm --assemble --run /dev/md0 /dev/sdc1 --force

note that we need to use force as mdadm will rightfully complain about the raid being degraded (its missing a disk).

This should generate /dev/md0 which we can mount normally :

mount /dev/md0 /mnt/external_disk

useful “cheat sheet ” for mdadm by ducea

 

Library missing during install of MongoDB :

gssapi/gssapi.h: No such file or directory

On Centos 7.6 it gets fixed by :

yum install krb5-devel.x86_64

krb5 is a development package for kerberos, a network authentication system;

Kerberos is a network authentication system. The krb5-devel package contains the header files and libraries needed for compiling Kerberos 5 programs. If you want to develop Kerberos-aware programs, you'll need to install this package.

 

A quick fix for a weird issue that in Proxmox unprivileged container fails to install filesystem update on Centos 7 :

[[email protected] ~]# yum install filesystem
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.belnet.be
 * extras: ftp.belnet.be
 * updates: ftp.belnet.be
Resolving Dependencies
--> Running transaction check
---> Package filesystem.x86_64 0:3.2-21.el7 will be updated
---> Package filesystem.x86_64 0:3.2-25.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                              Arch                                             Version                                                Repository                                      Size
===================================================================================================================================================================================================================
Updating:
 filesystem                                           x86_64                                           3.2-25.el7                                             base                                           1.0 M

Transaction Summary
===================================================================================================================================================================================================================
Upgrade  1 Package

Total download size: 1.0 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
filesystem-3.2-25.el7.x86_64.rpm                                                                                                                                                            | 1.0 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : filesystem-3.2-25.el7.x86_64                                                                                                                                                                    1/2
Error unpacking rpm package filesystem-3.2-25.el7.x86_64
error: unpacking of archive failed on file /sys: cpio: chown
  Verifying  : filesystem-3.2-25.el7.x86_64                                                                                                                                                                    1/2
filesystem-3.2-21.el7.x86_64 was supposed to be removed but is not!
  Verifying  : filesystem-3.2-21.el7.x86_64                                                                                                                                                                    2/2

Failed:
  filesystem.x86_64 0:3.2-21.el7                                                                           filesystem.x86_64 0:3.2-25.el7

Complete!

The fix, thanks to this post :

echo "%_netsharedpath /sys:/proc" >> /etc/rpm/macros.dist

and tada :

Updated:
  filesystem.x86_64 0:3.2-25.el7

 

Today, I had a new issue, a server blankly refused to reboot/shutdown.

[[email protected] ~] reboot -h now
Failed to start reboot.target: Connection timed out
See system logs and 'systemctl status reboot.target' for details.
Failed to open /dev/initctl: No such device or address
Failed to talk to init daemon.

I first tried :

[[email protected] ~]# systemctl --force reboot
Failed to execute operation: Activation of org.freedesktop.systemd1 timed out

which failed, so I had to get the big guns : systemctl --force --force reboot

[[email protected] ~]# systemctl --force --force reboot
Rebooting with argument 'now'.
packet_write_wait: Connection to 10.1.255.253 port 22: Broken pipe

As you see that restarted the server; weird issue, solved thanks to Luke

When a Client job fails and reports :

Error: VSS API failure calling "BackupComplete". ERR=Object is not initialized; called during restore or not called in correct sequence.

Try to start the Service : VSS manually;

This solved the issue for a few Clients;

After adding a bunch of clients, we realized that a good naming convention was required and that some funny names, where not the best choice after all. So we changed them. However, Bareos will remember those Jobs ran against the old Client. The ‘new’ name will be seen as a new client; Some of the scripts we made however found it annoying that some clients where not backed up for weeks. Basically cause they where no longer existing. There is however a way to completely remove them from database.

  • “Restore” the original configuration, or create a new client in the configuration with the old name; (Address / Password does not have to be correct, we just need a name in the configuration)
  • in bconsole :
    *purge jobs client=SVENND
    
    This command can be DANGEROUS!!!
    
    It purges (deletes) all Files from a Job,
    JobId, Client or Volume; or it purges (deletes)
    all Jobs from a Client or Volume without regard
    to retention periods. Normally you should use the
    PRUNE command, which respects retention periods.
    This command requires full access to all resources.
    Using Catalog "MyCatalog"
    Begin purging jobs from Client "SVENND"
    Found 19 Jobs for client "SVENND" in catalog "MyCatalog".
    Purge (yes/no)? yes
    
  • After that, the client record will be orphaned, as nothing is ran for it; Now you can clear the database using : bareos-dbcheck -v -f
[[email protected] ~]# bareos-dbcheck -v -f
Hello, this is the Bareos database check/correct program.
Modify database is on. Verbose is on.
Please select the function you want to perform.
     0) Quit
     1) Toggle modify database flag
     2) Toggle verbose flag
     3) Check for bad Filename records
     4) Check for bad Path records
     5) Check for duplicate Path records
     6) Check for orphaned Jobmedia records
     7) Check for orphaned File records
     8) Check for orphaned Path records
     9) Check for orphaned FileSet records
    10) Check for orphaned Client records
    11) Check for orphaned Job records
    12) Check for all Admin records
    13) Check for all Restore records
    14) Run ALL checks
Select function number: 10
Checking for orphaned Client entries.
Found 1 orphaned Client records.
Print them? (yes/no): yes
Orphaned ClientId=8 Name="SVENND"
Deleting 1 orphaned Client records.
Deleting: DELETE FROM Client WHERE ClientId=8

Of course the data of those users are still in the filesystem (or tape) for me that is not really an issue, so I did not look into it.