Posts Tagged ‘servers’

Disable NetworkManager automatic Ethernet Interface Management on Redhat Linux , CentOS 6 / 7 / 8

Friday, February 5th, 2021

Reading Time: 5minutes


Most of Linux distributions had introduced the NetworkManager service and are slowly trying to push out the old ways and use entirely it to manage network configs. Though at times this is very helpful stuff especially if you have Linux running on Laptop on servers is a guarantee for troubles.

If you are a system administrator like me and you need that needs to configure a New server with lets say 8 (Ethernet interface) LAN cards each to be configured with different IPs and you have a mixture of configuration where some eth1,eth2 etc. (4 of the interface IPs has to be static IPs and others has to be taken from a DHCP lease. NetworkManager is not something that you will want as usually you don't expect soon a network IP topology change. Below is example from a Living Hypervisor server machine that has 8 Network Interfaces configured together with few Virtual Interfaces used by the running KVM Virtual Machines.

[root@redhat :~ ]# ip address show |grep ": <"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master team0 state UP group default qlen 1000
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master team0 state UP group default qlen 1000
4: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br2 state UP group default qlen 1000
5: ens1f2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
6: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
7: ens1f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
8: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
9: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
10: venet0: <BROADCAST,POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
11: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
12: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
13: team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
14: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
15: host-routed: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
16: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
17: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
18: virbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
19: virbr1-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr1 state DOWN group default qlen 1000
26: vme52540019e701: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UNKNOWN group default qlen 1000
27: vme52540081868b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br1 state UNKNOWN group default qlen 1000
28: vme525400a13f03: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br2 state UNKNOWN group default qlen 1000

Having a NM managing so many LAN connected Ethernets can create you A LOT of surprises even if your servers are in a Highly Secured data center where chance of sudden IP change or network misbehaves are minimal. Even minimal some in Housing might do something wrong on the Rack mixing up with another server or switch andyour server might end up easily with unexplainable Network problems because of this NM service which is trying 'to balance' any network issues according to some algorithms …

Thus to save yourlself the troubles and completely disable NetworkManager (NM) Ethernets handling.

As a hint some of the troubles you might get especially if the System Hardware has issues with the Integrated Motherboard LAN Controllers such as of Dell PowerEdge R640 Rack Server.
I've recently observed one such Dell Rack mounted machine I had to configure from scratch which has out of the box 
NM preinstalled by a colleague and was doing strange stuff with the routings causing it to become remotely inacessible after reboot.
Even though I have started configuring the IPs and have double and triple check the configuration and machine had proper set of /etc/sysconfig/network-scripts/ifcfg-* configuration it still failed to boot with a network properly brought up and become unreachable via remote SSH connection immediately after sending machine to init 6 with /usr/sbin/init 6 (alias for shutdown -r now or reboot -f now :)

On Redhat 8 / CentOS 8 to Disabling permanently NM you have to disable NM systemd services permanently and add NM_CONTROLLED=no to each of the Ethernet configurations listed in network-scripts/ifcfg-eno3 eno4 eno1np0 etc. ifaces.

1. Disable completely Network Manager service and mask it

[root@redhat :~ ]# systemctl mask NetworkManager.service
[root@redhat :~ ]# systemctl stop NetworkManager.service
[root@redhat :~ ]# systemctl disable NetworkManager.service

2. Check if all systemd networkmanager components scripts are really disabled

# systemctl list-unit-files | grep NetworkManager

NetworkManager-dispatcher.service disabled
NetworkManager-wait-online.service enabled
NetworkManager.service disabled

NetworkManager-wait-online.service seems to be also enabled so we have to disable it.

[root@redhat :~ ]#  systemctl mask NetworkManager-wait-online.service
[root@redhat :~ ]#  systemctl disable NetworkManager-wait-online.service

Double check NM services

[root@redhat :~ ]#  systemctl list-unit-files | grep NetworkManager

3. Install / Enable old (legacy) network-scripts 

network-scripts is disabled by default due to it doesn't play well with NM.
Install the rpm package to enable it back

[root@redhat :~ ]#  yum install -y network-scripts 

4. Test if network-scripts is really enabled

Use Redhat's nmcli command for controlling network manager if it reports NM not running then you're fine

[root@redhat :~ ]#  nmcli device
Error: NetworkManager is not running.

5. Disable legacy use network-scripts print outs

Bring down some interface with ifdown Redhat script frontend to ifconfig and bring it up with ifup iface-name

# ifup eno4
WARN      : [ifup] You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
WARN      : [ifup] 'network-scripts' will be removed in one of the next major releases of RHEL.
WARN      : [ifup] It is advised to switch to 'NetworkManager' instead – it provides 'ifup/ifdown' scripts as well.

Notice the warnings they're harmless and safe to ignore however it is pretty annoying to see them, to disable them:

[root@redhat :~ ]#  touch /etc/sysconfig/disable-deprecation-warnings

6. Use network.service old-fashioned systemd service

From now on you can start using the good old well known and properly working network.service

[root@redhat :~ ]#  systemctl status network

To enable the network service to start after boot:

[root@redhat :~ ]#  systemctl enable network

7. Disable NetworkManager use from Network configuration scripts ifcfg-* for all server available configured ethernet cards

Open with text editor every network script and append NM_CONTROLLED="no" to the end of the file.

[root@redhat :~ ]#  vi /etc/sysconfig/network-scripts/ifcfg-ethernetX

To save yourself the time if you want to disable NetworkManager use for all /etc/sysconfig/network-scripts/ifcfg-* use a simple shell loop:

[root@redhat :~ ]# cd /etc/sysconfig/network-scripts/
[root@redhat :/etc/sysconfig/network-scripts ]# for i in *ifcfg*; do echo NM_CONTROLLED="no" >> $i; done

To load the new network settings do another network reload / restart

[root@redhat :~ ]# systemctl restart network

To disable NetworkManager on older CentOS 6 / Redhat 6 / SuSE / Fedora Linux where the OS still not systemd enabled instead of using systemctl you can straight do it with old and well known chkconfig redhat script.

[root@centos6 :~ ]# service NetworkManager stop
[root@centos6 :~ ]# chkconfig NetworkManager off

Hack: Using ssh / curl or wget to test TCP port connection state to remote SSH, DNS, SMTP, MySQL or any other listening service in PCI environment servers

Wednesday, December 30th, 2020

Reading Time: 5minutes


If you work on PCI high security environment servers in isolated local networks where each package installed on the Linux / Unix system is of importance it is pretty common that some basic stuff are not there in most cases it is considered a security hole to even have a simple telnet installed on the system. I do have experience with such environments myself and thus it is pretty daunting stuff so in best case you can use something like a simple ssh client if you're lucky and the CentOS / Redhat / Suse Linux whatever distro has openssh-client package installed.
If you're lucky to have the ssh onboard you can use telnet in same manner as netcat or the swiss army knife (nmap) network mapper tool to test whether remote service TCP / port is opened or not. As often this is useful, if you don't have access to the CISCO / Juniper or other (networ) / firewall equipment which is setting the boundaries and security port restrictions between networks and servers.

Below is example on how to use ssh client to test port connectivity to lets say the Internet, i.e.  Google / Yahoo search engines.

[root@pciserver: /home ]# ssh -oConnectTimeout=3 -v -p 23
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 23.
debug1: connect to address port 23: Connection timed out
debug1: Connecting to [2a00:1450:4017:80b::200e] port 23.
debug1: connect to address 2a00:1450:4017:80b::200e port 23: Cannot assign requested address
ssh: connect to host port 23: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -oConnectTimeout=3 -v -p 80
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: connect to address port 80: Connection timed out
debug1: Connecting to [2a00:1450:4017:807::200e] port 80.
debug1: connect to address 2a00:1450:4017:807::200e port 80: Cannot assign requested address
ssh: connect to host port 80: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -p 80
ssh_exchange_identification: Connection closed by remote host
root@pcfreak:/var/www/images# ssh -p 80 -v -oConnectTimeout=3
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: connect to address port 80: Connection timed out
debug1: Connecting to [2a00:1450:4017:80b::200e] port 80.
debug1: connect to address 2a00:1450:4017:80b::200e port 80: Cannot assign requested address
ssh: connect to host port 80: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -p 80 -v -oConnectTimeout=5
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: connect to address port 80: Connection timed out
debug1: Connecting to [2a00:1450:4017:80c::200e] port 80.
debug1: connect to address 2a00:1450:4017:80c::200e port 80: Cannot assign requested address
ssh: connect to host port 80: Cannot assign requested address
root@pcfreak:/var/www/images# ssh -p 80 -v
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 80.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: ssh_exchange_identification: HTTP/1.0 400 Bad Request


debug1: ssh_exchange_identification: Content-Type: text/html; charset=UTF-8

debug1: ssh_exchange_identification: Referrer-Policy: no-referrer

debug1: ssh_exchange_identification: Content-Length: 1555

debug1: ssh_exchange_identification: Date: Wed, 30 Dec 2020 14:13:25 GMT

debug1: ssh_exchange_identification:

debug1: ssh_exchange_identification: <!DOCTYPE html>

debug1: ssh_exchange_identification: <html lang=en>

debug1: ssh_exchange_identification:   <meta charset=utf-8>

debug1: ssh_exchange_identification:   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">

debug1: ssh_exchange_identification:   <title>Error 400 (Bad Request)!!1</title>

debug1: ssh_exchange_identification:   <style>

debug1: ssh_exchange_identification:     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(// 10
debug1: ssh_exchange_identification: 0% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.g
debug1: ssh_exchange_identification: no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(// no-repeat 0
debug1: ssh_exchange_identification: % 0%/100% 100%;-moz-border-image:url(// 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//
debug1: ssh_exchange_identification: color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}

debug1: ssh_exchange_identification:   </style>

debug1: ssh_exchange_identification:   <a href=//><span id=logo aria-label=Google></span></a>

debug1: ssh_exchange_identification:   <p><b>400.</b> <ins>That\342\200\231s an error.</ins>

debug1: ssh_exchange_identification:   <p>Your client has issued a malformed or illegal request.  <ins>That\342\200\231s all we know.</ins>

ssh_exchange_identification: Connection closed by remote host


Here is another example on how to test remote host whether a certain service such as DNS (bind) or telnetd is enabled and listening on remote local network  IP with ssh

[root@pciserver: /home ]# ssh -p 53 -v -oConnectTimeout=5
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 53.
debug1: connect to address port 53: Connection timed out
ssh: connect to host port 53: Connection timed out

[root@server: /home ]# ssh -p 23 -v -oConnectTimeout=5
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Connecting to [] port 23.
debug1: connect to address port 23: Connection timed out
ssh: connect to host port 23: Connection timed out

But what if Linux server you have tow work on is so paranoid that you even the ssh client is absent? Well you can use anything else that is capable of doing a connectivity to remote port such as wget or curl. Some web servers or application servers usually have wget or curl as it is integral part for some local shell scripts doing various operation needed for proper services functioning or simply to test locally a local or remote listener services, if that's the case we can use curl to connect and get output of a remote service simulating a normal telnet connection like this:

host:~# curl -vv 'telnet://remote-server-host5:22'
* About to connect() to remote-server-host5 port 22 (#0)
*   Trying… connected
* Connected to aflpvz625 ( port 22 (#0)

Now lets test whether we can connect remotely to a local net remote IP's Qmail mail server with curls telnet simulation mode:

host:~#  curl -vv 'telnet://'
* Expire in 0 ms for 6 (transfer 0x56066e5ab900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x56066e5ab900)
* Connected to ( port 25 (#0)
220 This is Mail Pc-Freak.NET ESMTP

Fine it works, lets now test whether a remote server who has MySQL listener service on standard MySQL port TCP 3306 is reachable with curl

host:~#  curl -vv 'telnet://'
* Expire in 0 ms for 6 (transfer 0x5601fafae900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x5601fafae900)
* Connected to ( port 3306 (#0)
Warning: Binary output can mess up your terminal. Use "–output -" to tell
Warning: curl to output it to your terminal anyway, or consider "–output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 107)
* Closing connection 0
root@pcfreak:/var/www/images#  curl -vv 'telnet://'
* Expire in 0 ms for 6 (transfer 0x5598ad008900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x5598ad008900)
* Connected to ( port 3306 (#0)
Warning: Binary output can mess up your terminal. Use "–output -" to tell
Warning: curl to output it to your terminal anyway, or consider "–output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 107)
* Closing connection 0

As you can see the remote connection is returning binary data which is unknown to a standard telnet terminal thus to get the output received we need to pass curl suggested arguments.

host:~#  curl -vv 'telnet://' –output –
* Expire in 0 ms for 6 (transfer 0x55b205c02900)
*   Trying…
* Expire in 200 ms for 4 (transfer 0x55b205c02900)
* Connected to ( port 3306 (#0)

The curl trick used to troubleshoot remote port to remote host from a Windows OS host which does not have telnet installed by default but have curl instead.

Also When troubleshooting vSphere Replication, it is often necessary to troubleshoot port connectivity as common Windows utilities are not available.
As Curl is available in the VMware vCenter Server Appliance command line interface.

On servers where curl is not there but you have wget is installed you can use it also to test a remote port


# wget -vv -O /dev/null –timeout=5
–2020-12-30 16:54:22–
Resolving (…, 2a00:1450:4017:80b::200e
Connecting to (||:554… failed: Connection timed out.
Connecting to (|2a00:1450:4017:80b::200e|:554… failed: Cannot assign requested address.

–2020-12-30 16:54:28–  (try: 2)
Connecting to (||:554… ^C

As evident from output the port 554 is filtered in google which is pretty normal.

If curl or wget is not there either as a final alternative you can either install some perl, ruby, python or bash script etc. that can opens a remote socket to the remote IP.

Add Zabbix time synchronization ntp userparameter check script to Monitor Linux servers

Tuesday, December 8th, 2020

Reading Time: 3minutes



How to add Zabbix time synchronization ntp userparameter check script to Monitor Linux servers?

We needed to set on some servers at my work an elementary check with Zabbix monitoring to check whether servers time is correctly synchronized with ntpd time service as well report if the ntp daemon is correctly running on the machine. For that a userparameter script was developed called userparameter_ntp.conf the script is simplistic and few a lines of bash shell scripting 
stuff is based on gresping information required from ntpq and ntpstat common ntp client commands to get information about the status of time synchronization on the servers.

[root@linuxserver ]# ntpstat
synchronised to NTP server ( at stratum 3
   time correct to within 47 ms
   polling server every 1024 s


[root@linuxserver ]# ntpq -c peers
     remote           refid      st t when poll reach   delay   offset  jitter
+timeserver1     2 u  319 1024  377   15.864    1.270   0.262
+timeserver2     2 u  591 1024  377   16.287   -0.334   1.748
*timeserver3     2 u   47 1024  377   15.613   -0.553   0.251
 timeserver4 .INIT.          16 u    – 1024    0    0.000    0.000   0.000

Below is Zabbix UserParameter script that does report us 3 important values we monitor to make sure time server synchronization works as expected the zabbix keys we set are ntp.offset, ntp.sync, ntp.exact in attempt to describe what we're fetching from ntp client:

[root@linuxserver ]# cat /etc/zabbix/zabbix-agent.d/userparameter_ntp.conf

UserParameter=ntp.offset,(/usr/sbin/ntpq -pn | /usr/bin/awk 'BEGIN { offset=1000 } $1 ~ /\*/ { offset=$9 } END { print offset }')
#UserParameter=ntp.offset,(/usr/sbin/ntpq -pn | /usr/bin/awk 'FNR==4{print $9}')
UserParameter=ntp.sync,(/usr/bin/ntpstat | cut -f 1 -d " " | tr -d ' \t\n\r\f')
UserParameter=ntp.exact,(/usr/bin/ntpstat | /usr/bin/awk 'FNR==2{print $5,$6}')

In Zabbix the monitored ntpd parameters set-upped looks like this:




!Note that in above userparameter example, the commented userparameter script is a just another way to do an ntpd offset returned value which was developed before the more sophisticated with more regular expression checks from the /usr/sbin/ntpd via ntpq, perhaps if you want to extend it you can also use another script to report more verbose information to Zabbix if that is required like ouput from ntpq -c peers command:

UserParameter=ntp.verbose,(/usr/sbin/ntpq -c peers)

Of course to make the Zabbix fetch necessery data from monitored hosts, we need to set-up further new Zabbix Template with the respective Trigger and Items.

Below are few screenshots including the triggers used.


  • ntpd.trigger


  • NTP Synchronization trigger




As you can see from history we have setup our items to Store history of reported data to Zabbix from parameter script for 90 days and update our monitor check, every 30 seconds from the monitored hosts to which Tempate is applied.

Well that's all folks, time synchronization issues we'll be promptly triggering a new Alarm in Zabbix !

Howto Upgrade IBM Spectrum Protect Backup Client TSM 7.X to 8.1.8, Update Tivoli 8.1.8 to 8.1.11 on CentOS and Redhat Linux

Thursday, December 3rd, 2020

Reading Time: 6minutes



Having another day of a system administrator boredom, we had a task to upgrade some Tivoli TSM Backup clients running on a 20+ machines powered by CentOS and RHEL Linux to prepare the systems to be on the latest patched IBM Spectrum Backup client version available from IBM. For the task of patching I've used a central server where, I've initially downloaded the provided TSM client binaries archives. From this machine, we have copied TivSM*.tar to each and every system that needs to be patched and then patched. The task is not too complex as the running TSM in the machines are all at the same version and all running a recent patched version of Linux. Hence to make sure all works as expected we have tested TSM is upgraded from 7.X.X to 8.X.X on one machine and then test 8.1.8 to 8.1.11 upgrade on another one. Once having confirmed that Backups works as expected after upgrade. We have proceeded to do it massively on each of the rest 20+ hosts.
Below article's goal is to help some lazy sysadmin with the task to prepare an TSM Backup upgrade procedure to standartize TSM Upgrade, which as many of the IBM's softwares is very specific and its upgrade requires, a bit of manual work and extra cautious as there seems to be no easy way (or at least I don't know it), to do the upgrade by simply adding an RPM repository and doing, something like yum install tivsm*.

0. Check if there is at least 2G free of space

According to documentation the minimum space you need to a functional install without having it half installed or filling up your filesystem is 2 Gigabytes of Free Memory on a filesystem where the .tar and rpms will be living.

Thus check what is the situation with your filesystem where you wills store the .tar archice and extract .RPM files / install the RPM files.

# df -h

1. Download the correct tarball with 8.1 Client

On one central machine you would need to download the Tivoli you can do that via wget / curl / lynx whatever is at hand on the Linux server.

As of time of writting this article TSM's 8.1.11 location is at

I've made a local download mirror of Tivoli TSM 8.1.11 here.
In case you need to install IBM Spectrum Backup Client to a PCI secured environment to a DMZ-ed LAN network from a work PC you can Download it first from your local PC and via Citrix client upload program or WinSCP upload it to a central replication host from where you will later copy to each of the other server nodes that needs to be upgraded.

Lets Copy archive to all Server hosts where you want it later installed, using a small hack

Assuming you already have an Excel document or a Plain text document with all the IPs of the affected hosts where you will need to get TSM upgraded. Extract this data and from it create a plain text file /home/user/hosts.txt containing all the machine IPs lined up separated with carriage return separations (\n), so you can loop over each one and use scp to send the files.

– Replicate Tivoli tar to all machine hosts where you want to get IBM Spectrum installed or upgraded.
Do it with a loop like this:

# for i in $(cat hosts.txt); do scp user@$i:/home/user/; done

 Copy to a Copy buffer temporary your server password assuming all your passwords to each machine are identical and paste your login user pass for each host to initiate transfer

2. SSH to each of the Machine hosts IPs

Once you login to the host you want to upgrade
Go to your user $HOME/home/user and create files where we'll temporary store Tivoli archive files and extract RPMs

[root@linux-server user]# mkdir -p ~/tsm/TSM_BCK/
[root@linux-server user]# mv ~/tsm
[root@linux-server user]# cd tsm
[root@linux-server user]# tar -xvvf

3. Create backup of old backup files

It is always a good idea to keep old backup files

[root@linux-server tsm]# cp -av /opt/tivoli/tsm/client/ba/bin/dsm.opt ~/tsm/TSM_BCK/dsm.opt_bak_$(date +'%Y_%M_%H')
[root@linux-server tsm]# cp -av /opt/tivoli/tsm/client/ba/bin/dsm.sys ~/tsm/TSM_BCK/dsm.sys_bak_$(date +'%Y_%M_%H')

[root@linux-server tsm]# [[ -f /etc/adsm/TSM.PWD ]] && cp -av /etc/adsm/TSM.PWD ~/TSM_BCK/ || echo 'file doesnt exist'

/etc/adsm/TSM.PWD this file is only there as legacy for TSM it contained encrypted passwords inver 7 for updates. In TSM v.8 encryption file is not there as new mechanism for sensitive data was introduced.
Be aware that from Tivoli 8.X it will return error

!! Note – if dsm.opt , dsm.sys files are on different locations – please use correct full path locations !!

4. Stop  dsmcad – TSM Service daemon

[root@linux-server tsm]# systemctl stop dsmcad

5. Locate and deinstall all old Clients

Depending on the version to upgrade if you're upgrading from TSM version 7 to 8, you will get output like.

[root@linux-server tsm]# rpm -qa | grep 'TIVsm-'

If you're one of this paranoid admins you can remove TIVsm packs  one by one.

[root@linux-server tsm]# rpm -e TIVsm-BA-7.1.6-2.x86_64
[root@linux-server tsm]# rpm -e TIVsm-API64-7.1.6-2.x86_64

Instead if upgrading from version 8.1.8 to 8.1.11 due to the Security CVE advisory recently published by IBM e.g. (IBM Runtime Vulnerability affects IBM Spectrum Backup archive Client) and  vulnerability in Apache Commons Log4J affecting IBM Spectrum Protect Backup Archive Client.

[root@linux-server tsm]# rpm -qa | grep 'TIVsm-'

Assuming you're not scared of a bit automation you can straight do it with below one liner too 🙂

# rpm -e $(rpm -qa | grep TIVsm)

[root@linux-server tsm]# rpm -qa | grep gsk
[root@linux-server tsm]# rpm -e gskcrypt64 gskssl64

6. Check uninstallation success:

[root@linux-server tsm]# rpm -qa | grep TIVsm
[root@linux-server tsm]# rpm -qa | grep gsk

Here you should an Empty output, if packages are not on the system, e.g. Empty output is good output ! 🙂

7. Install new client IBM Spectrum Client (Tivoli Storage Manager) and lib dependencies

[root@linux-server tsm]# rpm -ivh gskcrypt64-
[root@linux-server tsm]# rpm -ivh gskssl64-

 If you're lazy to type you can do as well

[root@linux-server tsm]# rpm -Uvh gsk*

Next step is to install main Tivoli SM components the the API files and BA (The Backup Archive Client)

[root@linux-server tsm]# rpm -ivh TIVsm-API64.x86_64.rpm
[root@linux-server tsm]# rpm -ivh TIVsm-BA.x86_64.rpm

If you have to do it on multiple servers and you do it manually following a guide like this, you might instead want to install them with one liner.

[root@linux-server tsm]# rpm -ivh TIVsm-API64.x86_64.rpm TIVsm-BA.x86_64.rpm

There are some Not mandatory "Common Inventory Technology" components (at some cases if you're using the API install it we did not need that), just for the sake if you need them on your servers due to backup architecture, install also below commented rpm files.

## rpm -ivh TIVsm-APIcit.x86_64.rpm

## rpm -ivh TIVsm-BAcit.x86_64.rpm

These packages not needed only for operation WebGUI TSM GUI management, (JBB) Journal Based Backup, BAhdw (the ONTAP library)

— TIVsm-WEBGUI.x86_64.rpm
— TIVsm-JBB.x86_64.rpm
— TIVsm-BAhdw.x86_64.rpm

8. Start and enable dsmcad service

[root@linux-server tsm]# systemctl stop dsmcad

You will get

##Warning: dsmcad.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[root@linux-server tsm]# systemctl daemon-reload

[root@linux-server tsm]# systemctl start dsmcad

## enable dsmcad – it is disabled by default after install

[root@linux-server ~]# systemctl enable dsmcad

[root@linux-server tsm]# systemctl status dsmcad

9. Check dmscad service is really running

Once enabled IBM TSM will spawn a process in the bacground dmscad if it started properly you should have the process backgrounded.

[root@linux-server tsm]# ps -ef|grep -i dsm|grep -v grep
root      2881     1  0 18:05 ?        00:00:01 /usr/bin/dsmcad

If process is not there there might be some library or something not at place preventing the process to start …

10. Check DSMCAD /var/tsm logs for errors

After having dsmcad process enabled and running in background

[root@linux-server tsm]# grep -i Version /var/tsm/sched.log|tail -1
12/03/2020 18:06:29   Server Version 8, Release 1, Level 10.000


[root@linux-server tsm]# cat /var/tsm/dsmerror.log

To see the current TSM configuration files we can  grep out comments *

[root@linux-server tsm]# grep -v '*' /opt/tivoli/tsm/client/ba/bin/dsm.sys

Example Configuration of the agent:
   *TSM SERVER NODE Location
   Servername           tsm_server
   COMMmethod           TCPip
   TCPPort              1400
   Passwordaccess       generate
   SCHEDLOGNAME         /var/tsm/sched.log
   MANAGEDServices      schedule
   ERRORLOGNAME         /var/tsm/dsmerror.log
   INCLEXCL             /opt/tivoli/tsm/client/ba/bin/inclexcl.tsm

11. Remove tsm install directory tar ball and rpms to save space on system

The current version of Tivoli service manager is 586 Megabytes.

[root@linux-server tsm]# du -hsc

Some systems are on purpose configured to have less space under their /home directory,
hence it is a good idea to clear up unnecessery files after completion.

Lets get rid of all the IBM Spectrum archive source files and the rest of RPMs used for installation.

[root@linux-server tsm]# rm -rf ~/tsm/{*.tar,*.rpm,*.gpg,*.htm,*.txt}

12. Check backups are really created on the configured remote Central backup server

To make sure after the upgrade the backups are continuously created and properly stored on the IBM Tivoly remote central backup server, either manually initiate a backup or wait for lets say a day and run dsmc client to show all created backups from previous day. To make sure you'll not get empty output you can on purpose modify some file by simply opening it and writting over without chaning anything e.g. modify your ~/.bashrc or ~/.bash_profile

## List all backups for '/' root directory from -fromdate='DD/MM/YY'

[root@linux-server tsm]# dsmc
IBM Spectrum Protect
Command Line Backup-Archive Client Interface
  Client Version 8, Release 1, Level 11.0
  Client date/time: 12/03/2020 18:14:03
(c) Copyright by IBM Corporation and other(s) 1990, 2020. All Rights Reserved.

Session established with server TSM2_SERVER: AIX
  Server Version 8, Release 1, Level 10.000
  Server date/time: 12/03/2020 18:14:04  Last access: 12/03/2020 18:06:29
Protect> query backup -subdir=yes "/" -fromdate=12/3/2020
           Size        Backup Date                Mgmt Class           A/I File
           —-        ———–                ———-           — —-
         6,776  B  12/03/2020 01:26:53             DEFAULT              A  /etc/freshclam.conf
         6,685  B  12/03/2020 01:26:53             DEFAULT              A  /etc/freshclam.conf-2020-12-02
         5,602  B  12/03/2020 01:26:53             DEFAULT              A  /etc/hosts
         5,506  B  12/03/2020 01:26:53             DEFAULT              A  /etc/hosts-2020-12-02
           398  B  12/03/2020 01:26:53             DEFAULT              A  /opt/tivoli/tsm/client/ba/bin/tsmstats.ini
       114,328  B  12/03/2020 01:26:53             DEFAULT              A  /root/.bash_history
           403  B  12/03/2020 01:26:53             DEFAULT              A  /root/.lesshst

How to add colorful random ASCII art picture and a bible verse on each SSH server login, joyout sysadmins life with cowsay, fortune, caca-utils and others

Tuesday, November 24th, 2020

Reading Time: 6minutes


There are pleny of console ASCII stuff out there that can make your console sysadmin boring life a little bit more funny and cherish some memories from the old times of 8 bit computers :).

One of this as I blogged earlier is cowsay and cowthink to generate a ascii picture with a cow with your custom message.
I've earlier blogged about that in my previous articleCreate ASCII Art Text bannners in Linux console / terminal with figlet and toilet

One of this cool things I'm using daily on my servers  is a cowsay console goodie together with a bash shell script that does visualize a random ASCII picture from a preset of pictures on each and every ssh login to my server.
The script I use is cowrand below is code:

# cowsay pix randomizer by hip0
# it shows random ascii from the cowsay prog during logging. :]
var=`ls -1 $dir | wc -l | awk '{ print $1}'`
let "number %= $var"
var1=`ls -1 $dir | head -n $number | tail -n 1 | head -n 1`
if [ -z “$var1” ]; then
/usr/bin/cowsay -f $var1 Welc0m3 t0 pC-fREAK … Enj0y.


The script is set as executable under /etc/cowrand

hipo@pcfreak:~$ ls -al /etc/cowrand
-rwxr-xr-x 1 hipo hipo 432 Nov 24 19:21 /etc/cowrand*

I've set this script to my /etc/profile to auto start on every login on my Debian Linux systems right after the comments like so:

hipo@pcfreak:~$ grep -i cowrand -A 2 -B 3 /etc/profile
# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
# and Bourne compatible shells (bash(1), ksh(1), ash(1), …).
echo '';
/etc/cowrand | lolcat
echo '';

As you can see to make my life even more funnier, I've installed another fun command lolcat


hipo@pcfreak:~$ apt-cache show lolcat |grep -i desc -A 3
Description-en: colorful `cat`
 lolcat concatenates files like the UNIX `cat` program, but colors it for the
 lulz in a rainbow animation. Terminals with 256 colors and animations are

Description-md5: 86f992d66ac74197cda39e0bbfcb549d
Ruby-Versions: all
Section: games

You can think of lolcat as a standard cat command that has been made to print in colors, this gives a funny results.


To add some spice to everything nice as a recipee for thethe creation of powerpuffgirls, I've come up with a way to use fortune
console tool that uses to print quotes out of a database to use as a source a big database containing the Holy Bible books of Old and New Testament Books. The fortune prints me out a quote extract from the bible on each and every remote SSH login to my machine. The content of this bible database for fortune bible_quotes_fortune.tar.gz can be downloaded and used from here.

The command used to print out a verse from the holy bible is:


hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
For if thou refuse to let them go, and wilt hold them still,
        — Exodus 9:2
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And when the queen of Sheba heard of the fame of Solomon concerning
the name of the LORD, she came to prove him with hard questions.
        — 1 Kings 10:1
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And Shelemiah, and Nathan, and Adaiah,
        — Ezra 10:39
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
For by thee I have run through a troop: by my God have I leaped
over a wall.
        — 2 Samuel 22:30
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
Unto the place of the altar, which he had make there at the first:
and there Abram called on the name of the LORD.
        — Genesis 13:4
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And there shall dwell in Judah itself, and in all the cities thereof
together, husbandmen, and they that go forth with flocks.
        — Jeremiah 31:24
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And he hath put a new song in my mouth, even praise unto our God:
many shall see it, and fear, and shall trust in the LORD.
        — Psalms 40:3
hipo@pcfreak:~$ /usr/games/fortune -s /usr/local/fortune/
And Jehoshaphat made peace with the king of Israel.
        — 1 Kings 22:44


The fortune is really awesome as it reminds me often of a verses from Holy Bible I often forget, the database is using the all famous King James Bible famous as (KJB) / (KJV) from 1611 this bible version that is like a protestant standard nowadays takes its name after James VI and I (James Charles Stuart; 19 June 1566 – 27 March 1625 – King of Scotland and Ireland) who was the sponsor of KJV collection and print.

Finally after adding the /usr/games/fortune -s /usr/local/fortune/ to the beginning of /etc/profile together with cowsay and cowrand I got this beautiful and educational result that combines fun with wisdom, below is example of what you will get after you  do a remote ssh login;






Those who have a Linux Graphical Environment desktop might also enjoy xcowsay

Another must I recommend to the text geeks is the caca-utils package which contains cool things such as aafire (cacafire)


Or (Image to text converter) img2txt / cacaview (a text console picture viewer) that could give you a raw idea on how a png / jpg picture looks like (or at least the picture shapes) without a need for a GUI picture viewer such as Eye of the Gnome.


Here is a original bear


And here is the one you'll see in cacaview 🙂
To read more about cacaview I have and its uses, check my previous article Viewing JPEG,GIF and PNG in ASCII with cacaview in Linux.
If you want to show off even more as a '1337 h4x0r' you might also show your sysadm 1337 5K!11Z to colleagues by showng them how you check weather via console (i've a separate article for how to ASCII art check colorful weather forecast via console / terminal ).

If you're too bored in your daily sys admin job, you might make some fun and take some useless effort to install ASCII Art Aquarium ASCIIQUARIUM




If you're crazy enough and want to torture your other sysadmin colleagues and a get a nice prank, you might install and set asciiquarium to auto run for their specific account on each and every login to some server until they control C or if you're a bit evil you can even set a small auto load on account login via ~/.bashrc shell script to 'Disable CTRL + C' combination 🙂

Of course there is plenty of other cool ASCII games and stuff. I've collected some of them by launching the Play Cool Ascii games service on my machine for ASCII art geeks to test out some ASCII games here.


Vodka! :)

Wednesday, September 12th, 2007

Reading Time: 2minutes

Yesterday I drinked 200 gr. of Vodka yesterday Night, it was pretty refreshing for me but I got drunk a little.I'm smoking again … Things are going bad in my life recently. I have health issues. And I intend to go to doctor today.Yesterday I went to the polyclinic but my personal Dr. Nikolay  was not there (I was angry, I went to doctor once in years and he is not there) so I'll try again today. I had pains somewhere around the stomach. At least at work things are going smoothly at least God hears my prayers about this. I'm very confused and I have completely no idea what to do with my life. Yesterday I was out with Lily and Kiril on the fountain. The previous day Nomen, I, Yavor, Kiro and Bino went to the "Kobaklyka" (a woody place which is close to Dobrich.) Well that's most of what's happening lately with my life. I wrote a little script to make that nautilus to get restarted if it starts burning the cpu. It's a dumb script (the bad thing is that I'm loosing form scripting, Well I don't script much lately). Here is the script The days before the 4 days weekend, I hat to spend a lot of time on one of the servers fighting with Spammers. Hate spammers really! I ended removing bounce messages at all for one of the domains, which fixed the bounce spam method spammers use (btw qmail's chkuser seems to not work properly for some reason) … Also I started watching Stargate – SG1. First I thought it's a stupid sci-fi serial. But after the first serie I now think it has it's good moments :]. Also I had something like a Mortification Day going on during Monday. The whole day I listened to Mortification (The first Christian Death Metal Band). I Liked much the "Hammer of God" album. In the evening Sabin (Bino) came home and we watched some Mortification videos at Youtube. Right now I listen again to "Ever – Idyll" a pretty great song. And yeah I keep listening to a lot, a great radio. Try it if you haven't!END—–

Remove old unused kernels and cleanup orphaned packages on CentOS / RHEL/ Fedora and Debian Linux

Friday, October 23rd, 2020

Reading Time: 4minutes


If you administer CentOS 7 / CentOS  8 bunch of servers it is very likely after one of the scheduled Patch days every 6 months or so, you end up with a multiple Linux OS kernels installed on the system.
In normal situation on a freshly installed CentOS machine only one rpm package is installed on the system with the kernel release shipped with CentOS / RHEL / Fedora distro:
The reason to remove the old unused kernels is very simple, you don't want to have a messy installation and after some of the updates to boot up in a revert back old kernel or if you're pedantic to simply save few megas of space.
Some people choose to have more than one kernel just to make sure, if the new installed one doesn't boot, after a restart from ILO / IDRAC remote console interface you can select to boot the proper kernel. I agree having the old kernel before the system *kernel* upgrade as backup recovery is a good thing but this is a good thing to the point the system gets booted after reboot (you know we sysadmins usually after each major system package upgrade), we like to reboot the system warmly praying and hoping it will boot up next time 🙂

1. Remove CentOS last XX kernels from the OS

Of course removal of old kernels could be managed by a simple

yum remove kernel


One more than one kernel is present you can hence leave only lets say the last 2 installed kernel on the CentOS host (some people prefer to have only one) but just for the sake of having a backup kernel I like more to have last two kernels installed present, to do so run package-cleanup which is contained in yum-utils rpm package CentOS – this is CentOS / Redhat ( RHEL) specific command.

[root@centos ~ ]:#package-cleanup –oldkernels –count=2


–count=number argument – tells how many from the  latest version kernels to get removed.

Note if you don't have the package-cleanup command install yum-utils package:

[root@centos ~ :]#  yum install -y yum-utils


2. RemoveOld kernels from Fedora Linux – leave only the latest 3 installed

This is done with dnf by setting the –-latest-limit arg to negative value to how many last kernels want to keep

[root@fedora ~ ]:# dnf remove $(dnf repoquery –installonly –latest-limit=-3 -q)


3. Set how many kernels you want to be present on system all the time after package upgrades

It is possible to tell CentOS / RHEL / Fedora's on how many kernels show be kept installed on the system, the default configured on Operating system install time is to keep the last 5 installed kernel on the OS. This is controlled from installonly_limit=5 value that is usually as of year 2020 RPM based distributions found under /etc/yum.conf (on CentOS / RHEL) and in /etc/dnf/dnf.conf (in Fedora) configuration file and sets the desired number of kernels present on system after issuing commands yum upgrade / dnf upgrade –refresh etc.
The minimum number to give to  installonly_limit is 2.

4. Remove orphan rpm packages from server

The next thing to do is to check the installed orphan packages to see if we can safely remove them; by orphaned packages we mean all packages which no longer serve a purpose of package dependencies.
Orphan packages are packages who left over from some old dependencies that are no longer needed on the system but just take up space and impose a possible security risk as some of them might end up with time with a public well known and hacked CVE vulnearbility.

Let me try to explain this concept with a quick example: package A is depended on package B, thus, in order to install package A the package B must also be installed. Once the package A is removed the package B might still be installed, hence the package B is now orphaned package.
Here’s how we can safely see the orphan packages we do have on our system:

[root@centos ~ :]# package-cleanup –quiet –leaves –exclude-bin

And here’s how we can delete them:

[root@centos ~ :]# package-cleanup –quiet –leaves –exclude-bin | xargs yum remove -y

The above commands should be launched multiple times, because the packages deleted with the first batch could create additional orphan packages, and so on: be sure to perform these tasks until no orphan packages appear anymore after the first package-cleanup command.


5. Delete Old Kernels and keep only last three ones on Debian / Ubuntu Linux

To do the same on a debian based distribution there is a command is provided by a deb package byobu, if you want to clean up old kernels on Debians :

$ sudo purge-old-kernels –keep 3

That's all folks enjoy ! 🙂


Deny DHCP Address by MAC on Linux

Thursday, October 8th, 2020

Reading Time: 4minutes

Deny DHCP addresses by MAC ignore MAC to not be DHCPD leased on GNU / Linux howto

I have not blogged for a long time due to being on a few weeks vacation and being in home with a small cute baby. However as a hardcore and a bit of dumb System administrator, I have spend some of my vacation and   worked on bringing up the the and the other Websites hosted as a high availvailability ones living on a 2 Webservers running on a Master to Master MySQL Replication backend database, this is oll hosted on  servers, set to run as a round robin DNS hosts on 2 servers one old Lenove ThinkCentre Edge71 as well as a brand new real Lenovo server Lenovo ThinkServer SD350 with 24 CPUs and a 32 GB of RAM
To assure Internet Connectivity is having a good degree of connectivity and ensure websites hosted on both machines is not going to die if one of the 2 pair configured Fiber Optics Internet Providers Bergon.NET has some Issues, I've rented another Internet Provider Line is set bought from the VIVACOM Mobile Fiber Internet provider – that is a 1 Gigabit Fiber Optics Line.
Next to that to guarantee there is no Database, Webserver, MailServer, Memcached and other running services did not hit downtimes due to Electricity power outage, two Powerful Uninterruptable Power Supplies (UPS)  FPS Fortron devices are connected to the servers each of which that could keep the machine and the connected switches and Servers for up to 1 Hour.

The machines are configured to use dhcpd to distributed IP addresses and the Main Node is set to distribute IPs, however as there is a local LAN network with more of a personal Work PCs, Wireless Devices and Testing Computers and few Virtual machines in the Network and the IPs are being distributed in a consequential manner via a ISC DHCP server.

As always to make everything work properly hence, I had again some a bit weird non-standard requirement to make some of the computers within the Network with Static IP addresses and the others to have their IPs received via the DHCP (Dynamic Host Configuration Protocol) and add some filter for some of the Machine MAC Addresses which are configured to have a static IP addresses to prevent the DHCP (daemon) server to automatically reassign IPs to this machines.

After a bit of googling and pondering I've done it and some of the machines, therefore to save others the efforts to look around How to set Certain Computers / Servers Network Card MAC (Interfaces) MAC Addresses  configured on the LAN network to use Static IPs and instruct the DHCP server to ingnore any broadcast IP addresses leases – if they're to be destined to a set of IGNORED MAcs, I came up with this small article.

Here is the DHCP server /etc/dhcpd/dhcpd.conf from my Debian GNU / Linux (Buster) 10.4


option domain-name "pcfreak.lan";
option domain-name-servers,,,;
max-lease-time 891200;
class "black-hole" {
    match substring (hardware, 1, 6);
    ignore booting;
subclass "black-hole" 18:45:91:c3:d9:00;
subclass "black-hole" 70:e2:81:13:44:11;
subclass "black-hole" 70:e2:81:13:44:12;
subclass "black-hole" 00:16:3f:53:5d:11;
subclass "black-hole" 18:45:9b:c6:d9:00;
subclass "black-hole" 16:45:93:c3:d9:09;
subclass "black-hole" 16:45:94:c3:d9:0d;/etc/dhcpd/dhcpd.conf
subclass "black-hole" 60:67:21:3c:20:ec;
subclass "black-hole" 60:67:20:5c:20:ed;
subclass "black-hole" 00:16:3e:0f:48:04;
subclass "black-hole" 00:16:3e:3a:f4:fc;
subclass "black-hole" 50:d4:f5:13:e8:ba;
subclass "black-hole" 50:d4:f5:13:e8:bb;
subnet netmask {
        option routers        ;
        option subnet-mask    ;
host think-server {
        hardware ethernet 70:e2:85:13:44:12;
default-lease-time 691200;
max-lease-time 891200;
log-facility local7;

To spend you copy paste efforts a file with Deny DHCP Address by Mac Linux configuration is here
Of course I have dumped the MAC Addresses to omit a data leaking but I guess the idea behind the MAC ADDR ignore is quite clear

The main configuration doing the trick to ignore a certain MAC ALenovo ThinkServer SD350ddresses that are reachable on the Connected hardware switch on the device is like so:

class "black-hole" {
    match substring (hardware, 1, 6);
    ignore booting;
subclass "black-hole" 18:45:91:c3:d9:00;

The Deny DHCP Address by MAC is described on distribution lists here but it seems the documentation on the topic on how to Deny / IGNORE DHCP Addresses by MAC Address on Linux has been quite obscure and limited online.

As you can see in above config the time via which an IP is freed up and a new IP lease is done from the server is severely maximized as often DHCP servers do use a max-lease-time like 1 hour (3600) seconds:, the reason for increasing the lease time to be to like 10 days time is that the IPs in my network change very rarely so it is a waste of CPU cycles to do a frequent lease.

default-lease-time 691200;
max-lease-time 891200;

As you see to Guarantee resolving works always as expected I have configured – Google Public DNS and OpenDNS IPs

option domain-name-servers,,,;

One hint to make is, after setting up all my desired config in the standard config location /etc/dhcp/dhcpd.conf it is always good idea to test configuration before reloading the running dhcpd process.


root@pcfreak: ~# /usr/sbin/dhcpd -t
Internet Systems Consortium DHCP Server 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit
Config file: /etc/dhcp/dhcpd.conf
Database file: /va/home/hipo/infor/lib/dhcp/dhcpd.leases
PID file: /var/run/

That's all folks with this sample config the IPs under subclass "black-hole", which are a local LAN Static IP Addresses will never be offered leasess anymore from the ISC DHCP.
Hope this stuff helps someone, enjoy and in case if you need a colocation of a server or a website hosting for a really cheap price on this new set High Availlability up described machines open an inquiry on


Set all logs to log to to physical console /dev/tty12 (tty12) on Linux

Wednesday, August 12th, 2020

Reading Time: 4minutes

tty linux-logo how to log everything to last console terminal tty12

Those who administer servers from the days of birth of Linux and who used actively GNU / Linux over the years or any other UNIX knows how practical could be to configure logging of all running services / kernel messages / errors and warnings on a physical console.

Traditionally from the days I was learning Linux basics I was shown how to do this on an old Debian Sarge 3.0 Linux without systemd and on all Linux distributions Redhat 9.0 / Calderas and Mandrakes I've used either as a home systems or for servers. I've always configured output of all messages to go to the last easy to access console /dev/tty12 (for those who never use it console switching under Linux plain text console mode is done with key combination of CTRL + ALT + F1 .. F12.

In recent times however with the introduction of systemd pretty much things changed as messages to console are not handled by /etc/inittab which was used to add and refresh physical consoles tty1, tty2 … tty7 (the default added one on Linux were usually 7), but I had to manually include more respawn lines for each console in /etc/inittab.
Nowadays as of year 2020 Linux distros /etc/inittab is no longer there being obsoleted and console print out of INPUT / OUTPUT messages are handled by systemd.

1. Enable Physical TTYs from TTY8 till TTY12 etc.

The number of default consoles existing in most Linux distributions I've seen is still from tty1 to tty7. Hence to add more tty consoles and be ready to be able to switch out  not only towards tty7 but towards tty12 once you're connected to the server via a remote ILO (Integrated Lights Out) / IdRAC (Dell Remote Access Controller) / IPMI / IMM (Imtegrated Management Module), you have to do it by telling systemd issuing below systemctl commands:


 # systemctl enable getty@tty8.serviceCreated symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty9.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty10.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty11.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty12.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

Once the TTYS tty7 to tty12 are enabled you will be able to switch to this consoles either if you have a physical LCD / CRT monitor or KVM switch connected to the machine mounted on the Rack shelf once you're in the Data Center or will be able to see it once connected remotely via the Management IP Interface (ILO)remote console.

2. Taking screenshot of the physical console TTY with fbcat

For example below is a screenshot of the 10th enabled tty10:


As you can in the screenshot I've used the nice tool fbcat that can be used to make a screenshot of remote console. This is very useful especially if remote access via a SSH client such as PuTTY / MobaXterm is not there but you have only a physical attached monitor access on a DCs that are under a heavy firewall that is preventing anyone to get to the system remotely. For example screenshotting the physical console in case if there is a major hardware failure occurs and you need to dump a hardware error message to a flash drive that will be used to later be handled to technicians to analyize it and exchange the broken server hardware part.

Screenshots of the CLI with fbcat is possible across most Linux distributions where as usual.

In Debian you have to first instal the tool via :

# apt install –yes fbcat

and on RedHats / CentOS / Fedoras

# yum install -y fbcat

Taking screenshot once tool is on the server of whatever you have printed on console is as easy as

# fbcat > tty_name.ppm

Note that you might want to convert the .ppm created picture to png with any converter such as imagemagick's convert command or if you have a GUI perhaps with GNU Image Manipulation Tool (GIMP).

3. Enabling every rsyslog handled message to log to Physical TTY12

To make everything such aserrors, notices, debug, warning messages  become instantly logging towards above added new /dev/tty12.

Open /etc/rsyslog.conf and to the end of the file append below line :

   *.=notice;*.=warn   /dev/tty12

To make rsyslog load its new config restart it:


# systemctl status rsyslog




rsyslog.service – System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-08-10 04:09:36 EEST; 2 days ago
     Docs: man:rsyslogd(8)
 Main PID: 671 (rsyslogd)
    Tasks: 4 (limit: 4915)
   Memory: 12.5M
   CGroup: /system.slice/rsyslog.service
           └─671 /usr/sbin/rsyslogd -n -iNONE


авг 12 00:00:05 pcfreak rsyslogd[671]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="671" x-info="https://www.rsyslo
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


systemctl restart rsyslog

That's all folks navigate by pressing simultaneously CTRL + ALT + F12 to get to TTY12 or use ALT + LEFT / ALT + RIGHT ARROW (console switch commands) till you get to the console where everything should be now logged.

Enjoy and if you like this article share to tell your sysadmin friends about this nice hack  ! 🙂




Check if server is Physical Bare Metal or a Virtual Machine and its type

Tuesday, March 17th, 2020

Reading Time: 4minutes


In modern times the IT employee system administrator / system engineer / security engineer or a developer who has to develop and test code remotely on UNIX hosts, we have to login to multiple of different servers located in separate data centers around the world situated in Hybrid Operating system environments running multitude of different Linux OSes. Often especially for us sysadmins it is important to know whether the remote machine we have SSHed to is physical server (Bare Metal) or a virtual machines running on top of different kind of Hypervisor node OpenXen / Virtualbox / Virtuosso  / VMWare etc.

Then the question comes how to determine whether A remote Installed Linux is Physical or Virtual ?

1. Using the dmesg kernel log utility

The good old dmesg that is used to examine and control the kernel ring buffer detects plenty of useful information which gives you the info whether a server is Virtual or Bare Metal. It is present and accessible on every Linux server out there, thus using it is the best and simplest way to determine the OS system node type.

To grep whether a machine is Virtual and the Hypervisor type use:


nginx:~# dmesg | grep "Hypervisor detected"
[0.000000] Hypervisor detected: KVM

As you see above OS installed is using the KVM Virtualization technology.

An empty output of this command means the Remote OS is installed on a physical computer.


2. Detecting the OS platform the systemd way

Systemd along with the multiple over-complication of things that nearly all sysadmins (including me hate) so much introduced something useful in the fact of hostnamectl command
that could give you the info about the OS chassis platform.


root@pcfreak:~# hostnamectl status
 Static hostname: pcfreak
         Icon name: computer-desktop
           Chassis: desktop
        Machine ID: 02425d67037b8e67cd98bd2800002671
           Boot ID: 34a83b9a79c346168082f7605c2f557c
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-amd64
      Architecture: x86-64



Below is output of a VM running on a Oracle Virtualbox HV.


linux:~# hostnamectl status
Static hostname: ubuntuserver
 Icon name: computer-vm
 Chassis: vm
 Machine ID: 2befe86cf8887ca098f509e457554beb
 Boot ID: 8021c02d65dc46b1885afb25fddcf18c
 Virtualization: oracle
 Operating System: Ubuntu 16.04.1 LTS
 Kernel: Linux 4.4.0-78-generic
 Architecture: x86-64


3. Detect concrete container virtualization with systemd-detect-virt 

Another Bare Metal or VM identify tool that was introducted some time ago by freedesktop project is systemd-detect-virt (usually command is part of systemd package).
It is useful to detect the exact virtualization on a systemd running OS systemd-detect-virt is capable to detect many type of Virtualization type that are rare like: IBM zvm S390 Z/VM, bochs, bhyve (a FreeBSD hypervisor), Mac OS's parallels, lxc (linux containers), docker containers, podman etc.

The output from the command is either none (if no virtualization is present or the VM Hypervisor Host type):


server:~# systemd-detect-virt


quake:~# systemd-detect-virt


4. Install and use facter to report per node facts


debian:~# apt-cache show facter|grep -i desc -A2
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and
 reports per-node facts, which are collected by the Puppet agent and are made

Description-md5: 88cdf9a1db3df211de4539a0570abd0a
Tag: devel::lang:ruby, devel::library, implemented-in::ruby,
root@jeremiah:/home/hipo# apt-cache show facter|grep -i desc -A1
Description-en: collect and display facts about the system
 Facter is Puppet’s cross-platform system profiling library. It discovers and

Description-md5: 88cdf9a1db3df211de4539a0570abd0a


– Install facter on Debian / Ubuntu / deb based Linux


# apt install facter –yes

– Install facter on RedHat / CentOS RPM based distros

# yum install epel-release


# yum install facter

– Install facter on OpenSuSE / SLES

# zypper install facter

Once installed on the system to find out whether the remote Operating System is Virtual:

# facter 2> /dev/null | grep virtual
is_virtual => false
virtual => physical

If the machine is a virtual machine you will get some different output like:

# facter 2> /dev/null | grep virtual
is_virtual => true
virtual => kvm

If you're lazy to grep you can use it with argument.

# facter virtual


6. Use lshw and dmidecode (list hardware configuration tool)

If you don't have the permissions to install facter on the system and you can see whether lshw (list hardware command) is not already present on remote host.

# lshw -class system  
    description: Computer
    width: 64 bits
    capabilities: smbios-2.7 vsyscall32

If the system is virtual you'll get an output similar to:

# lshw -class system  
 description: Computer
 product: VirtualBox
 vendor: innotek GmbH
 version: 1.2
 serial: 0
 width: 64 bits
 capabilities: smbios-2.5 dmi-2.5 vsyscall32
 configuration: family=Virtual Machine uuid=78B58916-4074-42E2-860F-7CAF39F5E6F5

Of course as it provides a verbosity of info on Memory / CPU type / Caches / Cores / Motherboard etc. virtualization used or not can be determined also with dmidecode / hwinfo and other tools that detect the system hardware this is described thoroughfully in my  previous article Get hardware system info on Linux.

7. Detect virtualziation using virt-what or imvirt scripts

imvirt is a little script to determine several virtualization it is pretty similar to virt-what the RedHat own script for platform identification. Even though virt-what is developed for RHEL it is available on other distros, Fedoda, Debian, Ubuntu, Arch Linux (AUR) just like is imvirt.

installing both of them is with the usual apt-get / yum or on Arch Linux with yay package manager (yay -S virt-what) …

Once run the output it produces for physical Dell / HPE / Fujitsu-Siemens Bare Metal servers would be just empty string.

# virt-what

Or if the system is Virtual Machine, you'll get the type, for example KVM (Kernel-based Virtual Machine) / virtualbox / qemu etc.




It was explained how to do a simple check whether the server works on a physical hardware or on a virtual Host hypervisor. The most basic and classic way is with dmesg. If no access to dmesg is due to restrictions you can try the other methods for systemd enabled OSes with hostnamectl / systemd-detect-virt. Other means if the tools are installed or you have the permissions to install them is with facter / lshw or with virt-what / imvirt scripts.
There definitely perhaps much more other useful tools to grasp hardware and virtualization information but this basics could be useful enough for shell scripting purposes.
If you know other tools, please share.