Posts Tagged ‘CentOS’

How to configure static DNS and Search domain for Redhat / CentOS and Redhat Linux

Wednesday, February 3rd, 2021

Reading Time: 2minutes

In latest Redhat based OS-es Fedora / CentOS / Redhat etc. just like on many other Linux distributions, we have /etc/resolv.conf being overwritten by NetworkManager and / or systemd configurations setup which since some time has been introduced to be a "more sophisticated" (default)  so the file is being written by Network Manager / dhcp or systemd config. Though the idea is good, having other programs modify /etc/resolv.conf is a real pain in the ass especially as you end up with an empty file because some service has overwritten what you have placed in the file and the DNS records and Search Domain is deleted forever. If you're not aware of this "new cool" linux feature you might first think that it was a bug that has ovewritten /etc/resolv.conf but ok guys as Bill Gates loved to say "this is not a bug it is a feature", so any attemps you make to manually change /etc/resolv.conf will be soon gone 🙂

This is pretty annoying for old school sysadmins which like to just set the necessery Domain name server resolving

search Subdomain.SearchDomain.Com
nameserver yyy.yyy.yyy.yyy

However as said Nowdays if you just place the desired config with in /etc/resolv.conf on next Server reboot or Network restart (or next fetch of DHCP if the ethernet interface IPs are being obtained via DHCP protocol) you will end up in a situation with an empty /etc/resolv.conf  with one commented line reading:

[root@redhat ~]# cat /etc/resolv.conf
# Generated by NetworkManager

To make the DNS and Search Domain be always presented on any network restart or reboot on the server hence you will need to define  DNS1 DNS2 DNS3 etc. and SEARCH variable inside the network configuration files for Bridge or Network interfaces located in /etc/sysconfig/network-scripts/ifcfg-br0/etc/sysconfig/network-scripts/ifcfg-eno1np0 etc.  that will automatically append above search / nameserver fields in /etc/resolv.conf on any NetworkManager or system restart.
Below is example with the variables added to a Network bridge configuration on Redhat 8.3 (Ootpa):

[root@redhat ~]# cat /etc/sysconfig/network-scripts/ifcfg-br0


To test the configuration does append proper records into /etc/resolv.conf on Network restart  or /sbin/reboot reload the network.

[root@redhat ~]# systemctl restart NetworkManager

The result is you should have a good looking resolv.conf as so

[root@redhat ~]#  cat /etc/resolv.conf
# Generated by NetworkManager


Debug and fix Virtuozzo / KVM broken Hypervisor error: ‘PrlSDKError(‘SDK error: 0x80000249: Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down.’ on CentOS Linux howto

Thursday, January 28th, 2021

Reading Time: 6minutes


I've recently yum upgraded a CentOS Linux server runinng Virtuozzo kernel and Virtuozzo virtualization Virtual Machines to the latest available CentOS Linux release 7.9.2009 (Core) just to find out after the upgrade there was issues with both virtuozzo (VZ) way to list installed VZ enabled VMs reporting Unable to connect to Virtuozzo error like below:

[root@CENTOS etc]# prlctl list -a
Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down. Contact your Virtuozzo administrator for assistance.

Even the native QEMU KVM VMs installed on the Hypervisor system failed to work to list and bring up the VMs producing another unexplainable error with virsh unable to connect to the hypervisor socket

[root@CENTOS etc]# virsh list –all
error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

In dmesg cmd kernel log messages the error found looked as so:

[root@CENTOS etc]# dmesg|grep -i sdk

[    5.314601] PrlSDKError('SDK error: 0x80000249: Unable to connect to Virtuozzo. You may experience a connection problem or the server may be down. Contact your Virtuozzo administrator for assistance.',)

To fix it I had to experiment a bit based on some suggestions from Google results as usual and what turned to be the cause is a now obsolete setting for disk probing that is breaking libvirtd

Disable allow_disk_format_probing in /etc/libvirt/qemu.conf

The fix to PrlSDKError('SDK error: 0x80000249: Unable to connect to Virtuozzo comes to commenting a parameter inside 


which for historical reasons seems to be turned on by default it is like this

allow_disk_format_probing = 1

Resolution is to either change the value to 0 or completely comment the line:

[root@CENTOS etc]# grep allow_disk_format_probing /etc/libvirt/qemu.conf
# If allow_disk_format_probing is enabled, libvirt will probe disk
#allow_disk_format_probing = 1
#allow_disk_format_probing = 1

Debug problems with Virtuozzo services and validate virtualization setup

What really helped to debug the issue was to check the extended status info of libvirtd.service vzevent vz.service libvirtguestd.service prl-disp systemd services

[root@CENTOS etc]# systemctl -l status libvirtd.service vzevent vz.service libvirtguestd.service prl-disp

Here I had to analyze the errors and googled a little bit about it

Once this is changed I had to of course restart libvirtd.service and rest of virtuozzo / kvm services

[root@CENTOS etc]# systemctl restart libvirtd.service ibvirtd.service vzevent vz.service libvirtguest.service prl-disp

Another useful tool part of a standard VZ install that I've used to make sure each of the Host OS Hypervisor components is running smoothly is virt-host-validate(tool is part of libvirt-client rpm package)

[root@CENTOS etc]# virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpu' controller mount-point                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'cpuset' controller mount-point                  : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'devices' controller mount-point                 : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for cgroup 'blkio' controller mount-point                   : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller mount-point                  : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpu' controller mount-point                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuacct' controller mount-point                 : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'cpuset' controller mount-point                  : PASS
   LXC: Checking for cgroup 'devices' controller support                     : PASS
   LXC: Checking for cgroup 'devices' controller mount-point                 : PASS
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking for cgroup 'blkio' controller mount-point                   : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

One thing to note here that virt-host-validate helped me to realize the  fuse (File system in userspace) module kernel support enabled on the HV was missing so I've enabled temporary for this boot with modprobe and permanently via a configuration like so:

# to load it one time
[root@CENTOS etc]#  modprobe fuse
# to load fuse permnanently on next boot

[root@CENTOS etc]#  echo fuse >> /etc/modules-load.d/fuse.conf

Disable selinux on CentOS HV

Another thing was selinux was enabled on the HV. Selinux is really annoying thing and to be honest I never used it on any server and though its idea is quite good the consequences it creates for daily sysadmin work are terrible so I usually disable it. It could be that a Hypervisor Host OS might work just normal with the selinux enabled but just in case I decided to remove it. This is how

[root@CENTOS etc]#  sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

To temporarily change the SELinux mode from targeted to permissive with the following command:

[root@CENTOS etc]#  setenforce 0

Edit /etc/selinux/config file and set the SELINUX mod to disabled

[root@CENTOS etc]# vim /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing – SELinux security policy is enforced.
#       permissive – SELinux prints warnings instead of enforcing.
#       disabled – No SELinux policy is loaded.
# SELINUXTYPE= can take one of these two values:
#       targeted – Targeted processes are protected,
#       mls – Multi Level Security protection.

Finally rebooted graceously the machine just in case with the good recommended way to reboot servers with shutdown command instead of /sbin/reboot

[root@CENTOS etc]# shutdown -r now

The advantage of shutdown is that it tries to shutdown each service by sending stop requests but usually this takes some time and even a shutdown request could take longer to proccess as each service such as a WebServer application is being waited to close all its network connections etc. |
However if you want to have a quick reboot and you don't care about any established network connections to third party IPs you can go for the brutal old fashioned /sbin/reboot 🙂

Listing installed RPMs by vendor installed on CentOS / RedHat Linux

Friday, January 8th, 2021

Reading Time: 3minutes

Listing installed RPMs by vendor installed on CentOS / RedHat Linux

Listing installed RPMs by vendor is useful sysadmin stuff if you have third party software installed that is not part of official CentOS / RedHat Linux and you want to only list this packages, here is how this is done


[root@redhat ~]# rpm -qa –qf '%{NAME} %{VENDOR} %{PACKAGER} \n' | grep -v 'CentOS' | sort

criu Virtuozzo Virtuozzo (
gskcrypt64 IBM IBM
gskssl64 IBM IBM
ipxe-roms-qemu Virtuozzo Virtuozzo (
libevent (none) (none)
libguestfs-appliance Virtuozzo Virtuozzo (
libguestfs-tools-c Virtuozzo Virtuozzo (
libguestfs Virtuozzo Virtuozzo (
libprlcommon Virtuozzo Virtuozzo (
libprlsdk-python Virtuozzo Virtuozzo (
libprlsdk Virtuozzo Virtuozzo (
libprlxmlmodel Virtuozzo Virtuozzo (
libtcmu Virtuozzo Virtuozzo (
libvcmmd Virtuozzo Virtuozzo (
libvirt-client Virtuozzo Virtuozzo (
libvirt-daemon-config-nwfilter Virtuozzo Virtuozzo (
libvirt-daemon-driver-interface Virtuozzo Virtuozzo (
libvirt-daemon-driver-network Virtuozzo Virtuozzo (
libvirt-daemon-driver-nodedev Virtuozzo Virtuozzo (
libvirt-daemon-driver-nwfilter Virtuozzo Virtuozzo (
libvirt-daemon-driver-qemu Virtuozzo Virtuozzo (
libvirt-daemon-driver-storage-core Virtuozzo Virtuozzo (
libvirt-daemon-driver-storage Virtuozzo Virtuozzo (
libvirt-daemon-kvm Virtuozzo Virtuozzo (
libvirt-daemon Virtuozzo Virtuozzo (
libvirt-libs Virtuozzo Virtuozzo (
libvirt-python Virtuozzo Virtuozzo (
libvirt Virtuozzo Virtuozzo (
libvzctl Virtuozzo Virtuozzo (
libvzevent Virtuozzo Virtuozzo (
openvz-logos Virtuozzo Virtuozzo (
p7zip-plugins Fedora Project Fedora Project
ploop-lib Virtuozzo Virtuozzo (
ploop Virtuozzo Virtuozzo (
prlctl Virtuozzo Virtuozzo (
prl-disk-tool Virtuozzo Virtuozzo (
prl-disp-service Virtuozzo Virtuozzo (
python2-lockfile Fedora Project Fedora Project
python2-psutil Fedora Project Fedora Project
python-daemon Fedora Project Fedora Project
python-subprocess32 Virtuozzo Virtuozzo (
qemu-img-vz Virtuozzo Virtuozzo (
qemu-kvm-common-vz Virtuozzo Virtuozzo (
qemu-kvm-vz Virtuozzo Virtuozzo (
qt Virtuozzo Virtuozzo (
rkhunter Fedora Project Fedora Project
seabios-bin Virtuozzo Virtuozzo (
seavgabios-bin Virtuozzo Virtuozzo (
spfs Virtuozzo Virtuozzo (
TIVsm-API64 IBM (none)
TIVsm-APIcit IBM (none)
TIVsm-BAcit IBM (none)
TIVsm-BA IBM (none)
vcmmd Virtuozzo Virtuozzo (
vmauth Virtuozzo Virtuozzo (
vzctl Virtuozzo Virtuozzo (
vzkernel Virtuozzo Virtuozzo (
vzkernel Virtuozzo Virtuozzo (
vztt_checker Virtuozzo Virtuozzo (
vztt_checker Virtuozzo Virtuozzo (
vztt-lib Virtuozzo Virtuozzo (
vztt Virtuozzo Virtuozzo (
zabbix-agent (none) (none)


That instructs rpm to output each package's name and vendor, then we exclude those from "Red Hat, Inc." (which is the exact string Red Hat conveniently uses in the "vendor" field of all RPMs they pacakge).

By default, rpm -qa uses the format '%{NAME}-%{VERSION}-%{RELEASE}', and it's nice to see version and release, and on 64-bit systems, it's also nice to see the architecture since both 32- and 64-bit packages are often installed. Here's how I did that:

[root@redhat ~]# rpm -qa –qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR} %{PACKAGER} \n' | grep -v 'CentOS' | sort

criu- Virtuozzo Virtuozzo (
gskcrypt64-8.0-55.17.x86_64 IBM IBM
gskssl64-8.0-55.17.x86_64 IBM IBM
ipxe-roms-qemu-20170123-1.git4e85b27.1.vz7.5.noarch Virtuozzo Virtuozzo (
libevent-2.0.22-1.rhel7.x86_64 (none) (none)
libguestfs-1.36.10-6.2.vz7.12.x86_64 Virtuozzo Virtuozzo (
libguestfs-appliance-1.36.10-6.2.vz7.12.x86_64 Virtuozzo Virtuozzo (
libguestfs-tools-c-1.36.10-6.2.vz7.12.x86_64 Virtuozzo Virtuozzo (
libprlcommon-7.0.162-1.vz7.x86_64 Virtuozzo Virtuozzo (
libprlsdk-7.0.226-2.vz7.x86_64 Virtuozzo Virtuozzo (
libprlsdk-python-7.0.226-2.vz7.x86_64 Virtuozzo Virtuozzo (
libprlxmlmodel-7.0.80-1.vz7.x86_64 Virtuozzo Virtuozzo (
libtcmu-1.2.0-16.2.vz7.x86_64 Virtuozzo Virtuozzo (
libvcmmd-7.0.22-3.vz7.x86_64 Virtuozzo Virtuozzo (
libvirt-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-client-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-config-nwfilter-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-driver-interface-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-driver-network-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-driver-nodedev-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-driver-nwfilter-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-driver-qemu-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-driver-storage-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-driver-storage-core-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-daemon-kvm-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-libs-3.9.0-14.vz7.38.x86_64 Virtuozzo Virtuozzo (
libvirt-python-3.9.0-1.vz7.1.x86_64 Virtuozzo Virtuozzo (
libvzctl-7.0.506-1.vz7.x86_64 Virtuozzo Virtuozzo (
libvzevent-7.0.7-5.vz7.x86_64 Virtuozzo Virtuozzo (
openvz-logos-70.0.13-1.vz7.noarch Virtuozzo Virtuozzo (
p7zip-plugins-16.02-10.el7.x86_64 Fedora Project Fedora Project
ploop-7.0.137-1.vz7.x86_64 Virtuozzo Virtuozzo (
ploop-lib-7.0.137-1.vz7.x86_64 Virtuozzo Virtuozzo (
prlctl-7.0.164-1.vz7.x86_64 Virtuozzo Virtuozzo (
prl-disk-tool-7.0.43-1.vz7.x86_64 Virtuozzo Virtuozzo (
prl-disp-service-7.0.925-1.vz7.x86_64 Virtuozzo Virtuozzo (
python2-lockfile-0.11.0-17.el7.noarch Fedora Project Fedora Project
python2-psutil-5.6.7-1.el7.x86_64 Fedora Project Fedora Project
python-daemon-1.6-4.el7.noarch Fedora Project Fedora Project
python-subprocess32-3.2.7-1.vz7.5.x86_64 Virtuozzo Virtuozzo (
qemu-img-vz-2.10.0-21.7.vz7.67.x86_64 Virtuozzo Virtuozzo (
qemu-kvm-common-vz-2.10.0-21.7.vz7.67.x86_64 Virtuozzo Virtuozzo (
qemu-kvm-vz-2.10.0-21.7.vz7.67.x86_64 Virtuozzo Virtuozzo (
qt-4.8.7-2.vz7.2.x86_64 Virtuozzo Virtuozzo (
rkhunter-1.4.6-2.el7.noarch Fedora Project Fedora Project
seabios-bin-1.10.2-3.1.vz7.3.noarch Virtuozzo Virtuozzo (
seavgabios-bin-1.10.2-3.1.vz7.3.noarch Virtuozzo Virtuozzo (
spfs-0.09.0010-1.vz7.x86_64 Virtuozzo Virtuozzo (
TIVsm-API64-8.1.11-0.x86_64 IBM (none)
TIVsm-APIcit-8.1.11-0.x86_64 IBM (none)
TIVsm-BA-8.1.11-0.x86_64 IBM (none)
TIVsm-BAcit-8.1.11-0.x86_64 IBM (none)
vcmmd-7.0.160-1.vz7.x86_64 Virtuozzo Virtuozzo (
vmauth-7.0.10-2.vz7.x86_64 Virtuozzo Virtuozzo (
vzctl-7.0.194-1.vz7.x86_64 Virtuozzo Virtuozzo (
vzkernel-3.10.0-862.11.6.vz7.64.7.x86_64 Virtuozzo Virtuozzo (
vzkernel-3.10.0-862.20.2.vz7.73.29.x86_64 Virtuozzo Virtuozzo (
vztt-7.0.63-1.vz7.x86_64 Virtuozzo Virtuozzo (
vztt_checker-7.0.2-1.vz7.i686 Virtuozzo Virtuozzo (
vztt_checker-7.0.2-1.vz7.x86_64 Virtuozzo Virtuozzo (
vztt-lib-7.0.63-1.vz7.x86_64 Virtuozzo Virtuozzo (
zabbix-agent-3.2.11-1.el7.x86_64 (none) (none)

Howto Upgrade IBM Spectrum Protect Backup Client TSM 7.X to 8.1.8, Update Tivoli 8.1.8 to 8.1.11 on CentOS and Redhat Linux

Thursday, December 3rd, 2020

Reading Time: 6minutes



Having another day of a system administrator boredom, we had a task to upgrade some Tivoli TSM Backup clients running on a 20+ machines powered by CentOS and RHEL Linux to prepare the systems to be on the latest patched IBM Spectrum Backup client version available from IBM. For the task of patching I've used a central server where, I've initially downloaded the provided TSM client binaries archives. From this machine, we have copied TivSM*.tar to each and every system that needs to be patched and then patched. The task is not too complex as the running TSM in the machines are all at the same version and all running a recent patched version of Linux. Hence to make sure all works as expected we have tested TSM is upgraded from 7.X.X to 8.X.X on one machine and then test 8.1.8 to 8.1.11 upgrade on another one. Once having confirmed that Backups works as expected after upgrade. We have proceeded to do it massively on each of the rest 20+ hosts.
Below article's goal is to help some lazy sysadmin with the task to prepare an TSM Backup upgrade procedure to standartize TSM Upgrade, which as many of the IBM's softwares is very specific and its upgrade requires, a bit of manual work and extra cautious as there seems to be no easy way (or at least I don't know it), to do the upgrade by simply adding an RPM repository and doing, something like yum install tivsm*.

0. Check if there is at least 2G free of space

According to documentation the minimum space you need to a functional install without having it half installed or filling up your filesystem is 2 Gigabytes of Free Memory on a filesystem where the .tar and rpms will be living.

Thus check what is the situation with your filesystem where you wills store the .tar archice and extract .RPM files / install the RPM files.

# df -h

1. Download the correct tarball with 8.1 Client

On one central machine you would need to download the Tivoli you can do that via wget / curl / lynx whatever is at hand on the Linux server.

As of time of writting this article TSM's 8.1.11 location is at

I've made a local download mirror of Tivoli TSM 8.1.11 here.
In case you need to install IBM Spectrum Backup Client to a PCI secured environment to a DMZ-ed LAN network from a work PC you can Download it first from your local PC and via Citrix client upload program or WinSCP upload it to a central replication host from where you will later copy to each of the other server nodes that needs to be upgraded.

Lets Copy archive to all Server hosts where you want it later installed, using a small hack

Assuming you already have an Excel document or a Plain text document with all the IPs of the affected hosts where you will need to get TSM upgraded. Extract this data and from it create a plain text file /home/user/hosts.txt containing all the machine IPs lined up separated with carriage return separations (\n), so you can loop over each one and use scp to send the files.

– Replicate Tivoli tar to all machine hosts where you want to get IBM Spectrum installed or upgraded.
Do it with a loop like this:

# for i in $(cat hosts.txt); do scp user@$i:/home/user/; done

 Copy to a Copy buffer temporary your server password assuming all your passwords to each machine are identical and paste your login user pass for each host to initiate transfer

2. SSH to each of the Machine hosts IPs

Once you login to the host you want to upgrade
Go to your user $HOME/home/user and create files where we'll temporary store Tivoli archive files and extract RPMs

[root@linux-server user]# mkdir -p ~/tsm/TSM_BCK/
[root@linux-server user]# mv ~/tsm
[root@linux-server user]# cd tsm
[root@linux-server user]# tar -xvvf

3. Create backup of old backup files

It is always a good idea to keep old backup files

[root@linux-server tsm]# cp -av /opt/tivoli/tsm/client/ba/bin/dsm.opt ~/tsm/TSM_BCK/dsm.opt_bak_$(date +'%Y_%M_%H')
[root@linux-server tsm]# cp -av /opt/tivoli/tsm/client/ba/bin/dsm.sys ~/tsm/TSM_BCK/dsm.sys_bak_$(date +'%Y_%M_%H')

[root@linux-server tsm]# [[ -f /etc/adsm/TSM.PWD ]] && cp -av /etc/adsm/TSM.PWD ~/TSM_BCK/ || echo 'file doesnt exist'

/etc/adsm/TSM.PWD this file is only there as legacy for TSM it contained encrypted passwords inver 7 for updates. In TSM v.8 encryption file is not there as new mechanism for sensitive data was introduced.
Be aware that from Tivoli 8.X it will return error

!! Note – if dsm.opt , dsm.sys files are on different locations – please use correct full path locations !!

4. Stop  dsmcad – TSM Service daemon

[root@linux-server tsm]# systemctl stop dsmcad

5. Locate and deinstall all old Clients

Depending on the version to upgrade if you're upgrading from TSM version 7 to 8, you will get output like.

[root@linux-server tsm]# rpm -qa | grep 'TIVsm-'

If you're one of this paranoid admins you can remove TIVsm packs  one by one.

[root@linux-server tsm]# rpm -e TIVsm-BA-7.1.6-2.x86_64
[root@linux-server tsm]# rpm -e TIVsm-API64-7.1.6-2.x86_64

Instead if upgrading from version 8.1.8 to 8.1.11 due to the Security CVE advisory recently published by IBM e.g. (IBM Runtime Vulnerability affects IBM Spectrum Backup archive Client) and  vulnerability in Apache Commons Log4J affecting IBM Spectrum Protect Backup Archive Client.

[root@linux-server tsm]# rpm -qa | grep 'TIVsm-'

Assuming you're not scared of a bit automation you can straight do it with below one liner too 🙂

# rpm -e $(rpm -qa | grep TIVsm)

[root@linux-server tsm]# rpm -qa | grep gsk
[root@linux-server tsm]# rpm -e gskcrypt64 gskssl64

6. Check uninstallation success:

[root@linux-server tsm]# rpm -qa | grep TIVsm
[root@linux-server tsm]# rpm -qa | grep gsk

Here you should an Empty output, if packages are not on the system, e.g. Empty output is good output ! 🙂

7. Install new client IBM Spectrum Client (Tivoli Storage Manager) and lib dependencies

[root@linux-server tsm]# rpm -ivh gskcrypt64-
[root@linux-server tsm]# rpm -ivh gskssl64-

 If you're lazy to type you can do as well

[root@linux-server tsm]# rpm -Uvh gsk*

Next step is to install main Tivoli SM components the the API files and BA (The Backup Archive Client)

[root@linux-server tsm]# rpm -ivh TIVsm-API64.x86_64.rpm
[root@linux-server tsm]# rpm -ivh TIVsm-BA.x86_64.rpm

If you have to do it on multiple servers and you do it manually following a guide like this, you might instead want to install them with one liner.

[root@linux-server tsm]# rpm -ivh TIVsm-API64.x86_64.rpm TIVsm-BA.x86_64.rpm

There are some Not mandatory "Common Inventory Technology" components (at some cases if you're using the API install it we did not need that), just for the sake if you need them on your servers due to backup architecture, install also below commented rpm files.

## rpm -ivh TIVsm-APIcit.x86_64.rpm

## rpm -ivh TIVsm-BAcit.x86_64.rpm

These packages not needed only for operation WebGUI TSM GUI management, (JBB) Journal Based Backup, BAhdw (the ONTAP library)

— TIVsm-WEBGUI.x86_64.rpm
— TIVsm-JBB.x86_64.rpm
— TIVsm-BAhdw.x86_64.rpm

8. Start and enable dsmcad service

[root@linux-server tsm]# systemctl stop dsmcad

You will get

##Warning: dsmcad.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[root@linux-server tsm]# systemctl daemon-reload

[root@linux-server tsm]# systemctl start dsmcad

## enable dsmcad – it is disabled by default after install

[root@linux-server ~]# systemctl enable dsmcad

[root@linux-server tsm]# systemctl status dsmcad

9. Check dmscad service is really running

Once enabled IBM TSM will spawn a process in the bacground dmscad if it started properly you should have the process backgrounded.

[root@linux-server tsm]# ps -ef|grep -i dsm|grep -v grep
root      2881     1  0 18:05 ?        00:00:01 /usr/bin/dsmcad

If process is not there there might be some library or something not at place preventing the process to start …

10. Check DSMCAD /var/tsm logs for errors

After having dsmcad process enabled and running in background

[root@linux-server tsm]# grep -i Version /var/tsm/sched.log|tail -1
12/03/2020 18:06:29   Server Version 8, Release 1, Level 10.000


[root@linux-server tsm]# cat /var/tsm/dsmerror.log

To see the current TSM configuration files we can  grep out comments *

[root@linux-server tsm]# grep -v '*' /opt/tivoli/tsm/client/ba/bin/dsm.sys

Example Configuration of the agent:
   *TSM SERVER NODE Location
   Servername           tsm_server
   COMMmethod           TCPip
   TCPPort              1400
   Passwordaccess       generate
   SCHEDLOGNAME         /var/tsm/sched.log
   MANAGEDServices      schedule
   ERRORLOGNAME         /var/tsm/dsmerror.log
   INCLEXCL             /opt/tivoli/tsm/client/ba/bin/inclexcl.tsm

11. Remove tsm install directory tar ball and rpms to save space on system

The current version of Tivoli service manager is 586 Megabytes.

[root@linux-server tsm]# du -hsc

Some systems are on purpose configured to have less space under their /home directory,
hence it is a good idea to clear up unnecessery files after completion.

Lets get rid of all the IBM Spectrum archive source files and the rest of RPMs used for installation.

[root@linux-server tsm]# rm -rf ~/tsm/{*.tar,*.rpm,*.gpg,*.htm,*.txt}

12. Check backups are really created on the configured remote Central backup server

To make sure after the upgrade the backups are continuously created and properly stored on the IBM Tivoly remote central backup server, either manually initiate a backup or wait for lets say a day and run dsmc client to show all created backups from previous day. To make sure you'll not get empty output you can on purpose modify some file by simply opening it and writting over without chaning anything e.g. modify your ~/.bashrc or ~/.bash_profile

## List all backups for '/' root directory from -fromdate='DD/MM/YY'

[root@linux-server tsm]# dsmc
IBM Spectrum Protect
Command Line Backup-Archive Client Interface
  Client Version 8, Release 1, Level 11.0
  Client date/time: 12/03/2020 18:14:03
(c) Copyright by IBM Corporation and other(s) 1990, 2020. All Rights Reserved.

Session established with server TSM2_SERVER: AIX
  Server Version 8, Release 1, Level 10.000
  Server date/time: 12/03/2020 18:14:04  Last access: 12/03/2020 18:06:29
Protect> query backup -subdir=yes "/" -fromdate=12/3/2020
           Size        Backup Date                Mgmt Class           A/I File
           —-        ———–                ———-           — —-
         6,776  B  12/03/2020 01:26:53             DEFAULT              A  /etc/freshclam.conf
         6,685  B  12/03/2020 01:26:53             DEFAULT              A  /etc/freshclam.conf-2020-12-02
         5,602  B  12/03/2020 01:26:53             DEFAULT              A  /etc/hosts
         5,506  B  12/03/2020 01:26:53             DEFAULT              A  /etc/hosts-2020-12-02
           398  B  12/03/2020 01:26:53             DEFAULT              A  /opt/tivoli/tsm/client/ba/bin/tsmstats.ini
       114,328  B  12/03/2020 01:26:53             DEFAULT              A  /root/.bash_history
           403  B  12/03/2020 01:26:53             DEFAULT              A  /root/.lesshst

Fix eth changing network interface names from new Linux naming scheme ens, eno, em1 to legacy eth0, eth1, eth2 on CentOS Linux

Thursday, January 16th, 2020

Reading Time: 7minutes


On CentOS / RHEL 7 / Fedora 19+ and other Linux distributions, the default network eth0, eth1 .. interface naming scheme has been changed and in newer Linux kernels OS-es to names such as – ens3 , eno1, enp5s2, em1 etc.,  well known old scheme for eth* is now considered a legacy.
This new Network card naming in Linux OS is due to changes made in Kernel / modules and udev  rules which resembles how Ethernet ifaces are named on other UNIX like systems.
The weird name is taken depending on the Hardware Network card vendor name and is a standard for years in FreeBSD and Mac OSX, however this was not so over the years,
so for old school sysadmins that's pretty annoying as, we're much used to the eth0 / eth1 / eth2 / eth3 naming standard which brought some clearness on the network card naming.

Also for systems which are upgraded from old Linux OS distro releases to a newer ones, that includes this great new "cool" feature, that fits so well the New age-of computing Cloud craziness.
That behaviour could create a number of problems, especially if the already Production working servers due to failure to bring up some of the network devices after the upgrade or, even if you fix that by editting the /etc/network* / etc/sysconfig/networking/* by hand still there is even more stuff that won't work properly, such as any custom made iptables / ipset firewalls rules, or any kind of custom used third party Shell / Perl scripts that depend on the old-school conventional and (convenient easy to remember!!!) eth0, eth2 etc. naming

For sysadmins who are using some kind of Application Clustering with something like corosync / pacemaker this new fuzzy improvement makes things even worse as having a changed interface name of the card will break the cluster …


1. Get list of the LAN Card Server hardware


To get a better view on the server installed and recognized LAN Cards use lspci / dmidecode commands:

 lspci |grep -i Ether -A1 -B1
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (r                                                                                                           ev 03)
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
05:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
05:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
05:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
05:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev                                                                                                            01)
7f:08.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)


lspci reports all attached LAN Cards to server which are plugged in on the Motherbord, since that specific server has a Motherboard integrated LAN Adapters too, we can see this one
via dmidecode.

# dmidecode |grep -i Ether -A 5 -B 5

Handle 0x00C5, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: Embedded LOM 1 Port 3
        Type: Ethernet
        Status: Enabled
        Type Instance: 3
        Bus Address: 0000:XX:00.X

Handle 0x00C6, DMI type 41, 11 bytes
Onboard Device
        Reference Designation: Embedded LOM 1 Port 4
        Type: Ethernet
        Status: Enabled
        Type Instance: 4
        Bus Address: 0000:0X:00.X

Handle 0x00C7, DMI type 41, 11 bytes

                HP Ethernet 1Gb 4-port 331T Adapter – NIC
                Slot 2

Handle 0x00E3, DMI type 203, 34 bytes
OEM-specific Type
        Header and Data:


The illustrate the eth0 changing name issue, here is example taken from server on how eth1 interface is named on a new CentOS install:

# ip addr show

eno1: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 6c:0b:84:6c:48:1c brd ff:ff:ff:ff:ff:ff
inet brd scope global eno1
inet6 2606:b400:c00:48:6e0b:84ff:fe6c:481c/128 scope global dynamic
valid_lft 2326384sec preferred_lft 339184sec
inet6 fe80::6e0b:84ff:fe6c:481c/64 scope link
valid_lft forever preferred_lft forever



2. Disable Network Manager on the server

To prevent potential problems for future with randomly changing Network card names order on reboots and other mess,
it is generally a good idea to disable Network Manager.


# systemctl disable NetworkManager
rm '/etc/systemd/system/'
rm '/etc/systemd/system/dbus-org.freedesktop.NetworkManager.service'
rm '/etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service'


3. Check and correct network configuration if necessery in  /etc/sysconfig/network-scripts/ifcfg-*

Either fix the naming across all files ifcfg-* to match eth0 / eth1 / ethXX or even better both change the NAME and DEVICE in files and completely rename the files ifcfg-eno1 to ifcfg-eth1 ..
ifcfg-enoXX to ifcfg-ethXX

server:~# cat /etc/sysconfig/network-scripts/ifcfg-eno1


4. Fix the interface scheme naming through passing a GRUB boot parameter to Kernel


a. Create backup of /etc/default/grub

cp -rpf /etc/default/grub /etc/default/grub_bak_date +"%Y_%m_%Y"

b. Edit /etc/default/grub

c. Find config parameter GRUB_CMDLINE_LINUX

d. Add net.ifnames=0 biosdevname=0 to the line


net.ifnames=0 biosdevname=0

After the change the line should look like

GRUB_CMDLINE_LINUX=" crashkernel=auto net.ifnames=0 biosdevname=0 rhgb quiet"


e. Regenerate GRUB loader to have included the new config

server:~# grub2-mkconfig -o /boot/grub2/grub.cfg

f. Reboot the sytem

server:~# shutdown -r now


5. Fix auto-generated inconvenient naming by modifying udev rules

The Mellanox Ehternet server card vendor's workaround to the ever changing eth names is modify udev rules to be able to have the ordinary eth0 / eth1 / eth2 … Lan card name scheme.
In short this is recommended for Mellanox but should work on any other Lan card device attached on a Linux powered server.

# cat /etc/sysconfig/network-scripts/ifcfg-eth1


# cat /etc/sysconfig/network-scripts/ifcfg-eth2


# vi /etc/udev/rules.d/70-persistent-net.rules

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="7c:fe:90:cb:76:02", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="7c:fe:90:cb:76:03", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"


Next step is to reboot.

# /sbin/reboot

After a while when the server boots check with ip or ifconfig the configuration to make sure the ethXX ordering is proper again.


# /sbin/ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet netmask broadcast
inet6 fe80::7efe:90ff:fecb:7602 prefixlen 64 scopeid 0x20<link>
ether 7c:fe:90:cb:76:02 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23 bytes 3208 (3.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 7c:fe:90:cb:76:03 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

On some Linux distributions, if it happens this udev extra configuration is not venerated, use net.ifnames=0 biosdevname=0 grub configuration.

6. Verify eth interfaces are present    

# ip addr show


eth0: [BROADCAST,MULTICAST,UP,LOWER_UP] mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 6c:0b:84:6c:48:1c brd ff:ff:ff:ff:ff:ff

inet brd scope global eno1

inet6 2606:b400:c00:48:6e0b:84ff:fe6c:481c/128 scope global dynamic

valid_lft 2326384sec preferred_lft 339184sec

inet6 fe80::6e0b:84ff:fe6c:481c/64 scope link

valid_lft forever preferred_lft forever

That's all this should put an end to the annoying auto generated naming lan device naming.




So what was explained up was how to resolve problems caused by autogenerated ethernet interface cards by a new functionality in the Linux kernel, so Network cards are again visible via ip address show / ifconfig again in a proper order eth0 / eth1 / eth2 / eth3 etc. instead of a vendor generated cryptic names as ens / eno / em etc. This is possible via either by editing udev rules or grub configuration. Doing so saves nerves and makes sysadmin life better, at least it did mine.
That's all this should put an end to the annoying auto generated naming.

How to install jcmd on CentOS 7 to diagnose running Java Virtual Machine crashing applications

Tuesday, January 14th, 2020

Reading Time: 3minutes


jcmd utility is well known in the Brane New wonderful world of Java but if you're like me a classical old school sysadmins and non-java developer you probably never heard it hence before going straight into how to install it on CentOS 7 Linux servers, I'll shortly say few words on what it is.
jcmd is used to send diagnostic requests to running Java Virtual Machine (JVM) it is available in both in Oracle Java as well as OpenJDK.
The requests jcmd sends to VM are based on the running Java PID ID and are pretty useful for controlling Java Flight Recordings, troubleshoot, and diagnose JVM and Java Applications. It must be used on the same machine where the JVM is running.

 Used without arguments or with the -l option, jcmd prints the list of running Java processes with their process id, their main class and their command line arguments.
When a main class is specified on the command line, jcmd sends the diagnostic command request to all Java processes for which the command line argument is a substring of the Java process' main class.
jcmd could be useful if the JConsoleJMX (Java Management Extensions) can't be used for some reason on the server or together with Java Visual VM (visual interface for viewing detailed info about Java App).

In most Linux distributionsas as  of year 2020 jcmd is found in  java-*-openjdk-headless.

To have jcmd on lets say Debian GNU / Linux, you're up to something like:


apt-get install –yes openjdk-12-jdk-headless


apt-get install openjdk-11-jdk-headless
however in CentOS
7 jcmd is not found in java*openjdk*headless but instead to have it on server, thus it take me a while to look up where it is foundso after hearing
from some online post it is part of package java*openjdk*devel* to make sure this so true, I've used the  –download-only option

 yum install –downloadonly –downloaddir=/tmp java-1.8.0-openjdk-devel


So the next question was how to inspect the downloaded rpm package into /tmp usually, this is possible via Midnight Commander (mc) easily to view contents, however as this
server did not have installed mc due to security policies I had to do it differently after pondering a while on how to to list the RPM package file content come up using following command



 rpm2cpio java-1.8.0-openjdk-devel- |cpio -idmv|grep -i jcmd|less


To then install  java-1.8.0-openjdk-devel- to do so run:




yum -y install  java-1.8.0-openjdk-devel-1.8.0*

Once the jcmd, I've created the following bash script  that was set that was tracking for application errors and checking whether the JBoss application server pool-available-count is not filled up and hence jboss refuses to serve connections  through query automatically launching jcmd to get various diagnostic data about Java Virtual Machine (e.g. a running snapshot) – think of it like the UNIX top for debugging or Windows System Monitor but run one time. 




# PID_OF_JAVA=$(pgrep -l java)
# jcmd $PID_OF_JAVA GC.heap_dump GC.heap_dump_file-$(date '+%Y-%m-%d_%H-%M-%S').jfr
# jcmd $PID_OF_JAVA Thread.print > Thread.print-$(date '+%Y-%m-%d_%H-%M-%S').jfr

The produced log files can then be used by the developer to visualize some Java specific stuff "Flight recordings" like in below screenshot:


If you're interested on some other interesting tools that can be used to Monitor  and Debug a Running Java VM take a look at Java's official documentation Monitoring Tools.
So that's all Mission Accomplished 🙂 Now the Java Application developer could observe the log and tell why exactly the application crashed after the multitude of thrown Exceptions in the JBoss server.log.

Ansible Quick Start Cheatsheet for Linux admins and DevOps engineers

Wednesday, October 24th, 2018

Reading Time: 6minutes


Ansible is widely used (Configuration management, deployment, and task execution system) nowadays for mass service depoyments on multiple servers and Clustered environments like, Kubernetes clusters (with multiple pods replicas) virtual swarms running XEN / IPKVM virtualization hosting multiple nodes etc. .

Ansible can be used to configure or deploy GNU / Linux tools and services such as Apache / Squid / Nginx / MySQL / PostgreSQL. etc. It is pretty much like Puppet (server / services lifecycle management) tool , except its less-complecated to start with makes it often a choose as a tool for mass deployment (devops) automation.

Ansible is used for multi-node deployments and remote-task execution on group of servers, the big pro of it it does all its stuff over simple SSH on the remote nodes (servers) and does not require extra services or listening daemons like with Puppet. It combined with Docker containerization is used very much for later deploying later on inside Cloud environments such as Amazon AWS / Google Cloud Platform / SAP HANA / OpenStack etc.


0. Instaling ansible on Debian / Ubuntu Linux

Ansible is a python script and because of that depends heavily on python so to make it running, you will need to have a working python installed on local and remote servers.

Ansible is as easy to install as running the apt cmd:


# apt-get install –yes ansible

The following additional packages will be installed:
  ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
Suggested packages:
  sshpass python-jinja2-doc ipython python-netaddr-docs python-gssapi
Recommended packages:
The following NEW packages will be installed:
  ansible ieee-data python-jinja2 python-kerberos python-markupsafe python-netaddr python-paramiko python-selinux python-xmltodict python-yaml
0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded.
Need to get 3,413 kB of archives.
After this operation, 22.8 MB of additional disk space will be used.

apt-get install –yes sshpass


Installing Ansible on Fedora Linux is done with:


# dnf install ansible –yes sshpass


On CentOS to install:

# yum install ansible –yes sshpass

sshpass needs to be installed only if you plan touse ssh password prompt authentication with ansible.

Ansible is also installable via python-pip tool, if you need to install a specific version of ansible you have to use it instead, the package is available as an installable package on most linux distros.

Ansible has a lot of pros and cons and there are multiple articles already written on people for and against it in favour of Chef or Puppet As I recently started learning Ansible. The most important thing to know about Ansible is though many of the things can be done directly using a simple command line, the tool is planned for remote installing of server services using a specially prepared .yaml format configuration files. The power of Ansible comes of the use of Ansible Playbooks which are yaml scripts that tells ansible how to do its activities step by step on remote server. In this article, I'm giving a quick cheat sheet to start quickly with it.

1. Remote commands execution with Ansible

First thing to do to start with it is to add the desired hostnames ansible will operate with it can be done either globally (if you have a number of remote nodes) to deploy stuff periodically by using /etc/ansible/hosts or use a custom host script for each and every ansible custom scripts developed.

a. Ansible main config files

A common ansible /etc/ansible/hosts definition looks something like that:


# cat /etc/ansible/hosts

Host to execute on can be also provided via a shell variable $ANSIBLE_HOSTS
b) is remote hosts reachable / execute commands on all remote host

To test whether hour hosts are properly configure from /etc/ansible/hosts you can ping all defined hosts with:


ansible all -m ping


This makes ansible try to remote to remote hosts (if you have properly configured SSH public key authorization) the command should return success statuses on every host.


ansible all -a "ifconfig -a"

If you don't have SSH keys configured you can also authenticate with an argument (assuming) all hosts are configured with same password with:


ansible all –ask-pass -a "ip all show" -u hipo –ask-pass


If you have configured group of hosts via hosts file you can also run certain commands on just a certain host group, like so:


ansible <host-group> -a <command>

It is a good idea to always check /etc/ansible/ansible.cfg which is the system global (main red ansible config file).

c) List defined host groups

ansible localhost -m debug -a 'var=groups.keys()'
ansible localhost -m debug -a 'var=groups'

d) Searching remote server variables


# Search remote server variables
ansible localhost -m setup -a 'filter=*ipv4*'



ansible localhost -m setup -a 'filter=ansible_domain'



ansible all -m setup -a 'filter=ansible_domain'



# uninstall package on RPM based distros
ansible centos -s -m yum -a "name=telnet state=absent"
# uninstall package on APT distro
ansible localhost -s -m apt -a "name=telnet state=absent"



2. Debugging – Listing information about remote hosts (facts) and state of a host


# All facts for one host
ansible -m setup
 # Only ansible fact for one host
-m setup -a 'filter=ansible_eth*'
# Only facter facts but for all hosts
ansible all -m setup -a 'filter=facter_*'

To Save outputted information per-host in separate files in lets say ~/ansible/host_facts


ansible all -m setup –tree ~/ansible/host_facts


3. Playing with Playbooks deployment scripts


a) Syntax Check of a playbook yaml


ansible-playbook –syntax-check

b) Run General Infos about a playbook such as get what a playbook would do on remote hosts (tasks to run) and list-hosts defined for a playbook (like above pinging).


ansible-playbook –list-hosts

To get the idea about what an yaml playbook looks like, here is example from official ansible docs, that deploys on remote defined hosts a simple Apache webserver.

– hosts: webservers
    http_port: 80
    max_clients: 200
  remote_user: root
  – name: ensure apache is at the latest version
      name: httpd
      state: latest
  – name: write the apache config file
      src: /srv/httpd.j2
      dest: /etc/httpd.conf
    – restart apache
  – name: ensure apache is running
      name: httpd
      state: started
    – name: restart apache
        name: httpd
        state: restarted

To give it a quick try save the file as webserver.yml and give it a run via ansible-playbook command

ansible-playbook -s playbooks/webserver.yml


The -s optioninstructs ansible to run play on remote server with super user (root) privileges.

The power of ansible is its modules, which are constantly growing over time a complete set of Ansible supported modules is in its official documenation.


There is a lot of things to say about playbooks, just to give the brief they have there own language like a  templates, tasks, handlers, a playbook could have one or multiple plays inside (for instance instructions for deployment of one or more services).

The downsides of playbooks are they're so hard to write from scratch and edit, because yaml syntaxing is much more stricter than a normal oldschool sysadmin configuration file.
I've stucked with problems with modifying and writting .yaml files and I should say the community in #ansible in was very helpful to help me debug the obscure errors.

yamllint (The YAML Linter tool) comes handy at times, when facing yaml syntax errors, to use it install via apt:

# apt-get install –yes yamllint

a) Running ansible in "dry mode" just show what ansible might do but not change anything

ansible-playbook playbooks/PLAYBOOK_NAME.yml –check

b) Running playbook with different users and separate SSH keys


ansible-playbook playbooks/your_playbook.yml –user ansible-user
ansible -m ping hosts –private-key=~/.ssh/keys/custom_id_rsa -u centos


c) Running ansible playbook only for certain hostnames part of a bigger host group


ansible-playbook playbooks/PLAYBOOK_NAME.yml –limit "host1,host2,host3"

d) Run Ansible on remote hosts in parallel

To run in raw of 10 hosts in parallel

# Run 10 hosts parallel
ansible-playbook <File.yaml> -f 10            

e) Passing variables to .yaml scripts using commandline

Ansible has ability to pre-define variables from .yml playbooks. This variables later can be passed from shell cli, here is an example:

# Example of variable substitution pass from command line the var in varsubsts.yaml if present is defined / replaced ansible-playbook playbooks/varsubst.yaml –extra-vars "myhosts=localhost gather=yes pkg=telnet"


4. Ansible Galaxy (A Docker Hub) like large repository with playbook (script) files


Ansible Galaxy has about 10000 active users which are contributing ansible automation playbooks in fields such as Development / Networking / Cloud / Monitoring / Database / Web / Security etc.

To install from ansible galaxy use ansible-galaxy

# install from galaxy the geerlingguy mysql playbook
ansible-galaxy install geerlingguy.mysql

The available packages you can use as a template for your purpose are not so much as with Puppet as Ansible is younger and not corporate supported like Puppet, anyhow they are a lot and does cover most basic sysadmin needs for mass deployments, besides there are plenty of other unofficial yaml ansible scripts in various github repos.

Install TeamViewer on latest Debian, Ubuntu, Fedora, CentOS Linux quick how to

Tuesday, October 31st, 2017

Reading Time: 3minutes


If you're a sysadmin who uses GNU / Linux as a Desktop as me you will certainly need to have TeamViewer installed and ready for use on your Linux desktop.

Even though TeamViewer is a proprietary application and I prefer not to use it I'm forced to have it installed because of every now and then a friend or customer would require you to login remotely to his Windows server and clean up the system either from spyware or viruses or just deploy some new software.

Nowdays most of people are running 64 bit( amd64 ) built operating system and the problem with TeamViewer on Linux 64bit is that it doesn't have an actual full featured 64 port of the application but only have a 32 bits install, besides that big part of components of TeamViewer are running using winewindows emulation and hencing making it work on Linux is sometimes not so trivial as we might have desired.

Because TeamViewer is a 32 bit application, it has a number of dependency libraries that are 32 bit in Linux that's the so called (i386) built libraries (packages).

Hence to make TeamViewer work on modern GNU / Linux operating systems such as Debian / Ubuntu / Mint Linux / Fedora / CentOS etc. it is necessery to have some i386 libraries and other 32 bit things pre-installed and only then you can have a working copy of teamviewer on your Linux.

1. Installing i386 applications required for TeamViewer operation

– On Debian / Ubuntu / Kubuntu / Xubuntu Linux run below commands:

First we need to add the i386 architecture to be supported by Linux


dpkg –add-architecture i386
apt update


Then on Debian and other deb based Linux we need to install following libraries

# apt install libjpeg62-turbo:i386 wget gdebi-core


2. Download latest teamviewer version from TeamViewer website

– On Debian, Ubuntu and other deb based Linux distros.

Download latest teamviewer version and install it:


# wget




On CentOS, Fedora, OpenSuSE other RPM based distros:

Download the Teamviewer package and package signature using wget


# wget
# wget



3. Insteall teamviewer with gdebi (Simple Tool to install deb files)


# gdebi teamviewer_i386.deb

Remote control and meeting solution.
 TeamViewer provides easy, fast and secure remote access and meeting solutions
 to Linux, Windows PCs, Apple PCs and various other platforms,
 including Android and iPhone.
 TeamViewer is free for personal use.
 You can use TeamViewer completely free of charge to access your private
 computers or to help your friends with their computer problems.
 To buy a license for commercial use, please visit
 This package contains Free Software components.
 For details, see /opt/teamviewer/doc/license_foss.txt
Do you want to install the software package? [y/N]:y


On Fedora, CentOS, SuSE RPM based ones:


# rpm –import TeamViewer_Linux_PubKey.asc



# rpm -i teamviewer_12.0.xxxxx.i686.rpm


or if you face some failed dependencies you better use zypper that will download any missing teamviewer dependencies.


# zypper install teamviewer_12.0.xxxxx.i686.rpm




4. Start Teamviewer





linux:~$ teamviewer
CheckCPU: SSE2 support: yes
XRandRWait: No value set. Using default.
XRandRWait: Started by user.
Checking setup…
wine: configuration in '/home/hipo/.local/share/teamviewer12' has been updated.
Launching TeamViewer …
Launching TeamViewer GUI …