Archive for the ‘System Administration’ Category

Reinstall all Debian packages with a copy of apt deb package list from another working Debian Linux installation

Wednesday, July 29th, 2020

Reinstall-all-Debian-packages-with-copy-of-apt-packages-list-from-another-working-Debian-Linux-installation

Few days ago, in the hurry in the small hours of the night, I've done something extremely stupid. Wanting to move out a .tar.gz binary copy of qmail installation to /var/lib/qmail with all the dependent qmail items instead of extracting to admin user /root directory (/root), I've extracted it to the main Operating system root / directrory.
Not noticing this, I've quickly executed rm -rf var with the idea to delete all directory tree under /root/var just 3 seconds later, I've realized I'm issuing the rm -rf var with the wrong location WITH a root user !!!! Being scared on what I've done, I've quickly pressed CTRL+C to immedately cancel the deletion operation of my /var.

wrong-system-var-rm-linux-dont-do-that-ever-or-your-system-will-end-up-irreversably-damaged

But as you can guess, since the machine has an Slid State Drive drive and SSD memory drive are much more faster in I/O operations than the classical ATA / SATA disks. I was not quick enough to cancel the operation and I've noticed already some part of my /var have been R.I.P-pped in the heaven of directories.

This was ofcourse upsetting so for a while I rethinked the situation to get some ideas on what I can do to recover my system ASAP!!! and I had the idea of course to try to reinstall All my installed .deb debian packages to restore system closest to the normal, before my stupid mistake.

Guess my unpleasent suprise when I have realized dpkg and respectively apt-get apt and aptitude package management tools cannot anymore handle packages as Debian Linux's package dependency database has been damaged due to missing dpkg directory 

/var/lib/dpkg 

Oh man that was unpleasent, especially since I've installed plenty of stuff that is custom on my Mate based desktop and, generally reinstalling it updating the sytem to the latest Debian security updates etc. will be time consuming and painful process I wanted to omit.

So of course the logical thing to do here was to try to somehow recover somehow a database copy of /var/lib/dpkg  if that was possible, that of course led me to the idea to lookup for a way to recover my /var/lib/dpkg from backup but since I did not maintained any backup copy of my OS anywhere that was not really possible, so anyways I wondered whether dpkg does not keep some kind of database backups somewhere in case if something goes wrong with its database.
This led me to this nice Ubuntu thred which has pointed me to the part of my root rm -rf dpkg db disaster recovery solution.
Luckily .deb package management creators has thought about situation similar to mine and to give the user a restore point for /var/lib/dpkg damaged database

/var/lib/dpkg is periodically backed up in /var/backups

A typical /var/lib/dpkg on Ubuntu and Debian Linux looks like so:
 

hipo@jeremiah:/var/backups$ ls -l /var/lib/dpkg
total 12572
drwxr-xr-x 2 root root    4096 Jul 26 03:22 alternatives
-rw-r–r– 1 root root      11 Oct 14  2017 arch
-rw-r–r– 1 root root 2199402 Jul 25 20:04 available
-rw-r–r– 1 root root 2199402 Oct 19  2017 available-old
-rw-r–r– 1 root root       8 Sep  6  2012 cmethopt
-rw-r–r– 1 root root    1337 Jul 26 01:39 diversions
-rw-r–r– 1 root root    1223 Jul 26 01:39 diversions-old
drwxr-xr-x 2 root root  679936 Jul 28 14:17 info
-rw-r—– 1 root root       0 Jul 28 14:17 lock
-rw-r—– 1 root root       0 Jul 26 03:00 lock-frontend
drwxr-xr-x 2 root root    4096 Sep 17  2012 parts
-rw-r–r– 1 root root    1011 Jul 25 23:59 statoverride
-rw-r–r– 1 root root     965 Jul 25 23:59 statoverride-old
-rw-r–r– 1 root root 3873710 Jul 28 14:17 status
-rw-r–r– 1 root root 3873712 Jul 28 14:17 status-old
drwxr-xr-x 2 root root    4096 Jul 26 03:22 triggers
drwxr-xr-x 2 root root    4096 Jul 28 14:17 updates

Before proceeding with this radical stuff to move out /var/lib/dpkg/info from another machine to /var mistakenyl removed oned. I have tried to recover with the well known:

  • extundelete
  • foremost
  • recover
  • ext4magic
  • ext3grep
  • gddrescue
  • ddrescue
  • myrescue
  • testdisk
  • photorec

Linux file deletion recovery tools from a USB stick loaded with a Number of LiveCD distributions, i.e. tested recovery with:

  • Debian LiveCD
  • Ubuntu LiveCD
  • KNOPPIX
  • SystemRescueCD
  • Trinity Rescue Kit
  • Ultimate Boot CD


but unfortunately none of them couldn't recover the deleted files … 

The reason why the standard file recovery tools could not recover ?

My assumptions is after I've done by rm -rf var; from sysroot,  issued the sync (- if you haven't used it check out man sync) command – that synchronizes cached writes to persistent storage and did a restart from the poweroff PC button, this should have worked, as I've recovered like that in the past) in a normal Sys V System with a normal old fashioned blocks filesystem as EXT2 . or any other of the filesystems without a journal, however as the machine run a EXT4 filesystem with a journald and journald, this did not work perhaps because something was not updated properly in /lib/systemd/systemd-journal, that led to the situation all recently deleted files were totally unrecoverable.

1. First step was to restore the directory skele of /var/lib/dpkg

# mkdir -p /var/lib/dpkg/{alternatives,info,parts,triggers,updates}

2. Recover missing /var/lib/dpkg/status  file

The main file that gives information to dpkg of the existing packages and their statuses on a Debian based systems is /var/lib/dpkg/status

# cp /var/backups/dpkg.status.0 /var/lib/dpkg/status

3. Reinstall dpkg package manager to make package management working again

Say a warm prayer to the Merciful God ! and do:

# apt-get download dpkg
# dpkg -i dpkg*.deb

4. Reinstall base-files .deb package which provides basis of a Debian system

Hopefully everything will be okay and your dpkg / apt pair will be in normal working state, next step is to:

# apt-get download base-files
# dpkg -i base-files*.deb

5. Do a package sanity and consistency check and try to update OS package list

Check whether packages have been installed only partially on your system or that have missing, wrong or obsolete control  data  or  files.  dpkg  should suggest what to do with them to get them fixed.

# dpkg –audit

Then resynchronize (fetch) the package index files from their sources described in /etc/apt/sources.list

# apt-get update


Do apt db constistency check:

#  apt-get check


check is a diagnostic tool; it updates the package cache and checks for broken dependencies.
 

Take a deep breath ! …

Do :

ls -l /var/lib/dpkg
and compare with the above list. If some -old file is not present don't worry it will be there tomorrow.

Next time don't forget to do a regular backup with simple rsync backup script or something like Bacula / Amanda / Time Vault or Clonezilla.
 

6. Copy dpkg database from another Linux system that has a working dpkg / apt Database

Well this was however not the end of the story … There were still many things missing from my /var/ and luckily I had another Debian 10 Buster install on another properly working machine with a similar set of .deb packages installed. Therefore to make most of my programs still working again I have copied over /var from the other similar set of package installed machine to my messed up machine with the missing deleted /var.

To do so …
On Functioning Debian 10 Machine (Working Host in a local network with IP 192.168.0.50), I've archived content of /var:

linux:~# tar -czvf var_backup_debian10.tar.gz /var

Then sftped from Working Host towards the /var deleted broken one in my case this machine's hostname is jericho and luckily still had SSHD and SFTP running processes loaded in memory:

jericho:~# sftp root@192.168.0.50
sftp> get var_backup_debian10.tar.gz

Now Before extracting the archive it is a good idea to make backup of old /var remains somewhere for example somewhere in /root 
just in case if we need to have a copy of the dpkg backup dir /var/backups

jericho:~# cp -rpfv /var /root/var_backup_damaged

 
jericho:~# tar -zxvf /root/var_backup_debian10.tar.gz 
jericho:/# mv /root/var/ /

Then to make my /var/lib/dpkg contain the list of packages from my my broken Linux install I have ovewritten /var/lib/dpkg with the files earlier backupped before  .tar.gz was extracted.

jericho:~# cp -rpfv /var /root/var_backup_damaged/lib/dpkg/ /var/lib/

7. Reinstall All Debian  Packages completely scripts

I then tried to reinstall each and every package first using aptitude with aptitude this is done with

# aptitude reinstall '~i'

However as this failed, tried using a simple shell loop like below:

for i in $(dpkg -l |awk '{ print $2 }'); do echo apt-get install –reinstall –yes $i; done

Alternatively, all .deb package reninstall is also possible with dpkg –get-selections and with awk with below cmds:

dpkg –get-selections | grep -v deinstall | awk '{print $1}' > list.log;
awk '$1=$1' ORS=' ' list.log > newlist.log
;
apt-get install –reinstall $(cat newlist.log)

It can also be run as one liner for simplicity:

dpkg –get-selections | grep -v deinstall | awk '{print $1}' > list.log; awk '$1=$1' ORS=' ' list.log > newlist.log; apt-get install –reinstall $(cat newlist.log)

This produced a lot of warning messages, reporting "package has no files currently installed" (virtually for all installed packages), indicating a severe packages problem below is sample output produced after each and every package reinstall … :

dpkg: warning: files list file for package 'iproute' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'brscan-skey' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libapache2-mod-php7.4' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libexpat1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libexpat1:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'php5.6-readline' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'linux-headers-4.19.0-5-amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libgraphite2-3:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libgraphite2-3:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libbonoboui2-0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libxcb-dri3-0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libxcb-dri3-0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'liblcms2-2:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'liblcms2-2:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libpixman-1-0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libpixman-1-0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'gksu' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'liblogging-stdlog0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'mesa-vdpau-drivers:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'mesa-vdpau-drivers:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libzvbi0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libzvbi0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libcdparanoia0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libcdparanoia0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python-gconf' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'php5.6-cli' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libpaper1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'mixer.app' missing; assuming package has no files currently installed

After some attempts I found a way to be able to work around the warning message, for each package by simply reinstalling the package reporting the issue with

apt –reinstall $package_name


Though reinstallation started well and many packages got reinstalled, unfortunately some packages such as apache2-mod-php5.6 and other php related ones  started failing during reinstall ending up in unfixable states right after installation of binaries from packages was successfully placed in its expected locations on disk. The failures occured during the package setup stage ( dpkg –configure $packagename) …

The logical thing to do is a recovery attempt with something like the usual well known by any Debian admin:

apt-get install –fix-missing

As well as Manual requesting to reconfigure (issue re-setup) of all installed packages also did not produced a positive result

dpkg –configure -a

But many packages were still failing due to dpkg inability to execute some post installation scripts from respective .deb files.
To work around that and continue installing the rest of packages I had to manually delete all files related to the failing package located under directory 

/var/lib/dpkg/info#

For example to omit the post installation failure of libapache2-mod-php5.6 and have a succesful install of the package next time I tried reinstall, I had to delete all /var/lib/dpkg/info/libapache2-mod-php5.6.postrm, /var/lib/dpkg/info/libapache2-mod-php5.6.postinst scripts and even sometimes everything like libapache2-mod-php5.6* that were present in /var/lib/dpkg/info dir.

The problem with this solution, however was the package reporting to install properly, but the post install script hooks were still not in placed and important things as setting permissions of binaries after install or applying some configuration changes right after install was missing leading to programs failing to  fully behave properly or even breaking up even though showing as finely installed …

The final solution to this problem was radical.
I've used /var/lib/dpkg database (directory) from ther other working Linux machine with dpkg DB OK found in var_backup_debian10.tar.gz (linux:~# host with a working dpkg database) and then based on the dpkg package list correct database responding on jericho:~# to reinstall each and every package on the system using Debian System Reinstaller script taken from the internet.
Debian System Reinstaller works but to reinstall many packages, I've been prompted again and again whether to overwrite configuration or keep the present one of packages.
To Omit the annoying [Y / N ] text prompts I had made a slight modification to the script so it finally looked like this:
 

#!/bin/bash
# Debian System Reinstaller
# Copyright (C) 2015 Albert Huang
# Copyright (C) 2018 Andreas Fendt

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# —
# This script assumes you are using a Debian based system
# (Debian, Mint, Ubuntu, #!), and have sudo installed. If you don't
# have sudo installed, replace "sudo" with "su -c" instead.

pkgs=`dpkg –get-selections | grep -w 'install$' | cut -f 1 |  egrep -v '(dpkg|apt)'`

for pkg in $pkgs; do
    echo -e "\033[1m   * Reinstalling:\033[0m $pkg"    

    apt-get –reinstall -o Dpkg::Options::="–force-confdef" -o Dpkg::Options::="–force-confold" -y install $pkg || {
        echo "ERROR: Reinstallation failed. See reinstall.log for details."
        exit 1
    }
done

 

 debian-all-packages-reinstall.sh working modified version of Albert Huang and Andreas Fendt script  can be also downloaded here.

Note ! Omitting the text confirmation prompts to install newest config or keep maintainer configuration is handled by the argument:

-o Dpkg::Options::="–force-confold


I however still got few NCurses Console selection prompts during the reinstall of about 3200+ .deb packages, so even with this mod the reinstall was not completely automatic.

Note !  During the reinstall few of the packages from the list failed due to being some old unsupported packages this was ejabberd, ircd-hybrid and a 2 / 3 more.
This failure was easily solved by completely purging those packages with the usual

# dpkg –purge $packagename

and reruninng  debian-all-packages-reinstall.sh on each of the failing packages.

Note ! The failing packages were just old ones left over from Debian 8 and Debian 9 before the apt-get dist-upgrade towards 10 Duster.
Eventually I got a success by God's grance, after few hours of pains and trials, ending up in a working state package database and a complete set of freshly reinstalled packages.

The only thing I had to do finally is 2 hours of tampering why GNOME did not automatically booted after the system reboot due to failing gdm
until I fixed that I've temprary used ligthdm (x-display-manager), to do I've

dpkg –reconfigure gdm3

lightdm-x-display-manager-screenshot-gdm3-reconfige

 to work around this I had to also reinstall few libraries, reinstall the xorg-server, reinstall gdm and reinstall the meta package for GNOME, using below set of commands:
 

apt-get install –reinstall libglw1-mesa libglx-mesa0
apt-get install –reinstall libglu1-mesa-dev
apt install –reinstallgsettings-desktop-schemas
apt-get install –reinstall xserver-xorg-video-intel
apt-get install –reinstall xserver-xorg
apt-get install –reinstall xserver-xorg-core
apt-get install –reinstall task-desktop
apt-get install –reinstall task-gnome-desktop

As some packages did not ended re-instaled on system because on the original host from where /var/lib/dpkg db was copied did not have it I had to eventually manually trigger reinstall for those too:

apt-get install –reinstall –yes vlc
apt-get install –reinstall –yes thunderbird
apt-get install –reinstall –yes audacity
apt-get install –reinstall –yes gajim
apt-get install –reinstall –yes slack remmina
apt-get install –yes k3b
pt-get install –yes gbgoffice
pt-get install –reinstall –yes skypeforlinux
apt-get install –reinstall –yes vlc
apt-get install –reinstall –yes libcurl3-gnutls libcurl3-nss
apt-get install –yes virtualbox-5.2
apt-get install –reinstall –yes vlc
apt-get install –reinstall –yes alsa-tools-gui
apt-get install –reinstall –yes gftp
apt install ./teamviewer_15.3.2682_amd64.deb –yes

Note that some of above packages requires a properly configured third party repositories, other people might have other packages that are missing from the dpkg list and needs to be reinstalled so just decide according to your own case of left aside working system present binaries that doesn't belong to any dpkg installed package.

After a bit of struggle everything is back to normal Thanks God! 🙂 !
 

How to install GUI on CentOS 7 Minimal and set Gnome Graphical Environment to automatically load on system boot

Wednesday, July 22nd, 2020

centos-linux-logo

I have installed CentOS 7.7 Minimal Server Linux on a VirtualBox Virtual Environment as a test bed machine.

The system got installed easily succesfully with the standard CentOS python based graphical installer, however I needed to place various software which was not there
and for that of course I needed to have a network enabled.

To make network working instead of the default Network NAT configuration for the Virtual Machine I needed to use the Network to be Attached to a Bridged Adapter in order to make
my Windows machine to provide network and (internet) access to VirtualMachine.

virtualbox-virtualmachine-bridged-networking-configuration-screenshot

Then to make networking work after booting into CentOS I had to manually fetch IP via DHCP protocol with command:
 

[root@centos :~]# dhclient enp0s3


ethernet0-interface-dhclient-get-ip-linux

To make the setting permanent I had to also of course modify /etc/sysconfig/network-scripts/ifcfg-enp0s3 file and change 

ONBOOT=no

to 

ONBOOT=yes


enable-dhclient-centos-linux-shot

On next reboot CentOS boots normally with networking as expected

As by default CentOS Minimal does not provide any graphical environment however I needed to have it in my VM in order to be able to use VboxLinuxAdditions.run (VirtualBox Guest Additions plugins) that enabled the CentOS Operating System to show in Virtualbox in fullscreen and to enable the Copy / Paste buffers to work from The Hypervisor (Windows in that case) and the Guest VM (the CentOS VM).

In CentOS terminology metapackages (a  grouped package under a certain name, alias) are called simply groups) there is a "GNOME Desktop" group that can be used to install the GNOME Graphical Command from that point on with yum, like so:
 

[root@centos :~]# yum -y groups install "GNOME Desktop"


In a while the graphical environment will be in place, the command will install about 1300+ RPM packages, this will take about 5 minutes or so depending on your bandwidth connectivity. Once all is installed and configured succesfully you can use the good old startx command to launch GNOME.

[root@centos :~]# startx


centos7-linux-graphical-environment-screenshot

This of course will make Xserver and GNOME to run one time and on next reboot, you will end up in a plain text mode environment, so perhaps you will need to make the autolaunch of GNOME environment automatically on each boot in CentOS just like in most modern Linux distributions that use SYSTEMD to handle runlevels, you will need to configure it by changing the systemd default configured target via systemctl:

[root@centos :~]# systemctl list-units –type target | egrep "eme|res|gra|mul" 
graphical.target       loaded active active Graphical Interface
multi-user.target      loaded active active Multi-User System

[root@centos :~]# systemctl set-default graphical.target
multi-user.target

[root@centos :~]# systemctl set-default
graphical.target

[root@centos :~]# systemctl set-default graphical.target
graphical.target


Next step was to enable the Guest Additions to do so I had to install in advance 2 RPM packages kernel-headers and kernel-devel


[root@centos :~]# yum install -y kernel-headers kernel-devel

Then I had to mount and run the VboxLinuxAdditions.run script to enable them, i.e.:

[root@centos :~]# mkdir /mnt/cdrom
[root@centos :~]# mount /dev/cdrom /mnt/cdrom

[root@centos :~]# cd /mnt/cdrom/
[root@centos :~]# sh VboxLinuxAdditions.run


virtualbox-linux-additions-install-screenshot-centos-7-linux

Get daily E-Mail Reports statistics on postfix Linux mail server

Tuesday, July 14th, 2020

https://pc-freak.net/images/Postfix-email-server-logo.svg-1

I've had today a task at work to monitor a postfix mail send and received emails (MAIL FROM / RPCT TO) and get out a simple statistics on what kind of emails are coming and going out from the Postfix SMTP on a server?

Below is shortly explained how I did it plus you will learn how you can use something more advanced to get server mail count, delivery status, errors etc. daily.
 

1. Using a simple script to process /var/log/messages

For that I made a small script to do the trick, the script simply checks mail delivery logged information from /var/log/maillog process a bit sort and logs in a separate log daily.

#!/bin/sh
# Process /var/log/maillog extract from= and to= mails sort
# And log mails to $LOGF
# Author Georgi Georgiev 14.07.2020

DATE_FORM=$(date +'%m_%d_%y_%H_%M_%S_%h_%m');
LOG='/home/gge/mail_from_to-mails';
LOGF="$LOG.$DATE_FORM.log";
CUR_DATE=$(date +'%m_%d_%y_%T');
echo "Processing /var/log/maillog";
echo "Processing /var/log/maillog" > $LOGF;
echo >>$LOGF
echo "!!! $CUR_DATE # Sent MAIL FROM: addresses: !!!" >> $LOGF;
grep -E 'from=' /var/log/maillog|sed -e 's#=# #g'|awk '{ print $8 }'|sed -e 's#<# #g' -e 's#># #g' -e 's#\,##'|sort -rn|uniq >> $LOGF;

echo "!!! $CUR_DATE # Receive RCPT TO: addresses !!!" >>$LOGF;
grep -E 'to=' /var/log/maillog|sed -e 's#=# #g'|awk '{ print $8 }'|sed -e 's#<# #g' -e 's#># #g' -e 's#\,##'|sort -rn|uniq >> $LOGF;


You can get a copy of the mail_from_to_collect_mails_postfix.sh script here.

I've set the script to run via a crond scheduled job once early in the mornthing and I'll leave it like that for 5 days or so to get a good idea on what are the mailboxes that are receiving incoming mail.

The cron I've set to use is as follows:

# crontab -u root -l 
05 03 * * *     sh /home/gge/mail_from_to.sh >/dev/null 2>&1

This will be necessery later for a Email Server planned migration to relay its mail via another MTA host.

2. Getting More Robust Postifx Mail Statistics from logs


My little script is of course far from best solution to get postfix mail statistics from logs.

If you want something more professional and you need to have a daily report on what mails sent to mail server and mails sent from the MTA to give you information about the Email delivery queue status, number of successful and failed emails from a mail sender / recipient and a whole bunch of useful info you can use something more advanced such as pflogsumm perl script to get daily / weekly monthly mail delivery statistics.

What can pflogsumm do for you ?

Pflogsumm is a log analyzer/summarizer for the Postfix MTA. It is
designed to provide an overview of Postfix activity, with just enough
detail to give the administrator a “heads up” for potential trouble
spots and fixing any SMTP and email related issues.

Pflogsumm generates summaries and, in some cases, detailed reports of
mail server traffic volumes rejected and bounced email and server
warnings, errors, and panics.

At the time of writting this article it is living on jimsun.linxnet.com just in case if pflogsumm.pl's official download location disappears at some time in future here is pflogsumm-1.1.3.tar.gz mirror stored on pc-freak.net

– Install pflogsumm

Use of pflogsumm is pretty straight forward, you download unarchive the script to some location such as /usr/local/bin/pflogsumm.pl  add the script executable flag and you run it to create a Postfix Mail Log statistics report for you

wget http://jimsun.linxnet.com/downloads/pflogsumm-1.1.3.tar.gz -O /usr/local/src/pflogsumm-1.1.3.tar.gz

# mkdir -p /usr/local/src/
# cd /usr/local/src/
# tar -zxvf pflogsumm-1.1.3.tar.gz
# cd pflogsumm-1.1.3/

# mv /usr/local/pflogsumm-1.1.3/pflogsumm.pl /usr/local/bin/pflogsumm
# chmod a+x /usr/local/bin/pflogsumm


That's all, assuming you have perl installed on the system with some standard modules, we're now good to go: 

To give it a test report to the command line:

# /usr/local/bin/pflogsumm -d today /var/log/maillog

pflogsumm-log-summary-screenshot-linux-received-forwarded-bounced-rejected

To generate mail server use report and launch to some email of choice do:

# /usr/local/bin/pflogsumm -d today /var/log/maillog | mail -s Mailstats your-mail@your-domain.com


To make pflogsumm report everyday various interesting stuff such as (message deferrals, message bounce, details, smtp delivery failures, fatal errors, recipients by message size etc. add some cronjob like below to the server:

# /usr/sbin/pflogsumm -d yesterday /var/log/maillog | mail -s Mailstats | mail -s Mailstats your-mail@your-domain.com

If you need a GUI graphical mail monitoring in a Web Browser, you will need to install a webserver with a perl / cgi support,  RRDTools and MailGraph.

linux-monitoring-mail-server-with-mailgraph.cgi

How to check how many processor and volume groups IBM AIX eServer have

Monday, July 13th, 2020

how-many-cpus-are-on-commands-Linux-sysadmin-and-user-show-know-AIX-logo
In daily sysadmin duties I have been usually administrating GNU / Linux or FreeBSD servers.
However now in my daily sysadmin jobs I've been added to do some minor sysadmin activities on  a few IBM AIX eServers UNIX machines.

As the eServers were completely unknown to me and I logged in for a first time I needed a way to get idea on what kind of hardware I'm logging in so I wanted to get information about the Central Processing UNIT CPUs on the host.

On Linux I'm used to do a cat /proc/cpuinfo or do dmidecode etc. to get the number of CPUs, however AIX does not have /proc/cpuinfo and has its own way to get information about the system hardware.
As I've red in the IBM AIX's RedBook to get system information on AIX there is the lscfg command.
 

aix:/# lscfg
INSTALLED RESOURCE LIST

The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
*   = Diagnostic support not available.

  Model Architecture: chrp
  Model Implementation: Multiple Processor, PCI bus

+ sys0                                                            System Object
+ sysplanar0                                                      System Planar
* vio0                                                            Virtual I/O Bus
* vscsi3           U8205.E6B.068D6AP-V4-C21-T1                    Virtual SCSI Client Adapter
* vscsi2           U8205.E6B.068D6AP-V4-C20-T1                    Virtual SCSI Client Adapter
* vscsi1           U8205.E6B.068D6AP-V4-C11-T1                    Virtual SCSI Client Adapter
* hdisk1           U8205.E6B.068D6AP-V4-C11-T1-L8100000000000000  Virtual SCSI Disk Drive
* vscsi0           U8205.E6B.068D6AP-V4-C10-T1                    Virtual SCSI Client Adapter
* hdisk0           U8205.E6B.068D6AP-V4-C10-T1-L8100000000000000  Virtual SCSI Disk Drive
* ent3             U8205.E6B.068D6AP-V4-C5-T1                     Virtual I/O Ethernet Adapter (l-lan)
* ent2             U8205.E6B.068D6AP-V4-C4-T1                     Virtual I/O Ethernet Adapter (l-lan)
* ent1             U8205.E6B.068D6AP-V4-C3-T1                     Virtual I/O Ethernet Adapter (l-lan)
* ent0             U8205.E6B.068D6AP-V4-C2-T1                     Virtual I/O Ethernet Adapter (l-lan)
* vsa0             U8205.E6B.068D6AP-V4-C0                        LPAR Virtual Serial Adapter
* vty0             U8205.E6B.068D6AP-V4-C0-L0                     Asynchronous Terminal
+ L2cache0                                                        L2 Cache
+ mem0                                                            Memory
+ proc0                                                           Processor
+ proc4                                                           Processor


To get the number of processors on the host I've had to use:

aix:/# lscfg|grep -i proc
  Model Implementation: Multiple Processor, PCI bus
+ proc0                                                           Processor
+ proc4                                                           Processor


Another way to get the CPU number is with:

aix:/# lsdev -C -c processor
proc0 Available 00-00 Processor
proc4 Available 00-04 Processor

aix:/# lsattr -EH -l proc4
attribute   value          description           user_settable

frequency   3720000000     Processor Speed       False
smt_enabled true           Processor SMT enabled False
smt_threads 4              Processor SMT threads False
state       enable         Processor state       False
type        PowerPC_POWER7 Processor type        False

aix:/# lsattr -EH -l proc0
attribute   value          description           user_settable

frequency   3720000000     Processor Speed       False
smt_enabled true           Processor SMT enabled False
smt_threads 4              Processor SMT threads False
state       enable         Processor state       False
type        PowerPC_POWER7 Processor type        False


As you can see each of the processor is multicore has 2 Cores and each of the cores have for Threads, to get the overall number of CPUs on the system including the threaded Virtual CPUs:

aix:/# bindprocessor -q
The available processors are:  0 1 2 3 4 5 6 7


This specific machine has overall of 8 CPUs cores.

lscfg can be used to get various useful other info of the iron:

aix:/# lscfg -s
INSTALLED RESOURCE LIST

The following resources are installed on the machine.
+/- = Added or deleted from Resource List.
*   = Diagnostic support not available.

  Model Architecture: chrp
  Model Implementation: Multiple Processor, PCI bus

+ sys0
        System Object
+ sysplanar0
        System Planar
* vio0
        Virtual I/O Bus
* vscsi3           U8305…………….
        Virtual SCSI Client Adapter
* vscsi2           U8305…………….
        Virtual SCSI Client Adapter
* vscsi1           U8305…………….
        Virtual SCSI Client Adapter
* hdisk1           U8305…………….
        Virtual SCSI Disk Drive
* vscsi0           U8305……………..
        Virtual SCSI Client Adapter
* hdisk0           U8305…………….
        Virtual SCSI Disk Drive
* ent3             U8305…………….
        Virtual I/O Ethernet Adapter (l-lan)
* ent2             U8305.E6B…………….
        Virtual I/O Ethernet Adapter (l-lan)
* ent1             U8305.E6B…………….
        Virtual I/O Ethernet Adapter (l-lan)
* ent0             U8305.E6B…………….
        Virtual I/O Ethernet Adapter (l-lan)
* vsa0             U8305.E7B…………….
        LPAR Virtual Serial Adapter
* vty0             U8305.E7B…………….
        Asynchronous Terminal
+ L2cache0
        L2 Cache
+ mem0
        Memory
+ proc0
        Processor
+ proc4
        Processor

aix:/# lscfg -p
INSTALLED RESOURCE LIST

The following resources are installed on the machine.

  Model Architecture: chrp
  Model Implementation: Multiple Processor, PCI bus

  sys0                                                            System Object
  sysplanar0                                                      System Planar
  vio0                                                            Virtual I/O Bus
  vscsi3           U8305.E7B…………….V6-C40-T1                    Virtual SCSI Client Adapter
  vscsi2           U8305.E7B…………….V6-C40-T1                     Virtual SCSI Client Adapter
  vscsi1           U8305.E7B…………….V6-C40-T1                    Virtual SCSI Client Adapter
  hdisk1           U8305.E7B…………….V6-C40-T1-L8500000000000000  Virtual SCSI Disk Drive
  vscsi0           U8305.E7B…………….V6-C40-T1                    Virtual SCSI Client Adapter
  hdisk0           U8305.E7B…………….V6-C40-T1-L8500000000000000  Virtual SCSI Disk Drive
  ent3             U8305.E7B…………….V6-C40-T1                     Virtual I/O Ethernet Adapter (l-lan)
  ent2             U8305.E7B…………….V6-C40-T1                     Virtual I/O Ethernet Adapter (l-lan)
  ent1             U8305.E7B…………….V6-C40-T1                     Virtual I/O Ethernet Adapter (l-lan)
  ent0             U8305.E7B…………….V6-C40-T1                     Virtual I/O Ethernet Adapter (l-lan)
  vsa0             U8305.E7B.069D7AP-V5-C1                        LPAR Virtual Serial Adapter
  vty0             U8305.E7B.069D7AP-V5-D1-L0                     Asynchronous Terminal
  L2cache0                                                        L2 Cache
  mem0                                                            Memory
  proc0                                                           Processor
  proc4                                                           Processor

  PLATFORM SPECIFIC

  Name:  IBM,8305-E7B
    Model:  IBM,8305-E7B
    Node:  /
    Device Type:  chrp

  Name:  openprom
    Model:  IBM,AL730_158
    Node:  openprom

  Name:  interrupt-controller
    Model:  IBM, Logical PowerPC-PIC, 00
    Node:  interrupt-controller@0
    Device Type:  PowerPC-External-Interrupt-Presentation

  Name:  vty
    Node:  vty@30000000
    Device Type:  serial
    Physical Location: …………………………………………..

  Name:  l-lan
    Node:  l-lan@30000002
    Device Type:  network
    Physical Location: …………………………………………..

  Name:  l-lan
    Node:  l-lan@30000003
    Device Type:  network
    Physical Location: …………………………………………..

  Name:  l-lan
    Node:  l-lan@30000004
    Device Type:  network
    Physical Location: …………………………………………..

  Name:  l-lan
    Node:  l-lan@30000005
    Device Type:  network
    Physical Location: …………………………………………..

  Name:  v-scsi
    Node:  v-scsi@3000005a
    Device Type:  vscsi
    Physical Location: …………………………………………..

  Name:  v-scsi
    Node:  v-scsi@3000005b
    Device Type:  vscsi
    Physical Location: …………………………………………..

  Name:  v-scsi
    Node:  v-scsi@30000014
    Device Type:  vscsi
    Physical Location: ………………………………..

  Name:  v-scsi
    Node:  v-scsi@30000017
    Device Type:  vscsi
    Physical Location: …………………………………


Another useful command I found is to list the equivalent of Linux's LVM Logical Volumes configured on the system, below is how:

aix:/# lspv hdisk0
00f68c6a84acb0d5 rootvg active hdisk1 00f69d6a85400468 dsvg active

To get more info on a volume group:

aix:/# lspv hdisk0 PHYSICAL VOLUME: hdisk0 VOLUME GROUP: rootvg PV IDENTIFIER: 00f68d6a85acb0d5 VG IDENTIFIER 00f68d6a00004c0000000131353444a5 PV STATE: active STALE PARTITIONS: 0 ALLOCATABLE: yes PP SIZE: 32 megabyte(s) LOGICAL VOLUMES: 12 TOTAL PPs: 959 (30688 megabytes) VG DESCRIPTORS: 2 FREE PPs: 493 (15776 megabytes) HOT SPARE: no USED PPs: 466 (14912 megabytes) MAX REQUEST: 256 kilobytes FREE DISTRIBUTION: 191..00..00..110..192 USED DISTRIBUTION: 01..192..191..82..00 MIRROR POOL: None


You can get which local configured partition is set on which ( PV )Physical Volume

aix:/# lspv -l hdisk0
hdisk0:
LV NAME               LPs     PPs     DISTRIBUTION          MOUNT POINT
lg_dumplv             64      64      00..64..00..00..00    N/A
hd8                   1       1       00..00..01..00..00    N/A
hd6                   16      16      00..16..00..00..00    N/A
hd2                   166     166     00..45..89..32..00    /usr
hd4                   29      29      00..11..18..00..00    /
hd3                   40      40      00..04..04..32..00    /tmp
hd9var                55      55      00..00..37..18..00    /var
hd10opt               74      74      00..37..37..00..00    /opt
hd1                   8       8       00..07..01..00..00    /home
hd5                   1       1       01..00..00..00..00    N/A

Stop SSH Bruteforce authentication attempt Attacks with fail2ban

Monday, July 6th, 2020

Fail2ban stop restrict ssh bruteforce authentication attempt attacks

Most of webmasters today have some kind of SSH console remote access to the server and the OpenSSH Secure Shell service is usually not filtered for specific Networks but fully accessbible on the internet. This is especially true for home brew Linux Web servers as well as small to mid sized websites and blogs hosted on a cheap dedicated servers hosted in UK2 / Contabo RackSpace etc.

Brute force password guess attack tools such as Hydra and a distributed password dictionary files have been circulating quite for a while and if the attacker has enough time as well as a solid dictionary base, as well as some kind of relatively weak password you can expect that sooner or later some of the local UNIX accounts can be breaked and the script kiddie can get access to your server and make quickly a havoc, if he is lucky enough to be able to exploit some local vulnerability and get root access …

If you're a sysadmin that has to manage the Linux server and you do a routine log reading on the machine, you will soon get annoyed of the ever growing amount of different users, that are trying to login unsucessfully to the SSH (TCP port 22) service filling up the logs with junk and filling up disk space for nothing as well as consuming some CPU and Memory resources for nothing, you will need some easy  solution to make brute force attacks from an IP get filtered after few unsuccessful login attempts.
 

The common way to protect SSH would then be to ban an IP address from logging in if there are too many failed login attempts based on an automatic firewall inclusion of any IP that tried to unsuccessfully login lets say 5 or 10 reoccuring times.

In Linux there is a toll called “fail2ban” (F2B) used to limit brute force authentication attempts.

F2B works with minimal configuration and besides being capable of protecting the SSH service, it can be set to protect a lot of other Server applications like;

 Apache / NGINX Web Servers with PHP / Mail Servers (Exim, Postfix, Qmail, Sendmail), POP3 IMAP / AUTH services (Dovecot, Courier, Cyrus), DNS Bind servers, MySQL DBs, Monitoring tools such as Nagios, FTP servers (ProFTPD / PureFTP), Proxy servers (Squid), WordPress sites (wp-login) brute force attacks, Web Mail services (Horde / Roundcube / OpenWebmail), jabber servers etc.

To get a better overview below is F2B package description:
 

linux:~# apt-cache show fail2ban|grep -i description-en -A 21
Description-en: ban hosts that cause multiple authentication errors
 Fail2ban monitors log files (e.g. /var/log/auth.log,
 /var/log/apache/access.log) and temporarily or persistently bans
 failure-prone addresses by updating existing firewall rules.  Fail2ban
 allows easy specification of different actions to be taken such as to ban
 an IP using iptables or hostsdeny rules, or simply to send a notification
 email.
 .
 By default, it comes with filter expressions for various services
 (sshd, apache, qmail, proftpd, sasl etc.) but configuration can be
 easily extended for monitoring any other text file.  All filters and
 actions are given in the config files, thus fail2ban can be adopted
 to be used with a variety of files and firewalls.  Following recommends
 are listed:
 .
  – iptables/nftables — default installation uses iptables for banning.
    nftables is also suported. You most probably need it
  – whois — used by a number of *mail-whois* actions to send notification
    emails with whois information about attacker hosts. Unless you will use
    those you don't need whois
  – python3-pyinotify — unless you monitor services logs via systemd, you
    need pyinotify for efficient monitoring for log files changes

 

Using fail2ban is easy as there is a multitude of preexisting filters and actions (that gets triggered on a filter match) already written by different people usually found in /etc/fail2ban/filter.d an /etc/fail2ban/action.d.
as well as custom action / filter / jails is easy to do.
Fail2ban is available as a standard distro RPM / DEB package
on most modern versions of Ubuntu (16.04 and later), Debian, Mint and CentOS 7, OpenSuSE, Fedora etc.
 

1. Install Fail2ban package


– On Deb based distros do the usual:

linux:~# apt update
linux:~# apt install –yes fail2ban


On RPM distros Fedora / CentOS / SuSE etc.
 

linux:~# sudo yum -y install epel-release
linux:~# sudo yum -y install fail2ban

2. Enable fail2ban ban rules for SSH failed authentication filtering

Create the file /etc/fail2ban/jail.local :
 

# cat > /etc/fail2ban/jail.local


Paste below content:
 

[DEFAULT]
# Ban hosts for one hour:
bantime = 360
# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport

[sshd]
enabled = true
maxretry = 10
findtime = 43200
# ban for 1 day
bantime = 3600
# ban for 1 day
#bantime = 86400

This config will ban for 1 day any IP that tries to access more than 10 unsuccesful times sshd daemon.
This works through IPTABLES indicated by config banaction = iptables-multiport config making fail2ban to automatically add a new iptables block rule valid for 1 day.

To close the file press CTRL + D simulanesly

  • maxretry controls the maximum number of allowed retries.
  • findtime specifies the time window (in seconds) which should be considered for banning an IP. (43200 seconds is 12 hours)
  • bantime specifies the time window (in seconds) for which the IP address will be banned (86400 seconds is 24 hours).

If SSH listens to a different port from 22 on the machine, you can specify the port number with port = <port_number> in this file.

3. Start fail2ban service


Either use the /etc/init.d/fail2ban start script or systemd systemctl
 

linux:~# systemctl enable fail2ban
linux:~# systemctl restart fail2ban

4. Fail2ban operational principle in short


Fail2ban uses Filters, Actions And Jails:

Filtersspecify certain patterns of text that Fail2ban should recognize in log files.
Actionsare things Fail2ban can do once a filter is matched.
Jailstell Fail2ban to match a filter on some logs. When the number of matches goes beyond a certain limit specified in the jail, Fail2ban takes an action specified in the jail.

If still wonder what is Fail2ban jail? Each configured jail tells fail2ban to look at system logs and take actions against attacks on a configured service, in our case OpenSSH service.
 

5. Blocking repeated unsuccessful password authentication attempts for longer periods


If more than number of failed ssh logins happen to occur (lets say 35 reoccuring ones in /var/log/auth.log (the debian failed ssh login file) or /var/log/secure (redhat distros failed ssh log file).
You will perhaps want to permanently block this IP for 3 days or so, here is how:
 

banaction = iptables-multiport
maxretry  = 35
findtime  = 259200
bantime   = 608400
enabled   = true
filter    = sshd

Add this to [sshlongterm] section to make it finally look like this:
 

[sshlongterm]
port      = ssh
logpath   = %(sshd_log)s
banaction = iptables-multiport
maxretry  = 35
findtime  = 259200
bantime   = 608400
enabled   = true
filter    = sshd


Of course to load new config restart fail2ban
 

# /etc/init.d/fail2ban restart


Default jail as well as the sshlongterm jail should now work together. Short term attacks will be handled by the default jail under [sshd], and the long term attacks handled by our own second jail [sshlongterm].
 

6. Checking which intruders were blocked by fail2ban


Fail2ban creates a separate chain f2b-sshd to which it adds each blocked IP for the period of time preset in the config, to list it:

linux:~# /sbin/iptables -L f2b-sshd
Chain f2b-sshd (1 references)
target     prot opt source               destination         
REJECT     all  —  165.22.2.95          anywhere             reject-with icmp-port-unreachable
REJECT     all  —  ec2-13-250-58-46.ap-southeast-1.compute.amazonaws.com  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  cable200-116-3-133.epm.net.co  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  mail.antracite.org   anywhere             reject-with icmp-port-unreachable
REJECT     all  —  139.59.57.2          anywhere             reject-with icmp-port-unreachable
REJECT     all  —  111.68.98.152.pern.pk  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  216.126.59.61        anywhere             reject-with icmp-port-unreachable
REJECT     all  —  104.248.4.138        anywhere             reject-with icmp-port-unreachable
REJECT     all  —  210.211.119.10       anywhere             reject-with icmp-port-unreachable
REJECT     all  —  85.204.118.13        anywhere             reject-with icmp-port-unreachable
REJECT     all  —  broadband.actcorp.in  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  104.236.33.155       anywhere             reject-with icmp-port-unreachable
REJECT     all  —  114.ip-167-114-114.net  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  142.162.194.168.rfc6598.dynamic.copelfibra.com.br  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  server.domain.cl     anywhere             reject-with icmp-port-unreachable
REJECT     all  —  ip202.ip-51-68-251.eu  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  188.166.185.157      anywhere             reject-with icmp-port-unreachable
REJECT     all  —  93.49.11.206         anywhere             reject-with icmp-port-unreachable
RETURN     all  —  anywhere             anywhere            

Conclusion


What we have seen here is how to make fail2ban protect Internet firewall unrestricted SSHD Service to filter out 1337 skript kiddie 'hackers' out of your machine. With a bit of tuning could not only break the occasional SSH Brute Force bot scanners that craw the new but could even mitigate massive big botnet initiated brute force attacks to servers.
Of course fail2ban is not a panacea and to make sure you won't get hacked one days better make sure to only allow access to SSH service only for a certain IP addresses or IP address ranges that are of your own PCs.

Sysadmin tip: How to force a new Linux user account password change after logging to improve security

Thursday, June 18th, 2020

chage-linux-force-password-expiry-check-user-password-expiry-setting

Have you logged in through SSH to remote servers with the brand new given UNIX account in your company just to be prompted for your current Password immediately after logging and forced to change your password?
The smart sysadmins or security officers use this trick for many years to make sure the default set password for new user is set to a smarter user to prevent default password leaks which might later impose a severe security risk for a company Demiliterized networks confidential data etc.

If you haven't seen it yet and you're in the beautiful world of UNIX / Linux as a developer qa tester or sysadmin sooner or later you will face it.
Here of course I'm talking about plain password local account authentication using user / pass credentials stored in /etc/passwd or /etc/shadow.

Lets Say hello to the main command chage that is used to do this sysadmin trick.
chage command is used to change user password expiry information and  set and alter password aging parameters on user accounts.

1. Force chage to make password expire on next user login for a new created user
 

# chage -d 0 {user-name} 


Below is a real life example
 

chage-force-user-account-password-expiry-linux

2. Get information on when account expires

[hipo@linux ~]$ chage -l hipo
Last password change                                    : Apr 03, 2020
Password expires                                        : Jul 08, 2020
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 90
Number of days of warning before password expires       : 14

3. Use chage to set user account password expiration

The most straight forward way to set an expiration date for an active user acct is with:

# chage -E 2020-08-16 username


To make the account get locked automatically if the password has expired and the user did not logged in to it for 2 days after its expiration.

# chage -I 2 username


– Set Password expire with Minimum days 7 (-n mindays 7), (-x maxdays 28) and (-w warndays 5)

# passwd -n 7 -x 28 -w 5 username

To check the passwod expiration settings use list command:

# chage -l username
Last password change                                    : юни 18, 2020
Password expires                                        : юли 16, 2020
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 7
Maximum number of days between password change          : 28
Number of days of warning before password expires       : 5

chage is a command is essential sysadmin command that is mentioned in every Learn Linux book out there, however due to its often rare used many people and sysadmins either, don't know it or learn of it only once it is needed. 
A note to make here is some sysadmins prefer to use usermod to set a password expire instead of chage.

usermod -e 2020-10-14 username

For those who wonder how to set password expiry on FreeBSD and other BSD-es is done, there it is done via the pw system user management tool as chage is not present there.

A note to make here is chage usually does not provide information for Linux user accounts that are stored in LDAP. To get information of such you can use ldapsearch with a query to the LDAP domain store with something like.
 

ldapsearch -x -ZZ -LLL -b dc=domain.com,dc=com objectClass=*


It is worthy to mention also another useful command when managing users this is getent used to get entries from Name Service Switch libraries. 
getent is useful to get various information from basic /etc/ stored db files such as /etc/services /etc/shadow, /etc/group, /etc/aliases, /etc/hosts and even do some simple rpc queries.

Monitoring Linux hardware Hard Drives / Temperature and Disk with lm_sensors / smartd / hddtemp and Zabbix Userparameter lm_sensors report script

Thursday, April 30th, 2020

monitoring-linux-hardware-with-software-temperature-disk-cpu-health-zabbix-userparameter-script

I'm part of a  SysAdmin Team that is partially doing some minor Zabbix imrovements on a custom corporate installed Zabbix in an ongoing project to substitute the previous HP OpenView monitoring for a bunch of Legacy Linux hosts.
As one of the necessery checks to have is regarding system Hardware, the task was to invent some simplistic way to monitor hardware with the Zabbix Monitoring tool.  Monitoring Bare Metal servers hardware of HP / Dell / Fujituse etc. servers  in Linux usually is done with a third party software provided by the Hardware vendor. But as this requires an additional services to run and sometimes is not desired. It was interesting to find out some alternative Linux native ways to do the System hardware monitoring.
Monitoring statistics from the system hardware components can be obtained directly from the server components with ipmi / ipmitool (for more info on it check my previous article Reset and Manage intelligent  Platform Management remote board article).
With ipmi
 hardware health info could be received straight from the ILO / IDRAC / HPMI of the server. However as often the Admin-Lan of the server is in a seperate DMZ secured network and available via only a certain set of routed IPs, ipmitool can't be used.

So what are the other options to use to implement Linux Server Hardware Monitoring?

The tools to use are perhaps many but I know of two which gives you most of the information you ever need to have a prelimitary hardware damage warning system before the crash, these are:
 

1. smartmontools (smartd)

Smartd is part of smartmontools package which contains two utility programs (smartctl and smartd) to control and monitor storage systems using the Self-Monitoring, Analysis and Reporting Technology system (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks

Disk monitoring is handled by a special service the package provides called smartd that does query the Hard Drives periodically aiming to find a warning signs of hardware failures.
The downside of smartd use is that it implies a little bit of extra load on Hard Drive read / writes and if misconfigured could reduce the the Hard disk life time.

linux:~#  /usr/sbin/smartctl -a /dev/sdb2
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-5-amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     KINGSTON SA400S37240G
Serial Number:    50026B768340AA31
LU WWN Device Id: 5 0026b7 68340aa31
Firmware Version: S1Z40102
User Capacity:    240,057,409,536 bytes [240 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 4
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Thu Apr 30 14:05:01 2020 EEST
SMART support is: Available – device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x11) SMART execute Offline immediate.
                                        No Auto Offline data collection support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        No Selective Self-test supported.
SMART capabilities:            (0x0002) Does not save SMART data before
                                        entering power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (  10) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x0032   100   100   000    Old_age   Always       –       100
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       –       2820
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       –       21
148 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       0
149 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       0
167 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       0
168 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       –       0
169 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       0
170 Unknown_Attribute       0x0000   100   100   010    Old_age   Offline      –       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       –       0
173 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       0
181 Program_Fail_Cnt_Total  0x0032   100   100   000    Old_age   Always       –       0
182 Erase_Fail_Count_Total  0x0000   100   100   000    Old_age   Offline      –       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       –       0
192 Power-Off_Retract_Count 0x0012   100   100   000    Old_age   Always       –       16
194 Temperature_Celsius     0x0022   034   052   000    Old_age   Always       –       34 (Min/Max 19/52)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       –       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       –       0
218 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       –       0
231 Temperature_Celsius     0x0000   097   097   000    Old_age   Offline      –       97
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       –       2104
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       –       1857
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       –       1141
244 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       32
245 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       107
246 Unknown_Attribute       0x0000   100   100   000    Old_age   Offline      –       15940

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

Selective Self-tests/Logging not supported

2. hddtemp

Usually if smartd is used it is useful to also use hddtemp which relies on smartd data.
 The hddtemp program monitors and reports the temperature of PATA, SATA
 or SCSI hard drives by reading Self-Monitoring Analysis and Reporting
 Technology (S.M.A.R.T.)
information on drives that support this feature.
 

linux:~# /usr/sbin/hddtemp /dev/sda1
/dev/sda1: Hitachi HDS721050CLA360: 31°C
linux:~# /usr/sbin/hddtemp /dev/sdc6
/dev/sdc6: KINGSTON SV300S37A120G: 25°C
linux:~# /usr/sbin/hddtemp /dev/sdb2
/dev/sdb2: KINGSTON SA400S37240G: 34°C
linux:~# /usr/sbin/hddtemp /dev/sdd1
/dev/sdd1: WD Elements 10B8: S.M.A.R.T. not available

3. lm-sensors / i2c-tools 

 Lm-sensors is a hardware health monitoring package for Linux. It allows you
 to access information from temperature, voltage, and fan speed sensors.
i2c-tools
was historically bundled in the same package as lm_sensors but has been seperated cause not all hardware monitoring chips are I2C devices, and not all I2C devices are hardware monitoring chips.

The most basic use of lm-sensors is with the sensors command

linux:~# sensors
i350bb-pci-0600
Adapter: PCI adapter
loc1:         +55.0 C  (high = +120.0 C, crit = +110.0 C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +28.0 C  (high = +78.0 C, crit = +88.0 C)
Core 0:         +26.0 C  (high = +78.0 C, crit = +88.0 C)
Core 1:         +28.0 C  (high = +78.0 C, crit = +88.0 C)
Core 2:         +28.0 C  (high = +78.0 C, crit = +88.0 C)
Core 3:         +28.0 C  (high = +78.0 C, crit = +88.0 C)

 


On CentOS Linux useful tool is also  lm_sensors-sensord.x86_64 – A Daemon that periodically logs sensor readings to syslog or a round-robin database, and warns of sensor alarms.

In Debian Linux there is also the psensors-server (an HTTP server providing JSON Web service which can be used by GTK+ Application to remotely monitor sensors) useful for developers
psesors-server

psensor-linux-graphical-tool-to-check-cpu-hard-disk-temperature-unix

If you have a Xserver installed on the Server accessed with Xclient or via VNC though quite rare,
You can use xsensors or Psensora GTK+ (Widget Toolkit for creating Graphical User Interface) application software.

With this 3 tools it is pretty easy to script one liners and use the Zabbix UserParameters functionality to send hardware report data to a Company's Zabbix Sserver, though Zabbix has already some templates to do so in my case, I couldn't import this templates cause I don't have Zabbix Super-Admin credentials, thus to work around that a sample work around is use script to monitor for higher and critical considered temperature.
Here is a tiny sample script I came up in 1 min time it can be used to used as 1 liner UserParameter and built upon something more complex.

SENSORS_HIGH=`sensors | awk '{ print $6 }'| grep '^+' | uniq`;
SENSORS_CRIT=`sensors | awk '{ print $9 }'| grep '^+' | uniq`; ;SENSORS_STAT=`sensors|grep -E 'Core\s' | awk '{ print $1" "$2" "$3 }' | grep "$SENSORS_HIGH|$SENSORS_CRIT"`;
if [ ! -z $SENSORS_STAT ]; then
echo 'Temperature HIGH';
else 
echo 'Sensors OK';
fi 

Of course there is much more sophisticated stuff to use for monitoring out there


Below script can be easily adapted and use on other Monitoring Platforms such as Nagios / Munin / Cacti / Icinga and there are plenty of paid solutions, but for anyone that wants to develop something from scratch just like me I hope this
article will be a good short introduction.
If you know some other Linux hardware monitoring tools, please share.

How to Import Remove List archive signing keys on CentOS / RHEL / Fedora RPM based Linux distributions

Wednesday, April 8th, 2020

how-to-import-remove-list-archiving-signing-keys-on-CentOS-RHEL-Fedora-rpm-based-Linux-distros-package
If you  plan to build and distribute  own RPMs securely, it is strongly recommended that all custom RPMs are signed using GNU Privacy Guard (GPG). Generating GPG keys and building GPG-signed packages matching it.
Hence, If you have to deal with some of the RPM based package management Linux distribution like CentOS / RHEL / Fedora etc. you will sooner or later end up in a situation where some of the archive signing keys for a package provided by some of the repositories is missing or it is not matching the keys provided for the RPM repo.

As a result you will be unable to install some package like lets say zabbix-sender or you won't be able to update a certain package to the latest available version, because the Archive Signing key is not found.
The usual naming for a RPM file with a GPG key in is YOUR-RPM-GPG-KEY.

A typical PGP Public key file content looks something like this:
 

—–BEGIN PGP PUBLIC KEY BLOCK—–

Version: GnuPG v1.0.0 (GNU/Linux)

Comment: For info see http://www.gnupg.org

mQGiBDfqVEqRBADBKr3Bl6PO8BQ0H8sJoD6p9U7Yyl7pjtZqioviPwXP+DCWd4u8

HQzcxAZ57m8ssA1LK1Fx93coJhDzM130+p5BG9mYSPShLabR3N1KXdXAYYcowTOM

GxdwYRGr1Spw8QydLhjVfU1VSl4xt6bupPbFJbyjkg5Z3P7BlUOUJmrx3wCgobNV

EDGaWYJcch5z5B1of/41G8kEAKii6q7Gu/vhXXnLS6m15oNnPVybyngiw/23dKjS

ti/PYrrL2J11P2ed0x7zm8v3gLrY0cue1iSba+8glY+p31ZPOr5ogaJw7ZARgoS8

BwjyRymXQp+8Dete0TELKOL2/itDOPGHW07SsVWOR6cmX4VlRRcWB5KejaNvdrE5

4XFtOd04NMgWI63uqZc4zkRa+kwEZtmbz3tHSdWCCE+Y7YVP6IUf/w6YPQFQriWY

FiA6fD10eB+BlIUqIw80EqjsBKmCwvKkn4jg8kibUgj4/TzQSx77uYokw1EqQ2wk

OZoaEtcubsNMquuLCMWijYhGBBgRAgAGBQI36lRyAAoJECGRgM3bQqYOhyYAnj7h

VDY/FJAGqmtZpwVp9IlitW5tAJ4xQApr/jNFZCTksnI+4O1765F7tA==

=3AHZ

—–END PGP PUBLIC KEY BLOCK—–

The usual naming for a RPM file with a GPG key in is YOUR-RPM-GPG-KEY
 

1. List RPM gpg keys installed on system

To list all the installed RPM gpg keys on the system do:

rpm -q gpg-pubkey


To get a list of the number of installed keys with verbose info with key description::

rpm -qa gpg-pubkey –qf "%{version}-%{release} %{summary}\n"|wc -l

 rpm -qa gpg-pubkey –qf "%{version}-%{release} %{summary}\n"
fdb19c98-56fd6333 gpg(Fedora 25 Primary (25) <fedora-25-primary@fedoraproject.org>)
7fac5991-4615767f gpg(Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>)
64dab85d-57d33e22 gpg(Fedora 26 Primary (26) <fedora-26-primary@fedoraproject.org>)
fa7a179a-562bcd6e gpg(RPM Fusion nonfree repository for Fedora (25) <rpmfusion-buildsys@lists.rpmfusion.org>)
6806a9cb-562bce39 gpg(RPM Fusion free repository for Fedora (25) <rpmfusion-buildsys@lists.rpmfusion.org>)
d38b4796-570c8cd3 gpg(Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>)

[root@host ~:]# rpm -q gpg-pubkey –qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-f4a80eb5-53a7ff4b    gpg(CentOS-7 Key (CentOS 7 Official Signing Key) )
gpg-pubkey-b6792c39-53c4fbdd    gpg(CentOS-7 Debug (CentOS-7 Debuginfo RPMS) )
gpg-pubkey-8fae34bd-538f1e51    gpg(CentOS-7 Testing (CentOS 7 Testing content) )

To list all OS installed gpg keys do:

[user@host ~:]$ $ rpm -qa | grep -i gpg
gpg-pubkey-db42a60e-37ea5438


2. Import RPM-GPG-KEY


A new key be it official archive keys issued from Fedora or a custom own build RPM package can be imported Redhat Package Manager like so:

[root@host ~:]# rpm –import RPM-GPG-KEY


It is possible to also import multiple GPG signature keys, for example on CentOS the usual path containg keys is /etc/pki/rpm-gpg/ to import all of the contained files there:

[root@host ~:]# rpm –import /etc/pki/rpm-gpg/*


3. Check package with imported gpg arch key


Once the RPM-GPG-KEY is imported you can compare whether a RPM package matches with the key signature.

[root@host ~:]# rpm –checksig package-1.3-3.src.rpm

[root@host ~:]# rpm –checksig xtoolwait-1.3-3.src.rpm
package-1.3-3.src.rpm: (sha1) dsa sha1 md5 gpg OK


4. Remove RPM installed arch key


If you have installed some gpg arch. key by mistake and you need to remove it:

[root@host ~:]#rpm -e gpg-pubkey-b6792c39-53c4fbdd


To make sure it is remove do a Listing once again signing archive keys, it should not show anymore:

[root@host ~:]# rpm -q gpg-pubkey –qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-f4a80eb5-53a7ff4b    gpg(CentOS-7 Key (CentOS 7 Official Signing Key) )
gpg-pubkey-8fae34bd-538f1e51    gpg(CentOS-7 Testing (CentOS 7 Testing content) )