Archive for the ‘Linux’ Category

Delete empty files and directories under directory tree in Linux / UNIX / BSD

Wednesday, October 21st, 2020


Sometimes it happens that you end up on your server with a multiple of empty files. The reason for that could be different for example it could be /tmp is overflown with some session store files on a busy website, or due to some programmers Web executed badly written PHP / Python / Perl / Ruby code bug or lets say Content Management System ( CMS ) based website based on WordPress / Joomla / Drupal / Magento / Shopify etc. due to a broken plugin some specific directory could get filled up with plenty of meaningless empty files, that never gets wiped out if you don't care. This could happen if you offer your users to share files online to a public sharing service as WebFTP and some of the local hacked UNIX user accounts decides to make you look like a fool and run an endless loop to create files in your Hard Drive until your small server HDD filesystem of few terabytes gets filled up with useless empty files and due to full inode count on the filesystem your machine running running services gets disfunctional …

Hence on servers with shared users or simply webservers it is always a good idea to keep an eye on filesystem used nodes count by system are and in case if notices a sudden increase of used FS inodes as part of the investigation process on what caused it to check the amount of empty files on the system attached SCSI / SSD / SAS whatever drive.

1. Show a list of free inodes on server

Getting inodes count after logged is done with df command

root@linux-server:~# df -i
Filesystem        Inodes   IUsed     IFree IUse% Mounted on
udev             2041464     516   2040948    1% /dev
tmpfs            2046343    1000   2045343    1% /run
/dev/sdb2       14655488 1794109  12861379   13% /
tmpfs            2046343       4   2046339    1% /dev/shm
tmpfs            2046343       8   2046335    1% /run/lock
tmpfs            2046343      17   2046326    1% /sys/fs/cgroup
/dev/sdc6        6111232  6111232   0   100% /var/www
/dev/sda1       30162944 3734710  26428234   13% /mnt/sda1
/dev/sdd1      122093568 8011342 114082226    7% /backups
tmpfs            2046343      13   2046330    1% /run/user/1000


2. Show all empty files and directories count


### count empty directories ### root@linux-server:~# find /path/ -empty -type d | wc -l

### count empty files only ### root@linux-server:~# find /path/ -empty -type f | wc -l


3. List all empty files in directory or root dir

As you can see on the server in above example the amount of inodes of empty inodes is depleted.
The next step is to anylize what is happening in that web directory and if there is a multitude of empty files taking up all our disk space.

root@linux-server:~# find /var/www -type f -empty > /root/empty_files_list.txt

As you can see I'm redirecting output to a file as with the case of many empty files, I'll have to wait for ages and console will get filled up with a data I'll be unable to easily analyze

If the problem is another directory in your case, lets say the root dir.

root@linux-server:~#  DIR='/';
root@linux-server:~# find $DIR -type f -empty > /root/empty_files_list.txt

4. Getting empty directories list

Under some case it might be that the server is overflowed with empty directories. This is also a thing some malicious cracker guy could do to your server if he can't root the server with some exploit but wants to bug you and 'show off his script kiddie 3l337 magic tricks' :). This is easily done with a perl / python or bash shell endless loop inside which a random file named millions of empty directories instead of files is created.

To look up for empty directories hence use:

root@linux-server:~# DIR='/home';
root@linux-server:~# find  $DIR . -type d -empty > /root/empty_directories_list.txt


5. Delete all empty files only to clean up inodes

Deletion of empty files will automatically free up the inodes occupied, to delete them.

root@linux-server:~# cd /path/containing/multiple/empty-dirs/
root@linux-server:~# find . -type f -empty -exec rm -fr {} \;


6. Delete all empty directories only to clean up inocommanddes

root@linux-server:~# find . -type d -empty -exec rm -fr {} \;


7. Delete all empty files and directories to clean up inodes

root@linux-server:~# cd /path/containing/multiple/empty-dirs/
root@linux-server:~# find . -empty -delete


8. Use find + xargs to delete if files count to delete is too high

root@linux-server:~# find . -empty | xargs rm -r

That's all folks ! Enjoy now your Filesystem to have retrieved back the lost inodes from the jump empty files or directories.

Happy cleaning  🙂

Deny DHCP Address by MAC on Linux

Thursday, October 8th, 2020

Deny DHCP addresses by MAC ignore MAC to not be DHCPD leased on GNU / Linux howto

I have not blogged for a long time due to being on a few weeks vacation and being in home with a small cute baby. However as a hardcore and a bit of dumb System administrator, I have spend some of my vacation and   worked on bringing up the the and the other Websites hosted as a high availvailability ones living on a 2 Webservers running on a Master to Master MySQL Replication backend database, this is oll hosted on  servers, set to run as a round robin DNS hosts on 2 servers one old Lenove ThinkCentre Edge71 as well as a brand new real Lenovo server Lenovo ThinkServer SD350 with 24 CPUs and a 32 GB of RAM
To assure Internet Connectivity is having a good degree of connectivity and ensure websites hosted on both machines is not going to die if one of the 2 pair configured Fiber Optics Internet Providers Bergon.NET has some Issues, I've rented another Internet Provider Line is set bought from the VIVACOM Mobile Fiber Internet provider – that is a 1 Gigabit Fiber Optics Line.
Next to that to guarantee there is no Database, Webserver, MailServer, Memcached and other running services did not hit downtimes due to Electricity power outage, two Powerful Uninterruptable Power Supplies (UPS)  FPS Fortron devices are connected to the servers each of which that could keep the machine and the connected switches and Servers for up to 1 Hour.

The machines are configured to use dhcpd to distributed IP addresses and the Main Node is set to distribute IPs, however as there is a local LAN network with more of a personal Work PCs, Wireless Devices and Testing Computers and few Virtual machines in the Network and the IPs are being distributed in a consequential manner via a ISC DHCP server.

As always to make everything work properly hence, I had again some a bit weird non-standard requirement to make some of the computers within the Network with Static IP addresses and the others to have their IPs received via the DHCP (Dynamic Host Configuration Protocol) and add some filter for some of the Machine MAC Addresses which are configured to have a static IP addresses to prevent the DHCP (daemon) server to automatically reassign IPs to this machines.

After a bit of googling and pondering I've done it and some of the machines, therefore to save others the efforts to look around How to set Certain Computers / Servers Network Card MAC (Interfaces) MAC Addresses  configured on the LAN network to use Static IPs and instruct the DHCP server to ingnore any broadcast IP addresses leases – if they're to be destined to a set of IGNORED MAcs, I came up with this small article.

Here is the DHCP server /etc/dhcpd/dhcpd.conf from my Debian GNU / Linux (Buster) 10.4


option domain-name "pcfreak.lan";
option domain-name-servers,,,;
max-lease-time 891200;
class "black-hole" {
    match substring (hardware, 1, 6);
    ignore booting;
subclass "black-hole" 18:45:91:c3:d9:00;
subclass "black-hole" 70:e2:81:13:44:11;
subclass "black-hole" 70:e2:81:13:44:12;
subclass "black-hole" 00:16:3f:53:5d:11;
subclass "black-hole" 18:45:9b:c6:d9:00;
subclass "black-hole" 16:45:93:c3:d9:09;
subclass "black-hole" 16:45:94:c3:d9:0d;/etc/dhcpd/dhcpd.conf
subclass "black-hole" 60:67:21:3c:20:ec;
subclass "black-hole" 60:67:20:5c:20:ed;
subclass "black-hole" 00:16:3e:0f:48:04;
subclass "black-hole" 00:16:3e:3a:f4:fc;
subclass "black-hole" 50:d4:f5:13:e8:ba;
subclass "black-hole" 50:d4:f5:13:e8:bb;
subnet netmask {
        option routers        ;
        option subnet-mask    ;
host think-server {
        hardware ethernet 70:e2:85:13:44:12;
default-lease-time 691200;
max-lease-time 891200;
log-facility local7;

To spend you copy paste efforts a file with Deny DHCP Address by Mac Linux configuration is here
Of course I have dumped the MAC Addresses to omit a data leaking but I guess the idea behind the MAC ADDR ignore is quite clear

The main configuration doing the trick to ignore a certain MAC ALenovo ThinkServer SD350ddresses that are reachable on the Connected hardware switch on the device is like so:

class "black-hole" {
    match substring (hardware, 1, 6);
    ignore booting;
subclass "black-hole" 18:45:91:c3:d9:00;

The Deny DHCP Address by MAC is described on distribution lists here but it seems the documentation on the topic on how to Deny / IGNORE DHCP Addresses by MAC Address on Linux has been quite obscure and limited online.

As you can see in above config the time via which an IP is freed up and a new IP lease is done from the server is severely maximized as often DHCP servers do use a max-lease-time like 1 hour (3600) seconds:, the reason for increasing the lease time to be to like 10 days time is that the IPs in my network change very rarely so it is a waste of CPU cycles to do a frequent lease.

default-lease-time 691200;
max-lease-time 891200;

As you see to Guarantee resolving works always as expected I have configured – Google Public DNS and OpenDNS IPs

option domain-name-servers,,,;

One hint to make is, after setting up all my desired config in the standard config location /etc/dhcp/dhcpd.conf it is always good idea to test configuration before reloading the running dhcpd process.


root@pcfreak: ~# /usr/sbin/dhcpd -t
Internet Systems Consortium DHCP Server 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit
Config file: /etc/dhcp/dhcpd.conf
Database file: /va/home/hipo/infor/lib/dhcp/dhcpd.leases
PID file: /var/run/

That's all folks with this sample config the IPs under subclass "black-hole", which are a local LAN Static IP Addresses will never be offered leasess anymore from the ISC DHCP.
Hope this stuff helps someone, enjoy and in case if you need a colocation of a server or a website hosting for a really cheap price on this new set High Availlability up described machines open an inquiry on


Set all logs to log to to physical console /dev/tty12 (tty12) on Linux

Wednesday, August 12th, 2020

tty linux-logo how to log everything to last console terminal tty12

Those who administer servers from the days of birth of Linux and who used actively GNU / Linux over the years or any other UNIX knows how practical could be to configure logging of all running services / kernel messages / errors and warnings on a physical console.

Traditionally from the days I was learning Linux basics I was shown how to do this on an old Debian Sarge 3.0 Linux without systemd and on all Linux distributions Redhat 9.0 / Calderas and Mandrakes I've used either as a home systems or for servers. I've always configured output of all messages to go to the last easy to access console /dev/tty12 (for those who never use it console switching under Linux plain text console mode is done with key combination of CTRL + ALT + F1 .. F12.

In recent times however with the introduction of systemd pretty much things changed as messages to console are not handled by /etc/inittab which was used to add and refresh physical consoles tty1, tty2 … tty7 (the default added one on Linux were usually 7), but I had to manually include more respawn lines for each console in /etc/inittab.
Nowadays as of year 2020 Linux distros /etc/inittab is no longer there being obsoleted and console print out of INPUT / OUTPUT messages are handled by systemd.

1. Enable Physical TTYs from TTY8 till TTY12 etc.

The number of default consoles existing in most Linux distributions I've seen is still from tty1 to tty7. Hence to add more tty consoles and be ready to be able to switch out  not only towards tty7 but towards tty12 once you're connected to the server via a remote ILO (Integrated Lights Out) / IdRAC (Dell Remote Access Controller) / IPMI / IMM (Imtegrated Management Module), you have to do it by telling systemd issuing below systemctl commands:


 # systemctl enable getty@tty8.service Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty9.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty10.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty11.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

systemctl enable getty@tty12.service

Created symlink /etc/systemd/system/ -> /lib/systemd/system/getty@.service.

Once the TTYS tty7 to tty12 are enabled you will be able to switch to this consoles either if you have a physical LCD / CRT monitor or KVM switch connected to the machine mounted on the Rack shelf once you're in the Data Center or will be able to see it once connected remotely via the Management IP Interface (ILO) remote console.

2. Taking screenshot of the physical console TTY with fbcat

For example below is a screenshot of the 10th enabled tty10:


As you can in the screenshot I've used the nice tool fbcat that can be used to make a screenshot of remote console. This is very useful especially if remote access via a SSH client such as PuTTY / MobaXterm is not there but you have only a physical attached monitor access on a DCs that are under a heavy firewall that is preventing anyone to get to the system remotely. For example screenshotting the physical console in case if there is a major hardware failure occurs and you need to dump a hardware error message to a flash drive that will be used to later be handled to technicians to analyize it and exchange the broken server hardware part.

Screenshots of the CLI with fbcat is possible across most Linux distributions where as usual.

In Debian you have to first instal the tool via :

# apt install –yes fbcat

and on RedHats / CentOS / Fedoras

# yum install -y fbcat

Taking screenshot once tool is on the server of whatever you have printed on console is as easy as

# fbcat > tty_name.ppm

Note that you might want to convert the .ppm created picture to png with any converter such as imagemagick's convert command or if you have a GUI perhaps with GNU Image Manipulation Tool (GIMP).

3. Enabling every rsyslog handled message to log to Physical TTY12

To make everything such as errors, notices, debug, warning messages  become instantly logging towards above added new /dev/tty12.

Open /etc/rsyslog.conf and to the end of the file append below line :

   *.=notice;*.=warn   /dev/tty12

To make rsyslog load its new config restart it:


# systemctl status rsyslog




rsyslog.service – System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2020-08-10 04:09:36 EEST; 2 days ago
     Docs: man:rsyslogd(8)
 Main PID: 671 (rsyslogd)
    Tasks: 4 (limit: 4915)
   Memory: 12.5M
   CGroup: /system.slice/rsyslog.service
           └─671 /usr/sbin/rsyslogd -n -iNONE


авг 12 00:00:05 pcfreak rsyslogd[671]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="671" x-info="https://www.rsyslo
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


systemctl restart rsyslog

That's all folks navigate by pressing simultaneously CTRL + ALT + F12 to get to TTY12 or use ALT + LEFT / ALT + RIGHT ARROW (console switch commands) till you get to the console where everything should be now logged.

Enjoy and if you like this article share to tell your sysadmin friends about this nice hack  ! 🙂




Check server Internet connectivity Speedtest from Linux terminal CLI

Friday, August 7th, 2020


If you are a system administrator of a dedicated server and you have no access to Xserver Graphical GNOME / KDE etc. environment and you wonder how you can track the bandwidth connectivity speed of remote system to the internet and you happen to have a modern Linux distribution, here is few ways to do a speedtest.

1. Use speedtest-cli command line tool to test connectivity


speedtest-cli is a tiny tool written in python, to use it hence you need to have python installed on the server.
It is available both for Redhat Linux distros and Debians / Ubuntus etc. in the list of standard installable packages.

a) Install speedtest-cli on Fedora / CentOS / RHEL

On CentOS / RHEL / Scientific Linux lower than ver 8:



$ sudo yum install python

On CentOS 8 / RHEL 8 user type the following command to install Python 3 or 2:



$sudo yum install python3
$ sudo yum install python2




On Fedora Linux version 22+



$ sudo dnf install python
$ sudo dnf install pytho3


Once python is at place download or in case if link is not reachable download mirrored version of on here



$ wget -O speedtest-cli
$ chmod +x speedtest-cli


Then it is time to run script speedtest-screenshot-linux-terminal-console-cli-cmd
To test enabled Bandwidth on the server



$ python speedtest-cli

b) Install speedtest-cli on Debian

On Latest Debian 10 Buster speedtest is available out of the box in regular .deb repositories, so fetch it with apt


# apt install –yes speedtest-cli


You can give now speedtest-cli a try with –bytes arguments to get speed values in bytes instead of bits or if you want to generate an image with test results in picture just like it will appear if you use inside a gui browser, use the –share option





2. Getting connectivity results of all defined speedtest test City Locations

Speedtest has a list of servers through which a Upload and Download speed is tested, to run speedtest-cli to test with each and every server and get a better picture on what kind of connectivity to expect from your server towards the closest region capital cities, fetch speedtest-servers.php list and use a small shell loop below is how:






root@pcfreak:~#  wget
–2020-08-07 16:31:34–
Преобразувам (…,,, …
Connecting to (||:80… успешно свързване.
HTTP изпратено искане, чакам отговор… 301 Moved Permanently
Адрес: [следва]
–2020-08-07 16:31:34–
Connecting to (||:443… успешно свързване.
HTTP изпратено искане, чакам отговор… 307 Temporary Redirect
Адрес: [следва]
–2020-08-07 16:31:35–
Преобразувам (…
Connecting to (||:443… успешно свързване.
HTTP изпратено искане, чакам отговор… 200 OK
Дължина: 211695 (207K) [text/xml]
Saving to: ‘speedtest-servers.php’
speedtest-servers.php                  100%[==========================================================================>] 206,73K  –.-KB/s    in 0,1s
2020-08-07 16:31:35 (1,75 MB/s) – ‘speedtest-servers.php’ saved [211695/211695]

Once file is there with below loop we extract all file defined servers id="" 's 

root@pcfreak:~# for i in $(cat speedtest-servers.php | egrep -Eo 'id="[0-9]{4}"' |sed -e 's#id="##' -e 's#"##g'); do speedtest-cli  –server $i; done
Retrieving configuration…
Testing from Vivacom (…
Retrieving server list…
Retrieving information for the selected server…
Hosted by Telecoms Ltd. (Varna) [38.88 km]: 25.947 ms
Testing download speed……………………………………………………………………..
Download: 57.71 Mbit/s
Testing upload speed…………………………………………………………………………………………
Upload: 93.85 Mbit/s
Retrieving configuration…
Testing from Vivacom (…
Retrieving server list…
Retrieving information for the selected server…
Hosted by GMB Computers (Constanta) [94.03 km]: 80.247 ms
Testing download speed……………………………………………………………………..
Download: 35.86 Mbit/s
Testing upload speed…………………………………………………………………………………………
Upload: 80.15 Mbit/s
Retrieving configuration…
Testing from Vivacom (…




For better readability you might want to add the ouput to a file or even put it to run periodically on a cron if you have some suspcion that your server Internet dedicated lines dies out to some general locations sometimes.

3. Testing UPlink speed with Download some big file from source location

In the past a classical way to test the bandwidth connectivity of your Internet Service Provider was to fetch some big file, Linux guys should remember it was almost a standard to roll a download of Linux kernel source .tar file with some test browser as elinks / lynx / w3c.
speedtest-screenshot-kernel-org-shot1 speedtest-screenshot-kernel-org-shot2
or if those are not at hand test connectivity on remote free shell servers whatever file downloader as wget or curl was used.
Analogical method is still possible, for example to use wget to get an idea about bandwidtch connectivity, let it roll below 500 mb from to /dev/null few times:


$ wget –output-document=/dev/null

$ wget –output-document=/dev/null

$ wget –output-document=/dev/null


# wget -O /dev/null –progress=dot:mega ; date
–2020-08-07 13:56:49–
Resolving (…
Connecting to (||:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 10485760 (10M) [application/octet-stream]
Saving to: ‘/dev/null’

     0K …….. …….. …….. …….. …….. …….. 30%  142M 0s
  3072K …….. …….. …….. …….. …….. …….. 60%  179M 0s
  6144K …….. …….. …….. …….. …….. …….. 90%  204M 0s
  9216K …….. ……..                                    100%  197M=0.06s

2020-08-07 13:56:50 (173 MB/s) – ‘/dev/null’ saved [10485760/10485760]

Fri 07 Aug 2020 01:56:50 PM UTC

To be sure you have a real picture on remote machine Internet speed it is always a good idea to run download of random big files on a certain locations that are well known to have a very stable Internet bandwidth to the Internet backbone routers.

4. Using Simple shell script to test Internet speed

Fetch and use


wget && chmod u+x && bash



5. Using iperf to test connectivity between two servers 


iperf is another good tool worthy to mention that can be used to test the speed between client and server.

To use iperf install it with apt and do on the server machine to which bandwidth will be tested:


# iperf -s 


On the client machine do:


# iperf -c 


where is the IP of the server where iperf was spawned to listen.

6. Using Netflix fast to determine Internet connection speed on host


fast is a service provided by Netflix. Its web interface is located at and it has a command-line interface available through npm (npm is a package manager for nodejs) so if you don't have it you will have to install it first with:

# apt install –yes npm


Note that if you run on Debian this will install you some 249 new nodejs packages which you might not want to have on the system, so this is useful only for machines that has already use of nodejs.


$ fast


     82 Mbps ↓

The command returns your Internet download speed. To get your upload speed, use the -u flag:


$ fast -u


   ⠧ 80 Mbps ↓ / 8.2 Mbps ↑


7. Use speedometer / iftop to measure incoming and outgoing traffic on interface

If you're measuring connectivity on a live production server system, then you might consider that the measurement output might not be exactly correct especially if you're measuring the Uplink / Downlink on a Heavy loaded webserver / Mail Server / Samba or DNS server.
If this is the case a very useful tools to consider to extract the already taken traffic used on your Incoming and Outgoing ( TX / RX ) Network interfaces
are speedometer and iftop, they're present and installable depending on the OS via yum / apt or the respective package manager.


To install on Debian server:




# apt install –yes iftop speedometer


The most basic use to check the live received traffic in a nice Ncurses like text graphic is with: 





# speedometer -r 


To generate real time ASCII art graph on RX / TX traffic do:



# speedometer -r eth0 -t eth0






# iftop -P -i eth0









Reinstall all Debian packages with a copy of apt deb package list from another working Debian Linux installation

Wednesday, July 29th, 2020


Few days ago, in the hurry in the small hours of the night, I've done something extremely stupid. Wanting to move out a .tar.gz binary copy of qmail installation to /var/lib/qmail with all the dependent qmail items instead of extracting to admin user /root directory (/root), I've extracted it to the main Operating system root / directrory.
Not noticing this, I've quickly executed rm -rf var with the idea to delete all directory tree under /root/var just 3 seconds later, I've realized I'm issuing the rm -rf var with the wrong location WITH a root user !!!! Being scared on what I've done, I've quickly pressed CTRL+C to immedately cancel the deletion operation of my /var.


But as you can guess, since the machine has an Slid State Drive drive and SSD memory drive are much more faster in I/O operations than the classical ATA / SATA disks. I was not quick enough to cancel the operation and I've noticed already some part of my /var have been R.I.P-pped in the heaven of directories.

This was ofcourse upsetting so for a while I rethinked the situation to get some ideas on what I can do to recover my system ASAP!!! and I had the idea of course to try to reinstall All my installed .deb debian packages to restore system closest to the normal, before my stupid mistake.

Guess my unpleasent suprise when I have realized dpkg and respectively apt-get apt and aptitude package management tools cannot anymore handle packages as Debian Linux's package dependency database has been damaged due to missing dpkg directory 




Oh man that was unpleasent, especially since I've installed plenty of stuff that is custom on my Mate based desktop and, generally reinstalling it updating the sytem to the latest Debian security updates etc. will be time consuming and painful process I wanted to omit.

So of course the logical thing to do here was to try to somehow recover somehow a database copy of /var/lib/dpkg  if that was possible, that of course led me to the idea to lookup for a way to recover my /var/lib/dpkg from backup but since I did not maintained any backup copy of my OS anywhere that was not really possible, so anyways I wondered whether dpkg does not keep some kind of database backups somewhere in case if something goes wrong with its database.
This led me to this nice Ubuntu thred which has pointed me to the part of my root rm -rf dpkg db disaster recovery solution.
Luckily .deb package management creators has thought about situation similar to mine and to give the user a restore point for /var/lib/dpkg damaged database

/var/lib/dpkg is periodically backed up in /var/backups

A typical /var/lib/dpkg on Ubuntu and Debian Linux looks like so:

hipo@jeremiah:/var/backups$ ls -l /var/lib/dpkg
total 12572
drwxr-xr-x 2 root root    4096 Jul 26 03:22 alternatives
-rw-r–r– 1 root root      11 Oct 14  2017 arch
-rw-r–r– 1 root root 2199402 Jul 25 20:04 available
-rw-r–r– 1 root root 2199402 Oct 19  2017 available-old
-rw-r–r– 1 root root       8 Sep  6  2012 cmethopt
-rw-r–r– 1 root root    1337 Jul 26 01:39 diversions
-rw-r–r– 1 root root    1223 Jul 26 01:39 diversions-old
drwxr-xr-x 2 root root  679936 Jul 28 14:17 info
-rw-r—– 1 root root       0 Jul 28 14:17 lock
-rw-r—– 1 root root       0 Jul 26 03:00 lock-frontend
drwxr-xr-x 2 root root    4096 Sep 17  2012 parts
-rw-r–r– 1 root root    1011 Jul 25 23:59 statoverride
-rw-r–r– 1 root root     965 Jul 25 23:59 statoverride-old
-rw-r–r– 1 root root 3873710 Jul 28 14:17 status
-rw-r–r– 1 root root 3873712 Jul 28 14:17 status-old
drwxr-xr-x 2 root root    4096 Jul 26 03:22 triggers
drwxr-xr-x 2 root root    4096 Jul 28 14:17 updates

Before proceeding with this radical stuff to move out /var/lib/dpkg/info from another machine to /var mistakenyl removed oned. I have tried to recover with the well known:

  • extundelete
  • foremost
  • recover
  • ext4magic
  • ext3grep
  • gddrescue
  • ddrescue
  • myrescue
  • testdisk
  • photorec

Linux file deletion recovery tools from a USB stick loaded with a Number of LiveCD distributions, i.e. tested recovery with:

  • Debian LiveCD
  • Ubuntu LiveCD
  • SystemRescueCD
  • Trinity Rescue Kit
  • Ultimate Boot CD

but unfortunately none of them couldn't recover the deleted files … 

The reason why the standard file recovery tools could not recover ?

My assumptions is after I've done by rm -rf var; from sysroot,  issued the sync (- if you haven't used it check out man sync) command – that synchronizes cached writes to persistent storage and did a restart from the poweroff PC button, this should have worked, as I've recovered like that in the past) in a normal Sys V System with a normal old fashioned blocks filesystem as EXT2 . or any other of the filesystems without a journal, however as the machine run a EXT4 filesystem with a journald and journald, this did not work perhaps because something was not updated properly in /lib/systemd/systemd-journal, that led to the situation all recently deleted files were totally unrecoverable.

1. First step was to restore the directory skele of /var/lib/dpkg

# mkdir -p /var/lib/dpkg/{alternatives,info,parts,triggers,updates}


2. Recover missing /var/lib/dpkg/status  file

The main file that gives information to dpkg of the existing packages and their statuses on a Debian based systems is /var/lib/dpkg/status

# cp /var/backups/dpkg.status.0 /var/lib/dpkg/status


3. Reinstall dpkg package manager to make package management working again

Say a warm prayer to the Merciful God ! and do:

# apt-get download dpkg
# dpkg -i dpkg*.deb


4. Reinstall base-files .deb package which provides basis of a Debian system

Hopefully everything will be okay and your dpkg / apt pair will be in normal working state, next step is to:

# apt-get download base-files
# dpkg -i base-files*.deb


5. Do a package sanity and consistency check and try to update OS package list

Check whether packages have been installed only partially on your system or that have missing, wrong or obsolete control  data  or  files.  dpkg  should suggest what to do with them to get them fixed.

# dpkg –audit

Then resynchronize (fetch) the package index files from their sources described in /etc/apt/sources.list

# apt-get update

Do apt db constistency check:

#  apt-get check

check is a diagnostic tool; it updates the package cache and checks for broken dependencies.

Take a deep breath ! …

Do :

ls -l /var/lib/dpkg
and compare with the above list. If some -old file is not present don't worry it will be there tomorrow.

Next time don't forget to do a regular backup with simple rsync backup script or something like Bacula / Amanda / Time Vault or Clonezilla.

6. Copy dpkg database from another Linux system that has a working dpkg / apt Database

Well this was however not the end of the story … There were still many things missing from my /var/ and luckily I had another Debian 10 Buster install on another properly working machine with a similar set of .deb packages installed. Therefore to make most of my programs still working again I have copied over /var from the other similar set of package installed machine to my messed up machine with the missing deleted /var.

To do so …
On Functioning Debian 10 Machine (Working Host in a local network with IP, I've archived content of /var:

linux:~# tar -czvf var_backup_debian10.tar.gz /var

Then sftped from Working Host towards the /var deleted broken one in my case this machine's hostname is jericho and luckily still had SSHD and SFTP running processes loaded in memory:

jericho:~# sftp root@
sftp> get var_backup_debian10.tar.gz

Now Before extracting the archive it is a good idea to make backup of old /var remains somewhere for example somewhere in /root 
just in case if we need to have a copy of the dpkg backup dir /var/backups

jericho:~# cp -rpfv /var /root/var_backup_damaged

jericho:~# tar -zxvf /root/var_backup_debian10.tar.gz 
jericho:/# mv /root/var/ /

Then to make my /var/lib/dpkg contain the list of packages from my my broken Linux install I have ovewritten /var/lib/dpkg with the files earlier backupped before  .tar.gz was extracted.

jericho:~# cp -rpfv /var /root/var_backup_damaged/lib/dpkg/ /var/lib/


7. Reinstall All Debian  Packages completely scripts


I then tried to reinstall each and every package first using aptitude with aptitude this is done with

# aptitude reinstall '~i'

However as this failed, tried using a simple shell loop like below:

for i in $(dpkg -l |awk '{ print $2 }'); do echo apt-get install –reinstall –yes $i; done

Alternatively, all .deb package reninstall is also possible with dpkg –get-selections and with awk with below cmds:

dpkg –get-selections | grep -v deinstall | awk '{print $1}' > list.log;
awk '$1=$1' ORS=' ' list.log > newlist.log
apt-get install –reinstall $(cat newlist.log)

It can also be run as one liner for simplicity:

dpkg –get-selections | grep -v deinstall | awk '{print $1}' > list.log; awk '$1=$1' ORS=' ' list.log > newlist.log; apt-get install –reinstall $(cat newlist.log)

This produced a lot of warning messages, reporting "package has no files currently installed" (virtually for all installed packages), indicating a severe packages problem below is sample output produced after each and every package reinstall … :

dpkg: warning: files list file for package 'iproute' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'brscan-skey' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libapache2-mod-php7.4' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libexpat1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libexpat1:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'php5.6-readline' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'linux-headers-4.19.0-5-amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libgraphite2-3:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libgraphite2-3:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libbonoboui2-0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libxcb-dri3-0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libxcb-dri3-0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'liblcms2-2:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'liblcms2-2:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libpixman-1-0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libpixman-1-0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'gksu' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'liblogging-stdlog0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'mesa-vdpau-drivers:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'mesa-vdpau-drivers:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libzvbi0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libzvbi0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libcdparanoia0:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libcdparanoia0:i386' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python-gconf' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'php5.6-cli' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libpaper1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package '' missing; assuming package has no files currently installed

After some attempts I found a way to be able to work around the warning message, for each package by simply reinstalling the package reporting the issue with

apt –reinstall $package_name

Though reinstallation started well and many packages got reinstalled, unfortunately some packages such as apache2-mod-php5.6 and other php related ones  started failing during reinstall ending up in unfixable states right after installation of binaries from packages was successfully placed in its expected locations on disk. The failures occured during the package setup stage ( dpkg –configure $packagename) …

The logical thing to do is a recovery attempt with something like the usual well known by any Debian admin:

apt-get install –fix-missing

As well as Manual requesting to reconfigure (issue re-setup) of all installed packages also did not produced a positive result

dpkg –configure -a

But many packages were still failing due to dpkg inability to execute some post installation scripts from respective .deb files.
To work around that and continue installing the rest of packages I had to manually delete all files related to the failing package located under directory 


For example to omit the post installation failure of libapache2-mod-php5.6 and have a succesful install of the package next time I tried reinstall, I had to delete all /var/lib/dpkg/info/libapache2-mod-php5.6.postrm, /var/lib/dpkg/info/libapache2-mod-php5.6.postinst scripts and even sometimes everything like libapache2-mod-php5.6* that were present in /var/lib/dpkg/info dir.

The problem with this solution, however was the package reporting to install properly, but the post install script hooks were still not in placed and important things as setting permissions of binaries after install or applying some configuration changes right after install was missing leading to programs failing to  fully behave properly or even breaking up even though showing as finely installed …

The final solution to this problem was radical.
I've used /var/lib/dpkg database (directory) from ther other working Linux machine with dpkg DB OK found in var_backup_debian10.tar.gz (linux:~# host with a working dpkg database) and then based on the dpkg package list correct database responding on jericho:~# to reinstall each and every package on the system using Debian System Reinstaller script taken from the internet.
Debian System Reinstaller works but to reinstall many packages, I've been prompted again and again whether to overwrite configuration or keep the present one of packages.
To Omit the annoying [Y / N ] text prompts I had made a slight modification to the script so it finally looked like this:

# Debian System Reinstaller
# Copyright (C) 2015 Albert Huang
# Copyright (C) 2018 Andreas Fendt

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU General Public License for more details.

# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <>.

# —
# This script assumes you are using a Debian based system
# (Debian, Mint, Ubuntu, #!), and have sudo installed. If you don't
# have sudo installed, replace "sudo" with "su -c" instead.

pkgs=`dpkg –get-selections | grep -w 'install$' | cut -f 1 |  egrep -v '(dpkg|apt)'`

for pkg in $pkgs; do
    echo -e "\033[1m   * Reinstalling:\033[0m $pkg"    

    apt-get –reinstall -o Dpkg::Options::="–force-confdef" -o Dpkg::Options::="–force-confold" -y install $pkg || {
        echo "ERROR: Reinstallation failed. See reinstall.log for details."
        exit 1
done working modified version of Albert Huang and Andreas Fendt script  can be also downloaded here.

Note ! Omitting the text confirmation prompts to install newest config or keep maintainer configuration is handled by the argument:


-o Dpkg::Options::="–force-confold

I however still got few NCurses Console selection prompts during the reinstall of about 3200+ .deb packages, so even with this mod the reinstall was not completely automatic.

Note !  During the reinstall few of the packages from the list failed due to being some old unsupported packages this was ejabberd, ircd-hybrid and a 2 / 3 more.
This failure was easily solved by completely purging those packages with the usual

# dpkg –purge $packagename

and reruninng on each of the failing packages.

Note ! The failing packages were just old ones left over from Debian 8 and Debian 9 before the apt-get dist-upgrade towards 10 Duster.
Eventually I got a success by God's grance, after few hours of pains and trials, ending up in a working state package database and a complete set of freshly reinstalled packages.

The only thing I had to do finally is 2 hours of tampering why GNOME did not automatically booted after the system reboot due to failing gdm
until I fixed that I've temprary used ligthdm (x-display-manager), to do I've

dpkg –reconfigure gdm3


 to work around this I had to also reinstall few libraries, reinstall the xorg-server, reinstall gdm and reinstall the meta package for GNOME, using below set of commands:

apt-get install –reinstall libglw1-mesa libglx-mesa0
apt-get install –reinstall libglu1-mesa-dev
apt install –reinstallgsettings-desktop-schemas
apt-get install –reinstall xserver-xorg-video-intel
apt-get install –reinstall xserver-xorg
apt-get install –reinstall xserver-xorg-core
apt-get install –reinstall task-desktop
apt-get install –reinstall task-gnome-desktop


As some packages did not ended re-instaled on system because on the original host from where /var/lib/dpkg db was copied did not have it I had to eventually manually trigger reinstall for those too:


apt-get install –reinstall –yes vlc
apt-get install –reinstall –yes thunderbird
apt-get install –reinstall –yes audacity
apt-get install –reinstall –yes gajim
apt-get install –reinstall –yes slack remmina
apt-get install –yes k3b
pt-get install –yes gbgoffice
pt-get install –reinstall –yes skypeforlinux
apt-get install –reinstall –yes vlc
apt-get install –reinstall –yes libcurl3-gnutls libcurl3-nss
apt-get install –yes virtualbox-5.2
apt-get install –reinstall –yes vlc
apt-get install –reinstall –yes alsa-tools-gui
apt-get install –reinstall –yes gftp
apt install ./teamviewer_15.3.2682_amd64.deb –yes


Note that some of above packages requires a properly configured third party repositories, other people might have other packages that are missing from the dpkg list and needs to be reinstalled so just decide according to your own case of left aside working system present binaries that doesn't belong to any dpkg installed package.

After a bit of struggle everything is back to normal Thanks God! 🙂 !


Send TCP / UDP strings and commands to Local and Remote Applications without netcat with Bash

Friday, July 24th, 2020


Did you ever needed to send TCP / UDP packets manually to send commands to local or remote applications, having a fully functional BASH Shell but not having the luxury to have NC (Netcat Swiss Army Knife of Networking) tool?
This happens if you have some Linux based embeded device as Arduino or a Linux server with a high security PCI requirement which can't affort to have Netcat in place or another portable hardware with a Linux kernel, that needs to communicate in UDP for any reason but you don't want to waste additional 28KB or physically you have access to a Linux device that doesn't have netcat but you want to be able to send UDP externally …

SInce some time in newer GNU Bash's releases support for TCP / UDP data sending is described in Bash's Manual and should be working it is not as good as you might expect but for a small things it could save you the day.

The syntax to use it is:


To open new socket connection to a UDP / TCP protocol with bash you have to simply open a new Shell handler (lets say 3) to:






1. Get GOOGLE HTML Source with simple BASH / Getting URL Index with bash sockets

If you happen to have access to a machine where no network downloader tool or a text browser such as curl, wget, lynx, links is available but you want to dump the content of a index.html or any other URL with simply bash you can do it like so:

exec 3<>/dev/tcp/
echo -e "GET / HTTP/1.1\r\n\r\n" >&3 

cat <&3

If you need to open a connection to a Internet Domain with bash and store the output into a separate .html file:

exec 3<>/dev/tcp/
echo -e "GET / HTTP/1.1\r\n\r\n" >&3 

cat <&3 | tee -a output.html

Note that this will work only if you're logged into into an interactive shell.
If you want instead do it from a shell script (and omit usage) of wget etc. use something like basic script :

exec 3<>/dev/tcp/
printf "GET /\r\n\r\n" >&3
while IFS= read -r -u3 -t2 line || [[ -n “$line” ]]; do echo "$line"; done
exec 3>&-
exec 3<&-


2. Sending UDP protocol data via bash socket

To send test  variables or commands data to localhost ( UDP listening service:

echo 'TEST COMMAND' > /dev/udp/


echo "Any UDP data" > /dev/udp/

If you happent to have netcat or running on a bash shell that doesn't properly support TCP / UDP sending you can always do it netcat way:

echo "Command" | nc -u -w0 3000

Of course this little hack is useful just for simple things and eventually for more comlex stuff and scripting you would like to use a fully functional HTML reader ( W3C compliant Web Browser )
still  for a quick dirty stuff Bash socketing from the console rocks pretty much ! 🙂


How to install GUI on CentOS 7 Minimal and set Gnome Graphical Environment to automatically load on system boot

Wednesday, July 22nd, 2020


I have installed CentOS 7.7 Minimal Server Linux on a VirtualBox Virtual Environment as a test bed machine.

The system got installed easily succesfully with the standard CentOS python based graphical installer, however I needed to place various software which was not there
and for that of course I needed to have a network enabled.

To make network working instead of the default Network NAT configuration for the Virtual Machine I needed to use the Network to be Attached to a Bridged Adapter in order to make
my Windows machine to provide network and (internet) access to VirtualMachine.


Then to make networking work after booting into CentOS I had to manually fetch IP via DHCP protocol with command:

[root@centos :~]# dhclient enp0s3



To make the setting permanent I had to also of course modify /etc/sysconfig/network-scripts/ifcfg-enp0s3 file and change 









On next reboot CentOS boots normally with networking as expected

As by default CentOS Minimal does not provide any graphical environment however I needed to have it in my VM in order to be able to use (VirtualBox Guest Additions plugins) that enabled the CentOS Operating System to show in Virtualbox in fullscreen and to enable the Copy / Paste buffers to work from The Hypervisor (Windows in that case) and the Guest VM (the CentOS VM).

In CentOS terminology metapackages (a  grouped package under a certain name, alias) are called simply groups) there is a "GNOME Desktop" group that can be used to install the GNOME Graphical Command from that point on with yum, like so:

[root@centos :~]# yum -y groups install "GNOME Desktop"

In a while the graphical environment will be in place, the command will install about 1300+ RPM packages, this will take about 5 minutes or so depending on your bandwidth connectivity. Once all is installed and configured succesfully you can use the good old startx command to launch GNOME.



[root@centos :~]# startx


This of course will make Xserver and GNOME to run one time and on next reboot, you will end up in a plain text mode environment, so perhaps you will need to make the autolaunch of GNOME environment automatically on each boot in CentOS just like in most modern Linux distributions that use SYSTEMD to handle runlevels, you will need to configure it by changing the systemd default configured target via systemctl:


[root@centos :~]# systemctl list-units –type target | egrep "eme|res|gra|mul"       loaded active active Graphical Interface      loaded active active Multi-User System


[root@centos :~]# systemctl set-default


[root@centos :~]# systemctl set-default


[root@centos :~]# systemctl set-default

Next step was to enable the Guest Additions to do so I had to install in advance 2 RPM packages kernel-headers and kernel-devel

[root@centos :~]# yum install -y kernel-headers kernel-devel

Then I had to mount and run the script to enable them, i.e.:


[root@centos :~]# mkdir /mnt/cdrom
[root@centos :~]# mount /dev/cdrom /mnt/cdrom

[root@centos :~]# cd /mnt/cdrom/
[root@centos :~]# sh






Get daily E-Mail Reports statistics on postfix Linux mail server

Tuesday, July 14th, 2020

I've had today a task at work to monitor a postfix mail send and received emails (MAIL FROM / RPCT TO) and get out a simple statistics on what kind of emails are coming and going out from the Postfix SMTP on a server?

Below is shortly explained how I did it plus you will learn how you can use something more advanced to get server mail count, delivery status, errors etc. daily.

1. Using a simple script to process /var/log/messages

For that I made a small script to do the trick, the script simply checks mail delivery logged information from /var/log/maillog process a bit sort and logs in a separate log daily.

# Process /var/log/maillog extract from= and to= mails sort
# And log mails to $LOGF
# Author Georgi Georgiev 14.07.2020

DATE_FORM=$(date +'%m_%d_%y_%H_%M_%S_%h_%m');
CUR_DATE=$(date +'%m_%d_%y_%T');
echo "Processing /var/log/maillog";
echo "Processing /var/log/maillog" > $LOGF;
echo >>$LOGF
echo "!!! $CUR_DATE # Sent MAIL FROM: addresses: !!!" >> $LOGF;
grep -E 'from=' /var/log/maillog|sed -e 's#=# #g'|awk '{ print $8 }'|sed -e 's#<# #g' -e 's#># #g' -e 's#\,##'|sort -rn|uniq >> $LOGF;

echo "!!! $CUR_DATE # Receive RCPT TO: addresses !!!" >>$LOGF;
grep -E 'to=' /var/log/maillog|sed -e 's#=# #g'|awk '{ print $8 }'|sed -e 's#<# #g' -e 's#># #g' -e 's#\,##'|sort -rn|uniq >> $LOGF;

You can get a copy of the script here.

I've set the script to run via a crond scheduled job once early in the mornthing and I'll leave it like that for 5 days or so to get a good idea on what are the mailboxes that are receiving incoming mail.

The cron I've set to use is as follows:

# crontab -u root -l 
05 03 * * *     sh /home/gge/ >/dev/null 2>&1


This will be necessery later for a Email Server planned migration to relay its mail via another MTA host.


2. Getting More Robust Postifx Mail Statistics from logs

My little script is of course far from best solution to get postfix mail statistics from logs.

If you want something more professional and you need to have a daily report on what mails sent to mail server and mails sent from the MTA to give you information about the Email delivery queue status, number of successful and failed emails from a mail sender / recipient and a whole bunch of useful info you can use something more advanced such as pflogsumm perl script to get daily / weekly monthly mail delivery statistics.

What can pflogsumm do for you ?



Pflogsumm is a log analyzer/summarizer for the Postfix MTA. It is
designed to provide an overview of Postfix activity, with just enough
detail to give the administrator a “heads up” for potential trouble
spots and fixing any SMTP and email related issues.

Pflogsumm generates summaries and, in some cases, detailed reports of
mail server traffic volumes rejected and bounced email and server
warnings, errors, and panics.

At the time of writting this article it is living on just in case if's official download location disappears at some time in future here is pflogsumm-1.1.3.tar.gz mirror stored on

– Install pflogsumm

Use of pflogsumm is pretty straight forward, you download unarchive the script to some location such as /usr/local/bin/  add the script executable flag and you run it to create a Postfix Mail Log statistics report for you

wget -O /usr/local/src/pflogsumm-1.1.3.tar.gz


# mkdir -p /usr/local/src/
# cd /usr/local/src/
# tar -zxvf pflogsumm-1.1.3.tar.gz
# cd pflogsumm-1.1.3/

# mv /usr/local/pflogsumm-1.1.3/ /usr/local/bin/pflogsumm
# chmod a+x /usr/local/bin/pflogsumm

That's all, assuming you have perl installed on the system with some standard modules, we're now good to go: 

To give it a test report to the command line:

# /usr/local/bin/pflogsumm -d today /var/log/maillog


To generate mail server use report and launch to some email of choice do:

# /usr/local/bin/pflogsumm -d today /var/log/maillog | mail -s Mailstats

To make pflogsumm report everyday various interesting stuff such as (message deferrals, message bounce, details, smtp delivery failures, fatal errors, recipients by message size etc. add some cronjob like below to the server:

# /usr/sbin/pflogsumm -d yesterday /var/log/maillog | mail -s Mailstats | mail -s Mailstats

If you need a GUI graphical mail monitoring in a Web Browser, you will need to install a webserver with a perl / cgi support,  RRDTools and MailGraph.


Linux Send Monitoring Alert Emails without Mail Server via relay SMTP with ssmtp / msmtp

Friday, July 10th, 2020


If you have to setup a new Linux server where you need to do a certain local running daemons monitoring with a custom scripts on the local machine Nagios / Zabbix / Graphana etc. that should notify about local running custom programs or services in case of a certain criteria is matched or you simply want your local existing UNIX accounts to be able to send outbound Emails to the Internet.

Then usually you need to install a fully functional SMTP Email server that was Sendmail or QMAIL in old times in early 21st century andusually postfix or Exim in recent days and configure it to use as as a Relay mail server some Kind of SMTP.

The common Relay smtp setting would be such as Google's, Yahoo!'s relay host, or External configured MTA Physical server with proper PTR / MX records or a SMTP hosted on a virtual machine living in Amazon's AWS or m$ Azure that is capable to delivere EMails to the Internet.

Configuring the local installed Mail Transport Agent (MTA) as a relay server is a relatively easy task to do but of course why should you have a fully stacked MTA service with a number of unnecessery services such as Email Queue, Local created mailboxes, Firewall rules, DNS records, SMTP Auth, DKIM keys etc. and even the ability to acccept any emails back in case if you just want to simply careless send and forget with a confirmation that remote email was send successfully?

This is often the case for some machines and especially with the inclusion of technologies such as Kubernettes / Clustered environments / VirtualMachines small proggies such as ssmtp / msmtp that could send mail without a Fully functional mail server installed on localhost ( ) is true jams.

ssmtp program is Simple Send-only sendMail emulator  has been around in Debian GNU / Linux, Ubuntu, CentOS and mostly all Linuxes for quite some a time but recently the Debian package has been orphaned so to install it on a deb based server host you need to use instead msmtp.

1. Install ssmtp on CentOS / Fedora / RHEL Linux

In RPM distributions you can't install until epel-release repository is enabled.

[root@centos:~]# yum –enablerepo=extras install epel-release

[root@centos:~]# yum install ssmtp

2. Install ssmp / msmtp Debian / Ubuntu Linux

If you run older version of Debian based distribution the package to install is ssmtp, e.g.:

root@debian:~# apt-get install –yes ssmtp

On Newer Debians as of Debian 10.0 Buster onwards install instead

root@debian:~# apt install –yes msmtp-mta

can save you a lot of effort to keep an eye on a separately MTA hanging around and running as a local service eating up resources that could be spared.

3. Configure Relay host for ssmtp

A simple configuration to make ssmtp use SMTP servers as a relay host below:

linux:~# cat << EOF > /etc/ssmtp/ssmtp.conf
# /etc/ssmtp/ssmtp.conf
# The user that gets all the mails (UID < 1000, usually the admin)
# The full hostname.  Must be correctly formed, fully qualified domain name or GMail will reject connection.
# The mail server (where the mail is sent to), both port 465 or 587 should be acceptable
# See also

# The address where the mail appears to come from for user authentication.
# Email 'From header's can override the default domain?


# Username/Password
# Use SSL/TLS before starting negotiation
logfile        ~/.msmtp.log


This configuration is very basic and it is useful only if you don't want to get delivered mails back as this functionality is also supported even though rarely used by most.

One downside of ssmtp is mail password will be plain text, so make sure you set proper permissions to /etc/ssmtp/ssmtp.conf

– If your Gmail account is secured with two-factor authentication, you need to generate a unique App Password to use in ssmtp.conf. You can do so on your App Passwords page. Use Gmail username (not the App Name) in the AuthUser line and use the generated 16-character password in the AuthPass line, spaces in the password can be omitted.

– If you do not use two-factor authentication, you need to allow access to unsecure apps.

4. Configuring different msmtp for separate user profiles

SSMTP is capable of respecting multiple relays for different local UNIX users assuming each of whom has a separate home under /home/your-username

To set a certain user lets say georgi to relay smtp sent emails with mail or mailx command create ~/.msmtprc


linux:~# vim ~/.msmtprc

Append configuration like:

# Set default values for all following accounts.
port 587
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
account gmail
from <user>
auth on
user <user>
passwordeval gpg –no-tty -q -d ~/.msmtp-gmail.gpg
# Set a default account

account default : gmail

To add it for any different user modify the respective fields and set the different Mail hostname etc.

5. Using mail address aliases

msmtp also supports mail aliases, to make them work you will need to have file /etc/msmptrc with

aliases               /etc/aliases

Standard aliasses them should work 

linux:~# cat /etc/aliases
# Example aliases file
# Send root to Joe and Jane
# Send everything else to admin
default: admin@domain.example


6. Get updated when your Debian servers have new packages to update 

msmpt can be used for multiple stuff one example use would be to use it together with cron to get daily updates if there are new debian issued security or errata update pending packages, to do so you can use the apticron shell script.

To use it on debian install the apticron pack:

root@debian:~# apt-get install –yes apticron

apticron has the capability to:

 * send daily emails about pending upgrades in your system;
 * give you the choice of receiving only those upgrades not previously notified;
 * automatically integrate to apt-listchanges in order to give you by email the
   new changes of the pending upgrade packages;
 * handle and warn you about packages put on hold via aptitude/dselect,
   avoiding unexpected package upgrades (see #137771);
 * give you all these stuff in a simple default installation;


To configure it you have to place a config copy the one from /usr/lib/apticron/apticron.conf to /etc/apticron/apticron.conf

The only important value to modify in the config is the email address to which an apt-listchanges info for new installable debs from the apt-get dist-upgrade command. Output from them will be be send to the configured EMAIL field  in apticron.conf.


The timing at which the offered new pending package update reminder will be sent is controlled by /etc/cron.d/apticron

debian:~# cat /etc/cron.d/apticron
# cron entry for apticron

48 * * * * root if test -x /usr/sbin/apticron; then /usr/sbin/apticron –cron; else true; fi

apticron will use the local previous ssmtp / msmpt program to deliver to configured mailbox.
To manually trigger apticron run:

root@debian:~# if test -x /usr/sbin/apticron; then /usr/sbin/apticron –cron; else true; fi

7. Test whether local mail send works to the Internet

To test mail sent we can use either mail / mailx or sendmail command or some more advanced mailer as alpine or mutt.

Below is few examples.

linux:~$ echo -e "Subject: this is the subject\n\nthis is the body" | mail

To test attachments to mail also works run:

linux:~$ mail -s "Subject" < mail-content-to-attach.txt


Prepare the mail you want to send and send it with sendmail

linux:~$ vim test-mail.txt
Subject: Test Email
This is a test mail.

linux:~$ sendmail -t < test-mail.txt

Sending encoded atacchments with uuencode is also possible but you will need sharutils Deb / RPM package installed.

To attach lets say 2 simple text files uuencoded:

linux:~$ uuencode file.txt myfile.txt | sendmail

echo "

To: From: Subject: A test Hello there." > test.mail

linux:~$ cat test.mail | msmtp -a default <username>

That's all folks, hope you learned something, if you know of some better stuff like ssmtp please shar e it.

Stop SSH Bruteforce authentication attempt Attacks with fail2ban

Monday, July 6th, 2020

Fail2ban stop restrict ssh bruteforce authentication attempt attacks

Most of webmasters today have some kind of SSH console remote access to the server and the OpenSSH Secure Shell service is usually not filtered for specific Networks but fully accessbible on the internet. This is especially true for home brew Linux Web servers as well as small to mid sized websites and blogs hosted on a cheap dedicated servers hosted in UK2 / Contabo RackSpace etc.

Brute force password guess attack tools such as Hydra and a distributed password dictionary files have been circulating quite for a while and if the attacker has enough time as well as a solid dictionary base, as well as some kind of relatively weak password you can expect that sooner or later some of the local UNIX accounts can be breaked and the script kiddie can get access to your server and make quickly a havoc, if he is lucky enough to be able to exploit some local vulnerability and get root access …

If you're a sysadmin that has to manage the Linux server and you do a routine log reading on the machine, you will soon get annoyed of the ever growing amount of different users, that are trying to login unsucessfully to the SSH (TCP port 22) service filling up the logs with junk and filling up disk space for nothing as well as consuming some CPU and Memory resources for nothing, you will need some easy  solution to make brute force attacks from an IP get filtered after few unsuccessful login attempts.

The common way to protect SSH would then be to ban an IP address from logging in if there are too many failed login attempts based on an automatic firewall inclusion of any IP that tried to unsuccessfully login lets say 5 or 10 reoccuring times.

In Linux there is a toll called “fail2ban” (F2B) used to limit brute force authentication attempts.

F2B works with minimal configuration and besides being capable of protecting the SSH service, it can be set to protect a lot of other Server applications like;

 Apache / NGINX Web Servers with PHP / Mail Servers (Exim, Postfix, Qmail, Sendmail), POP3 IMAP / AUTH services (Dovecot, Courier, Cyrus), DNS Bind servers, MySQL DBs, Monitoring tools such as Nagios, FTP servers (ProFTPD / PureFTP), Proxy servers (Squid), WordPress sites (wp-login) brute force attacks, Web Mail services (Horde / Roundcube / OpenWebmail), jabber servers etc.

To get a better overview below is F2B package description:

linux:~# apt-cache show fail2ban|grep -i description-en -A 21
Description-en: ban hosts that cause multiple authentication errors
 Fail2ban monitors log files (e.g. /var/log/auth.log,
 /var/log/apache/access.log) and temporarily or persistently bans
 failure-prone addresses by updating existing firewall rules.  Fail2ban
 allows easy specification of different actions to be taken such as to ban
 an IP using iptables or hostsdeny rules, or simply to send a notification
 By default, it comes with filter expressions for various services
 (sshd, apache, qmail, proftpd, sasl etc.) but configuration can be
 easily extended for monitoring any other text file.  All filters and
 actions are given in the config files, thus fail2ban can be adopted
 to be used with a variety of files and firewalls.  Following recommends
 are listed:
  – iptables/nftables — default installation uses iptables for banning.
    nftables is also suported. You most probably need it
  – whois — used by a number of *mail-whois* actions to send notification
    emails with whois information about attacker hosts. Unless you will use
    those you don't need whois
  – python3-pyinotify — unless you monitor services logs via systemd, you
    need pyinotify for efficient monitoring for log files changes


Using fail2ban is easy as there is a multitude of preexisting filters and actions (that gets triggered on a filter match) already written by different people usually found in /etc/fail2ban/filter.d an /etc/fail2ban/action.d.
as well as custom action / filter / jails is easy to do.
Fail2ban is available as a standard distro RPM / DEB package
on most modern versions of Ubuntu (16.04 and later), Debian, Mint and CentOS 7, OpenSuSE, Fedora etc.

1. Install Fail2ban package

– On Deb based distros do the usual:


linux:~# apt update
linux:~# apt install –yes fail2ban

On RPM distros Fedora / CentOS / SuSE etc.

linux:~# sudo yum -y install epel-release
linux:~# sudo yum -y install fail2ban


2. Enable fail2ban ban rules for SSH failed authentication filtering


Create the file /etc/fail2ban/jail.local :

# cat > /etc/fail2ban/jail.local

Paste below content:

# Ban hosts for one hour:
bantime = 360
# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport


enabled = true
maxretry = 10
findtime = 43200
# ban for 1 day
bantime = 3600
# ban for 1 day
#bantime = 86400


This config will ban for 1 day any IP that tries to access more than 10 unsuccesful times sshd daemon.
This works through IPTABLES indicated by config banaction = iptables-multiport config making fail2ban to automatically add a new iptables block rule valid for 1 day.

To close the file press CTRL + D simulanesly

  • maxretry controls the maximum number of allowed retries.
  • findtime specifies the time window (in seconds) which should be considered for banning an IP. (43200 seconds is 12 hours)
  • bantime specifies the time window (in seconds) for which the IP address will be banned (86400 seconds is 24 hours).

If SSH listens to a different port from 22 on the machine, you can specify the port number with port = <port_number> in this file.


3. Start fail2ban service

Either use the /etc/init.d/fail2ban start script or systemd systemctl

linux:~# systemctl enable fail2ban
linux:~# systemctl restart fail2ban


4. Fail2ban operational principle in short

Fail2ban uses Filters, Actions And Jails:

Filtersspecify certain patterns of text that Fail2ban should recognize in log files.
Actionsare things Fail2ban can do once a filter is matched.
Jailstell Fail2ban to match a filter on some logs. When the number of matches goes beyond a certain limit specified in the jail, Fail2ban takes an action specified in the jail.

If still wonder what is Fail2ban jail? Each configured jail tells fail2ban to look at system logs and take actions against attacks on a configured service, in our case OpenSSH service.

5. Blocking repeated unsuccessful password authentication attempts for longer periods

If more than number of failed ssh logins happen to occur (lets say 35 reoccuring ones in /var/log/auth.log (the debian failed ssh login file) or /var/log/secure (redhat distros failed ssh log file).
You will perhaps want to permanently block this IP for 3 days or so, here is how:

banaction = iptables-multiport
maxretry  = 35
findtime  = 259200
bantime   = 608400
enabled   = true
filter    = sshd


Add this to [sshlongterm] section to make it finally look like this:

port      = ssh
logpath   = %(sshd_log)s
banaction = iptables-multiport
maxretry  = 35
findtime  = 259200
bantime   = 608400
enabled   = true
filter    = sshd

Of course to load new config restart fail2ban

# /etc/init.d/fail2ban restart

Default jail as well as the sshlongterm jail should now work together. Short term attacks will be handled by the default jail under [sshd], and the long term attacks handled by our own second jail [sshlongterm].

6. Checking which intruders were blocked by fail2ban

Fail2ban creates a separate chain f2b-sshd to which it adds each blocked IP for the period of time preset in the config, to list it:

linux:~# /sbin/iptables -L f2b-sshd
Chain f2b-sshd (1 references)
target     prot opt source               destination         
REJECT     all  —          anywhere             reject-with icmp-port-unreachable
REJECT     all  —  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  anywhere             reject-with icmp-port-unreachable
REJECT     all  —   anywhere             reject-with icmp-port-unreachable
REJECT     all  —          anywhere             reject-with icmp-port-unreachable
REJECT     all  —  anywhere             reject-with icmp-port-unreachable
REJECT     all  —        anywhere             reject-with icmp-port-unreachable
REJECT     all  —        anywhere             reject-with icmp-port-unreachable
REJECT     all  —       anywhere             reject-with icmp-port-unreachable
REJECT     all  —        anywhere             reject-with icmp-port-unreachable
REJECT     all  —  anywhere             reject-with icmp-port-unreachable
REJECT     all  —       anywhere             reject-with icmp-port-unreachable
REJECT     all  —  anywhere             reject-with icmp-port-unreachable
REJECT     all  —  anywhere             reject-with icmp-port-unreachable
REJECT     all  —     anywhere             reject-with icmp-port-unreachable
REJECT     all  —  anywhere             reject-with icmp-port-unreachable
REJECT     all  —      anywhere             reject-with icmp-port-unreachable
REJECT     all  —         anywhere             reject-with icmp-port-unreachable
RETURN     all  —  anywhere             anywhere            



What we have seen here is how to make fail2ban protect Internet firewall unrestricted SSHD Service to filter out 1337 skript kiddie 'hackers' out of your machine. With a bit of tuning could not only break the occasional SSH Brute Force bot scanners that craw the new but could even mitigate massive big botnet initiated brute force attacks to servers.
Of course fail2ban is not a panacea and to make sure you won't get hacked one days better make sure to only allow access to SSH service only for a certain IP addresses or IP address ranges that are of your own PCs.