Posts Tagged ‘root server’

Update reverse sshd config with cronjob to revert if sshd reload issues

Friday, February 12th, 2021

Reading Time: 2minutes


Say you're doing ssh hardening modifying /etc/ssh/sshd_config for better system security or just changing options in sshd due to some requirements. But you follow the wrong guide and you placed some ssh variable which is working normally on newer SSH versions ssh OpenSSH_8.0p1 / or 7 but the options are applied on older SSH server and due to that restarting sshd via /etc/init.d/… or systemctl restart sshd cuts your access to remote server located in a DC and not attached to Admin LAN port, and does not have a working ILO or IDRAC configured and you have to wait for a couple of hours for some Support to go to the server Room / Rack / line location to have access to a Linux physical tty console and fix it by reverting the last changes you made to sshd and restarting.

Thus logical question comes what can you do to assure yourself you would not cut your network access to remote machine after modifying OpenSSHD and normal SSHD restart?

There is an old trick, I'm using for years now but perhaps if you're just starting with Linux as a novice system administrator or a server support guy you would not know it, it is as simple as setting a cron job for some minutes to periodically overwrite the sshd configuration with a copy of the old working version of sshd before modification.

Here is this nice nify trick which saved me headache of call on technical support line to ValueWeb when I was administering some old Linux servers back in the 2000s

root@server:~# crontab -u root -e

# create /etc/ssh/sshd_config backup file
cp -rpf /etc/ssh/sshd_config /etc/ssh/sshd_config_$(date +%d-%m-%y)
# add to cronjob to execute every 15 minutes and ovewrite sshd with the working version just in case
*/15 * * * * /bin/cp -rpf /etc/ssh/sshd_config_$(date +%d-%m-%y) /etc/ssh/sshd_config && /bin/systemctl restart sshd
# restart sshd 
cp -rpf /etc/ssh/sshd_config_$(date +%d-%m-%y) /etc/ssh/sshd_config && /bin/systemctl restart sshd

Copy paste above cron definitions and leave them on for some time. Do the /etc/ssh/sshd_config modifications and once you're done restart sshd by lets say

root@server:~#  killall -HUP sshd 

If the ssh connectivity continues to work edit the cron job again and delete all lines and save again.
If you're not feeling confortable with vim as a text editor (in case you're a complete newbie and you don't know) how to get out of vim. Before doing all little steps you can do on the shell with  export EDITOR=nano or export EDITOR=mcedit cmds,this will change the default text editor on the shell. 

Hope this helps someone… Enjoy 🙂

How to make sure your Linux system users won’t hide or delete their .bash_history / Securing .bash_history file – Protect Linux system users shell history

Monday, July 19th, 2010

Reading Time: 4minutes

If you're running multi user login Linux system, you have probably realized that there are some clever users that prefer to prevent their command line executed commands to be logged in .bash_history.
To achieve that they use a number of generally known methodologist to prevent the Linux system from logging into their $HOME/.bash_history file (of course if running bash as a default user shell).
This though nice for the user is a real nightmare for the sysadmin, since he couldn't keep track of all system command events executed by users. For instance sometimes an unprivilegd user might be responsible for executing a malicious code which crashes or breaks your server.
This is especially unpleasent, because you will find your system crashed and if it's not some of the system services that causes the issue you won’t even be able to identify which of all the users is the malicious user account and respectively the code excecuted which fail the system to the ground.
In this post I will try to tell you a basic ways that some malevolent users might use to hide their bash history from the system administrator.
I will also discuss a few possible ways to assure your users .bash_history keeps intact and possibly the commands executed by your users gets logged in in their.
The most basic way that even an unexperienced shell user will apply if he wants to prevent his .bash_history from sys admins review would be of directly wiping out the .bash_history file from his login account or alternatively emptying it with commands like:

malicious-user@server:~$ rm -f. bash_history
ormalicious-user@server:~# cat /dev/null > ~/.bash_history

In order to prevent this type of attack against cleaning the .bash_history you can use the chattr command.
To counter attack this type of history tossing method you can set your malicious-user .bash_history’s file the (append only flag) with chattr like so:

root@server:~# cd /home/malicious-user/
root@server:~# chattr +a .bash_history

It’s also recommended that the immunable flag is placed to the file ~/.profile in user home

root@server:~# chattr +i ~/.profile

It would be probably also nice to take a look at all chattr command attributes since the command is like swiss army knife for the Linux admin:
Here is all available flags that can be passed to chattr
append only (a)
compressed (c)
don~@~Yt update atime (A)
synchronous directory updates (D)
synchronous updates (S)
data journalling (j)
no dump (d)
top of directory hierarchy (T)
no tail-merging (t)
secure deletion (s)
undeletable (u)
immutable (i)

It’s also nice that setting the “append only” flag in to the user .bash_history file prevents the user to link the .bash_history file to /dev/null like so:

malicious-user@server:~$ ln -sf /dev/null ~/.bash_history
ln: cannot remove `.bash_history': Operation not permitted

malicious-user@server:~$ echo > .bash_history
bash: .bash_history: Operation not permitted

However this will just make your .bash_history append only, so the user trying to execute cat /dev/null > .bash_history won’t be able to truncate the content of .bash_history.

Unfortunately he will yet be able to delete the file with rm so this type of securing your .bash_history file from being overwritten is does not completely guarantee you that user commands will get logged.
Also in order to prevent user to play tricks and escape the .bash_history logging by changing the default bash shell variables for HISTFILE an d HISTFILESIZE, exporting them either to a different file location or a null file size.
You have to put the following bash variables to be loaded in /etc/bash.bashrc or in /etc/profile
# #Prevent unset of histfile, /etc/profile
# Don't let the users enter commands that are ignored# in the history file
readonly HISTFILE
readonly HISTSIZE

everytime a user logs in to your Linux system the bash commands above will be set.
The above tip is directly taken from Securing debian howto which by the way is quite an interesting and nice reading for system administrators 🙂

If you want to apply an append only attribute to all user .bash_history to all your existing Linux server system users assuming the default users directory is /home in bash you can execute the following 1 liner shell code:

#Set .bash_history as attr +a
2. find /home/ -maxdepth 3|grep -i bash_history|while read line; do chattr +a "$line"; done

Though the above steps will stop some of the users to voluntary clean their .bash_history history files it won’t a 100% guaranttee that a good cracker won’t be able to come up with a way to get around the imposed .bash_history security measures.

One possible way to get around the user command history prevention restrictions for a user is to simply using another shell from the ones available on the system:
Here is an example:

malicious-user:~$ /bin/csh

csh shell logs by default to the file .history

Also as far as I know it should be possible for a user to simply delete the .bash_history file overwritting all the .bash_history keep up attempts up-shown.
If you need a complete statistics about accounting you’d better take a look at The GNU Accounting Utilities

In Debian the GNU Accounting Utilities are available as a package called acct, so installation of acct on Debian is as simple as:

debian:~# apt-get install acct

I won’t get into much details about acct and would probably take a look at it in my future posts.
For complete .bash_history delete prevention maybe the best practice is to useg grsecurity (grsec)

Hopefully this article is gonna be a step further in tightening up your Server or Desktop Linux based system security and will also give you some insight on .bash_history files 🙂 .

Check linux install date / How do I find out how long a Linux server OS was installed?

Wednesday, March 30th, 2016

Reading Time: 2minutes


To find out the Linux install date, there is no one single solution according to the Linux distribution type and version, there are some common ways to get the Linux OS install age.
Perhaps the most popular way to get the OS installation date and time is to check out when the root filesystem ( / ) was created, this can be done with tune2fs command


server:~# tune2fs -l /dev/sda1 | grep 'Filesystem created:'
Filesystem created:       Thu Sep  6 21:44:22 2012


server:~# ls -alct /|tail -1|awk '{print $6, $7, $8}'
sep 6 2012


root home directory is created at install time


server:~# ls -alct /root


root@server:~# ls -lAhF /etc/hostname
-rw-r–r– 1 root root 8 sep  6  2012 /etc/hostname


For Debian / Ubuntu and other deb based distributions the /var/log/installer directory is being created during OS install, so on Debian the best way to check the Linux OS creation date is with:

root@server:~# ls -ld /var/log/installer
drwxr-xr-x 3 root root 4096 sep  6  2012 /var/log/installer/
root@server:~# ls -ld /lost+found
drwx—— 2 root root 16384 sep  6  2012 /lost+found/


On Red Hat / Fedora / CentOS, redhat based Linuces , you can use:


rpm -qi basesystem | grep "Install Date"


basesystem is the package containing basic Linux binaries many of which should not change, however in some cases if there are some security updates package might change so it is also good to check the root filesystem creation time and compare whether these two match.

Howto Fix “sysstat Cannot open /var/log/sysstat/sa no such file or directory” on Debian / Ubuntu Linux

Monday, February 15th, 2016

Reading Time: 2minutes

I really love sysstat and as a console maniac I tend to install it on every server however by default there is some <b>sysstat</b> tuning once installed to make it work, for those unfamiliar with <i>sysstat</i> I warmly recommend to check, it here is in short the package description:<br /><br />

server:~# apt-cache show sysstat|grep -i desc -A 15
Description: system performance tools for Linux
 The sysstat package contains the following system performance tools:
  – sar: collects and reports system activity information;
  – iostat: reports CPU utilization and disk I/O statistics;
  – mpstat: reports global and per-processor statistics;
  – pidstat: reports statistics for Linux tasks (processes);
  – sadf: displays data collected by sar in various formats;
  – nfsiostat: reports I/O statistics for network filesystems;
  – cifsiostat: reports I/O statistics for CIFS filesystems.
 The statistics reported by sar deal with I/O transfer rates,
 paging activity, process-related activities, interrupts,
 network activity, memory and swap space utilization, CPU
 utilization, kernel activities and TTY statistics, among
 others. Both UP and SMP machines are fully supported.


If you happen to install sysstat on a Debian / Ubuntu server with:

server:~# apt-get install –yes sysstat

, and you try to get some statistics with sar command but you get some ugly error output from:


server:~# sar Cannot open /var/log/sysstat/sa20: No such file or directory

And you wonder how to resolve it and to be able to have the server log in text databases periodically the nice sar stats load avarages – %idle, %iowait, %system, %nice, %user, then to FIX that Cannot open /var/log/sysstat/sa20: No such file or directory

You need to:

server:~# vim /etc/default/sysstat

By Default value you will find out sysstat stats it is disabled, e.g.:


Switch the value to "true"


Then restart sysstat init script with:

server:~# /etc/init.d/sysstat restart

However for those who prefer to do things from menu Ncurses interfaces and are not familiar with Vi Improved, the easiest way is to run dpkg reconfigure of the sysstat:

server:~# dpkg –reconfigure



root@server:/# sar
Linux 2.6.32-5-amd64 (pcfreak) 15.02.2016 _x86_64_ (2 CPU)

0,00,01 CPU %user %nice %system %iowait %steal %idle
0,15,01 all 24,32 0,54 3,10 0,62 0,00 71,42
1,15,01 all 18,69 0,53 2,10 0,48 0,00 78,20
10,05,01 all 22,13 0,54 2,81 0,51 0,00 74,01
10,15,01 all 17,14 0,53 2,44 0,40 0,00 79,49
10,25,01 all 24,03 0,63 2,93 0,45 0,00 71,97
10,35,01 all 18,88 0,54 2,44 1,08 0,00 77,07
10,45,01 all 25,60 0,54 3,33 0,74 0,00 69,79
10,55,01 all 36,78 0,78 4,44 0,89 0,00 57,10
16,05,01 all 27,10 0,54 3,43 1,14 0,00 67,79

Well that's it now sysstat error resolved, text reporting stats data works again, Hooray! 🙂

How to check if newly installed SSL certificate for IMAP and IMAPS is properly installed

Tuesday, June 28th, 2011

Reading Time: < 1minute
Did you have to regenerate your SSL certificate for your mail server’s IMAP and IMAP SSL service?
Did you have to find out if the newly installed certificates are fine after install?

Here is how:

           root@server-hosting [/usr/local ]# openssl s_client -connect
root@server-hosting [/usr/local ]# openssl s_client -connect -starttls imap

The output returned by this two commands will be the imap and imaps configured certificates as well as extensive info concerning the installed SSL, the last chunk of info to be spit is most crucial to know if certificate is fine.
It should be something like:

New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: 0B69E91022CB56D64F56CFA08405944D9C4C0069EE4097890B98F1406CF084D5
Master-Key: 13745B94E0C5A0604EB7529E7409251961DFD5F4134F3A8F
Key-Arg : None
Start Time: 1309265383
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)
. OK CAPABILITY completed