Posts Tagged ‘tiny shell’

Create simple proxy http server with netcat ( nc ) based tiny shell script

Tuesday, January 26th, 2021

Reading Time: 4minutes

use-Netcat_proxy-picture

The need of proxy server is inevitable nowadays especially if you have servers located in a paranoid security environments. Where virtually all is being passed through some kind of a proxy server. In my work we have recently started a  CentOS Linux release 7.9.2009 on HP Proliant DL360e Gen8 (host named rhel-testing).

HP DL360e are quite old nowadays but since we have spare servers and we can refurnish them to use as a local testing internal server Hypervisor it is okay for us. The machine is attached to a Rack that is connected to a Secured Deimilitarized Zone LAN (DMZ Network) which is so much filtered that even simple access to the local company homebrew RPM repository is not accessible from the machine.
Thus to set and remove software from the machine we needed a way to make yum repositories be available, and it seems the only way was to use a proxy server (situated on another accessible server which we use as a jump host to access the testing machine).

Since opening additional firewall request was a time consuming non-sense and the machine is just for testing purposes, we had to come with a solution where we can somehow access a Local repository RPM storage server http://rpm-package-server-repo.com/ for which we have a separate /etc/yum.repos.d/custom-rpms.repo definition file created.

This is why we needed a simplistic way to run a proxy but as we did not have the easy way to install privoxy / squid / haproxy or apache webserver configured as a proxy (to install one of those of relatively giant piece of software need to copy many rpm packages and manually satisfy dependencies), we looked for a simplistic way to run a proxy server on jump-host machine host A.

A note to make here is jump-host that was about to serve as a proxy  had already HTTP access towards the RPM repositories http://rpm-package-server-repo.com and could normally fetch packages with curl or wget via it …

For to create a simple proxy server out of nothing, I've googled a bit thinking that it should be possible either with BASH's TCP/IP capabilities or some other small C written tool compiled as a static binary, just to find out that netcat swiss army knife as a proxy server bash script is capable of doing the trick.

Jump host machine which was about to be used as a proxy server for http traffic did not have enabled access to tcp/port 8888 (port's firewall policies were prohibiting access to it).Since 8888 was the port targetted to run the proxy to allow TCP/IP port 8888 accessibility from the testing RHEL machine towards jump host, we had to issue first on jump host:

[root@jump-host: ~ ]# firewall-cmd –permanent –zone=public –add-port=8888/tcp

To run the script once placed under /root/tcp-proxy.sh on jump-host we had to run a never ending loop in a GNU screen session to make sure it runs forever:

Original tcp-proxy.sh script used taken from above article is:
 

#!/bin/sh -e

 

if [ $# != 3 ]
then
    echo "usage: $0 <src-port> <dst-host> <dst-port>"
    exit 0
fi

TMP=`mktemp -d`
BACK=$TMP/pipe.back
SENT=$TMP/pipe.sent
RCVD=$TMP/pipe.rcvd
trap 'rm -rf "$TMP"' EXIT
mkfifo -m 0600 "$BACK" "$SENT" "$RCVD"
sed 's/^/ => /' <"$SENT" &
sed 's/^/<=  /' <"$RCVD" &
nc -l -p "$1" <"$BACK" | tee "$SENT" | nc "$2" "$3" | tee "$RCVD" >"$BACK"

 

Above tcp-proxy.sh script you can download here.

I've tested the script one time and it worked, the script syntax is:

 [root@jump-host: ~ ]#  sh tcp-proxy.sh
usage: tcp-proxy.sh <src-port> <dst-host> <dst-port>


To make it work for one time connection I've run it as so:

 

 [root@jump-host: ~ ]# sh tcp-proxy.sh 8888 rpm-package-server-repo.com 80

 

 

To make the script work all the time I had to use one small one liner infinite bash loop which goes like this:

[root@jump-host: ~ ]#  while [ 1 ]; do sh tcp-proxy.sh 8888 rpm-package-server-repo.com 80; done​

On rhel-testing we had to configure for yum and all applications to use a proxy temporary via
 

[root@rhel-tresting: ~ ]# export http_proxy=jump-host_machine_accessibleIP:8888


And then use the normal yum check-update && yum update to apply to rhel-testing machine latest RPM package security updates.

The nice stuff about the tcp-proxy.sh with netcat in a inifite loop is you will see the binary copy of traffic flowing on the script which will make you feel like in those notorious Hackers movies ! 🙂

The stupid stuff is that sometimes some connections and RPM database updates or RPMs could be cancelled due to some kind of network issues.

To make the connection issues that are occuring to the improvised proxy server go away we finally used a slightly modified version from the original netcat script, which read like this.
 

#!/bin/sh -e

 

if [ $# != 3 ]
then
    echo "usage: $0 <src-port> <dst-host> <dst-port>"
        exit 0
        fi

        TMP=`mktemp -d`
        BACK=$TMP/pipe.back
        SENT=$TMP/pipe.sent
        RCVD=$TMP/pipe.rcvd
        trap 'rm -rf "$TMP"' EXIT
        mkfifo -m 0600 "$BACK" "$SENT" "$RCVD"
        sed 's/^/ => /' <"$SENT" &
        sed 's/^/<=  /' <"$RCVD" &
        nc –proxy-type http -l -p "$1" <"$BACK" | tee "$SENT" | nc "$2" "$3" | tee "$RCVD" >"$BACK"


Modified version tcp-proxy1.sh with –proxy-type http argument passed to netcat script u can download here.

With –proxy-type http yum check-update works normal just like with any normal fully functional http_proxy configured.

Next step wasto make the configuration permanent you can either add to /root/.bashrc or /etc/bashrc (if you need the setting to be system wide for every user that logged in to Linux system).

[root@rhel-tresting: ~ ]#  echo "http_proxy=http://jump-host_machine_accessibleIP:8888/" > /etc/environment


If you need to set the new built netcat TCP proxy only for yum package update tool include proxy only in /etc/yum.conf:

[root@rhel-tresting: ~ ]# vi /etc/yum.conf
proxy=http_proxy=http://jump-host_machine_accessibleIP:8888/


That's all now you have a proxy out of nothing with just a simple netcat enjoy.

Linux: How to change recursively directory permissions to executable (+x) flag

Monday, September 2nd, 2013

Reading Time: < 1minute

change recursively permissions of directories and subdirectories Linux and Unix with find command
I had to copy large directory from one Linux server to windows host via SFTP proto (with WinSCP). However some of directories to be copied lacked executable flag, thus WinSCP failed to list and copy them.

Therefore I needed way to set recursively, all sub-directories under directory /mirror (located on Linux server) to +x executable flag.

There are two ways to do that one is directly through find cmd, second by using find with xargs
Here is how to do it with find:

# find /mirror -type d -exec chmod 755 {} + Same done with find + xargs:

# find /path/to/base/dir -type d -print0 | xargs -0 chmod 755
To change permissions only to all files under /mirror server directory with find

# find /path/to/base/dir -type f -exec chmod 644 {} +

Same done with find + xargs:
# find /path/to/base/dir -type f -print0 | xargs -0 chmod 644

Also, tiny shell script that recursively changes directories permissions (autochmod_directories.sh) is here

Make daily Linux MySQL database backups with shell script

Thursday, May 23rd, 2013

Reading Time: 2minutes

Creating database backup with MySQL with mysqlbackupper and mysqlback shell scripts easy create mysql backups

Some time ago, I've written a tiny shell script which does dumps of Complete (SQL Script) MySQL databases. There are plenty of ways to backup MySQL database and plenty of scripts on the net but I like doing it my own way. I have few backup scripts. I prefer script database over keeping binary logs, or using some un-traditional backup methods like backing all binary data in /var/lib/mysql.

One was intended to backup with mysqldump whole database and later upload to a central server running tsh (shell). Using tsh maybe not the best method to upload, but the script can easily be modified to use ssh passwordless authentication as a method to upload.

I'm not a pro shell scripter, but MySQLBackupper script can be used as useful for learning some simple bash  shell scripting.

To use the script as intended you will have to build tsh from source. Tsh is in very early development stage (ver 0.2) but as far as I tested it before some years it does great what it is intended for. You can  MySQLBackupper.sh script from here.
Earlier, I used MysqlBackupper.sh to upload all SQL dumps to /backups directory on central backup storage server, thus I had written secondary script to classify uploaded backups based on backup archive name. Script used is mysqldumps-classify.sh and can be viewed here. Though this way of making backups, needs a bit of custom work for managing backups up to 10 / 20 servers it worked well.

I have written also another mysqlbackup script which is much more simplistic and only dumps with mysqldump and stores copies on hard disk in tar.gz archive. You can download my other simple mysqkbackup.sh here.

Only inconvenient thing about above scripts is they dump all SQL databases. Hence whether necessary to get content for single database from (complete) All database SQL (script backup), I use SED (stream editor) one liner script.

It is interesting to hear how others prepare their MySQL db backups.

screen -d Fix “Must run suid root for multiuser support.” su user detach error

Thursday, March 28th, 2013

Reading Time: < 1minute

I had to run a shell script to run automatically in detached screen during Linux system boot up via /etc/rc.local. This is needed because the server uses the tiny shell script to fetch data from remote host database and fill information into local MySQL server.

My idea was to su from root to www-data (Apache) user – the script has requirements to run with Apache user, then it has to run detached using GNU screen (multi terminal emulator. The tiny one line script I imagined would do the trick is like so:

# tty=$(tty); su www-data -c 'cd /home/user/www/enetpulse; screen -d /home/user/www/enetpulse/while_true.sh'; chmod 0720 $tty

I run this as root user to test whether it will work or not before I put it in /etc/rc.local but for my surprise got an error:
 

Must run suid root for multiuser support.

After a quick investigation on what is causing it I came across the solution which is to include screen arguments (-m -S shared). The working variant that gets around the error – i.e. successfully changes user privileges to Debian Apache user (www-data) and then detach with screen is:

# tty=$(tty); chmod a+rw $tty; su www-data -c 'cd /home/user/www/enetpulse; screen -d -m -S shared /home/user/www/enetpulse/while_true.sh'; chmod 0720 $tty;

That's all now script works out as planned on next server reboot

Script to Automatically change current MySQL server in wp-config.php to another MySQL host to minimize WordPress and Joomla downtimes

Friday, July 20th, 2012

Reading Time: 2minutes

I'm running a two servers for a couple of home hosted websites. One of the servers is serving as Apache host1 and has configured MySQL running on it and the second is used just for database host2 – (has another MySQL configured on it).
The MySQL servers are not configured to run as a MySQL MASTER and MySQL SLAVE (no mysql replication), however periodically (daily), I have a tiny shell script that is actualizing the data from the active SQL host2 server to host1.

Sometimes due to electricity problems or CPU overheats the active MySQL host at host2 gets stoned and stops working causing the 2 WordPress based websites and One joomla site inaccessible.
Until I manually get to the machine and restart host2 the 3 sites are down from the net and as you can imagine this has a very negative impact on the existing website indexing (PageRank) in Google.

When I'm at home, this is not a problem as I have physical access to the servers and if somethings gets messy I fix it quickly. The problem comes, whether I'm travelling or in another city far from home and there is no-one at home to give the hanged host hard reboot ….

Lately the problems with hang-ups of host2 happaned 3 times or so for 2 weeks, as a result the websites were inaccessible for hours and since there is nobody to reboot the server for hours; the websites keep hanging until the DB host is restarted ;;;;

To work-around this I came with the idea to write a tiny shell script to check if host2 is ping-able in order to assure the Database host is not down and then if script determines host2 (mysql) host is down it changes wp-config.php (set to use host2) to a wp-config.php (which I have beforehand configured to use) host1.

Using the script is a temporary solution, since I have to actually find the real hang-up causing troubles, but at least it saves me long downtimes. Here is a download link to the script I called change_blog_db.sh .
I've configured the script to be run on the Apache node (host1) via a crontab calling the script every 10 minutes, here is the crontab:
 

*/10 * * * * /usr/sbin/change_blog_db.sh > /dev/null 2>&1

The script is written in a way so if it determins host2 is reachable a copy of wp-config.php and Joomla's configuration.php tuned to use host2 is copied over the file config originals. In order to use the script one has to configured the head variables script section, e.g.:

host_to_ping='192.168.0.2';
blog_dir='/var/www/blog';
blog_dir2='/var/www/blog1';blog_dir3='/var/www/joomla';
notify_mail='hipo@pc-freak.net';
wp_config_orig='wp-config.php';
wp_config_localhost='wp-config-localhost.php';
wp_config_other_host='wp-config-192.168.0.2.php';
joomla_config_orig='configuration.php';
joomla_config_other_host='configuration-192.168.0.2.php';

You will have to manually prepare;;;

wp-config-localhost.php, wp-config-192.168.0.2.php ,configuration-192.168.0.2.php, wp-config-localhost.php to be existing files configured to with proper host1 and host2 IP addresses.
Hope the script will be useful to others, experiencing database downtimes with WordPress or Joomla installs.
 

How to add a range of virtual IPs to a CentOS and Fedora Linux server

Monday, July 18th, 2011

Reading Time: < 1minute
Recently I had the task to add a range of few IP addresses to as a virtual interface IPs.

The normal way to do that is of course using the all well known ifconfig eth0:0, ifconfig eth0:1 or using a tiny shell script which does it and set it up to run through /etc/rc.local .

However the Redhat guys could omit all this mambo jambo and do it The Redhat way TM 😉 by using a standard method documented in CentOS and RHEL documentation.
Here is how:

# go to network-script directory[root@centos ~]# cd /etc/sysconfig/network-scripts
# create ifcfg-eth0-range (if virtual ips are to be assigned on eth0 lan interface[root@centos network-scripts]# touch ifcfg-eth0-range

Now inside ifcfg-eth0-range, open up with a text editor or use the echo command to put inside:

IPADDR_START=192.168.1.120
IPADDR_END=192.168.1.250
NETMASK=255.255.255.25
CLONENUM_START=0

Now save the /etc/sysconfig/network-scripts/ifcfg-eth0-range file and finally restart centos networking via the network script:

[root@centos network-scripts]# service network restart

That’s all now after the network gets reinitialized all the IPs starting with 192.168.1.120 and ending in 192.168.1.250< will get assigned as virtual IPs for eth0 interface
Cheers 😉