Posts Tagged ‘good’

Howto Pass SSH traffic through a Secured Corporate Proxy server with corkscrew, using sshd as a standalone proxy service with no proxy installed on remote Linux server or VPS

Tuesday, November 19th, 2019

Reading Time: 11 minutes

howto pass ssh traffic through proxy to remote server use remote machine as a proxy for connecting to the Internet

Working in the big bad corporate world (being employed in  any of the Fortune 500) companies, especially in an IT delivery company is a nasty thing in terms of User Personal Data Privacy because usually when employeed in any of a corporation, the company ships you with a personal Computer with some kind of pre-installed OS (most often this is Windows) and the computer is not a standalone one but joined in Active Directory (AD) belonging to Windows Domain and centrally administered by whoever.

As part of the default deplyed configuration in this pre-installed OS and software is that part or all your network traffic and files is being monitored in some kind of manner as your pre-installed Windows or Linux notebook given by the Corporation is having a set of standard software running in the background, and even though you have Windows Administrator there are many things you have zero control or even if you have changed it once the Domain Policy is triggered your custom made changes / Installed Programs that happen to be against the company policy are being automatically deleted, any registry changes made are being rewinded etc. Sometimes even by trying to manually clean up your PC from the corporate crapware,  you might breaks access to the corporate DMZ firewalled network. A common way to secure their employee PC data large companies have a Network seperation, your PC when not connected to the Corporate VPN is having a certain IP configuration and once connected to the Demilitarized Zone VPN those configuration changes and the PC have access to internal company infrastructure servers / router / switches / firewalls / SANs etc. Access to corporate Infrastructure is handled via crypted VPN clinet such as Cisco AnyConnect Secure Mobility Client which is perhaps one of the most used ones out there.

Part of the common software installed to Monitor your PC for threats / viruses / trojans among which is MCafee / EMET (Enhandced Mitigation Experience Toolkit) the PC is often prebundled with some kind of anti-malware (crapware) :). But the tip of the iceberg on user surveillance where most of surveillance happens is the default installed proxy on the PC which usually does keep track of all your remote accessed HTTP Website URLs accessed in plain text – traffic flowing on Port 80 and crypted one on standard (SSL) Port 443. This Web Traffic is handled by the Central Corporate proxy that is being deployed via some kind of Domain policy, every time the Computer joins the Windows domain. 

This of course is a terrible thing for your Browsing security and together with the good security practice to run your browser in Incognito mode, which makes all your browsing activity such as access URLs History or Saved Cookies data to be cleared up on a Browser close it is important to make sure you run your own personal traffic via a separate browser which you will use only for your own concern browsing such as Accessing your Bank Money Accounts to check your Monthly Sallary / Purchase things online via /, whether all of the rest traffic company related is trafficed via the default set corporate central proxy.
This is relatively easy sometimes in companies, where security is not of a top concern but in corporations with tightened security accessing remote proxy, or accessing even common daily news and Public Email websites or social media sites / Twitter / Youtube will be filtered so the only way to reach them will be via some kind of Proxy and often this proxy is the only way out to the Free world from the corporate jail.

Here is where the good old SSH comes as a saving grace as it turns out SSH traffic could be trafficed over a proxy. In below article I will give you a short insight on how Proxy through SSH could be achieved to Secure your dailty web traffic and use SSH to reach your own server on the Internet as well as how you can copy securely data via SSH through corporate Proxy. 

1. How to view your corporate used (default) proxy / Check Proxy.pac file definitions


To get an idea what is the used proxy on your Corporate PC (as most corporate employee given notebooks are running some kind of M$ Windows)  you can go to:

Windows Control Panel -> Internet Options -> Connections -> Lan Settings


Under the field Proxy server (check out the Proxy configured Address and Port number )


Having that as browsers venerate the so-called Proxy.pac file, to be rawly aware on some general Company Proxy configured definitions you can access in a browser the proxy itself fething the proxy.pac file for example.




This is helpful as some companies Proxies have some proxy rules that reveal some things about its Internet architecture and even some have some badly configured proxy.pac files which could be used to fool the proxy under some circumstances 🙂

2. Few of the reasons corporations proxy all their employee's work PC web traffic


The corporate proxying of traffic has a number of goals, some of which are good hearted and others are for mostly spying on the users.


1. Protect Corporate Employees from malicious Viruses / Trojans Horses / Malware / Badware / Whatever ware – EXCELLENT
2. Prevent users from acessing a set of sources that due to the corporate policy are considered harmful (e.g. certain addresses 
of information or disinformation of competitors, any Internet source that might preach against the corporation, hacking ralated websites etc.) – NOT GOOD (for the employee / user) and GOOD for the company
3.Spy on the users activity and be able to have evidence against the employee in case he decided to do anything harmful to the company evidences from proxy could even later be used in court if some kind of corpoate infringment occurs due to misbehave of the employee. – PERFECT FOR COMPANY and Complete breach of User privacy and IMHO totally against European Union privacy legislation such as GDRP
4. In companies that are into the field of Aritificial Intelligence / Users behavior could even be used to advance Self-learning bots and mechanisms – NASTY ! YAECKES


3. Run SSH Socks proxy to remote SSHd server running on common SSL 443 port


Luckily sysadmins who were ordered the big bosses to sniff on your Web behaviour and preferences could be outsmarted with some hacks.

To protect your Browsing behaviours and Secure your privacy perhaps the best option is to use the Old but gold practice o Securing your Networkf traffic using SSH Over Proxy and SSH Dynamic tunnel as a Proxy as explained in my previous article here.


In short the quest way to have your free of charge SOCKS  Remote proxy to your Home based Linux installed OS server / VPN with a Public Internet address is to use ssh as so:


ssh -D 3128 UserName@IP-of-Remote-SSHD-Host -p 443


This will start the SOCKS Proxy tunnel from Corporate Work PC to your Own Home brew server.

For some convenience it is useful to set up an .alias (for cygwin) / linux users in .bashrc file:


alias proxy='ssh -D 3128 UserName@IP-of-Remote-SSHD-Host -p 443';


To start using the Proxy from browser, I use a plugin called FoxyProxy in Chrome and Firefox browsers
set-up to connect to localhost – for All Protocols as a SOCKs v5 Proxy.

The sshd Socks proxy can be used for multiple others for example, using it you can also pass on traffic from Mail client such as Thunderbird to your Email server if you're behind a firewall prohibiting access to the common POP3 port 110 or IMAP port TCP 143. 

4. How to access SSH through Proxy using jumphost SSH hop

If you're like me and you have on your Home Linux machine only one Internet address and you have already setupped an SSL enabled service (lets say Webmail) to listen to that Public Internet IP and you don't have the possibility to run another instance of /usr/bin/sshd on port 443 via configuration or manually one time by issuing:


/usr/sbin/sshd -p 443


Then you can use another ssh another Linux server as a jump host to your own home Linux sshd server. This can be done even by purchasing a cheap VPS server for lets say 3 dollars month etc. or even better if you have a friend with another Linux home server, you can ask him to run you sshd on TCP port 443 and add you an ssh account.
Once you have the second Linux machine as JumpHost to reach out to your own machine use:


ssh -J -v


To easify this a bit long line it is handy to use some kind of alias like:


alias sshhome='ssh -J -v'


The advantage here is just by issuing this sshd tunnel and keeping it open in a terminal or setting it up as Plink Putty tunnel you have all your Web Traffic Secured
between your Work Corporate PC and your Home Brew Server, keeping the curious eyes of your Company Security Officers from your own Web traffic, hence
separating the corporate privacy from your own personal privacy. Using the just established own SSH Proxy Tunnel to home for your non-work stuff browsing habits
from the corporate systems which are accessed by switching with a button click in FoxyProxy to default proxy settings.

5. How to get around paranoid corporate setup where only remote access to Corporate proxy on TCP Port 80 and TCP 443 is available in Browser only


Using straight ssh and to create Proxy will work in most of the cases but it requires SSH access to your remote SSH running server / VPS on TCP Port 22, however under some Fort-Nox like financial involved institutions and companies for the sake of tightened security, it is common that all Outbound TCP Ports are prohibited except TCP Port 80 and SSL 443 as prior said, so what can you do then to get around this badful firewall and access the Internet via your own server Proxy? 
The hack to run SSH server either on tcp port 80 or tcp port 443 on remote Host and use 443 / 80 to acess SSHD should work, but then even for the most paranoid corporations the ones who are PCI Compliant – PCI stands for (Payment Card Industry), e.g. works with Debit and Credit Card data etc, accessing even 80 or 443  ports with something like telnet client or netcat will be impossible. 
Once connected to the corporate VPN,  this 2 two ports firewall exceptions will be only accessible via the Corporate Proxy server defined in a Web Browser (Firefox / IE / Chrome etc.) as prior explained in article.

The remedy here is to use a 3rd party tools such as httptunnel or corkscrew that  are able to TUNNEL SSH TRAFFIC VIA CORPORATE PROXY SERVER and access your own resource out of the DMZ.

Both httptunnel and corkscrew are installable both on most Linux distros or for Windows users via CygWin for those who use MobaXterm.

Just to give you better idea on what corkscrew and (hts) httptunnel does, here is Debian packages descriptions.

# apt-cache show​ corkscrew
" corkscrew is a simple tool to tunnel TCP connections through an HTTP
 proxy supporting the CONNECT method. It reads stdin and writes to
 stdout during the connection, just like netcat.
 It can be used for instance to connect to an SSH server running on
 a remote 443 port through a strict HTTPS proxy.


# apt-cache show httptunnel|grep -i description -A 7
Description-en: Tunnels a data stream in HTTP requests
 Creates a bidirectional virtual data stream tunnelled in
 HTTP requests. The requests can be sent via a HTTP proxy
 if so desired.
 This can be useful for users behind restrictive firewalls. If WWW
 access is allowed through a HTTP proxy, it's possible to use
 httptunnel and, say, telnet or PPP to connect to a computer

Description-md5: ed96b7d53407ae311a6c5ef2eb229c3f
Tag: implemented-in::c, interface::commandline, interface::daemon,
 network::client, network::server, network::vpn, protocol::http,
 role::program, suite::gnu, use::routing
Section: net
Priority: optional
Filename: pool/main/h/httptunnel/httptunnel_3.3+dfsg-4_amd64.deb

Windows cygwin users can install the tools with:

apt-cyg install –yes corkscrew httptunnel

Linux users respectively with:

apt-get install –yes corkscrew httptunnel


yum install -y corkscrew httptunnel


You will then need to have the following configuration in your user home directory $HOME/.ssh/config file

ProxyCommand /usr/bin/corkscrew your-corporate-firewall-rpoxy-url 8080 %h %p



Picture Copyright by Daniel Haxx

The best picture on how ssh traffic is proxied is the one found on Daniel Haxx's website which is a great quick tutorial which originally helped to get the idea of how corkscrew works in proxying traffic I warmly recommend you take a quick look at his SSH Through or over Proxy article. could be also and IP if you don't have your own domain name in case if using via some cheap VPN Linux server with SSH, or alternatively
if you don't want to spend money on buying domain for SSH server (assuming you don't have such yet) you can use Dyn DNS or NoIP.

Another thing is to setup the proper http_proxy / https_proxy / ftp_proxy variable exports in $HOME/.bashrc in my setup I have the following:

export ftp_proxy="http://your-corporate-firewall-rpoxy-url:8080"
export https_proxy="https://your-corporate-firewall-rpoxy-url:8080"
export http_proxy="http://your-corporate-firewall-rpoxy-url:8080"
export HTTP_PROXY="http://your-corporate-firewall-rpoxy-url:8080"
export HTTPS_PROXY="http://your-corporate-firewall-rpoxy-url:8080"


6. How to Transfer Files / Data via SSH Protocol through  Proxy with SCP and SFTP

Next logical question is how to Transfer your own personal encrypted files (that contains no corporate sensitive information) between your Work laptop and home brew Linux ssh server or cheap VPN.

It took me quite a lot of try-outs until finally I got it how Secure Copy (scp) command can be used toto transfer files between my Work Computer and my Home brew server using JumpHost, here is how:

scp -o 'ProxyJump' ~/file-or-files-to-copy*

I love using sftp (Secure FTP) command Linux client to copy files and rarely use scp so I have a lot of try-outs to connect interacitvely via the Corporate Proxy server over a Jump-Host:443 to my Destination home machine, 


I've tried using netcat as it was pointed in many articles online, like so to traffic my sftp traffic via my localhost binded SSH Socks proxy on :3128 together with netcat as shown in article prior example, using following line:

sftp -oProxyCommand='/bin/nc -X connect -x %h %p' 22


Also tried proxy connect like this:


sftp -o ProxyCommand="proxy-connect -h localhost -p 3128 %h %p"


Moreover, tried to use the ssh  command (-s) argument capability to invoke SSH protocol subsystem feature which is used to facilitiate use of SSH secure transport for other application

ssh -v -J hipo@Jump-Host:443 -s sftp -v

open failed: administratively prohibited: open failed


Finally decided to give a try to the same options arguments as in scp and thanks God it worked and I can even access via the Corporate Proxy through the Jump Host SSH interactively via Secure FTP 🙂


sftp -o 'ProxyJump'

To save time from typing this long line every time, I've setup the following alias to ~/.bashrc

alias sftphome='sftp -o 'ProxyJump''



Of course using own Proxy via your Home brew SSH Machine as well as transferring your data securely from your Work PC (notebook) to Home does not completely make you Surveillance free, as the Corporate Windows installed OS image is perhaps prebundled with its own integrated Keylogger as well as the Windows Domain administrators have certainly access to connect to your PC and run various commands, so this kind of Security is just an attempt to make company has less control and know less on your browsing habits and the best solution where possible to secure your privacy and separate your Personal Space form Work space by using a second computer (if having the ability to work from home) with a KVM Switch device and switch over your Work PC and Home PC via it or in some cases (where companies) allows it, setup something like VNC server (TightVNC / RealVNC) on work PC and leave it all time running in office and connect remotely with vncviewer from your own controlled secured computer.

In article I've explained shortly common scenario found in corporate Work computers proxy setup, designed to Surveil all your move, mentioned few common softwares running by default to protect from Viruses and aimed to Protect user from malicious hacking tools, explained how to view your work notebook configured Proxy, shortly mentioned on Proxy.pac and hinted how to view proxy.pac config as well as gave few of the reasons why all web traffic is being routed over central proxy.

That's all folks, Enjoy the Freedom to be less surveilled !

How to Set MySQL MariaDB server root user to be able to connect from any host on the Internet / Solution to ‘ ERROR 1045 (28000): Access denied for user ‘root’@’localhost’ (using password: YES) ‘

Tuesday, September 3rd, 2019

Reading Time: 5 minutes


In this small article, I'll shortly explain on how I setup a Standard default package MariaDB Database server on Debian 10 Buster Linux and how I configured it to be accessible from any hostname on the Internet in order to make connection from remote Developer PC with MySQL GUI SQL administration tools such as MySQL WorkBench / HeidiSQL / Navicat / dbForge   as well as the few set-backs experienced in the process (e.g. what was the reason for ' ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) '  error and its solution.

Setting new or changing old MariaDB (MySQL) root server password


I've setup a brand new MariaDB database (The new free OpenSource software fork of MySQL) mariadb-server-10.3 on a Debian 10, right after the OS was installed with the usual apt command:

# apt install mariadb-server

Next tep was to change the root access password which was set to empty pass by default, e.g. connected with mysql CLI locally while logged via SSH on server and run:

MariaDB [(none)]> mysql -u root -p

use mysql;
update user set authentication_string=PASSWORD("MyChosenNewPassword") where User='root';

There was requirement by the customer, that MySQL server is not only accessed locally but be accessed from any IP address from anywhere on the Internet, so next step was to do so.

Allowing access to MySQL server from Anywhere

Allowing access from any host to MariaDB SQL server  is a bad security practice but as the customer is the King I've fulfilled this weird wish too, by changing the listener for MariaDB (MySQL) on Debian 10 codenamed Buster
changing the default listener
to be not the default (localhost) but any listener is done by modifying the bind-address directive in conf /etc/mysql/mariadb.conf.d/50-server.cnf:

root@linux:~# vim /etc/mysql/mariadb.conf.d/50-server.cnf

Then comment out

bind-address  =

and  add instead (any listener)


bind-address  =
root@linux:/etc/mysql/mariadb.conf.d# grep -i bind-address 50-server.cnf
##bind-address            =
bind-address    =

Then to make the new change effective restart MariaDB (luckily still using the old systemV init script even though systemd is working.

root@linux:~# /etc/init.d/mysql restart
[ ok ] Restarting mysql (via systemctl): mysql.service.

To make sure it is properly listening on MySQL defaults TCP port 3306, then as usual used netcat.

root@pritchi:~# netstat -etna |grep -i 3306
tcp        0      0  *               LISTEN      109        1479917  


By the way the exact mariadb.cnf used on this middle-sized front-backend server is here – the serveris planned to be a Apache Web server + Database host with MySQL DB of a middle range to be able to serve few thousand of simultaneous unique customers.

To make sure no firewall is preventing MariaDB to be accessed, I've checked for any reject rules iptables and ipset definitions, e.g.:

root@linux:~# iptables -L |gre -i rej

root@linux:~# ipset list


Then to double make sure the MySQL is allowed to access from anywhere, used simple telnet from my Desktop Laptop PC (that also runs Debian Linux) towards the server .

hipo@jeremiah:~$ telnet 3306
Connected to
Escape character is '^]'.
Connection closed by foreign host.


As telnet is not supporting the data encryption after TCP proto connect, in a few seconds time, remote server connection is terminated.


Setting MySQL user to be able to connect to local server MySQL from any remote hostname

I've connected locally to MariaDB server with mysql -u root -p and issued following set of SQL commands to make MySQL root user be able to connect from anywhere:


CREATE USER 'root'@'%' IDENTIFIED BY 'my-secret-pass';
GRANT ALL ON *.* TO 'root'@'localhost';
GRANT ALL ON *.* TO 'root'@'%';


Next step, I've took was to try logging in with root (admin) MariaDB superuser from MySQL CLI (Command Line Interface) on my desktop just to find out, I'm facing a nasty error.

hipo@jeremiah:~$ mysql -u root -H -p
Enter password:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

My first guess was something is wrong with my root user created in MySQL's mysql.user table (In MySQL this is the privileges table that stores, how MySQL user credentials are handled by mysqld local OS running process.


Changing the MySQL root (admin) password no longer possible on Debian 10 Buster?


The standard way ot change the MySQL root password well known via a simple dpkg-reconfigure (provided by Debian's debconf is no longer working so below command produces empty output instead of triggering the good old Ncurses text based interface well-known over the years …


root@linux:~# /usr/sbin/dpkg-reconfigure mariadb-server-10.3



Viewing MariaDB (MySQL) username / password set-up from the CLI


To list how this set-privileges looked like I've used following command:


MariaDB [mysql]> select * from mysql.user where User = 'root';
| Host      | User | Password                                  | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Reload_priv | Shutdown_priv | Process_priv | File_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Repl_slave_priv | Repl_client_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Create_user_priv | Event_priv | Trigger_priv | Create_tablespace_priv | Delete_history_priv | ssl_type | ssl_cipher | x509_issuer | x509_subject | max_questions | max_updates | max_connections | max_user_connections | plugin                | authentication_string | password_expired | is_role | default_role | max_statement_time |
| localhost | root | *E6D338325F50177F2F6A15EDZE932D68C88B8C4F | Y           | Y           | Y           | Y           | Y           | Y         | Y           | Y             | Y            | Y         | Y          | Y               | Y          | Y          | Y            | Y          | Y                     | Y                | Y            | Y               | Y                | Y                | Y              | Y                   | Y                  | Y                | Y          | Y            | Y                      | Y                   |          |            |             |              |             0 |           0 |               0 |                    0 | mysql_native_password |                       | N                | N       |              |           0.000000 |
| %         | root | *E6D338325F50177F2F6A15EDZE932D68C88B8C4F | Y           | Y           | Y           | Y           | Y           | Y         | Y           | Y             | Y            | Y         | N          | Y               | Y          | Y          | Y            | Y          | Y                     | Y                | Y            | Y               | Y                | Y                | Y              | Y                   | Y                  | Y                | Y          | Y            | Y                      | Y                   |          |            |             |              |             0 |           0 |               0 |                    0 |                       |                       | N                | N       |              |           0.000000 |


The hashed (encrypted) password string is being changed from the one on the server, so please don't try to hack me (decrypt it) 🙂
As it is visible from below output the Host field for root has the '%' string which means, any hostname is authorized to be able to connect and login to the MySQL server, so this was not the problem.

After quite some time on reading on what causes
' ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
I've spend some time reading various forum discussions online on the err such as the one on StackOverflow here's  how to fix access denied for user 'root'@'localhost' and one on's – ERROR 1045(28000) : Access denied for user 'root@localhost' (using password: no ) and after a while finally got it, thanks to a cool IRC.FREENODE.NET guy nicknamed, hedenface who pointed me I'm that, I'm trying to use the -H flag (Prodice HTML) instead of -h (host_name), it seems somehow I ended up with the wrong memory that the -H stands for hostname, by simply using -h I could again login Hooray!!!


root@linux:~$ mysql -u root -h -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 14
Server version: 10.3.15-MariaDB-1 Debian 10


Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

I've further asked the customer to confirm, he can connect also from his Microsoft Windows 10 PC situated on a different LAN network and got his confirmation. Few notes to make here is I've also installed phpmyadmin on the server using phpmyadmin php source code latest version, as in Debian 10 it seems the good old PHP is no longer available (as this crazy developers again made a mess and there is no phpmyadmin .deb package in Debian Buster – but that's a different story I'll perhaps try to document in some small article in future.

Howto debug and remount NFS hangled filesystem on Linux

Monday, August 12th, 2019

Reading Time: 6 minutes


If you're using actively NFS remote storage attached to your Linux server it is very useful to get the number of dropped NFS connections and in that way to assure you don't have a remote NFS server issues or Network connectivity drops out due to broken network switch a Cisco hub or other network hop device that is routing the traffic from Source Host (SRC) to Destination Host (DST) thus, at perfect case if NFS storage and mounted Linux Network filesystem should be at (0) zero dropped connectios or their number should be low. Firewall connectivity between Source NFS client host and Destination NFS Server and mount should be there (set up fine) as well as proper permissions assigned on the server, as well as the DST NFS should be not experiencing I/O overheads as well as no DNS issues should be present (if NFS is not accessed directly via IP address).
In below article which is mostly for NFS novice admins is described shortly few of the nuances of working with NFS.

1. Check nfsstat and portmap for issues

One indicator that everything is fine with a configured NFS mount is the number of dropped NFS connections
or with a very low count of dropped connections, to check them if you happen to administer NFS



linux:~# nfsstat -o net
Server packet stats:
packets    udp        tcp        tcpconn
0          0          0          0  

nfsstat is useful if you have to debug why occasionally NFS mounts are getting unresponsive.

As NFS is so dependent upon portmap service for mapping the ports, one other point to check in case of Hanged NFSes is the portmap service whether it did not crashed due to some reason.


linux:~# service portmap status
portmap (pid 7428) is running…   [portmap service is started.]


linux:~# ps axu|grep -i rpcbind
_rpc       421  0.0  0.0   6824  3568 ?        Ss   10:30   0:00 /sbin/rpcbind -f -w

A useful commands to debug further rcp caused issues are:

On client side:


rpcdebug -m nfs -c


On server side:


rpcdebug -m nfsd -c


It might be also useful to check whether remote NFS permissions did not changed with the good old showmount cmd

linux:~# showmount -e rem_nfs_server_host

Also it is useful to check whether /etc/exports file was not modified somehow and whether the NFS did not hanged due to attempt of NFS daemon to reload the new configuration from there, another file to check while debugging is /etc/nfs.conf – are there group / permissions issues as well as the usual /var/log/messages and the kernel log with dmesg command for weird produced NFS client / server or network messages.

nfs-utils disabled serving NFS over UDP in version 2.2.1. Arch core updated to 2.3.1 on 21 Dec 2017 (skipping over 2.2.1.) If UDP stopped working then, add udp=y under [nfsd] in /etc/nfs.conf. Then restart nfs-server.service.

If the remote NFS server is running also Linux it is useful to check its /etc/default/nfs-kernel-server configuration

At some stall cases it might be also useful to remount the NFS (but as there might be a process on the Linux server) trying to read / write data from the remote NFS mounted FS it is a good idea to check (whether a process / service) on the server is not doing I/O operations on the NFS and if such is existing to kill the process in question with fuser

linux:~# fuser -k [mounted-filesystem]


2. Diagnose the problem interactively with htop

    Htop should be your first port of call. The most obvious symptom will be a maxed-out CPU.
    Press F2, and under "Display options", enable "Detailed CPU time". Press F1 for an explanation of the colours used in the CPU bars. In particular, is the CPU spending most of its time responding to IRQs, or in Wait-IO (wio)?

3. Get more extensive Mount info with mountstats


nfs-utils package contains mountstats command which is very useful in debugging further the issues identified

$ mountstats
Stats for example:/tank mounted on /tank:
  NFS mount options: rw,sync,vers=4.2,rsize=524288,wsize=524288,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,proto=tcp,port=0,timeo=15,retrans=2,sec=sys,,local_lock=none
  NFS server capabilities: caps=0xfbffdf,wtmult=512,dtsize=32768,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x803,acl=0x3,sessions,pnfs=notconfigured
  NFS security flavor: 1  pseudoflavor: 0


NFS byte counts:
  applications read 248542089 bytes via read(2)
  applications wrote 0 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 171375125 bytes via NFS READ
  client wrote 0 bytes via NFS WRITE

RPC statistics:
  699 RPC requests sent, 699 RPC replies received (0 XIDs not found)
  average backlog queue length: 0

    338 ops (48%)
    avg bytes sent per op: 216    avg bytes received per op: 507131
    backlog wait: 0.005917     RTT: 548.736686     total execute time: 548.775148 (milliseconds)
    115 ops (16%)
    avg bytes sent per op: 199    avg bytes received per op: 240
    backlog wait: 0.008696     RTT: 15.756522     total execute time: 15.843478 (milliseconds)
    93 ops (13%)
    avg bytes sent per op: 203    avg bytes received per op: 168
    backlog wait: 0.010753     RTT: 2.967742     total execute time: 3.032258 (milliseconds)
    32 ops (4%)
    avg bytes sent per op: 220    avg bytes received per op: 274
    backlog wait: 0.000000     RTT: 3.906250     total execute time: 3.968750 (milliseconds)
    25 ops (3%)
    avg bytes sent per op: 268    avg bytes received per op: 350
    backlog wait: 0.000000     RTT: 2.320000     total execute time: 2.360000 (milliseconds)
    24 ops (3%)
    avg bytes sent per op: 224    avg bytes received per op: 176
    backlog wait: 0.000000     RTT: 30.250000     total execute time: 30.291667 (milliseconds)
    23 ops (3%)
    avg bytes sent per op: 220    avg bytes received per op: 160
    backlog wait: 0.000000     RTT: 6.782609     total execute time: 6.826087 (milliseconds)
    4 ops (0%)
    avg bytes sent per op: 224    avg bytes received per op: 14372
    backlog wait: 0.000000     RTT: 198.000000     total execute time: 198.250000 (milliseconds)
    2 ops (0%)
    avg bytes sent per op: 172    avg bytes received per op: 164
    backlog wait: 0.000000     RTT: 1.500000     total execute time: 1.500000 (milliseconds)
    1 ops (0%)
    avg bytes sent per op: 172    avg bytes received per op: 164
    backlog wait: 0.000000     RTT: 2.000000     total execute time: 2.000000 (milliseconds)
    1 ops (0%)
    avg bytes sent per op: 164    avg bytes received per op: 116
    backlog wait: 0.000000     RTT: 1.000000     total execute time: 1.000000 (milliseconds)

nfs-utils disabled serving NFS over UDP in version 2.2.1. Arch core updated to 2.3.1 on 21 Dec 2017 (skipping over 2.2.1.) If UDP stopped working then, add udp=y under [nfsd] in /etc/nfs.conf. Then restart nfs-server.service.

4. Check for firewall issues

If all fails make sure you don't have any kind of firewall issues. Sometimes firewall changes on remote server or somewhere in the routing servers might lead to stalled NFS mounts.


To use properly NFS as you should know as a minimum you need to have opened as ports is Port 111 (TCP and UDP) and 2049 (TCP and UDP) on the NFS server (side) as well as any traffic inspection routers on the road from SRC (Linux client host) and NFS Storage destination DST server.

There are also ports for Cluster and client status (Port 1110 TCP for the former, and 1110 UDP for the latter) as well as a port for the NFS lock manager (Port 4045 TCP and UDP) but having this opened or not depends on how the NFS is configured. You can further determine which ports you need to allow depending on which services are needed cross-gateway.

5. How to Remount a Stalled unresponsive NFS filesystem mount


At many cases situation with remounting stalled NFS filesystem is not so easy but if you're lucky a standard mount and remount should do the trick.

Most simple way to remout the NFS (once you're sure this might not disrupt any service) – don't blame me if you break something is with:

umount -l /mnt/NFS_mnt_point
mount /mnt/NFS_mnt_point

Note that the lazy mount (-l) umount opt is provided here as very often this is the only way to unmount a stalled NFS mount.

Sometimes if you have a lot of NFS mounts and all are inacessible it is useful to remount all NFS mounts, if the remote NFS is responsive this should be possible with a simple for bash loop:

for P in $(mount | awk '/type nfs / {print $3;}'); do echo $P; echo "sudo umount $P && sudo mount $P" && echo "ok :)"; done

If you cd /mnt/NFS_mnt_point and try ls and you get

$ ls
.: Stale File Handle


You will need to unmount the FS with forceful mount flag

umount -f /mnt/NFS_mnt_point

Sum it up

In this article, I've shown you a few simple ways to debug what is wrong with a Stalled / Hanged NFS filesystem present on a NFS server mounted on a Linux client server.
Above was explained the common issues caused by NFS portmap (rpcbind) dependency, how to its status is fine, some further diagnosis with htop and mountstat was pointed. I've pointed the minimum amount of TCP / UDP ports 2049 and 111 that needs to be opened for the NFS communication to work and finally explained on how to remount a stalled NFS single or all attached mount on a NFS client to restore to normal operations.
As NFS is a whole ocean of things and the number of ways it is used are too extensive this article is just a general info useful for the NFS dummy admin for more robust configs read some good book on NFS such as Managing NFS and NIS, 2nd Edition – O'Reilly Media and for Kernel related NFS debugging make sure you check as a minimum ArchLinux's NFS troubleshooting guide and sourceforge's NFS Troubleshoting and Optimizing NFS Performance guides.


Howto create Linux Music Audio CD from MP3 files / Create playable WAV format Audio CD Albums from MP3s

Tuesday, July 16th, 2019

Reading Time: 8 minutes


Recently my Mother asked me to prepare a Music Audio CD for her from a popular musician accordionist Stefan Georgiev from Dobrudja who has a unique folklore Bulgarian music.

As some of older people who still remember the age of the CD and who had most likely been into the CD burning Copy / Piracy business so popular in the countries of the ex-USSR so popular in the years 1995-2000 audio ,  Old CD Player Devices were not able to play the MP3 file format due to missing codecs (as MP3 was a proprietary compression that can't be installed on every device without paying the patent to the MP3 compression rights holder.

The revolutionary MP3 compression used to be booming standard for transferring Music data due to its high compression which made an ordinary MP3 of 5 minutes of 5MB (10+ times more compression than an ordinary classic WAV Audio the CPU intensiveness of MP3 files that puts on the reading device, requiring the CD Player to have a more powerful CPU.

Hence  due to high licensing cost and requirement for more powerful CPU enabled Audio Player many procuders of Audio Players never introduced MP3 to their devices and MP3 Neve become a standard for the Audio CD that was the standard for music listening inside almost every car out there.

Nowdays it is very rare need to create a Audio CD as audio CDs seems to be almost dead (As I heard from a Richard Stallman lecture In USA nowadays there is only 1 shop in the country where you can still buy CD or DVD drives) and only in third world as Africa Audio CDs perhaps are still in circulation.

Nomatter that as we have an old Stereo CD player on my village and perhaps many others, still have some old retired CD reading devices being able to burn out a CD is a useful thing.

Thus to make mother happy and as a learning excercise, I decided to prepare the CD for her on my Linux notebook.
Here I'll shortly describe the takes I took to make it happen which hopefully will be useful for other people that need to Convert and burn Audio CD from MP3 Album.


1. First I downloaded the Album in Mp3 format from Torrent tracker

My homeland Bulgaria and specific birth place place the city of Dobrich has been famous its folklore:  Galina Durmushlijska and Stefan Georgiev are just 2 of the many names along with Оркестър Кристал (Orchestra Crystal) and the multitude of gifted singers. My mother has a santiment for Stefan Georgiev, as she listened to this gifted accordinist on her Uncle's marriage.

Thus In my case this was (Стефан Георгиев Хора и ръченици от Добруджа) the album full song list here If you're interested to listen the Album and Enjoy unique Folklore from Dobrudja (Dobrich) my home city, Stefan Georgiev's album Hora and Rachenica Dances is available here


I've downloaded them from Bulgarian famous torrent tracker in MP3 format.
Of course you need to have a CD / DVD readed and write device on the PC which nowdays is not present on most modern notebooks and PCs but as a last resort you can buy some cheap External Optical CD / DVD drive for 25 to 30$ from Amazon / Ebay etc.


2. You will need to install a couple of programs on Linux host (if you don't have it already)

To be able to convert from command line from MP3 to WAV you will need as minimum ffmpeg and normalize-audio packages as well as some kind of command line burning tool like cdrskin  wodim which is
the fork of old good known cdrecord, so in case if you you're wondering what happened with it just
use instead wodim.

Below is a good list of tools (assuming you have enough HDD space) to install:


root@jeremiah:/ # apt-get install –yes dvd+rw-tools cdw cdrdao audiotools growisofs cdlabelgen dvd+rw-tools k3b brasero wodim ffmpeg lame normalize-audio libavcodec58


Note that some of above packages I've installed just for other Write / Read operations for DVD drives and you might not need that but it is good to have it as some day in future you will perhaps need to write out a DVD or something.
Also the k3b here is specific to KDE and if you're a GNOME user you could use Native GNOME Desktop app such brasero or if you're in a more minimalistic Linux desktop due to hardware contrains use XFCE's native xfburn program.

If you're a console / terminal geek like me you will definitely enjoy to use cdw

root@jeremiah:/ # apt-cache show cdw|grep -i description -A 1
Description-en: Tool for burning CD's – console version
 Ncurses-based frontend for wodim and genisoimage. It can handle audio and

Description-md5: 77dacb1e6c00dada63762b78b9a605d5


3. Selecting preferred CD / DVD / BD program to use to write out the CD from Linux console

cdw uses wodim (which is a successor of good old known console cdrecord command most of use used on Linux in the past to burn out new Redhat / Debian / different Linux OS distro versions for upgrade purposes on Desktop and Server machines.

To check whether your CD / DVD drive is detected and ready to burn on your old PC issue:


root@jeremiah:/# wodim -checkdrive
Device was not specified. Trying to find an appropriate drive…
Detected CD-R drive: /dev/cdrw
Using /dev/cdrom of unknown capabilities
Device type    : Removable CD-ROM
Version        : 5
Response Format: 2
Capabilities   :
Vendor_info    : 'HL-DT-ST'
Identification : 'DVDRAM GT50N    '
Revision       : 'LT20'
Device seems to be: Generic mmc2 DVD-R/DVD-RW.
Using generic SCSI-3/mmc   CD-R/CD-RW driver (mmc_cdr).
Supported modes: TAO PACKET SAO SAO/R96P SAO/R96R RAW/R16 RAW/R96P RAW/R96R

You can also use xorriso (whose added value compared to other console burn cd tools is is not using external program for ISO9660 formatting neither it use an external or an external burn program for CD, DVD or BD (Blue Ray) drive but it has its own libraries incorporated from libs.

Below output is from my Thinkpad T420 notebook. If the old computer CD drive is there and still functional in most cases you should not get issues to detect it.

cdw ncurses text based CD burner tool's interface is super intuitive as you can see from below screenshot:


CDW has many advanced abilities such as “blanking” a disk or ripping an audio CD on a selected folder. To overcome the possible problem of CDW not automatically detecting the disk you have inserted you can go to the “Configuration” menu, press F5 to enter the Hardware options and then on the first entry press enter and choose your device (by pressing enter again). Save the setting with F9.

4. Convert MP3 / MP4 Files or whatever format to .WAV to be ready to burn to CD

Collect all the files you want to have collected from the CD album in .MP3 a certain directory and use a small one liner loop to convert files to WAV with ffmpeg:

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

for i in $( ls *.mp3); do ffmpeg -i $i $i.wav; done

If you don't have ffmpeg installed and have mpg123 you can also do the Mp3 to WAV conversion with mpg123 cmd like so:


for i in $( ls ); do mpg123 -w $i.wav $i.mp3; done

Another alternative for conversion is to use good old lame (used to create Mp3 audio files but abling to also) decode
mp3 to wav.


lame –decode somefile.mp3 somefile.wav

In the past there was a burn command tool that was able to easily convert MP3s to WAV but in up2date Linux modern releases it is no longer available most likely due to licensing issues, for those on older Debian Linux 7 / 8 / 9 / Ubuntu 8 to 12.XX / old Fedoras etc. if you have the command you can install burn and use it (and not bother with shell loops):

apt-get install burn


yum install burn

Once you have it to convert


$ burn -A -a *.mp3


5. Fix file naming to remove empty spaces such as " " and substitute to underscores as some Old CD Players are
unable to understand spaces in file naming with another short loop.


for f in *; do mv "$f" `echo $f | tr ' ' '_'`; done


6. Normalize audio produced .WAV files (set the music volume to a certain level)

In case if wondering why normalize audio is needed here is short extract from normalize-audio man page command description to shed some light.

"normalize-audio  is  used  to  adjust  the volume of WAV or MP3 audio files to a standard volume level.  This is useful for things like creating mp3 mixes, where different recording levels on different albums can cause the volume to  vary  greatly from song to song."

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

normalize-audio -m *.wav


7. Burn the produced normalized Audio WAV files to the the CD


wodim -v -fix -eject dev='/dev/sr0' -audio -pad *.wav

Alternatively you can conver all your MP3 files to .WAV with anything be it audacity
or another program or even use 
GNOME's CDBurn tool brasero (if gnome user) or KDE's CDBurn which in my opinion is
the best CD / DVD burning application for Linux K3B.

Burning Audio CD with K3b is up to few clicks and super easy and even k3b is going to handle the MP3 to WAV file Conversion itself. To burn audio with K3B just run it and click over 'New Audio CD Project'.


For those who want to learn a bit more on CD / DVD / Blue-Ray burning on GNU / Linux good readings are:
Linux CD Burning Mini Howto, is Linux's CD Writing Howto on ibiblio (though a bit obsolete) or Debian's official documentation on BurnCD.

8. What we learned here

Though the accent of this tutorial was how to Create Audio Music CD from MP3 on GNU / Linux, the same commands are available in most FreeBSD / NetBSD / OpenBSD ports tree so you can use the same method to build prepare Audio Music CD on *BSDs.

In this article, we went through few basic ways on how to prepare WAV files from MP3 normalize the new created WAV files on Linux, to prepare files for creation of Audio Music CD for the old mom or grandma's player or even just for fun to rewind some memories. For GUI users this is easily done with  k3b,  brasero or xfburn.

I've pointed you to cdw a super useful text ncurses tool that makes CD Burninng from plain text console (on servers) without a Xorg / WayLand  GUI installed super easy. It was shortly reviewed what has changed over the last few years and why and why cdrecord was substituted for wodim. A few examples were given on how to handle conversion through bash shell loops and you were pointed to some extra reading resources to learn a bit more on the topic.
There are plenty of custom scripts around for doing the same CD Burn / Covnersion tasks, so pointing me to any external / Shell / Perl scripts is mostly welcome.

Hope this learned you something new, Enjoy ! 🙂

Into great depression – What is like to live in the Balkans?

Wednesday, May 18th, 2011

Reading Time: 3 minutes

I'm so depressed these days that I'm trying to write something decent here but everytime I try I do stop and delete all I have written and start from scratch again.
It's terrible, I believe everyone have this days and they're so dark that even the smallest ray of light is gone somewhere…

The causes for depression are multiple, I know we're entering into the season period and that could be a factor, but truly for a long time I haven't felt that bad and I really cannot find the true cause. It's like hunting the unexplained.
Being a citizen of a countries on the balkans brings a lot of questions which cannot get answered. Why we the Balkan and more specificly most of the Orthodox Christian countries are suffering so badly and economically in constant crisis and recession?
From a material perspective Bulgaria is one of the worst countries one can live in, we the people on the balkans are chronically depressed and it really seems like a downward spiral
We've been gone through so far, when I was a child we were teached in the spirit of communism and a believe in a hard material realities.

Communism has taught us we're all fleshly brothers and we should live in groups and stick to the group, now as the democracy come it's on the contraty, we're being constantly re-taught that we should leave behind the group kind of thinking and all built from communism destroy it all and build the new society… We're told by individualist nations like USA and Western europe that the only thing for the good of a person is to (get an absolute individualistic life and only exist for the greater goodness of each ones self as individuals..

As with everything the Balkans are notable for being a very unordered place. Living here is like living in chaos…
The social security policies here are not working, the jurisdiction is working on behalf of the rich, the police force is seriously disfunctional and easily bribable. Put next to all this shit a high levels of unemployment and a lot of unhappy depressed people crawling around the streets and you get the picture …
As a normal consequence most of the young people have entered a dark ways of alcoholism and hard-core nihillism.
There are high level of people who are oriented into the new dark realities of Metal or underground music.
Each philosophy that is being put in from the west is being adopted here and being multiplied million times and mostly the bad things are being adopted and less rarely the good ones…
It's so mixed up that nobody can explain why it is happening as it is here.
I really am trying hard to convince myself for a years now that it is worthy to live here but the more I live here in Bulgaria the more I see all is getting worser than getting for good.

I wonder for how long it will go this pointless way, we the balkan people are living in ruins literally.

The only light we still have is the Church, but very sadly most people has left behind the faith and prefer to follow the fake American dream than to obey to our old ways and traditions.

Globalisation has entered in the Balkans in a full-force and is destroying our ancient culture and traditions and building the fakeness of the coca-cola culture that most of the people prefer to adore nowdays …

Bulgaria's population is mostly based of old people and we're a dying nation, if a miracle doesn't happen then we definitely will be gone.

Ubuntu 9.04 Jaunty on Toshiba L300 PLSBGE Laptop

Sunday, September 20th, 2009

Reading Time: 2 minutes


Today I had the task to Install Ubuntu GNU / Linux on Toshiba L300 PLSBGE Laptop.

I had already installed Windows Vista Enterprise on the notebook. Thus I used Acronis Disk Directory Suite to partition the harddrive for Ubuntu even though I could have used the gparted included on Ubuntu’s installation CD. I’ve asked specificly if
I could use the Ubuntu CD to repartition my hard drive in #ubuntu in
The answer was positive it’s absolutely safe to resize an NTFS partition using Ubuntu’s installation LiveCD and gparted.That seems like a good news for all of us the free software users / enthusiasts / hobbyists etc.
I hated the default Ubuntu behaviour it automatically decided to install itself on a automatically created 2.5 GB partition.
I thought it won’t be so dumb to install itself on such a tiny partition. Well guess what I was wrong IT WAS SO DUMB!
Even though I had already 80 GB ext3 partition. Ubuntu’s default behaviour was to install on an automatically created 2.5GB partition.
Right after the installation I was stunned just to realize there was no free space on the drive where the dumbass installed itself.
After that I had some issues deleting the already cretead partitions, which for some reason messed up. Luckily the good old fdisk fixed the situation so I could easily delete all the partitions except the vista and recreate them again using gparted with Ubuntu’s install LiveCD. The install was completely flawless. Everything worked out of the box, no external efforts to fix broken stuff like usually happens in Linux fantastic! The only broken thing was that switching to plain console with ctrl+alt+f1 would visualize a non working display output and I couldn’t see the ttys at all. That wasn’t such a hassle since the system is planned to work only in GUI mode. I guess Ubuntu is going through a real development with a positive end results. Even external USB printer Canon PIXMA IP3300 worked by simply plugging it in. For some weird reason to make the printer work correctly I had to use driver for Canon PIXMA IP3000. Anyways it worked with it and the printer started printing correctly even though officially on some of the linux printers databases it’s reported the printer won’t work correctly never ever.END—–

Optimizing Linux TCP/IP Networking to increase Linux Servers Performance

Tuesday, April 8th, 2008

Reading Time: 3 minutes


Some time ago I thought of ways to optimize my Linux Servers network performance.

Even though there are plenty of nice articles on the topic on how to better optimize Linux server performance by tunning up the kernel sysctl (variables).

Many of the articles I found was not structed in enough understandable way so I decided togoogle around and  found few interesting websites which gives a good overview on how one can speed up a bit and decrease overall server loads by simply tuning few basic kernel sysctl variables.

Below article is a product of my research on the topic on how to increase my GNU / Linux servers performance which are mostly running LAMP (Linux / Apache / MySQL / PHP) together with Qmail mail servers.

The article is focusing on Networking as networking is usual bottleneck for performance.
Below are the variables I found useful for optimizing the Linux kernel Network stack.

Implementing the variables might reduce your server load or if not decrease server load times and CPU utilization, they would at lease increase thoroughput so more users will be able to access your servers with (hopefully) less interruptions.
That of course would save you some Hardware costs and raise up your Servers efficiency.

Here are the variables themselves and some good example:

# = 0 ( Turn off IP Forwarding )

net.ipv4.conf.default.rp_filter = 1

# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0

# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0 ( same as above )
net.ipv4.conf.default.accept_source_route = 0

# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
( - || - )net.ipv4.tcp_fin_timeout = 40

# ( Decrease FIN timeout ) - Useful on busy/high load
serversnet.ipv4.tcp_keepalive_time = 4000 ( keepalive tcp timeout )
net.core.rmem_default = 786426 - Receive memory stack size ( a good idea to increase it if your server receives big files )
net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608 ( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
( maximum amount of option memory buffers )

# like a homework investigate by yourself what the variables below stand for :)
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024


# Below are newly added experimental
#net.core.rmem_max = 16777216
#net.core.wmem_max = 16777216
##kernel.msgmni = 1024
##kernel.sem = 250 256000 32 1024


Also a good sysctl.conf file which one might want to substitite or use as a skele for some productive server is ready for download here

Even if you can't reap out great CPU reduction benefits from integrating above values or similar ones, your overall LAMP performance to end customers should increase – at some occasions dramatically, at others little bit but still noticable.

If you're unsure on exact kernel variable values to use check yourself what should be the best values that fits you according to your server Hardware – usually this is done by experimenting and reading the kernel documentation as provided for each one of uplisted variables.

Above sysctl.conf is natively created to run on Debian and on other distributions like CentOS, Fedora Slackware some values might either require slight modifications.

Hope this helps and gives you some idea of how network optimization in Linux is usually done. Happy (hacking) tweakening !

Testament of a great saint – Testament of Saint John of Rila – Spiritual guidance for people from last ages

Thursday, August 8th, 2013

Reading Time: 16 minutes


[1.] I, John, the humble and sinful, who has never done anything good on earth, when I came into this wilderness of Rila, I found no man over here, but only wild animals and impenetrable thickets. I settled alone in it among the wild animals, without food nor shelter, but the sky was my shelter and the earth my bed and the herbs my food. But the good Lord, for the love of whom I disregarded everything and endured hunger and thirst, frost, the heat of the sun, and corporal nakedness, did not abandon me, but like a merciful and child-loving father he lavishly satisfied all my needs. What shall I contribute to the Lord for all he has given me? Many are his benefactions to me, for he looked from his holy height at my humbleness (cf. Luke 1:48) and lent his support to me to go through everything—not I, but the might of Christ, which is in me—because every good gift and every perfect gift is from him (James 1:17).

[2.] Seeing you today gathered together in the Lord here, where, as I told you, no man has dwelled until now, but only wild animals, and foreseeing that the end of my life here is soon coming on, because of this I made up my mind, before my departure (II Tim. 4:6) from life here, to leave you the present fatherly testament of mine, just as carnal fathers leave their children an earthly inheritance of silver and gold and other property, so that when you commemorate your father in the Holy Spirit, you do not forget his testament.

[3.] I know, my beloved children in God, I know you very well, that you, being beginners, are not confirmed yet in the monk’s life, but fear not, for the Lord’s "power is made perfect in weakness" (II Cor. 12:9). Just because of this I made up my mind to write for you this rough and ignorant testament of mine, so that you will keep it always in your minds to become stronger in body and soul, in the Lord, and go forward through the virtues in fear of God. Because I believe in my God, whom I have served since my youth and to whom I submitted zealously, after my departure, this wilderness, which until now was terrible and uninhabited, will be inhabited by a multitude of desert-citizens. What was written about it will be fulfilled: "The desolate hath many more children than she which hath a husband" (Is. 54:1; Gal. 4:27).

Св. Йоан Рилски. Стенопис от XIV в. в църквата на Земенския манастир.[4.] Because of this I beg you, my children, whom I have gathered in the Lord, I beg of you, my flesh and blood, do not neglect your father’s admonition and together with the apostle I say: “I am in travail again until Christ be formed in you” (Gal. 4:19). I beg you and make you swear on the dread name of God not to violate or abandon anything after my death, but everything I have written let be carried out, as it is written and as you have promised before God. Whosoever oversteps or violates something of it, let him be damned and separated from the Father and the Son and the Holy Spirit, to have no share with the saints, who were pleasing to God ages ago, but let his share be with those who had crucified the Lord of Glory (Acts 7:2) and with his betrayer Judas, to be erased from “the book of life” (Phil. 4:3) and not to be inscribed [in it] with the righteous.

[5.] First of all, I bequeath to you the obligation to preserve the holy faith immaculate and unaffected by any false teaching, just as we received it from the holy fathers, without "being led away with diverse and strange teachings" (Heb. 13:9). Hold fast and keep the traditions you have heard and seen from me. Do not deviate either to the right, or to the left, but walk along the royal road. Keep yourselves carefully away from worldly fascinations and always remember why you have come out of the world, and why you have despised it and worldly things.

[6.] Now again, keep yourselves away from the avaricious snake, “for the love of money is the root of all evil” (I Tim. 6:10), according to the apostle, who calls it a second idolatry. Because for the hermit wealth consists not in silver and gold,5 but in perfect poverty, in the denial of his personal will, and in lofty humbleness. I am not telling you this as my commandments, but [I am] recalling for you the commandments of Christ. For he told his holy disciples and through them everybody who had renounced the world: "Take no gold, nor silver, nor a bag, nor copper in your belts" (Matt. 10:9) and so on. For gold and silver are great enemies of the monk and bite those who have them like a snake.

holy relics of-one of greatest saints of all times Bulgarian saint John of Rila Rila mountain bulgaria

[7.] If we, however, have undoubted hope in God, he will not leave us deprived of anything, for he himself says: “A woman may forget her children, yet will I not forget thee” (Is. 49:15). Also in another place: “But seek first the kingdom of God, and his righteousness; and all these things shall be yours as well” (Matt. 6:33). For in the beginning, when I came to this wilderness, the sly enemy attempted to allure me, for the pious king sent to me a lot of gold.6 For the sake of God I refused to see him, for I understood that it was a perfidy of the devil. I did not accept it, but returned it to those who sent it, for I thought to myself: “If I wished to have gold and silver, and suchlike things, why came I into this terrible and impenetrable wilderness, where I found no man, but wild animals?” So I saved myself from the intrigues of the sly tempter, who endeavors to trip us up in those things, which we renounced willfully. That is why you are not to look for any of these things, “for your heavenly Father knows that you need them all” (Matt. 6:32) before your prayer [is offered].

[8.] Nor look to be recognized and beloved by earthly kings and princes, nor put your hope in them, leaving the heavenly King, with whom you enlisted to be soldiers and "wrestle not against flesh and blood," but "against the ruler of the darkness of this world" (Eph. 6:12). For the prophet Jeremiah also threatens us speaking so: "Cursed be the man that hopeth in man" and the rest. Enumerating the evils, he adds that "blessed is the man that hopeth in the Lord" (Jer. 17:5-8). Do not say: "What shall we eat, or drink, or in what shall we be dressed?" for the gentiles seek after these things. "Look at the birds of the air: for they neither sow nor reap, nor gather into barns; yet your heavenly father feeds them. Are you not of more value than they?" (Matt. 6:26). As soon as you have come out of the world, do not go back, neither with your body, nor with your mind, for, as it is said, "No man, having put his hand to the plough, and looking back, is fit for the Kingdom of Heaven" (Luke 9:62).

[9.] The Apostle [Paul] too, however, teaches us to "forget what lies behind and strain froward to what lies ahead" (Phil. 3:13). What does "forgetting those things which are behind" mean, my children? Nothing else except to deliver to oblivion all those things which, coming out of the world for God’s sake, we have left and despised, and to strive towards the feat which lies before us, to which we were called by our taskmaster, our most gracious God and Lord Jesus Christ, who has enabled us to endure his gentle yoke, "For his yoke is easy, and his burden is light" (Matt. 11:30).

[10.] As the grace of the Holy Spirit brought you together, so must you endeavor to live with one heart and one mind and one spirit, directing your eyes only towards the eternal reward, which God has prepared for those who have loved him. The communal life is in every way more useful for monks than the solitary one, for solitude is not suitable for the many, but only for a few who are perfect in all monastic virtues. The common life, on the other hand, is useful in general for everybody, about which the patristic books tell us and teach us sufficiently. The spirit-speaking prophet David glorified it saying: "See now what is so good and so pleasant as for brethren to dwell together in unity!" (Ps. 133:1). In addition to this, one spirit-moved ecclesiastical hymn writes in this way: "Because in this the Lord promised eternal life." But also our good Master Lord God Jesus Christ, does he not say to us himself, by his immaculate lips: "Where two or three are gathered together in my name, there I am in the midst of them"? (Matt. 18:20). Our God-bearing fathers say for the solitary life: "Woe to him that is alone when he falls; and there is not a second to lift him up" (Eccl. 4:10).

[11.] That is why, children, as the Holy Spirit through the mouth of the prophet glorifies the communal life, do you not neglect it either, but on the contrary, confirm it and be like "one body in the Lord" (Rom. 12:5), which has different members. Some of them form, however, the head which governs, others the feet which toil and bear, so that there is formed from all a single spiritual body in the Lord, created with a single mind and logical spirit, and directed by spiritual reasoning, in no wise having divisions. When such a dwelling and life in God is arranged, then he himself will be in the midst of you, governing you invisibly.

[12.] Do not seek the first place and authority, but remember those who have said: "If one would be first, he must be last of all, and servant of all" (Mark 9:35). Elect for yourselves preceptors and appoint superiors, whom God will show you, that is, men "of good report" (Acts 10:22) among everybody in spiritual matters and surpassing everybody in intelligence and spiritual discernment, and able to pasture well and comfortably the flock entrusted to them down the meadows of piety and of the life-giving commands of Christ. For these men it is proper to seek confirmation more from God than from our opinion.

      7. According to Goshev, "Zavetât," pp. 449–61, this author’s Parainesis had been available in a Slavonic translation since the reign of Symeon (893–927).

[13.] If, as our great father and monastic preceptor, the reverend Ephraem Syrus says,7 all of you begin to desire authority and presidencies, and all of you to be abbots, and all of you preceptors, and interpreters, and teachers, and among you spring up rivalries, quarrels, disputes, zealousness, calumnies, haughtinesses, envy and other passions indecorous for monks, then certainly be aware that Christ is not among you, for Christ is not the teacher of discord and dissent, but of peace and unity. For he prays to God the father for his holy disciples to be united, that is, of one mind—they themselves and everybody who believes in him through them, and says as follows: "Holy Father, keep them in thy name that they may be one, as we are" (John 17:11). In another place: "I do not pray for them only but also for those who believe in me through their word that all may be one" (John 17:20–21). If you will be one, be at peace one with another. For he said to his disciples, "Peace I leave with you, my peace I give you." (John 14: 27) For such is this peace of Christ, children, that again he speaks, saying, "Not as the world gives, do I give to you" (John 14:27). But this peace of Christ surpasses every mind. This is the peace, about which the prophet talks: "And his peace has no bounds." But also the apostle teaches us saying: "Strive for peace with all men and for the holiness, without which no man shall see God" (Heb. 12:14). May you have such a peace, now, among you, and let you arrange everything for God with great unity of mind and heart, so as not to enrage your own God and master.

[14.] If somebody is found among you who sows weeds, discords and other temptations, you have to eliminate at once such a man from your assembly, so that this will not be transfigured into a devouring canker, according to the apostle, and not to spread the evil among the good ones, and “lest any root of bitterness spring up and cause trouble by it, and the many be defiled” (Heb. 12:5); and the wicked wolf not trouble the peaceful flock of Christ, because this sort [of men] will appear. For of them Christ prophesies saying: "For it is necessary that temptations come; but woe to the world for temptations to sin!" (Matt. 16:7). For this and you, children, keep away from these things and do not allow them to live among you, but divert them away from yourselves as the shepherd chases away the scabby sheep from the pure flock.

[15.] Living together for the Lord’s sake and bearing the burdens of one another, do not neglect those who live in solitude and "wandering over deserts and in mountains, and in dens, and in caves of the earth, of whom the world was not worthy" (Heb. 11:38), but supply them as much as you can, in order to hold them as your petitioners before God, for the prayer of the pious may achieve much.

      8. For the translation of some of the late antique classics of ascetic literature into Slavonic, see Dujcev, "Réforme," p. 262.

[16.] Instruct yourselves in the Lord’s law day and night (Ps. 1:2). Read often the patristic books and try to be imitators of our holy fathers Antony, Theodosios and the others, who shone like lamps in the world with their good deeds.8 Hold firmly to the church rule, leaving or neglecting nothing of this, which is established by the holy fathers.

[17.] Manual labor must not be neglected by you, however, but work must be in your hands, and the prayer “Lord Jesus Christ, Son of God, have mercy on me, a sinner” must be permanently on your lips, as well as the memory of death in your mind. This was the practice of the ancient desert fathers. They did not eat their bread in vain, and they not only lived themselves by labor of their own hands, but they gave to the needy too, and so they were not disappointed in their hope. “For,” says the apostle [Paul], "it is well that the heart be strengthened by grace; not with foods which have not benefited their adherents" (Heb. 13:9). He says too: "Let brotherly love continue. Do not neglect to show hospitality to strangers; for thereby some have entertained angels unawares" (Heb. 13:1–2).

[18.] Establish the newly enlightened from your own race in the faith and instruct them to abandon the indecent pagan rites and the evil customs which they keep even after the acceptance of the holy faith. But they do this because of ignorance, and thus they need to be brought to their senses.

[19.] I had much more to say to you, my beloved children in the Lord, but it is impossible to write everything. I deliver you to him who is the source of all wisdom and reason, and the true Comforter— to the Holy and life-giving Spirit, in order that he himself gives you wisdom, to bring you to your senses, to enlighten you, to teach and instruct you in every good deed.

[20.] Now I leave you our beloved brother Gregory for instructor and superior in place of me, about whom all of you testify that he is able to govern you well and according to God, and you elect him by consensus as superior, even though he does not want it, but because of obedience and humility he acquiesces to your request. After him, [choose] whomever God will show you. As for myself, I wish henceforth to live in quiet and silence, to repent my sins and to beg mercy of God. Have mercy on me, your sinful father, always in your prayers that I may receive mercy on judgment day, for I have done nothing good on earth and fear that judgment and torment prepared for sinners like me. So may the blessing of God be with you all, guarding and protecting you from all evils. Amen.

I have written this in the year from the creation of the world 6449 ( = A.D. 941) on the twenty-fifth day of the month of March.

I, the humble and most sinful John, first inhabitant of the wilderness of Rila, sign with my own hand and confirm the above-written [testament].

© 2000 Dumbarton Oaks
Trustees for Harvard University
Washington, D.C.
Printed in the United States of America

Rila: Testament of John of Rila

Date: 941 Translator: Ilija Iliev

Here is also some more information of Rila Monastery for those who want to learn more what is consequence of a living of a great saint.

The monk John laid the foundations for what was to become the greatest monastery of medieval Bulgaria circa 930–31 in the mountains to the east of the Struma river valley in western Bulgaria.2 Born around 876–880, not much more than a dozen years after Boris-Michael (852–889), ruler of the Bulgars, had accepted Christianity in 865, John began his monastic career at the monastery of St. Dimiter near his birthplace, then lived for many years as a hermit. His final settlement was a site north of the Rila river, to the east of the present Rila monastery. Remains of the foundations of the first buildings are to be seen in the meadows south of the hermitage dedicated to St. Luke.3 Jealous of his independence, John refused to welcome the Bulgarian ruler Peter (927–969), who came to pay him homage. John’s Testament, translated below, was issued March 25, 941 to regulate the cenobitic community and is his only literary work. John then retired to his accustomed solitary life, and died on August 18, 946. He was a popular subject among hagiographers; seven lives in Bulgarian and two in Greek were composed between the twelfth and the nineteenth centuries.


B. Subsequent History of the Monastery in Medieval Times

Little is known about the Rila monastery during the Byzantine dominion over Bulgaria (1018– 1185). The earliest Slavonic life of John of Rila, the so-called "Popular Life," was composed in Bulgarian towards the end of this period, as was the first life in Greek, authored by George Skylitzes, an official on the staff of the Byzantine governor at Srédetz (modern Sofia) during the reign of Emperor Manuel I Komnenos (1143–1180), that now survives only in a Slavonic translation.5 The monastery’s fortunes revived considerably later during the Second Bulgarian Empire (1186–1396). Rila and several other monasteries, richly endowed by the Bulgarian rulers with new lands and villages, seem to have enjoyed considerable prosperity in the fourteenth century.6 There is also a charter of 1378 preserved in the monastery of the last Bulgarian king, Ivan Shishman (1371– 1393), that confirms the tax exemptions of the monastery’s existing properties and awards new lands as well; it refers to similar charters now lost that earlier Bulgarian monarchs had awarded to the monastery dating back to the middle of the thirteenth century.

Earlier in the fourteenth century, Rila had benefited also from a local patron, the protosebast Hreljo, a local lord and sometime vassal of the Serbian tsar Stephen Dusan (1331–1355), who erected a new monastery on the site of the existing Rila monastery, to the west of John’s original foundation, which continued in operation as the "Old Hermitage." Hreljo built a 75-foot protective tower, still preserved, in 1335. It included living quarters for Hreljo and his family as well as a chapel on the top floor dedicated to the Transfiguration.7 A brick inscription records Hreljo’s erection of this structure. In 1343 he also built a stone church, which survived until 1834. There were similar towers built at this time for the monasteries on Mount Athos (see (51) Koutloumousi [A4]), and there is one still existing at the Hilandar monastery. Forced to become a monk at the order of Dusan, who distrusted his loyalty, Hreljo was strangled to death by hired assassins in his tower in 1343, probably also at Dusan’s instigation. Hreljo’s gravestone, broken into many pieces, is preserved in the monastery’s museum and speaks of his entry into the monastery and unnatural death.8

In 1385, Dometian, the monastery’s superior, had John’s Testament recopied while hiding away the original along with the foundation’s other valuables for fear of the Turks, who had taken Srédetz in 1382. At about this time too Evtimij, the last Bulgarian patriarch of Turnovo, wrote his widely popular version of the Life of John of Rila.


C. Rila under Ottoman Rule

The Turkish sultans Beyazid I (1389–1402) and Mehmet I (1413–1421) issued firmans confirming the privileges Rila had received earlier from Bulgarian monarchs, but this did not save the monastery from later depredations, with the result that it was abandoned by the middle of the fifteenth century.9 There was a revival, however, in the second half of the century. Around 1460, the three brothers David, Joasaf and Teofan, sons of a certain Jakov, bishop of Krupnik, worked to strengthen and repair the damaged buildings. Shortly thereafter, a pact was reached in 1466 with the Russian monastery of St. Panteleemon on Mount Athos obliging Rila and the former institution to assist one another as needed in the future. Permission was obtained from the Turkish authorities in 1469 to transport the relics of John of Rila from Turnovo, the old capital of the Second Bulgarian Empire, where they had been since 1195. The translation considerably increased the Rila monastery’s prestige. A dependency (metoh) dedicated to Sts. Peter and Paul was built to the southwest of the main monastery in 1478.

Firmans issued by sultans Beyazid II (1481–1512) in 1498, Selim I (1512–1520) in 1519, and Murad III (1574–1595) confirmed the monastery in the possession of its properties, but like the earlier series of firmans, these did not succeed in protecting the foundation from the depredations of various brigands.10 Beginning in 1558–59, the monks succeeded in establishing direct relations with Russia, whose rulers they hoped would be sympathetic to their complaints of oppression at the hands of their Ottoman masters.

Despite extremely difficult conditions, which continued well into the second half of the eighteenth century, including attacks by robbers in 1766 and 1779, the monastery not only managed to survive but served as a kind of center of Bulgarian culture. Additional churches were built as dependencies towards the end of the eighteenth century and in the early years of the nineteenth century, then a complete reconstruction of the main monastery was begun in 1816.11 A fire in 1833, however, destroyed all the buildings there except for Hreljo’s tower and his fourteenthcentury stone church. The monastery was rebuilt once again in 1834 while the church was torn down to make room for a larger structure; both the monastery and the nineteenth-century church still stand today, along with Hreljo’s tower, the only medieval structure preserved on the site. Analysis Experts have endorsed the essential authenticity of the document.12 It is an example of the testamentary genre of monastic foundation documents, whose author seems to have made some use of (3) Theodore Studites and even (4) Stoudios.13 There are indeed some resemblances to the former document, such as the statement of purpose [3], the prohibition of changes [4], and the admonition to preserve the faith [5]. The use of the wilderness topos in the brief foundation history [1] also has a close parallel in (29) Kosmosoteira [1], a twelfth-century document. On the whole, however, this is a distinct document with its own concerns for the ordering of monastic life at Rila.


A. Lives of the Monks

Like his Stoudite predecessors, John of Rila endorses [10] the cenobitic lifestyle, but also urges his monks to establish [15] relations with and support neighboring solitaries. This coexistence of cenobitic and eremitic lifestyles, prefigured in John’s own career, would be one of the notable characteristics of Byzantine monasticism. The author demonstrates an acquaintance with the ascetic tradition of late antiquity, quoting [13] Ephraem Syrus and recommending [16] the study of patristic literature, in particular the Lives of St. Antony, founder of anchoritic monasticism, Theodosios the Koinobiarch, “and others” as well as respecting canon law. John also invokes [17] patristic authority for the practice of manual labor.


B. Constitutional Matters

While there is genuine disciplinary content in this document, its chief purpose, as in most testaments, was to designate [20] a successor, here the monk Gregory. John then announces his intent to retire into seclusion as part of an arrangement for assuring an orderly succession to the superiorship that is similar to that proposed in the eleventh century in (22) Evergetis [13]. C. Financial Matters

Aside from the commitment to self-sufficiency that seems implicit in his endorsement of manual labor, there are no indications of how John expected the foundation to support itself financially. He proudly asserts [7] that he refused a royal donation, perhaps an annuity like the solemnia attested in Byzantium in the tenth century, and he advises [8] his community not to seek favors from “earthly kings and princes” [8]. This deliberate shunning of material support is unusual. Many later founders did not fear for the independence of their foundations when accepting imperial largess or tax exemptions (e.g., (13) Ath. Typikon [36] or (19) Attaleiates [22]), nor did Rila itself long after John’s death. As Dujcev ("Réforme," p. 263) surmised, John probably was concerned about Bulgarian monasticism being too submissive to secular authority, understandably given the prior history of Bulgarian monasticism under royal patronage.


Unique MenuetOS – Free Software 32 / 64 bit OS entirely written in assembly language

Wednesday, July 10th, 2013

Reading Time: 3 minutes


unique operating-system menuetos written-in-assembler-programming-logo

Something very unique, I stumbled on some time ago and worthy to mention and recommend for everyone to test is MenuetOS. Can you imagine, someone might write an operating system entirely from scratch in 32 / 64 bit Assemler? Idea sounds crazy and impossible but in fact developers of MenuetOS already achieved it!

Unique OS - menuetos asm free os start-menu screenshot

Normally every modern operating system nowadays is based on some kind of UNIX / Linux / or NT (Windows) technology or at least follows some kind of POSIX standartization.
 The design goal of MenuetOS since the first release in year 2000, is to remove the extra layers between different parts of an OS. The more the layers more complicated the programming behind is and therefore this creates bugs more bugs. MenuetOS follows the idea of KISS model (Keep It Simple Stupid). Its amazing what people can write in pure asm programming!! 64 bit version of menuet is also backward compatible with 32 bit. MenuetOS supports mostly all any other modern OS does. Here is list of Supported Features:





  • – Pre-emptive multitasking with 1000hz scheduler, multithreading, multiprocessor, ring-3 protection
  • – Responsive GUI with resolutions up to 1920×1080, 16 million colours
  • – Free-form, transparent and skinnable application windows, drag'n drop
  • – SMP multiprocessor support with currently up to 8 cpus
  • – IDE: Editor/Assembler for applications
  • – USB 2.0 HiSpeed Classes: Storage, Printer, Webcam Video and TV/Radio support
  • – USB 1.1 Keyboard and Mouse support
  • – TCP/IP stack with Loopback & Ethernet drivers
  • – Email/ftp/http/chess clients and ftp/mp3/http servers
  • – Hard real-time data fetch
  • – Fits on a single floppy, boots also from CD and USB drives

MenuetOS has fully functional Graphic interface (environment). Though it is so simple it is much more fast (as written in assembler) and behaves more stable than other OS-es written in C / C++.
Its bundled with a POP3 / Imap mail client soft

menuetos assmebly OS mail client
As of time even some major legendary Games like DoomQuake, Sokoban and Chess are ported to MenuetOS !!!


MenuetOS Doom

quake legendary game running on Menuetos asm free OS

Quake I port on MenuetOS

Below are some more screenshots of Apps and stuff running

Maniac Mansion running on MenuetOS assembler build free Operating system

The world famous Maniac Mansion (1987)

Prince of Persia running on 32 64 bit assembler written GPL free-OS

Arcade Classic of 16 bit and 8 bit computers Prince of Persia running on top of dosbox on MenuetOS

For those who like to program old school MenuetOS has BASIC compiler, C library (supports C programming), debuggers, Command Prompt.

It even supports Networking and has some  most popular network adapters drivers as well as has basic browsing support through HTTP application.


You can listen music with CD Player but no support for mp3 yet.
To give MenuetOS a try just like any other Live Linux distribution it has Bootable LiveCD version – you can download it from here
MenuetOS is a very good for people interested to learn good 32 bit and 64 bit Assembler Programming.
Enjoy this unique ASM true hacker OS 😉

The Color of Pomegranates (1968) фильм “Цвет граната”/Նռան գույնը (1969г.)

Thursday, April 11th, 2013

Reading Time: < 1 minute

Those who enjoy post-modern art and odd movies should definitely check out The Color of Pomegranates. This movie is one among top Greatest Soviet Movies. There is a saying in Bulgaria "Is the movie Good or Russian?". In terms of surrealism in movie genre most of Soviet produced movies hit 10. In terms of normal viewer they're almost impossible to understand and very unimportant. The good thing about the movie is it shows some traditional things from some of ex USSR countries. Anyways I don't like the paganism in movie. Few of the Christian old paintings and things are worthy to see.


The Color of Pomegranates (1968) фильм "Цвет граната"/Նռան գույնը (1969г.)