Posts Tagged ‘left’

Why du and df reporting different on a filesystem / How to fix inconsistency between used space on FS and disk showing full strangeness

Wednesday, July 24th, 2019

linux-why-du-and-df-shows-different-result-inconsincy-explained-filesystem-full-oddity

If you're a sysadmin on a large server environment such as a couple of hundred of Virtual Machines running Linux OS on either physical host or OpenXen / VmWare hosted guest Virtual Machine, you might end up sometimes at an odd case where some mounted partition mount point reports its file use different when checked with
df
cmd than when checked with du command, like for example:
 

root@sqlserver:~# df -hT /var/lib/mysql
Filesystem   Type  Size Used Avail Use% Mounted On
/dev/sdb5      ext4    19G  3,4G    14G  20% /var/lib/mysql

Here the '-T' argument is used to show us the filesystem.

root@sqlserver:~# du -hsc /var/lib/mysql
0K    /var/lib/mysql/
0K    total

1. Simple debug on what might be the root cause for df / du inconsistency reporting

Of course the basic thing to do when in that weird situation is to be totally shocked how this is possible and to investigate a bit what is the biggest first level sub-directories that eat up the space on the mounted location, with du:

# du -hkx –max-depth=1 /var/lib/mysql/|uniq|sort -n
4       /var/lib/mysql/test
8       /var/lib/mysql/ezmlm
8       /var/lib/mysql/micropcfreak
8       /var/lib/mysql/performance_schema
12      /var/lib/mysql/mysqltmp
24      /var/lib/mysql/speedtest
64      /var/lib/mysql/yourls
144     /var/lib/mysql/narf
320     /var/lib/mysql/webchat_plus
424     /var/lib/mysql/goodfaithair
528     /var/lib/mysql/moonman
648     /var/lib/mysql/daniel
852     /var/lib/mysql/lessn
1292    /var/lib/mysql/gallery

The given output is in Kilobytes so it is a little bit hard to read, if you're used to Mbytes instead, do

 # du -hmx –max-depth=1 /var/lib/mysql/|uniq|sort -n|less

I've also investigated on the complete /var directory contents sorted by size with:

 # du -akx ./ | sort -n
5152564    ./cache/rsnapshot/hourly.2/localhost
5255788    ./cache/rsnapshot/hourly.2
5287912    ./cache/rsnapshot
7192152    ./cache


Even after finding out the bottleneck dirs and trying to clear up a bit, continued facing that inconsistently shown in two commands and if you're likely to be stunned like me and try … to move some files to a different filesystem to free up space or assigned inodes with a hope that shown inconsitency output will be fixed as it might be caused  due to some kernel / FS caching ?? and this will eventually make the mounted FS to refresh …

But unfortunately, if you try it you'll figure out clearing up a couple of Megas or Gigas will make no difference in cmd output.

In my exact case /var/lib/mysql is a separate mounted ext4 filesystem, however same issue was present also on a Network Filesystem (NFS) and thus, my first thought that this is caused by a network failure problem or NFS bug turned to be wrong.

After further short investigation on the inodes on the Filesystem, it was clear enough inodes are available:
 

# df -i /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5      1221600  2562 1219038   1% /var/lib/mysql

So the filled inodes count assumed issue also has been rejected.
P.S. (if you're not well familiar with them read manual, i.e. – man 7 inode).
 

– Remounting the mounted filesystem

To make sure the filesystem shown inconsistency between du and df is not due to some hanging network mount or bug, first logical thing I did is to remount the filesytem showing different in size, in my case this was done with:
 

# mount -o remount,rw -t ext4 /var/lib/mysql

For machines with NFS remote mounted storage locations, used:

# mount -o remount,rw -t nfs /var/www


FS remount did not solved it so I continued to ponder what oddity and of course I thought of a workaround (in case if this issues are caused by kernel bug or OS lib issue) reboot might be the solution, however unfortunately restarting the VMs was not a wanted easy to do solution, thus I continued investigating what is wrong …

Next check of course was to check, what kind of network connections are opened to the affected hosts with:
 

# netstat -tupanl


Did not found anything that might point me to the reported different Megabytes issue, so next step was to check what is the situation with currently opened files by running processes on the weird df / du reported systems with lsof, and boom there I observed oddity such as multiple files

# lsof -nP | grep '(deleted)'

COMMAND   PID   USER   FD   TYPE DEVICE    SIZE NLINK  NODE NAME
mysqld   2588  mysql    4u   REG 253,17      52     0  1495 /var/lib/mysql/tmp/ibY0cXCd (deleted)
mysqld   2588  mysql    5u   REG 253,17    1048     0  1496 /var/lib/mysql/tmp/ibOrELhG (deleted)
mysqld   2588  mysql    6u   REG 253,17       777884290     0  1497 /var/lib/mysql/tmp/ibmDFAW8 (deleted)
mysqld   2588  mysql    7u   REG 253,17       123667875     0 11387 /var/lib/mysql/tmp/ib2CSACB (deleted)
mysqld   2588  mysql   11u   REG 253,17       123852406     0 11388 /var/lib/mysql/tmp/ibQpoZ94 (deleted)

Notice that There were plenty of '(deleted)' STATE files shown in memory an overall of 438:

# lsof -nP | grep '(deleted)' |wc -l
438


As I've learned a bit online about the problem, I found it is also possible to find deleted unlinked files only without any greps (to list all deleted files in memory files with lsof args only):

# lsof +L1|less


The SIZE field (fourth column)  shows a number of files that are really hard in size and that are kept in open on filesystem and in memory, totally messing up with the filesystem. In my case this is temp files created by MYSQLD daemon but depending on the server provided service this might be apache's www-data, some custom perl / bash script executed via a cron job, stalled rsync jobs etc.
 

2. Check all the list open files with the mysql / root user as part of the the server filesystem inconsistency debugging with:

– Grep opened files on server by user

# lsof |grep mysql
mysqld    1312                       mysql  cwd       DIR               8,21       4096          2 /var/lib/mysql
mysqld    1312                       mysql  rtd       DIR                8,1       4096          2 /
mysqld    1312                       mysql  txt       REG                8,1   20336792   23805048 /usr/sbin/mysqld
mysqld    1312                       mysql  mem       REG               8,21      24576         20 /var/lib/mysql/tc.log
mysqld    1312                       mysql  DEL       REG               0,16                 29467 /[aio]
mysqld    1312                       mysql  mem       REG                8,1      55792   14886933 /lib/x86_64-linux-gnu/libnss_files-2.28.so

# lsof | grep root
COMMAND    PID   TID TASKCMD          USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
systemd      1                        root  cwd       DIR                8,1       4096          2 /
systemd      1                        root  rtd       DIR                8,1       4096          2 /
systemd      1                        root  txt       REG                8,1    1489208   14928891 /lib/systemd/systemd
systemd      1                        root  mem       REG                8,1    1579448   14886924 /lib/x86_64-linux-gnu/libm-2.28.so

Other command that helped to track the discrepancy between df and du different file usage on FS is:
 

# du -hxa  / | egrep '^[[:digit:]]{1,1}G[[:space:]]*'
 

3. Fixing large files kept in memory filesystem problem


What is the real reason for ending up with this file handlers opened by running backgrounded programs on the Linux OS?
It could be multiple  but most likely it is due to exceeded server / client interactions or breaking up RAM or HDD drive with writing plenty of logs on the FS without ending keeping space occupied or Programming library bugs used by hanged service leaving the FH opened on storage.

What is the solution to file system files left in memory problem?

The best solution is to first fix custom script or hanged service and then if possible to simply restart the server to make the kernel / services reload or if this is not possible just restart the problem creation processes.

Once the process is identified like in my case this was MySQL on systemd enabled newer OS distros, just do:

# systemctl restart mysqld.service


or on older init.d system V ones:

# /etc/init.d/service restart


For custom hanged scripts being listed in ps axuwef you can grep the pid and do a kill -HUP (if the script is written in a good way to recognize -HUP and restart the sub-running process properly – BE EXTRA CAREFUL IF YOU'RE RESTARTING BROKEN SCRIPTS as this might cause your running service disruptions …).

# pgrep -l script.sh
7977 script.sh


# kill -HUP PID

Now finally this should either mitigate or at best case completely solve the reported disagreement between df and du, after which the calculated / reported disk space should be back to normal and show up approximately the same (note that size changes a bit as mysql service is writting data) constantly extending the size between the two checks.

# df -hk /var/lib/mysql; du -hskc /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5        19097172 3472744 14631296  20% /var/lib/mysql
3427772    /var/lib/mysql
3427772    total

What we learned?

What I've explained in this article is why and how it comes that 'zoombie' files reside on a filesystem
appearing to be eating disk space on a mounted local or network partition, giving strange inconsistent
reports, leading to system service disruptions and impossibility to have correctly shown information on used
disk space on mounted drive.

I went through with some standard logic on debugging service / filesystem / inode issues up explainat, that led me to the finding about deleted files being kept in filesystem and producing the filesystem strange sized / showing not correct / filled even after it was extended with tune2fs and was supposed to have extra 50GBs.

Finally it was explained shortly how to HUP / restart hanging script / service to fix it.

Some few good readings that helped to fix the issue:

What to do when du and df report different usage is here
df in linux not showing correct free space after file removal is here
Why do “df” and “du” commands show different disk usage?
 

No space left on device with free disk space / Why no space left on device while there is plenty of disk space on drive – Running out of Inodes

Tuesday, November 17th, 2015

no_space_left-on-device-while-there-is-disk-space-running-out-of-file-inodes-unix_linux_file_system_diagram.gif

On one of the servers, I'm administrating the websites started showing some Mysql database table corrup errors like:
 

Table './database_name/site_news_list_com' is marked as crashed and last (automatic?) repair failed

The server is using Oracle MySQL server community stable edition on Debian GNU / Linux 6.0, so I first thought during work the server crashed either due to some bug issue in MySQL or it crashed due to some PHP cron job that did something messy. Thus to solve the crashed tables, tried using mysqlcheck tool which helped pretty fine, at many times whether there were database / table corruptions. I've run the following set of mysqlcheck commands with root (superuser) in a bash shell after logging in through SSH:

:

server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log


In order for above commands to work, I've created the /root/.my.cnf containing my root (mysql CLI) mysql username and password, e.g. file has content like below:

[client]
user=root
password=MySecretPassword8821238

Btw a good note here is its generally a good idea (if you want to have consistent mysql databases) to automatically execute via a cron job 2 times a month, I've in root cronjob the following:

crontab -u root -l |grep -i mysqlcheck
04 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 07 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 12 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 17 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log


Strangely I got a lot of errors that some .MYI / .MYD .frm temp files, necessery for the mysql tables recovery can't be written inside /home/mysql/database_name

That was pretty weird and I thought there might be some issues with permissions, causing the inability to write, due to some bug or something so I went straight and checked /home/mysql/database_name permissions, e.g.::

server:/home/mysql/database_name# ls -ld soccerfame
drwx—— 2 mysql mysql 36864 Nov 17 12:00 soccerfame
server:/home/mysql/database_name# ls -al1|head -n 10
total 1979012
drwx—— 2 mysql mysql 36864 Nov 17 12:00 .
drwx—— 36 mysql mysql 4096 Nov 17 11:12 ..
-rw-rw—- 1 mysql mysql 8712 Nov 17 10:26 1_campaigns_diez.frm
-rw-rw—- 1 mysql mysql 14672 Jul 8 18:57 1_campaigns_diez.MYD
-rw-rw—- 1 mysql mysql 1024 Nov 17 11:38 1_campaigns_diez.MYI
-rw-rw—- 1 mysql mysql 8938 Nov 17 10:26 1_campaigns.frm
-rw-rw—- 1 mysql mysql 8738 Nov 17 10:26 1_campaigns_logs.frm
-rw-rw—- 1 mysql mysql 883404 Nov 16 22:01 1_campaigns_logs.MYD
-rw-rw—- 1 mysql mysql 330752 Nov 17 11:38 1_campaigns_logs.MYI


As seen from above output, all was perfect with permissions, so it should have been something else, so I decided to try to create a random file with touch command inside /home/mysql/database_name directory:

touch /home/mysql/database_name/somefile-to-test-writtability.txt touch: cannot touch ‘/scr1/data/somefile-to-test-writtability.txt‘: No space left on device


Then logically I thought the /home/mysql/ mounted ext4 partition got filled, because of crashed SQL database or a bug thus, checked with disk free command df whether there is enough space on server:

server:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 20G 7.6G 11G 42% /
udev 10M 0 10M 0% /dev
tmpfs 13G 1.3G 12G 10% /run
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/md2 256G 134G 110G 55% /home

Well that's weird? Obviously only 55% of available disk space is used and available 134G which was more than enough so I got totally puzzled why, files can't be written.

Then very logically, I thought it might be that /home directory has remounted as read only, because the SSD memory disk on server is failing and checked for errors in dmesg, i.e.:

server:~# dmesg|grep -i error


Also checked how exactly was partition mounted, to check whether it is (RO) read-only:

server:~# mount -l|grep -i /home
/dev/md2 on /home type ext4 (rw,relatime,discard,data=ordered)


Now everything become even more weirder, as obviously the disk continued to be claiming no space left on device, while in reality there was plenty of disk space.

Then after running a quick research on the internet for the no space left on device with free disk space, I've come across this great superuser.com thread which let me realize the partition run out of inodes and that's why no new file inodes could be assigned and therefore, the linux kernel is refusing to write the file on ext4 partition.

For those who haven't heard of Linux Partition Inodes here is link to Wikipedia and a quick quote:

In a Unix-style file system, the inode is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. Each inode stores the attributes and disk block location(s) of the filesystem object's data.[1] Filesystem object attributes may include manipulation metadata (e.g. change,[2] access, modify time), as well as owner and permission data (e.g. group-id, user-id, permissions).[3]
Directories are lists of names assigned to inodes. The directory contains an entry for itself, its parent, and each of its children.


Once I understood it is the inodes, I checked how many of them are occupied with cmd:

server:~# df -i /home
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 17006592 0 100% /home


You see, there were 0 (zero) free file inodes on server and that was the reason for no space left on device while there was actually free disk space

To clean up (free) some inodes on partition, first thing I did is to delete all old logs which were inside /home and files I positively know not to be necessery, then to find which directories allocating most innodes used:

server:~# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n


If you're on a regular old fashined IDE Hard Drive and not SSD or you have too much files inside this command will take really long …:

Therefore a better solution might be to frist:

a) Try to find root folders with large inodes count:

for i in /home/*; do echo $i; find $i |wc -l; done
Try to find specific folders:


You should get output like:

/home/new_website
606692
/home/common
73
/home/pcfreak
5661
/home/hipo
33
/home/blog
13570
/home/log
123
/home/lost+found
1

b) Then once you know the directory allocating most inodes, run the command again to see the sub-directories with most files (eating) partition innodes:

for i in /home/webservice/*; do echo $i; find $i |wc -l; done

One usual large folder which could free you some nodes is the linux source headers, but in my case it was simply a lot of tiny old logs being logged on the system for few years in the past without cleaning:

After deleting the log dirs and cache folder in my case /home/new_website/{log,cache}:

server:~# rm -rf /home/new_website/log/*
server:~# rm -rf /home/new_website/cache/*

 

 

a) Then, stopping Apache webserver to check prevent Apache to use MySQl databases while running database repair and restaring MySQL:
 

server:~# /etc/init.d/apache2 stop Restarting MySQL server
..
server:~# /etc/init.d/mysql restart
..


b) And re-issuing MySQL Check / Repair / Optimize database commands:
 

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

c) And finally starting the Apache Webserver again:
 

server:~# /etc/init.d/apache2 start


Some innodse got freed up:
 

server:~# df -i /home Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 16797196 209396 99% /home


And hooray by God's Grace and with help of prayers of The most Holy Theotokos (Virgin) Mary, websites started again !

Apache Webserver: No space left on device: Couldn’t create accept lock /var/lock/apache2/accept.lock – Fix

Wednesday, April 8th, 2015

Apache-http-server-no-space-left-on-device-semaphores-quotes-hard-disk-space-resolve-fix-howto
If out of a sudden your Apache webserver crashes and is refusing to start up by manually trying to restart it through its init script on Debian Linux servers – /etc/init.d/apache2 and RPM based ones: /etc/init.d/httpd

Checking in php_error.log there was no shown errors related to loading PHP modules, however apache's error.log show following errors:

[Wed Apr 08 14:20:14 2015] [error] [client 180.76.5.149] client denied by server configuration: /var/www/sploits/info/trojans_info/tr_data/y3190.html
[Wed Apr 08 14:20:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:20:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.15974) (5)
[Wed Apr 08 14:25:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:25:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16790) (5)
[Wed Apr 08 14:27:03 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:03 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16826) (5)
[Wed Apr 08 14:27:53 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:53 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16852) (5)
[Wed Apr 08 14:30:48 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:30:48 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17710) (5)
[Wed Apr 08 14:31:21 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:31:21 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17727) (5)
[Wed Apr 08 14:32:40 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:32:40 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17780) (5)
[Wed Apr 08 14:38:32 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?

As you can read the most likely reason behind above errors preventing for apache to start is /var/run/apache2.pid  unable to be properly written due to lack of disk space or due to disk quota set for users including for userID with which Apache is running.

First thing I did is of course to see how much free space is on the server:

df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg00-rootvol       4.0G  1.7G  2.2G  44% /
udev                           7.8G  204K  7.8G   1% /dev
tmpfs                           24G     0   24G   0% /dev/shm
/dev/sda1                      486M   40M  422M   9% /boot
/dev/mapper/vg00-lv_crashdump 1008M   34M  924M   4% /crashdump
/dev/mapper/vg00-homevol       496M   26M  445M   6% /home
/dev/mapper/vg00-lv_opt         12G  1.4G  9.9G  13% /opt
/dev/mapper/vg00-tmpvol        2.0G   68M  1.9G   4% /tmp
/dev/mapper/vg00-varvol        7.9G  609M  6.9G   8% /var
/dev/mapper/vg00-crashvol      1.9G   35M  1.8G   2% /var/crash
/dev/mapper/vg00-auditvol      124M  5.6M  113M   5% /var/log/audit
/dev/mapper/vg00-webdienste     60G   12G   48G  19% /webservice

As visible from above df command output , there is enough disk on HDD, so this is definitely not the issue:

Then I Checked whether there is Quota enabled on the Linux server with repquota command shows there are no quotas enabled:

# repquota / var/
repquota: Mountpoint (or device) / not found or has no quota enabled.
repquota: Mountpoint (or device) /var not found or has no quota enabled.
repquota: Not all specified mountpoints are using quota.

 

So obviously the only few left possible reason for Apache failing to start after invoked via init script is  either due to left tainted semaphores or due to some server hardware  RAM problem / or a dying  hard disk with bad blocks.

So what are Semaphores? Generally speaking Semaphores are apparatus for conveying information by means of visual signals between applications (something like sockets).They're used for communicating between the active processes of a certain application. In the case of Apache, they’re used to communicate between the parent and child processes, hence if Apache can’t properly write and coordinate these things down, then it can’t communicate properly with all of the processes it starts and hence the Main HTTPD process can't spawn probably its childs preventing Webserver to enter "started mode" and write its PID file.

To check general information about system semaphore arrays there is the ipcs -s command, however my experience is that ipcs -a is more useful (because it lists generally all kind of semaphores) including Semaphore Shared Memory Signals which are the most likely to cause you the problem.

ipcs -a

—— Shared Memory Segments ——–
key        shmid      owner      perms      bytes      nattch     status

—— Semaphore Arrays ——–
key        semid      owner      perms      nsems
0x00000000 22970368   www-data   600        1

—— Message Queues ——–
key        msqid      owner      perms      used-bytes   messages

As you see in my case there is a Semaphore Arrays which had to be cleaned to make Apache2 be able to start again.
 

To clean all left semaphores (arrays) preventing Apache from start properly, use below for one liner bash loop:
 

for i in `ipcs -s | awk '/www-data/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


Note that above for loop is specific to Debian on CentOS / Fedora / RHEL and other Linuxes the username with which stucked semaphores might stay will be apache or httpd

Depending on the user with which the Apache Webserver is running, run above loop like so:

For RPM based distros (CentOS / RHEL):

for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


For other distros such as Slackware or FreeBSD or any custom compiled Apache webserver:

for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


If there is also Shared Memory Segments you can remove them with ipcrm i.e.:

ipcrm -m 0x63637069


An alternative way to get rid of left uncleaned semaphores is with xargs:
 

ipcs -s | grep nobody | awk ‘ { print $2 } ‘ | xargs ipcrm


Even though this fixes the issue I understood my problems were due to exceeding semaphores, to check default number of set semaphores on Linux Kernel level as well as few Semaphore related values run below sysctl:

sysctl -a | egrep kernel.sem\|kernel.msgmni
kernel.msgmni = 15904

kernel.sem = 250        32000   32      128


As you can see the number of maximum semaphores is quite large so in my case the failure because of left semaphores was most likely due to some kind of Cracker / Automated bot scanner attack or someone trying malicious against the webserver or simply because of some kind of Apache bug or enormous high load the server faced.

Apache Webserver disable file extension – Forbid / Deny access in Apache config to certain file extensions for a Virtualhost

Monday, March 30th, 2015


If you're a Webhosting company sysadmin like me and you already have configured directory listing for certain websites / Vhosts and those files are mirrored from other  development webserver location but some of the uploaded developer files extensions which are allowed to be interptered such as php include files .inc / .htaccess mod_rewrite rules / .phps / .html / .txt need to be working on the dev / test server but needs to be disabled (excluded) from delivery or interpretting for some directory on the prod server.

Open Separate host VirtualHost file or Apache config (httpd.conf / apache2.conf)  if all Vhosts  for which you want to disable certain file extensions and add inside:

<Directory "/var/www/sploits">
        AllowOverride All

</Directory>

Extension Deny Rules such as:

For disabling .inc files from inclusion from other PHP sources:
 

<Files  ~ "\.inc$">
  Order allow,deny
  Deny from all
</Files>


To Disable access to .htaccess single file only

<Files ~ "^\.htaccess">
  Order allow,deny
  Deny from all
</Files>


To Disable .txt from being served by Apache and delivered to requestor browser:

<Files  ~ "\.txt$">
  Order allow,deny
  Deny from all
</Files>


To Disable any left intact .html from being delivered to client:

<Files  ~ "\.html$">
  Order allow,deny
  Deny from all
</Files>

Do it for as many extensions as you need.
Finally to make changes affect restart Apache as usual:

If on Deb based Linux issue:

/etc/init.d/apach2 restart


On CentOS / RHEL and other Redhats / RedHacks 🙂

/etc/init.d/httpd restart

Nessebar – Ancient Christian city and a wonderful rest resort

Wednesday, April 9th, 2014

nessebar-an-ancient-Christian-city-a-great-resort-place
Last Friday together with my best friend Mitko together with an old-school good friend Samuel and another friend Geоrgi travelled to Pomorie Monastery. The reasons to go there was to have a short pilgrimage journey and to baptize Samuil who had the good desire to receive Baptism. As always it is a God's blessing to spend time in monastery and this was time it was not different. We started the trip from Sofia to Pomorie in about 08:15 and with a few little breaks we reached theMonastery about 12:30. Before we start journey I call Pomorie Monastery's abbot father Ierotey to ask if he will bless our pilgrimage (for those who never was in monastery's everything  happens wtih a blessing and it is best before a monastic trip to ask a blessing). Once arrived the novice monk Milen met us and accomodated us in two of the monastic rooms.

On next day, there was the standard morning prayer service and the bells rung to wake us up, after the service we had a small talk with father Sergiy, met some of the brothers and with abbots blessing together with father Sergiy, we went for a few hours pilgrimage journey to Nessebar.

Nessebar (the town ancient name is Mesambria) is an ancient city situated 16 km from Pomorie (known in ancient times as Anchialo / Anchialo). Both Pomorie and Nesebar was a great historical Christian sites from very ancient times, a cities where first civilization started before Christ. Here according to excavation Christianity started somewhere in the early II-nd century and undergoes а bloom until the XI, XIII century. Here in those places according to some historic datas used to even have an archibiship seat. It is less known fact that Nessebar is one of the most ancient cities in all Europe and started its existence about 3200 years B.C.!
In ancient times before Christianity Mesambria used to be inhabited with Thracians.
On the road to Nesebar we passed through Aheloy (Achelous) – a town where occured the Battle of Achelous (y. 917) which is the biggest battle in medieval European history (120 000 troops participated in battle).

Bulgarians_defeat_the_Byzantines_at_Anchialos_battle_biggest-battle-of-10-th-century-medieval-times

Nesebar is a significant historical city and thus part of UNESCO's world heritage world site. It is divided in new and old city, whether new city's doesn't shiny with architecture, the old town architecture is preserved and absolutely unique. The fact that Nessebar used to be a important Christian center is still evident as even though the town area is situated in peninsula (consisting of only 850 meters width and 350 meters hight), it has 12 Churches !

Nesebar_Church-of-Christ-Pantokrator
Nessebar Church of Christ Pantokrator

Some of the ancient Churches in nessebar are dated from around VI to VIII century. Many of the Churches are in a style specific for Bulgarian empire build in analogy with the  Tarnovo capital of Bulgaria at that time architecture, such architecture is very common for Bulgaria and Byzantine empire in the XIII and XVI centuries. Churches dated from the XIII c. is St. Parascheva, (XIII c.) St. Theodor (XIV c.), St. Archangel Michael and Gabriel (XVI).
По това време са построени църквите “Св. Параскева (XIII в), “Св. Теодор” (XIV в), “Св. Архангели Михаил и Гавраил” (XIV в), имащи преки аналогии в столичната търновска архитектура.
 

ancient-church-in-Nessebar-1

Unfortunately though there are many Churches in Nessebar, many of them are just historical monuments nowadays and others are turned into Painting Galleries. In all ancient city only 1 Orthoox Church – The Dormition of Virgin Mary is functional with regular Holy Liturgies served. The Church has a miracle making of the Theotokos. Many people have found relief and cure or fast help from God after praying in front of the miraculous icon.

miracle-making-holy-icon-of-Virgin-Mary-Nessebar-Bulgaria
Nessebar Miracle Making Icon of Holy Theotokos

To give thanks to Mother Mary many people who received cure or whose prayers came true by praying in front of the miraculous icon deliberately left their gold earings, necklaces and even war medals.
In the Church there are holy relics of number of Christian saints – saint Cyprian and Justina, saint Tatyiana, st. mrtr. Marina, st. Vlasij, st. Macarious

Father Sergij walk us through the city telling a bit of history of each of the Churhes one of the Basilicas was much bigger the rest and fr. Sergius explained that this used to be a Church where a Metropolitan or a Bishop was serving.

We learned that in Nessebar was a very desired king region a place, a highly spiritual place with an overall of 40 Churches!
On the entrance of old nessebar there are remains of the old city fortress walls, the whole city houses and architecture is renessance with a lot of wooden houses.
fortifications_in_entrance_of_Nesebar
Nessebar city entry fortress remains

Nesebar_-_Wooden_Houses
Wooden Houses in Nesebar

Besides the beautiful Churches the sea side is breath taking and from sea shore you can in the distance another resort city Sunny Beach.

nessebar-hills picture
Nessebar winter sea coast view

In Nesebar there are plenty of souvenir shops, caffeterias, small ethnographic styled restaurants assuring a great time for every touries.Very neart to Nessebar is situated also a beautiful rest resort village Ravda.
About 13:00 we left Nessebar and headed back to Pomorie with fr. Sergij who told us a what of Christian faith stories rich in wisdom. On next day Sunday after the end of Holy Liturgy the Abbot of Pomorie Monastery (fr. Ierotey)  baptized Samuel and we had a lunch together with the brotherhood.In early afternoon we  headed back to Sofia. As always father archimandrit Ierotey and fr. Sergius presented us with Christian literature as a gift and a CDs with faith related movies.

I work for money if you want loyalty hire a dog – thinking that can solve your internal workplace (dilemmas) issues :)

Wednesday, February 12th, 2014

I-work-for-money-if-you-want-loyalty-hire-a-dog

I WORK FOR MONEY IF YOU WANT LOYALTY – HIRE A DOG!

I was told about those picture few months ago by a friend (Ivan Pushkaroff – Pushi) while we were drinking beer together in  (Delfincheto – a pub more known among people under the name "Ribkata (The Fish)" – A Metal Music pub in Studentski Grad (Sofia's Student City). By the way if you're in Sofia and near "Studentski Grad" be sure to drop by there – they serve great and always fresh fish also for ex-metal heads for me it is always nice to get some memories about  "the good old metal years"

when I tried to complain how difficult it is for me on my job place. I'm a quite sensitive person and a perfectionist and always try to do a great job. Often however there are many obstacles to do your job in right way (as you depend on so many bureaucracy and external people that even if you do your best someone else can mix it up and screw all your effort by not doing his part of the job. The failure of project or Task due to inadeaquacy of someone else involved to do his job right can leave you with a feeling that you have failed even though it is completely not your fault. As these days I'm quite stressed on my job and I have to work on a lot of badly designed projects (and improperly) assigned tasks – yes in big companies it is like these "the right hand doesn't know what the left one is doing". I remembered my dear friend advice to not take it seriously or personally and decided to put these funny picture in hope that these will help me and others to deal with situation of hardships in job in a humorous way After all I'm just a mercenary and I'm doing my best right now in my job at HP … I work for money if they expect higher degree of loyalty let them hire a dog 🙂

Abe’s Amazing Adventure, A Dangerous Dave 2 like GNU Linux / FreeBSD Arcade Game substitute

Wednesday, December 14th, 2011

Abe's Amazing Adventure Game menu screenshot, A Dangerous Dave like Linux arcade game
While looking over the installable packages in System -> Administration -> Software Center on my my Debian powered notebook, I’ve accidently rolled over a game called Abe’s Amazing Adventure , as a big oldschool arcade game fan, I’ve immediately clicked on Install being impatient to see the overall game atmosphere and gameplay as well as storyline.

Abe’s Amazing Adventure is available under the debian package name abe and can be installed by issuing:

debian:~# apt-get install abe I was pleasently surprised to find out the gameplay and the overall game feels like the golden classical arcade game Dangerous Dave

For youngsters who never played the amazing absolute killing arcade ! as well as for people who feel nostalgic about Dangerous Dave 2 here is a nice gameplay screenshot:

Dangerous Dave 2 screenshot

Here is also a screenshot of Abe’s Amazing Adventure gameplay:

Abes Amazing Adventure Dangerous Dave like Linux / BSD Arcade Game like screenshot

You can see the great resemblense between the two games even by the screenshot; In both games you have to collect keys to open doors and diamonds which gives you points. Of course there are some differences as you can see Abe’s Amazing Adventure ‘s graphics has some additional items like Baloons with which your main character can fly over distances (for a limited time). There is also more diversity in the enemy types you face on the road to unlock the different doors. The game has also a red stars which while taken saves the game so if you quit the game on next game load up starting the game automatically starts you from the last save point.

Abes Amazing Adventure Star gameplay Screenshot on Debian

The ( 6 ) baloons seen in the screenshot on the left upper corner can be used occasionally to fly some distance by pressing Enter .
There are health recovery kits one can take to recover, some health damaged by hitting obstacles on the way.
The game story is short but entertaining, just like the general game feel:

Abes amazing adventure game storyThe game music is relaxing and more or less in the spirit of arcade games, the keyboard interactivety is pretty good as well.One thing I see can be improved to make the game better are the graphics, they’re a bit childish and too cubical, where sometimes some more general characters and obstacles animation will give the game a better look. Anyways for the year 2003 and for a free software arcade the game is not bad. For children and growing kids the game is just perfect, even for big kids like me its good to spend an hour of jump and run fun 😉

W3 total Cache a nice wordpress “all in one” wordpress Caching plugin

Friday, September 23rd, 2011

w3 total cache logo wordpress speed up w3tc

In my recent question of looking for best ways to optimize my wordpress blog, as well as other wordpress based websites, I’m managing I’ve come across a great plugin called W3 Total Cache or W3TC as it’s widely known among wordpress geeks.

The full caching wordpress solution in face of W3TC is already actively deployed among many major wordpress powered websites, to name a few:
stevesouders.com, mattcutts.com, mashable.com

I gave a try W3 Total Cache and was amazed about the rich caching functionality it provides. Having the W3TC plugin installed adds a whole menu on the left wordpress admin panel reading Performance , clicking on it shows a menu with thorough choices on numerous things which deal with the running of wordpress on the server.
Here is a screenshot on the menus W3 Total Cache provides:

w3 total cache whole wordpress caching pages and sql solution

The plugin has support for html caching, sql caching, CDN, CSS and Javascript compress & minify etc.
To test the plugin adequately I disabled HyperCache and Db Cache . My observations is that with W3TC wordpress renders php and the overall user experience and download times in wordpress are better. Therefore I’ll probably use only W3 Total Cache as a cacher for wordpress installs.
Besides that I had some issues on some wordpress based websites with the Db Cache SQL caching.
On these websites after enabling Db Cache, suddenly page edditing of the created existing pages started returning empty pages. This could be due to the fact that these wordpress based websites was customly tailered and many code was wiped out, however it could also be a Db Cache bug. So to conclude W3TC is the perfect solution for wordpress caching 😉

st. Georgi’s day in Pomorie Monastery st. great Martyr Georgi pilgrimage of the relics of saint great martyr George

Saturday, May 7th, 2011

the relics of saint great martyr GeorgePomorie Monastery st. (George) Georgi miraculous craeting icon of saint George

Currently I’m once again in Pomorie’s monastery for the Saint George’s day

It is the second year I take the time to come for a pilgrimage in Pomorie’s Monastery st. great Martyr George (Georgi).<</b>

I’m named after the great Christian saint (It’s my Name day today :)!) and I do wanted to receive a spiritual blessing here in the monastery.

The believing Orthodox Christians this year could receive a spiritual blessing this year from the Monastery as some of the Holy Relics of many great saints are temporary in the monastery for the Monastery’s feast day which is the day we commemorate our great Orthodox Christian martyr saint Georgi.

It’s unique that this year by God’s grace the monastery has temporary received a small particle of the holy relics of st. George.
The pilgrims could come here and does venerate the great saint martyrs for which I thank God for being allowed by his providence to be here in the monastery and venerate the holy relics.

Here are few photos of the Holy relics from the monastery, on the pictures you see the holy relics of Saint Cyprian & St. Justina on the right, st. George’s holy relics in the center and st. Macrina’s holy relics on the left.
Saint Georgi (George) holy relics in Pomorie Monastery Bulgaria

Abbot of Pomorie Monastery Father Ierotei and holy relics of st. Macrina

On the picture above you see the Abbot of Pomorie monastery Father Ierotei (who is a truly great father and a wonderful spirited man !) with the holy relics (bones) of saint Macrina on his left.

On this last picture you can see the holy relics of the Pomorie’s monastery embedded into icons (the two icons on the right).

Pomoriiski Manastir Holy relics

I thank God for blessing me with being here in this holy cloister and blessing me with being able to venerate the holy martyrs relics and through this bless me and strengthen me.!

It’s important to say that we the orthodox Christians had the tradition to venerate our saint holy remains as a way to venerate God himself in accordance to the holy bible psalm which says Great and Glorious is God in his saints!

I’m sorry to provide my readers with this bad quality pictures, I’ll try to put some better quality pictures of the holy remains in very short time.
Being able to show this holy remains to other orthodox christian brother and sisters is also a great grace and blessing which by God’s grace was given to us the sinners! Just to close I’ll say glory be to our God – The Holy Trinity now and forever and ever amen!