Posts Tagged ‘blog’

Improve wordpress admin password encryption authentication keys security with WordPress Unique Authentication Keys and Salts

Friday, October 9th, 2020

Reading Time: 5minutes


Having a wordpress blog or website with an admistrator and access via a Secured SSL channel is common nowadays. However there are plenty of SSL encryption leaks already out there and many of which are either slow to be patched or the hosting companies does not care enough to patch on time the libssl Linux libraries / webserver level. Taking that in consideration many websites hosted on some unmaintained one-time run not-frequently updated Linux servers are still vulneable and it might happen that, if you paid for some shared hosting in the past and someone else besides you hosted the website and forget you even your wordpress installation is still living on one of this SSL vulnerable hosts. In situations like that malicious hackers could break up the SSL security up to some level or even if the SSL is secured use MITM (MAN IN THE MIDDLE) attack to simulate your well secured and trusted SSID Name WIFi network to  redirects the network traffic you use (via an SSL transparent Proxy) to connect to WordPress Administrator Dashbiard via Once your traffic is going through the malicious hax0r even if you haven't used the password to authenticate every time, e.g. you have saved the password in browser and WordPress Admin Panel authentication is achieved via a Cookie the cookies generated and used one time by Woddpress site could be easily stealed one time and later from the vicious 1337 h4x0r and reverse the hash with an interceptor Tool and login to your wordpress …

Therefore to improve the wordpress site security it very important to have configured WordPress Unique Authentication Keys and Salts (known also as the WordPress security keys).

They're used by WordPress installation to have a uniquely generated different key and Salt from the default one to the opened WordPress Blog / Site Admin session every time.

So what are the Authentication Unique Keys and Salts and why they are Used?

Like with almost any other web application, when PHP session is opened to WordPress, the code creates a number of Cookies stored locally on your computer.

Two of the cookies created are called:


First  cookie is used only in the admin pages (WordPress dashboard), while the second cookie is used throughout WordPress to determine if you are logged in to WordPress or not. Note: [hash] is a random hashed value typically assigned to your session, therefore in reality the cookies name would be named something like wordpress_ffc02f68bc9926448e9222893b6c29a9.

WordPress session stores your authentication details (i.e. WordPress username and password) in both of the above mentioned cookies.

The authentication details are hashed, hence it is almost impossible for anyone to reverse the hash and guess your password through a cookie should it be stolen. By almost impossible it also means that with today’s computers it is practically unfeasible to do so.

WordPress security keys are made up of four authentication keys and four hashing salts (random generated data) that when used together they add an extra layer to your cookies and passwords. 

The authentication details in these cookies are hashed using the random pattern specified in the WordPress security keys. I will not get into too much details but as you might have heard in Cryptography Salts and Keys are important – an indepth explanation on Salts Cryptography (here). A good reading for those who want to know more on how does the authentication based and salts work is on stackexchange.

How to Set up Salt and Key Authentication on WordPress

To be used by WP Salts and Key should be configured under wp-config.php usually they look like so:


!!! Note !!!  that generating (manually or generated via a random generator program), the definition strings you have to use a random string value of more than 60 characters to prevent predictability 

The default on any newly installed WordPress Website is to have the 4 definitions with _KEY and the four _SALTs to be unconfigured strings looks something like:


Most people never ever take a look at wp-config.php as only the Web GUI Is used for any maintainance, tasks so there is a great chance that if you never heard specifically by some WordPress Security Expert forum or some Security plugin (such as WP Titan Anti Spam & Security) installed to report the WP KEY / SALT you might have never noticed it in the config.

There are 8 WordPress security keys in current WP Installs, but not all of them have been introduced at the same time.
Historically they were introduced in WP versions in below order:

WordPress 2.7: NONCE_KEY

Setting a custom random generated values is an easy task as there is already online Wordpress Security key Random generator.
You can visit above address and you will get an automatic randomly generated values which could be straight copy / pasted to your wp-config.php.

Howeever if you're a paranoic on the guessability of the random generator algorithm, I would advice you use the generator and change some random values yourself on each of the 8 line, the end result in the configuration should be something similar to:


define('AUTH_KEY',         '|w+=W(od$V|^hy$F5w)g6O-:e[WI=NHY/!Ez@grd5=##!;jHle_vFPqz}D5|+87Q');
define('SECURE_AUTH_KEY',  'rGReh.<%QBJ{DP )p=BfYmp6fHmIG~ePeHC[MtDxZiZD;;_OMp`sVcKH:JAqe$dA');
define('LOGGED_IN_KEY',    '%v8mQ!)jYvzG(eCt>)bdr+Rpy5@t fTm5fb:o?@aVzDQw8T[w+aoQ{g0ZW`7F-44');
define('NONCE_KEY',        '$o9FfF{S@Z-(/F-.6fC/}+K 6-?V.XG#MU^s?4Z,4vQ)/~-[D.X0<+ly0W9L3,Pj');
define('AUTH_SALT',        ':]/2K1j(4I:DPJ`(,rK!qYt_~n8uSf>=4`{?LC]%%KWm6@j|aht@R.i*ZfgS4lsj');
define('SECURE_AUTH_SALT', 'XY{~:{P&P0Vw6^i44Op*nDeXd.Ec+|c=S~BYcH!^j39VNr#&FK~wq.3wZle_?oq-');
define('LOGGED_IN_SALT',   '8D|2+uKX;F!v~8-Va20=*d3nb#4|-fv0$ND~s=7>N|/-2]rk@F`DKVoh5Y5i,w*K');
define('NONCE_SALT',       'ho[<2C~z/:{ocwD{T-w+!+r2394xasz*N-V;_>AWDUaPEh`V4KO1,h&+c>c?jC$H');



Once above defines are set, do not forget to comment or remove old AUTH_KEY / SECURE_AUTH_KEY / LOGGED_IN_KEY / AUTH_SALT / SECURE_AUTH_SALT / LOGGED_IN_SALT /NONCE_SALT keys.

The values are configured one time and never have to be changed, WordPress installation automatic updates or Installed WP Plugins will not tamper the value with time.
You should never expand or show your private generated keys to anyone otherwise this could be used to hack your website site.
It is also a good security practice to change this keys, especially if you have some suspects someone has somehow stolen your wp-onfig keys. 


Having AUTH KEYs and Properly configured is essential step to improve your WordPress site security. Anytime having any doubt for a browser hijacked session (or if you have logged in) to your /wp-admin via unsecured public Computer with a chance of a stolen site cookies you should reset keys / salts to a new random values. Setting the auth keys is not a panacea and frequent WP site core updates and plugins should be made to secure your install. Always do frequent audits to WP owned websites with a tool such as WPScan is essential to keep your WP Website unhacked.



Fix X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied – Finally a working solution and what causes the error

Wednesday, July 22nd, 2015

Reading Time: 7minutes


I've lost a whole day and was angry and irritated after moving (migrating) a Working Qmail installation in a binary form from a Debian Lenny 5.0 to a Debian 7.0 Wheezy Linux. The whole migration exercise was quite of a crazy move and I can tell you it didn't worth the effort as I lost much more time than even if I went on installing the server following Thibs Great Qmail Tutorial.

Yes I know many would say why do you still bother with an old and unsupported Qmail Mail server and not go with Postfix, the logic is correct however the whole issue is the previous installation has a number of domains already running VirtualMail hosting using VPopMail, so migrating all the old mailboxes from Qmail to Postfix are not worthy IMHO. Plus I honestly love qmail for being so stable even today (even without support). After all most of Qmail is secure enough already 🙂
And to be honest I don't so much care about security as in the old days as I know NSA, does already have access to any server on planet 🙂

Almost always when a Qmail migration happens I end up swearing and sweating and generally getting crazy, but anyways …

The overall migration of binaries went quite OK I just copied every binary and all the related libraries from the old Debian 5.0 to Debian 7.0 and installed the following long list of perl deb binaries using apt-get:



I've also installed with apt-get daemontools and daemontools-run ucspi and some others to get all the necessery binaries qmail needs the whole list of apt installed packages is here

I've also copied all the old binaries from /usr/local/lib from server1 to server2, some others from /usr/local/share and /usr/share as well as /usr/lib/courier/usr/lib/courier-authlib /usr/local/libexec /usr/local/sbin also had to link a couple of libraries such as /usr/lib/libcrypto* , link  /usr/lib/ to /usr/lib/ and copy /usr/lib/ and few others which i don't exactly remember.

Well anyways once I've copied everything Qmail looked fined except I had a couple of permission issues and had to clean up and fix the queue with qfixq, I've also used qmail-scanner*/contrib/ script to test whether qmail-scanner was running fine, e.g.:

./ -doit

As well as




script, thanks to which I've captured and resolved few of permission problems 

Finally I've stuck upon this shitty errors (appearing) in /var/log/syslog and /var/log/messages



Jul 21 22:04:19 vps186637 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:27 vps186637 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:38 vps186637 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:11:17 vps186637 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:16:09 vps186637 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:19:15 vps186637 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:38:59 ns2 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:42:33 ns2 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:43:49 ns2 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:46:05 ns2 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:50:40 ns2 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:53:08 ns2 X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied




There is plenty of things written about this: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied

But all written is too obscure and too old already somewhere between 2004 and 2010, I've been digging through Gentoo Forums, Fedora Debian and other Linux installs and everyone used to be pointing a permission issue with /var/spool/qscan/ said theoretically to be causing the error, however all looked perfectly fine with my /var/spool/qscan , e.g.:



roo@ns2:/usr/local/src# ls -ld /var/spool/qscan/
drwxr-s— 6 qscand qscand 4096 Jul 21 23:07 /var/spool/qscan/





ls -al /var/spool/qscan/
total 244904
drwxr-s— 6 qscand qscand      4096 Jul 21 23:07 .
drwxr-xr-x 4 root   root        4096 Jul 20 21:17 ..
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 archives
-rwxr-x— 1 qscand qscand      1434 Oct 12  2011
-rw——- 1 qscand qscand 249731919 Jul 21 23:11 qmail-queue.log
-rw——- 1 qscand qscand    398225 Oct 28  2011 qmail-queue.log.1
-rw-rw—- 1 root   qscand        74 Jul 21 23:07 qmail-scanner-queue-version.txt
lrwxrwxrwx 1 root   qscand        16 Jul 21 23:07 qscan -> /var/spool/qscan
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 quarantine
-rw-r—– 1 root   qscand     12288 Jul 21 23:07 quarantine-events.db
-rw-r—– 1 qscand qscand     10443 Oct 12  2011 quarantine-events.txt
-rw-rw—- 1 qscand qscand    580033 Jul 21 23:07 quarantine.log
-rw-r—– 1 qscand qscand      2739 Oct 12  2011 settings_per_domain.txt
drwxr-x— 3 qscand qscand      4096 Jul 21 23:11 tmp
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 working


Some suggested that /var/spool/qscan should be owned by qscand:clamav instead so I tried this but it didn't help,
others recommended adding clamav groupid into qscand's but this didn't help either:

root@ns2:/usr/local/src# grep -i clamav /etc/group


Besides that I was getting also this shitty error:


Jul 21 20:05:42 vps186637 X-Qmail-Scanner-2.08st:[ns2143750194279012466] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:06:51 vps186637 X-Qmail-Scanner-2.08st:[ns2143750201179013125] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:15:42 vps186637 X-Qmail-Scanner-2.08st:[ns214375025407906015] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:16:06 vps186637 X-Qmail-Scanner-2.08st:[ns2143750256579011980] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:18:54 vps186637 X-Qmail-Scanner-2.08st:[ns2143750273479014847] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:21:03 vps186637 X-Qmail-Scanner-2.08st:[ns2143750286379028491] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 22:07:47 vps186637 X-Qmail-Scanner-2.08st:[ns2143750926779010097] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status 512/2


All the time in logs, so I thought something might be wrong with clamdscan and followed and tried some suggestions from as described here, however none of the fixes described there worked for me …

I also tried to reinstall clamav from source using a bit of modified version of this tutorial

This didn't help either … I saw some suggestions online that the permission issues are caused by some wrong clamd.conf and freshclamd.conf configuration options – failing clamdscan, but this didn't work either. I also tried to remove clamdscan and substitute it with clamscan  as a suggested workaround but this didn't work either …

I spend about 6 hours trying to catch what is causing this issues so finally I went on and re-installed bigger part of Qmail using Thibs tutorial over the old installation.

I've also tried in mean time multiple time to rebuild qmail scanner database with:

setuidgid qscand /var/qmail/bin/ -g

Played around with permissions in /etc/clamav e.g.



chown -R qscand:clamav /var/log/clamav /var/lib/clamav /var/run/clamav
chown qscand:qscand /etc/clamav/freshclam.conf





/etc/cron.daily/qmail-scanner with following content


cat /etc/cron.daily/qmail-scanner
setuidgid qscand /var/qmail/bin/ -z

However in /var/log/qmail/qmail-smtpd/current and /var/log/qmail-send/current I continously got:

Qmail 451 qq temporary problem (#4.3.0) error

Interestingly during looking for a solution to the 451 qq temporary problem and:

status: qmail-todo stop processing asap / status: exiting – I've stumbled to my own blog post here 🙂


Finally, I tried to reinstall qmail-scanner and in mean time update it to  Version: 2.11 – st – patch – 20130319
Just to realized something was wrong with suidperl, e.g. in Debian 7.0.* Wheezy perl-suid binary is no longer in repositories so only way to have suidperl there is either to re-compile perl from source manually which is too much work and I think in most cases not worthy the effort or to use a small suid-wrapper:

#include <unistd.h>
#include <errno.h>

main( int argc, char ** argv, char ** envp )
              if( setgid(getegid()) ) perror( "setgid" );
              if( setuid(geteuid()) ) perror( "setuid" );
              envp = 0; /* blocks IFS attack on non-bash shells */
              system( "/usr/bin/perl", argv, envp );
              perror( argv[0] );
              return errno;


Create it into a file lets say suid-wrapperc and compile the file with GNU C Compiler:

$ gcc -o suid-wrapper suid-wrapper.c

Then move the suid-wrapper produced binary into /usr/bin/suidperl e.g.



$ mv suid-wrapper /usr/bin/suidperl

Last you will need to edit /var/qmail/bin/





# vim /var/qmail/bin/

And substitute





#!/usr/bin/perl -T





Note!!! that permissions should be suid and owned by qscand:qscand as follows:





ls -al /var/qmail/bin/
-rwsr-sr-x 1 qscand qscand 159727 Jul 21 23:11 /var/qmail/bin/

Finally to resolve the error I had to restart qmail via qmailctl start / stop script:





root@ns2:/var/qmail/bin# qmailctl restart
Restarting qmail:
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.


Finally to test emails are sent and receiver properly I used good old mail command part of bsd-mailx deb package


# mail -s "testing 12345678"



I've also tested with plain telnet to verify no errors because often the mail command doesn't return (show) errors on email sent and errors are written only in /var/log/mail.log or /var/log/qmail/* logs



# telnet localhost 25
Connected to localhost.
Escape character is '^]'.
220 servername.localdomain.tld ESMTP
mail from: <>
250 ok
rcpt to: <>
250 ok
354 go ahead
From: Test_sender <>
To: Test_receiver <>
Subject: Just a stupid SMTP test


Just a test !
250 ok 1279384489 qp 3711
221 servername.localdomain.tld
Connection closed by foreign host.

One other thing which probably helped I did was:

# qmailctl doqueue


Thanks God this time, it worked out without any QQ errors 🙂 !


Finding top access IPs in Webserver or how to delay connects from Bots (Web Spiders) to your site to prevent connect Denial of Service

Friday, September 15th, 2017

Reading Time: 8minutes


If you're a sysadmin who has to deal with cracker attemps for DoS (Denial of Service) on single or multiple servers (clustered CDN or standalone) Apache Webservers, nomatter whether working for some web hosting company or just running your private run home brew web server its very useful thing to inspect Web Server log file (in Apache HTTPD case that's access.log).

Sometimes Web Server overloads and the follow up Danial of Service (DoS) affect is not caused by evil crackers (mistkenly often called hackers but by some data indexing Crawler Search Engine bots who are badly configured to aggressively crawl websites and hence causing high webserver loads flooding your servers with bad 404 or 400, 500 or other requests, just to give you an example of such obstructive bots.

1. Dealing with bad Search Indexer Bots (Spiders) with robots.txt

Just as I mentioned hackers word above I feel obliged to expose the badful lies the press and media spreading for years misconcepting in people's mind the word cracker (computer intruder) with a hacker, if you're one of those who mistakenly call security intruders hackers I recommend you read Dr. Richard Stallman's article On Hacking to get the proper understanding that hacker is an cheerful attitude of mind and spirit and a hacker could be anyone who has this kind of curious and playful mind out there. Very often hackers are computer professional, though many times they're skillful programmers, a hacker is tending to do things in a very undstandard and weird ways to make fun out of life but definitelely follow the rule of do no harm to the neighbor.

Well after the short lirical distraction above, let me continue;

Here is a short list of Search Index Crawler bots with very aggressive behaviour towards websites:


# mass download bots / mirroring utilities
1. webzip
2. webmirror
3. webcopy
4. netants
5. getright
6. wget
7. webcapture
8. libwww-perl
11. Teleport / TeleportPro
12. Zeus

Note that some of the listed crawler bots are actually a mirroring clients tools (wget) etc., they're also included in the list of server hammering bots because often  websites are attempted to be mirrored by people who want to mirror content for the sake of good but perhaps these days more often mirror (duplicate) your content for the sake of stealing, this is called in Web language Content Stealing in SEO language.

I've found a very comprehensive list of Bad Bots to block on Mike's tech blog his website provided example of bad robots.txt file is mirrored as plain text file here

Below is the list of Bad Crawler Spiders taken from his site:


# robots.txt to prohibit bad internet search engine spiders to crawl your website
# Begin block Bad-Robots from robots.txt
User-agent: asterias
User-agent: BackDoorBot/1.0
User-agent: Black Hole
User-agent: BlowFish/1.0
User-agent: BotALot
User-agent: BuiltBotTough
User-agent: Bullseye/1.0
User-agent: BunnySlippers
User-agent: Cegbfeieh
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CherryPickerElite/1.0
User-agent: CherryPickerSE/1.0
User-agent: CopyRightCheck
User-agent: cosmos
User-agent: Crescent
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
User-agent: DittoSpyder
User-agent: EmailCollector
User-agent: EmailSiphon
User-agent: EmailWolf
User-agent: EroCrawler
User-agent: ExtractorPro
User-agent: Foobot
User-agent: Harvest/1.5
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: InfoNaviRobot
User-agent: JennyBot
User-agent: Kenjin Spider
User-agent: Keyword Density/0.9
User-agent: LexiBot
User-agent: libWeb/clsHTTP
User-agent: LinkextractorPro
User-agent: LinkScan/8.1a Unix
User-agent: LinkWalker
User-agent: LNSpiderguy
User-agent: lwp-trivial
User-agent: lwp-trivial/1.34
User-agent: Mata Hari
User-agent: Microsoft URL Control – 5.01.4511
User-agent: Microsoft URL Control – 6.00.8169
User-agent: MIIxpc
User-agent: MIIxpc/4.2
User-agent: Mister PiX
User-agent: moget
User-agent: moget/2.1
User-agent: mozilla/4
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
User-agent: mozilla/5
User-agent: NetAnts
User-agent: NICErsPRO
User-agent: Offline Explorer
User-agent: Openfind
User-agent: Openfind data gathere
User-agent: ProPowerBot/2.14
User-agent: ProWebWalker
User-agent: QueryN Metasearch
User-agent: RepoMonkey
User-agent: RepoMonkey Bait & Tackle/v1.01
User-agent: RMA
User-agent: SiteSnagger
User-agent: SpankBot
User-agent: spanner
User-agent: suzuran
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: TeleportPro
User-agent: Telesoft
User-agent: The Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: toCrawl/UrlDispatcher
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: turingos
User-agent: URLy Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: Web Image Collector
User-agent: WebAuto
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCopier
User-agent: WebEnhancer
User-agent: WebmasterWorldForumBot
User-agent: WebSauger
User-agent: Website Quester
User-agent: Webster Pro
User-agent: WebStripper
User-agent: WebZip
User-agent: WebZip/4.0
User-agent: Wget
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: WWW-Collector-E
User-agent: Xenu’s
User-agent: Xenu’s Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Crawl-delay: 20
# Begin Exclusion From Directories from robots.txt
Disallow: /cgi-bin/

Veryimportant variable among the ones passed by above robots.txt is

Crawl-Delay: 20


You might want to tune that variable a Crawl-Delay of 20 instructs all IP connects from any Web Spiders that are respecting robots.txt variables to delay crawling with 20 seconds between each and every connect client request, that is really useful for the Webserver as less connects means less CPU and Memory usage and less degraded performance put by aggressive bots crawling your site like crazy, requesting resources 10 times per second or so …

As you can conclude by the naming of some of the bots having them disabled would prevent your domain/s clients from Email harvesting Spiders and other not desired activities.


2. Listing IP addresses Hits / How many connects per IPs used to determine problematic server overloading a huge number of IPs connects

After saying few words about SE bots and I think it it is fair to also  mention here a number of commands, that helps the sysadmin to inspect Apache's access.log files.
Inspecting the log files regularly is really useful as the number of malicious Spider Bots and the Cracker users tends to be
raising with time, so having a good way to track the IPs that are stoning at your webserver and later prohibiting them softly to crawl either via robots.txt (not all of the Bots would respect that) or .htaccess file or as a last resort directly form firewall is really useful to know.

– Below command Generate a list of IPs showing how many times of the IPs connected the webserver (bear in mind that commands are designed log fields order as given by most GNU / Linux distribution + Apache default logging configuration;


webhosting-server:~# cd /var/log/apache2 webhosting-server:/var/log/apache2# cat access.log| awk '{print $1}' | sort | uniq -c |sort -n

Below command provides statistics info based on whole access.log file records, sometimes you will need to have analyzed just a chunk of the webserver log, lets say last 12000 IP connects, here is how:

webhosting-server:~# cd /var/log/apache2 webhosting-server:/var/log/apache2# tail -n 12000 access.log| awk '{print $1}' | sort | uniq -c |sort -n

You can combine above basic bash shell parser commands with the watch command to have a top like refresh statistics every few updated refreshing IP statistics of most active customers on your websites.

Here is an example:


webhosting-server:~# watch "cat access.log| awk '{print $1}' | sort | uniq -c |sort -n";


Once you have the top connect IPs if you have a some IP connecting with lets say 8000-10000 thousand times in a really short interval of time 20-30 minues or so. Hence it is a good idea to investigate further where is this IP originating from and if it is some malicious Denial of Service, filter it out either in Firewall (with iptables rules) or ask your ISP or webhosting to do you a favour and drop all the incoming traffic from that IP.

Here is how to investigate a bit more about a server stoner IP;
Lets assume that you found IP: to be having too many connects to your webserver:

webhosting-server:~# grep -i /var/log/apache2/access.log|tail -n 1 – – [12/Sep/2017:07:42:13 +0300] "GET / HTTP/1.1" 403 371 "-" "Mozilla/5.0 (compatible;; +"


webhosting-server:~# host domain name pointer


webhosting-server:~# whois|less


The outout you will get would be something like:

% This is the RIPE Database query service.
% The objects are in RPSL format.
% The RIPE Database is subject to Terms and Conditions.
% See

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to ' –'

% Abuse contact for ' –' is ''

inetnum: –
netname:        HETZNER-RZ15
descr:          Hetzner Online GmbH
descr:          Datacenter 15
country:        DE
admin-c:        HOAC1-RIPE
tech-c:         HOAC1-RIPE
status:         ASSIGNED PA
mnt-by:         HOS-GUN
mnt-lower:      HOS-GUN
mnt-routes:     HOS-GUN
created:        2012-03-12T09:45:54Z
last-modified:  2015-08-10T09:29:53Z
source:         RIPE

role:           Hetzner Online GmbH – Contact Role
address:        Hetzner Online GmbH
address:        Industriestrasse 25
address:        D-91710 Gunzenhausen
address:        Germany
phone:          +49 9831 505-0
fax-no:         +49 9831 505-3
remarks:        *************************************************
remarks:        * For spam/abuse/security issues please contact *
remarks:        *, not this address. *
remarks:        * The contents of your abuse email will be *
remarks:        * forwarded directly on to our client for *

3. Generate list of directories and files that are most called by clients

webhosting-server:~# cd /var/log/apache2; webhosting-server:/var/log/apache2# awk '{print $7}' access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10

( take in consideration that this info is provided only on current records from /var/log/apache2/ and is short term for long term statistics you have to merge all existing gzipped /var/log/apache2/access.log.*.gz )

To merge all the old gzipped files into one single file and later use above shown command to analyize run:


cd /var/log/apache2/
cp -rpf *access.log*.gz apache-gzipped/
cd apache-gzipped
for i in $(ls -1 *access*.log.*.gz); do gzip -d $i; done
rm -f *.log.gz;
for i in $(ls -1 *|grep -v access_log_complete); do cat $i >> access_log_complete; done

Though the accent of above article is Apache Webserver log analyzing, the given command examples can easily be recrafted to work properly on other Web Servers LigHTTPD, Nginx etc.

Above commands are about to put a higher load to your server during execution, so on busy servers it is a better idea, to first go and synchronize the access.log files to another less loaded servers in most small and midsized companies this is being done by a periodic synchronization of the logs to the log server used usually only to store log various filesand later used to do various analysis our run analyse software such as Awstats, Webalizer, Piwik, Go Access etc.

Worthy to mention one great text console must have Apache tool that should be mentioned to analyze in real time for the lazy ones to type so much isApache-top but those script will be not installed on most webhosting servers and VPS-es, so if you don't happen to own a self-hosted dedicated server / have webhosting company etc. – (have root admin access on server), but have an ordinary server account you can use above commands to get an overall picture of abusive webserver IPs.


If you have a Linux with a desktop GUI environment and have somehow mounted remotely the weblog server partition another really awesome way to visualize in real time the connect requests to  web server Apache / Nginx etc. is with Logstalgia

Well that's all folks, I hope that article learned you something new. Enjoy

Thanks for article neo-tux picture

How to show country flag, web browser type and Operating System in WordPress Comments

Wednesday, February 15th, 2012

Reading Time: 2minutes



I've come across a nice WordPress plugin that displays country flag, operating system and web browser used in each of posted comments blog comments.
Its really nice plugin, since it adds some transperancy and colorfulness to each of blog comments 😉
here is a screenshot of my blog with Comments Info Detector "in action":

Example of Comments Info Detector in Action on wordpress blog comments

Comments Info Detector as of time of writting is at stable ver 1.0.5.
The plugin installation and configuration is very easy as with most other WP plugins. To install the plugin;

1. Download and unzip Comments Info Detector

linux:/var/www/blog:# cd wp-content/plugins
linux:/var/www/blog/wp-content/plugins:# wget
linux:/var/www/blog/wp-content/plugins:# unzip

Just for the sake of preservation of history, I've made a mirror of comments-info-detector 1.0.5 wp pluginfor download here
2. Activate Comment-Info-Detector

To enable the plugin Navigate to;
Plugins -> Inactive -> Comment Info Detector (Activate)

After having enabled the plugin as a last 3rd step it has to be configured.

3. Configure comment-info-detector wp plugin

By default the plugin is disabled. To change it to enabled (configure it) by navigating to:

Settings -> Comments Info Detector

Next a a page will appear with variout fields and web forms, where stuff can be changed. Here almost all of it should be left as it is the only change should be in the drop down menus near the end of the page:

Display Country Flags Automatically (Change No to Yes)
Display Web Browsers and OS Automatically (Change No to Yes

Comments Info Detector WordPress plugin configuration Screenshot

After the two menus are set to "Yes" and pressing on Save Changes the plugin is enabled it will immediately start showing information inside each comment the GeoIP country location flag of the person who commented as well as OS type and Web Browser 🙂

No space left on device with free disk space / Why no space left on device while there is plenty of disk space on drive – Running out of Inodes

Tuesday, November 17th, 2015

Reading Time: 5minutes



On one of the servers, I'm administrating the websites started showing some Mysql database table corrup errors like:


Table './database_name/site_news_list_com' is marked as crashed and last (automatic?) repair failed

The server is using Oracle MySQL server community stable edition on Debian GNU / Linux 6.0, so I first thought during work the server crashed either due to some bug issue in MySQL or it crashed due to some PHP cron job that did something messy. Thus to solve the crashed tables, tried using mysqlcheck tool which helped pretty fine, at many times whether there were database / table corruptions. I've run the following set of mysqlcheck commands with root (superuser) in a bash shell after logging in through SSH:


server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log
server:~# /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

In order for above commands to work, I've created the /root/.my.cnf containing my root (mysql CLI) mysql username and password, e.g. file has content like below:




Btw a good note here is its generally a good idea (if you want to have consistent mysql databases) to automatically execute via a cron job 2 times a month, I've in root cronjob the following:


crontab -u root -l |grep -i mysqlcheck
04 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 07 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 12 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log 17 06 5,10,15,20,25,1 * * /usr/bin/mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases –silent -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

Strangely I got a lot of errors that some .MYI / .MYD .frm temp files, necessery for the mysql tables recovery can't be written inside /home/mysql/database_name

That was pretty weird and I thought there might be some issues with permissions, causing the inability to write, due to some bug or something so I went straight and checked /home/mysql/database_name permissions, e.g.::


server:/home/mysql/database_name# ls -ld soccerfame
drwx—— 2 mysql mysql 36864 Nov 17 12:00 soccerfame
server:/home/mysql/database_name# ls -al1|head -n 10
total 1979012
drwx—— 2 mysql mysql 36864 Nov 17 12:00 .
drwx—— 36 mysql mysql 4096 Nov 17 11:12 ..
-rw-rw—- 1 mysql mysql 8712 Nov 17 10:26 1_campaigns_diez.frm
-rw-rw—- 1 mysql mysql 14672 Jul 8 18:57 1_campaigns_diez.MYD
-rw-rw—- 1 mysql mysql 1024 Nov 17 11:38 1_campaigns_diez.MYI
-rw-rw—- 1 mysql mysql 8938 Nov 17 10:26 1_campaigns.frm
-rw-rw—- 1 mysql mysql 8738 Nov 17 10:26 1_campaigns_logs.frm
-rw-rw—- 1 mysql mysql 883404 Nov 16 22:01 1_campaigns_logs.MYD
-rw-rw—- 1 mysql mysql 330752 Nov 17 11:38 1_campaigns_logs.MYI

As seen from above output, all was perfect with permissions, so it should have been something else, so I decided to try to create a random file with touch command inside /home/mysql/database_name directory:


touch /home/mysql/database_name/somefile-to-test-writtability.txt touch: cannot touch ‘/scr1/data/somefile-to-test-writtability.txt‘: No space left on device

Then logically I thought the /home/mysql/ mounted ext4 partition got filled, because of crashed SQL database or a bug thus, checked with disk free command df whether there is enough space on server:

server:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1 20G 7.6G 11G 42% /
udev 10M 0 10M 0% /dev
tmpfs 13G 1.3G 12G 10% /run
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/md2 256G 134G 110G 55% /home

Well that's weird? Obviously only 55% of available disk space is used and available 134G which was more than enough so I got totally puzzled why, files can't be written.

Then very logically, I thought it might be that /home directory has remounted asread only, because the SSD memory disk on server is failing and checked for errors in dmesg, i.e.:


server:~# dmesg|grep -i error

Also checked how exactly was partition mounted, to check whether it is (RO) read-only:


server:~# mount -l|grep -i /home
/dev/md2 on /home type ext4 (rw,relatime,discard,data=ordered)

Now everything become even more weirder, as obviously the disk continued to be claiming no space left on device, while in reality there was plenty of disk space.

Then after running a quick research on the internet for the no space left on device with free disk space, I've come across this great thread which let me realize the partition run out of inodes and that's why no new file inodes could be assigned and therefore, the linux kernel is refusing to write the file on ext4 partition.

For those who haven't heard of Linux Partition Inodes here is link to Wikipedia and a quick quote:


In a Unix-style file system, the inode is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. Each inode stores the attributes and disk block location(s) of the filesystem object's data.[1] Filesystem object attributes may include manipulation metadata (e.g. change,[2] access, modify time), as well as owner and permission data (e.g. group-id, user-id, permissions).[3]
Directories are lists of names assigned to inodes. The directory contains an entry for itself, its parent, and each of its children.

Once I understood it is the inodes, I checked how many of them are occupied with cmd:


server:~# df -i /home
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 17006592 0 100% /home

You see, there were 0 (zero) free file inodes on server and that was the reason for no space left on device while there was actually free disk space

To clean up (free) some inodes on partition, first thing I did is to delete all old logs which were inside /home and files I positively know not to be necessery, then to find which directories allocating most innodes used:


server:~# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

If you're on a regular old fashined IDE Hard Drive and not SSD or you have too much files inside this command will take really long …:

Therefore a better solution might be to frist:

a) Try to find root folders with large inodes count:

for i in /home/*; do echo $i; find $i |wc -l; done
Try to find specific folders:

You should get output like:



b) Then once you know the directory allocating most inodes, run the command again to see the sub-directories with most files (eating) partition innodes:


for i in /home/webservice/*; do echo $i; find $i |wc -l; done


One usual large folder which could free you some nodes is the linux source headers, but in my case it was simply a lot of tiny old logs being logged on the system for few years in the past without cleaning:

After deleting the log dirs and cache folder in my case /home/new_website/{log,cache}:

server:~# rm -rf /home/new_website/log/*
server:~# rm -rf /home/new_website/cache/*



a) Then, stopping Apache webserver to check prevent Apache to use MySQl databases while running database repair and restaring MySQL:

server:~# /etc/init.d/apache2 stop Restarting MySQL server
server:~# /etc/init.d/mysql restart

b) And re-issuing MySQL Check / Repair / Optimize database commands:


mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–check –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf –analyze –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–auto-repair –optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

mysqlcheck –defaults-extra-file=/etc/mysql/debian.cnf \–optimize –all-databases -u root -p`grep -i password /root/.my.cnf |sed -e 's#password=##g'`>> /var/log/cronwork.log

c) And finally starting the Apache Webserver again:

server:~# /etc/init.d/apache2 start

Some innodse got freed up:

server:~# df -i /home Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md2 17006592 16797196 209396 99% /home

And hooray by God's Grace and with help of prayers of The most Holy Theotokos (Virgin) Mary, websites started again !

Fix MySQL ibdata file size – ibdata1 file growing too large, preventing ibdata1 from eating all your server disk space

Thursday, April 2nd, 2015

Reading Time: 4minutes


If you're a webhosting company hosting dozens of various websites that use MySQL with InnoDB  engine as a backend you've probably already experienced the annoying problem of MySQL's ibdata1 growing too large / eating all server's disk space and triggering disk space low alerts. Theibdata1 file, taking up hundreds of gigabytes is likely to be encountered on virtually all Linux distributions which run default MySQL server <= MySQL 5.6 (with default distro shipped my.cnf). The excremental ibdata1 raise appears usually due to a application software bug on how it queries the database. In theory there are no limitation for ibdata1 except maximum file size limitation set for the filesystem (and there is no limitation option set in my.cnf) meaning it is quite possible that under certain conditionsibdata1 grow over time can happily fill up your server LVM (Storage) drive partitions.

Unfortunately there is no way to shrink the ibdata1 file and only known work around (I found) is to set innodb_file_per_table option in my.cnf to force the MySQL server create separate *.ibd files under datadir (my.cnf variable) for each freshly created InnoDB table.

1. Checking size of ibdata1 file

On Debian / Ubuntu and other deb based Linux servers datadir is /var/lib/mysql/ibdata1

server:~# du -hsc /var/lib/mysql/ibdata1
45G     /var/lib/mysql/ibdata1
45G     total

2. Checking info about Databases and Innodb storage Engine

server:~# mysql -u root -p

| Database           |
| information_schema |
| bible              |
| blog               |
| blog-sezoni        |
| blogmonastery      |
| daniel             |
| ezmlm              |
| flash-games        |

Next step is to get some understanding about how many existing InnoDB tables are present within Database server:


mysql> SELECT COUNT(1) EngineCount,engine FROM information_schema.tables WHERE table_schema NOT IN ('information_schema','performance_schema','mysql') GROUP BY engine;
| EngineCount | engine |
|         131 | InnoDB |
|           5 | MEMORY |
|         584 | MyISAM |
3 rows in set (0.02 sec)

To get some more statistics related to InnoDb variables set on the SQL server:

mysqladmin -u root -p'Your-Server-Password' var | grep innodb

Here is also how to find which tables use InnoDb Engine

mysql> SELECT table_schema, table_name
    -> WHERE engine = 'innodb';

| table_schema | table_name               |
| blog         | wp_blc_filters           |
| blog         | wp_blc_instances         |
| blog         | wp_blc_links             |
| blog         | wp_blc_synch             |
| blog         | wp_likes                 |
| blog         | wp_wpx_logs              |
| blog-sezoni  | wp_likes                 |
| icanga_web   | cronk                    |
| icanga_web   | cronk_category           |
| icanga_web   | cronk_category_cronk     |
| icanga_web   | cronk_principal_category |
| icanga_web   | cronk_principal_cronk    |

3. Check and Stop any Web / Mail / DNS service using MySQL

server:~# ps -efl |grep -E 'apache|nginx|dovecot|bind|radius|postfix'

Below cmd should return empty output, (e.g. Apache / Nginx / Postfix / Radius / Dovecot / DNS etc. services are properly stopped on server).

4. Create Backup dump all MySQL tables with mysqldump

Next step is to create full backup dump of all current MySQL databases (with mysqladmin):

server:~# mysqldump –opt –allow-keywords –add-drop-table –all-databases –events -u root -p > dump.sql
server:~# du -hsc /root/dump.sql
940M    dump.sql
940M    total


If you have free space on an external backup server or remotely mounted attached (NFS or SAN Storage) it is a good idea to make a full binary copy of MySQL data (just in case something wents wrong with above binary dump), copy respective directory depending on the Linux distro and install location of SQL binary files set (in my.cnf).
To check where are MySQL binary stored database data (check in my.cnf):

server:~# grep -i datadir /etc/mysql/my.cnf
datadir         = /var/lib/mysql

If server is CentOS / RHEL Fedora RPM based substitute in above grep cmd line /etc/mysql/my.cnf with /etc/my.cnf

if you're on Debian / Ubuntu:

server:~# /etc/init.d/mysql stop
server:~# cp -rpfv /var/lib/mysql /root/mysql-data-backup

Once above copy completes, DROP all all databases except, mysql, information_schema (which store MySQL existing user / passwords and Access Grants and Host Permissions)

5. Drop All databases except mysql and information_schema

server:~# mysql -u root -p



DROP DATABASE wordpress;
DROP DATABASE micropcfreak;
DROP DATABASE statusnet;

          etc. etc.

ACHTUNG !!!DON'T execute!DROP database mysql; DROP database information_schema; !!! – cause this might damage your User permissions to databases

6. Stop MySQL server and add innodb_file_per_table and few more settings to prevent ibdata1 to grow infinitely in future

server:~# /etc/init.d/mysql stop

server:~# vim /etc/mysql/my.cnf

Delete files taking up too much space –ibdata1 ib_logfile0 and ib_logfile1

server:~# cd /var/lib/mysql/
server:~#  rm -f ibdata1 ib_logfile0 ib_logfile1
server:~# /etc/init.d/mysql start
server:~# /etc/init.d/mysql stop
server:~# /etc/init.d/mysql start
server:~# ps ax |grep -i mysql


You should get no running MySQL instance (processes), so above ps command should return blank.

7. Re-Import previously dumped SQL databases with mysql cli client

server:~# cd /root/
server:~# mysql -u root -p < dump.sql

Hopefully import should went fine, and if no errors experienced new data should be in.

Altearnatively if your database is too big and you want to import it in less time to mitigate SQL downtime, instead import the database with:

server:~# mysql -u root -p
mysql> SOURCE /root/dump.sql;


If something goes wrong with the import for some reason, you can always copy over sql binary files from /root/mysql-data-backup/ to /var/lib/mysql/

8. Connect to mysql and check whether databases are listable and re-check ibdata file size

Once imported login with mysql cli and check whther databases are there with:

server:~# mysql -u root -p

Next lets see what is currently the size of ibdata1, ib_logfile0 and ib_logfile1

server:~# du -hsc /var/lib/mysql/{ibdata1,ib_logfile0,ib_logfile1}
19M     /var/lib/mysql/ibdata1
1,1G    /var/lib/mysql/ib_logfile0
1,1G    /var/lib/mysql/ib_logfile1
2,1G    total

Now ibdata1 will grow, but only contain table metadata. Each InnoDB table will exist outside of ibdata1.
To better understand what I mean, lets say you have InnoDB table named blogdb.mytable.
If you go into /var/lib/mysql/blogdb, you will see two files
representing the table:

  •     mytable.frm (Storage Engine Header)
  •     mytable.ibd (Home of Table Data and Table Indexes for blogdb.mytable)

Now construction will be like that for each of MySQL stored databases instead of everything to go to ibdata1.
MySQL 5.6+ admins could relax as innodb_file_per_table is enabled by default in newer SQL releases.

Now to make sure your websites are working take few of the hosted websites URLs that use any of the imported databases and just browse.
In my case ibdata1 was 45GB after clearing it up I managed to save 43 GB of disk space!!!

Enjoy the disk saving! 🙂

Speed up WordPress / Joomla CMS and MySQL server on Linux with tmpfs ram file system / Decrease Website pageload times with RAM caching

Wednesday, March 4th, 2015

Reading Time: 9minutes

As a WordPress blog owner and an sys admin that has to deal with servers running a lot of WordPress / Joomla / Droopal and other custom CMS installed on servers, performoing slow or big enough to put a significant load on servers
and I love efficiency and hardware cost saving is essential for my daily job, I'm constantly trying to find new ways to optimize Customer Website (WordPress) and rest of sites in order to utilize better our servers and improve our clients sites speed (and hence satisfaction). 

There is plenty of little things to do on servers but probably among the most crucial ones which we use nowadays that save us a lot of money is tmpfs, and earlier (ramfs) – previously known as shmfs).
TMPFS is a (Temporary File Storage Facility) Linux kernel technology based on ramfs (used by Linux kernel initrd / initramfs on boot time in order to load and store the Linux kernel in memory, before system hard disk partition file systems are mounted) which is heavily used by virtually all modern popular Linux distributions. 

Using ramfs (cramfs variation – Compressed ROM filesystem) has been used to store different system environment kernel and Desktop components of many Linux environment / applications and used by a lot of the Linux BootCD such as the most famous (Klaus Knopper's) KNOPPIX LiveCD and Trinity Rescue Kit Linux (TRK uses /dev/shm which btw can be seen on most modern Linux distros and is actually just another mounted tmpfs).
If you haven't tried Live Linux yet try it out as me and a lot of sysadmins out there use some kind of LiveLinux at least few times on yearly basis  to Recover Unbootable Linux servers after some applied remote Updates as well as for Rescuing (Save) Data from Linux server failing to properly boot because of hard disk (bad blocks) failures. As I said earlier TMPFS is also used on almost any distribution for the /dev/ filesystem which is kept in memory.

You can see which tmpfs partitions is used on your Linux server with:


debian-server:~# mount |grep -i tmpfs
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)


Above is an output from a standard Debian Linux server. On CentOS 7 standard mounted tmpfs are as follows:


[root@centos ~]# mount |grep -i tmpfs
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1016332k,nr_inodes=254083,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)


[root@centos ~]# df -h|grep -i tmpfs
devtmpfs                 993M     0  993M   0% /dev
tmpfs                   1002M   92K 1002M   1% /dev/shm
tmpfs                   1002M  8.8M  993M   1% /run
tmpfs                   1002M     0 1002M   0% /sys/fs/cgroup

The /run tmpfs mounted directory is also to be seen also on latest Ubuntus and Fedoras and is actually the good old /var/run ( where applications keep there pids and some small app related files) stored in tmpfs filesystem stored in memory.

If you're wondering what is /dev/shm and why it appears mounted on every single Linux Server / Desktop you've ever used this is a special filesystem shared memory which various running programs (processes) can use to transfer data quick and efficient between each other to preven the slow disk swapping. People using Linux for the rest 15 years should remember /dev/shm has been a target of a lot of kernel exploits as historically it had a lot of security issues.

While writting this article I've just checked about KNOPPIX developed amd just for info as of time of writting this distro has already 1000+ programs on CD version and 2600+ packages / application on DVD version.
Nowadays Knoppixis mostly used mostly as USB Live Flash drive as a lot of people are dropping CD / DVD use (many servers doesn't have a CD / DVD Drive) and for USB Live Flash Linux distros tmpfs is also key technology used as this gives the end user an amazing fast experience (Desktop applications run much fasten on Live USBs when tmpfs is used than when the slow 7200 RPM HDDs are used).

Loading big parts of the distribution within RAM (with tmpfs from Linux Kernel 2.4+ onwards) is also heavily used by a lot of Cluster vendors in most of Clustered (Cloud) Linux based environemnts, cause TMPFS gives often speeds up improvements to x30 times and decreases greatly I/O HDD. FreeBSD users will be happy to know that TMPFS is already ported and could be used on from FreeBSD 7.0+ onward.

In this small article I will give you example use on how I use tmpfs to speed up our WordPress Websites which use WP Caching plugins such as W3 Total Cache and WP Super Cache
and Hyper Cache / WP Super Cache disk caching and MySQL server as a Database backend.
Below example is wordpress specific but since it can be easily applied to JoomlaDrupal or any other CMS out there that uses mySQL server to make a lot of CPU expensive memory hungry(LEFT JOIN) queries which end up using a slow 7200 RPM hard disk.


1. Preparing tmpfs partitions for WordPress File Cache directory

If you want to give tmpfs a test drive, I recommend you try to create / mount a 20 Megabyte partition. To create a tmpfs partition you don't need to use a tool like mkfs.ext3 / mkfs.ext4 as TMPFS is in reality a virtual filesystem that is mapped in the server system physical RAM (volatile memory). TMPFS is very nice because if you run out of free RAM system starts a combination of RAM use + some Hard disk SWAP 
The great thing about TMPFS is it never uses all of the available RAM and SWAP, which would not halt your server if TMPFS partition gets filled, but instead you will start getting the usual "Insufficient Disk Space", just like with a physical HDD parititon. RAMFS cares much less about server compared to TMPFS, because if RAMFS is historically older.

ramfsfile systems cannot be limited in size like a disk base file system which is limited by it’s capacity, thus ramfswill continue using memory storage until the system runs out of RAM and likely crashes or becomes unresponsive. This is a problem if the application writing to the file system cannot be limited in total size, so in my opinion you better stay away from RAMFS except you have a good idea what you're doing. Another disadvantage of RAMFS compared to TMPFS is you cannot see the size of the file system in df and it can only be estimated by looking at the cached entry in free.

Note that before proceeding to use TMPFS or RAMFS you should know besides having advantages, there are certain serious disadvantage that if the server using tmpfs (in RAM) to store files crashes the customer might loose his data, therefore using RAM filesystems on Production servers is best to be used just for caching folders which are regularly synchronized with (rsync) to some folder to assure no data will be lost on server reboot or crash.

Memory of fast storage areas are ideally suited for applications which need repetitively small data areas for caching or using as temporary space such as Jira (Issue and Proejct Tracking Software) Indexing  As the data is lost when the machine reboots the tmpfsstored data must not be data of high importance as even scheduling backups cannot guarantee that all the data will be replicated in the even of a system crash.

To test mounting a tmpfs virtual (memory stored) filesystem issue:

mount -t tmpfs tmpfs -o size=256m /mnt/tmpfs

If you want to test mount a ramfs instead:


 mount -t ramfs -o size=256m ramfs /mnt/ramfs


debian-server:~#  mount |grep -i -E "ramfs|tmpfs"
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /mnt/tmpfs type tmpfs (rw,size=256m)
ramfs on /mnt/ramfs type tmpfs (rw,size=256m)


Once mounted tmpfs can be used in the same way as any ext4 / reiserfs filesystem. In the same way to make mounts permanent, its necessery to add a line to /etc/fstab

To illustrate better a tmpfs use case on my blog running WordPress with W3TotalCache (W3TC) plugin cache folder in /var/www/blog/wp-content/w3tc to get advantage of tmpfs to store w3tc files.

a) Stop Apache

On Debian

debian-server:~# /etc/init.d/apache stop

On CentOS 

[root@centos ~]# /etc/init.d/httpd stop

b) Move w3tc dir to w3tc-bak


debian-server:~# cd /var/www/blog/wp-content/
debian-server:~# mv w3tc w3tc-bak


c) Create w3tc directory

debian-server:/var/www/blog/wp-content# mkdir w3tc
debian-server:/var/www/blog/wp-content# chown -R www-data:www-data w3tc

d) Add tmpfs record to /etc/fstab

My W3TC Cache didn't grow bigger than 2Gigabytes so I create a 2Giga directory for it by adding following in /etc/fstab 

debian-server:~# vim /etc/fstab


tmpfs /var/www/blog/wp-content/w3tc tmpfs defaults,size=2g,noexec,nosuid,uid=33,gid=33,mode=1755 0 0

You might also want to add the nr_inodes (option) to tmpfs while mounting. nr_inodes is the maximum inode for instance. Default is half the number of your physical RAM pages, (on a machine with highmem) the number of lowmem RAM page, some common option that should work is nr_inodes=5k, if you're unsure what this option does you can safely skip it 🙂

e) Mount new added tmpfs folder

Then to mount the newly added filesystem issue:

mount -a

Or if you're on a CentOS / RHEL server use httpd Apache user instead and whenever you have docroot and wordpress installed.


[root@centos ~]# chown -R apache:apache: w3tc

If you're using Apache SuPHP use whatever the UID / GID is proper.

On CentOS you will need to set proper UID and GID (UserID / GroupID), to find out which ones to to use check in /etc/passwd:

[root@centos ~]# grep -i apache /etc/passwd

f) Move old w3tc cache from w3tc-bak to w3tc


debian-server:/var/www/blog/wp-content# mv w3tc-bak/* w3tc/


g) Start again Apache

On Debian:


debian-server:~# /etc/init.d/apache2 start


On CentOS:

[root@centos~]# /etc/init.d/httpd start

h) Keeping w3tc cache site folder synced

As I said earlier the biggest problem with caching (the reason why many hosting providers) and site admins refuse to use it is they might loose some data, to prevent data loss or at least mitigate the data loss to few minutes intervals it is a good idea to synchronize tmpfs kept folders somewhere to disk with rsync.

To achieve that use a cronjob like this:

debian-server:~# crontab -u root -e
*/5 * * * * /usr/bin/ionice -c3 -n7 /usr/bin/nice -n 19 /usr/bin/rsync -ah –stats –delete /var/www/blog/wp-content/w3tc/ /backups/tmpfs/cache/ 1>/dev/null

Note that you will need to have the /backups/tmpfs/cache folder existing, create it with:


debian-server:~# mkdir -p /backups/tmpfs/cache

You will also need to add a rsync synchronization from backupped folder to tmpfs (in case if the server gets accidently rebooted because it hanged or power outage), place in



ionice -c3 -n7 nice -n 19 rsync -ahv –stats –delete /backups/tmpfs/cache/ /var/www/blog/wp-content/w3tc/ 1>/dev/null

(somewhere before exit 0) line

0 05 * * * /usr/bin/ionice -c3 -n7 /bin/nice -n 19 /usr/bin/rsync -ah –stats –delete /var/www/blog/wp-content/w3tc/ /backups/tmpfs/cache/ 1>/dev/null



2. Preparing tmpfs partitions for MySQL server temp File Cache directory

Its common that MySQL servers had to serve a lot of long and heavy SQL JOIN Queries mostly by related posts WP plugins such as (Zemanta Related Posts) and Contextual Related posts though MySQLs are well optimized  to work as much as efficient using mysql tuner (tuning primer) still often SQL servers get a lot of temp tables created to disk (about 25% to 30%) of all SQL queries use somehow HDD to serve queries and as this is very slow and there is file lock created the overall MySQL performance becomes sluggish at times to fix (resolve) that without playing with SQL code to optimize the slow queries the best way I found is by using TMPFS as MySQL temp folder.

To do so I create a TMPFS usually the size of 256 MB because this is usually enough for us, but other hosting companies might want to add bigger virtual temp disk:

a) Add tmpfs new dir to /etc/fstab

In /etc/fstab add below record with vim editor:

debian-server:~# vim /etc/fstab


tmpfs /var/mysqltmp tmpfs rw,gid=111,uid=108,size=256M,nr_inodes=10k,mode=0700 0 0


Note that the uid / and gid 105 and 114 are taken again from /etc/passwd

On Debian

debian-server:~# grep -i mysql /etc/passwd
mysql:x:108:111:MySQL Server,,,:/var/lib/mysql:/bin/false

On CentOS

[root@centos ~]# grep -i mysql /etc/passwd
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/bash

b) Create folder /var/mysqltmp or whenever you want to place the tmpfs memory kept SQL folder


debian-server:~# mkdir /var/mysqltmp
debian-server:~# chown mysql:mysql /var/mysqltmp


debian-server:~# mount|grep -i tmpfs
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /var/www/blog/wp-content/w3tc type tmpfs (rw,noexec,nosuid,size=2g,uid=33,gid=33,mode=1755)
tmpfs on /var/mysqltmp type tmpfs (rw,gid=108,uid=111,size=256M,nr_inodes=10k,mode=0700)

c) Add new path to tmpfs created folder in my.cnf 

Then  edit /etc/mysql/my.cnf


debian-server:~# vim /etc/mysql/my.cnf

# * Basic Settings
user        = mysql
pid-file    = /var/run/mysqld/
socket      = /var/run/mysqld/mysqld.sock
port        = 3306
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /var/mysqltmp


On CentOS edit and change tmpdir in same way within /etc/my.cnf

d) Finally Restart Apache and MySQL to make mysql start using new set tmpfs memory kept folder

On Debian:

debian-server:~# /etc/init.d/apache2 stop; /etc/init.d/mysql restart; /etc/init.d/apache2 start

On CentOS:

[root@centos ~]# /etc/init.d/httpd stop; /etc/init.d/mysqld restart; /etc/initd/httpd start

Now monitor your server and check your pagespeed increase for me such an optimization usually improves site performance so site becomes +50% faster, to see the difference you can test your website before applying tmpfs caching for site and after that by using Google PageInsight (PageSpeed) Online Test. Though this example is for MySQL and WordPress you can easily adopt the same for Joomla if you have Joomla Caching enabled to some folder, same goes for any other CMS such as Drupal that can take use of Disk Caching. Actually its a small secret of many Hosting providers that allow clients to create sites via CPanel and Kloxo this tmpfs optimizations are already used for sites and by this the provider is able to offer better website service on lower prices. VPS hosting providers also use heavy caching. A lot of people are using TMPFS also to accelerate Sites that have enabled Google Pagespeed as Cacher and accelerator, as PageSpeed module puts a heavy HDD I/O load that can easily stone the server. Many admins also choose to use TMPFS for  /tmp, /var/run, and /var/lock directories as this leads often to significant overall server services operations improvement.
Once you have tmpfs enabled, It is a good idea to periodically monitor your SWAP used space with (df -h), because if you allocate bigger tmpfs partitions than your physical memory and tmpfs's full size starts to be used your machine will start swapping heavily and this could have a very negative performance affect.

debian-server:~# df -h|grep -i tmpfs
tmpfs            3,9G     0   3,9G   0% /lib/init/rw
tmpfs            3,9G     0   3,9G   0% /dev/shm
tmpfs            2,0G  1,4G   712M  66% /var/www/blog/wp-content/w3tc
tmpfs            256M     0   256M   0% /mnt/tmpfs
tmpfs            256M  236K   256M   1% /var/mysqltmp

The applications of tmpfs to accelerate services is up to your imagination, so I will be glad to hear from other admins on any interesting other application or problems faced while using TMPFS.

 Enjoy! 🙂

Why Russophobes hates Putin – How situation changed in Russia during Vladimir Putin presidency

Thursday, April 3rd, 2014

Reading Time: 6minutes



  • For last 12 years of government Putin increated Russia's budget 22 times.
  • Increated warfare spendings 30 times.
  • Increated GDP 12 times (by GDP Russia moved from 36th to 6th place in the World).
  • Increated Russian golden reserves 48 times.
  • Returned back 256 oil, petrol and other natural resources sources / mine-yards (under non-Russian government curretnly are only 3 of
  • Russia's source for natural resources.
  • Nationalized 65% of oil industry and 95% of gas industry.
  • For a 5th consequential year 2nd / 3rd place in export of grain (just for a comparison USA is currently ranked 4th largest weed exporter). The avarage sallary of national institution employed increased 18.5 times.
  • Avarage pension increased 14 times.
  • Reduce of population decreased from 1.5 million per year in year 1999 to 21 000 in 2011, i.e. 71 times.
  • Prohibited deputies in Government to have bank accounts in foreign banks.Prevented American attack against Syria.
  • Put an end to war in Chechnya.

From January y. 2000 to present times Russian ruble rate changed from 28 Rubles per dollar to 29 Rubles per dollar – i.e. severe inflation in Russia ended.Present day Russia is a normal European country not that poor country where approimate pension was 20 dollars and where masters was the financial pyramids and the International Monatery Fund

In 1992 Eltsin cancelled completely export duty of oil products.
In 23 January E. Primakov government forced again oil taxes.
In export price of 9.5 dollars per barrel custom taxes were 2.5 euro per tone and in price 12,5 dollars per barrel 5 euro / tone.Such a minor increase in taxes produced 14 billion rubles in already empty Russian budget.

In August 1999 Eltsin assigned Putin for prime minister.
In just a month later the export taxes Putin increased duty taxes to 7.5 euro/tone and in 8 december to 15 euro/tone. Till then incomes from oil taxes has been steadily increasing and nowadays exporters calculate in national budget half of incomes origin from oil prices and export taxes.
From January to November 2007 Russian customs influxed in national budget 2.57 trillion Rubles.
Oil export takes has drastically raised incomes of citizens.This had major impact on construction business.All Russia nowadays is in reality enormous "construction yard", to illustrate from January to September 2007 375 009 homes were built occupying 34 million square meters.
Cement factories cannot satisfy local market requirements and Russia is forced to import cement from China.
Increased incomes of population led to increase in estates search, this increased apartment prices and as a consequence increased incomes from building activies.
Result is in consutriction business are invested enormous amounts of capital and a real construction boom is observed.
Another consequence of increased income was increase in demand for automobiles. Just for 2006 the quantity of demanded automobiles in Russia increased with 45% and reached 32 billion dollars with a count of sold new cars numbering 2 million. By indicator of sold new cars Russia orders on 5th place after in Europe taking place right after Germany, Great Britain, Italy and France.

Currently are being build a couple of new automobile plants, and existing ones increase production volume.
All this is consequence of increase in demand and therefore from increase in citizens income.

rn>For 10 years budets expenditure for social politics (pensions and social help) increased with 30%.

Before Putin pensions were under existence-minimum with appr. 25% and in 90th pensions were not paid at all.
Now pensions are 50% above existence-minimum and is constantly increasing.
In 2000 approximate sallary in Russia was 2223 Rubles (appr. 80 dollars).
Now approximate sallary in Russia 19174 rubles (apprx. 660 dollars).
Purchase of domestic goods for 10 years increased 10 times. Number of own automobilse increased 3 times.
Putin nationalized YUKOS, without 'making nervous' emering Russian busness in a market manner – with bankruptcy and auction. All this happened lawful, following laws adopted by democratic parliament.
The president doesn't have the right to use other  means. Formal occasion for arest of Hodorkovski were taxation frauds of YUKOS. In such machinations are involved practically all large private companies and this is the reason why nobody believes that excuse. It is unbelievable. However Putin simply defended Russia's interests.




The proof for that is transmission of actives of YUKOS to national corporation "Rosneft". It would have been more righteous if this actives were just confiscated … but there are laws and Putin had just stick to them. After all the President can't go out of framework of his jurisdiction.
It can be just added that after Khodorkovsky  was injailed, collectivity (incomes)of taxes of ex-actives of YUKOS increased 80 TIMES!
In y. 2004 Putin finally removed law "Agreement for Separation of Production  (Separation Agreement)". This law was annexed during Eltsin's regime, in order to benefit Oligarchs (Khodorkovsky, Gusinsky, Beresovsky etc.) in order to make possible Russian oil reserves to be possessed by Western (American and British) oil corporations.
By the power of this law Russian oil and natural fields went into international jurisdiction, and therefore the money from Russian oil doesn't entered budget of Russia but influxed in Western companies.
Money from oil drills went mainly into Dutch "Shell" for covering of corporation expenses. Only after something remained from that they sold it to Russia. In 2006 Putin declared following in that connection "And now we don't get anything from them and if they increase their profit we will not receive it even in 10 years from now."
In fact to this moment Russia didn't get any money from their own oil.
After the law was removed in 2004, revenues in budeg increased from 3 to 4 times.
After cancellation of contracts for oil fields "Sakhalin-1" and "Sakhalin-2" Russia's loans to American company calculated to 700 million dollars, for that time this was too much. The whole Anglo-Saxon world pricked against Putin because of a simple reason: "UK planned to assure its oil reserves for years to come in expense of Russia – only Germany and France who didn't have a direct interest in that process kept neutral …
In 1992 – 1995 the head aparatus of Russia formed its view based on foreign advisors. All legislation from 1990's was hence written by them. In Russian country administration was working 10 000 foreign coaches.  George Soros was financing writting of student history books where the Battle for Stalingrad was mentioned in only 2 pages and about the meeting between Elbe and Soviet and American soldiers (Elbe Day) in 10 pages.



: In that mournful times on pupils notebook you can see portrainst of 4 American presidents of USA.
Until this very day there are left relics from that anti-Russian propaganda but hopefully with time Putin will throw away american propaganda from education system.
But why Putin cannot immediately dismiss all this hostile to Russia clerks? The reason is simple: Constitution of Russia written under dictation of Western coaches, does not allow quick changse into it.
Nowadays the President is just one of many clerks with resticted power. Yes truly president power is a bit bigger than other clerks, but country head can't influence everything. The president can't even define his ministers, even though by law this is part of his jurisdiction.

Overfulfilment of budet in times of Putin govern allowed craetion of country Stabilization fund. Nowdays is collected huge golden-currency reserve and practically Russia doesn't have external debt.

War in Kavkaz is over, separatists were destroyed. All famous terrorist leaders were liquidated physically.
Even Zelimkhan_Yandarbiyev was killed in Qatar, Basaev and Umarov were also destoyed. Putin promised "to drawn them if necessary even in their own toilet dish" and he fulfilled his promise. Of course, separatism is not completely destroyed, such conflicts cannot be quickly solved, but nowadays situation in Kakvaz is the best possible. If Chechnya's elite feels the power of Moscow and benefits of being a surrender – then separatism will fade away. This is exactly what happens. The attempts of western spy centrals to supply terrorists are still leading to separate terr. acts but this is the maximum – there will be no war anymore.
For 10 years Putin increased political influence of Russia in world and lifted its image. Today Russia follows its own interests, and not Western ones.
It is not coincidence that Putin was recognized as the most influential world politic of 2013. He prevented Russia's devastation led country out of catastrophe caused by Gorbachov and Eltsin. That's the rason why Western Medias abhor him and compare him with devil.
Everyone interested in political life in Russia seems, that the battle against corruption took unforeseen measures.
At least 2-3 times weekly on TV are shown arrests  of clergy and policeman and news inform about sentences against government employees.
Lets not forget Crimea, here is how it was given to Ukraine.

In 1992 on signature of Treaty of Belovesh fordissolution of USSR Ukrainian representative Leonid Kravchuk noticed that Boris Eltsin is "in play with Vodka" and is delaying signature, he urged him with words: –

"Borya if you like take Crimea, please just  sign the contract!".

Drunk Eltsin nobly waved hand:

"What for I need Crimea? Here it is your present!"

And signed and by one scratch he "killed" efforts of Prince Grigory Potemkin, Catherina the Great, heros of Sevastopol in 1856, heroes defenders of Sevastopol in 1941 …

Few days ago Putin removed consequences of this "joke with history" – and people of Crimea sung and danced on squares, returning to their motherland.

Here is Why Russophobes hates Putin!



Source Materials from
Translated by: Georgi Georgiev
This translation is copyrighted and copying can only be done with explicit allowance of author Author or link to original translation


Finding spam sending php scripts on multiple sites servers – Tracing and stopping spammer PHP scripts

Monday, April 14th, 2014

Reading Time: 4minutes

Spam has become a severe issue for administrators, not only for mail server admins but also for webshosting adms. Even the most secure spam protected mail server can get affected by spam due to fact it is configured to relay mail from other servers acting as web hosting sites.

Webhosting companies almost always suffer seriously from spam issues and often their mail servers gets blocked (enter spam blacklists), because of their irresponsible clients uploading lets say old vulnerable Joomla, WordPress without Akismet or proper spam handling plugin,a CMS which is not frequently supported / updated or custom client insecure php code.

What I mean is Shared serverA is often configured to sent mail via (mail) server B. And often some of the many websites / scripts hosted on server A gets hacked and a spam form is uploaded and tons of spam start being shipped via mail server B.

Of course on mail server level it is possible to configure delay between mail sent and adopt a couple of policies to reduce spam, but the spam protection issue can't be completely solved thus admin of such server is forced to periodically keep an eye on what mail is sent from hosting server to mail server.

If you happen to be one of those Linux(Unix) webhosting admins who find few thousand of spammer emails into mail server logs or your eMail server queue and you can't seem to find what is causing it, cause there are multiple websites shared hosting using mainly PHP + SQL and you can't identify what php script is spamming by reviewing  Apache log / PHP files. What you can do is get use of:

PHP mail.log directive

Precious tool in tracking spam issues is a PHP Mail.log parameter, mail log paramater is available since PHP version >= 5.3.0 and above.
PHP Mail.log parameter records all calls to the PHP mail() function including exact PHP headers, line numbers and path to script initiating mail sent.

Here is how it is used:

1. Create empty PHP Mail.log file

touch /var/log/phpmail.log

File has to be writtable to same user with which Apache is running in case of Apache with SuPHP running file has to be writtable by all users.

On Debian, Ubunut Linux:

chown www:data:www-data /var/log/phpmail.log

On CentOS, RHEL, SuSE phpmail.log has to be owned by httpd:

chown httpd:httpd /var/log/phpmail.log

On some other distros it might be chown nobody:nobody etc. depending on the user with which Apache server is running.


2. Add to php.ini configuration following lines

mail.add_x_header = On
mail.log = /var/log/phpmail.log

PHP directive instructs PHP to log complete outbund Mail header sent by mail() function, containing the UID of the web server or PHP process and the name of the script that sent the email;

(X-PHP-Originating-Script: 33:mailer.php)

i.e. it will make php start logging to phpmail.log stuff like:


mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: — Headers: From: "Priority Mail" <status_93@pomoriemon> X-Mailer: MailMagic2.0 Reply-To: "Priority Mail" <> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="——
mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: — Headers: From: "One Day Shipping" <status_44@pomoriemonastery.
org> X-Mailer: CSMTPConnectionv1.3 Reply-To: "One Day Shipping" <> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="—
mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: — Headers: From: "Logistics Services" <customer.> X-Mailer: TheBat!(v3.99.27)UNREG Reply-To: "Logistics Services" <> Mime-Version: 1.0 Content-Type: mult
mail() on [/var/www/pomoriemonasteryorg/components/com_xmap/2ktdz2.php:1]: To: — Headers: From: "Priority Mail" <status_73@pomoriemonaster> X-Mailer: FastMailer/Webmail(versionSM/1.2.6) Reply-To: "Priority Mail" <> Mime-Version: 1.0 Content-Type: multipart/alternativ


On Debian / Ubuntu Linux to enable this logging, exec:

echo 'mail.add_x_header = On' >> /etc/php5/apache2/php.ini
echo 'mail.log = /var/log/phpmail.log' >> /etc/php5/apache2/php.ini

I find it useful to symlink /etc/php5/apache2/php.ini to /etc/php.ini its much easier to remember php location plus it is a standard location for many RPM based distros.

ln -sf /etc/php5/apache2/php.ini /etc/php.ini

Or another "Debian recommended way" to enable mail.add_x_header logging on Debian is via:

echo 'mail.add_x_header = On' >> /etc/php5/conf.d/mail.ini
echo 'mail.log = /var/log/phpmail.log' >> /etc/php5/conf.d/mail.ini

On Redhats (RHEL, CentOS, SuSE) Linux issue:

echo 'mail.add_x_header = On' >> /etc/php.ini
echo 'mail.log = /var/log/phpmail.log' >> /etc/php.ini

3. Restart Apache

On Debian / Ubuntu based linuces:

/etc/init.d/apache2 restart

P.S. Normally to restart Apache without interrupting client connections graceful option can be used, i.e. instead of restarting do:

/etc/init.d/apache2 graceful

On RPM baed CentOS, Fedora etc.:

/sbin/service httpd restart


apachectl graceful

4. Reading the log

To review in real time exact PHP scripts sending tons of spam tail it:

tail -f /var/log/phpmail.log


mail() on [/var/www/remote-admin/wp-includes/class-phpmailer.php:489]: To: — Headers: Date: Mon, 14 Apr 2014 03:27:23 +0000 Return-Path: From: WordPress Message-ID: X-Priority: 3 X-Mailer: PHPMailer ( [version 2.0.4] MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="UTF-8"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: — Headers: From: "Manager Elijah Castillo" <> X-Mailer: Mozilla/5.0 (Windows; U; Windows NT 5.0; es-ES; rv: Gecko/20100111 Thunderbird/3.0.1 Reply-To: "Manager Elijah Castillo" <> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A76017CC"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: — Headers: From: "Manager Justin Murphy" <> X-Mailer: Opera Mail/10.62 (Win32) Reply-To: "Manager Justin Murphy" <> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A7603ED6"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: — Headers: From: "Manager Elijah Castillo" <> X-Mailer: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; pl; rv: Gecko/20100317 Thunderbird/3.0.4 Reply-To: "Manager Elijah Castillo" <> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A7606308"
mail() on [/var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php:1]: To: — Headers: From: "Manager Justin Murphy" <> X-Mailer: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; pl; rv: Gecko/20100317 Thunderbird/3.0.4 Reply-To: "Manager Justin Murphy" <> Mime-Version: 1.0 Content-Type: multipart/alternative;boundary="———-1397463670534B9A76086D1"


As you can see there is a junky spam mails sent via some spammer script uploaded under name .3a1a1c.php, so to stop the dirty bastard, deleted the script:

rm -f /var/www/pomoriemonasteryorg/media/rsinstall_4de38d919da01/admin/js/tiny_mce/plugins/inlinepopups/skins/.3a1a1c.php

It is generally useful to also check (search) for all hidden .php files inside directoring storing multiple virtualhost websites, as often a weirdly named hidden .php is sure indicator of either a PHP Shell script kiddie tool or a spammer form.

Here is how to Find all Hidden Perl / PHP scripts inside /var/www:

find . -iname '.*.php'

find . -iname '.*.pl*'


Reviewing complete list of all hidden files is also often useful to determine shitty cracker stuff

 find . -iname ".*"

Debugging via  /var/log/phpmail.log enablement is useful but is more recommended on development and staging (QA) environments. Having it enable on productive server with high amounts of mail sent via PHP scripts or just on dedicated shared site server could cause both performance issues, hard disk could quickly get and most importantly could be a severe security hole as information from PHP scripts could be potentially exposed to external parties.