Archive for August 30th, 2011

How to Secure Apache on FreeBSD against Range header DoS vulnerability (affecting Apache 1.3/2.x)

Tuesday, August 30th, 2011

How to Secure Apache webserver on FreeBSD and CentOS against Range: header Denial of Service attack

Recently has become publicly known for the serious hole found in all Apache webserver versions 1.3.x and 2.0.x and 2.2.x. The info is to be found inside the security CVE-2011-3192 https://issues.apache.org/bugzilla/show_bug.cgi?id=51714

Apache remote denial of service is already publicly cirtuculating, since about a week and is probably to be used even more heavily in the 3 months to come. The exploit can be obtained from exploit-db.com a mirror copy of #Apache httpd Remote Denial of Service (memory exhaustion) is for download here

The DoS script is known in the wild under the name killapache.pl
killapache.pl PoC depends on perl ForkManager and thus in order to be properly run on FreeBSD, its necessery to install p5-Parallel-ForkManager bsd port :


freebsd# cd /usr/ports/devel/p5-Parallel-ForkManager
freebsd# make install && make install clean
...

Here is an example of the exploit running against an Apache webserver host.


freebsd# perl httpd_dos.pl www.targethost.com 50
host seems vuln
ATTACKING www.targethost.com [using 50 forks]
:pPpPpppPpPPppPpppPp
ATTACKING www.targethost.com [using 50 forks]
:pPpPpppPpPPppPpppPp
...

In about 30 seconds to 1 minute time the DoS attack with only 50 simultaneous connections is capable of overloading any vulnerable Apache server.

It causes the webserver to consume all the machine memory and memory swap and consequently makes the server to crash in most cases.
During the Denial of Service attack is in action access the websites hosted on the webserver becomes either hell slow or completely absent.

The DoS attack is quite a shock as it is based on an Apache range problem which started in year 2007.

Today, Debian has issued a new versions of Apache deb package for Debian 5 Lenny and Debian 6, the new packages are said to have fixed the issue.

I assume that Ubuntu and most of the rest Debian distrubtions will have the apache’s range header DoS patched versions either today or in the coming few days.
Therefore work around the issue on debian based servers can easily be done with the usual apt-get update && apt-get upgrade

On other Linux systems as well as FreeBSD there are work arounds pointed out, which can be implemented to close temporary the Apache DoS hole.

1. Limiting large number of range requests

The first suggested solution is to limit the lenght of range header requests Apache can serve. To implement this work raround its necessery to put at the end of httpd.conf config:


# Drop the Range header when more than 5 ranges.
# CVE-2011-3192
SetEnvIf Range (?:,.*?){5,5} bad-range=1
RequestHeader unset Range env=bad-range
# We always drop Request-Range; as this is a legacy
# dating back to MSIE3 and Netscape 2 and 3.
RequestHeader unset Request-Range
# optional logging.
CustomLog logs/range-CVE-2011-3192.log common env=bad-range
CustomLog logs/range-CVE-2011-3192.log common env=bad-req-range

2. Reject Range requests for more than 5 ranges in Range: header

Once again to implement this work around paste in Apache config file:

This DoS solution is not recommended (in my view), as it uses mod_rewrite to implement th efix and might be additionally another open window for DoS attack as mod_rewrite is generally CPU consuming.


# Reject request when more than 5 ranges in the Range: header.
# CVE-2011-3192
#
RewriteEngine on
RewriteCond %{HTTP:range} !(bytes=[^,]+(,[^,]+){0,4}$|^$)
# RewriteCond %{HTTP:request-range} !(bytes=[^,]+(?:,[^,]+){0,4}$|^$)
RewriteRule .* - [F]

# We always drop Request-Range; as this is a legacy
# dating back to MSIE3 and Netscape 2 and 3.
RequestHeader unset Request-Range

3. Limit the size of Range request fields to few hundreds
To do so put in httpd.conf:


LimitRequestFieldSize 200

4. Dis-allow completely Range headers: via mod_headers Apache module

In httpd.conf put:


RequestHeader unset Range
RequestHeader unset Request-Range

This work around could create problems on some websites, which are made in a way that the Request-Range is used.

5. Deploy a tiny Apache module to count the number of Range Requests and drop connections in case of high number of Range: requests

This solution in my view is the best one, I’ve tested it and I can confirm on FreeBSD works like a charm.
To secure FreeBSD host Apache, against the Range Request: DoS using mod_rangecnt, one can literally follow the methodology explained in mod_rangecnt.c header:


freebsd# wget http://people.apache.org/~dirkx/mod_rangecnt.c
..
# compile the mod_rangecnt module
freebsd# /usr/local/sbin/apxs -c mod_rangecnt.c
...
# install mod_rangecnt module to Apache
freebsd# /usr/local/sbin/apxs -i -a mod_rangecnt.la
...

Finally to load the newly installed mod_rangecnt, Apache restart is required:


freebsd# /usr/local/etc/rc.d/apache2 restart
...

I’ve tested the module on i386 FreeBSD install, so I can’t confirm this steps works fine on 64 bit FreeBSD install, I would be glad if I can hear from someone if mod_rangecnt is properly compiled and installed fine also on 6 bit BSD arch.

Deploying the mod_rangecnt.c Range: Header to prevent against the Apache DoS on 64 bit x86_amd64 CentOS 5.6 Final is also done without any pitfalls.


[root@centos ~]# uname -a;
Linux centos 2.6.18-194.11.3.el5 #1 SMP Mon Aug 30 16:19:16 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@centos ~]# /usr/sbin/apxs -c mod_rangecnt.c
...
/usr/lib64/apr-1/build/libtool --silent --mode=link gcc -o mod_rangecnt.la -rpath /usr/lib64/httpd/modules -module -avoid-version mod_rangecnt.lo
[root@centos ~]# /usr/sbin/apxs -i -a mod_rangecnt.la
...
Libraries have been installed in:
/usr/lib64/httpd/modules
...
[root@centos ~]# /etc/init.d/httpd configtest
Syntax OK
[root@centos ~]# /etc/init.d/httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]

After applying the mod_rangecnt patch if all is fine the memory exhaustion perl DoS script‘s output should be like so:


freebsd# perl httpd_dos.pl www.patched-apache-host.com 50
Host does not seem vulnerable

All of the above pointed work-arounds are only a temporary solution to these Grave Apache DoS byterange vulnerability , a few days after the original vulnerability emerged and some of the up-pointed work arounds were pointed. There was information, that still, there are ways that the vulnerability can be exploited.
Hopefully in the coming few weeks Apache dev team should be ready with rock solid work around to the severe problem.

In 2 years duration these is the second serious Apache Denial of Service vulnerability after before a one and a half year the so called Slowloris Denial of Service attack was capable to DoS most of the Apache installations on the Net.

Slowloris, has never received the publicity of the Range Header DoS as it was not that critical as the mod_range, however this is a good indicator that the code quality of Apache is slowly decreasing and might need a serious security evaluation.

Monitoring Windows hosts with Nagios on Debian GNU/Linux

Tuesday, August 30th, 2011

Nagios logo install and configure nagios to monitor Windows hosts with on Debian GNU/Linux

In this article in short, I’ll explain how I configured Nagios on a Debian GNU/Linux release (Squeeze 6) to monitor a couple of Windows hosts running inside a local network. Now let’s start.

1. Install necessery nagios debian packages

apt-get install nagios-images nagios-nrpe-plugin nagios-nrpe-server nagios-plugins nagios-plugins-basic nagios-plugins-standard
nagios3 nagios3-cgi nagios3-common nagios3-core

2. Edit /etc/nagios-plugins/config/nt.cfg

In the File substitute:

define command { command_name check_nt command_line /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -v '$ARG1$' }

With:

define command {
command_name check_nt
command_line /usr/lib/nagios/plugins/check_nt -H '$HOSTADDRESS$' -p 12489 -v $ARG1$ $ARG2$
}

3. Modify nrpe.cfg to put in allowd hoss to connect to the Nagions nrpe server

vim /etc/nagios/nrpe.cfg

Lookup inside for nagios’s configuration directive:

allowed_hosts=127.0.0.1

In order to allow more hosts to report to the nagios nrpe daemon, change the value to let’s say:

allowed_hosts=127.0.0.1,192.168.1.4,192.168.1.5,192.168.1.6

This config allows the three IPs 192.168.1.4-6 to be able to report for nrpe.

For the changes to nrpe server to take effect, it has to be restrarted.

debian:~# /etc/init.d/nagios-nrpe-server restart

Further on some configurations needs to be properly done on the nrpe agent Windows hosts in this case 192.168.1.4,192.168.1.5,192.168.1.6

4. Install the nsclient++ on all Windows hosts which CPU, Disk, Temperature and services has to be monitored

Download the agent from http://sourceforge.net/projects/nscplus and launch the installer, click twice on it and follow the installation screens. Its necessery that during installation the agent has the NRPE protocol enabled. After the installation is complete one needs to modify the NSC.ini
By default many of nsclient++ tracking modules are not enabled in NSC.ini, thus its necessery that the following DLLs get activated in the conf:

FileLogger.dll
CheckSystem.dll
CheckDisk.dll
NSClientListener.dll
SysTray.dll
CheckEventLog.dll
CheckHelpers.dll

Another requirement is to instruct the nsclient++ angent to have access to the Linux installed nagios server again with adding it to the allowed_hosts config variable:

allowed_hosts=192.168.1.1

In my case the Nagios runs on Debian Lenny (Squeeze) 6 and possess the IP address of 192.168.1.1
To test the intalled windows nsclient++ agents are properly installed a simple telnet connection from the Linux host is enough:

5. Create necessery configuration for the nagios Linux server to include all the Windows hosts which will be monitored

There is a window.cfg template file located in /usr/share/doc/nagios3-common/examples/template-object/windows.cfg on Debian.

The file is a good start point for creating a conf file to be understand by nagios and used to periodically refresh information about the status of the Windows hosts.

Thus it’s a good idea to copy the file to nagios3 config directory:

debian:~# mkdir /etc/nagios3/objects
debian:~# cp -rpf /usr/share/doc/nagios3-common/examples/template-object/windows.cfg /etc/nagios3/objects/windows.cfg

A sample windows.cfg content, (which works for me fine) and monitor a couple of Windows nodes running MS-SQL service and IIS and makes sure the services are up and running are:

define host{
use windows-server ; Inherit default values from a template
host_name Windows1 ; The name we're giving to this host
alias Iready Server ; A longer name associated with the host
address 192.168.1.4 ; IP address of the host
}
define host{
use windows-server ; Inherit default values from a template
host_name Windows2 ; The name we're giving to this host
alias Iready Server ; A longer name associated with the host
address 192.168.1.4 ; IP address of the host
}
define hostgroup{
hostgroup_name windows-servers ; The name of the hostgroup
alias Windows Servers ; Long name of the group
}
define hostgroup{
hostgroup_name IIS
alias IIS Servers
members Windows1,Windows2
}
define hostgroup{
hostgroup_name MSSQL
alias MSSQL Servers
members Windows1,Windows2
}
define service{
use generic-service
host_name Windows1
service_description NSClient++ Version
check_command check_nt!CLIENTVERSION
}
define service{ use generic-service
host_name Windows1
service_description Uptime
check_command check_nt!UPTIME
}
define service{ use generic-service
host_name Windows1
service_description CPU Load
check_command check_nt!CPULOAD!-l 5,80,90
}
define service{
use generic-service
host_name Windows1
service_description Memory Usage
check_command check_nt!MEMUSE!-w 80 -c 90
define service{
use generic-service
host_name Windows1
service_description C: Drive Space
check_command check_nt!USEDDISKSPACE!-l c -w 80 -c 90
}
define service{
use generic-service
host_name Windows1
service_description W3SVC
check_command check_nt!SERVICESTATE!-d SHOWALL -l W3SVC
}
define service{
use generic-service
host_name Windows1
service_description Explorer
check_command check_nt!PROCSTATE!-d SHOWALL -l Explorer.exe
}
define service{
use generic-service
host_name Windows2
service_description NSClient++ Version
check_command check_nt!CLIENTVERSION
}
define service{ use generic-service
host_name Windows2
service_description Uptime
check_command check_nt!UPTIME
}
define service{ use generic-service
host_name Windows2
service_description CPU Load
check_command check_nt!CPULOAD!-l 5,80,90
}
define service{
use generic-service
host_name Windows2
service_description Memory Usage
check_command check_nt!MEMUSE!-w 80 -c 90
define service{
use generic-service
host_name Windows2
service_description C: Drive Space
check_command check_nt!USEDDISKSPACE!-l c -w 80 -c 90
}
define service{
use generic-service
host_name Windows2
service_description W3SVC
check_command check_nt!SERVICESTATE!-d SHOWALL -l W3SVC
}
define service{
use generic-service
host_name Windows2
service_description Explorer
check_command check_nt!PROCSTATE!-d SHOWALL -l Explorer.exe
}
define service{ use generic-service
host_name Windows1
service_description SQL port Check
check_command check_tcp!1433
}
define service{
use generic-service
host_name Windows2
service_description SQL port Check
check_command check_tcp!1433
}
The above config, can easily be extended for more hosts, or if necessery easily setup to track more services in nagios web frontend.
6. Test if connectivity to the nsclient++ agent port is available from the Linux server

debian:~# telnet 192.168.58.6 12489
Trying 192.168.58.6...
Connected to 192.168.58.6.
Escape character is '^]'.
asd
ERROR: Invalid password.

Another good idea is to launch on the Windows host the NSClient++ (system tray) , e.g.:

Start, All Programs, NSClient++, Start NSClient++ (system tray).

Test Nagios configuration from the Linux host running nagios and nrpe daemons to check if the check_nt, can succesfully authenticate and retrieve data generated from the nsclient++ on the Windows host:

debian:~# /usr/lib/nagios/plugins/check_nt -H 192.168.1.5 -p 12489 -v CPULOAD -w 80 -c 90 -l 5,80,90,10,80,90

If everything is okay and the remote Windows system 192.168.1.5 has properly configured and running NSClient++ the above command should return an output like:

CPU Load 1% (5 min average) 1% (10 min average) | '5 min avg Load'=1%;80;90;0;100 '10 min avg Load'=1%;80;90;0;100

In case of the command returns:

could not fetch information from server

instead this means that probably there is some kind of problem with authentication or handshake of the Linux host’s nagios check_nt to the Windows server’s running on 12489.

This is sometimes caused by misconfigured NSC.ini file, however in other occasions this error is caused by misconfigured Windows Firewall or because the NSClient++ is not running with Administrator user.

By the way important note to make about Windows 2008r2 is that if NSClient++ is running there it’s absolutely required to Login with Windows Administrator and run the NSClient++ /start , if it’s run through the Run As Adminsitrator with an admin privileged user the aforementioned error might appear, so be careful.
I’ve experienced this error myself and it took me about 40 minutes to find that I have to run it directly with Administrator user after logging as Administrator.

7. Create nagios web iface Apache configuration

nagios debian pachage is shipped with a config which is suitable to be set debian:~# cp -rpf /usr/share/doc/nagios3-common/examples/apache2.conf /etc/apache2/sites-avalable/nagios
debian:~# ln -sf /etc/apache2/sites-available/nagios /etc/apache2/sites-enabled/nagios

The /etc/apache2/sites-available/nagios can easily be configured to work on Virtualhost, to do so the above copied file need to be wrapped inside a VirtualHost directive. For that put in the beginning of the file;

<VirtualHost *:80>

and in the end of the file:

<VirtualHost *:80>

8. Restart nagios server and Apache for the new settings to take effect

debian:~# /etc/init.d/apache2 restart
...
debian:~# /etc/init.d/nagios3 restart

If some custom configuration about tracking the Debian Linux nagios host running services needs to be made, its also helpful for one to check in /etc/nagios3/conf.d

Well that’s mostly what I had to do to make the Nagios3 server to keep track of a small Windows network on Debian GNU/Linux Squeeze 6, hope this small article helps. Cheers 😉