Posts Tagged ‘Debian Lenny Linux’

Create SVN (Subversion) web statistics on Debian Lenny Linux with mpy-svn-stats and svnstats.sh

Monday, July 5th, 2010

I’ve recently desired to have a visualized statistics on how many commits, imports, people who commit into subversion’s repositories,graphs showing up the most active comitters, commits into the all subversion repositories grouped by month, week etc.
This kind of valuable information can give you insight, on a projects code life cycle. It can also help you to find out who takes most active participation into a certain project code development etc. and therefore could play a vital role in finding out about the work efficiency within your IT company or IT department.

There are plenty of softwares that can generate you some shiny statistics on how often and by whom are commits into your repositories as well as general statistics concerning your repositories accessibility.

Some of the projects suggested by most Linux users online, who had to resolve the same task e.g. (find some decent software to generate them good statistics on the svn use.) are:

1. statsvn

Here is a description on what statsvn is directly taken from its website:

StatSVN retrieves information from a Subversion repository and generates various tables and charts describing the project development, e.g.

StatSVN looks really promising, however what I find personally repulsive about it is that it depends on a Sun Java virtual machine
I have a bad taste for third party software that depends on java and therefore the software uses an XML dump generated from svn log –xml -v path/to/repos > svn-logfile.xml after which it’s necessary to pass the generated svn-logfile.xml file to statsvn, for instance:

statsvn [options] svn-logfile.xml path/to/repos

though a debian of statsvn is available and packaged for Debian in /usr/share/doc/statsvn/README.Debian we read:

Notes to Debian users:

* the jtreemap-based report has been disabled as jtreemap is currently
not packaged for Debian, and Debian cannot ship the applet without
its sources (not included in statsvn’s sources).

— Vincent Fourmond <fourmond@debian.org>, Tue, 4 Mar 2008 21:14:14 +0100

What I understood from statsvn documentation is that jtreemap is absolutely necessary in order to have a running statsvn, regardless if you have or you don’t have a java vm installed.

To take a general idea on what kind of Repo Roadmap does svnstat generates with jtreemap check out the following link

Since jtreemap is not available prepackaged for Debian I decided not to use svnstats though it looked quite superb.

Some further research on the kind of softwares available online able to generate me some statistics from cvs or subversion source repositories led me to,

2. svnplot

svnplot stroke me with it’s perfect looking many graphics generated on the Lines of Code commited, contribution of different authors to the repository, File count, avarage commit file sizes, common activity, author commit activity etc. etc.

I think tt’s worthy to check out some example statistics about a sample repository statistics generated by svnplot to get a better idea what to expect if you choose to install it on your server.

Even though svnplot looked also promising It wasn’t actually my choice because I think it’s not really mature enough as a software, the second reason which hold me back from installing it on my debian server was that I find it too much as a work in progress still.

Since neither svnstast nor svnplot didn’t well match my expectation and lacked a debian package I finally choose:

3. mpy svn stats as a solution to generate and graph information about svn usage

There are few reasons I finally took svn-mpy-stats to be the solution of choice.

1. It is available as a package in Debian Linux and easily installable via apt-get
2. It is written in Python and therefore doesn’t require a java virtual machine or some extra cosmetics to make it work3. It’s really simple and straight forward to configure and already tested and reported that it works well in Debian GNU/Linux

So here is the few simple steps I took to install mpy-svn-stats on Debian Lenny (in Debian Sid / Squeeze I suppose the procedure would be analogous.

– Install mpy-svn-stats via apt-get or aptitude

debian-server:~# apt-get install mpy-svn-stats

Run it for your svn repository with a command like:

debian-server:~# mkdir /var/www/svnstats
/usr/bin/mpy-svn-stats -o /var/www/svnstats/ file:///var/svn-repos/repository_name

In the above command substitute /var/www/svnstats/ and /var/svn-repos/repository_name with a directory of choice where you like to generate the html statistics for the svn usage as well as the full path and name of your repository.

Of course it’s a good idea to make mpy-svn-stats run periodically with for instance crontab or at or any other unix task cheduler available for your Linux system.

So far so good. You have probably already noticed that it’s rather inconvenient because you have to execute mpy-svn-stats command to each of your svn repositories individually.
This is absolute madness if your company is creating new svn source repository projects often, like let’s say everyday, because you will have to generate statistics for each of the repositories either manually or add new repositories manually to a script which will be later invoked by a crontab rule.

To get around this constrain, I’ve come up with a tiny shell script svnstats.sh which takes care for everything on it’s own.

It automatically will loop in your main subversion repositories directory through all the sub-repositories and generate individual html statistics in a separate automatically created directory by the script.

So to make your life easier and automate the process of generating stats with mpy-svn-stats consider downloading svnstats.sh and installing it as a separate rule like so:

debian-server:~# crontab -u root -e

Include therein the following:

# generate svn statistics everyday in 05:20 a.m.20 5 * * * /usr/sbin/svnstats.sh >/dev/null >>&1

Now everyday at 05:20 your mpy-svn-stats will generate a nice graphs and statistics for your subversion repository usage in /var/www/svstats, if you consider generating the data into a different location consider editting the head of mpy-svn-stats svnstats.sh script and change according to your likings.

Now let’s create an Alias in Apache to enable the (mpy-svn-stats generated by svnstats.sh) to be visualized via web:

– Edit VirtualHost configuration file of choice and put there, something like:

Alias /svnstats/ /var/www/svnstats/

Lastly it might be a good idea to use htaccess to protect your url with a password, afterwards you can enjoy your mpy svn statistics.

Tightening PHP Security on Apache 2.2 with ModSecurity2 on Debian Lenny Linux

Monday, April 26th, 2010

Tightening-PHP-Security-on-Apache-2.2-2.4-with-Apache-ModSecurity2
In this article you'll learn how I easily installed and configured the ModSecurity 2 on a Debian Lenny system.
First let me give you a few introductionary words to modsecurity, what is it and why it's a good idea to install and use it on your Apache Webserver.

ModSecurity is an Apache module that provides intrusion detection and prevention for web applications. It aims at shielding web applications from known and unknown attacks, such as SQL injection attacks, cross-site scripting, path traversal attacks, etc.

As you can see from ModSecurity’s description it’s a priceless module add on to Apache that is able to protect your PHP Applications and Apache server from a huge number of hacker attacks undertook against your Online Web Application or Webserver.
The only thing I don’t like about this module is that it is actually a 3rd party module (e.g. not officially part of Apache). Some time ago I remember there was even an exploit for one of the versions of the module.
So in some cases the ModSecurity could also pose a security risk, so beware!
However if you know what you'rre doing and you keep a regular track of security news on some major security websites, that shouldn’t be a concern for you.
Now let'ss proceed to the install of the ModSecurity module itself.
The install is a piece of cake on Debian though you'll be required to use the Debian Lenny backports

Here is the install of the module step by step:

1. First add the gpg key of the backports repository to your install

debian-server:~# gpg --keyserver pgp.mit.edu --recv-keys C514AF8E4BA401C3
# another possible way to add the repository as the website describes is through the command
debian-server:~# wget -O - http://backports.org/debian/archive.key | apt-key add -

2. Install the libapache-mod-security package from the backports Debian Lenny repository

debian-server~:~# apt-get -t lenny-backports install libapache2-mod-security2

Now as a last step of the install ModSeccurity install procedure you have to add some configuration directives to Apache and restart the server afterwards.

– Open your /etc/apache2/apache2.conf and place in it the following configurations


<IfModule mod_security2.c>
# Basic configuration options
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess Off

# Handling of file uploads
# TODO Choose a folder private to Apache.
# SecUploadDir /opt/apache-frontend/tmp/
SecUploadKeepFiles Off

# Debug log
SecDebugLog /var/log/apache2/modsec_debug.log
SecDebugLogLevel 0

# Serial audit log
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus ^5
SecAuditLogParts ABIFHZ
SecAuditLogType Serial
SecAuditLog /var/log/apache2/modsec_audit.log

# Maximum request body size we will
# accept for buffering
SecRequestBodyLimit 131072

# Store up to 128 KB in memory SecRequestBodyInMemoryLimit 131072
# Buffer response bodies of up to # 512 KB in length SecResponseBodyLimit 524288
</IfModule>

The ModSecurity2 module would be properly installed and configured as an Apache module.
3.All left is to restart Apache in order the new module and configurations to take effect.

debian-server:~# /etc/init.d/apache restart

Don’t forget to check the apache conf file for errors before restarting the Apache with the above command for that to happen issue the command:
debian-server:~# apache2ctl -t

If all is fine you should get as an output:

Syntax OK

4. Next to find out if the Apache ModSecurity2 module is enabled and already used by Apache as a mean of protection you,
you might want to check if the log files modsec_audit.log and modsec_debug.log files has grown and doesfeed a new content.
If they’re growing and you see messages concerning the operation of the ModSecurity2 Apache module that’s a sure sign all is fine.
5. As we have the Mod Security Apache module configured on our Debian Server, now we will need to apply some ModSecurity Core Rules .
In short ModSecurity Core Rules are some critical protection rules against attacks across almost every web architecture.
Another really neat thing about Core Rules (CRS) for ModSecurity is that they are written with a performance in mind.
So enabling this filter rules won’t be a too heavy load for your Apache server.

Here is how to install the core rules:

6. Download latest ModSecurity Code Rules

Download them from the following Code Rule url
At the time of writting this article the latest code rules are version modsecurity-crs_2.0.6.tar.gz

To download and install this rules issue some commands like:

debian-server:~# wget http://sourceforge.net/projects/mod-security/files/modsecurity-crs/0-CURRENT/modsecurity-crs_2.0.6.tar.gz/download
debian-server:~# cp -rpf ~/modsecurity-crs_2.0.6.tar.gz /etc/apache2/
debian-server:~# cd /etc/apache2/; tar -zxvvf modsecurity-crs_2.0.6.tar.gz

Besides physically storing the unarchived modsecirity-crs in your /etc/apache2 it’s also necessery to add to your Apache Ifmodule mod_security.c block of code the following two lines:

Include /etc/apache2/modsecurity-crs_2.0.6/*.conf
Include /etc/apache2/modsecurity-crs_2.0.6/base_rules/*.conf

Thus ultimately the configuration concerning ModSecurity in your Apache Server configuration should look like the following:

<IfModule mod_security2.c>
# Basic configuration options
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess Off

# Handling of file uploads
# TODO Choose a folder private to Apache.
# SecUploadDir /opt/apache-frontend/tmp/
SecUploadKeepFiles Off

# Debug log
SecDebugLog /var/log/apache2/modsec_debug.log
SecDebugLogLevel 0

# Serial audit log
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus ^5
SecAuditLogParts ABIFHZ
SecAuditLogType Serial
SecAuditLog /var/log/apache2/modsec_audit.log

# Maximum request body size we will
# accept for buffering
SecRequestBodyLimit 131072

# Store up to 128 KB in memory
SecRequestBodyInMemoryLimit 131072
SecRequestBodyInMemoryLimit 131072

# Buffer response bodies of up to
# 512 KB in length
SecResponseBodyLimit 524288
Include /etc/apache2/modsecurity-crs_2.0.6/*.conf
Include /etc/apache2/modsecurity-crs_2.0.6/base_rules/*.conf
</Ifmodule>

Once again you have to check if everything is fine with Apache configurations with:

debian-server:~# apache2ctl -t

If it’s showing once again an OK status. Then you’re ready to restart the Webserver.
debian-server:~# /etc/init.d/apache2 restart

One example goodness of setting up the ModSecurity + the Core rule sets are that after the above described installationis fully functional.

ModSecurity will be able to track if somebody tries to execute PHP Shell on your server .
ModSecurity will catch, log and block (forbid) requests to r99.txt, r59, safe0ver and possibly other hacked modifications of the php shell script

That’s it! Now Enjoy your tightened Apache Security and Hopefully catch the script kiddie trying to h4x0r yoU 🙂