☩ Walking in Light with Christ – Faith, Computing, Diary

The aim of this small post is to give just a brief idea of how I fix my qmail server after breaking it or in case it is broken after mail bomb attacks, etc.

Most common cases when I break my qmail queue myself, are after I’m implementing some new patches and reinstall parts of the qmail server with a patched version of default qmail binaries.
On other occasions, I simply used the qmailctl to start or stop the server as a part of some routine tasks necessery for the administration of the qmail server.

Everybody who has already experience with qmail should have experienced, that qmail is very fragile and could break even with a simple changes, though if it works once it’s rock solid piece of mail servant.

Below I explain few ways I used through my days as a qmail sys admin to deal with broken or messed queues.

1. Fixing a broken qmail queue using automatic tools There are few handy tools which in most cases are able to solve issues with the queue, one very popular one isqueue-repair – check http://pyropus.ca/software/queue-repair/.
Installation of qmail-repair is dead easy, but it needs to be installed from source as no official debian package is available:

linux:/usr/local/src# wget http://pyropus.ca/software/queue-repair/queue-repair-0.9.0.tar.gz
linux:/usr/local/src# tar -xzvvf queue-repair-0.9.0.tar.gzdrwxr-xr-x charlesc/qcc 0 2003-10-22 16:54 queue-repair-0.9.0/
-rw-r--r-- charlesc/qcc 268 2003-10-22 16:54 queue-repair-0.9.0/TODO
-rw-r--r-- charlesc/qcc 1700 2003-10-22 16:54 queue-repair-0.9.0/CHANGELOG
-rw-r--r-- charlesc/qcc 18007 2003-10-22 16:54 queue-repair-0.9.0/COPYING
-rw-r--r-- charlesc/qcc 1098 2003-10-22 16:54 queue-repair-0.9.0/BLURB
-rwxr-xr-x charlesc/qcc 26286 2003-10-22 16:54 queue-repair-0.9.0/queue_repair.py

To check if there are issues fixable within the qmail queue it’s as easy as:

linux:/usr/local/src# cd queue-repair-0.9.0
linux:/usr/local/src/queue-repair-0.9.0# ./queue-repair -t
...
checking files...
checking queue/mess files...
checking split locations...

The tool will walk through the mail sub-directories containing mail queued files in /var/qmail/queue and will list any issues found.
It’s recommended that the qmail server is stopped before any queue modify operations are issued on the server:

linux:/usr/local/src# qmailctl stop
...

Further on in order to solve any found issues with the queue, there is the “-r”/repair option:

linux:/usr/local/src/queue-repair-0.9.0# ./queue-repair -r
...

Another tool which comes handy whether a repair of a messed qmail queue is needed is qmhandle – http://sourceforge.net/projects/qmhandle/

The use of qmhandle is also pretty easy, all one has to do is to follow the usual classical steps of a download the source & compile:

linux:/usr/local/src# wget https://www.pc-freak.net/files/qmhandle-1.3.2.tar.gz
linux:/usr/local/src# tar -zxvvf qmhandle-1.3.2
...
linux:/usr/local/src# cd qmhandle-1.3.2

Once again it’s necessery that the qmail server is stopped via its init script before qmHandle tool is used, e.g.:

linux:~# qmailctl stop
...

There is a difference between qmail queue repair tool and qmail handle , while qmail queue-repair tool is used to fix improper permissions of queued files with the qmail queue, qmhandle ‘s application is to completely delete the stored mail contents of a broken queue.

Deleting all the qmail queue content is in some cases the only option to fix the queue.
Often such a drastic measure is required after a heavy mail server overload, let’s say a result of spammers or caused by virus infected mail users which send a massive amounts of spam mails.

Thus at many cases when queue-repair was unable to solve a queue mess, I use qmhandble and sacrifice all the queued emails by completely wiping them out like so:

linux:/usr/local/src/qmhandle-1.3.2# ./qmhandle -D
...

Above command would eradicate all queued emails. Hopefully after the qmail server gets launched again with qmailctl start all the mail server operations should be back to normal.

Note that the use of qmhandle’s queue delete capabilities is pretty dangerous, if you forgot to stop the qmail server before issuing the above command!

Note that in order to use both qmHandle and queue-repair tools you will need to install python interpreter as both of the tools are written in python.

To check what is currently in the queue in Qmail, there are also native tools available, as you should probably know if you have dealt with qmail, e.g.:

debian:~# qmail-qstat
debian:~# qmail-qstat
messages in queue: 2
messages in queue but not yet preprocessed: 0

Often when there are problems with Qmail and more specificly with qmail server queue the qmail-qstat command does show messages in queue, however when an attempt to check what kind of messages are in the queue with qmail-qread no messages are shown, for instance below you see an example of that, even though qmail-qstat claims 2 messages are in the queue, qmail-qread is unable to list the messages:

debian:~# qmail-qread
debian:~#

If all is fine with qmail queue above’s qmail-qread command should have returned something similar to:

debian:~# qmail-qread
26 May 2011 07:46:47 GMT #659982 3517 <hipo@www.pc-freak.net>
remote somemail@gmail.nl
26 May 2011 07:46:47 GMT #659983 3517 <hipo@www.pc-freak.net>

2. Fixing qmail queue manually This is very dangerous initiative, so before you try anything, make sure that you know what you’re doing, the possibility that you make the situation worst if you attempt to tamper manually the qmail queue is quite high 🙂

However if you’re still convinced to try fixing it manually, take a look at /var/qmail/queue it’s very likely that there are permission issues with some of the queued files, in order to fix the situation it’s necessery that the following directories:

/var/qmail/queue/mess/
/var/qmail/queue/remote/
/var/qmail/queue/bounce
/var/qmail/queue/info

gets explored with midnight commander / mc or some kind of convenient file explorer.

If there are queued files owned by users different from qmailq and user group qmail , for instance if owned by the root user, a simple chown qmailq:qmail to the wrong permissions file, should be able to resolve the issues.

Apart from all I explain above, there are many other ways suggested online on howto clean a qmail queue, one very popular one is using James’s qfixq shell script.

This script as of this very date is not working on Debian based systems, the script is dedicated initially to run on Fedora and Redhat based Linuces

Moreover myy experience with qfixq was never successful.

One very important note which is often a cause of many problems, is always make sure you stop and start the qmail server with an interval of at least of 10 seconds.

I’ve managed many servers which after an immediate (undelayed) qmailctl stop and qmailctl start was unable to run the whole engine of the qmail server (and either email sending or email receiving was not properly working) afterwards.

In that cases many weird behaviours are common, consider this seriously if you deal with the qmail-queue, it might happen that even if you have fixed your qmail queue, after a restart the qmail might breaks up.
I’ve experienced this kind of oddities numerous times, thus when I do changes to qmail I always make sure I restart the server a couple of times (at least 5 times 😉 ) always with a good delay between the HUPs.

And as always with qmail prayer is always needed, this server is complex, you never know what will happen next 🙂