web analytics

Qmail – An introduction

Qmail
=====

Qmail is a mail transfer agent (MTA) for Unix like operating systems. When I say MTA, naturally the question comes – what is an MTA? For that, first we need to know how internt email works.

Suppose Robert wants to send a mail to Ann. He opens his email client, enters the from field, types in the subject and matter. Once he clicks the send button, the mail is sent. Similarly Ann receives it at her mailbox. As far as they both are concerned, the system works fine, very well as desired. But behind the scenes, many small programs work together making it possible for the email to be moved from Robert’s machine to Ann’s.

MTA(Mail Transfer Agent) is the program which moves the mail from the sender’s system to the recipient’s system. It does this with the help of some other programs called Mail User Agents(MUA). The email client (for eg. Outlook Express, Thunderbird etc) with the help of which Robert and Ann send and received mails are called Mail User Agents. A user does not internet with his MTA directly. Instead he uses the MUA for it. A user uses his MUA to send mail, which hand over the mail to his Mail Transfer Agent. This MTA in turn contacts the recipient’s MTA. When the recipient pops his mail using his email client, the MTA transfers the mail to his MUA. This is how the mail system works.

In early days of Internet there were not much internet users and as such the security risk was less. In modern internet there are millions of users exploring internet daily, billions of cash transactions take place daily and sendmail can longer support the security aspect. In 1995, Dan Bernstein, a mathematics student at the University of Columbia designed and implemented an MTA for modern internet and it was named Qmail. Qmail is just a built in replacement to sendmail.

Features of Qmail

Compared to Sendmail, Qmail has got a lot more features.

1)Security – Sendmail uses a single large complex program for all functions. As such a small bug in the program may lead to problems with the entire mail functions. Qmail has got a modular architecture and each function is performed by each module. These modules also do not trust the output from each other
2)Reliability – Qmail is said to be reliable. Suppose a power failure occurs leaving unsent messages in the queue. As soon as the power is restored and server is brought online, qmail starts delivery from where it had left the messages. Thus it ensures that no messages are lost.
3)Performance – Sendmail can perform mail deliveries only one at a time. Qmail parallelizes mail deliveries by delivering almost 20 mails at a time. Also, due to the modular architecture, the resources are utilized most efficiently.

Modular architecture

As said earlier, Qmail has got a modular architecture. This feature of qmail helps it to perform mail deliveries faster, utilizing minimum resources and in a fast and efficient way. The various qmail modules are :

qmail-inject – this module is responsible for reading a message from its standard output. It then hands over the message to the qmail-queue after adding the necessary headers
qmail-smtpd – it accepts mail via smtp
qmail-queue – this the qmail module responsible for qeueing messages.
qmail-send – this is the most important module which is responsible for mail transaction. It has got two functions – analyzing the recipients as local or remote and delivering the mails to them.

qmail-lspawn, qmail-local – these are the qmail modules responsible for local deliveries.
qmail-rspawn,qmail-remote – these are the qmail modules responsible for remote deliveries.
qmail-clean – this module cleans up the queue directory.

Qmail File Structure

/var/qmail is the root
alias – qmail files for system wide aliases
bin – program binaries and scripts
boot – startup scripts
control – configuration files
doc – documentation
man – man pages
queue – queue of unsent messages
users – the qmail users’ database files

Structure of Qmail queue

/var/qmail/queue is the root

mess* – message files
todo – the envelope, where the message came from, where it is going
intd – the envelope, under construction by qmail-queue
info* – the envelope sender address
local* – local envelope recipient addresses
remote* – remote envelope recipient addresses
bounce – permanent delivery errors

The asterisk symbol along with some of the folders show that these folders are further subdivided into small folders numbering from 0 to 22. This helps when the mail server is busy handling messages.

Configuration Files

/var/qmail/control holds the configuration files. Some of the important ones are:

1)me – holds the hostname
2)locals – it contains the list of domains for which the server should treat mails as local.
3)virtualdomains – list of domains for which this server is a backup mail server
4)rcpthosts – This is a very important file without which the server will be an open relay. This file contains the list of all domains for which the server should accept mail and the domains for which this server is a secondary mail server. In a nutshell, this file contains all the domains listed in locals and virtualdomains file.
5)morercpthosts – It has the same function as that of rcpthosts file. If your rcpthosts file contains a large number of domains, just keep the most important 50 domains in rcpthosts and keep the rest in morercpthosts.
6)badmailfrom – This file contains the domain names from which the server should reject mails.
7)concurrenylocal – Shows the maximum number of simultaneous local deliveries allowed.
8)concurrenyremote – Shows the maximum number of simultaneous remote deliveries allowed.
9)databytes – The maximum size of the message allowed
10) queuelifetime – The maximum time a message can stay in the queue.

How Qmail works

The working of qmail can be well depicted with the following diagram.
The qmail-smtpd and the qmail-inject modules are constantly looking for incoming messages. Qmail-inject checks for locally generated mails whereas qmail-smtpd checks for smtp connections. Once they receive a mail they pass it to qmail-queue for queuing the mails. qmail-queue puts a new message in the queue and pulls a trigger. The queue resides on disk. It’s typically in /var/qmail/queue and it has several subdirectories. When new message is received, qmail-queue creates a new file in queue/pid with a unique name. This new file has some i-number, say 123456. The new file is moved to mess/123456. qmail-queue writes the message contents to mess/123456. It writes the envelope information to intd/123456. It links intd/123456 to todo/123456. The message can now said to be queued.

Once the message is queued, the function of the receiving modules is over. qmail-queue then pulls a trigger to wake up qmail-send. qmail-send is always running, usually firing off deliveries. If it has nothing to do, it may go to sleep and wakes up when the trigger is pulled. It checks the todo folder for the new message and goes through it. It writes the sender information to info/123456. Qmail-send has two functions. It classifies the recipients as remote or local and then delivers the mail to it. If the recipient domain is present in the locals file or the virtualdomains file, then the recipient is local. Otherwise it is a remote recipient.

Let’s see how it handles a remote recipient. It the recipient is seen as remote, it writes it to remote/123456. Qmail-send then invokes qmail-rspawn which in turn invokes qmail-remote. It is the qmail-remote module that contacts the remote mail server for mail delivery. Whatever be the result, whether the delivery was successful or failure is send back to qmail-rspawn by qmail-remote, which in turn is send to qmail-send. Qmail-send then modifies qmail-remote accordingly. A sample remote/123456 will look as follows.

Dabc@abc.com^@Ttest@test.com^@….

The email addresses are separated by a null character and not by space, tab etc. The letter ‘D’ before the email address shows that the recipient has been done with, or that the mail has been delivered. ‘T’ means that it is still to-do. It will try for a mail delivery later and if successful will replace T with D. Once all the recipients are marked done, it will remove the remote/123456.

Local deliveries also follow the same method. The messages are passed to qmail-lspawn which in turn calls qmail-local and the results are passed viceversa.

Qmail Log Analysis

This is how the qmail log goes when the mail is accepted till its delivery.

a) TIMESTAMP new msg 12345
b) TIMESTAMP info msg 12345: bytes 3189 from <sender@domain.xyz> qp 27763 uid 46
c) TIMESTAMP starting delivery 37051: msg 12345 to local recip@domain
TIMESTAMP starting delivery 37051: msg 12345 to remote recip@domain
d) TIMESTAMP delivery 37051: success: did_1+0+0/
TIMESTAMP delivery 37051: deferral: Sorry,_I_wasn’t_able_to_establish_an_SMTP_connection._(#4.4.1)/
TIMESTAMP delivery 37051: failure: 1.2.3.4_does_not_like_recipient./Remote_host_said:_550_recip@domain_User_unknown/Giving_up_on_1.2.3.4./
e) TIMESTAMP end msg 12345

Let me explain the various steps.
a) TIMESTAMP new msg 12345
This message means that qmail-send has just become aware that the indicated message number exists in the queue. This usually happens shortly after qmail-queue is finished adding the message to the queue. The number 12345 is the message number. This number is the same as the file name present in the various subfolders of /var/qmail/queue related to that message. At a given time, you will see only a single message with the same message number. Another message is assigned this number only when the current one has done with it.

b) TIMESTAMP info msg 12345: bytes 3189 from <sender@domain.xyz> qp 27763 uid 46
12345 is the message number as said earlier. The email address is the sender address. The qp number (27763 in this example) is the process ID of the qmail-queue process which added the message to the queue. The uid number (46 in this exmple) is the numeric uid under which the qmail-queue process was

running when the message was added to the queue.

c) TIMESTAMP starting delivery 37051: msg 12345 to local recip@domain
TIMESTAMP starting delivery 37051: msg 12345 to remote recip@domain

This message means that a delivery attempt has started. The number 37051 is just a sequence number which is assigned to each delivery attempt. The message tells you whether the recipient is local or remote as well as the recipient address.

d) TIMESTAMP delivery 37051: success: did_1+0+0/
TIMESTAMP delivery 37051: deferral: Sorry,_I_wasn’t_able_to_establish_an_SMTP_connection._(#4.4.1)/
TIMESTAMP delivery 37051: failure: 1.2.3.4_does_not_like_recipient./Remote_host_said:_550_recip@domain_User_unknown/Giving_up_on_1.2.3.4./

This message tells the status of the message. There are three status codes – success, deferral and failure.
i)Success – For a local delivery, the success status code as above tells you that the mail was successfully delivered to one maildir, forwarded to 0 mail address and 0 programs were processed through. It depends on whether it found a .qmail file and what its contents were. In case of remote deliveries, these parameters will not be present. It will say only that qmail-remote could successfully connect to the remote mail server and the other mail server accepted the message. It cannot guarantee that the other mail server will deliver the message.
ii) Deferral – The deferral status means that the mail delivery met with a temporary failure and it will try again later. The extra information along with it will give more explanation on why it was not delivered.
iii)Failure – This means that all the delivery attempts met with a permanent failure and it will not try delivering the mail again. It will contain an extra information which will tell you why the delivery failed.

e) TIMESTAMP end msg 12345
With this the message 12345 will be deleted from the queue. Any occurance of the same message number again will indicate another message.

Qmail Monitoring Tools

There are various monitoring tools available for qmail. It helps you manage the queue in an efficient way as well as find out the culprits in case of spamming. Some of the monitoring tools are as follows.

a) Qmhandle :- It has got various options as follows. Just try qmhandle -option to see the result.

-a : try to send queued messages now
* -l : list message queues
* -L : list local message queue
* -R : list remote message queue
* -s : show some statistics
* -mN : display message number N
* -dN : delete message number N
*-Stext : delete all messages that have/contain text as Subject
* -D : delete all messages in the queue (local and remote)
* -N : list message numbers only

b) qmail-qstat :- Just try /var/qmail/bin/qmail-qstat. It will display the number of messages in the local queue as well as in the remote queue. If the number is too large then you can understand that there is some problem and can use the various other tools to find the reason.

c)qmail-qread :- /var/qmail/bin/qmail-qread is a wonderful utility which will show you the status of the mail send, whether it is local or remote and the email addresses.

Conclusion

This doc is only meant to provide a basic introduction to qmail. It will not contain any advanced features of qmail. Those who does not know what qmail is, may be benefitted from this. Others, please don’t lose what you have now 😀

One comment to Qmail – An introduction

  • Sherin  says:

    Excellent Article 🙂

Leave a reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>