Setting up Mailman with the Courier MTA

Most recently I went through the troublesome procedure of setting up Mailman 2.15 on a Server running Debian Wheezy and the Courier Mail Server MTA. While everything needed could be easily installed from the official Wheezy Repo, it did by far not work out-of-the-box. Getting the Mailman GUI served via an Apache Server was a five minute task but when it came to the email processing it clearly seems that Mailman is more optimized to work with the more widely used Postfix MTA than with my favorite, the Courier Mail Server. Nevertheless some research and a few moments of debugging will get you a working setup. I would like to briefly sum up my experiences as I couldn’t find a single working tutorial for this, but lots of bits and pieces.

Let’s begin with something easy and do the Apache configuration first. This is pretty straight forward as there is a well-documented sample configuration file in /etc/mailman/apache.conf that I mostly used. I ended up with this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

<VirtualHost *:80>
        ServerName lists.domain1.tld
        ServerAlias lists.domain2.tld
        DocumentRoot /var/www

        <Directory "/usr/lib/cgi-bin/mailman/">
                AllowOverride None
                Options ExecCGI
                AddHandler cgi-script .cgi
                Order Allow,Deny
                Allow from all
        </Directory>

        <Directory "/var/lib/mailman/archives/public/">
                Options FollowSymlinks
                AllowOverride None
                Order Allow,Deny
                Allow from all
                DirectoryIndex index.html
        </Directory>

        <Directory "/usr/share/images/mailman/">
                AllowOverride None
                Order Allow,Deny
                Allow from all
        </Directory>

        <Directory "/var/lib/mailman/archives/">
                Options FollowSymLinks
                AllowOverride None
        </Directory>

        Alias /pipermail/ /var/lib/mailman/archives/public/
        Alias /images/mailman/ /usr/share/images/mailman/
        ScriptAlias /admin /usr/lib/cgi-bin/mailman/admin
        ScriptAlias /admindb /usr/lib/cgi-bin/mailman/admindb
        ScriptAlias /confirm /usr/lib/cgi-bin/mailman/confirm
        ScriptAlias /create /usr/lib/cgi-bin/mailman/create
        ScriptAlias /edithtml /usr/lib/cgi-bin/mailman/edithtml
        ScriptAlias /listinfo /usr/lib/cgi-bin/mailman/listinfo
        ScriptAlias /options /usr/lib/cgi-bin/mailman/options
        ScriptAlias /private /usr/lib/cgi-bin/mailman/private
        ScriptAlias /rmlist /usr/lib/cgi-bin/mailman/rmlist
        ScriptAlias /roster /usr/lib/cgi-bin/mailman/roster
        ScriptAlias /subscribe /usr/lib/cgi-bin/mailman/subscribe
        ScriptAlias /mailman/ /usr/lib/cgi-bin/mailman/
</VirtualHost>

Note that you have to set the DocumentIndex Option to add index.html for the public archive directory, just in case you also unset this on the global level.

The configuration of Mailman itself is also not too challenging. There is just the one /etc/mailman/mm_cfg.py configuration file. There you need to set the DEFAULT_EMAIL_HOST=’lists.domain1.tld‘ and DEFAULT_URL_HOST=’lists.domain1.tld‘ to some working hosts before actually setting up the first mailing list. If you already set up the required ‚mailman‘ site list and you are having trouble with the default localhost.localdomain setting, then you can still easily fix this with [1]

1

withlist -l -a -r fix_url

Additionally I adjusted the DEFAULT_URL_PATTERN=’http://%s/mailman/‘ option to remove the cgi-bin part from the url, actually this is not important, it just has to be consistent with the Apache configuration. After this you can add the site list with

1

newlist mailman

and start the mailman daemon. You may just ignore the stuff about the aliases after creating the list or you may even go ahead and uncomment the MTA=None line in the mm_cfg.py to suppress this output, but I have not tested this. Now we should have Mailman running and Apache serving a working GUI.

In the next step we need to find a good solution to handle the incoming emails that need to be forwarded to the Mailman Server. While this theoretically can be done via aliases, this is quite troublesome and aliases have to be updated whenever a new list is created. There is a short note from 2003 on how this can be accomplished [2], however this failed due to a recent bug in the makealiases script [3]. The second solution that I came across involves forwarding all emails that Courier cannot deliver to any local mailbox or alias to a Python script courier-to-mailman.py [4] which is also around for more than a decade and recently has finally made it into the Mailman source [5], where you may download it (the script is found in the /contrib folder). I followed the pattern of the postfix-to-mailman.py script, copied the file to /etc/mailman/courier-to-mailman.py and linked it to be available from /usr/lib/mailman/bin/courier-to-mailman.py. While this is just a modified version of the postfix-to-mailman.py, it needed some tweaking in order to work properly. First of all, as I didn’t use the configure script from the source, you may have to manually insert the proper paths in the courier-to-mailman.py. For Wheezy, replace @prefix@ by /usr/lib/mailman and @VAR_PREFIX@ by /var/lib/mailman. Also, the hardcoded sendmail path didn’t work, but I am not quite sure if this is a general problem in Wheezy or just on my specific setup, where I had to adjust the path to /usr/sbin/sendmail. Last, ensure that the script is executable, which means including something like

1

#!/usr/bin/env python

If this is not already included (this depends on which version you downloaded) and setting permissions accordingly.

Now we have to configure the Courier MTA, to forward all the emails intended to be processed by Mailman to the script. Here I found the comments in the script somewhat misleading. First, the (Sub-)Domains receiving the emails have to be added to /etc/courier/esmtpacceptmailfor.dir/esmtpacceptmailfor and /etc/courier/hosteddomains. Don’t forget to run makeacceptmailfor and makehosteddomains and ensure that the configuration gets properly reloaded at this point. Then we simply create /etc/courier/aliasdir/.courier-default with the following content

1

|/usr/bin/preline /usr/lib/mailman/bin/courier-to-mailman.py

This forwards all incoming emails that cannot be delivered to any local mailbox or alias to the respective script. This method however comes with the disadvantage, that from now we accept all incoming emails with a „250 OK“ message and since we want to avoid producing backscatter spam we are not able to bounce undeliverable emails at a later point (depending on your use-case this might however be legally required). Theoretically we could also create a .courier-listname-default file with the very same content to reduce the amount of garbage emails we forward to the script and have them dropped earlier, but this file only processes emails for the pattern listname-xyz@lists.domain.tld. For Domains listed in hosteddomains it is not possible to have a /etc/courier/aliasdir/.courier-listname file which takes care of the emails addressed to listname@lists.domain.tld. For this to work we would have to list the respective Domain in /etc/courier/locals instead of hosteddomains, which would imply that all emails addressed to any-system-account@lists.domain.tld would be delivered to the corresponding system account [6]. This would again most likely increase the amount of garbage emails especially addressed to root@… or other well-known accounts. The further procedure however is the same for both methods. Ensure that the /etc/courier/aliasdir/ directory is owned by daemon or the respective user that is running courier, otherwise you will later get an „511 maildrop: Home directory owned by wrong user“ error [7]. The last thing we have to teach Courier in order to let Mailman send emails is to allow relaying via the local IPv6 address (::1). This also turned out to be somewhat tricky as there is a uncommon notation used in /etc/courier/smtpaccess/default [8] and there are mostly wrong solutions for this posted. This might has changed in current releases but in Courier 0.68.2 that comes with Wheezy the only solution that works is exactly adding the line

1

:0000:0000:0000:0000:0000:0000:0000:0001<TAB>allow,RELAYCLIENT

to smtpaccess/default – replace with an actual TAB, don’t use any additional spaces, multiple TABs etc. and don’t forget to run makesmtpaccess afterwards. Alternatively you may run Mailman using authenticated SMTP to send emails [9], but I have not tested this.

Go ahead and play around a little bit sending emails to mailman@lists.domain.tld, adding some externally hosted email addresses to this list and make sure everything is working. It is not? Well, then you might have stumbled upon one of those two problems:

Getting an „IOError: [Errno 13] Permission denied“ error in /var/log/mailman/error, stating an permission error in somewhere in the /var/lib/mailman/archives directory? Then make sure the entire archive directory is owned by the user list.

Playing around with aliases and getting an „456 Address temporarily unavailable“ error in your courier log, even after changing the config and restarting courier-mta? Then check the mailq, make sure to cancel all pending undeliverable messages with cancelmsg and the clean the /var/lib/courier/track directory. There, Courier keeps track of undeliverable addresses for a certain amount of time or until the message could be delivered and blocks any further incoming message to this address. This is especially annoying if you play around with different configurations.

Last, I implemented a host filter into courier-to-mailman.py, in the hope to decrease the wasted computing power on incoming garbage emails. The accepted Domains are hardcoded, but can be easily adjusted (see comments in the script). This is the final courier-to-mailman.py script that I came up with.

Finally there is to note positively that the update to Mailman 2.1.18 from the wheezy-backports went smooth as an usual Debian update does and negatively that Mailman uses a global namespace, which makes it impossible to provide two separate lists with the same name under two different domains running a single instance [10]. Although I don’t want to take care about this until the first naming conflict arises, I already stumbled upon a possible solution [11].

[1] https://www.progclub.org/blog/2012/02/01/mailman-fix_url-py/
[2] https://mail.python.org/pipermail/mailman-users/2003-March/027187.html
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=703570
[4] https://mail.python.org/pipermail/mailman-developers/2006-November/019274.html
[5] https://ftp.gnu.org/gnu/mailman/
[6] http://www.courier-mta.org/dot-courier.html
[7] http://ehc.ac/p/courier/mailman/message/18549088/
[8] http://www.courier-mta.org/couriertcpd.ht
[9] https://mail.python.org/pipermail/mailman-users/2005-October/047086.html
[10] http://wiki.list.org/DOC/4.47%20Virtual%20domain%20hosting%20with%20Mailman%3F
[11] https://code.launchpad.net/~msapiro/mailman/vhost

This article is licensed under CC BY-NC-SA 4.0

Unwrapping the Courier Filter Process

The Courier Mail Server comes along with some quite powerful filter and post-processing mechanisms. It turned out that the rather complex structure requires detailed knowledge about the possibilities of each step to figure out where to apply certain filtering or post-processing tasks for incoming emails. The overview I gained over the mechanisms during my approaches to solve the following tasks – First to filter incoming emails according to common DNS blacklists for certain accounts as well as alias addresses that are just forwarded, and Secound to automatically encrypt every incoming email for certain accounts – should be roughly summarized in this post.

The filter process can be divided into the filtering during receiving the email – at this point the filter’s main task is to decide whether to accept the email or drop the connection right away – and the post-processing of the email after fully receiving it. For the latter, the possibilities are basically endless, as this step may be carried out by any local Mail Delivery Agent (MDA), which is specified in the DEFAULTDELIVERY variable in /etc/courier/courier. In this case maildrop is used as the MDA.

In the first step the BLACKLISTS are queried and the results are reported by setting the specified environment variables. In the default example:

1

BLACKLISTS='-block=zen.spamhaus.org,BLOCK2 -block=cbl.abuseat.org,BLOCK2'

Spamhaus is queried and the BLOCK2 environment variable is set if the senders IP is listed. If the BLOCK environment variable is specified instead, esmtps immediatly drops the connection and refuses to receive the email. If this behaviour is too strict and one would additionally like to log the mails dropped, one may set any other unused environment varible, in most examples BLOCK2 is used.

The courierfilter supplies an API to external applications that may be enabled in /etc/courier/enablefiltering for specified receive channels, and soft-linked via the filterctl command to /etc/courier/filter/active/. As I have never used this mechanism as it is said to be highly buggy, I would recommend the official documentaion [1].

In the following esmtpd checks whether /etc/courier/maildropfilter exists. This file should contain the path to the maildrop binary that is included in the courier distribution, in this case it is /usr/bin/maildrop. If this file exists maildrop is executed in the embedded mode. In this mode the functionality of maildrop is somewhat limitted and some commands, such as xfilter, are not available. See [2] for a full list. However there is a workaround for this limitation, as it is possible to include any mailfilter script that resides within /etc/courier/maildroprcs/ for which the limitations are suspended [2]. In this step the file $HOME/.mailfilters/ rcptfilter is evaluated first, right after receiving the header of the email. If this filter returns with EXITCODE = 0 the email is accepted and the transfer of the DATA section is continued. If the EXITCODE = 99 the transfer of the DATA section is also continued but before it is acknowledged $HOME/.mailfilters/smtpfilter is evaluated and the email is only accepted if it returns EXITCODE = 0. In any other case the connection is dropped. At this point the BLOCK2 environment variable may be queried using the import statement. For alias addresses specified in /etc/courier/aliases/ that are just forwarded to a local account or an external address, there is usually no proper $HOME specified. In this case /etc/courier/aliasfilteracct may specify a $HOME for alias addresses. The $HOME given in /etc/courier/aliasfilteracct must be a valid $HOME of an existing user (no virtual user), otherwise courier will fail to evaluate any rcptfilter and exit with „400 stat() failed on aliasfilteracct“. For aliases only the rcptfilter can be used and files have to be named $HOME/.mailfilters/rcptfilter-alias-name, where name is the full alias address [3]. For local aliases which don’t name a domain /etc/courier/me is appended. Be aware that each period ‚.‘ in name must be replaced by a colon ‚:‘ e.g. rcptfilter-alias-sample@markuspetermann:net. One may also create an rcptfilter-default file which is used if no other file matches the alias. There are quite a few more features, especially regarding sub-addresses that won’t be covered at this point.

After the email has been fully received esmtpd invokes the Mail Delivery Agent, which is specified via the DEFAULTDELIVERY variable in /etc/courier/courierd. The default value points to ./Maildir which just places the email in the Maildir within the users $HOME. If one wishes to implement further post-processing of received emails, one may use maildrop in delivery mode to perform such tasks. This gives us rather endless possibilities to process and modify the incoming email. If maildrop is specified as local MDA it will first evaluate /etc/courier/maildroprc which contains the part of the maildrop script that applies globally to every incoming email. In addition to this one may place a $HOME/.mailfilter which contains any further part of the script that only applies to the emails of the specific user. At this point the BLOCK2 and other environment variables are not available anymore, as this is a completely different instance of maildrop running. Here, one may for example use the xfilter command to encrypt every incoming email (that is unencrypted) with the PGP key of the user, before delivering it.

[1] https://www.courier-mta.org/courierfilter.html
[2] https://www.courier-mta.org/maildrop.html
[3] https://www.courier-mta.org/localmailfilter.html

This article is licensed under CC BY-NC-SA 4.0