Server-side IMAP filtering done right

Posted on May 22, 2011

Several weeks ago, my colleague Paddy told me about a new software find of him: imapfilter. After several configuration sessions, we finally arrived at a configuration which promises to solve all mail filtering requirements of us, once and for all. Even better: almost without server support.

But what’s the problem in the first place? Today every decent mail program is able to apply filter rules — well mutt isn’t able, but delegates that task to other programs. Those of us, who don’t want to read their email through more or less capable web interfaces, tend to have multiple computers, smartphones, or other devices they use to access emails on their IMAP servers. At the same time we all face the dreaded problem of too many emails per today. In order to stay sane, I try to filter as much as possible. To do that, I have two choices, either to mantain a growing set of filtering rules on four different computers, using three different operating systems and three different mail programs. I’ll never do that. I am simply to lazy. The other alternative would be to rely on server-side filtering. Well yes, that seems to be ideal. I would have to maintain only one set of filter rules. So far the theory. In practice, every single implementation of server-side mail filtering — I have used so far — was either unreliable or simply not capable of expressing my filtering needs.

On the other side, I don’t want to maintain my own mail server. I did that years ago and it was quite some work to achieve the level of availability I expected.

Here, imapfilter enters the scene. It’s a simple Lua-based mail filtering engine, which connects to an IMAP server and does its work. Since it has idle support, it is able to handle emails almost instantanious. As if it were a filtering engine integrated into the mail server itself. The configuration of imapfilter is a simple piece of Lua code.

The last missing piece of this approach to fake server-side filtering, was drawn from the wish to have a clean interaction with programs like Apple’s and Mozilla Thunderbird. A program which simply waits for new mails to arrive in the inbox and then move them, will not prohibit or Thunderbird to pick up the new mail play and play some sound or show some popup. Just to notify only a split second later that the emails has already vanished. Being notified about every mail which is to be filtered away would destroy large parts of the benefit of true server-side filtering. To avoid this, we separate the inbox, which is watched by imapfilter from the inbox watched by your mail reader.

If your imap server has at least some mail filtering capabilities you can use that to deliver new mails into some folder, e.g. PreInbox, which is not shown by your mail reader. imapfilter will pick up mails here, filter them and move the rest to real inbox, which triggers your mail reader to bring it to your attention.

Requirements: - some IMAP server with poor mail filtering support
  1. Installs imapfilter. There is a package in Ubuntu, but I installed from source, because the version in Ubuntu was much older. I had to change the Makefile a bit, but I don’t remember the details at the moment. If you have problems installing contact me or ask the author of imapfilter to improve it’s build system.

  2. Create the configuration file in your home dir at “~/.imapfilter/config.lua”. See my example configuration below, the complete API is documented in the man page “imapfilter_config”. Obviously, you show change the server, username, and password. I’ve included some of my own rules to get you started. Keep the loops and the code after “end of the example rules”

--  Options  --

options.timeout = 120
options.subscribe = true
options.expunge = true

--  Accounts  --

account = IMAP {
  server = '',
  username = '',
  password = 'joshua',
  ssl = 'ssl3'

--  Main Loop  --

-- if the server supports idle this will always be true

    -- save a list of all messages currently in there
    initial = account.PreInbox:select_all()

    --- begin of example rules
    res = account.PreInbox:contain_from('')

    res = account.PreInbox:contain_to('')
	    + account.PreInbox:contain_cc('')

    res = account.PreInbox:contain_subject('[Haskell]')

    res = account.PreInbox:contain_subject('[Haskell-cafe]')

    res = account.PreInbox:contain_to('')
	    + account.PreInbox:contain_cc('')
	    + account.PreInbox:contain_field('Reply-To','')
	    + account.PreInbox:contain_cc('')
	    + account.PreInbox:contain_cc('')
	    + account.PreInbox:contain_field('Reply-To','')

    --- end of example rules

    --  Remaining  --

    res = initial * account.PreInbox:select_all()

    res = account.PreInbox:select_all() - initial

  until #res == 0
until not account.PreInbox:enter_idle()
  1. Configure your IMAP server to deliver all mails into the folder “PreInbox”, create it if necessary.
  2. Test your configuration: Execute imapfilter. It should run and apply your rules to every mail in PreInbox.
  3. Let upstart run imapfilter. Create the file “/etc/init/imapfilter.con” with the following contents. Replace “jmg” with your user name.
description     "imapfilter lua script remotely sorting my mails"

start on runlevel [2345]
stop on runlevel [!2345]

respawn limit 1 10

exec su -c 'imapfilter' jmg

That’s it! Now, you should enjoy a full featured, blazingly fast mail filtering capability with any old imap server indenpendent of any mail reader software. Since, I’ve deployed this configuration 4 weeks ago I’ve started to actually listen to my phone, when it plays a sound to notify me about new mail. Since it became such a rare event.

Don’t hesitate to contact me when you have question to this configuration.

Now, I am tired writing. So I post this article in this condition and will update it later.