Server-side IMAP filtering done right

Several weeks ago, my colleague Paddy told me about a new software find of him: imapfilter. After several configuration sessions, we finally arrived at a configuration which promises to solve all mail filtering requirements of us, once and for all. Even better: almost without server support.

But what’s the problem in the first place? Today every decent mail program is able to apply filter rules — well mutt isn’t able, but delegates that task to other programs. Those of us, who don’t want to read their email through more or less capable web interfaces, tend to have multiple computers, smartphones, or other devices they use to access emails on their IMAP servers. At the same time we all face the dreaded problem of too many emails per today. In order to stay sane, I try to filter as much as possible. To do that, I have two choices, either to mantain a growing set of filtering rules on four different computers, using three different operating systems and three different mail programs. I’ll never do that. I am simply to lazy. The other alternative would be to rely on server-side filtering. Well yes, that seems to be ideal. I would have to maintain only one set of filter rules. So far the theory. In practice, every single implementation of server-side mail filtering — I have used so far — was either unreliable or simply not capable of expressing my filtering needs.

On the other side, I don’t want to maintain my own mail server. I did that years ago and it was quite some work to achieve the level of availability I expected.

Here, imapfilter enters the scene. It’s a simple Lua-based mail filtering engine, which connects to an IMAP server and does its work. Since it has idle support, it is able to handle emails almost instantanious. As if it were a filtering engine integrated into the mail server itself. The configuration of imapfilter is a simple piece of Lua code.

The last missing piece of this approach to fake server-side filtering, was drawn from the wish to have a clean interaction with programs like Apple’s Mail.app and Mozilla Thunderbird. A program which simply waits for new mails to arrive in the inbox and then move them, will not prohibit Mail.app or Thunderbird to pick up the new mail play and play some sound or show some popup. Just to notify only a split second later that the emails has already vanished. Being notified about every mail which is to be filtered away would destroy large parts of the benefit of true server-side filtering. To avoid this, we separate the inbox, which is watched by imapfilter from the inbox watched by your mail reader.

If your imap server has at least some mail filtering capabilities you can use that to deliver new mails into some folder, e.g. PreInbox, which is not shown by your mail reader. imapfilter will pick up mails here, filter them and move the rest to real inbox, which triggers your mail reader to bring it to your attention.

Requirements:

  • some IMAP server with poor mail filtering support
  • some other server capable of running imapfilter 24/7
  • imapfilter version 2.2.3
  1. Installs imapfilter. There is a package in Ubuntu, but I installed from source, because the version in Ubuntu was much older. I had to change the Makefile a bit, but I don’t remember the details at the moment. If you have problems installing contact me or ask the author of imapfilter to improve it’s build system.
  2. Create the configuration file in your home dir at “~/.imapfilter/config.lua”. See my example configuration below, the complete API is documented in the man page “imapfilter_config”. Obviously, you show change the server, username, and password. I’ve included some of my own rules to get you started. Keep the loops and the code after “end of the example rules”
    ---------------
    --  Options  --
    ---------------

    options.timeout = 120
    options.subscribe = true
    options.expunge = true

    ----------------
    --  Accounts  --
    ----------------

    account = IMAP {
      server = 'some-clever.server.com',
      username = 'foobar@some-clever.server.com',
      password = 'joshua',
      ssl = 'ssl3'
    }


    -----------------
    --  Main Loop  --
    -----------------

    -- if the server supports idle this will always be true
    repeat
      repeat

        -- save a list of all messages currently in there
        initial = account.PreInbox:select_all()

        --- begin of example rules
        res = account.PreInbox:contain_from('wordpress@gaillourdet.net')
        res:move_messages(account['Computer/Blog'])

        res = account.PreInbox:contain_to('glasgow-haskell-users@haskell.org')
          + account.PreInbox:contain_cc('glasgow-haskell-users@haskell.org')
        res:move_messages(account['Mailinglisten/GHC-Users'])

        res = account.PreInbox:contain_subject('[Haskell]')
        res:move_messages(account['Mailinglisten/Haskell'])

        res = account.PreInbox:contain_subject('[Haskell-cafe]')
        res:move_messages(account['Mailinglisten/Haskell-Cafe'])


        res = account.PreInbox:contain_to('vim_use@googlegroups.com')
          + account.PreInbox:contain_cc('vim_use@googlegroups.com')
          + account.PreInbox:contain_field('Reply-To','vim_use@googlegroups.com')
          + account.PreInbox:contain_cc('vim_mac@googlegroups.com')
          + account.PreInbox:contain_cc('vim_mac@googlegroups.com')
          + account.PreInbox:contain_field('Reply-To','vim_mac@googlegroups.com')
        res:mark_seen()
        res:move_messages(account['Mailinglisten/Vim'])

        --- end of example rules

        -----------------
        --  Remaining  --
        -----------------

        res = initial * account.PreInbox:select_all()
        res:move_messages(account.INBOX)

        res = account.PreInbox:select_all() - initial

      until #res == 0
    until not account.PreInbox:enter_idle()
  3. Configure your IMAP server to deliver all mails into the folder “PreInbox”, create it if necessary.
  4. Test your configuration: Execute imapfilter. It should run and apply your rules to every mail in PreInbox.
  5. Let upstart run imapfilter. Create the file “/etc/init/imapfilter.con” with the following contents. Replace “jmg” with your user name.
    description     "imapfilter lua script remotely sorting my mails"

    start on runlevel [2345]
    stop on runlevel [!2345]

    respawn
    respawn limit 1 10

    exec su -c 'imapfilter' jmg

That’s it! Now, you should enjoy a full featured, blazingly fast mail filtering capability with any old imap server indenpendent of any mail reader software. Since, I’ve deployed this configuration 4 weeks ago I’ve started to actually listen to my phone, when it plays a sound to notify me about new mail. Since it became such a rare event.

Don’t hesitate to contact me when you have question to this configuration.

Now, I am tired writing. So I post this article in this condition and will update it later.

This entry was posted in Linux and tagged , , , , , , , . Bookmark the permalink.

5 Responses to Server-side IMAP filtering done right

  1. Gustavo Cordova says:

    DUDE!! You really saved my ass. I have such a volume of email that it became just white noise, and started missing important messages from my users. I found imapfilter but was unsatisfied with the configuration I cooked up, but after reading your configuration, in the famous words of I-don’t-know-who: “It Suddenly Makes Absolute Sense!”.

    Thanks again!!

    -gus

  2. jmg says:

    You’re welcome. Thanks for your encouraging feedback.

  3. This seems like a complex solution that you may not really need.

    Many of the IMAP mail servers now have Sieve capability. Sieve is a standard for mail filtering and works well. Even better is that there is a remote service standard called ManageSieve that allows users to control their own scripts along side the server mandated global and per-user scripts.

    There are plugins for both Thunderbird and Roundcube (web mail) amongst others. On the server side, the Dovecot IMAP/POP mail server ccertainly supports these services.

    Regards,
    Julian Knight

  4. jmg says:

    Well, yes many decent IMAP server implementations support sieve, but only few available IMAP hosting services support it. That’s the reason why I’ve come to that solution.

  5. wedding says:

    I like to read gaillourdet.net often, keep doing fine work ! :)

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>