Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

This is the fifth and final part of a five part tutorial that will show you how to install a full featured email server on your Raspberry Pi.

This tutorial covers how to automatically sort spam emails into the spam folder using Dovecot’s Local Mail Transfer Protocol (LMTP) and Sieve rules.

The parts are:
The Introduction & Contents Page (read first)
Raspberry Pi Email Server Part 1: Postfix
Raspberry Pi Email Server Part 2: Dovecot
Raspberry Pi Email Server Part 3: Squirrelmail
Raspberry Pi Email Server Part 4: Spam Detection with Spamassassin
Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

Intro

If you followed the previous tutorial, you currently have an email server that automatically scans incoming emails using Spamassassin. However, in its current state, Spam and Ham alike are delivered to the inbox, which is annoying. Since Spamassassin only marks emails based on their spam score, we need to use an external program to handle sorting & delivery.

The best tool for the job is Sieve: a set of rules defined by each user to determine how incoming emails are filtered. Sieve rules can sort emails based on a myriad of things: headers, the body of the email, various tags added by external programs, sender address… the list goes on.

In this case, we are going to use Sieve rules to send emails that have been marked by Spamassassin with spam flags like this (“X-Spam-Flag: YES“) in the headers straight into the Spam folder.

At present, emails are delivered to the inbox by Postfix. However, since Sieve is a Dovecot plugin, the final step of the delivery must be handled by Dovecot. This gives us two choices:

  1. Use Dovecot’s Local Delivery Agent (LDA)
  2. Use Dovecot’s Local Mail Transfer Protocol (LMTP)

For our setup, LDA is not ideal because it doesn’t run as root and therefore can’t access the individual inboxes of each user due to permissions. There are some other differences too…from the Dovecot Wiki:

The main difference is that the LDA is a short-running process, started as a binary from command line, while LMTP is a long-running process started by Dovecot’s master process.

In other words, with LDA a new process is started each time an email needs to be delivered, whereas with LMTP a process is always running and it handles a queue of emails. From what I have read on mailing lists etc. it seems that LMTP is more efficient.

So… LMTP it is!

Install & Configure Dovecot LMTP

First, install dovecot-lmtpd:

sudo apt-get update
sudo apt-get install dovecot-lmtpd

This will create a new config file at /etc/dovecot/conf.d/20-lmtp.conf.

Now to change the config in a few files:

/etc/dovecot/dovecot.conf

Append this to enable lmtp:

protocols = imap lmtp

/etc/dovecot/conf.d/20-lmtp.conf

Add this line to enable address extensions:

lmtp_save_to_detail_mailbox = yes

This means that if you send an email to you+folder@yourdomain.com it should be automatically placed in the “folder” folder. Cool, eh? You can use this for loads of things, but here’s a typical student example from a recent graduate ;) …

Change your email address for Pizza takeaway companies to you+pizza@yourdomain.com and create a folder called “pizza”. Now all your emails about pizza go into a separate folder, instead of cluttering your inbox. Awesome :)

Note: folder names are case sensitive, and the folder must be top level (not a folder within your inbox).

Now change the lmtp protocol block to look like this:

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = postmaster@yourdomain.com
}

/etc/dovecot/conf.d/10-master.conf

Now find the service lmtp {… block and then change the line unix_listener lmtp {… to look like this. This will allow postfix to access Dovecot’s LMTP from within its chroot:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0666
  }
}

/etc/dovecot/conf.d/10-auth.conf

By default, Dovecot will try to look up “you@yourdomain.com” in your user database, when it should be looking up just the first bit (“you”). This setting instructs Dovecot to strip the domain name before doing the lookup, and convert the username to all lowercase letters:

auth_username_format = %Ln

(the L is the lowercase part and the n drops the domain name).

/etc/dovecot/conf.d/10-director.conf

I’m not sure if this bit is necessary, but I commented out the “protocol lmtp {…” block completely.

/etc/postfix/main.cf

We still need to instruct Postfix to hand over control to Dovecot’s LMTP for the final stage of delivery.

Comment out:

mailbox_command=

…and add:

mailbox_transport = lmtp:unix:private/dovecot-lmtp

Sieve Rules

Dovecot's sieve is already installed, you can check by running:

sudo apt-get install dovecot-sieve

Now we need to change one more parameter in /etc/dovecot/conf.d/90-sieve.conf:

Uncomment this setting:

recipient_delimiter = +

We still need to reload/restart Postfix and Dovecot to make that all the changes are loaded:

sudo service postfix reload
sudo service dovecot reload

The default place to put the sieve script is in the user's home folder: ~/.dovecot.sieve.

Create it like this:

sudo nano /home/user/.dovecot.sieve

and add this:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" "YES" {
  fileinto "Spam";
  # Stop here - if there are other rules, ignore them for spam messages
  stop;
}

Or, if you want spam messages to be marked as read as well as moved:

require ["fileinto","imap4flags"];
if header :contains "X-Spam-Flag" "YES" {
        addflag "\\Seen";
        fileinto "Spam";
        stop;
}

Now chown the file to the owner of the mailbox, e.g.:

sudo chown sam:sam /home/user/.dovecot.sieve

When Spamassassin marks emails as Spam it adds X-Spam-Flag: YES to the headers. This rule checks the headers and sends mail to the spam folder if that flag exists.

Testing: GTUBE SPAM email

Here's how to send an email to your server using Telnet that will definitely be marked as spam. There's a neat trigger called GTUBE (Generic Trigger for Unsolicited Bulk Email) that is implemented in spamassassin.

All we need to do is send a message that contains this line in the body:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

We can use Telnet to send this email like we did when testing in parts 1 and 2:

feathers-mcgraw@Hobbs-T440s:~$ telnet yourdomain.com 25
Trying 192.168.1.174...
Connected to yourdomain.com.
Escape character is '^]'.
220 yourdomain.com ESMTP Postfix (Debian/GNU)
ehlo randomdomain.com
250-yourdomain.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: test@randomdomain.com
250 2.1.0 Ok
rcpt to: pi
250 2.1.5 Ok
data
354 End data with .
Subject: test spam email
This should set it off...
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
.
250 2.0.0 Ok: queued as DDFDEDA77
quit
221 2.0.0 Bye
Connection closed by foreign host.

That email should land in your Spam folder. Note that if you're using Squirrelmail you won't be automatically subscribed to the Spam folder, you have to add it yourself.

Thanks to Jens for finding this test!

Optional: Managesieve

Managesieve is a service that will allow you to log in remotely with a compatible email client and manage your sieve scripts.

First, we need to install the necessary package:

sudo apt-get update
sudo apt-get install dovecot-managesieved

You will notice that this creates a new configuration file: /etc/dovecot/conf.d/20-managesieve.conf, which you can leave as it is.

One more configuration change to make: open /etc/dovecot/dovecot.conf and add sieve to the protocols line:

protocols = imap lmtp sieve

and restart Dovecot:

sudo service dovecot restart

The managesieve service uses port 4190, so log in to your router's admin page and forward this port to your Pi.

You can now test if the service is running with telnet:

sam@samhobbs:/etc/dovecot$ telnet localhost 4190
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Dovecot (Ubuntu) Pigeonhole"
"SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave"
"NOTIFY" "mailto"
"SASL" "PLAIN LOGIN"
"STARTTLS"
"VERSION" "1.0"
OK "Dovecot (Ubuntu) ready."

If it's up and running, then you can try connecting with a compatible email client, like Kmail, the KDE email client that ships with distributions like Kubuntu. There is also a free software sieve plugin for Thunderbird that you might like to try if that's your client of choice.

To enable sieve in Kmail, go to:
Settings --> configure Kmail --> accounts --> Modify --> Filtering

Check "Server supports Sieve" and "Reuse host and login configuration".

kmail-sieve.png

Now you can select Settings --> "Manage Sieve Scripts" and edit your scripts using Kmail's editor, complete with syntax highlighting.

kmail-sieve-2.png

If you write multiple scripts, they will be stored in ~/sieve and the active one will be symlinked from ~/.dovecot.sieve. If you had a script at ~/.dovecot.sieve before you set up managesieve, it will be saved as dovecot.orig when you create a new one.

kmail-sieve-3.png

Thanks to the members of Kubuntuforums, without whom I would never have even heard of sieve rules!

Please leave a comment, I'd love to hear how you're getting on!

Type: 

Comments

this is great.
i wish to do it by myself.

and include an email client instead or plus squirrelmail like http://www.claws-mail.org/
do u know for another email client for complement this proyect?
exists an kind of cpanel for admin all this? for people dont know SSH or final users?
I imagine your proyect as a end user product: the mail pi :)
--put it on kickstarter --

Hi!

You can use any email client you like with your server, just use the settings I gave you. Unless you mean another webmail program? The only one I have tested with the RasPi is squirrelmai. Since I upgraded my server to an Intel NUC I've been using Roundcube, which is nice, but I'm not sure how slow it would be on the Pi.

Kickstarting it is an interesting idea, but I think doing it yourself is a valuable experience, because you learn so much!

I don't know anything about Cpanel, I've never used a hosting company, preferring to do it myself at home.

I love your guide and I have my mailserver up and running! All I have left is to switch from my free host to my final hostname. Want to put together a guide for configuring the DNS settings? :)

In other words, with LMTP a new process is started each time an email needs to be delivered, whereas with LMTP a process is always running and it handles a queue of emails. From what I have read on mailing lists etc. it seems that LMTP is more efficient.

first LMTP should be LDA.

in /etc/dovecot/conf.d/10-director.conf, you shouldn't have to comment out the entire block. There are plenty of other places in the configuration with the contents commented out but the brackets left.

you didn't mention any changes to /etc/dovecot/conf.d/15-lda.conf. the header for 15-lda says "LDA specific settings (also used by LMTP)." In 20-lmtp it says to try to save to that folder if it exists, but 15-lda gives the option lda_mailbox_autocreate and lda_mailbox_autosubscribe. Likely these are only used for lda. I tested just now with a cron email, and the mail folder auto-created without the 15-lda settings altered.

Hello and congratulations! Thanks for pointing out the error, always nice to know someone is reading it carefully enough to notice! :)

I don't think I can do a guide on how to set up DNS because the process will depend on your DNS provider's website GUI, and will be different for each provider. What is it that you want to know about DNS specifically? I think I should probably write something generally about DNS A / PTR / MX records: why they're important, what they should look like, how they work and how to test them etc. Does that sound useful?

I think you're right about 10-director.conf. Doesn't seem to make a difference, but yours is probably better (more explicit).

As for 15-lda.conf, I don't think I made any changes. Your experiment is interesting, thanks for sharing!

Sam

Something general should be good. Something I didn't check well enough before I chose my registrar was that they don't seem to do their own DNS records (or I couldn't find it). I ended up searching around online for free dns providers, and that worked in the end. With some clicking around I see you're using namecheap for dns hosting, and they offer free dns services, which works out well. I also used this site: http://www.webmonkey.com/2010/02/set_up_a_dns_name_server/ as a template for filling out some of the required information.

My next piece of the project is to figure out how to get gmail to accept emails from me.

I think you mean dynamic DNS? If you had no DNS records at all then your domain name would not resolve to your IP address.

I'm not using dynamic DNS any more, because I paid my ISP a £5 admin fee to set up a static IP address.

Gmail may not be accepting your emails because they are coming from a dynamic IP address (dynamic ranges are often blacklisted), and also because you don't have a PTR record.

Sam

Hi again Sam

Thanks again for your help and excellent work :-)

Just a quickie: when I created this sieve filter script from the terminal it created it with owner root, so you have to chown it to the mailbox owner before (in my case) Thunderbird plugin would cooperate with it, or else I suppose configure the plugin to access it with its existing permissions.

Hello Sam !

First of all, I would like to congratulate you for this tutorial, which is very clear and easy to follow. Thanks to you I have a new mail server on my Raspberry Pi !

The only problem I had was with the SquirrelMail interface. I use Nginx instead of Apache2, and I already run a web host for a OwnCloud service, which can be reached by typing mydomain.com from the outside, or raspberry or the ip adress from my local network. My idea was to switch services by an URL adress like domain.com/owncloud or domain.com/squirrel. But I am not very comfortable with the notion of virtual hosts, I have to learn. The only thing I can do now is to have or a Nginx host for OwnCloud, or another one for SquirrelMail but not both at the same time. Quite annoying.

I read your page on Multiple Websites on Apache2, and you said that you have two different domain names : www.samhobbs.co.uk and tomhobbs.co.uk, even for the same IP adress, for the two websites. At the moment I have only one domain name.

Do you think it is possible to have both services on one domain name (with Apache or Nginx) ? Or there can be a conflict between the two different hosts listening on the same adress (domain.com) and the same ports (80 and 443), even if a part of the URL is different ? (And so I have to ask no-ip for another domain name)

Thanks a lot for your help, and thanks again for your very helpful tutorials !
Best regards from France !
Pierre

Hi Pierre!

Thanks for the feedback :)

It's definitely possible to do what you want on both Apache and Nginx. According to the Nginx documentation the equivalent of a Virtual Host in Nginx is a "Server Block".

BTW, I have chosen to configure Apache so webmail owncloud and statistics are all on different sub-domains (like mail.samhobbs.co.uk), but you can easily serve them at different URLs. In the squirrelmail tutorial that's what the "Alias /squirrelmail /usr/share/squirrelmail" line did before we commented out - almost like symlinking /usr/share/squirrelmail to /var/www/squirrelmail.

By the way, it's a bit misleading when we say there are two different hosts "listening" at ports, because it's not like each virtual host is a separate daemon. It's only Apache listening, and it decides which content to serve based on the header information in the browser request.

Sam

It is a great guide, I have followed and have managed to set up my raspberry pi, I have encountered some problems, bounce incoming message

<tst @ mydomainc.>: host Firemaker [private / dovecot-lmtp] said: 550 5.1.1
     <tst @ mydomain .---> User does not exist: tst@mydomain.--- (in reply to
     RCPT TO command)

Final-Recipient: rfc822; tst@mydomain.---
Action: failed
Status: 5.1.1
Remote-MTA: dns; Firemaker
Diagnostic-Code: smtp; 550 5.1.1 <tst @ mydomain .---> User does not exist:
     tst@mydomain.---

When comment the line "mailbox_transport = lmtp:unix:private/dovecot-lmtp" in /etc/postfix/main.cf incoming mail works..

i understand that the problem is dovecot lmtp, I surfed the net and it appears to be well, any suggestions?

Greetings and thanks

Hi!

Can you have a look in /var/log/mail.log and see if there's anything relevant?

Sam

The problem is that the raspberry install without domainname, i reconfigure the network respecting host.domain.com convention and mail works great, sorry for my bad english.

for future references, the log shows

Sep 20 15:49:57 firemaker postfix/cleanup[21500]: 491FB44323: message-id=<CANKacKnwtQ8iRWx=x8V5xGPVv6Ysm-_s6iYTyHfkqtz3t6jyJw@mail.gmail.com>
Sep 20 15:49:57 firemaker postfix/qmgr[21485]: 491FB44323: from=<micuenta@gmail.com>, size=2060, nrcpt=1 (queue active)
Sep 20 15:49:57 firemaker dovecot: lmtp(21507): Connect from local
Sep 20 15:49:57 firemaker postfix/lmtp[21506]: 491FB44323: to=<tst@dominio.mx>, relay=firemaker[private/dovecot-lmtp], delay=0.63, delays=0.08/0.12/0.25/0.17, dsn=5.1.1, status=bounced (host firemaker[private/dovecot-lmtp] said: 550 5.1.1 <tst@dominio.mx> User doesn't exist: tst@dominio.mx (in reply to RCPT TO command))
Sep 20 15:49:57 firemaker dovecot: lmtp(21507): Disconnect from local: Client quit (in reset)
Sep 20 15:49:57 firemaker postfix/cleanup[21500]: DE6E644322: message-id=<20140920204957.DE6E644322@firemaker>
Sep 20 15:49:57 firemaker postfix/qmgr[21485]: DE6E644322: from=<>, size=3918, nrcpt=1 (queue active)
Sep 20 15:49:57 firemaker postfix/bounce[21508]: 491FB44323: sender non-delivery notification: DE6E644322
Sep 20 15:49:57 firemaker postfix/qmgr[21485]: 491FB44323: removed
Sep 20 15:49:58 firemaker spamd[16235]: prefork: child states: II

Thanks

Hi Sam.
First of all thanks for the great and clear tutorial.
I'm new to Linux and the Raspberry but I came this far with your tutorial without any problems.
However the last step fails.
When I did the sudo apt-get install dovcecot-managesieved I got an error.
E: Unable to locate package dovcecot-managesieved
I searched the internet for answers but that didn't take me further than that the package is not in the repository.
Don't know how to proceed. Hopefully you know the answer.

Greetings

Hi,

I misspelled Dovecot in the package name and have corrected it now: the correct package name is dovecot-managesieved.

Thanks for your comment!

Sam

Thanks for your quick response.
I should have seen the typo but I didn't :(
But now it works.
Thanks again.

If something like that happens again when you're following a guide, you can search for packages using this command:

apt-cache search keyword

For example,

apt-cache search sieve

Would have turned up a list of packages related to sieve. Obviously if it's a guide on this site then please tell me, but sometimes that isn't possible on other sites.

Sam

Hello!

I had a problem with getting sieve to create a folder and shunt an email to it using LMTP. Investigating the server logs, I get errors like this:


error: msgid=: failed to store into mailbox 'trash': Mailbox doesn't exist: trash.

How does one create folders and subfolders for the mail directory using the IMAP server?

Hi Colin

It depends whether you're asking about automatic mailbox creation or not. To create a mailbox manually you can either use your email client, or from the shell use this command:

sudo maildirmake.dovecot /home/you/Maildir/.Trash

However, if you followed part 1 of the tutorial you should already have that folder. Careful with the sieve scripts, I think they're case sensitive.

Sam

Sam,

You are correct, I have those folders; however, when I send an email to user+folder@mydomain.com, even if that folder exists, the email goes to my Inbox in Thunderbird. I suspect this is a problem with the Sieve plugin, as it hangs when I try to modify any message filters with Sieve. Do you have any experience with this plugin?

Colin,

It shouldn't be the sieve plugin, because the the sieving actually happens on the server before your email client even fetches the email with IMAP... the plugin could have created a bad sieve script, of course.

Can you cat your sieve script for me please? I will look into this in more detail when I get home, I don't have shell access to my server at the moment.

Sam

Hi Sam,

This is a really great tutorial, thanks for spending the time putting it
together.

I have the same problem exactly as Mario who posted on 20th September.
I'm getting the same bounce issue but when I comment out mailbox_transport
= lmtp:unix:private/dovecot-lmtp" in /etc/postfix/main.cf incoming mail
works.

I'm trying to figure out how he fixed it but it's a bit unclear.. any ideas?

Thanks

I'm not exactly sure what he meant either, sorry. Can you paste your service lmtp { block from /etc/dovecot/conf.d/10-master.conf please?

Sam

I had an issue with the same symptoms, and solved it with
sudo chown postfix:postfix /var/spool/postfix/private/dovecot-lmtp

I'm very much a beginner so the mistake is probably mine, but following this guide, that file was created root:root and was causing issues.

Although that may have solved your problem, I don't think it's the right way to do it - on my machine, that file has permissions 666 and is owned by root:root.

The LMTP process itself runs as root, so it makes sense that root owns the file not postfix.

I wonder if there's some small syntax error you (and others) have made - I've had quite a few other people who have followed the tutorial and had no problems, so it seems unlikely that it should work for most people and break for just a few with no differences.

Sam

Had a nightmare getting spam assassin to mark the emails, this page, specifically the part labelled "Debian bug #739738" seems to have resolved it.
https://workaround.org/ispmail/jessie/spamassassin

As I say, I'm rather new to Linux in general and don't have a good understanding of what this fix is actually doing, but thought it was relevant to this guide.

That's interesting, thanks - it seems to be a fix for a different problem to the first one we were discussing though? Or does this solve the initial problem?

Hopefully they will fix the bug and release a new version of the package soon. In the meantime, if I find anyone else with problems on Jessie I'll point them this way.

Sam

It is for a different issue, sorry should have made that more clear.
With the amount of issues I seem to find, even when following a comprehensive guide, I should consider a career in software testing.

I have almost everything working now but cannot get spam assassin to start properly,
"sudo service spamassassin start" seems to do nothing, while using
"sudo service spamassassin restart" loads it perfectly.
This wouldn't be too much of an issue, but it is also causing spam assassin not to function after rebooting.
Any ideas?

I had the same problem with spamassassin not reloading after cron update and first start on reboot. My solution was to force a second activation of the start despite the system told met that it was activated on boot.

systemctl enable spamassassin.service

Pages

Add new comment