Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

This is the fifth and final part of a five part tutorial that will show you how to install a full featured email server on your Raspberry Pi.

This tutorial covers how to automatically sort spam emails into the spam folder using Dovecot’s Local Mail Transfer Protocol (LMTP) and Sieve rules.

The parts are:
The Introduction & Contents Page (read first)
Raspberry Pi Email Server Part 1: Postfix
Raspberry Pi Email Server Part 2: Dovecot
Raspberry Pi Email Server Part 3: Squirrelmail
Raspberry Pi Email Server Part 4: Spam Detection with Spamassassin
Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

Intro

If you followed the previous tutorial, you currently have an email server that automatically scans incoming emails using Spamassassin. However, in its current state, Spam and Ham alike are delivered to the inbox, which is annoying. Since Spamassassin only marks emails based on their spam score, we need to use an external program to handle sorting & delivery.

The best tool for the job is Sieve: a set of rules defined by each user to determine how incoming emails are filtered. Sieve rules can sort emails based on a myriad of things: headers, the body of the email, various tags added by external programs, sender address… the list goes on.

In this case, we are going to use Sieve rules to send emails that have been marked by Spamassassin with spam flags like this (“X-Spam-Flag: YES“) in the headers straight into the Spam folder.

At present, emails are delivered to the inbox by Postfix. However, since Sieve is a Dovecot plugin, the final step of the delivery must be handled by Dovecot. This gives us two choices:

  1. Use Dovecot’s Local Delivery Agent (LDA)
  2. Use Dovecot’s Local Mail Transfer Protocol (LMTP)

For our setup, LDA is not ideal because it doesn’t run as root and therefore can’t access the individual inboxes of each user due to permissions. There are some other differences too…from the Dovecot Wiki:

The main difference is that the LDA is a short-running process, started as a binary from command line, while LMTP is a long-running process started by Dovecot’s master process.

In other words, with LDA a new process is started each time an email needs to be delivered, whereas with LMTP a process is always running and it handles a queue of emails. From what I have read on mailing lists etc. it seems that LMTP is more efficient.

So… LMTP it is!

Install & Configure Dovecot LMTP

First, install dovecot-lmtpd:

sudo apt-get update
sudo apt-get install dovecot-lmtpd

This will create a new config file at /etc/dovecot/conf.d/20-lmtp.conf.

Now to change the config in a few files:

/etc/dovecot/dovecot.conf

Append this to enable lmtp:

protocols = imap lmtp

/etc/dovecot/conf.d/20-lmtp.conf

Add this line to enable address extensions:

lmtp_save_to_detail_mailbox = yes

This means that if you send an email to you+folder@yourdomain.com it should be automatically placed in the “folder” folder. Cool, eh? You can use this for loads of things, but here’s a typical student example from a recent graduate ;) …

Change your email address for Pizza takeaway companies to you+pizza@yourdomain.com and create a folder called “pizza”. Now all your emails about pizza go into a separate folder, instead of cluttering your inbox. Awesome :)

Note: folder names are case sensitive, and the folder must be top level (not a folder within your inbox).

Now change the lmtp protocol block to look like this:

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = postmaster@yourdomain.com
}

/etc/dovecot/conf.d/10-master.conf

Now find the service lmtp {… block and then change the line unix_listener lmtp {… to look like this. This will allow postfix to access Dovecot’s LMTP from within its chroot:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0666
  }
}

/etc/dovecot/conf.d/10-auth.conf

By default, Dovecot will try to look up “you@yourdomain.com” in your user database, when it should be looking up just the first bit (“you”). This setting instructs Dovecot to strip the domain name before doing the lookup, and convert the username to all lowercase letters:

auth_username_format = %Ln

(the L is the lowercase part and the n drops the domain name).

/etc/dovecot/conf.d/10-director.conf

I’m not sure if this bit is necessary, but I commented out the “protocol lmtp {…” block completely.

/etc/postfix/main.cf

We still need to instruct Postfix to hand over control to Dovecot’s LMTP for the final stage of delivery.

Comment out:

mailbox_command=

…and add:

mailbox_transport = lmtp:unix:private/dovecot-lmtp

Sieve Rules

Dovecot's sieve is already installed, you can check by running:

sudo apt-get install dovecot-sieve

Now we need to change one more parameter in /etc/dovecot/conf.d/90-sieve.conf:

Uncomment this setting:

recipient_delimiter = +

We still need to reload/restart Postfix and Dovecot to make that all the changes are loaded:

sudo service postfix reload
sudo service dovecot reload

The default place to put the sieve script is in the user's home folder: ~/.dovecot.sieve.

Create it like this:

sudo nano /home/user/.dovecot.sieve

and add this:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" "YES" {
  fileinto "Spam";
  # Stop here - if there are other rules, ignore them for spam messages
  stop;
}

Or, if you want spam messages to be marked as read as well as moved:

require ["fileinto","imap4flags"];
if header :contains "X-Spam-Flag" "YES" {
        addflag "\\Seen";
        fileinto "Spam";
        stop;
}

Now chown the file to the owner of the mailbox, e.g.:

sudo chown sam:sam /home/user/.dovecot.sieve

When Spamassassin marks emails as Spam it adds X-Spam-Flag: YES to the headers. This rule checks the headers and sends mail to the spam folder if that flag exists.

Testing: GTUBE SPAM email

Here's how to send an email to your server using Telnet that will definitely be marked as spam. There's a neat trigger called GTUBE (Generic Trigger for Unsolicited Bulk Email) that is implemented in spamassassin.

All we need to do is send a message that contains this line in the body:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

We can use Telnet to send this email like we did when testing in parts 1 and 2:

feathers-mcgraw@Hobbs-T440s:~$ telnet yourdomain.com 25
Trying 192.168.1.174...
Connected to yourdomain.com.
Escape character is '^]'.
220 yourdomain.com ESMTP Postfix (Debian/GNU)
ehlo randomdomain.com
250-yourdomain.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: test@randomdomain.com
250 2.1.0 Ok
rcpt to: pi
250 2.1.5 Ok
data
354 End data with .
Subject: test spam email
This should set it off...
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
.
250 2.0.0 Ok: queued as DDFDEDA77
quit
221 2.0.0 Bye
Connection closed by foreign host.

That email should land in your Spam folder. Note that if you're using Squirrelmail you won't be automatically subscribed to the Spam folder, you have to add it yourself.

Thanks to Jens for finding this test!

Optional: Managesieve

Managesieve is a service that will allow you to log in remotely with a compatible email client and manage your sieve scripts.

First, we need to install the necessary package:

sudo apt-get update
sudo apt-get install dovecot-managesieved

You will notice that this creates a new configuration file: /etc/dovecot/conf.d/20-managesieve.conf, which you can leave as it is.

One more configuration change to make: open /etc/dovecot/dovecot.conf and add sieve to the protocols line:

protocols = imap lmtp sieve

and restart Dovecot:

sudo service dovecot restart

The managesieve service uses port 4190, so log in to your router's admin page and forward this port to your Pi.

You can now test if the service is running with telnet:

sam@samhobbs:/etc/dovecot$ telnet localhost 4190
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Dovecot (Ubuntu) Pigeonhole"
"SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave"
"NOTIFY" "mailto"
"SASL" "PLAIN LOGIN"
"STARTTLS"
"VERSION" "1.0"
OK "Dovecot (Ubuntu) ready."

If it's up and running, then you can try connecting with a compatible email client, like Kmail, the KDE email client that ships with distributions like Kubuntu. There is also a free software sieve plugin for Thunderbird that you might like to try if that's your client of choice.

To enable sieve in Kmail, go to:
Settings --> configure Kmail --> accounts --> Modify --> Filtering

Check "Server supports Sieve" and "Reuse host and login configuration".

kmail-sieve.png

Now you can select Settings --> "Manage Sieve Scripts" and edit your scripts using Kmail's editor, complete with syntax highlighting.

kmail-sieve-2.png

If you write multiple scripts, they will be stored in ~/sieve and the active one will be symlinked from ~/.dovecot.sieve. If you had a script at ~/.dovecot.sieve before you set up managesieve, it will be saved as dovecot.orig when you create a new one.

kmail-sieve-3.png

Thanks to the members of Kubuntuforums, without whom I would never have even heard of sieve rules!

Please leave a comment, I'd love to hear how you're getting on!

Type: 

Comments

Love your tutorial and followed it perfectly and I was getting e-mails just fine until I got to the installing the lmtp. Now e-mails are not coming in and when I check the /var/log/mail.log log I see the following entries...


Aug 6 18:57:44 mail postfix/lmtp[2901]: 7090542E78: to=, orig_to=, relay=none, delay=2113, delays=2113/0.11/0.08/0, dsn=4.4.1, status=deferred (connect to mail[private/dovecot-lmtp]: No such file or directory)
Aug 6 18:57:44 mail postfix/lmtp[2902]: C976E42E77: to=, relay=none, delay=2128, delays=2128/0.12/0.07/0, dsn=4.4.1, status=deferred (connect to mail[private/dovecot-lmtp]: No such file or directory)

I have no idea what the issue is and I've checked all the configuration files and they are all configured per your directions.

Any ideas?

Check the unix_listener path in 10-master.conf, I'm 90% sure the issue will be a typo in there, or the wrong path. Without setting the path to be inside Postfix's chroot, Postfix can't connect to Dovecot. Sam

Thanks, I did that and there was an error. But once I fixed it I was still having issues so I just decided to wipe the system and start over and everything worked out afterwards.

One thing I would suggest adding would be a section on adding and configuring the SquirrelMail "filters" buttons for use with Spamassassin.

Hi Sam,

It is exceedingly rare to find a howto on the internet that Just Works, and this is one of those gems. I had not trouble getting my mailserver up and running using your guide.

Thank you!

Jeroen

For people who have the error:

User doesn't exsist with lmpt

Change this value in /etc/dovecot/conf.d/10-auth.conf:

In 10-auth.conf I have the setting "auth_username_format = %Lu".

When changing it to "auth_username_format = %n" the local users can be
found but not the virtual users anymore.

Hi Sam,
too bad I just finished and everything seems to work fine (I'm not using SquirrelMail and plan to use RoundCube to have a calendar). Thank you for all, your work is great.
Are you planning to add a part 6 with an antivirus (ClamAV) and a part 7 for DKIM, SPF, DMARC?
Don't worry, replying no is fine too ;-)

Hey Sam

I just wanted to send my thanks. I've used your Pi email server tutorials from beginning to end and your DNS tutorial too. I have had no major issues at all. I am very pleased with the end result. Your advice is clear, thorough and easy to understand. In my opinion, given the general standard of tutorials I have seen on the internet, yours are a truly excellent resource. I especially appreciated the context you give to each step. I have learned a lot. You could easily be a teacher if you wanted!

Thanks again

Rob

Rob,

Thank you for the wonderful feedback. I'm glad you've learned lots, I have no formal training so this is just a hobby for me. Typing random things you don't understand into a terminal is not my idea of fun, so I always try to explain the why of it too - that's what makes the whole thing rewarding.

Hope you continue to find my tutorials useful!

Sam

Hi Sam,

Completed the lmtp-sieve tutorial, Thx again for everything.

One question:

You state the following after the telnet via port 25...
"That email should land in your Spam folder. Note that if you're using Squirrelmail you won't be automatically subscribed to the Spam folder, you have to add it yourself."

How do I add the Spam folder to Squirrelmail? I've looked online, but it seems the create a folder option inside Squirrelmail might not point to the Spam folder in Maildir...please point me in the right direction for this...thx

Jo

If I remembr correctly you need to subscribe to the folder in the squirrelmail settings. The folder should already exist in the Maildir if you did the Maildir creation part properly.

Sam

Hi Sam,

Found it! I did have Spam in Maildir. Then to subscribe I went to the Squirrelmail settings as suggested, see below:

I logged into Squirrel Mail-->Folders at the top
Scrolled down to Unsubscribe/ Subscribe
Scrolled down the list and subscribed to the "spam" folder

Shows up now in Squirrelmail...thx

Regards,

Jo

Hi Sam,
Love your work! Was just wondering if a shortcut for future users could be made by placing /home/user/.dovecot.sieve into /etc/skel/? Also, if we did this, would there be any problem with having to chown it as users are created?

Cheers,
Jack

That's a good idea! I think files in /etc/skel are automatically chowned, so it should work fine.

Someone else also mentioned using a system wide default sieve script. Not sure which option is best to be honest. Having individual scripts with everything in by default gives users more freedom and makes the whole thing transparent which is good, but it seems less elegant than one default config for everywhere - the advantage of that approach is if you want to change the script for all users, you only have to change it in one place. Maybe a global config along with a file in /etc/skel containing a comment like "edit this file to add your own personalised sieve rules" is the best of both.

Anyway, thanks for the interesting comment!

Sam

Hello again Sam,
The Pi server is running smoothly and SpamAssassin is picking up the spam (there is a lot isn't there...).
I am now up to the final leg, and have received the following error when installing dovecot-sieve:

Job for dovecot.service failed. See 'systemctl status dovecot.service' and 'journalctl -xn' for details.
invoke-rc.d: initscript dovecot, action "start" failed.
dpkg: error processing package dovecot-core (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of dovecot-sieve:
dovecot-sieve depends on dovecot-core (= 1:2.2.13-12~deb8u1); however:
Package dovecot-core is not configured yet.

dpkg: error processing package dovecot-sieve (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
dovecot-core
dovecot-sieve
E: Sub-process /usr/bin/dpkg returned an error code (1)

Do you have any suggestions? I am not sure how to open the log files if they would assist, so let me know and i'll do more digging.
Thanks again. I am just loving having my own server.
Spencer

Hi mate,

That's a strange error. Maybe we could try reconfiguring dovecot-core. This command:

sudo dpkg --configure -a

Should reconfigure any packages with problems. Let me know what happens!

Sam

I had this same issue. I figured out that when I commented out the “protocol lmtp {…” block in /etc/dovecot/conf.d/10-director.conf, I forgot to comment out the closing "}". As soon as I commented that out everything worked perfectly.

I know this is over 2 years later but hopefully this will help someone else out.

Hi Sam,
I have been looking for these instructions to setup email on raspberry pi. Finally I have found it. Thank you soooo much for creating and posting this knowledge. I have 2 questions:
1 - How many users or clients email boxes max can this server supports?
2 - How to make this Raspberry email server wireless access?

Happy New year 2016!
Thanks,
Nzola

You're welcome :)

1) You can add as many users as you like, each one is a system user with a login. You should be fine with about 10 users if the pi's only job is a mail server, more if it's a pi2. The space on your SD card is more likely to be a limiting factor.

2) There are loads of guides for setting up wifi elsewhere, just use one of those and then give the pi a static LAN IP address. I wouldn't recommend using wifi though, it's unreliable and slow. Ethernet cable all the way.

Sam

Hi,
Another cool option with delimiter and redirect mail to details folder, is lda_mailbox_autocreate from lda.
In 15-lda.conf, you can add lda_mailbox_autocreate = yes to make dovecot create the folder if it doesn't exist.

Will

Hi Sam,

I've been going through all the instructions again (manually checking everything) to see why my spam isn't landing in the spam folder. I found one manual entry error that I forgot to type in...

The other issue I seem to have is the /home/user/.dovecot.sieve part to place the sieve script in the default location. I don't seem to have the /home/user/ folder...

It looks like I created mine in /home/jo/.dovecot.sieve Will it still work? I tried creating it like you say as: sudo nano /home/user/.dovecot.sieve but it says that there is no such file or directory when I try to save it.

Any ideas?

Thanks

Jo

That path is just an example, where you replace user with your username, so yes that will work :)

Sam

Looks like this Sam:-

drwxr-xr-x 3 root root 4096 Feb 16 18:48 .
drwxr-xr-x 85 root root 4096 Feb 16 22:19 ..
-rw-r--r-- 1 root root 939 Jan 31 2015 65_debian.cf
-rw-r--r-- 1 root root 1289 Jan 31 2015 init.pre
-rw-r--r-- 1 root root 2215 Feb 16 19:14 local.cf
-rw-r--r-- 1 root root 118 Jan 31 2015 sa-compile.pre
drwxr-xr-x 2 root root 4096 Jan 31 2015 sa-update-hooks.d
-rw-r--r-- 1 root root 2524 Jan 31 2015 v310.pre
-rw-r--r-- 1 root root 1194 Jan 31 2015 v312.pre
-rw-r--r-- 1 root root 2416 Jan 31 2015 v320.pre
-rw-r--r-- 1 root root 1237 Jan 31 2015 v330.pre
-rw-r--r-- 1 root root 1020 Jan 31 2015 v340.pre

Maybe that 65_debian.cf file?

Nah, I have that one too, and it doesn't have any parameters in that are likely to be duplicates of the ones you are trying to change.

Did you reload spamassassin since you made the changes to the file?

Sam

Sure did. However, I've restarted and things seem to be picking up my local.cf configuration. :) I also realised that I hadn't given executable perms to my spamassassin-learn script. It would seem everything is working now!

Thanks as always Sam. :D

Hi there,
Wonderful instructions, I've succesfully managed to setup my webmail flawlessly. Thank you very much, it even feels quite safe after following your security advices.
However, I wanted to use roundcube to access my mail, and somehow after installation it won't log in. It keeps saying my either my name or password are wrong, but they work from another mail clients. I think there may be some piece of configuration of round cube that I'm missing. Any help?

Sorry, I might write a tutorial for Roundcube one day but not today.

Try changing your login between you and you@yourdomain.com and see if that makes a difference.

Sam

Pages

Add new comment