Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

This is the fifth and final part of a five part tutorial that will show you how to install a full featured email server on your Raspberry Pi.

This tutorial covers how to automatically sort spam emails into the spam folder using Dovecot’s Local Mail Transfer Protocol (LMTP) and Sieve rules.

The parts are:
The Introduction & Contents Page (read first)
Raspberry Pi Email Server Part 1: Postfix
Raspberry Pi Email Server Part 2: Dovecot
Raspberry Pi Email Server Part 3: Squirrelmail
Raspberry Pi Email Server Part 4: Spam Detection with Spamassassin
Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

Intro

If you followed the previous tutorial, you currently have an email server that automatically scans incoming emails using Spamassassin. However, in its current state, Spam and Ham alike are delivered to the inbox, which is annoying. Since Spamassassin only marks emails based on their spam score, we need to use an external program to handle sorting & delivery.

The best tool for the job is Sieve: a set of rules defined by each user to determine how incoming emails are filtered. Sieve rules can sort emails based on a myriad of things: headers, the body of the email, various tags added by external programs, sender address… the list goes on.

In this case, we are going to use Sieve rules to send emails that have been marked by Spamassassin with spam flags like this (“X-Spam-Flag: YES“) in the headers straight into the Spam folder.

At present, emails are delivered to the inbox by Postfix. However, since Sieve is a Dovecot plugin, the final step of the delivery must be handled by Dovecot. This gives us two choices:

  1. Use Dovecot’s Local Delivery Agent (LDA)
  2. Use Dovecot’s Local Mail Transfer Protocol (LMTP)

For our setup, LDA is not ideal because it doesn’t run as root and therefore can’t access the individual inboxes of each user due to permissions. There are some other differences too…from the Dovecot Wiki:

The main difference is that the LDA is a short-running process, started as a binary from command line, while LMTP is a long-running process started by Dovecot’s master process.

In other words, with LDA a new process is started each time an email needs to be delivered, whereas with LMTP a process is always running and it handles a queue of emails. From what I have read on mailing lists etc. it seems that LMTP is more efficient.

So… LMTP it is!

Install & Configure Dovecot LMTP

First, install dovecot-lmtpd:

sudo apt-get update
sudo apt-get install dovecot-lmtpd

This will create a new config file at /etc/dovecot/conf.d/20-lmtp.conf.

Now to change the config in a few files:

/etc/dovecot/dovecot.conf

Append this to enable lmtp:

protocols = imap lmtp

/etc/dovecot/conf.d/20-lmtp.conf

Add this line to enable address extensions:

lmtp_save_to_detail_mailbox = yes

This means that if you send an email to you+folder@yourdomain.com it should be automatically placed in the “folder” folder. Cool, eh? You can use this for loads of things, but here’s a typical student example from a recent graduate ;) …

Change your email address for Pizza takeaway companies to you+pizza@yourdomain.com and create a folder called “pizza”. Now all your emails about pizza go into a separate folder, instead of cluttering your inbox. Awesome :)

Note: folder names are case sensitive, and the folder must be top level (not a folder within your inbox).

Now change the lmtp protocol block to look like this:

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = postmaster@yourdomain.com
}

/etc/dovecot/conf.d/10-master.conf

Now find the service lmtp {… block and then change the line unix_listener lmtp {… to look like this. This will allow postfix to access Dovecot’s LMTP from within its chroot:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0666
  }
}

/etc/dovecot/conf.d/10-auth.conf

By default, Dovecot will try to look up “you@yourdomain.com” in your user database, when it should be looking up just the first bit (“you”). This setting instructs Dovecot to strip the domain name before doing the lookup, and convert the username to all lowercase letters:

auth_username_format = %Ln

(the L is the lowercase part and the n drops the domain name).

/etc/dovecot/conf.d/10-director.conf

I’m not sure if this bit is necessary, but I commented out the “protocol lmtp {…” block completely.

/etc/postfix/main.cf

We still need to instruct Postfix to hand over control to Dovecot’s LMTP for the final stage of delivery.

Comment out:

mailbox_command=

…and add:

mailbox_transport = lmtp:unix:private/dovecot-lmtp

Sieve Rules

Dovecot's sieve is already installed, you can check by running:

sudo apt-get install dovecot-sieve

Now we need to change one more parameter in /etc/dovecot/conf.d/90-sieve.conf:

Uncomment this setting:

recipient_delimiter = +

We still need to reload/restart Postfix and Dovecot to make that all the changes are loaded:

sudo service postfix reload
sudo service dovecot reload

The default place to put the sieve script is in the user's home folder: ~/.dovecot.sieve.

Create it like this:

sudo nano /home/user/.dovecot.sieve

and add this:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" "YES" {
  fileinto "Spam";
  # Stop here - if there are other rules, ignore them for spam messages
  stop;
}

Or, if you want spam messages to be marked as read as well as moved:

require ["fileinto","imap4flags"];
if header :contains "X-Spam-Flag" "YES" {
        addflag "\\Seen";
        fileinto "Spam";
        stop;
}

Now chown the file to the owner of the mailbox, e.g.:

sudo chown sam:sam /home/user/.dovecot.sieve

When Spamassassin marks emails as Spam it adds X-Spam-Flag: YES to the headers. This rule checks the headers and sends mail to the spam folder if that flag exists.

Testing: GTUBE SPAM email

Here's how to send an email to your server using Telnet that will definitely be marked as spam. There's a neat trigger called GTUBE (Generic Trigger for Unsolicited Bulk Email) that is implemented in spamassassin.

All we need to do is send a message that contains this line in the body:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

We can use Telnet to send this email like we did when testing in parts 1 and 2:

feathers-mcgraw@Hobbs-T440s:~$ telnet yourdomain.com 25
Trying 192.168.1.174...
Connected to yourdomain.com.
Escape character is '^]'.
220 yourdomain.com ESMTP Postfix (Debian/GNU)
ehlo randomdomain.com
250-yourdomain.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: test@randomdomain.com
250 2.1.0 Ok
rcpt to: pi
250 2.1.5 Ok
data
354 End data with .
Subject: test spam email
This should set it off...
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
.
250 2.0.0 Ok: queued as DDFDEDA77
quit
221 2.0.0 Bye
Connection closed by foreign host.

That email should land in your Spam folder. Note that if you're using Squirrelmail you won't be automatically subscribed to the Spam folder, you have to add it yourself.

Thanks to Jens for finding this test!

Optional: Managesieve

Managesieve is a service that will allow you to log in remotely with a compatible email client and manage your sieve scripts.

First, we need to install the necessary package:

sudo apt-get update
sudo apt-get install dovecot-managesieved

You will notice that this creates a new configuration file: /etc/dovecot/conf.d/20-managesieve.conf, which you can leave as it is.

One more configuration change to make: open /etc/dovecot/dovecot.conf and add sieve to the protocols line:

protocols = imap lmtp sieve

and restart Dovecot:

sudo service dovecot restart

The managesieve service uses port 4190, so log in to your router's admin page and forward this port to your Pi.

You can now test if the service is running with telnet:

sam@samhobbs:/etc/dovecot$ telnet localhost 4190
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Dovecot (Ubuntu) Pigeonhole"
"SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave"
"NOTIFY" "mailto"
"SASL" "PLAIN LOGIN"
"STARTTLS"
"VERSION" "1.0"
OK "Dovecot (Ubuntu) ready."

If it's up and running, then you can try connecting with a compatible email client, like Kmail, the KDE email client that ships with distributions like Kubuntu. There is also a free software sieve plugin for Thunderbird that you might like to try if that's your client of choice.

To enable sieve in Kmail, go to:
Settings --> configure Kmail --> accounts --> Modify --> Filtering

Check "Server supports Sieve" and "Reuse host and login configuration".

kmail-sieve.png

Now you can select Settings --> "Manage Sieve Scripts" and edit your scripts using Kmail's editor, complete with syntax highlighting.

kmail-sieve-2.png

If you write multiple scripts, they will be stored in ~/sieve and the active one will be symlinked from ~/.dovecot.sieve. If you had a script at ~/.dovecot.sieve before you set up managesieve, it will be saved as dovecot.orig when you create a new one.

kmail-sieve-3.png

Thanks to the members of Kubuntuforums, without whom I would never have even heard of sieve rules!

Please leave a comment, I'd love to hear how you're getting on!

Type: 

Comments

I’m struggling to get my head around permissions a bit.

From what I can figure out the command will be:
sudo ls -s /home/callum/Mailbox /not/mail/callum

Would that be correct? What would the owner/permissions need to be set as? I’ve got the mail usb mounted and fstabbed already so it’s just making this symlink, I’ve never done it before.

Cheers

Ok, so I would first format it ext4 so the filesystem supports linux permissions, and modify /etc/fstab to suit, and then:

Make sure you own the whole flash drive

sudo chown -R callum:callum /mnt/mail

Copy your existing maildir to the flash drive and make sure it's only readable by you

cp ~/Maildir /mnt/mail/Maildir
chmod 700 /mnt/mail/Maildir

Move your old maildir and replace it with a symlink to the folder on the flash drive:

mv ~/Maildir ~/Maildir.bak
ln -s /mnt/mail/Maildir ~/Maildir

Test.

Note that these instructions assume you are the only user whose Maildir you want to store on the flash drive.

Sam

Hi Sam

Thanks for your reply. I've tried the above but now I'm not able to send or receive any emails from this account. It can see the mailbox content which it had previously but doesn't look like it's updating it with anything new.

Cheers

Callum,

Run tail -f /var/log/mail.log and send yourself a test email. See if there are any relevant log messages?

Sam

I’m getting a warning from postfix/smtpd, not enough free space in mail quite : 0bytes.
Also from postfix/cleanup: write queue file no space left on device.

Cheers

Hello,

I've followed your guide, it is very thorough. I'm currently testing my spam assassin service. Unfortunately when I try to send an email from randomdomain.com the server blocks me:

root@host:~$ openssl s_client -connect domain --quiet
depth=...
verify return:1
depth=...
verify return:1
depth=...
verify return:1
220 localhost ESMTP Postfix (Raspbian)
ehlo randomdomain.com
250-localhost
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: hello@randomdomain.com
250 2.1.0 Ok
Subject: Hello
221 2.7.0 Error: I can break rules, too. Goodbye.

I've tried sending myself and email via squirrelmail but it did not put the spam mail in the spam folder

a quick question, why openssl s_client -connect domain doesn't work when I'm running it from my own mail server?

Thank you very much

Hi,

The server closed the session because you missed the RCPT TO: step (you broke the SMTP rules).

'Domain' is not fully qualified so how would openssl know you mean localhost?

Sam

Pages

Add new comment