Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

This is the fifth and final part of a five part tutorial that will show you how to install a full featured email server on your Raspberry Pi.

This tutorial covers how to automatically sort spam emails into the spam folder using Dovecot’s Local Mail Transfer Protocol (LMTP) and Sieve rules.

The parts are:
The Introduction & Contents Page (read first)
Raspberry Pi Email Server Part 1: Postfix
Raspberry Pi Email Server Part 2: Dovecot
Raspberry Pi Email Server Part 3: Squirrelmail
Raspberry Pi Email Server Part 4: Spam Detection with Spamassassin
Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

Intro

If you followed the previous tutorial, you currently have an email server that automatically scans incoming emails using Spamassassin. However, in its current state, Spam and Ham alike are delivered to the inbox, which is annoying. Since Spamassassin only marks emails based on their spam score, we need to use an external program to handle sorting & delivery.

The best tool for the job is Sieve: a set of rules defined by each user to determine how incoming emails are filtered. Sieve rules can sort emails based on a myriad of things: headers, the body of the email, various tags added by external programs, sender address… the list goes on.

In this case, we are going to use Sieve rules to send emails that have been marked by Spamassassin with spam flags like this (“X-Spam-Flag: YES“) in the headers straight into the Spam folder.

At present, emails are delivered to the inbox by Postfix. However, since Sieve is a Dovecot plugin, the final step of the delivery must be handled by Dovecot. This gives us two choices:

  1. Use Dovecot’s Local Delivery Agent (LDA)
  2. Use Dovecot’s Local Mail Transfer Protocol (LMTP)

For our setup, LDA is not ideal because it doesn’t run as root and therefore can’t access the individual inboxes of each user due to permissions. There are some other differences too…from the Dovecot Wiki:

The main difference is that the LDA is a short-running process, started as a binary from command line, while LMTP is a long-running process started by Dovecot’s master process.

In other words, with LDA a new process is started each time an email needs to be delivered, whereas with LMTP a process is always running and it handles a queue of emails. From what I have read on mailing lists etc. it seems that LMTP is more efficient.

So… LMTP it is!

Install & Configure Dovecot LMTP

First, install dovecot-lmtpd:

sudo apt-get update
sudo apt-get install dovecot-lmtpd

This will create a new config file at /etc/dovecot/conf.d/20-lmtp.conf.

Now to change the config in a few files:

/etc/dovecot/dovecot.conf

Append this to enable lmtp:

protocols = imap lmtp

/etc/dovecot/conf.d/20-lmtp.conf

Add this line to enable address extensions:

lmtp_save_to_detail_mailbox = yes

This means that if you send an email to you+folder@yourdomain.com it should be automatically placed in the “folder” folder. Cool, eh? You can use this for loads of things, but here’s a typical student example from a recent graduate ;) …

Change your email address for Pizza takeaway companies to you+pizza@yourdomain.com and create a folder called “pizza”. Now all your emails about pizza go into a separate folder, instead of cluttering your inbox. Awesome :)

Note: folder names are case sensitive, and the folder must be top level (not a folder within your inbox).

Now change the lmtp protocol block to look like this:

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = postmaster@yourdomain.com
}

/etc/dovecot/conf.d/10-master.conf

Now find the service lmtp {… block and then change the line unix_listener lmtp {… to look like this. This will allow postfix to access Dovecot’s LMTP from within its chroot:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0666
  }
}

/etc/dovecot/conf.d/10-auth.conf

By default, Dovecot will try to look up “you@yourdomain.com” in your user database, when it should be looking up just the first bit (“you”). This setting instructs Dovecot to strip the domain name before doing the lookup, and convert the username to all lowercase letters:

auth_username_format = %Ln

(the L is the lowercase part and the n drops the domain name).

/etc/dovecot/conf.d/10-director.conf

I’m not sure if this bit is necessary, but I commented out the “protocol lmtp {…” block completely.

/etc/postfix/main.cf

We still need to instruct Postfix to hand over control to Dovecot’s LMTP for the final stage of delivery.

Comment out:

mailbox_command=

…and add:

mailbox_transport = lmtp:unix:private/dovecot-lmtp

Sieve Rules

Dovecot's sieve is already installed, you can check by running:

sudo apt-get install dovecot-sieve

Now we need to change one more parameter in /etc/dovecot/conf.d/90-sieve.conf:

Uncomment this setting:

recipient_delimiter = +

We still need to reload/restart Postfix and Dovecot to make that all the changes are loaded:

sudo service postfix reload
sudo service dovecot reload

The default place to put the sieve script is in the user's home folder: ~/.dovecot.sieve.

Create it like this:

sudo nano /home/user/.dovecot.sieve

and add this:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" "YES" {
  fileinto "Spam";
  # Stop here - if there are other rules, ignore them for spam messages
  stop;
}

Or, if you want spam messages to be marked as read as well as moved:

require ["fileinto","imap4flags"];
if header :contains "X-Spam-Flag" "YES" {
        addflag "\\Seen";
        fileinto "Spam";
        stop;
}

Now chown the file to the owner of the mailbox, e.g.:

sudo chown sam:sam /home/user/.dovecot.sieve

When Spamassassin marks emails as Spam it adds X-Spam-Flag: YES to the headers. This rule checks the headers and sends mail to the spam folder if that flag exists.

Testing: GTUBE SPAM email

Here's how to send an email to your server using Telnet that will definitely be marked as spam. There's a neat trigger called GTUBE (Generic Trigger for Unsolicited Bulk Email) that is implemented in spamassassin.

All we need to do is send a message that contains this line in the body:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

We can use Telnet to send this email like we did when testing in parts 1 and 2:

feathers-mcgraw@Hobbs-T440s:~$ telnet yourdomain.com 25
Trying 192.168.1.174...
Connected to yourdomain.com.
Escape character is '^]'.
220 yourdomain.com ESMTP Postfix (Debian/GNU)
ehlo randomdomain.com
250-yourdomain.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: test@randomdomain.com
250 2.1.0 Ok
rcpt to: pi
250 2.1.5 Ok
data
354 End data with .
Subject: test spam email
This should set it off...
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
.
250 2.0.0 Ok: queued as DDFDEDA77
quit
221 2.0.0 Bye
Connection closed by foreign host.

That email should land in your Spam folder. Note that if you're using Squirrelmail you won't be automatically subscribed to the Spam folder, you have to add it yourself.

Thanks to Jens for finding this test!

Optional: Managesieve

Managesieve is a service that will allow you to log in remotely with a compatible email client and manage your sieve scripts.

First, we need to install the necessary package:

sudo apt-get update
sudo apt-get install dovecot-managesieved

You will notice that this creates a new configuration file: /etc/dovecot/conf.d/20-managesieve.conf, which you can leave as it is.

One more configuration change to make: open /etc/dovecot/dovecot.conf and add sieve to the protocols line:

protocols = imap lmtp sieve

and restart Dovecot:

sudo service dovecot restart

The managesieve service uses port 4190, so log in to your router's admin page and forward this port to your Pi.

You can now test if the service is running with telnet:

sam@samhobbs:/etc/dovecot$ telnet localhost 4190
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Dovecot (Ubuntu) Pigeonhole"
"SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave"
"NOTIFY" "mailto"
"SASL" "PLAIN LOGIN"
"STARTTLS"
"VERSION" "1.0"
OK "Dovecot (Ubuntu) ready."

If it's up and running, then you can try connecting with a compatible email client, like Kmail, the KDE email client that ships with distributions like Kubuntu. There is also a free software sieve plugin for Thunderbird that you might like to try if that's your client of choice.

To enable sieve in Kmail, go to:
Settings --> configure Kmail --> accounts --> Modify --> Filtering

Check "Server supports Sieve" and "Reuse host and login configuration".

kmail-sieve.png

Now you can select Settings --> "Manage Sieve Scripts" and edit your scripts using Kmail's editor, complete with syntax highlighting.

kmail-sieve-2.png

If you write multiple scripts, they will be stored in ~/sieve and the active one will be symlinked from ~/.dovecot.sieve. If you had a script at ~/.dovecot.sieve before you set up managesieve, it will be saved as dovecot.orig when you create a new one.

kmail-sieve-3.png

Thanks to the members of Kubuntuforums, without whom I would never have even heard of sieve rules!

Please leave a comment, I'd love to hear how you're getting on!

Type: 

Comments

This was an excellent tutorial.. I really appreciate you putting the time into making it. Everything is up and running and working great, first time around. Thanks again.

Hi Sam,

First of all thanks for the great tutorial.
The mailserver running on the Raspberry Pi 3 on Raspbian jessica lite.
When i use tail -f /var/log/mail.log i see the following lines:

Apr 2 14:10:14 kluivingh postfix/smtpd[1156]: disconnect from dub004-omc4s33.hotmail.com[157.55.2.108]
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [127.0.0.1]:53 failed: Connection refused, failing over to [ ::1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [::1]:53 failed: Connection refused, failing over to [127.0. 0.1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [127.0.0.1]:53 failed: Connection refused, failing over to [ ::1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [::1]:53 failed: Connection refused, failing over to [127.0. 0.1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [127.0.0.1]:53 failed: Connection refused, failing over to [ ::1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [::1]:53 failed: Connection refused, failing over to [127.0. 0.1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [127.0.0.1]:53 failed: Connection refused, failing over to [ ::1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [::1]:53 failed: Connection refused, failing over to [127.0. 0.1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [127.0.0.1]:53 failed: Connection refused, failing over to [ ::1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [::1]:53 failed: Connection refused, failing over to [127.0. 0.1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [127.0.0.1]:53 failed: Connection refused, failing over to [ ::1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [::1]:53 failed: Connection refused, failing over to [127.0. 0.1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [127.0.0.1]:53 failed: Connection refused, failing over to [ ::1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: sendto() to [::1]:53 failed: Connection refused, failing over to [127.0. 0.1]:53
Apr 2 14:10:14 kluivingh spamd[680]: dns: bad dns reply: Connection refused
Apr 2 14:10:14 kluivingh spamd[680]: dns: bad dns reply: Connection refused

Do you know how to fix this?
Did i forget something?

And i got another question Spamassassin doesn't auto start, any idea how to get it auto started?

Hi Daniel,

I haven't seen the DNS sendto error before, but it looks like the connection to your DNS resolver (on localhost) is failing, what's in your /etc/resolv.conf, and do you have BIND running locally or are your DNS settings unchanged from the default?

Autostarting spamassassin is achieved by setting ENABLED=1 in /etc/default/spamassassin - did you miss that step?

Sam

Tnx for fast reply.

/etc/resolv.conf
# Generated by resolvconf
domain dynamic.ziggo.nl
nameserver 192.168.0.1

Do i need to change this?

Spamassassin:
ENABLED=1 in /etc/default/spamassassin this step i did and checkt it.

But still no autostart :(

No that looks fine, it's just strange that the perl module is trying to use localhost when resolv.conf says to use your router. In general you shouldn't change resolv.conf by hand, because various programs like networkmanager overwrite the changes.

What happens if you try using dig (sudo apt-get install dnsutils) to look up a hostname:

dig @localhost samhobbs.co.uk

I think maybe the autostart issue is due to the switch to systemd, can you run:

sudo systemctl enable spamassassin

and then reboot, see if that makes it autostart?

Sam

Spamassasin is now booting at restart tnx.

When i use the dig command the answer is

; (2 servers found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Aha, so there's a problem with DNS resolution in general on your server.

First, make sure your router can resolve the dns query:

dig @192.168.0.1 samhobbs.co.uk

If that fails, then the problem must be your router or the nameservers that your router is using. If not, then maybe the problem could be because jessie lite is missing some packages I have assumed are installed by default... you could try:

sudo apt-get install dnsmasq resolvconf

And then repeat the dig command... do you get a different result?

Sam

Yeah no problems any more, thanks for the fast help :).

First command works but when i used localhost it didn't.

After i used: sudo apt-get install dnsmasq resolvconf

localhost is working as well.

That's great :)

If you're curious, what we did is install a local dns resolver, so that programs can query localhost, and localhost will forward the query to your router and return the answer to the original program. The perl module must expect a local DNS resolver, but there wasn't one installed previously.

You might find my BIND9 tutorial interesting if you want to read more about DNS.

Anyway, thanks for the interesting questions :)

Sam

Hi Sam,
I complete all. But, I have a question about "postmaster" this setting. I don't know that. Does it mean I can send a mail in an alias name to send to all my members? I sent it. However, the mail seems suspended on the air, no place to go. Do I get the right meaning what you mention?

Hi Jeff,

Postmaster is just an address that receives messages about failed mail delivery etc., have a look in /etc/aliases, I think by default it's an alias for root (i.e. failed delivery messages go to root's inbox) but you can change it to your own user if you want and run sudo newaliases.

Sam

Hi Sam,
Thank you. The other derivative question is Alises. I refer to your article, CAcerts tutorial. There are three email address set up, individually postmaster, webmaster and root pointing to an account. This makes me wonder a setup on Part 5 Spam Soring with LMTP & Sieve, there a setting about this:

protocol lmtp {
mail_plugins = $mail_plugins sieve
postmaster_address = postmaster@yourdomain.com
}

Do I need to add more to increment, webmaster_address = webmaster@xxx.com and root_address = root@xxx.com? (I will not set up root this account.)

No you don't need to add webmaster or root to the lmtp configuration, it's only necessary to add postmaster because lmtp needs to notify the postmaster (person in charge of the mail server) if it fails to deliver local mail.

Sam

Hi Sam,
Great tutorial! I had everything working perfectly for several months until a brownout fried my SD card. I've been trying to resurrect my Pi and am stuck on getting dovecot to work correctly. I get this error in /var/log/mail.log:

Aug 6 20:18:28 mailbox postfix/lmtp[14870]: 75BC5C1456: to=, relay=none, delay=4.8, delays=4.8/0/0/0, dsn=4.4.1, status=deferred (connect to mailbox.local[private/dovecot-lmtp]: No such file or directory)

Any idea what might be causing that? I have mailbox_transport = lmtp:unix:private/dovecot-lmtp set in /etc/postfix/main.cf ...

Many thanks!
Benn

Thank you for such good tutorial, but when I follow to apt-get install dovecot-lmptd, and return to fail to locate the package, what should I do, is there any mirror site will get this package

Thank you for all the work you did here ! Most of the things are well explained.

Only 2 things I'm missing are anti-virus checks and content blocking in postfix

I fully implemented this today. It worked perfect!!! Thank you so much. It is amazing that you figured this all out and posted it.

Hi thanks for assisting me through this setup. I have come to the end of it and most things are working. I do however have to small issues i am working through, i made a new account on my pi to make a new email address. I can login and send emails using this account, but whe i try and send an email to this account it gets bounced back to me saying the recipient does not exist on the domain? Not sure why this is happening. The second problem is with spamassassin, good mail gets through fine, but the spam test you demonstrated does not show up in either my spam folder or my normal inbox?

Thanks in advnace i am nearly there now, just this last bit
Connor

Hi Connor,

Can you send a test email to the new account while watching the mail log, and paste the output (not all of it, just the relevant lines)?

If you look at the headers in the email (most email clients will let you do this), can you see anything related to spamassassin? It should have scored them... if it has scored them but they aren't being sorted correctly, then it's a sieve problem. If it hasn't been scored, it's a spamassassin problem.

Sam

i decided to try my old email account the one i know has been working throughout this tutorial, and to my amazement even this is no longer working all email accounts i create have the same error, the information received says the emails cannot be delivered as the users do not exist? i don't understand why this all of a sudden is an issue. It even happening to the email i know to be working. just to clarify i can send emails out to other servers fine. but when i try and send emails to my server the server is saying the accounts dont exist.

thnks so much,
connor

Hi Connor,

Can you tell me what the current IP address of your server is, and your domain name? That way I can test that your DNS records are correct. If you tell me your email address on the server I can attempt to send you an email address manually too.

Sam

I found an issue to be with configuration of spamassassin (on my part) something really small and now both the spam filters and the emails are all working! now i am working on getting an SSL certificate from GoDaddy.

Thank you for all your assistance in this you have been so helpful and thankyou for the great, indepth tutorial!
Connor

You're welcome, well done for spotting it, sometimes the small errors can be really difficult to spot.

Sam

It's 2017 and this guide is still both useful and relevant, granted there are a few issues following the guide to the letter now that subsequent updates to packages have been made, but a bit of googling not only helped me identify the problem, it also furthered my knowledge on the setup. (Which is the point I believe you were going for from the start - i.e. Whats the point in providing a full configuration script if the user doesnt understand it?) I've skipped the spamassasin setup for now (until I actually receive spam) however I'm thinking about looking into linking with Spamhaus DNSBL - I previously had one of my emails blocked through here due to my dynamic IP, and thought this may be useful going forward!

Anyway, onto my next project for the PI, probably look into some sort of backup solution and perhaps HA setup, maybe even a little online store if thats at all possible!

Thanks again for sharing your work!

Rob

Hi Rob,

Out of interest, do you remember what you had to change?

You're right about the intended approach, I'm in the business of teaching people how to fish ;) wouldn't want you to be stuck if I was unable to spend the time helping any more.

Sam

I had problems around the access restrictions, I saw errors within the mail.log when I entered the following;

smtpd_recipient_restrictions =
permit_sasl_authenticated,
permit_mynetworks,
reject_unauth_destination

However, changing to the following format;

smtpd_relay_restrictions = permit_mynetworks,permit_sasl_authenticated,defer_unauth_destination

seemed to fix it for me. (postconf -n and dovecot -n are two very useful commands in diagnosing potential issues!)

Then I had issues with "no SASL authentication mechanisms", I believe I changed my dovecot/conf.d/10-master.conf to the following;
# Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
mode = 0666
}

Also,
Then the whole subdomain thing threw me at first, I didnt realise I had to setup CNAME's with my domain registrar to be able to get to another site I'm hosting. (Plus I'm using noip.com so basically I needed to point the new cname at my ddns)

mxtoolbox.com and wormly.com/test-smtp-server are two fantastic websites/tools to keep in your favourites

If I run through the setup process from scratch again, I'll be sure to document the gotcha's next time!

Thanks

Thanks for that, interesting to know where the tutorial could be improved.

The format for restriction lists with one item per line works for me, but I think it's sensistive to where you put the whitespace and you need a tab or spaces in front of the items in the list (and no whitespace after the =, just a carriage return).

Sounds like the CNAME thing is specific to your DNS provider's setup, if you have your own domain name you should be able to do it all with MX and A records. Do you not find that all of your email gets stuck in spam filters when using noip?

Sam

There was certainly a delay with email from google coming through, I think it was more their DNS records updating which was the problem. Outlook / Microsoft plain ourright rejected the email via Spamhaus, so once I removed myself from the block list, all appears ok.

I'll keep monitoring my ISP for IP changes and see how the emails behave when it does happen. Its not currently important for me, so the ddns solution is fine for now.

Thanks,

Rob

Pages

Add new comment