Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

Powered by Drupal
Submitted by Sam Hobbs on

This is the fifth and final part of a five part tutorial that will show you how to install a full featured email server on your Raspberry Pi. This tutorial covers how to automatically sort spam emails into the spam folder using Dovecot’s Local Mail Transfer Protocol (LMTP) and Sieve rules.

The parts are:

The Introduction & Contents Page (read first)

Raspberry Pi Email Server Part 1: Postfix

Raspberry Pi Email Server Part 2: Dovecot

Raspberry Pi Email Server Part 3: Squirrelmail

Raspberry Pi Email Server Part 4: Spam Detection with Spamassassin

Raspberry Pi Email Server Part 5: Spam Sorting with LMTP & Sieve

Intro

If you followed the previous tutorial, you currently have an email server that automatically scans incoming emails using Spamassassin. However, in its current state, Spam and Ham alike are delivered to the inbox, which is annoying. Since Spamassassin only marks emails based on their spam score, we need to use an external program to handle sorting & delivery. The best tool for the job is Sieve: a set of rules defined by each user to determine how incoming emails are filtered. Sieve rules can sort emails based on a myriad of things: headers, the body of the email, various tags added by external programs, sender address… the list goes on. In this case, we are going to use Sieve rules to send emails that have been marked by Spamassassin with spam flags like this (“X-Spam-Flag: YES“) in the headers straight into the Spam folder. At present, emails are delivered to the inbox by Postfix. However, since Sieve is a Dovecot plugin, the final step of the delivery must be handled by Dovecot. This gives us two choices:

  1. Use Dovecot’s Local Delivery Agent (LDA)
  2. Use Dovecot’s Local Mail Transfer Protocol (LMTP)

For our setup, LDA is not ideal because it doesn’t run as root and therefore can’t access the individual inboxes of each user due to permissions. There are some other differences too…from the Dovecot Wiki:

The main difference is that the LDA is a short-running process, started as a binary from command line, while LMTP is a long-running process started by Dovecot’s master process.

In other words, with LDA a new process is started each time an email needs to be delivered, whereas with LMTP a process is always running and it handles a queue of emails. From what I have read on mailing lists etc. it seems that LMTP is more efficient. So… LMTP it is!

Install & Configure Dovecot LMTP

First, install dovecot-lmtpd:

sudo apt-get update
sudo apt-get install dovecot-lmtpd

This will create a new config file at /etc/dovecot/conf.d/20-lmtp.conf. Now to change the config in a few files:

/etc/dovecot/dovecot.conf

Append this to enable lmtp:

protocols = imap lmtp

/etc/dovecot/conf.d/20-lmtp.conf

Add this line to enable address extensions:

lmtp_save_to_detail_mailbox = yes

This means that if you send an email to you+folder@yourdomain.com it should be automatically placed in the “folder” folder. Cool, eh? You can use this for loads of things, but here’s a typical student example from a recent graduate ;) … Change your email address for Pizza takeaway companies to you+pizza@yourdomain.com and create a folder called “pizza”. Now all your emails about pizza go into a separate folder, instead of cluttering your inbox. Awesome :) Note: folder names are case sensitive, and the folder must be top level (not a folder within your inbox). Now change the lmtp protocol block to look like this:

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = postmaster@yourdomain.com
}

/etc/dovecot/conf.d/10-master.conf

Now find the service lmtp {… block and then change the line unix_listener lmtp {… to look like this. This will allow postfix to access Dovecot’s LMTP from within its chroot:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0666
  }
}

/etc/dovecot/conf.d/10-auth.conf

By default, Dovecot will try to look up “you@yourdomain.com” in your user database, when it should be looking up just the first bit (“you”). This setting instructs Dovecot to strip the domain name before doing the lookup, and convert the username to all lowercase letters:

auth_username_format = %Ln

(the L is the lowercase part and the n drops the domain name).

/etc/dovecot/conf.d/10-director.conf

I’m not sure if this bit is necessary, but I commented out the “protocol lmtp {…” block completely.

/etc/postfix/main.cf

We still need to instruct Postfix to hand over control to Dovecot’s LMTP for the final stage of delivery. Comment out:

mailbox_command=

…and add:

mailbox_transport = lmtp:unix:private/dovecot-lmtp

Sieve Rules

Dovecot's sieve is already installed, you can check by running:

sudo apt-get install dovecot-sieve

Now we need to change one more parameter in /etc/dovecot/conf.d/90-sieve.conf: Uncomment this setting:

recipient_delimiter = +

We still need to reload/restart Postfix and Dovecot to make that all the changes are loaded:

sudo service postfix reload
sudo service dovecot reload

The default place to put the sieve script is in the user's home folder: ~/.dovecot.sieve. Create it like this:

sudo nano /home/user/.dovecot.sieve

and add this:

require ["fileinto"];
# Move spam to spam folder
if header :contains "X-Spam-Flag" "YES" {
  fileinto "Spam";
  # Stop here - if there are other rules, ignore them for spam messages
  stop;
}

Or, if you want spam messages to be marked as read as well as moved:

require ["fileinto","imap4flags"];
if header :contains "X-Spam-Flag" "YES" {
        addflag "\\Seen";
        fileinto "Spam";
        stop;
}

Now chown the file to the owner of the mailbox, e.g.:

sudo chown sam:sam /home/user/.dovecot.sieve

When Spamassassin marks emails as Spam it adds X-Spam-Flag: YES to the headers. This rule checks the headers and sends mail to the spam folder if that flag exists.

Testing: GTUBE SPAM email

Here's how to send an email to your server using Telnet that will definitely be marked as spam. There's a neat trigger called GTUBE (Generic Trigger for Unsolicited Bulk Email) that is implemented in spamassassin. All we need to do is send a message that contains this line in the body:

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

We can use Telnet to send this email like we did when testing in parts 1 and 2:

feathers-mcgraw@Hobbs-T440s:~$ telnet yourdomain.com 25
Trying 192.168.1.174...
Connected to yourdomain.com.
Escape character is '^]'.
220 yourdomain.com ESMTP Postfix (Debian/GNU)
ehlo randomdomain.com
250-yourdomain.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: test@randomdomain.com
250 2.1.0 Ok
rcpt to: pi
250 2.1.5 Ok
data
354 End data with .
Subject: test spam email
This should set it off...
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
.
250 2.0.0 Ok: queued as DDFDEDA77
quit
221 2.0.0 Bye
Connection closed by foreign host.

That email should land in your Spam folder. Note that if you're using Squirrelmail you won't be automatically subscribed to the Spam folder, you have to add it yourself. Thanks to Jens for finding this test!

Optional: Managesieve

Managesieve is a service that will allow you to log in remotely with a compatible email client and manage your sieve scripts. First, we need to install the necessary package:

sudo apt-get update
sudo apt-get install dovecot-managesieved

You will notice that this creates a new configuration file: /etc/dovecot/conf.d/20-managesieve.conf, which you can leave as it is. One more configuration change to make: open /etc/dovecot/dovecot.conf and add sieve to the protocols line:

protocols = imap lmtp sieve

and restart Dovecot:

sudo service dovecot restart

The managesieve service uses port 4190, so log in to your router's admin page and forward this port to your Pi. You can now test if the service is running with telnet:

sam@samhobbs:/etc/dovecot$ telnet localhost 4190
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
"IMPLEMENTATION" "Dovecot (Ubuntu) Pigeonhole"
"SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave"
"NOTIFY" "mailto"
"SASL" "PLAIN LOGIN"
"STARTTLS"
"VERSION" "1.0"
OK "Dovecot (Ubuntu) ready."

If it's up and running, then you can try connecting with a compatible email client, like Kmail, the KDE email client that ships with distributions like Kubuntu. There is also a free software sieve plugin for Thunderbird that you might like to try if that's your client of choice. To enable sieve in Kmail, go to: Settings --> configure Kmail --> accounts --> Modify --> Filtering Check "Server supports Sieve" and "Reuse host and login configuration". kmail-sieve.png Now you can select Settings --> "Manage Sieve Scripts" and edit your scripts using Kmail's editor, complete with syntax highlighting. kmail-sieve-2.png If you write multiple scripts, they will be stored in ~/sieve and the active one will be symlinked from ~/.dovecot.sieve. If you had a script at ~/.dovecot.sieve before you set up managesieve, it will be saved as dovecot.orig when you create a new one. kmail-sieve-3.png Thanks to the members of Kubuntuforums, without whom I would never have even heard of sieve rules! Please leave a comment, I'd love to hear how you're getting on!

Comments

Hi again Sam
I'm all sorted now. I reimaged my pi and started from scratch, too much went wrong yesterday and I've managed to rebuild it all today as well as getting this mail server working. Now I have it working - is it possible to move the location of my mailbox?
I have a 32gb stick attached where I would like to store my IMAP mailbox if possible, rather than on the Pi's SD card. Now that the server is all set up and working, what would I have to do?

Thanks

You can move the whole filesystem onto the USB drive if you like, or if you just want the Maildir on the flash drive then edit /etc/fstab to mount the USB drive somewhere consistent like /media/yourusbdrive and then create a symlink from ~/Maildir to /media/yourusbdrive/Maildir and make sure the permissions and ownership of the drive are correct. You will need to use a filesystem on the USB drive that supports linux permissions (like ext4). Sam

I’m struggling to get my head around permissions a bit.

From what I can figure out the command will be:
sudo ls -s /home/callum/Mailbox /not/mail/callum

Would that be correct? What would the owner/permissions need to be set as? I’ve got the mail usb mounted and fstabbed already so it’s just making this symlink, I’ve never done it before.

Cheers

Ok, so I would first format it ext4 so the filesystem supports linux permissions, and modify /etc/fstab to suit, and then: Make sure you own the whole flash drive
sudo chown -R callum:callum /mnt/mail
Copy your existing maildir to the flash drive and make sure it's only readable by you
cp ~/Maildir /mnt/mail/Maildir
chmod 700 /mnt/mail/Maildir
Move your old maildir and replace it with a symlink to the folder on the flash drive:
mv ~/Maildir ~/Maildir.bak
ln -s /mnt/mail/Maildir ~/Maildir
Test. Note that these instructions assume you are the only user whose Maildir you want to store on the flash drive. Sam

Hi Sam

Thanks for your reply. I've tried the above but now I'm not able to send or receive any emails from this account. It can see the mailbox content which it had previously but doesn't look like it's updating it with anything new.

Cheers

Callum, Run tail -f /var/log/mail.log and send yourself a test email. See if there are any relevant log messages? Sam

I’m getting a warning from postfix/smtpd, not enough free space in mail quite : 0bytes.
Also from postfix/cleanup: write queue file no space left on device.

Cheers

Hello,

I've followed your guide, it is very thorough. I'm currently testing my spam assassin service. Unfortunately when I try to send an email from randomdomain.com the server blocks me:

root@host:~$ openssl s_client -connect domain --quiet
depth=...
verify return:1
depth=...
verify return:1
depth=...
verify return:1
220 localhost ESMTP Postfix (Raspbian)
ehlo randomdomain.com
250-localhost
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-AUTH PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
mail from: hello@randomdomain.com
250 2.1.0 Ok
Subject: Hello
221 2.7.0 Error: I can break rules, too. Goodbye.

I've tried sending myself and email via squirrelmail but it did not put the spam mail in the spam folder

a quick question, why openssl s_client -connect domain doesn't work when I'm running it from my own mail server?

Thank you very much

Hi, The server closed the session because you missed the RCPT TO: step (you broke the SMTP rules). 'Domain' is not fully qualified so how would openssl know you mean localhost? Sam

Nick Sharp

Wed, 02/28/2018 - 00:00

I've recently moved been moving as many of my services off of power hungry servers to RasPi's. The last to do was my mail server which used to be Zentyal running on a very noisy hot server that was totally overkill for the 4 mailboxes I was running.

Ran through your guides with little problem, didn't worry about the web-mail client. It works great with Thunderbird on the PC and Bluemail on my tablet/phones.

Cheers
Nick.

My current setup..
RasPi 3 hosting Postfix/Dovecot/Spamassasin and Sieve
Raspi 0W hosting bind/dhcpd
Raspi 0W hosting fault tolerant bind/dhcpd and LDAP
Raspi 2 hosting Softether LT2P/IPSEC vpn server
Raspi 2 Hosting OpenElec
Raspi 3 TVBackend with 3 Tuners
Raspi 3 Hassbian for my Home automation system
Raspi 2 Hooked up to two 1TB drives as a NAS

Hi Sam, I was really interested to get this working, and super glad when your blog came up in the search results!

I followed your instructions but when i send email to my username+pizza@mydomain.com - even though I have a top level ‘pizza’ folder, the email is still arriving in my inbox.

I’m not interested in the SpamAssasin stuff yet and so implemented a rule from here instead: https://wiki.dovecot.org/Pigeonhole/Sieve/Examples#Plus_Addressed_mail_…

subaddress"];

if envelope :is :user "to" "sales" {
if envelope :matches :detail "to" "*" {
/* Save name in ${name} in all lowercase except for the first letter.
* Joe, joe, jOe thus all become 'Joe'.
*/
set :lower :upperfirst "name" "${1}";
}

if string :is "${name}" "" {
/* Default case if no detail is specified */
fileinto "NotMatched";
} else {
/* For sales+joe@ this will become users/Joe */
fileinto "users/${name}";
}
}

Which as i understand it should create folders on the fly! Where can i check for errors etc as clearly something is amiss!

Cheers,
Nick

Hi Sam,
Great tutorial and followed it perfectly and I was getting e-mails just fine until I got to the installing the lmtp. Now following entries...
If go back to before i configure lmtp all is ok or so it seams, I have tried changing the first_valid_uid = 114 in /etc/dovecot/conf.d/10-mail.conf

As I'm new to Linux system I'm find it a bit hard going so any help would be great.

From /var/log/syslog
Apr 11 06:50:41 gunas postfix/qmgr[907]: 19EF8822E9: from=, size=1443, nrcpt=1 (queue active)
Apr 11 06:50:41 gunas dovecot: lmtp(17285): Connect from local
Apr 11 06:50:41 gunas dovecot: lmtp(17285, opendmarc): Error: Mail access for users with UID 114 not permitted (see first_valid_uid in config file, uid from userdb lookup).
Apr 11 06:50:41 gunas dovecot: lmtp(17285): Disconnect from local: Successful quit
Apr 11 06:50:41 gunas postfix/lmtp[17284]: 19EF8822E9: to=, orig_to=, relay=mail.gunas.co.uk[private/dovecot-lmtp], delay=107435, delays=107435/0.1/0.21/0.1, dsn=4.3.0, status=deferred (host mail.gunas.co.uk[private/dovecot-lmtp] said: 451 4.3.0 Temporary internal error (in reply to end of DATA command))
Apr 11 06:51:08 gunas kernel: [19012.592470] [UFW BLOCK] IN=eth0 OUT= MAC=01:00:5e:00:00:01:30:b5:c2:fe:93:22:08:00:46:00:00:24:00:00:40:00:01:02:43:29 SRC=192.168.1.1 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0x00 TTL=1 ID=0 DF PROTO=2
Apr 11 06:52:17 gunas postfix/smtpd[17366]: connect from unknown[185.234.218.134]

From /var/log/mail.log
Apr 12 09:19:52 gunas postfix/qmgr[971]: 0B7DD44116C: from=, size=1445, nrcpt=1 (queue active)
Apr 12 09:19:52 gunas dovecot: lmtp(1317): Connect from local
Apr 12 09:19:52 gunas dovecot: lmtp(opendmarc): q/V2EShKsFwlBQAAPzfMEA: msgid=<20190412000006.0B7DD44116C@mail.gunas.co.uk>: saved mail to INBOX
Apr 12 09:19:52 gunas dovecot: lmtp(1317): Disconnect from local: Successful quit
Apr 12 09:19:52 gunas postfix/lmtp[1316]: 0B7DD44116C: to=, orig_to=, relay=mail.gunas.co.uk[private/dovecot-lmtp], delay=29986, delays=29986/0.12/0.14/0.13, dsn=2.0.0, status=sent (250 2.0.0 q/V2EShKsFwlBQAAPzfMEA Saved)
Apr 12 09:19:52 gunas postfix/qmgr[971]: 0B7DD44116C: removed

I'm starting to follow to your guide, and I'm puzzled...
/etc/dovecot/dovecot.conf contains nothing concerning particular protocols support.
Instead it refers to separate files within /usr/share/dovecot/ subdirectory:
!include_try /usr/share/dovecot/protocols.d/*.protocol
Thus, it's not required to add anything into the main config. All necessary protocols are enumerated in appropriate files.
Particularly, LMTP protocol is added automatically, when you install appropriate package dovecot-lmtpd

Thank you.

Everything is OK but sieve... Just can't make the test spam to be moved into .Spam/ subdirectory.
BTW, what exactly should be written in the fileinto "Spam"; directive? "Spam" or ".Spam"?
As I suppose, the problem is in the path. I have virtual users with mailboxes, located at
/var/mail/vhosts/mydomain.com/user/ for user@mydomain.com. At the same time, AFAICS, sieve recognizes system paths only, like ~/.dovecot.sieve. All the mailboxes have the same owner in my case, it's virtual:virtual. I've tried to add sieve script into the /home/virtual/.dovecot.sieve without a luck.
So, how I could apply sieve script to virtual mailboxes?
Or, at least, how I can check if sieve yet works? Any log records?

Thank you.

Alex, This tutorial is written as part of a series, with a local user setup. You can't just take one part and use it on any server as the config will be different! I'm sorry but I don't know how to confiugre it for virtual mailboxes. Sam

Alex

Mon, 01/20/2020 - 19:30

In reply to by Sam Hobbs

Everything is done now. After some trials and mistakes I've configured it. The problem was just in Postfix => Dovecot bridge. I should configure LMTP service, and Sieve is enabled there. Everything works! As soon as your guide doesn't cover virtual mailboxes, how about to add yet another part like "Virtual mailboxes" to this very good manual? I could share some materials, as soon as they'll be ready.

Right now I could share just few lines, which helped me to resolve my problem:

If you've got the next error message after sudo postfix restart:
(/var/log/mail.log):
postfix/postfix-script[14828]: fatal: the Postfix mail system is already running
then do the next:
-- check the existence of /var/spool/postfix/pid/master.pid
if it's there then rm master.pid
-- then restart Postfix again: sudo service postfix restart
-- check the same log file for this error: postfix/master[18648]: fatal: open lock file /var/lib/postfix/master.lock: unable to set exclusive lock: Resource temporarily unavailable
Do the next, if it's there:
-- check existence of /var/lib/postfix/master.lock
-- fuser /var/lib/postfix/master.lock (you can use htop as well)
-- find the proc_id of the process, which has locked the file
-- kill proc_id
Then try restart Postfix again:
sudo service postfix restart
Look at the /var/log/mail.log to ensure that errors have gone.

You might see similar error in Dovecot log. Delete this file if you see an error there:
sudo rm -f /var/run/dovecot/master.pid
service dovecot restart

These small tricks helped me a lot, because otherwise I'd need to reboot the RPi to restart Postfix/Dovecot.

Add new comment

The content of this field is kept private and will not be shown publicly.

Filtered HTML

  • Web page addresses and email addresses turn into links automatically.
  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.