Hybrid-Cloud Email with Amazon SES and Dovecot¶

The best way to make your online presence look professional is with a custom email address. You can also easily give different addresses to every site you sign up to, (so you'll know whose fault it is that you're on that spam newsletter!). I want to use my pxeger.com domain for my personal email, and this is how I did it.

Self-hosted email is a massive pain to set up and maintain. Many IP addresses will be on blacklists, you need proper DKIM signing and PTR records in order to not get your emails in people's Spam folders, it's very hard to secure well, and if it ever goes down you'll have no way of knowing if your monitoring and health-check emails use it! Not to mention spam and malware filtering!

Unfortunately, all of the available email services I could find either suffer from the same problems, compromise on privacy and security (about which I am very passionate), or are too expensive. I wanted a middle-ground, and I think I've found it here. You can skip the hard parts of self-hosting email, but do it mostly for free or pay-as-you-use, and retaining control of your personal data.

This setup will accept emails to any address at your domain.

Amazon Web Services¶

AWS really does have a service for everything, and its email service, Simple Email Service (SES), is excellent. SES is unique among cloud providers in its pricing model: for the first year (as part of the AWS Free Tier) you get 1,000 incoming emails per month, and after that it's pennies. You can send messages super-easily using their SDKs or good old SMTP, and put received emails into S3 buckets, trigger Lambda functions, or publish to SNS topics, so you can glue them to other services simply as well.

Requirements¶

decent familiarity with Linux servers and the command-line, and DNS fundamentals
a domain name on which you can put DNS records easily. (If you don't have one, I can highly recommend Porkbun (not sponsored))
some sort of Linux server. This could be a VPS hosted on Amazon EC2 or any other provider, but it can be as low-power as a Raspberry Pi, and can be hosted at home if you want (although you will have to forward some ports on your router).
- all commands here should be run as root
- your kernel must support FUSE; some VPSs may not support this
a Let's Encrypt certificate for a subdomain of that domain
- consider setting this to auto-renew
an Amazon Web Services account.

Choose whichever of these AWS regions is closest to you, and stick with it for the rest of the setup process
AWS has many other regions, but these are the only ones that will work for this setup.

Setting up your domain in SES¶

Point your subdomain where you have your letsencrypt to the IP address of the server you plan to use to store email on.

Go to the Amazon SES dashboard and verify your domain name (not the subdomain pointing to your server, but the one you want an email address at). (Instructions) Make sure to generate and set up DKIM records too.

Set an MX DNS record on your domain that points to the appropriate one of:

inbound-smtp.eu-west-1.amazonaws.com for Ireland
inbound-smtp.us-west-2.amazonaws.com for Oregon
inbound-smtp.us-east-1.amazonaws.com for Virginia

Set an SPF record on your domain according to these instructions.

Dovecot¶

Create a user on your server that you'll use to access the Dovecot server:

# create the user
useradd SOMEUSER
# prevent the user from logging in normally
chsh -s /usr/bin/nologin SOMEUSER
# set a password for the user
passwd SOMEUSER

For security, you should use a separate user account from the one you use to configure and manage your server. And definitely don't use root!

Create or overwrite the file /etc/pam.d/dovecot with the following contents:

#%PAM-1.0

# authenticate the user with a password
auth    required  pam_unix.so
account required  pam_unix.so
# ensure the user is allowed according to the /etc/emailuser file
auth    required  pam_listfile.so  item=user sense=allow file=/etc/emailuser onerr=fail
account required  pam_listfile.so  item=user sense=allow file=/etc/emailuser onerr=fail

This file will be used in the Dovecot configuration. It configures how users can login to the Dovecot server.

Create the /etc/emailuser file containing just the username of the user you created. Set the correct permissions with chmod 644

Install Dovecot using your distribution's package manager (apt, yum, pacman, etc.) Edit the Dovecot configuration file at /etc/dovecot/dovecot.conf. Delete the existing contents and replace them with:

ssl = required
ssl_cert = </etc/letsencrypt/live/YOUR_MAILSERVER_DOMAIN/fullchain.pem
ssl_key = </etc/letsencrypt/live/YOUR_MAILSERVER_DOMAIN/privkey.pem

# Plaintext login. This is safe and easy thanks to SSL.
auth_mechanisms = plain login

# POP3 doesn't work very well with this setup
protocols = imap

# use PAM for authentication
# /etc/pam.d/dovecot must exist and work correctly
passdb {
        driver = pam
        args = dovecot
}
# ensure dovecot knows where the user's home dir is
userdb {
        driver = passwd
}

# mail will be in ~/mail, and the inbox will be ~/mail/Inbox
mail_location = maildir:~/mail:INBOX=~/mail/Inbox:LAYOUT=fs
namespace inbox {
        inbox = yes
        mailbox Drafts {
                special_use = \Drafts
                auto = subscribe
        }
        mailbox Junk {
                special_use = \Junk
                auto = subscribe
                autoexpunge = 30d
        }
        mailbox Sent {
                special_use = \Sent
                auto = subscribe
        }
        mailbox Trash {
                special_use = \Trash
        }
        mailbox Archive {
                special_use = \Archive
        }
}

Be sure to replace YOUR_MAILSERVER_DOMAIN with the domain you set a LetsEncrypt certificate for.

Completely remove the /etc/dovecot/conf.d directory. Finally, reload the Dovecot service: systemctl reload dovecot.

If you are hosting this at home, you'll need to forward port 143 on your router in order to access the mail server from the internet.

Receiving emails¶

Go to the Rule Sets page on the SES dashboard and create a receipt rule:

In the recipient box, enter just your domain name
Add an action:
Action type: S3
S3 bucket: create a new bucket (e.g. foobar-emails-1). NOTE: don't include dots in your bucket name because s3fs doesn't work well with them
SNS topic: create SNS topic (e.g. foobar-emails-1)
Do not encrypt message (I haven't got it to work with encryption enabled yet, if you do please let me know)
Object key prefix: leave blank
Enabled: yes
Require TLS: yes
Spam and virus scanning: yes

You can use the optional SNS topic to send notifications or trigger automations if you want.

This setup assumes you want emails to any address at your domain to reach the same inbox. If you only want to use certain addresses, you can change the Recipient to an email address instead of just your domain. See the AWS documentation for more details.

The final piece of the puzzle is getting emails from Amazon S3 to your Dovecot server. This is the most complex part.

Here's how to set up access to S3 from your server:

Create an AWS User that can access the S3 bucket
1. Go to the IAM console
2. Create a user and enter a username
3. Choose "Programmatic access"
4. Choose "Attach existing policies directly"
5. Create a policy
  1. Service: S3
  2. Actions: HeadBucket, ListBucket, GetObject, GetObjectAcl, DeleteObject
  3. Resources:
    1. Bucket: choose the bucket you created earlier
    2. Object: tick the "Any" box
  4. Request conditions: optionally you can restrict this access token to the IP address of your server
6. Go back to the Add user page and attach the policy you created
Create credentials for the user
Install s3fs-fuse
Create the file /etc/passwd-s3fs containing the credentials you generated, in the format ACCESS_KEY_ID:SECRET_ACCESS_KEY. Make sure its permissions are 600.
Add an entry to /etc/fstab to mount your bucket somewhere, e.g. /mnt/my_bucket_mountpoint (the directory must exist):
1
my_bucket_name /mnt/my_bucket_mountpoint fuse.s3fs _netdev,allow_other 0 0
Then run mount -a to mount it.

Next, you need to create a webhook listener that can

Set up a webserver and CGI such as Apache or nginx. Tutorials for this are available on the internet. Alternatively, you could use a framework such as Express for Node.js, or Flask for Python. If you are running this server at home you'll need to forward port 80/443 on your router

Create a script running on your webserver that can be accessed from the internet. For security, you should have some simple authentication with a GET parameter. The script will need to move files from the mounted S3 bucket into the mail/Inbox/new/ directory of the IMAP user you created earlier.

Here is an example of such a script, written using CGI and Python:

#!/usr/bin/python
import hashlib
import os
import sys
import traceback
import urllib.parse


KEY = "50a43ff3bdaec2884af24b6d3719ec4447d9fdb7d1a7cb895be00f9b5a94406a0737b088baf97bd4093b92b96cdf04b9d04d9c7d7746c947a9d97ab66ccaad81"


def run():
    os.system("/usr/bin/sudo /srv/copyemail.sh")


def main():
    print("Content-Type: text/plain")
    qs = os.getenv("QUERY_STRING")
    if not qs:
        raise ValueError()
    q = urllib.parse.parse_qsl(qs)
    for k, v in q:
        if k == "key":
            if hashlib.sha512(v.encode()).hexdigest() == KEY:
                print("\nOK")
                run()
                sys.exit()
    raise ValueError()


if __name__ == "__main__":
    try:  
        main()
    except Exception as e:
        print("Status: 400 Bad Request\n")
        print("ERROR")
        traceback.print_exc(file=sys.stderr)
        sys.exit()

Then a script to actually move the emails (/srv/copyemails.sh):

#!/bin/sh
/usr/bin/mv /mnt/my_bucket_mountpoint/* /home/SOMEUSER/mail/Inbox/new/
/usr/bin/chmod 600 /home/SOMEUSER/mail/Inbox/new/*
/usr/bin/chown SOMEUSER:SOMUSER /home/SOMEUSER/mail/Inbox/new/*

Make both scripts executable. Finally, grant passwordless sudo access to your webserver's user for the copyemails.sh script - add this line to the sudoers file using visudo:
1
www-data ALL=(ALL): NOPASSWD /srv/copyemail.sh
Subscribe this webhook to the Amazon SNS topic you created earlier from the SNS dashboard
Create a subscription - choose the topic
Choose HTTP or HTTPS for the protocol (HTTPS should be preferred)
Leave the rest of the fields as default
Confirm the subscription
You will need to modify your CGI script to report the confirmation request
Go to the Subscriptions page on the SNS dashboard, choose the one you created, and click "Request confirmation"
Get the Confirmation URL from the request sent to your script and visit it in a web browser
Check that the subscription is confirmed correctly on the SES dashboard, then return your script to how it was befores

SMTP access to SES¶

Create SMTP credentials using the Amazon SES dashboard. You'll need these to send any email from your email client.

Your email client¶

This was the hardest part of the setup for me - figuring out how to get Thunderbird to use the right server settings (the same idea should apply to whatever email client you use). Here's what you need:

Incoming mail:

Server: the domain name of your Dovecot server
IMAP
Port: 143
STARTTLS
Password Authentication using the username and password for the account

Outgoing mail:

SMTP
Server: the appropriate one of:
- email-smtp.eu-west-1.amazonaws.com
- email-smtp.us-west-2.amazonaws.com
- email-smtp.us-east-1.amazonaws.com
Port: 587
STARTTLS
Password Authentication using the SMTP username and password generated by AWS in the previous step

The Catch¶

As you may have noticed while testing your shiny new system, you can only send emails to addresses that you've verified. If you just want to use your domain for email aliases that you provide to unstrustworthy websites, or for junk, that might be fine. However, most people will at some point want to send emails to other people too. Luckily working around this is not too hard, but it does require talking to a human.

Go to the Amazon's Sender Limit Increase form and explain your purpose. I was quickly and helpfully moved out of the Sandbox, removing this limitation (and letting me send 50,000 emails per day which I certainly don't need!). For reference, here's what I entered in the form:

Limit Type: SES Sending Limits
Mail Type: Other
Website URL: https://www.pxeger.com/2020-07-02-hybrid-cloud-email-with-amazon-ses-and-dovecot/
Describe, in detail, how you will only send to recipients who have specifically requested your mail:

I will only send emails manually, in very low volumes, and only every in response to people who contact me.
Describe, in detail, the process you will follow when you receive bounce and complaint notifications:

I will attempt to resolve the particular issue by tracing and correcting the root cause, and I will review my systems to ensure bounces and complaints don't happen in the future
Region: choose the same region you used for the whole process (for me it's eu-west-1, Ireland)
Limit: Desired Daily Sending Quota
- New limit value: 50 (this seems like the most I could expect to send on a busy workday if I began using this email system for work communication)
Use case description:

I use Amazon SES to implement hybrid-cloud-hosted email on a custom domain for my personal brand. You can learn more about the setup at https://www.pxeger.com/2020-07-02-hybrid-cloud-email-with-amazon-ses-and-dovecot/. I will only send emails in response to emails I receive, or to contact people who have requested me to contact them. All emails will be sent manually using an SMTP client, and in very low volumes. If I recieve any bounces or complaints, which is unlikely given my use case, I will follow up the cause of these issues, address the root cause, apologise if appropriate/necessary, and review my monitoring to prevent further incidents.

Conclusion¶

I've been running this system for approximately 15 months, and it hasn't failed me yet.

If you have any feedback, feel free to email me to let me know about any more.

Next Steps¶

From here, you could consider:

modifying your webhook script with rules to automatically move emails marked as spam by Amazon SES (identified by the X-SES-Spam-Verdict: FAIL and/or X-SES-Virus-Verdict: FAIL headers)
creating a DMARC policy
setting up PGP encryption on your email (because email is inherently an insecure communication system)

Known Issues¶

Here is a list of known issues with this system:

Amazon SES limits emails to 10MB, including attachments
AWS will scan all your emails for viruses. If you are a privacy extremist this might be an issue, but personally I trust them not to exploit this

First published: 2020-07-02
Last updated: 2021-11-27