Hybrid-Cloud Email with Amazon SES and Dovecot¶
The best way to make your online presence look professional is with a custom email address. You can also easily give different addresses to every site you sign up to, (so you'll know whose fault it is that you're on that spam newsletter!). I want to use my
pxeger.com domain for my personal email, and this is how I did it.
Self-hosted email is a massive pain to set up and maintain. Many IP addresses will be on blacklists, you need proper DKIM signing and PTR records in order to not get your emails in people's Spam folders, it's very hard to secure well, and if it ever goes down you'll have no way of knowing if your monitoring and health-check emails use it! Not to mention spam and malware filtering!
Unfortunately, all of the available email services I could find either suffer from the same problems, compromise on privacy and security (about which I am very passionate), or are too expensive. I wanted a middle-ground, and I think I've found it here. You can skip the hard parts of self-hosting email, but do it mostly for free or pay-as-you-use, and retaining control of your personal data.
This setup will accept emails to any address at your domain.
Amazon Web Services¶
AWS really does have a service for everything, and its email service, Simple Email Service (SES), is excellent. SES is unique among cloud providers in its pricing model: for the first year (as part of the AWS Free Tier) you get 1,000 incoming emails per month, and after that it's pennies. You can send messages super-easily using their SDKs or good old SMTP, and put received emails into S3 buckets, trigger Lambda functions, or publish to SNS topics, so you can glue them to other services simply as well.
- decent familiarity with Linux servers and the command-line, and DNS fundamentals
- a domain name on which you can put DNS records easily. (If you don't have one, I can highly recommend Porkbun (not sponsored))
- some sort of Linux server. This could be a VPS hosted on Amazon EC2 or any other provider, but it can be as low-power as a Raspberry Pi, and can be hosted at home if you want (although you will have to forward some ports on your router).
- all commands here should be run as root
- your kernel must support FUSE; some VPSs may not support this
- a Let's Encrypt certificate for a subdomain of that domain
- consider setting this to auto-renew
an Amazon Web Services account.
Choose whichever of these AWS regions is closest to you, and stick with it for the rest of the setup process
AWS has many other regions, but these are the only ones that will work for this setup.
Setting up your domain in SES¶
Point your subdomain where you have your letsencrypt to the IP address of the server you plan to use to store email on.
Go to the Amazon SES dashboard and verify your domain name (not the subdomain pointing to your server, but the one you want an email address at). (Instructions) Make sure to generate and set up DKIM records too.
Set an MX DNS record on your domain that points to the appropriate one of:
Set an SPF record on your domain according to these instructions.
Create a user on your server that you'll use to access the Dovecot server:
1 2 3 4 5 6
Create or overwrite the file
/etc/pam.d/dovecot with the following contents:
1 2 3 4 5 6 7 8
/etc/emailuser file containing just the username of the user you created. Set the correct permissions with
Install Dovecot using your distribution's package manager (
pacman, etc.) Edit the Dovecot configuration file at
/etc/dovecot/dovecot.conf. Delete the existing contents and replace them with:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
Completely remove the
/etc/dovecot/conf.d directory. Finally, reload the Dovecot service:
systemctl reload dovecot.
If you are hosting this at home, you'll need to forward port 143 on your router in order to access the mail server from the internet.
Go to the Rule Sets page on the SES dashboard and create a receipt rule:
- In the recipient box, enter just your domain name
- Add an action:
- Action type: S3
- S3 bucket: create a new bucket (e.g. foobar-emails-1). NOTE: don't include dots in your bucket name because s3fs doesn't work well with them
- SNS topic: create SNS topic (e.g. foobar-emails-1)
- Do not encrypt message (I haven't got it to work with encryption enabled yet, if you do please let me know)
- Object key prefix: leave blank
- Enabled: yes
- Require TLS: yes
- Spam and virus scanning: yes
You can use the optional SNS topic to send notifications or trigger automations if you want.
This setup assumes you want emails to any address at your domain to reach the same inbox. If you only want to use certain addresses, you can change the Recipient to an email address instead of just your domain. See the AWS documentation for more details.
The final piece of the puzzle is getting emails from Amazon S3 to your Dovecot server. This is the most complex part.
Here's how to set up access to S3 from your server:
- Create an AWS User that can access the S3 bucket
- Go to the IAM console
- Create a user and enter a username
- Choose "Programmatic access"
- Choose "Attach existing policies directly"
- Create a policy
- Service: S3
- Actions: HeadBucket, ListBucket, GetObject, GetObjectAcl, DeleteObject
- Bucket: choose the bucket you created earlier
- Object: tick the "Any" box
- Request conditions: optionally you can restrict this access token to the IP address of your server
- Go back to the Add user page and attach the policy you created
- Create credentials for the user
- Install s3fs-fuse
- Create the file
/etc/passwd-s3fscontaining the credentials you generated, in the format
ACCESS_KEY_ID:SECRET_ACCESS_KEY. Make sure its permissions are 600.
- Add an entry to
/etc/fstabto mount your bucket somewhere, e.g.
/mnt/my_bucket_mountpoint(the directory must exist):Then run
my_bucket_name /mnt/my_bucket_mountpoint fuse.s3fs _netdev,allow_other 0 0
mount -ato mount it.
Next, you need to create a webhook listener that can
- Set up a webserver and CGI such as Apache or nginx. Tutorials for this are available on the internet. Alternatively, you could use a framework such as Express for Node.js, or Flask for Python. If you are running this server at home you'll need to forward port 80/443 on your router
Create a script running on your webserver that can be accessed from the internet. For security, you should have some simple authentication with a GET parameter. The script will need to move files from the mounted S3 bucket into the
mail/Inbox/new/directory of the IMAP user you created earlier.
- Here is an example of such a script, written using CGI and Python:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
#!/usr/bin/python import hashlib import os import sys import traceback import urllib.parse KEY = "50a43ff3bdaec2884af24b6d3719ec4447d9fdb7d1a7cb895be00f9b5a94406a0737b088baf97bd4093b92b96cdf04b9d04d9c7d7746c947a9d97ab66ccaad81" def run(): os.system("/usr/bin/sudo /srv/copyemail.sh") def main(): print("Content-Type: text/plain") qs = os.getenv("QUERY_STRING") if not qs: raise ValueError() q = urllib.parse.parse_qsl(qs) for k, v in q: if k == "key": if hashlib.sha512(v.encode()).hexdigest() == KEY: print("\nOK") run() sys.exit() raise ValueError() if __name__ == "__main__": try: main() except Exception as e: print("Status: 400 Bad Request\n") print("ERROR") traceback.print_exc(file=sys.stderr) sys.exit()
- Then a script to actually move the emails (
1 2 3 4
#!/bin/sh /usr/bin/mv /mnt/my_bucket_mountpoint/* /home/SOMEUSER/mail/Inbox/new/ /usr/bin/chmod 600 /home/SOMEUSER/mail/Inbox/new/* /usr/bin/chown SOMEUSER:SOMUSER /home/SOMEUSER/mail/Inbox/new/*
- Make both scripts executable. Finally, grant passwordless sudo access to your webserver's user for the
copyemails.shscript - add this line to the sudoers file using
www-data ALL=(ALL): NOPASSWD /srv/copyemail.sh
- Subscribe this webhook to the Amazon SNS topic you created earlier from the SNS dashboard
- Create a subscription - choose the topic
- Choose HTTP or HTTPS for the protocol (HTTPS should be preferred)
- Leave the rest of the fields as default
- Confirm the subscription
- You will need to modify your CGI script to report the confirmation request
- Go to the Subscriptions page on the SNS dashboard, choose the one you created, and click "Request confirmation"
- Get the Confirmation URL from the request sent to your script and visit it in a web browser
- Check that the subscription is confirmed correctly on the SES dashboard, then return your script to how it was befores
- Here is an example of such a script, written using CGI and Python:
SMTP access to SES¶
Create SMTP credentials using the Amazon SES dashboard. You'll need these to send any email from your email client.
Your email client¶
This was the hardest part of the setup for me - figuring out how to get Thunderbird to use the right server settings (the same idea should apply to whatever email client you use). Here's what you need:
- Server: the domain name of your Dovecot server
- Port: 143
- Password Authentication using the username and password for the account
Server: the appropriate one of:
- Password Authentication using the SMTP username and password generated by AWS in the previous step
As you may have noticed while testing your shiny new system, you can only send emails to addresses that you've verified. If you just want to use your domain for email aliases that you provide to unstrustworthy websites, or for junk, that might be fine. However, most people will at some point want to send emails to other people too. Luckily working around this is not too hard, but it does require talking to a human.
Go to the Amazon's Sender Limit Increase form and explain your purpose. I was quickly and helpfully moved out of the Sandbox, removing this limitation (and letting me send 50,000 emails per day which I certainly don't need!). For reference, here's what I entered in the form:
- Limit Type: SES Sending Limits
- Mail Type: Other
- Website URL: https://www.pxeger.com/2020-07-02-hybrid-cloud-email-with-amazon-ses-and-dovecot/
Describe, in detail, how you will only send to recipients who have specifically requested your mail:
I will only send emails manually, in very low volumes, and only every in response to people who contact me.
Describe, in detail, the process you will follow when you receive bounce and complaint notifications:
I will attempt to resolve the particular issue by tracing and correcting the root cause, and I will review my systems to ensure bounces and complaints don't happen in the future
Region: choose the same region you used for the whole process (for me it's
- Limit: Desired Daily Sending Quota
- New limit value: 50 (this seems like the most I could expect to send on a busy workday if I began using this email system for work communication)
Use case description:
I use Amazon SES to implement hybrid-cloud-hosted email on a custom domain for my personal brand. You can learn more about the setup at https://www.pxeger.com/2020-07-02-hybrid-cloud-email-with-amazon-ses-and-dovecot/. I will only send emails in response to emails I receive, or to contact people who have requested me to contact them. All emails will be sent manually using an SMTP client, and in very low volumes. If I recieve any bounces or complaints, which is unlikely given my use case, I will follow up the cause of these issues, address the root cause, apologise if appropriate/necessary, and review my monitoring to prevent further incidents.
I've been running this system for approximately 15 months, and it hasn't failed me yet.
If you have any feedback, feel free to email me to let me know about any more.
From here, you could consider:
- modifying your webhook script with rules to automatically move emails marked as spam by Amazon SES (identified by the
- creating a DMARC policy
- setting up PGP encryption on your email (because email is inherently an insecure communication system)
Here is a list of known issues with this system:
- Amazon SES limits emails to 10MB, including attachments
- AWS will scan all your emails for viruses. If you are a privacy extremist this might be an issue, but personally I trust them not to exploit this
First published: 2020-07-02
Last updated: 2021-11-27