How to Set Up Mail Server Using Docker

There are many different ways of setting up a mail server in Linux. I have chosen to use Docker as it streamlines the setup process and provides the same setup environment regardless of the underlying OS. I am using the mailserver/docker-mailserver image. It doesn’t need any SQL database and it has great documentation at https://docker-mailserver.github.io/docker-mailserver/edge/.

I am running my mailserver on a Hetzner VPS. I cannot set a reverse DNS for my public IP address at home, but I can using a Hetzner VPS. This is an important authentication step for sending mail. Without the reverse DNS, there is a higher chance of mail providers marking your mail as spam.

Installation

Create a folder for the mailserver’s configuration files to be stored and enter this folder.

mkdir mailserver && cd mailserver

Enter this command to pull the necessary files from the github:

DMS_GITHUB_URL='https://raw.githubusercontent.com/docker-mailserver/docker-mailserver/master'
wget "${DMS_GITHUB_URL}/docker-compose.yml"
wget "${DMS_GITHUB_URL}/mailserver.env"
wget "${DMS_GITHUB_URL}/setup.sh"
chmod a+x ./setup.sh

This downloads the docker compose file, the environment file and a setup script.

Setup

Open the docker compose file, and change the hostname and domain name accordingly.

The rest of the options can be left at default. The data files are stored in the current directory. All mail ports are opened, but this is okay as we will setup secure authentication using the correct ports later.

The container can now be started.

docker compose up -d

A mail account needs to be setup, so we can use the setup script.

./setup.sh email add test@example.com

You will be prompted to enter a password.

Firewall

To run a mailserver, you must open the appropriate ports on the firewall. I am using UFW. This command opens all the required ports:

sudo ufw allow 25,143,165,587,993/tcp

Of course you may also need to forward ports on your router depending on your setup.

You should be able to receive emails now. Sending mail successfully requires more work.

Configuration

DKIM

DKIM is used to prevent email spoofing. It is recommended to activate it.

A DKIM signature is enabled with the command:

./setup.sh config dkim

The mailserver now needs to be restarted.

A TXT record called mail._domainkey needs to be added to your DNS server. Its content can be found in docker-data/dms/config/opendkim/keys/example.com/mail.txt. The record is stored in four sets of quotes. Everything inside the quotes can be pasted in order into the TXT record. This is an example:

v=DKIM1; k=rsa; p=AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN

DMARC

DMARC is used to verify email senders. It is pre-configured by docker-mailserver. A TXT record called _dmarc has to be added to your DNS server.

This record is a lenient policy to start with while setting up:

v=DMARC1; p=none; rua=mailto:dmarcreport@example.com; ruf=mailto:dmarcreport@example.com; sp=none; ri=86400

Or this is a stricter policy for when the mailserver is fully set up:

v=DMARC1; p=quarantine; rua=mailto:dmarcreport@example.com; ruf=mailto:dmarcreport@example.com; fo=0; adkim=r; aspf=r; pct=100; rf=afrf; ri=86400; sp=quarantine

SPF

SPF is an validation system to detect email spoofing using DNS. It comprises of a TXT record called example.com in your DNS.

This enables SoftFail mode which lets all mail through:

v=spf1 mx ~all

After setup, the HardFail policy should be enforced:

v=spf1 mx -all

mx refers to the mail servers pointed to by the DNS.

SSL/TLS

SSL can be set up in many different ways, but the easiest way is to use letsencrypt with Certbot.

A volume and environment variable must be added to docker-compose.yml:

services:
    mailserver:
        hostname: mail
        domainname: example.com
        environment:
            - SSL_TYPE=letsencrypt
        volumes:
            - ./docker-data/certbot/certs:/etc/letsencrypt

The environment variable can also be set in mailserver.env.

A new certbot container must also be added:

services:
    certbot:
        container_name: certbot
        image: certbot/certbot:latest
        volumes:
            - ./docker-data/certbot/certs:/etc/letsencrypt

To get a certificate from letsencrypt:

docker compose run --rm certbot certonly --webroot --webroot-path /var/www/certbot -m example@mail.com --agree-tos --no-eff-email -v -d mail.example.com

A crontab can be used to run a command to renew the certificate:

docker compose run --rm certbot renew

Environment Variables

All the integrated services this image comes with can be enabled and configured in the environment file mailserver.env, as well as other configuration options. Environment variables can also be set directly in docker-compose.yml. The environment file can be specified in docker-compose.yml:

env_file: mailserver.env

Here are some useful variable to set:

ENABLE_SPAMASSASSIN=1
ENABLE_CLAMAV=1
ENABLE_FAIL2BAN=1

Spamassassin is used to detect spam.

ClamAV is used to detect malware including viruses.

Fail2Ban bans IP addresses for too many failed attempts.

Conclusion

When all of these steps are completed you can use a website like mail-tester.com to check if everything is configured correctly.