Deploying a public mail server on Docker

In this guide, we'll deploy a production-ready public mail server using Docker Mailserver. We'll configure DNS records, SPF, DKIM and DMARC so emails can reach Outlook and Gmail mailboxes.

Ressource requirements

In idle, the container uses approx 1.5G of memory, my hardware recommendations for a personnal instance would be:

At least 2GB of RAM
2vCores
20G of storage

Check your IP reputation and SMTP accessibility

Before deploying an SMTP server at home or on a VPS, it is important to check the reputation of your IP. It can sometimes be hard to improve the reputation of an IP, so be sure to start with a clean one.
You can use this tool to check your IP presence in reputation lists : apivoid.com

Sometimes, VPS providers block outcoming SMTP, use this command to check that your instance can talk to other SMTP servers on the internet.

nc -vz gmail-smtp-in.l.google.com 25

Email security mechanics

Domain authentication

To prevent unauthorized servers from sending emails using spoofed domains, three security mechanisms have been implemented in the mailing system.
These mechanisms are important to understand, as major email providers such as Google, Outlook, and Yahoo use them to authenticate incoming messages. If they are incorrectly configured, emails sent from our server may be marked as spam or, in some cases, rejected entirely.

DKIM

DomainKeys Identified Mail is an email authentication method that adds a cryptographic signature to outgoing messages. The sending mail server signs parts of the email with a private key, and receiving servers verify the signature using a public key published in DNS. This ensures the message was not altered in transit and confirms it was genuinely sent by the domain it claims to represent.

SPF

Sender Policy Framework is a mechanism that defines which mail servers are allowed to send emails on behalf of a domain. It works by publishing a DNS record listing authorized IP addresses or services. When an email is received, the recipient checks whether the sending server’s IP is in that list, helping prevent spoofed sender addresses.

DMARC

Domain-based Message Authentication, Reporting and Conformance builds on SPF and DKIM by defining how receiving servers should handle emails that fail authentication checks. It enforces a policy (none, quarantine, or reject) and ensures alignment between the visible “From” address and authenticated domains. It also provides reporting, allowing domain owners to monitor abuse and delivery issues.

Deploying docker-mailserver

Creating the certificates

Before deploying the mail server, we will need to request SSL certificates from letsencrypt.
For this, we will use acme.sh

apt install acme.sh

Request the certificate using this command

acme.sh --issue -d "mail.example.com" --keylength 4096 --server letsencrypt

Copy the certificate to /etc/ssl directory

export DOMAIN=mail.example.com
acme.sh --install-cert -d ${DOMAIN} \
  --key-file /etc/ssl/${DOMAIN}/privkey.pem \
  --fullchain-file /etc/ssl/${DOMAIN}/fullchain.pem \
  --cert-file /etc/ssl/${DOMAIN}/cert.pem

The compose file

docker-compose.yml

services:
  mail.example.com: # Replace with the domain of your mail instance
    image: ghcr.io/docker-mailserver/docker-mailserver:15.1.0
    container_name: mail.example.com # Replace with the domain of your mail instance
    hostname: mail
    domainname: example.com # Replace with the domain of your mail instance
    restart: unless-stopped
    networks:
      - mailserver
    ports:
      - "25:25"     # SMTP
      - "465:465"   # SMTPS
      - "587:587"   # Submission
      - "143:143"   # IMAP
      - "993:993"   # IMAPS
    volumes:
      - ./docker-data/mail-data:/var/mail
      - ./docker-data/mail-state:/var/mail-state
      - ./docker-data/mail-logs:/var/log/mail
      - ./docker-data/config:/tmp/docker-mailserver
      - /etc/ssl/example.com:/etc/ssl/example.com:ro # Replace with your SSL certificate directory
      - /etc/localtime:/etc/localtime:ro

    env_file:
      - path: config.env
        required: true
    cap_add:
      - NET_ADMIN
    stop_grace_period: 1m
networks:
  mailserver: # Used for other containers to connect via SMTP and IMAP (Webmail app, notifications, ...)
    external: true

config.env

ENABLE_SPAMASSASSIN=1 # For spam detection
ENABLE_CLAMAV=1 # Antivirus
ENABLE_FAIL2BAN=1 # Scan filter
ENABLE_POSTGREY=0
ENABLE_OPENDKIM=1 # Domain Key identification
ENABLE_OPENDMARC=1 # For email authentication policy
ENABLE_POLICYD_SPF=1 # Sender Policy
ENABLE_IMAP=1 # For email consulting
SPOOF_PROTECTION=1 # Prevent users from sending emails with another address
DOVECOT_MAILBOX_FORMAT=maildir
MOVE_SPAM_TO_JUNK=1 # Move spam messages to Junk folder
SSL_TYPE=manual
SSL_CERT_PATH="/etc/ssl/example.com/fullchain.pem" # Change with your own SSL certificate
SSL_KEY_PATH="/etc/ssl/example.com/privkey.pem" # Change with your own SSL private key
ONE_DIR=1 # Store all emails in a single directory
PERMIT_DOCKER=network
POSTMASTER_ADDRESS="postmaster@example.com"
OVERRIDE_HOSTNAME=mail.example.com # The content of the PTR record pointing to your instance

Create the docker network

docker network create mailserver

Start the mail server

docker compose up -d

Creating the DNS records and configuring the mail server

All configuration actions will be done using the setup command in the mailserver container.

Creating the MX record for mail reception

Example record for the example.com zone, replace example.com with your domain.

@ IN MX 10 mail.example.com.
mail IN A 203.0.113.10

Creating DKIM keys

docker exec mail.example.com setup config dkim domain 'example.com'

Then copy the domain-key TXT record to the clipboard

cat docker-data/config/openskim/keys/example.com/mail.txt

Connect to your DNS server and create a new TXT record for the DKIM key

mail._domainkey IN TXT ( "v=DKIM1; h=sha256; k=rsa; " "p=MIIBIj..." "KbDuomP..." )

Setting up SPF and DMARC policy

SPF and DMARC policies will be declared through TXT records on our domain
Configuring DMARC policy with notification to postfaster in case of mail spoofing

@ IN TXT "v=DMARC1; p=quarantine; rua=mailto:postmaster@example.com"

Configuring SPF policy (replace 203.0.113.10 with your instance public IP).
The -all at the end of the record tells mail servers to drop all emails from unauthorized IPs. You can also use ~all to tell mail servers to only mark them as suspect.

@ IN TXT "v=spf1 ip4:203.0.113.10 -all"

Creating users

Creating an email account

docker exec mail.example.com setup email add user@example.com "Password123!"

Creating email aliases

docker exec mail.example.com setup alias add admin@example.com user@example.com
docker exec mail.example.com setup alias add postmaster@example.com user@example.com