self-hosted email mail server Postfix Dovecot SPF DKIM DMARC deliverability security

Self-Hosted Mail Server: The Complete, Low-Level Guide (AWS + NoServerMail Edition)

A deep, system-level walkthrough of how to run your own mail server on AWS covering DNS, SES, Lambda-based routing, S3/DynamoDB storage, TLS, authentication, spam control, logging, and deliverability tuning with NoServerMail.

NoServerMail Team
2025-02-09

Self-Hosted Mail Server: The Complete, Low-Level Guide

Running your own mail server is absolutely possible but to do it right you need to understand all the moving pieces: DNS, MTAs/MDAs (or managed equivalents), TLS, authentication, anti-spam, queue management, and monitoring. This guide breaks down the nitty gritty with an AWS-first view and shows how NoServerMail wires these components together so you control the domain and data while offloading server babysitting.

1) Core Building Blocks (Know These Terms Cold)

Component What It Does In AWS with NoServerMail
MTA (Mail Transfer Agent) Receives mail from the internet and relays it onward. Handles SMTP, queues, retries, bounces. Amazon SES handles SMTP ingress/egress; NoServerMail orchestrates policies and retries.
MDA/LDA (Mail Delivery Agent) Final delivery to the mailbox. Speaks LMTP/LDA, applies filters. Lambda-based delivery pipelines fan messages into S3 and DynamoDB for storage/indexing.
MUA (Mail User Agent) The client users see (webmail or desktop/mobile). NoServerMail web inbox + API; IMAP/POP optional via gateway.
IMAP/POP Service Exposes stored mail to MUAs with authentication. Provided by NoServerMail’s managed interface or optional IMAP bridge.
Spam/Malware Filter Scores, tags, or rejects messages. SES spam/virus scanning plus optional rspamd/SpamAssassin Lambda hooks.

Mental model: The MTA is your mail router, the MDA is your local post office, and IMAP is the counter where users pick up mail.

2) Infrastructure Prerequisites (AWS View)

  • Domain ownership in Route 53 (or any registrar) and ability to add DNS records (MX, SPF, DKIM, DMARC, MTA-STS, TLSRPT).
  • SES-enabled region with production access (e.g., us-east-1, eu-west-1). Remove sandbox limits by verifying domain and requesting production.
  • NoServerMail stack (CloudFormation/Terraform) deployed with Lambda, S3, DynamoDB, and API Gateway endpoints.
  • TLS certificates for web and submission endpoints (ACM for HTTPS/API, SES manages SMTP TLS; DKIM keys provisioned via SES/NoServerMail).
  • IAM least privilege: execution roles for Lambda, restricted S3 buckets, DynamoDB tables with KMS encryption.

3) DNS Records Checklist (AWS + NoServerMail)

Record Purpose Example
MX Points to your mail server example.com. IN MX 10 inbound-smtp.${region}.amazonaws.com. (SES)
A/AAAA Resolves submission/API hostname mail.example.com. IN A 203.0.113.10 or CloudFront/ALB target
PTR Reverse DNS for your IP Handled when using static IPs/Elastic IPs for custom MTAs; SES-managed IPs come pre-set
SPF (TXT) Authorizes sending hosts v=spf1 include:amazonses.com include:_spf.noservermail.io -all
DKIM (TXT) Publishes public key selector1._domainkey.example.com. IN TXT "v=DKIM1; p=..."
DMARC (TXT) Aligns SPF/DKIM & sets policy _dmarc.example.com. IN TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com"
MTA-STS (TXT + HTTPS) Advertises TLS policy _mta-sts.example.com TXT "v=STSv1; id=20250209" + https://mta-sts.example.com/.well-known/mta-sts.txt (served from S3+CloudFront)
TLSA (DANE) (Optional) pins certs for SMTP Rare with SES; more relevant for self-IP MTAs

Tip: Update the DMARC rua address to a mailbox you actually monitor; aggregate reports are invaluable for spotting spoofing and alignment issues.

4) TLS and Certificates (SES + ACM)

  • Inbound SMTP: SES terminates TLS with modern ciphers; you control the policy by region. STARTTLS is opportunistic but widely honored.
  • Submission/API: Use ACM certificates on API Gateway/ALB/CloudFront for web inbox and submission endpoints. Force TLS 1.2+.
  • DKIM keys: Managed by SES/NoServerMail; selectors rotated automatically and published as TXT.
  • Strict transport: Serve MTA-STS policy via S3 static site + CloudFront; publish TLSRPT TXT for reports.

5) Authentication and Access Control (Users + Pipelines)

  • User auth: NoServerMail issues credentials/tokens for web/API submission; optionally integrates with your IdP (OIDC/SAML) via Cognito.
  • Envelope enforcement: Sender restrictions enforced in Lambda before SES handoff to prevent spoofing from compromised users.
  • Password policy: Enforce strong secrets and MFA for admin access; store hashes with Argon2/scrypt; rotate API keys.
  • Rate limiting: API Gateway + Lambda throttles abusive clients; SES sending quotas prevent runaway abuse.
  • Access isolation: S3 buckets are private; DynamoDB tables require signed IAM requests; CloudWatch logs restricted to ops roles.

6) Queueing, Retries, and Bounces (Managed by SES + Lambda)

  • SES maintains SMTP queues and retries with sane defaults; you monitor via CloudWatch metrics (Delivery, Bounce, Complaint).
  • NoServerMail subscribes to SES event destinations (SNS/SQS) and drives Lambda processors to classify hard vs. soft bounces.
  • Customizable bounce templates and automatic suppression lists prevent repeat failures; logs land in S3 for auditability.

7) Spam and Malware Controls (Layered)

  • SES inbound filters provide baseline spam/virus detection.
  • Lambda content hooks let you inject rspamd/SpamAssassin scoring and ClamAV scans before storage.
  • RBL/DBL checks can run in Lambda (cached lookups) to avoid latency; whitelist major providers to reduce false positives.
  • Authentication-Results headers added by NoServerMail pipelines to aid debugging; failing SPF/DKIM/DMARC can be quarantined to a review folder.
  • Outbound rate limits via SES sending quotas and per-user throttles protect your domain reputation.

8) Deliverability Factors You Control (AWS-Specific Levers)

  1. Reputation of IP and domain: Avoid sending from fresh or previously abused ranges; warm up gradually (start with <500 messages/day, then ramp).
  2. Alignment: Ensure SPF and DKIM align with the visible From: domain; DMARC policy helps receivers trust you.
  3. Content hygiene: No URL shorteners, avoid spammy phrases, include plain-text and HTML parts, and include an unsubscribe link for bulk mail.
  4. Feedback loops (FBLs): Register with major providers (Microsoft SNDS/JMRP, Yahoo CFL) to receive complaint data.
  5. Consistent HELO: SES-managed IPs already present proper HELO; with dedicated IPs, set custom MAIL FROM domains for alignment.
  6. TLS success rate: SES STARTTLS success is high; monitor CloudWatch Reputation.BounceRate and TLS metrics where available.

9) Storage, Backups, and Retention (S3 + DynamoDB)

  • Message bodies stored in S3 with bucket policies/KMS encryption; versioning optional for legal hold.
  • Metadata and search indexes in DynamoDB with TTL-based expiry for spam/junk folders.
  • Backups: Cross-region replication for S3; Point-in-Time Recovery (PITR) for DynamoDB. Export logs to S3 for long-term audit.
  • Retention policies: Lifecycle rules on S3 for auto-expiry; DynamoDB TTL for folders with shorter retention.

10) Observability and Operations (CloudWatch-Centric)

  • Logs: SES event logs + Lambda logs in CloudWatch; ship to S3 or OpenSearch if you prefer longer retention or full-text search.
  • Metrics: SES provides sending/delivery/bounce/complaint metrics; NoServerMail surfaces queue depth, processing latency, TLS success, and auth failures.
  • Tracing: X-Ray can wrap Lambda functions for cold-start and downstream latency; use structured JSON logs for fast querying.
  • Health checks: Synthetic canaries (Lambda or external) send/receive from major providers to verify DNS/TLS/auth alignment continuously.

11) Minimal Postfix + Dovecot Blueprint (For DIY EC2)

# Hostname & packages
hostnamectl set-hostname mail.example.com
apt update && apt install postfix dovecot-imapd dovecot-lmtpd opendkim opendkim-tools rspamd

# Postfix main.cf (highlights)
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
mydestination = localhost
mynetworks = 127.0.0.0/8 [::1]/128
inet_interfaces = all
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_security_level = may
smtp_tls_security_level = dane
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_recipient_restrictions =
  permit_mynetworks,
  permit_sasl_authenticated,
  reject_non_fqdn_recipient,
  reject_unknown_recipient_domain,
  reject_unauth_destination

# Dovecot (highlights)
protocols = imap lmtp
mail_location = maildir:~/Maildir
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0600
    user = postfix
    group = postfix
  }
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}
auth_mechanisms = plain login
passdb {
  driver = pam
}
userdb {
  driver = passwd
}

Adjust ownership (vmail users), storage paths, and SASL backends (SQL/LDAP) as needed. Reload services after edits and run postfix check for syntax validation. For most users, the SES + Lambda + S3 + DynamoDB path above eliminates the need to run and patch these daemons yourself.

12) Common Failure Modes and Fixes

  • “Relay access denied”: Your smtpd_recipient_restrictions lack permit_sasl_authenticated, or the sender is not in allowed domains.
  • “Connection refused” on port 25: ISP/host blocks outbound SMTP; you may need a smart host/relay service.
  • Bounces for SPF fail: Missing or incorrect SPF record; ensure the sending IP is listed and From: domain matches.
  • DKIM signature invalid: Clock skew, modified body (mismatched canonicalization), or wrong selector in DNS.
  • High spam score: Check RBL listings, enable PTR, warm up IP, and tune content (plain-text part, unsubscribe links).

13) Hardening Checklist

  • [ ] Enforce submission on 587 with STARTTLS + SASL; disable plain 25 for authenticated users.
  • [ ] Enable postscreen or fail2ban for abusive IPs.
  • [ ] Lock down SSH (keys only, firewall, Fail2ban) to protect the host itself.
  • [ ] Rotate DKIM keys yearly; monitor DMARC alignment.
  • [ ] Keep packages patched; watch for CVEs in OpenSSL, Postfix, Dovecot, rspamd.

14) How NoServerMail Maps the Moving Parts on AWS

  • DNS & identity: Wizard guides you through adding MX/SPF/DKIM/DMARC/MTA-STS/TLSRPT records; verifies SES identities automatically.
  • Ingress: SES receives SMTP on AWS edge; STARTTLS supported by default. Dedicated IPs optional for reputation isolation.
  • Routing: SES event destinations -> SNS/SQS -> Lambda processors that classify, scan, and deliver.
  • Storage & search: S3 for bodies/attachments (KMS encrypted); DynamoDB for metadata, search indexes, and per-folder TTLs.
  • Submission: Authenticated web/API endpoints fronted by CloudFront/API Gateway with ACM certs; policy enforcement before SES send.
  • Observability: CloudWatch metrics/alarms for send rates, bounces, complaints, latency; structured logs shipped to S3/OpenSearch.
  • Security & resilience: IAM least privilege, VPC endpoints for S3/DynamoDB (optional), multi-AZ Lambda, lifecycle policies, cross-region replication options.
  • Operational guardrails: Automated DKIM rotation, bounce/complaint suppression, rate limiting, and synthetic probes to keep deliverability healthy.

15) Takeaways

  • A production mail server demands DNS alignment, TLS, auth, spam controls, and observability skip any piece and you invite deliverability or security problems.
  • On AWS, SES + Lambda + S3 + DynamoDB cover the heavy lifting; NoServerMail stitches them together with policy, UI, and automation.
  • If you want control with minimal ops toil, deploy NoServerMail and focus on your product instead of queue drain charts and blocklists.
Back to all articles Go to homepage