#21: make the stack deployable, and document how

Three things stood between this and a real deployment.

A missing JWT_SECRET fell back to a constant that is published in this
repository, so any deployment that forgot the variable would have accepted
session tokens anyone could mint, including an admin one. The auth service
now refuses to start without a secret of at least 32 characters. The test
suites set it through vitest config, because ESM hoists imports and an
assignment inside a test file runs too late.

Session cookies were issued without the Secure attribute: it is driven by
NODE_ENV, which compose never set. Behind a TLS terminator that is a
downgrade waiting to happen.

Nothing created the first mailbox. A fresh install has no domain and no
account, and every route into the console needs one - so there was no
first login. There is now a create-admin command, which also provisions
the standard folders and the four system aliases. It is a deliberate
command rather than a bootstrap on boot: creating an administrator as a
side effect of starting a container is how a deleted account quietly comes
back.

Also: the production images were shipping compiled test files, since the
build compiled everything under src. A separate build config excludes
them while lint still type-checks them.

Documentation. README.md covers what this is, how it is put together, and
what is not finished - stated plainly, because a mail server that half
works loses mail. docs/DEPLOYMENT.md is the Coolify path in the order it
has to happen, starting with the three prerequisites no code can satisfy:
outbound port 25, the PTR record, and the DNSSEC delegation.
docs/OPERATIONS.md is the runbook - health states, certificates, backups,
reading a delivery failure, DKIM rotation, and an explicit list of what is
not automated. .env.production.example replaces guessing which of the dev
defaults are safe: none of them are.

docs/ was gitignored as a place for internal notes; only docs/internal/ is
now, so these guides ship with the code.

Every command the documentation tells an operator to run was executed
against a running stack: create-admin, preflight, the health endpoint, the
published mail ports, and the Secure cookie.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.env.production.example

@@ -0,0 +1,86 @@
+# LambdaMail - production environment
+#
+# Copy this into Coolify's environment editor, or to .env on the host, and
+# replace every value marked CHANGE. The stack refuses to start without the
+# required ones rather than falling back to a default, because a default
+# secret in a published repository is not a secret.
+#
+# Generate each secret with:  openssl rand -base64 36
+
+# --- Container images -------------------------------------------------------
+# Published by .github/workflows/release.yml for linux/amd64 and linux/arm64.
+# Pin IMAGE_TAG to a released version or a sha- tag to roll back.
+IMAGE_NAMESPACE=ghcr.io/lucasrguerra/lambdamail
+IMAGE_TAG=latest
+
+# --- Identity ---------------------------------------------------------------
+# MAIL_DOMAIN is the domain you send and receive as.
+# PRIMARY_MAIL_HOST is the host the MX record points at; its PTR must match.
+MAIL_DOMAIN=example.com                        # CHANGE
+PRIMARY_MAIL_HOST=mail.example.com             # CHANGE
+WEBMAIL_HOST=mail.example.com                  # CHANGE - where the web UI is served
+ADMIN_HOST=                                    # optional separate host for /admin
+PUBLIC_IPV4=203.0.113.10                       # CHANGE - the VPS public IPv4
+# Leave empty unless IPv6 has a matching PTR and is covered by SPF. Half
+# configured IPv6 is a common cause of rejection (PLAN.md premises).
+PUBLIC_IPV6=
+
+# --- Secrets ----------------------------------------------------------------
+POSTGRES_DB=lambdamail
+POSTGRES_USER=lambdamail
+POSTGRES_PASSWORD=CHANGE-generate-with-openssl-rand
+REDIS_PASSWORD=CHANGE-generate-with-openssl-rand
+# Signs webmail and admin sessions. Minimum 32 characters; the auth service
+# refuses to start below that.
+JWT_SECRET=CHANGE-generate-with-openssl-rand-at-least-32-chars
+# Encrypts DKIM private keys, TOTP secrets and provider tokens at rest.
+# LOSING THIS MAKES EVERY STORED SECRET UNRECOVERABLE - back it up separately
+# from the database (PLAN.md risk R9).
+LAMBDAMAIL_MASTER_KEY=CHANGE-generate-with-openssl-rand-at-least-32-chars
+
+# --- DNS automation ---------------------------------------------------------
+# Token scope: Zone.DNS:Edit plus Zone.Zone:Read on the zone. Leave empty to
+# manage the 13 records by hand; the service then logs that sync is off.
+CLOUDFLARE_API_TOKEN=
+DNS_SYNC_INTERVAL_SECONDS=21600
+
+# --- TLS --------------------------------------------------------------------
+# traefik: read the certificate Coolify's proxy already manages (default).
+# acme:    obtain it here via DNS-01. Required for DANE, and only then.
+TLS_MODE=traefik
+COOLIFY_PROXY_DIR=/data/coolify/proxy
+COOLIFY_PROXY_NETWORK=coolify
+TRAEFIK_ACME_FILENAME=acme.json
+CERT_POLL_INTERVAL_SECONDS=60
+PROTOCOLS_SPOOL_DIR=/var/lambdamail/spool
+# Used only when TLS_MODE=acme.
+ACME_EMAIL=
+ACME_DIRECTORY_URL=
+
+# --- Transport security -----------------------------------------------------
+# Start at testing. Promote to enforce only once TLS-RPT reports come back
+# clean: enforce with a broken certificate silently blackholes inbound mail.
+MTA_STS_MODE=testing
+OUTBOUND_MTA_STS_ENABLED=true
+# DANE stays off under TLS_MODE=traefik. The key changes at every renewal and
+# the published TLSA record stops matching, which permanently rejects mail at
+# every validating MTA (PLAN.md section 5.1).
+OUTBOUND_DANE_ENABLED=false
+DNSSEC_RESOLVER=1.1.1.1:53
+MAX_MESSAGE_BYTES=52428800
+ARC_SEAL_ENABLED=false
+
+# --- Smarthost --------------------------------------------------------------
+# Most VPS providers block outbound port 25. If the preflight reports it
+# blocked, set a relay here; it is added to the published SPF automatically.
+RELAY_HOST=
+RELAY_PORT=587
+RELAY_USER=
+RELAY_PASS=
+
+# --- Storage ----------------------------------------------------------------
+STORAGE_DRIVER=local
+S3_ENDPOINT=
+S3_BUCKET=
+S3_ACCESS_KEY=
+S3_SECRET_KEY=

.gitignore

@@ -1,7 +1,10 @@
-# Internal planning docs — never published with the code
+# Internal planning docs - never published with the code
 PLAN.md
-docs/
+DEVELOPMENT_RULES.md
 .superpowers/
+# docs/ holds the deployment and operations guides and IS published; keep
+# internal notes out of it by name rather than ignoring the whole directory.
+docs/internal/
 
 # Dependencies
 node_modules/

Makefile

@@ -1,4 +1,4 @@
-.PHONY: up down logs test lint migrate-up migrate-down gen-dev-cert seed
+.PHONY: up down logs test lint migrate-up migrate-down gen-dev-cert seed preflight create-admin reset-password
 
 up:
 	docker compose --env-file .env up --build
@@ -35,3 +35,23 @@ gen-dev-cert:
 
 seed:
 	docker compose exec -T db psql -U $${POSTGRES_USER:-lambdamail} -d $${POSTGRES_DB:-lambdamail} < scripts/dev-seed.sql
+
+# Environment diagnostics (PLAN.md section 15). Run this before pointing DNS at
+# the host: it reports outbound port 25, PTR/FCrDNS, RBL listings and the
+# Cloudflare token scope.
+preflight:
+	docker compose run --rm --entrypoint /app/lambdamail-protocols protocols preflight
+
+# First login on a fresh deployment. Nothing else creates a mailbox, so without
+# this there is no way into the admin console.
+#   make create-admin EMAIL=you@example.com PASSWORD='...'
+create-admin:
+	@test -n "$(EMAIL)" || (echo "EMAIL is required" && exit 1)
+	@test -n "$(PASSWORD)" || (echo "PASSWORD is required" && exit 1)
+	docker compose run --rm --entrypoint node auth dist/cli.js create-admin "$(EMAIL)" "$(PASSWORD)"
+
+# Recovers a locked-out account. Every session it had is signed out.
+reset-password:
+	@test -n "$(EMAIL)" || (echo "EMAIL is required" && exit 1)
+	@test -n "$(PASSWORD)" || (echo "PASSWORD is required" && exit 1)
+	docker compose run --rm --entrypoint node auth dist/cli.js reset-password "$(EMAIL)" "$(PASSWORD)"

README.md

@@ -0,0 +1,148 @@
+# LambdaMail
+
+A self-hosted mail server with an integrated webmail and admin console, built to
+run on [Coolify](https://coolify.io) from a single `docker-compose.yaml`.
+
+SMTP, IMAP, POP3 and ManageSieve, with SPF, DKIM, DMARC, ARC, MTA-STS and
+TLS-RPT, DNS automation through Cloudflare, and spam and virus filtering. No
+paid tiers and no artificial limits on domains or mailboxes.
+
+> **Status: pre-release.** The mail path, the security posture and the web
+> interfaces work end to end, and there is an automated test suite behind them.
+> Several console panels are still read-only, and some features are recorded but
+> not yet acted on — see [What is not finished](#what-is-not-finished) before
+> putting real mail through it.
+
+---
+
+## What it does
+
+| Area | |
+| :--- | :--- |
+| **Protocols** | SMTP MX (25), Submission (587 STARTTLS, 465 implicit TLS), IMAP (143/993) with UID, flags, IDLE and CONDSTORE, POP3 (110/995), ManageSieve (4190) |
+| **Authentication** | SPF, DKIM (Ed25519 + RSA dual signature), DMARC, ARC sealing, DANE/TLSA and MTA-STS on delivery, TLS-RPT ingestion |
+| **Filtering** | Rspamd scoring with greylisting, RBLs and rate limiting; ClamAV; Sieve |
+| **Delivery** | Durable outbound queue in Postgres with the retry ladder and RFC 3464 bounces; smarthost fallback for hosts with port 25 blocked |
+| **DNS** | Cloudflare reconciliation of the 13 records, idempotent, with drift detection |
+| **Web** | Webmail at `/user/*` and an admin console at `/admin/*`, isolated by session audience, with TOTP two-factor, recovery codes, app passwords, and English, Portuguese and Spanish |
+
+## Architecture
+
+Two runtimes, split by what each is good at:
+
+- **`protocols`** (Go) — every protocol listener, the delivery worker, TLS
+  certificate handling, DNS reconciliation, and the webmail's mail API. It owns
+  folder, UID, flag and blob semantics.
+- **`auth`** (TypeScript) — logins, sessions, two-factor, account management and
+  the admin console API.
+- **`web`** (Next.js) — both web surfaces, and the proxy that gives the browser a
+  single origin.
+- **`storage`** (TypeScript) — health only today.
+
+Postgres is the source of truth for everything, including the outbound queue.
+Redis is a cache and dispatch index; losing it loses no mail.
+
+```
+                    ┌──────────┐
+  browser ─────────▶│   web    │──┬─▶ auth      (sessions, accounts, admin)
+                    └──────────┘  └─▶ protocols (mail read/send, DKIM, TLS)
+                                        │
+  internet ──25/587/465/143/993──────────┘
+                                        │
+                              Postgres ─┴─ Redis ─ Rspamd ─ ClamAV
+```
+
+## Deploying
+
+For a real VPS, follow **[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md)**. It covers the
+three things no code can do for you — reverse DNS, the DNSSEC delegation and
+getting port 25 unblocked — and the order the rest has to happen in.
+
+The short version:
+
+```bash
+cp .env.production.example .env      # then replace every CHANGE
+make preflight                       # checks port 25, PTR, RBLs, token scope
+docker compose up -d
+make create-admin EMAIL=you@example.com PASSWORD='...'
+```
+
+`make create-admin` is not optional: a fresh install has no mailbox, and every
+route into the console needs one.
+
+## Running locally
+
+```bash
+cp .env.example .env
+make gen-dev-cert                    # self-signed certificate for the listeners
+docker compose up --build
+make seed                            # a postmaster@example.test fixture
+```
+
+The webmail is on `http://localhost:3000`. The seeded password is
+`dev-password-only` — it is a fixture, not a credential.
+
+## Testing
+
+```bash
+make test        # Go suite, auth and webmail
+make lint        # go vet, tsc, and the language lint
+```
+
+The Go and auth suites include integration tests that need Postgres. They skip
+without `TEST_DATABASE_URL` and run against a real database in CI, which is
+where the security regressions are enforced — wrong password, wrong TOTP code,
+session revocation, surface isolation, cross-mailbox reads.
+
+## Configuration
+
+Every variable is documented inline in
+[`.env.production.example`](.env.production.example). The ones without a safe
+default:
+
+| Variable | |
+| :--- | :--- |
+| `JWT_SECRET` | Signs sessions. At least 32 characters; the service refuses to start below that rather than fall back to a constant. |
+| `LAMBDAMAIL_MASTER_KEY` | Encrypts DKIM keys, TOTP secrets and provider tokens. **Back it up separately from the database** — losing it makes every stored secret unrecoverable. |
+| `POSTGRES_PASSWORD`, `REDIS_PASSWORD` | No defaults; compose fails without them. |
+| `MAIL_DOMAIN`, `PRIMARY_MAIL_HOST`, `PUBLIC_IPV4` | Identity. `PRIMARY_MAIL_HOST` must match the PTR of `PUBLIC_IPV4`. |
+
+## Operating
+
+| | |
+| :--- | :--- |
+| `make preflight` | Port 25 egress, PTR and FCrDNS, RBL listings, Cloudflare token scope, certificate presence |
+| `make create-admin EMAIL=… PASSWORD='…'` | First administrator on a fresh install |
+| `make reset-password EMAIL=… PASSWORD='…'` | Recovers a locked-out account and signs out its sessions |
+| `docker compose logs -f protocols` | Delivery, TLS and DNS reconciliation |
+| `GET /health` on `protocols` | `200` healthy, `200` with `X-LambdaMail-Health: degraded` when a certificate needs attention, `503` when a dependency is down |
+
+[docs/OPERATIONS.md](docs/OPERATIONS.md) covers backups, certificate rotation,
+what the alerts mean and how to read a delivery failure.
+
+## What is not finished
+
+Stated plainly, because a mail server that half works loses mail:
+
+- **Recorded but not acted on.** Vacation responder, signature and the Sieve
+  rules built in the UI are stored, but nothing consumes them yet — enabling a
+  vacation reply does not send one.
+- **Read-only console panels.** The guided domain onboarding checklist, the DNS
+  desired-vs-actual diff and the CSV import report on state rather than driving
+  it.
+- **Not implemented.** JMAP, BIMI, CalDAV/CardDAV, multi-node. Backups are
+  documented but not automated.
+- **DANE** is off unless `TLS_MODE=acme`, and deliberately so: under a
+  Traefik-managed certificate the key changes at every renewal and the published
+  TLSA record stops matching, which permanently rejects mail at every validating
+  MTA.
+
+## Contributing
+
+- All code, comments and identifiers are in English. `make lint` enforces it.
+- Tests come with the change that needs them; the suites above must stay green.
+- Commits reference their issue: `#N: what changed and why`.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

docker-compose.yaml

@@ -123,6 +123,10 @@ services:
     restart: *restart
     logging: *default-logging
     environment:
+      # Drives the Secure attribute on session cookies. Without it the cookies
+      # are issued unsecured, which behind a TLS terminator is a downgrade
+      # waiting to happen.
+      NODE_ENV: ${NODE_ENV:-production}
       PORT: 3001
       DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}?sslmode=disable
       REDIS_URL: redis://:${REDIS_PASSWORD}@redis:6379
@@ -153,6 +157,7 @@ services:
     restart: *restart
     logging: *default-logging
     environment:
+      NODE_ENV: ${NODE_ENV:-production}
       PORT: 3002
       DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}?sslmode=disable
       REDIS_URL: redis://:${REDIS_PASSWORD}@redis:6379
@@ -262,6 +267,7 @@ services:
     restart: *restart
     logging: *default-logging
     environment:
+      NODE_ENV: ${NODE_ENV:-production}
       PORT: 3000
       AUTH_SERVICE_URL: http://auth:3001
       STORAGE_SERVICE_URL: http://storage:3002

docs/DEPLOYMENT.md

@@ -0,0 +1,202 @@
+# Deploying LambdaMail on Coolify
+
+This is the order things have to happen in. Several steps depend on DNS
+propagation or on your provider, so read the whole page before starting — a
+couple of them are worth kicking off early.
+
+## Before anything else
+
+Three prerequisites are not automatable, and without them the deployment will
+come up but mail will not work. Start them now, because two involve waiting on
+somebody else.
+
+### 1. Outbound port 25
+
+Most providers block it by default — AWS, GCP, Azure, Oracle, Vultr, and
+DigitalOcean on newer accounts. Open a support ticket asking for outbound SMTP
+to be unblocked, and say what the host is for.
+
+If the answer is no, LambdaMail can still send through a relay: set `RELAY_HOST`
+and friends, and the relay is added to the published SPF record automatically.
+Receiving is unaffected either way.
+
+### 2. Reverse DNS (PTR)
+
+The PTR for your public IPv4 must resolve to `PRIMARY_MAIL_HOST`, and that name
+must resolve back to the same address. This is called FCrDNS, and Gmail and
+Outlook will refuse or spam-folder mail without it.
+
+**The PTR is set by whoever owns the IP block — your VPS provider, not
+Cloudflare.** It is usually a field in the server's control panel.
+
+```bash
+dig -x 203.0.113.10 +short        # must print mail.example.com.
+dig mail.example.com +short       # must print 203.0.113.10
+```
+
+### 3. DNSSEC — only if you want DANE
+
+Signing the zone is a prerequisite for DANE. It is enabled in the Cloudflare
+dashboard and needs a DS record published at your registrar, which is manual.
+
+Skip this for a first deployment. DANE stays off under `TLS_MODE=traefik`
+anyway, and turning it on without understanding the rollover is the single most
+effective way to make every validating MTA reject your mail permanently.
+
+### IPv6
+
+Either configure it properly — AAAA record, matching PTR, and covered by SPF —
+or leave `PUBLIC_IPV6` empty. Half-configured IPv6 is a common cause of
+rejection, and empty is a safe answer.
+
+---
+
+## Step 1 — Point DNS at the host
+
+Two records must exist before the first deploy, because Coolify's proxy needs
+them to obtain certificates:
+
+| Type | Name | Value |
+| :--- | :--- | :--- |
+| `A` | `mail` | your public IPv4, **DNS only, not proxied** |
+| `A`/`CNAME` | `mta-sts` | the same host (may be proxied) |
+
+The `mail` record must not be behind Cloudflare's proxy. SMTP is not HTTP;
+proxying it breaks the mail ports and hides your real address from the
+certificate challenge.
+
+The rest of the 13 records are created by the service once you give it a
+Cloudflare token, or you can add them by hand later.
+
+## Step 2 — Create the application in Coolify
+
+1. **New Resource → Docker Compose**, pointed at this repository.
+2. Coolify will read `docker-compose.yaml`. Leave the compose file alone; all
+   configuration is environment.
+3. Under **Environment Variables**, paste the contents of
+   `.env.production.example` and replace every value marked `CHANGE`.
+
+Generate each secret separately:
+
+```bash
+openssl rand -base64 36
+```
+
+`LAMBDAMAIL_MASTER_KEY` deserves particular care. It encrypts every DKIM
+private key, TOTP secret and provider token in the database. **Store a copy
+somewhere other than the database backup.** If you lose it, those secrets are
+unrecoverable and you will be reissuing DKIM keys and re-enrolling every user's
+two-factor.
+
+### Ports
+
+Coolify does not map mail ports for you. In the service settings, make sure the
+host publishes:
+
+```
+25, 465, 587      SMTP
+143, 993          IMAP
+110, 995          POP3
+4190              ManageSieve
+```
+
+These are already declared in `docker-compose.yaml`; what matters is that the
+VPS firewall allows them inbound.
+
+### The proxy network
+
+The compose file joins Coolify's proxy network, named `coolify` by default. If
+yours differs, set `COOLIFY_PROXY_NETWORK`.
+
+## Step 3 — Deploy and check the preflight
+
+Deploy, then run the diagnostics from the host:
+
+```bash
+docker compose run --rm --entrypoint /app/lambdamail-protocols protocols preflight
+```
+
+It reports outbound port 25, PTR and FCrDNS, whether your IP is on Spamhaus,
+SpamCop or SORBS, the Cloudflare token's scope, and whether a certificate
+exists for the mail host. Fix anything it flags before going further —
+particularly an RBL listing, which will not fix itself.
+
+## Step 4 — Create the first administrator
+
+Nothing else creates a mailbox, so without this there is no way into the
+console:
+
+```bash
+make create-admin EMAIL=you@example.com PASSWORD='a long passphrase'
+```
+
+This creates the domain if needed, the mailbox as `SUPER_ADMIN`, its standard
+folders, and the four system aliases (`postmaster@`, `abuse@`, `dmarc@`,
+`tlsrpt@`) that the published DNS records point at.
+
+Then:
+
+1. Sign in to the webmail at `https://<WEBMAIL_HOST>/user/login`.
+2. Go to **Settings** and enrol a second factor. The admin console requires
+   one and will refuse to open until it exists.
+3. Sign in to `https://<WEBMAIL_HOST>/admin/login`.
+
+## Step 5 — DNS records
+
+With a Cloudflare token set, the service reconciles the zone on start and every
+six hours after that. Watch it:
+
+```bash
+docker compose logs -f protocols | grep "DNS reconcile"
+```
+
+It is idempotent, and it never deletes records it did not create — anything
+unexpected is reported as a conflict rather than overwritten.
+
+Without a token, add the records by hand. The admin console's Domains page
+lists what is expected.
+
+## Step 6 — Verify
+
+Send mail to a Gmail address and check the headers show `spf=pass`,
+`dkim=pass`, `dmarc=pass`. Then run a round trip through
+[mail-tester.com](https://www.mail-tester.com) and read every point it deducts.
+
+A first deployment will usually be short a few points until the DNS records
+have propagated and the IP has some sending history.
+
+---
+
+## Upgrading
+
+Images are published to GHCR for `linux/amd64` and `linux/arm64` on every merge
+to `main`.
+
+```bash
+docker compose pull
+docker compose up -d
+```
+
+Migrations run automatically on start, before the services that depend on them.
+To pin a version, set `IMAGE_TAG` to a release version or a `sha-` tag rather
+than `latest`.
+
+## Troubleshooting
+
+**The web UI is up but nothing can log in.** The auth service refuses to start
+without a `JWT_SECRET` of at least 32 characters. `docker compose logs auth`
+will say so.
+
+**Mail is accepted but never arrives.** Check `docker compose logs -f protocols`
+for the delivery worker. A `4.7.5` deferral means the destination published a
+TLS policy that could not be satisfied — that is deliberate; the message is
+retried rather than sent in the clear.
+
+**`/health` returns `X-LambdaMail-Health: degraded`.** The stack is running but
+a certificate needs attention: usually no certificate exists for
+`PRIMARY_MAIL_HOST` because no router was declared for it, or one is expiring
+within 24 hours. The body says which.
+
+**Everything is fine but Gmail still spam-folders you.** That is reputation, not
+configuration. A new IP has none. Send low volumes to engaged recipients first
+and watch the DMARC reports in the admin console.

docs/OPERATIONS.md

@@ -0,0 +1,165 @@
+# Operating LambdaMail
+
+What to watch, what the alerts mean, and how to recover. For the initial
+install see [DEPLOYMENT.md](DEPLOYMENT.md).
+
+## Health
+
+`GET /health` on the `protocols` service answers three ways, and the difference
+matters:
+
+| Response | Meaning |
+| :--- | :--- |
+| `200`, `X-LambdaMail-Health: ok` | Healthy |
+| `200`, `X-LambdaMail-Health: degraded` | Running, but something needs an operator. The body says what. |
+| `503` | A dependency is down — usually Postgres. |
+
+Degraded is deliberately not a failure: restarting cannot fix a certificate the
+proxy never issued, and a restart loop would only hide it.
+
+## Certificates
+
+Under `TLS_MODE=traefik` the mail listeners read the certificate Coolify's proxy
+manages. Two failure modes are worth understanding.
+
+**No certificate for the mail host.** The proxy only holds certificates for
+hosts it routes, so `mail.example.com` needs a router even though nothing is
+served over HTTP there. `docker-compose.yaml` declares one; if you removed it,
+the listeners fall back to a self-signed certificate and `/health` reports
+degraded.
+
+**A certificate that stopped renewing.** The watcher re-reads the store every
+60 seconds and reports:
+
+- `WARNING` when the soonest expiry is under 7 days
+- `CRITICAL` under 24 hours, or when it has already expired
+
+Watch for these in `docker compose logs protocols`. The admin console's TLS
+panel shows the same state, including when the watcher last managed to read the
+store — a timestamp that stops advancing means it has gone blind, which is the
+failure that shows up 90 days later as an expired certificate.
+
+## Backups
+
+Two things must be backed up, and **they must not live in the same place**:
+
+1. **The database.** `pg_dump` covers mailboxes, messages metadata, DKIM keys
+   (encrypted), sessions and the queue.
+2. **`LAMBDAMAIL_MASTER_KEY`.** Every DKIM private key, TOTP secret and provider
+   token in that dump is encrypted with it. A backup you cannot decrypt is not a
+   backup.
+
+Message bodies live in the `mail_storage` volume (or S3, with
+`STORAGE_DRIVER=s3`). The database references them by hash; restoring one
+without the other leaves rows pointing at blobs that are not there.
+
+```bash
+# Database
+docker compose exec -T db pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > lambdamail-$(date +%F).sql.gz
+
+# Message bodies
+docker run --rm -v lambdamail_mail_storage:/data -v "$PWD:/backup" \
+  alpine tar czf /backup/mail-$(date +%F).tar.gz -C /data .
+```
+
+Restoring has not been rehearsed in this repository. Before relying on it,
+restore into a scratch environment and confirm a message opens — an untested
+restore is a hypothesis.
+
+## Accounts
+
+```bash
+make create-admin EMAIL=… PASSWORD='…'     # first admin, or --force for another
+make reset-password EMAIL=… PASSWORD='…'   # recovers a lockout, ends its sessions
+```
+
+An account locks itself for 15 minutes after 5 failed passwords. That is the
+lockout working, not a fault; `reset-password` clears it.
+
+If somebody loses their second factor and their recovery codes, there is no
+self-service path today. Clear the enrolment directly and have them enrol again:
+
+```sql
+DELETE FROM mfa_totp WHERE mailbox_id = (SELECT id FROM mailboxes WHERE email_address = 'them@example.com');
+```
+
+## Delivery
+
+The outbound queue lives in Postgres, so it survives a restart of anything. The
+retry ladder is 5 min, 15 min, 1 h, then 2/4/8/12 h with jitter, giving up after
+5 days with a bounce. A delay notice goes out at the fourth attempt.
+
+```sql
+-- What is stuck and why
+SELECT destination_domain, status, attempt, last_smtp_code, left(last_error, 80)
+  FROM outbound_jobs WHERE status IN ('DEFERRED','BOUNCED')
+ ORDER BY created_at DESC LIMIT 20;
+```
+
+Reading the failures:
+
+- **`4.7.5` with a TLS message** — the destination published a DANE or MTA-STS
+  policy that could not be satisfied. The message is deferred rather than sent
+  in the clear, which is the point. If it persists, the destination's
+  certificate is broken, not yours.
+- **`5.1.1`** — no such recipient. Permanent, bounced immediately.
+- **A dial or connection error with no code** — the destination is unreachable;
+  it will retry.
+
+The admin console's Queue page shows the same data with retry and cancel.
+
+## DNS drift
+
+With a Cloudflare token, the zone is reconciled on start and every six hours.
+It is idempotent, and records it did not create are reported as conflicts
+rather than overwritten.
+
+```bash
+docker compose logs protocols | grep -E "DNS reconcile|DNS conflict"
+```
+
+`PARTIAL` or `DRIFT` in that output means the desired state and the zone
+disagree — most often because a record was edited by hand.
+
+## Spam filtering
+
+Rspamd holds its state in Redis and its thresholds in `system_config`, editable
+from the admin console. The defaults reject at 15, tag at 6 and greylist at 4.
+
+A message the filter routes to `Junk` is filed there; one over the reject
+threshold is refused during the SMTP session, so the sender is told. Nothing is
+accepted and silently discarded — a silent discard loses legitimate mail with
+nobody aware of it.
+
+## Rotating a DKIM key
+
+From the admin console, or:
+
+```bash
+curl -X POST https://<host>/api/v1/admin/dkim/rotate \
+  -H 'Content-Type: application/json' -b 'lm_admin_session=…' \
+  -d '{"domain":"example.com","selector":"lmail2","algorithm":"rsa2048"}'
+```
+
+The previous key is retired with a 7-day overlap rather than deleted: mail
+already sent under it is still being verified by receivers who may take a while
+to stop resolving the old selector. **Publish the new selector's TXT record
+before the overlap ends**, or signatures start failing when the old key goes.
+
+## Rotating the master key
+
+There is no automated path today. It requires decrypting every sealed secret
+with the old key and resealing with the new one, and getting it wrong destroys
+them. Plan a maintenance window and rehearse against a copy first.
+
+## What is not automated
+
+Being explicit so nothing is assumed:
+
+- Backups are documented, not scheduled. Set up a cron job.
+- Restores are not rehearsed.
+- There is no metrics endpoint or alerting yet; the signals above are log lines
+  and the health endpoint.
+- Master key rotation is manual.
+- Log retention is whatever Docker's json-file driver is configured for
+  (10 MB × 3 per service, in `docker-compose.yaml`).

services/storage/package.json

@@ -4,7 +4,7 @@
   "private": true,
   "type": "module",
   "scripts": {
-    "build": "tsc -p tsconfig.json",
+    "build": "tsc -p tsconfig.build.json",
     "start": "node dist/server.js",
     "test": "vitest run",
     "lint": "tsc --noEmit"

services/storage/tsconfig.build.json

@@ -0,0 +1,7 @@
+{
+  // The production build excludes tests: they were being compiled into dist
+  // and shipped inside the image. Type checking still covers them through
+  // tsconfig.json, which "npm run lint" uses.
+  "extends": "./tsconfig.json",
+  "exclude": ["node_modules", "dist", "**/*.test.ts"]
+}